Node Gossip & Failure Detection Protocols in Cassandra 4.x/5.x

Gossip and failure detection are the membership control plane of every Apache Cassandra deployment: they decide which nodes are UP, which are DOWN, and therefore which nodes are eligible to serve reads, accept writes, receive hints, and participate in streaming. A mistuned suspicion threshold or an unmonitored state transition does not fail loudly — it cascades quietly into stalled compactions, repair backlogs, and silent replica divergence. This guide is for DBAs and distributed-systems engineers who need deterministic control over the node lifecycle, failure-detection tuning, and gossip-gated automation on Cassandra 4.x and 5.x. It sits under Cassandra Architecture & Compaction Fundamentals; read that first if you have not yet mapped how the storage engine and cluster fabric interact. Every threshold, command, and code path below is validated against Cassandra 4.0/4.1 and 5.0, with version drift called out inline.

Concept: gossip convergence and the Phi Accrual math

Cassandra’s gossip protocol runs on a fixed one-second cadence. Each round, every node picks a small random set of peers (always including a seed with some probability) and performs a three-way digest exchange that reconciles membership, load, schema version, and heartbeat state. Because the fan-out is randomized and epidemic, a state change propagates to the whole ring in O(log N) rounds — a few seconds even on large clusters. Two monotonic counters guarantee that stale state never overwrites fresh state: generation (a wall-clock stamp set once per process start) and version (incremented on every local state mutation). A peer only accepts an update whose (generation, version) tuple is strictly newer than what it holds.

Each gossip round between two peers is the three-way exchange that disseminates membership state and the heartbeats the failure detector consumes, as shown below.

Failure detection is deliberately not a static timeout. Cassandra uses the Phi Accrual Failure Detector, which converts the arrival history of a peer’s heartbeats into a continuous suspicion level, phi. For a peer whose last heartbeat arrived t_since milliseconds ago, phi is defined as:

phi(t_since) = -log10( P_later(t_since) )

where P_later(t) is the probability, estimated from a sliding window of recent inter-arrival intervals, that the next heartbeat arrives later than t. Intuitively, phi rises the longer a heartbeat is overdue relative to how regular that peer has historically been. A phi of 8 corresponds to a false-positive probability of roughly 10^-8; a phi of 1 means a mislabel probability near 10%. A node is convicted (marked DOWN) the moment phi crosses phi_convict_threshold. Because the estimate is adaptive, a peer on a jittery cross-datacenter link is granted more slack automatically, while a rock-steady LAN peer is convicted quickly when it genuinely stops responding.

The endpoint itself moves through a small state machine — BOOT → NORMAL, with LEAVING/LEFT on decommission, MOVING on token moves, and an orthogonal UP/DOWN liveness flag driven purely by phi. Gossip state gates eligibility for every heavy operation: a peer that is DOWN or LEAVING is excluded from read/write coordination and from streaming, which is exactly why premature or delayed convictions ripple into compaction and repair.

The phi_convict_threshold default of 8 suits stable LANs. Lowering it below 5 provokes premature evictions during ordinary JVM GC pauses; raising it above 12 delays real failure detection, holding stale replicas in the coordination set and stretching the window over which read repair and anti-entropy diverge. Tuning it is therefore a direct lever on the consistency-versus-availability trade-off, not a cosmetic knob.

Gossip state, compaction, and repair interdependencies

Gossip liveness gates compaction and repair eligibility directly. A node marked DOWN or LEAVING is pulled out of streaming, so its share of background merges under the underlying LSM tree mechanics in Cassandra stops making progress and its SSTable count drifts upward. When the node returns and gossip flips it back to NORMAL, it must reconcile its memtables, commit log, and SSTables against the current ring; if that transition is declared prematurely, you risk serving unmerged fragments or tombstones that have slipped past garbage collection.

Repair scheduling has to respect the failure-detection window. Firing nodetool repair while phi_convict_threshold is set too aggressively invites streaming retries that saturate inter-node bandwidth every time a peer flaps. Align repair cadence with the compaction strategy in force: the trade-offs between STCS, LCS, and TWCS determine how aggressively tombstones are purged and therefore how sensitive a table is to a stale replica. Time-window tables tolerate longer gossip convergence because compaction expires whole windows on schedule; leveled tables demand tighter detection to keep overlapping SSTables — and read amplification — from accumulating while a replica is wrongly considered live.

Failure state also decides which reconciliation mechanism runs. The distinction between read repair and anti-entropy repair matters here: blocking read repair fixes a single digest mismatch synchronously at query time, whereas scheduled anti-entropy repair rebuilds whole ranges via Merkle-tree comparison. The probabilistic read_repair_chance and dclocal_read_repair_chance table options were removed in Cassandra 4.0 and replaced by the per-table read_repair option ('BLOCKING', the default, or 'NONE'); blocking read repair still reconciles replicas at query time on a digest mismatch, but only among the replicas gossip currently considers UP. If a convicted-but-actually-healthy node is excluded, its correct value never enters the comparison — one more reason accurate detection precedes durable consistency.

Configuration reference

The keys below are the failure-detection and recovery surface in cassandra.yaml, plus the JVM startup property that governs how long a joining node listens before trusting the ring. Cassandra 5.0 accepts human-readable duration and data-size units; 4.x uses the older _in_ms / _in_kb suffixes, noted inline.

A minimal, production-safe cassandra.yaml fragment for a two-datacenter deployment:

# cassandra.yaml — failure detection & recovery surface (Cassandra 5.0 syntax)
phi_convict_threshold: 8            # raise to 10-12 only on jittery cross-DC links
hinted_handoff_enabled: true
max_hint_window: 3h                 # 4.x: max_hint_window_in_ms: 10800000
hinted_handoff_throttle: 1024KiB    # 4.x: hinted_handoff_throttle_in_kb: 1024
endpoint_snitch: GossipingPropertyFileSnitch
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.0.0.1,10.0.1.1"  # one stable seed per datacenter, not every node

There is deliberately no nodetool getter/setter for phi_convict_threshold: change it persistently in cassandra.yaml (requires a restart) or adjust it at runtime by invoking setPhiConvictThreshold on the JMX FailureDetector MBean (org.apache.cassandra.net:type=FailureDetector). A JMX change is in-memory only and reverts on restart, so persist any value you intend to keep. In multi-datacenter clusters, keep token ownership consistent so state transitions route cleanly — validate the data partitioning and token ring basics map before touching detection thresholds, because a ring mismatch and a gossip flap look nearly identical in the logs.

Step-by-step: gossip-gated failure response

The goal of a safe node-lifecycle runbook is to react to a suspected failure only after gossip has genuinely stabilized, and to never launch an I/O-heavy operation into an unstable ring. The following procedure gates repair on gossip state and uses only read-only inspection until an explicit action point.

1. Snapshot ring liveness. Confirm every peer’s gossip view before anything else.

   Copy

   nodetool status
   # Expected: every node shows UN (Up/Normal). DN = Down/Normal is a live conviction.
   nodetool gossipinfo | grep -E "^/|STATUS|LOAD"

2. Read per-endpoint phi. Phi values are not exposed through CQL; source them from nodetool failuredetector or the JMX FailureDetector MBean.

   Copy

   nodetool failuredetector
   # Rows: "/10.0.0.7   8.4   true"  ->  phi above phi_convict_threshold means convicted.

3. Correlate with load, never token drift. Before treating high phi as a failure, rule out a bootstrap or move: a node in BOOT, JOINING, or MOVING is expected to look unstable. Check nodetool describecluster for a single schema version and nodetool netstats for active streams.

4. Gate the response in code. Only when gossip is stable do you allow a repair to fire. The daemon below polls gossip state, requires a real stability interval (phi is a dimensionless suspicion score, not a time window, so gate on continuous seconds of stability), and defers otherwise.

   Copy

   #!/usr/bin/env python3
   # requirements: Python 3.10+, Apache Cassandra 4.x/5.x nodetool on PATH.
   # Read-only until the explicit repair action; safe to run on a cron.
   import subprocess
   import time
   import sys
   from typing import Dict
   
   GOSSIP_STABLE_STATES: frozenset[str] = frozenset({"NORMAL", "LEAVING"})
   MIN_STABLE_SECONDS: int = 45          # ~45 one-second gossip rounds of quiet
   COOLDOWN_MINUTES: int = 30
   
   def run_nodetool(*args: str) -> str:
       """Execute nodetool with a hard timeout and explicit error surfacing."""
       try:
           result = subprocess.run(
               ["nodetool", *args],
               capture_output=True, text=True, timeout=15,
           )
       except subprocess.TimeoutExpired as exc:
           raise RuntimeError("nodetool timed out") from exc
       if result.returncode != 0:
           raise RuntimeError(f"nodetool {' '.join(args)} failed: {result.stderr.strip()}")
       return result.stdout
   
   def parse_gossip_state() -> Dict[str, str]:
       """Map endpoint -> STATUS token parsed from `nodetool gossipinfo`."""
       states: Dict[str, str] = {}
       current: str | None = None
       for line in run_nodetool("gossipinfo").splitlines():
           line = line.strip()
           if line.startswith("/"):
               # Endpoint header "/10.0.0.1"; strip "/" and any ":port".
               current = line[1:].split(":")[0]
               states[current] = "UNKNOWN"
           elif line.startswith("STATUS") and current:
               # "STATUS:<version>:NORMAL,<token>" -> take NORMAL.
               states[current] = line.split(":")[-1].split(",")[0].strip()
       return states
   
   def gossip_is_stable() -> bool:
       """Require MIN_STABLE_SECONDS of continuously healthy gossip state."""
       deadline = time.monotonic() + MIN_STABLE_SECONDS
       while time.monotonic() < deadline:
           unstable = [ip for ip, s in parse_gossip_state().items()
                       if s not in GOSSIP_STABLE_STATES]
           if unstable:
               print(f"[DEFER] unstable endpoints: {unstable}")
               return False
           time.sleep(5)
       return True
   
   def gated_repair() -> None:
       if not gossip_is_stable():
           print(f"[COOLDOWN] sleeping {COOLDOWN_MINUTES}m before re-evaluation")
           time.sleep(COOLDOWN_MINUTES * 60)
           return
       print("[OK] gossip stable; running incremental primary-range repair")
       # 4.x/5.x: incremental repair is the default; -pr limits to this node's
       # primary range. Pass -full to force a full repair when needed.
       run_nodetool("repair", "-pr")
   
   if __name__ == "__main__":
       try:
           gated_repair()
       except RuntimeError as exc:
           print(f"[FATAL] {exc}", file=sys.stderr)
           sys.exit(1)

5. Escalate manual eviction with care. If a peer is genuinely dead and stuck (a LEFT/removed ghost that will not clear), only then reach for nodetool assassinate <ip> — and always after confirming token-ring consistency with nodetool describecluster. Never assassinate during an active flap; you can partition data by evicting a node that is about to recover.

Verification & observability

Confirm that a state transition landed the way you intended, and that gossip is quiet, using these surfaces:

• Ring convergence. nodetool status should show every node UN; a returning node briefly shows UJ (Up/Joining) before UN. Cross-check nodetool gossipinfo on two different nodes — their views must agree.
• Failure-detector health. nodetool failuredetector phi values should sit near 0–2 for healthy peers under steady state; a peer parked above phi_convict_threshold is convicted.
• Gossip thread pool. On 4.x/5.x, query the virtual table SELECT name, active_tasks, pending_tasks, blocked_tasks FROM system_views.thread_pools WHERE name = 'GossipStage'; — sustained pending_tasks on GossipStage means gossip is queueing behind CPU or I/O and phi will inflate as a symptom, not a cause.
• Log patterns. Grep system.log for the transition trail: Node /10.0.0.7 state jump to NORMAL, InetAddress /10.0.0.7 is now DOWN / is now UP, and Convicting /10.0.0.7 with accrued failure detector value. Repeated up/down pairs for one endpoint are the flapping signature.

Backlog on GossipStage and elevated phi frequently trace back to compaction I/O starvation; when a returning node struggles, correlate against its queue depth using compaction backlog analysis and alerting before touching detection thresholds.

Failure modes & rollback

Flapping under GC pauses (phi_convict_threshold too low). A stop-the-world pause longer than the effective conviction window makes a healthy node miss heartbeats and get marked DOWN, only to bounce back UP seconds later. Each flap triggers redundant streaming and hint replay. Detection: alternating is now DOWN / is now UP lines for one endpoint in system.log, correlated with long GC entries in gc.log. Rollback: raise phi_convict_threshold to 10–12 (persist in cassandra.yaml or set via the FailureDetector MBean), then attack the root cause by tuning heap and pause targets; revert the threshold to 8 once pauses are controlled so you do not permanently blunt detection.

Zombie / stuck endpoint after assassinate. Forcing nodetool assassinate on a node that other peers still gossip as alive can leave a ghost entry that reappears after gossip re-propagates, or a token range with no clear owner. Detection: nodetool status disagrees between nodes, or a removed IP keeps resurfacing in nodetool gossipinfo. Rollback: stop assassinating; let gossip converge for several minutes, confirm a single schema version with nodetool describecluster, and if the ghost persists, run nodetool removenode <host-id> (using the host ID, not the IP) so the ring owner is reassigned cleanly.

Seed-partition split view (schema/ring disagreement). If seeds become unreachable from one datacenter, that side can converge on a divergent membership or schema view, so reads at QUORUM/LOCAL_QUORUM start failing even though local nodes are healthy. Detection: nodetool describecluster reports multiple schema versions; nodetool gossipinfo shows one DC missing peers the other DC still sees. Rollback: restore seed reachability first (never “fix” this by assassinating peers), then let each side re-gossip; if schema stays split, run nodetool resetlocalschema on the minority nodes to pull the majority version.

FAQ

What phi_convict_threshold should I actually run?

Keep the default 8 on stable LAN clusters — it corresponds to a roughly 10^-8 false-positive probability and rarely misfires. Raise it to 10–12 only when you have measured genuine cross-datacenter jitter or unavoidable GC pauses that are causing flaps. Going below 5 is almost never correct: you trade a few seconds of detection latency for a flood of false convictions, redundant streaming, and repair thrash.

Why is a healthy node being marked DOWN and then UP repeatedly?

That flapping pattern is the classic signature of a conviction window shorter than your worst-case JVM pause. The Phi Accrual detector treats a long stop-the-world GC exactly like a dead node, because from the peer’s point of view heartbeats simply stopped arriving. Correlate is now DOWN/is now UP log lines with gc.log; fix the pauses and, as a stopgap, raise phi_convict_threshold.

Can I read or change phi_convict_threshold with nodetool?

No — there is no nodetool getter or setter for it. Set it persistently in cassandra.yaml (which needs a restart), or change it at runtime by invoking setPhiConvictThreshold on the JMX FailureDetector MBean at org.apache.cassandra.net:type=FailureDetector. Remember the JMX change is in-memory and is lost on restart, so mirror it into cassandra.yaml if it should survive.

How long should I wait after a node rejoins before running repair?

Gate on gossip stability, not a fixed guess. Wait until nodetool status shows the returning node as UN on every peer and its GossipStage pending tasks have drained — in practice a continuous quiet window of 30–60 seconds (dozens of one-second gossip rounds). Launching nodetool repair into an unconverged ring causes streaming retries that saturate inter-node bandwidth and can re-trigger flapping.

Does gossip run independently per datacenter in a multi-DC cluster?

Gossip is a single logical membership plane, but with GossipingPropertyFileSnitch each node knows its own DC/rack and cross-DC heartbeats traverse higher-latency links, which naturally inflates phi for remote peers. The Phi Accrual estimate adapts to that jitter, but you should still keep seeds reachable across DCs and monitor cross-DC latency separately, because a link problem and a genuine node failure produce very similar phi curves.

Related

• Cassandra Architecture & Compaction Fundamentals — the parent guide mapping how the storage engine and cluster fabric interact end to end.
• Diagnosing gossip protocol failures in multi-DC clusters — deterministic triage workflows for cross-datacenter false positives and clock skew.
• Read repair vs anti-entropy repair — which reconciliation mechanism runs, and why gossip liveness decides it.
• Data partitioning & token ring basics — token ownership that state transitions and streaming depend on.
• Compaction backlog analysis & alerting — spotting the I/O starvation that inflates phi and stalls a recovering node.