Replacing Dead Nodes in Cassandra 4.x/5.x with replace_address

When a Cassandra node dies for good — a failed disk controller, a dead motherboard, a terminated cloud instance whose local storage is gone — the deployment is left with a token range that no longer has its full replica set. This guide is the operational reference for bringing that range back to health by standing up a replacement node that adopts the dead node’s exact token ownership rather than bootstrapping fresh tokens. It sits under Node Lifecycle Automation, the parent guide for the full set of add, remove, replace, and restart operations; read that first if you are building end-to-end automation across all four. Here the focus is narrow and deep: detecting a truly dead node, using the -Dcassandra.replace_address_first_boot JVM boot option, understanding why the replacement keeps the dead node’s tokens, streaming from surviving replicas, choosing replace over nodetool removenode or assassinate, and running repair afterward to close any consistency gaps. Everything is validated against Apache Cassandra 4.0, 4.1, and 5.0, with version drift called out inline.

Concept: why replace, and why it keeps the dead node’s tokens

A running Cassandra cluster distributes data by hashing partition keys onto a token ring, and every node owns a set of token ranges determined by its virtual nodes. When you bootstrap a brand-new node, the deployment allocates it a fresh set of tokens and streams the corresponding slices of data from existing replicas, shrinking everyone else’s ownership. That is exactly the wrong behaviour for a replacement: the failed node’s ranges are already under-replicated, and adding new tokens elsewhere does nothing to restore them. It also doubles the streaming work, because the ring rebalances twice — once when you would remove the dead node and again when the new one bootstraps.

Replacement avoids that churn. Booting with -Dcassandra.replace_address_first_boot=<dead_ip> tells the joining node to assume the identity of the dead node in the ring: it reads the dead node’s token assignments from gossip state held by surviving peers, claims those exact tokens, and streams the data those tokens map to from the other replicas that still hold it. The ring topology is unchanged — same tokens, same ownership map — so no other node gives up or gains a range. From the deployment’s perspective one host address swapped in for another at the same position on the ring. Because the token distribution mechanics matter here, the underlying model is worth revisiting in data partitioning and the token ring if virtual nodes are unfamiliar.

The replacement streams from surviving replicas of the dead node’s ranges. With a replication factor of 3, each range the dead node owned still has two live copies; the joining node pulls from those to rebuild a third copy. This is where consistency subtleties enter. Any writes that were acknowledged only at the dead replica during a period of degraded consistency, or any hinted handoffs that expired, will not be present on the survivors, so the replacement can start life with small gaps. That is normal and expected — it is precisely why an anti-entropy repair after the join is mandatory, not optional. The replace operation restores availability of the range; repair restores consistency.

One hard rule governs the whole procedure: the node you are replacing must be genuinely, permanently dead. replace_address boots a node that deliberately does not gossip its intent to the target address, and if the “dead” node is merely partitioned and comes back, you will have two hosts claiming the same tokens — a split-brain that corrupts ownership and requires manual surgery to unwind. Confirming death is the first and most important gate, covered next.

Detecting a truly dead node

Cassandra’s failure detector marks a peer as down when it stops receiving gossip heartbeats, but “down” is not the same as “dead”. A node under GC pressure, a saturated network link, or a firewall change can all produce a DN (Down/Normal) marker while the host is perfectly recoverable. Before you commit to replacement, distinguish a transient outage from permanent loss.

Start with the ring view from a healthy node:

# Cassandra 4.x/5.x — status of every node; look for DN in the first column.
nodetool status

Datacenter: dc1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns   Host ID                               Rack
UN  10.0.1.11      412.6 GiB  256     ?      3a1e...c9                             rack1
UN  10.0.1.12      408.1 GiB  256     ?      7b2f...4a                             rack1
DN  10.0.1.13      401.9 GiB  256     ?      d4c8...11                             rack2

A DN line is your candidate. Capture the dead node’s Address and Host ID now — you need both later. Then confirm the node is unreachable from multiple healthy peers, not just one, so you are not chasing a one-sided network partition:

# Run from two or more surviving nodes.
nodetool gossipinfo | grep -A6 10.0.1.13

The STATUS and LOAD lines under the dead endpoint, plus the heartbeat generation, tell you whether gossip still considers the host alive anywhere. If every survivor agrees the node is down and the underlying host is confirmed lost — the disk is unrecoverable, the instance is terminated, the hardware is condemned — you have a true DN and may replace it. The failure-detection mechanics behind these markers are detailed in node gossip and failure detection protocols; consult it if a node flaps between UN and DN rather than staying reliably down, because a flapping node is a candidate for a restart, not a replace.

Configuration reference

Replacement is driven almost entirely by a JVM boot flag rather than cassandra.yaml keys, but several settings around it govern whether the join succeeds and how fast it streams. The table below covers the parameters that matter for a replace operation.

The JVM option is the operative control. In Cassandra 4.x and 5.x it is set either as a one-off -D argument at process start or as a persistent line in jvm.options:

# jvm.options (or jvm-server.options on some packages) — Cassandra 4.x/5.x
# Add ONE line for the first boot of the replacement, then remove it after join.
-Dcassandra.replace_address_first_boot=10.0.1.13

# cassandra.yaml on the replacement — must mirror the dead node.
auto_bootstrap: true
num_tokens: 16
# Identity fields (listen_address / rpc_address) should use the NEW host's own
# IP; only the replace_address flag references the dead node's address.

Prefer replace_address_first_boot over the older replace_address. The _first_boot variant is consumed exactly once: after the node successfully joins, Cassandra records that the replace completed and ignores the flag on every subsequent restart even if the line is still present. The bare replace_address has no such guard — if you leave it set, the node tries to replace again on its next restart and fails because the address is now itself. Using _first_boot removes an entire class of “the node won’t restart” incidents.

Step-by-step replacement procedure

Run this on the new host that will take the dead node’s place. It must be reachable on the deployment network, run the identical Cassandra version, and start with empty data, commitlog, and saved-caches directories.

1. Confirm the target is truly dead and capture its identity. From a healthy node, verify the DN marker across peers and record the dead IP and Host ID.

   Copy

   nodetool status | grep DN
   nodetool gossipinfo | grep -B1 -A6 10.0.1.13

   Expected: a single DN line for 10.0.1.13 and consistent down-state across survivors. If the node shows UN anywhere, stop — it is not dead.

2. Verify surviving replica count for the dead ranges. For every keyspace the dead node held, at least one other replica must be UN, or the replacement has nothing to stream from.

   Copy

   nodetool status <keyspace>   # per-keyspace ownership; confirm ≥1 UN peer per range

3. Prepare the replacement host. Install the matching Cassandra version, copy the deployment’s cassandra.yaml with this host’s own listen_address/rpc_address, and ensure num_tokens matches the dead node. Clear any pre-existing state:

   Copy

   sudo systemctl stop cassandra
   sudo rm -rf /var/lib/cassandra/data/* \
               /var/lib/cassandra/commitlog/* \
               /var/lib/cassandra/saved_caches/*

4. Set the replace flag. Add the boot option to jvm.options on the replacement host:

   Copy

   echo '-Dcassandra.replace_address_first_boot=10.0.1.13' | sudo tee -a /etc/cassandra/jvm.options

5. Start Cassandra and watch it join. The node enters UJ (Up/Joining), inherits the tokens, and streams. Do not restart it mid-join.

   Copy

   sudo systemctl start cassandra
   tail -f /var/log/cassandra/system.log | grep -Ei "replac|bootstrap|stream|JOINING"

   Expected log progression: a line confirming it will replace /10.0.1.13, followed by streaming plans and per-range receipt messages.

6. Confirm the join completed. From a healthy node, the new host should now appear UN at the dead node’s token position, and the old DN entry should be gone (the Host ID transfers to the new address).

   Copy

   nodetool status | grep 10.0.1.

7. Remove the flag. Delete the replace_address_first_boot line from jvm.options so future restarts are clean. With _first_boot this is belt-and-braces, but keep the file tidy.

8. Run repair to close consistency gaps. The replacement holds whatever the survivors had; reconcile any missing writes with a full repair scoped to the local datacenter.

   Copy

   nodetool repair --full -pr

Verification & observability

A replacement is not “done” when the node shows UN; it is done when the range is fully replicated and consistent. Verify both. Ownership and load first:

# The new address should carry a Load comparable to its rack peers.
nodetool status
nodetool ring | head -20

The replacement’s Load should climb during the join and settle near its rack peers’ values once streaming finishes; a load far below peers signals an incomplete stream. Confirm no streaming sessions are stuck:

nodetool netstats | grep -iE "receiving|stream|repair"

During the join you should see receiving sessions with progressing byte counts; after it, they should be gone. On Cassandra 5.0 you can also read streaming and task state from the virtual tables without JMX:

SELECT peer, keyspace_name, table_name, progress_percentage
FROM system_views.streaming;

Finally, prove consistency was restored rather than merely assumed. After the post-replace repair completes, spot-check a few partitions the dead node owned at LOCAL_QUORUM and confirm no DigestMismatchException churn in the logs. Because the replacement briefly served reads with slightly stale data before repair, watch application read latency and error rates through the join window; a spike in ReadTimeoutException during streaming is expected and should subside as the node settles.

Failure modes & rollback

The “dead” node was only partitioned. If the target rejoins gossip while the replacement is booting or after it has joined, two hosts claim the same tokens and the ring is corrupted. Detection: both addresses appear at the same token position in nodetool ring, or the survivors log Nodes ... and ... have the same token. Rollback: stop the replacement immediately, and if the original has genuinely recovered, discard the replacement and let the original resume; if the original is truly dead, stop it hard (it should never come back), then restart the replacement. Never let both run. The safeguard is the detection gate in step 1 — confirm death across multiple peers before you start.

Streaming fails partway with StreamException. A network blip, a survivor going down mid-join, or throttling misconfiguration can abort the rebuild, leaving the replacement joined but under-filled. Detection: StreamException or Stream failed in system.log, and a Load well below rack peers in nodetool status. Rollback: the safest recovery is to wipe and re-run — stop Cassandra, clear the data directories again, re-add the replace_address_first_boot flag, and restart so streaming replans from scratch. A partially streamed replacement should not be left in service; repair alone will not reliably backfill a large aborted stream.

Chose replace when removenode or assassinate was the right tool. Replace is for restoring a dead node’s ranges with a like-for-like host. If you are permanently shrinking the deployment — not replacing hardware — you do not want a replacement at all. Use nodetool removenode <host-id> to have survivors re-stream the dead node’s ranges among themselves and rebalance the ring downward; reserve nodetool assassinate <ip> for a last resort when removenode hangs and gossip state must be forcibly evicted, accepting that it skips streaming and therefore requires a follow-up repair. Detection of the wrong choice: you replaced a node but actually wanted fewer nodes, leaving an over-provisioned cluster. Rollback: decommission the surplus node cleanly rather than assassinating it.

FAQ

What is the difference between replace_address and replace_address_first_boot?

Both boot a node as a replacement that inherits a dead node’s tokens, but replace_address_first_boot is consumed exactly once: Cassandra marks the replace complete after a successful join and ignores the flag on every later restart, even if the line stays in jvm.options. The bare replace_address has no such guard, so leaving it set makes the node attempt another replace on its next restart and fail. Always prefer replace_address_first_boot in Cassandra 4.x and 5.x.

Why does the replacement keep the dead node’s tokens instead of getting new ones?

Because the dead node’s ranges are already under-replicated, and adding fresh tokens elsewhere would not restore them — it would only rebalance the ring twice and double the streaming. By reading the dead node’s token assignments from gossip and claiming them exactly, the replacement rebuilds the missing replica in place, leaving every other node’s ownership untouched. That is the whole point of replace versus a plain bootstrap.

When should I use removenode or assassinate instead of replace?

Use replace when you are swapping in like-for-like hardware and want the dead node’s ranges restored. Use nodetool removenode <host-id> when you are permanently shrinking the deployment and want survivors to re-stream and rebalance among themselves. Reserve nodetool assassinate <ip> for when removenode hangs and you must forcibly evict a ghost entry from gossip; because it skips streaming, always follow assassinate with a full repair.

Do I still need to run repair after a successful replace?

Yes, always. The replacement streams from surviving replicas, so any write acknowledged only at the dead node during degraded consistency, or any hint that expired, will be missing. Replace restores availability of the range; a full repair restores consistency. Run nodetool repair --full -pr on the replacement once it reaches UN.

Can I replace a node that is only down temporarily?

No. replace_address assumes the target is permanently gone. If the node is merely partitioned or restarting and it rejoins, two hosts will claim the same tokens and corrupt ownership. Confirm the host is truly and permanently dead across multiple peers first; if it merely flaps between UN and DN, treat it as a restart candidate, not a replacement.

Related

• Node Lifecycle Automation — the parent guide spanning bootstrap, decommission, replace, and rolling-restart operations end to end.
• Replacing a dead node with replace_address — the concrete, copy-paste procedure for a same-datacenter replacement via inherited tokens.
• Rebuilding a replacement node from another datacenter — when to stream a fresh node’s data from a specific DC with nodetool rebuild instead.
• Read repair vs anti-entropy repair — why a full repair after replace is mandatory to close the gaps streaming leaves behind.
• Node gossip and failure detection protocols — how DN markers are produced and how to tell a dead node from a flapping one.
• Data partitioning and the token ring — the vnode ownership model that makes token inheritance possible.