Replacing a Dead Cassandra Node with replace_address (4.x/5.x)

This is the concrete, run-it-now procedure for replacing one permanently dead node with a fresh host that inherits the dead node’s tokens, using the -Dcassandra.replace_address_first_boot boot option. The scenario is a same-datacenter swap: an instance in dc1 lost its storage, its replicas still live on other dc1 nodes, and you want a new host to take over the exact ring position and stream the missing copy back from those local replicas. It is the applied companion to the broader replacing dead nodes guide — read that for the concept of token inheritance and the decision between replace, removenode, and assassinate; this page is the checklist and code. Prerequisites: the replacement runs the same Cassandra version (down to the minor release) as the rest of the ring, uses the same listen/broadcast configuration pattern as its peers with its own IP, and starts with empty data, commitlog, and saved-caches directories. Everything below is validated against Cassandra 4.0, 4.1, and 5.0.

The reason this same-datacenter path is preferable to bootstrapping a brand-new node is that the replacement adopts the dead node’s exact token ranges from gossip and streams only the copies that are missing, from replicas that sit on the same low-latency network. Nothing else on the ring gives up or gains ownership, so there is no double rebalance and no cross-region traffic bill. The trade-off is strictness: because the replacement deliberately does not gossip to the dead address, the target must be genuinely and permanently gone, and every gate below exists to prove that before a single byte moves.

Pre-conditions & safety gates

These checks are read-only and run from a surviving node. Do not touch the replacement host until every gate passes. Treat them as a hard sequence — if any one fails, the whole operation stops, because a replacement started against a live or ambiguous node is far harder to unwind than one you never began.

# 1. Confirm the target is DN, not merely down for a moment. Run on 2+ survivors.
nodetool status | awk '$1=="DN"'

Safety Check: The same address must show DN from multiple healthy nodes. A node that shows UN anywhere is not dead — abort. Expected Output:

DN  10.4.2.37  388.2 GiB  16  ?  b91c7f04-2d55-4a9e-9c1a-77e3f0a1d820  rack3

# 2. Capture the dead node's identity for the record and for removenode fallback.
nodetool gossipinfo | grep -B1 -A8 10.4.2.37

Safety Check: Confirm the STATUS line reads a shutdown/left state and the heartbeat generation is stale (not advancing). Record the HOST_ID. Expected Output: a gossip block for /10.4.2.37 whose heartbeat does not change across two runs a minute apart.

# 3. Verify surviving replica count for every keyspace the node held.
for ks in $(cqlsh -e "DESC KEYSPACES" 10.4.2.11 | tr ' ' '\n' | grep -v '^$'); do
  echo "== $ks =="; nodetool status "$ks" | awk '$1=="UN"' | wc -l
done

Safety Check: Each application keyspace must report at least replication_factor − 1 UN replicas so there is a source to stream from. With RF=3 that means ≥2 live nodes per range. Expected Output: a count per keyspace; every application keyspace ≥ 2.

If any gate fails — the node flaps to UN, gossip still shows it alive, or a range has no surviving replica — stop. A flapping node belongs in a restart workflow, and a range with zero survivors is a data-loss incident that replacement cannot fix. When gossip markers are ambiguous, the detection logic in node gossip and failure detection protocols resolves whether the node is genuinely gone.

Implementation

The replacement is prepared on the new host. The script below is idempotent: it refuses to run against a non-empty data directory, refuses to proceed if the target is not DN, sets the flag, starts the process, waits for the join, then strips the flag. Set the two addresses at the top and run it as root on the replacement host.

#!/usr/bin/env bash
# replace_node.sh — same-DC dead-node replacement for Cassandra 4.x/5.x.
# Run as root ON THE REPLACEMENT HOST. Requires: matching Cassandra version,
# a survivor reachable for status checks, empty data dirs.
set -euo pipefail

DEAD_IP="10.4.2.37"          # the permanently dead node
SURVIVOR="10.4.2.11"         # any UN node, for cluster-state checks
JVM_OPTS="/etc/cassandra/jvm.options"
DATA_DIRS=(/var/lib/cassandra/data /var/lib/cassandra/commitlog /var/lib/cassandra/saved_caches)
FLAG="-Dcassandra.replace_address_first_boot=${DEAD_IP}"

# --- Guard 1: never replace a live node. ---------------------------------
if nodetool -h "$SURVIVOR" status | awk -v ip="$DEAD_IP" '$2==ip {print $1}' | grep -qx UN; then
  echo "ABORT: ${DEAD_IP} is UN (alive). Not a replacement candidate." >&2
  exit 1
fi

# --- Guard 2: idempotency — refuse to clobber an existing dataset. --------
if [ -n "$(find /var/lib/cassandra/data -mindepth 2 -type f 2>/dev/null | head -1)" ]; then
  echo "ABORT: data directory is not empty. This host already holds data." >&2
  exit 1
fi

# --- Guard 3: don't double-add the flag. ----------------------------------
if grep -qF "replace_address" "$JVM_OPTS"; then
  echo "ABORT: a replace_address flag is already present in ${JVM_OPTS}." >&2
  exit 1
fi

echo "Clearing state directories..."
systemctl stop cassandra || true
for d in "${DATA_DIRS[@]}"; do rm -rf "${d:?}/"*; done

echo "Setting first-boot replace flag for ${DEAD_IP}..."
printf '\n%s\n' "$FLAG" >> "$JVM_OPTS"

echo "Starting Cassandra (joining as replacement)..."
systemctl start cassandra

Once the process is up it enters UJ and begins streaming. Monitor the join from a survivor rather than tailing the log by eye; the Python watcher below polls nodetool status until the replacement’s own IP reports UN, with a bounded timeout so it never hangs forever.

#!/usr/bin/env python3
# requirements: stdlib only (subprocess). Python 3.10+.
# Poll a survivor until the replacement node reaches UN, then signal to
# strip the replace flag and run repair.
import subprocess
import sys
import time

SURVIVOR = "10.4.2.11"      # UN node to query
REPLACEMENT = "10.4.2.38"   # the new host's own IP
TIMEOUT_S = 6 * 60 * 60     # 6h ceiling for a large rebuild
POLL_S = 30


def node_state(survivor: str, target_ip: str) -> str | None:
    """Return the Up/Down+state token (e.g. 'UN', 'UJ') for target_ip, or None."""
    try:
        out = subprocess.run(
            ["nodetool", "-h", survivor, "status"],
            capture_output=True, text=True, timeout=20, check=True,
        ).stdout
    except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
        print(f"[warn] status query failed: {e}", file=sys.stderr)
        return None
    for line in out.splitlines():
        cols = line.split()
        if len(cols) >= 2 and cols[1] == target_ip:
            return cols[0]
    return None


def wait_for_join() -> bool:
    deadline = time.monotonic() + TIMEOUT_S
    while time.monotonic() < deadline:
        state = node_state(SURVIVOR, REPLACEMENT)
        print(f"[info] {REPLACEMENT} state = {state or 'not yet in ring'}")
        if state == "UN":
            return True
        if state == "DN":
            print("[error] replacement went DN during join — investigate.", file=sys.stderr)
            return False
        time.sleep(POLL_S)
    print("[error] timed out waiting for the replacement to reach UN.", file=sys.stderr)
    return False


if __name__ == "__main__":
    sys.exit(0 if wait_for_join() else 1)

When the watcher exits 0, finish on the replacement host: remove the flag so restarts are clean, then run a scoped full repair to reconcile anything the survivors were missing.

# On the replacement host, after it reports UN.
sed -i '/replace_address_first_boot/d' /etc/cassandra/jvm.options
nodetool repair --full -pr        # primary-range full repair closes the gaps

Removing the flag matters even though _first_boot is self-clearing: leaving stray replace options in jvm.options invites confusion during the next maintenance window and, if someone later swaps the line for the non-first_boot variant, can turn a routine restart into an accidental replace attempt. The repair is the step operators most often skip and most often regret — the streamed copy reflects only what the surviving replicas held at stream time, so any mutation that was acknowledged solely at the dead node before it failed, or any hint that expired during the outage, exists nowhere until repair rebuilds it. Run the repair to completion before the node carries a meaningful share of LOCAL_QUORUM reads.

Verification

Confirm the swap succeeded on three axes: ring position, streamed volume, and consistency. A node that merely shows UN has not necessarily finished — a stalled stream can leave it live but under-filled, and only checking load and streaming state together proves the rebuild actually completed.

# 1. The new IP holds the dead node's slot; the old DN line is gone.
nodetool -h 10.4.2.11 status | grep -E "10.4.2.37|10.4.2.38"

Expected: one UN line for 10.4.2.38 and no line for 10.4.2.37. The Host ID carried over to the new address.

# 2. Load is comparable to rack peers — a sign streaming completed.
nodetool -h 10.4.2.11 status | awk '$1=="UN"{print $2, $3, $4}'

Expected: the replacement’s Load sits within a normal band of its peers, not a fraction of them.

# 3. No streaming sessions remain hung.
nodetool -h 10.4.2.38 netstats | grep -iE "receiving|stream" || echo "no active streams"

Expected: no active streams once the join and repair are done.

Troubleshooting

Cannot replace_address /10.4.2.37 because it doesn't exist in gossip. The replacement booted but the survivors no longer carry any gossip record of the dead node, so there are no tokens to inherit. Root cause: the dead entry was already evicted — often because someone ran nodetool removenode or assassinate on it earlier, or gossip aged it out after a long outage. Fix: you can no longer replace this address; instead bootstrap the new host normally (remove the replace flag entirely and let it take fresh tokens), or if you must restore the exact ranges, bring the node in empty with auto_bootstrap and follow with a full repair. Never re-add the dead entry by hand.
The replacement joins as a NEW node instead of replacing. It reaches UN but nodetool status still shows the old DN line and the ring gained a node rather than swapping one. Root cause: the flag never took effect — a typo in the option name, the line landed in the wrong jvm.options file for the package, or the process was already past its first boot when the flag was added, so _first_boot was ignored. Fix: decommission the wrongly-joined node with nodetool decommission, wipe its data directories, correct the exact flag -Dcassandra.replace_address_first_boot=<dead_ip> in the active jvm.options, and restart from a clean state so the first boot consumes it.
StreamException during the rebuild. The join aborts partway with StreamException: Stream failed in system.log, leaving the replacement under-filled. Root cause: a survivor went down mid-stream, a network partition interrupted the session, or stream throughput was set so high it saturated the link and timed out. Fix: do not leave a partially streamed node in service — stop Cassandra, clear the data directories, lower stream_throughput_outbound (5.0) / stream_throughput_outbound_megabits_per_sec (4.x) to a rate the link sustains, re-add the replace flag, and restart so streaming replans from scratch. A follow-up repair alone will not reliably backfill a large aborted stream.

Replacing dead nodes — the parent guide covering token inheritance and the replace-vs-removenode decision.
Rebuilding a replacement node from another datacenter — the alternative when you stream a fresh node’s data from a different DC with nodetool rebuild.
Read repair vs anti-entropy repair — why the post-replace full repair is required to close streaming gaps.
Node gossip and failure detection protocols — reading DN markers and confirming a node is genuinely dead.

Back to Replacing Dead Nodes