Compaction Strategy Cassandra Ops & Automation

Rebuilding a Cassandra Replacement Node From Another Datacenter (4.x/5.x)

Sometimes the fastest, safest way to fill a fresh node is not to stream from its local peers at all, but to pull its entire dataset from a different datacenter with nodetool rebuild <source_dc>. This is the standard move when you are standing up or replacing a node in a datacenter whose local replicas are thin, degraded, or under heavy write load, while another datacenter holds a healthy full copy. The pattern is deliberate: bring the node into the ring without bootstrapping (auto_bootstrap: false), so it claims its token ranges but stays empty, then run rebuild to stream those ranges from a named source DC on your schedule. It is the cross-datacenter counterpart to the same-DC token inheritance described in replacing dead nodes; read that guide for the replace decision and use this page when the data should come from another region. Prefer rebuild over repair here because rebuild does a clean bulk stream of the full range from one chosen source, whereas repair does a comparison-heavy reconciliation across all replicas — far slower and more I/O-intensive for filling an empty node. Validated against Cassandra 4.0, 4.1, and 5.0.

Pre-conditions & safety gates

Run these before starting the new node. The source DC must be healthy and its name must be exactly right — a wrong DC name is the single most common cause of a rebuild that streams nothing.

# 1. Confirm the exact datacenter names as Cassandra sees them.
nodetool status | grep -i "Datacenter"

Safety Check: Note the precise, case-sensitive DC name you will pass to rebuild (e.g. dc-east, not DC-East). The name must match nodetool status output exactly. Expected Output:

Datacenter: dc-east
Datacenter: dc-west

# 2. Verify the source DC is fully healthy and holds the data.
nodetool status dc-east | awk '$1!="UN" && $1 ~ /^[UD]/'

Safety Check: The command should print nothing — every node in the source DC must be UN. Rebuilding from a DC with down nodes risks streaming an incomplete range. Expected Output: empty (all source nodes UN).

# 3. Confirm the keyspaces actually replicate into BOTH datacenters.
cqlsh -e "SELECT keyspace_name, replication FROM system_schema.keyspaces;" dc-east-node

Safety Check: Every application keyspace must use NetworkTopologyStrategy with a replica count > 0 in the target DC. If the target DC has no replicas defined, rebuild has no ranges to fill — fix replication first with ALTER KEYSPACE. Expected Output: each keyspace shows NetworkTopologyStrategy with entries for both dc-east and dc-west.

If the target DC is new, alter every keyspace to include it and let the schema propagate before the node starts. Because token ownership in the target DC governs which ranges the node will request, the vnode model in data partitioning and the token ring is worth reviewing when the DC is being expanded rather than merely repaired.

There is a real cost decision embedded in these gates. Streaming across datacenters traverses a WAN link that is usually slower, more expensive, and shared with live cross-region replication traffic, so a rebuild that pulls hundreds of gigabytes can degrade the source DC and inflate egress costs if it runs unthrottled at peak. The gates above exist not only to make the rebuild succeed but to make it safe: a confirmed-healthy source, keyspaces that genuinely replicate into the target, and an exact DC name together mean the one long streaming operation you kick off does useful work on the first try rather than failing silently or hammering a busy region.

Implementation

The sequence is: configure the node with auto_bootstrap: false, start it so it joins empty, throttle inter-DC streaming, run rebuild naming the source DC, then repair. The auto_bootstrap: false line is what makes this distinct from an ordinary join — it tells the node to claim tokens without pulling data, so rebuild becomes the sole, controllable streaming step.

# cassandra.yaml on the new/replacement node in the TARGET dc (dc-west).
# Setting auto_bootstrap false lets the node join empty; rebuild fills it.
auto_bootstrap: false
endpoint_snitch: GossipingPropertyFileSnitch
# cassandra-rackdc.properties on the new node — pin it to the target DC/rack.
# dc=dc-west
# rack=rack1

Start the node, confirm it is UN but nearly empty, then set a conservative inter-DC stream rate so the rebuild does not saturate the WAN link between datacenters:

systemctl start cassandra
# It should reach UN quickly because it streamed nothing on join.
nodetool status dc-west | awk '$1=="UN"'

# Throttle inter-DC streaming (megabits/s). Record the old value first.
nodetool getinterdcstreamthroughput
nodetool setinterdcstreamthroughput 200

Now drive the rebuild. The Python wrapper below runs nodetool rebuild <source_dc> as a long-lived job, guards against an empty/blank DC name, streams progress from netstats, and refuses to declare success unless the node’s load actually grew — catching the silent “streamed 0 bytes” failure before you move on.

#!/usr/bin/env python3
# requirements: stdlib only. Python 3.10+.
# Rebuild a fresh node from a named source DC, then verify it actually filled.
import re
import subprocess
import sys
import time

SOURCE_DC = "dc-east"       # must match nodetool status EXACTLY
SELF = "10.7.1.40"          # this node's own IP


def nt(args: list[str], timeout: int = 30) -> tuple[int, str]:
    try:
        r = subprocess.run(["nodetool", *args], capture_output=True,
                           text=True, timeout=timeout, check=True)
        return 0, r.stdout
    except subprocess.CalledProcessError as e:
        return e.returncode, e.stderr
    except subprocess.TimeoutExpired:
        return -1, "timeout"


def load_bytes(ip: str) -> float:
    """Parse this node's Load (GiB) from nodetool status, as bytes."""
    _, out = nt(["status"])
    units = {"KiB": 1024, "MiB": 1024**2, "GiB": 1024**3, "TiB": 1024**4}
    for line in out.splitlines():
        c = line.split()
        if len(c) >= 4 and c[1] == ip:
            m = re.match(r"([\d.]+)", c[2])
            return float(m.group(1)) * units.get(c[3], 1) if m else 0.0
    return 0.0


def run_rebuild() -> bool:
    if not SOURCE_DC.strip():
        print("[error] SOURCE_DC is blank — rebuild would stream nothing.", file=sys.stderr)
        return False

    before = load_bytes(SELF)
    print(f"[info] load before rebuild: {before / 1024**3:.2f} GiB")

    # rebuild blocks until streaming completes; give it a long ceiling.
    rc, out = nt(["rebuild", "--", SOURCE_DC], timeout=12 * 60 * 60)
    if rc != 0:
        print(f"[error] rebuild failed: {out}", file=sys.stderr)
        return False

    after = load_bytes(SELF)
    print(f"[info] load after rebuild: {after / 1024**3:.2f} GiB")

    # Guard against the silent 0-byte rebuild (wrong DC name, no replicas here).
    if after <= before * 1.01:
        print("[error] load did not grow — check the source DC name and "
              "that keyspaces replicate into this DC.", file=sys.stderr)
        return False
    return True


if __name__ == "__main__":
    ok = run_rebuild()
    if ok:
        # Reconcile anything missed, then this node is ready to serve.
        nt(["repair", "--full", "-pr"], timeout=12 * 60 * 60)
    sys.exit(0 if ok else 1)

While the rebuild runs you can watch inbound streams live from another shell; nodetool netstats shows each receiving session and its byte progress. After it finishes, the wrapper runs a primary-range full repair to reconcile writes that landed after the rebuild snapshot began.

Two properties of this flow are worth calling out. First, rebuild is idempotent in the sense that matters operationally — if it aborts partway or you are unsure it finished, running it again re-streams only the ranges the node still needs, so a re-run is safe and cheap relative to a full repair. Second, the load-growth guard in the wrapper is deliberate defence against the failure mode that wastes the most time: a rebuild against a mistyped source DC exits with status zero and streams nothing, and without checking that Load actually increased you can spend an hour believing a node is filled when it is empty. Comparing the byte load before and after turns that silent failure into an immediate, actionable error.

Verification

# 1. Load now matches a healthy same-DC or cross-DC peer.
nodetool status dc-west | awk '$1=="UN"{print $2, $3, $4}'

Expected: the rebuilt node’s Load is in the same band as its peers, not near zero.

# 2. No streaming session is stuck; rebuild has drained.
nodetool netstats | grep -iE "receiving|rebuild|stream" || echo "no active streams"

Expected: no active streams after completion.

# 3. Restore the inter-DC throughput you throttled.
nodetool setinterdcstreamthroughput 0   # 0 = unlimited (default), or your prior value

Expected: command succeeds; confirm with nodetool getinterdcstreamthroughput.

Spot-check consistency by reading a few known partitions at LOCAL_QUORUM against the source DC; because rebuild plus the follow-up repair should have reconciled everything, results must match. If they diverge, the repair did not complete — re-run it before putting the node in the read path. Until you are confident both the rebuild and the repair finished, keep the node out of the application’s coordinator rotation for that DC, or configure the client’s load-balancing policy to avoid it, so no user request is served from a range that is still filling.

Troubleshooting

  • Rebuild returns immediately and streams 0 bytes. The command exits cleanly but the node’s Load never grows. Root cause: almost always a source DC name that does not match nodetool status exactly (case or hyphenation), or the target DC has no replicas because the keyspaces were never altered to replicate into it. Fix: re-check the DC name character-for-character, confirm each keyspace’s NetworkTopologyStrategy includes the target DC with a non-zero count, and re-run nodetool rebuild -- <exact_source_dc>. Rebuild is safely repeatable — running it again only re-streams ranges it still needs.

  • Source-datacenter overload during the stream. The rebuild works but the source DC’s read latency and CPU spike, hurting live traffic there. Root cause: inter-DC stream throughput set too high for the WAN link, so a handful of source nodes pin their disks and NICs feeding the rebuild. Fix: lower it live with nodetool setinterdcstreamthroughput <mbits> — the change applies to the in-flight rebuild — and schedule large rebuilds during the source DC’s off-peak window. Streaming from a DC that is itself under compaction pressure compounds the problem, so confirm the source’s compaction queue is calm first.

  • Inconsistency after rebuild requires a follow-up repair. Reads against the rebuilt node return slightly stale values compared to the source. Root cause: rebuild streams a point-in-time view of each range, so mutations written during and just after the stream are not guaranteed to land, and no repair ran. Fix: run nodetool repair --full -pr on the rebuilt node and let it complete before routing reads to it. Rebuild restores bulk data quickly; only a full anti-entropy repair guarantees the node is consistent with its replicas.

Back to Replacing Dead Nodes