Bootstrapping New Nodes Safely into a Live Cassandra 4.x/5.x Cluster

Expanding a running Apache Cassandra cluster looks deceptively simple — start a process, watch it appear in nodetool status — but the join is one of the few operations that silently redistributes ownership of live data while clients keep reading and writing. Done wrong, a bootstrap leaves ranges under-replicated, strands stale data on the nodes that used to own it, or brings a node up empty while it advertises itself as a replica. This guide is the operational reference for adding a single node to a healthy Cassandra 4.0, 4.1, or 5.0 ring without dropping a request, and it forms part of the broader node lifecycle automation practice that also covers decommissioning, dead-node replacement, and rolling upgrades. It assumes you already understand how partition ownership is derived from the token ring — if not, read the data partitioning and token ring basics first, because every streaming decision below flows from it. Version drift is called out inline where 4.1 renamed configuration keys.

How a bootstrap actually works

When a fresh node starts with auto_bootstrap enabled and empty data directories, it does not simply announce itself and begin serving traffic. It joins the ring in a distinct provisional state — UJ, meaning Up / Joining — calculates the token ranges it is about to own, and pulls every one of those ranges from the current replicas before it will accept a single read. Only when the last stream lands and the node finalizes its position does it flip to UN (Up / Normal) and start participating in the read path. That streaming-before-serving contract is the entire point of a safe bootstrap: it guarantees the new node holds a complete copy of its ranges the moment it becomes a replica, so the deployment never drops below its configured replication factor for the data it now owns.

Gossip is what makes this coordinated rather than chaotic. The joining node contacts the seed nodes listed in its cassandra.yaml, learns the current ring topology through the gossip and failure-detection protocols, and announces its own tokens. Existing nodes see a new endpoint in the BOOT state, compute which of their ranges now belong to it, and open streaming sessions to hand those ranges over. Because ownership is being recalculated across the whole ring, Cassandra enforces a hard constraint: only one node may join at a time while cassandra.consistent.rangemovement is true (the default). Attempting a second concurrent join aborts with Other bootstrapping/leaving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true. Respecting that constraint is not optional — it is what preserves consistency guarantees during range movement.

Two rules about seeds are non-negotiable. First, a seed node never bootstraps. A node that finds its own address in its seed list skips the streaming phase entirely, comes up immediately in UN, and claims ownership of ranges for which it holds no data. That is how operators accidentally introduce silent data loss. Second, you promote a freshly joined node to seed status only after it has finished bootstrapping and been repaired — never before. The safe sequence is always: join as a non-seed, reach UN, verify, and only then edit seed lists if the topology warrants it. Where a node lands on the ring — and therefore how much it streams — is governed by token allocation, which the token allocation and cluster rebalancing guide covers in depth.

The final, frequently-skipped half of a bootstrap happens on the old nodes. Once the new node owns a slice of ranges, the previous replicas still physically hold that data on disk — Cassandra does not delete it during the join. Those keys are now out-of-range for the old nodes, and until you run nodetool cleanup on each of them, they waste disk, inflate compaction, and can resurrect deleted data if the topology changes again. Cleanup is a mandatory closing step, not a cosmetic one.

The five phases of a safe bootstrap: one node joins, gossips its tokens, streams its ranges, reaches UN, and only then triggers cleanup on the pre-existing replicas.

Configuration reference

The parameters below live in the joining node’s cassandra.yaml and shape how it takes its position and how fast it pulls data. Streaming throughput and connection counts are the two levers you tune per environment; the token settings must match the rest of the ring.

Parameter	Type	Default	Recommended	Impact on the join
`auto_bootstrap`	boolean	`true` (implicit)	`true` for every join	When true the node streams all owned ranges before serving; set false only for the very first node of a brand-new datacenter
`num_tokens`	int	`16`	`16` (match existing nodes)	Number of vnode tokens claimed; must be identical across nodes or ownership skews
`allocate_tokens_for_local_replication_factor`	int	commented (random)	RF of your primary keyspace, e.g. `3`	Switches token selection from random to the allocation algorithm for even ownership and smaller streams
`stream_throughput_outbound_megabits_per_sec`	int (Mb/s)	`200`	tune to NIC/disk headroom	Caps outbound bandwidth each replica uses to feed the join; renamed to `stream_throughput_outbound` (unit-typed, e.g. `24MiB/s`) in 4.1+
`streaming_connections_per_host`	int	`1`	`1`–`4`	Parallel socket connections opened per peer during streaming; higher speeds large joins but raises contention

A minimal, join-ready block on the new node looks like this. Note that seeds points at existing seed nodes, never at the joining node itself:

# cassandra.yaml on the JOINING node — Cassandra 4.x / 5.x
cluster_name: 'prod-analytics'
num_tokens: 16
allocate_tokens_for_local_replication_factor: 3
auto_bootstrap: true            # implicit default; shown for clarity
streaming_connections_per_host: 2

# 4.0 and earlier:
# stream_throughput_outbound_megabits_per_sec: 200
# 4.1+ (unit-typed rename):
stream_throughput_outbound: 24MiB/s

seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.0.1.10,10.0.1.11"   # EXISTING seeds only — not this node

The endpoint_snitch, cluster_name, and datacenter/rack assignment must match the deployment the node is joining, or it will refuse to gossip or land in the wrong topology.

Step-by-step: joining one node safely

Run these in order on the specified host. Do not start the join until every pre-flight gate passes.

Confirm the deployment is uniformly healthy. A bootstrap should only begin against an all-UN ring with a single schema version. Run on any existing node:
```
nodetool status
nodetool describecluster
```
Safety gate: every node must read UN. If any node is DN, UJ, or UL, stop — you may not add a node while another is joining or leaving. Expected output:
```
Datacenter: dc1
==============
Status=Up/Down  State=Normal/Leaving/Joining/Moving
--  Address     Load     Tokens  Owns    Host ID   Rack
UN  10.0.1.10   412 GiB  16      33.4%   6b1f...   rack1
UN  10.0.1.11   405 GiB  16      33.1%   9c2a...   rack1
UN  10.0.1.12   418 GiB  16      33.5%   1d3e...   rack1
```
describecluster must list exactly one schema version under Schema versions:. More than one means schema disagreement — resolve it before joining.
Prepare the new node. Install the identical Cassandra version, confirm the data, commitlog, and hints directories are empty, and write the cassandra.yaml from the reference above with the correct seeds, snitch, and token settings. An empty data directory is what tells Cassandra this is a genuine bootstrap.
```
ls -la /var/lib/cassandra/data /var/lib/cassandra/commitlog
```
Safety gate: if the data directory is non-empty, wipe it (rm -rf the contents) or the node will come up as an existing replica instead of bootstrapping.
Optionally throttle streaming on the source nodes. If the deployment is under heavy write load, cap outbound streaming so the join does not starve client I/O. This is reversible and applies live:
```
# Run on each existing node; value is megabits/s on 4.0, unit-typed on 4.1+.
nodetool setstreamthroughput 200
```
Start the joining node and watch it enter UJ. Bring the process up and immediately tail the log:
```
systemctl start cassandra
grep -E "JOINING|bootstrap|Starting to bootstrap" /var/log/cassandra/system.log
```
Expected output: log lines progressing through JOINING: waiting for ring information, JOINING: schema complete, ready to bootstrap, and JOINING: Starting to bootstrap. On any existing node, nodetool status now shows the newcomer as UJ:
```
UJ  10.0.1.13   1.2 GiB  16      ?       f4a7...   rack1
```
Monitor streaming to completion. The join is I/O work; track it rather than guessing. From the joining node:
```
nodetool netstats | grep -E "Mode|Receiving|Bootstrap|total files"
```
Expected output: Mode: JOINING with a list of Receiving N files sessions whose byte counts climb toward completion. Detailed progress-and-ETA tooling lives in the companion guide on monitoring streaming progress during bootstrap.
Confirm the transition to UN. When the last stream lands, the node finalizes and flips to UN. Verify from an existing node:
```
nodetool status | grep 10.0.1.13
```
Safety gate: do not proceed to cleanup until the new node reads UN with a non-trivial Load and a sensible Owns percentage. A node that jumped to UN in seconds with near-zero load skipped bootstrap — treat that as a failure (see below).
Run cleanup on the pre-existing nodes — one at a time. Only after the newcomer is UN do you reclaim the now-out-of-range data from the old replicas. Cleanup is a full compaction of each table, so serialize it:
```
# On each ORIGINAL node, sequentially — never in parallel across the deployment.
nodetool cleanup
```
Safety gate: run cleanup on a single node at a time and confirm it finishes (nodetool compactionstats returns to pending tasks: 0) before moving to the next, so you never sacrifice more than one replica’s read performance at once.

Verification & observability

After the join, prove three things: the node is a full-fledged replica, ownership rebalanced, and no streaming is still pending. Ownership should now be roughly even across all nodes:

nodetool status

A healthy post-join status shows every node UN, Owns percentages evened out (for the analytics ring above, roughly 25% each across four nodes), and Load on the new node in the same order of magnitude as its peers. Confirm streaming has fully drained:

# Should report "Mode: NORMAL" and no active Receiving/Sending sessions.
nodetool netstats

Cross-check that gossip sees the node as normal and that schema is still converged:

nodetool gossipinfo | grep -A2 10.0.1.13   # STATUS should read NORMAL
nodetool describecluster                    # still one schema version

Finally, because the new node has never been repaired, its data is only as consistent as what streamed to it. Schedule an anti-entropy repair on the new node during a low-traffic window to reconcile any writes that raced the join.

Failure modes & rollback

Second join attempted while one is in progress. Starting a second node while another is UJ aborts the new process with Other bootstrapping/leaving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true. Detection: the second node exits at startup and logs that message; nodetool status shows one UJ already present. Rollback: none needed — the failed node never joined. Wait for the first node to reach UN, verify, then start the second. Do not work around this by disabling consistent range movement on a live cluster.

Streaming stalls and the node is stuck in UJ. A network drop, a saturated source node, or a StreamException can leave the joining node in UJ indefinitely with netstats showing a session frozen at partial completion. Detection: nodetool netstats byte counts stop advancing for many minutes and system.log shows StreamException: Stream failed. Rollback: resume the interrupted transfer with nodetool bootstrap resume, which retries only the incomplete ranges. If resume fails repeatedly, stop the node, wipe its data/commitlog/hints directories, and restart to bootstrap cleanly from scratch — a half-streamed node must never be forced to UN.

Node joined as a seed and skipped bootstrap. If the new node’s address appears in its own seed list, it comes up UN almost instantly, owning ranges it never streamed. Detection: nodetool status shows the node UN with Load far below its peers moments after start, and reads for its ranges miss data. Rollback: immediately nodetool decommission the node (or nodetool removenode <host-id> from another node if it is already down), correct its cassandra.yaml to list only existing seeds, wipe its data directories, and rejoin it as a proper non-seed bootstrap.

FAQ

Why must I only bootstrap one node at a time?

Because a join recalculates range ownership across the whole ring, and Cassandra’s consistent range movement guarantees only hold when a single node moves at once. With cassandra.consistent.rangemovement at its default of true, a second concurrent bootstrap aborts with Other bootstrapping/leaving nodes detected. Joining two nodes simultaneously risks both claiming overlapping new ranges and streaming from stale sources, which can leave data under-replicated. Always wait for UN before starting the next join.

Do I really need nodetool cleanup after every bootstrap?

Yes, on the pre-existing nodes. During a join, Cassandra streams a copy of the newly-owned ranges to the new node but does not delete the originals from the old replicas — that data is now out of their range but still on disk. Until you run nodetool cleanup on each original node, it wastes storage, inflates compaction, and can even resurrect deleted rows if ownership shifts again. Run it one node at a time after the newcomer reaches UN.

What is the difference between the UJ and UN states?

UJ is Up/Joining: the node process is running and gossiping, but it is still streaming its owned ranges and will not serve reads. UN is Up/Normal: streaming has completed, the node has finalized its ring position, and it is a full read/write replica. A safe bootstrap always passes through UJ before UN. A node that reaches UN without ever showing UJ skipped bootstrap — usually because it was listed as a seed — and must be rejoined.

Should a new node ever be a seed node?

Not during its join. A node that finds itself in its own seed list bypasses bootstrap entirely, coming up UN with no streamed data and silently under-replicating its ranges. Bootstrap the node as a non-seed, let it reach UN, repair it, and only then — if your topology needs another seed — add its address to the seed lists across the deployment.

How do I speed up a slow bootstrap without hurting clients?

Raise streaming throughput and parallelism incrementally, watching client latency. nodetool setstreamthroughput <Mb/s> lifts the cap live on the source nodes, and streaming_connections_per_host (set before start) opens more parallel sockets per peer. Push these only as far as the source nodes’ disk and NIC headroom allows; if client p99 latency climbs, throttle back. Never chase speed by disabling consistent range movement.

Node lifecycle automation — the parent guide tying bootstrap, decommission, replacement, and rolling upgrades into one operational discipline.
Automating node bootstrap with Python — an idempotent orchestrator that runs this entire procedure end to end with safety gates.
Monitoring streaming progress during bootstrap — parse netstats and JMX to compute per-stream progress and an ETA for the join.
Token allocation and cluster rebalancing — how token selection decides where a joining node lands and how much it streams.
Data partitioning and token ring basics — the ownership model every streaming decision here derives from.

Related guides

Back to Node Lifecycle Automation