Decommissioning Nodes Safely with nodetool decommission in Cassandra 4.x/5.x

Retiring a node from a running Apache Cassandra ring is one of the few lifecycle operations that moves real data while the deployment keeps serving traffic, so it punishes shortcuts. When you run nodetool decommission, the target node does not simply shut down — it enters a leaving state and hands every token range it owns to the replicas that must inherit them, streaming that data out before it exits the ring. Do it without checking per-datacenter replication factor, receiver disk headroom, and concurrency, and you can drop a partition below its live replica count mid-stream, exhaust a neighbour’s disk, or leave orphaned ranges that only a full repair will reconcile. This guide is for DBAs and SREs who need to remove a healthy node from a Cassandra 4.0, 4.1, or 5.0 cluster deterministically. It sits under node lifecycle automation; read that overview first if you have not yet mapped how bootstrap, decommission, replacement, and rolling restart interlock. Every command below is validated against 4.x and 5.x, with version drift flagged inline.

Concept: what “decommission” actually does

A Cassandra cluster distributes ownership across a token ring, and each node is the primary owner of the ranges assigned to its tokens plus a replica for ranges owned by others, as governed by the keyspace replication factor. Decommissioning is the reverse of bootstrap: instead of a joining node claiming ranges and pulling data in, a leaving node relinquishes its ranges and pushes data out. The node stays up and reachable for the entire operation — clients keep reading and writing through it — but its gossip state transitions so the rest of the ring knows a handover is under way.

That state is the crux of the whole procedure. In nodetool status a healthy node shows UN (Up / Normal). The moment decommission begins it flips to UL — Up / Leaving — and stays there until the last owned range has streamed to its new home. Only then does the process transition the node to LEFT, release its tokens, and terminate. Contrast this with UJ (Up / Joining), the bootstrap state, and DN (Down / Normal), a node that has simply failed. The gossip and failure-detection protocols propagate the UL marker cluster-wide so no coordinator routes new ownership to a node that is on its way out, and so peers know to expect inbound streams.

Streaming direction is what makes decommission safe when it is done right and dangerous when it is not. For every range the leaving node owns, Cassandra computes which surviving nodes must now hold a replica to preserve the configured replication factor, then streams the SSTables for that range to them. A three-node rack running RF=3 will, after one node leaves, need the two survivors to hold what the third held; if those survivors were already at capacity, the inbound stream can fill their data volumes. The operation is therefore only as safe as the receiving side. This is the opposite failure surface from a joining node, and it is why decommission and bootstrap must never run concurrently on overlapping ranges.

The four-stage lifecycle below is what you are steering.

There is a second, distinct operation for a node that is already gone. nodetool removenode retires a node that is DN — the process is dead and cannot stream anything itself, so the surviving replicas reconstruct the missing data among themselves from their existing copies. Use decommission when the node is alive and you can drain it gracefully; use removenode (or assassinate as a last resort) only when it is unrecoverable. The two are not interchangeable: running removenode against a live node, or decommission against a dead one, corrupts ring state. This guide covers the graceful path; the dead-node path belongs with replacing dead nodes.

Configuration reference

Decommission speed and safety are governed by outbound streaming throughput, per-host connection fan-out, and — above everything — the replication factor of every keyspace the node participates in. The streaming knobs live in cassandra.yaml and are also adjustable at runtime through nodetool; the replication constraints are structural and cannot be tuned away.

Note the units: stream_throughput_outbound_megabits_per_sec and the nodetool setstreamthroughput argument are megabits per second, whereas setcompactionthroughput takes megabytes per second — a common cross-wiring that halves or octuples the rate you intended. The replication-factor row is the one you cannot negotiate: if a datacenter runs RF=3 and you decommission a node, that DC must still contain at least three live, UN nodes when the operation completes, or every LOCAL_QUORUM operation loses its guarantee. A minimal runtime block to pace an outbound drain:

# cassandra.yaml — Cassandra 4.1+ / 5.0. Applied at startup.
# Pre-4.1 uses the same key names for these streaming settings.
stream_throughput_outbound_megabits_per_sec: 200
streaming_connections_per_host: 2
inter_dc_stream_throughput_outbound_megabits_per_sec: 100

# Pace the drain at runtime without a restart (value is megabits/sec).
nodetool setstreamthroughput 200
nodetool getstreamthroughput   # confirm the applied ceiling

Before you touch either, confirm the surviving topology can absorb the load. The check that decides go/no-go — live replica count per DC versus RF, plus receiver disk headroom — is important enough to run as its own gate; the dedicated pre-flight validation script formalizes it and returns an explicit verdict.

Step-by-step: draining a live node out of the ring

Run this on the node you intend to retire unless noted otherwise. Each step is safe to inspect before the irreversible streaming begins at step 5.

1. Confirm the whole cluster is healthy and identify the target. A decommission that starts while another node is DN risks dropping a range below its live replica count as soon as the leaving node’s data moves.

   Copy

   # Every node must read UN. Note the target's Host ID and per-DC layout.
   nodetool status

   Expected output (abridged):

   Datacenter: dc-east
   ==================
   --  Address     Load       Tokens  Owns    Host ID                               Rack
   UN  10.0.2.11   412.3 GiB  256     33.6%   9a1f...                               rack1
   UN  10.0.2.12   408.9 GiB  256     33.1%   b7c2...                               rack1
   UN  10.0.2.13   401.2 GiB  256     33.3%   c4d9...                               rack1
   

2. Verify replication-factor safety for every keyspace in the DC. Count the live nodes that will remain in the target’s datacenter and compare against each keyspace’s RF for that DC. Removing a node from a three-node RF=3 DC drops you to two live replicas and breaks LOCAL_QUORUM writes — do not proceed.

   Copy

   # List replication for user keyspaces; check RF per DC against surviving node count.
   cqlsh -e "SELECT keyspace_name, replication FROM system_schema.keyspaces;"

   Safety gate: for the target’s DC, (live UN nodes in DC) − 1 must be >= max(RF across keyspaces in that DC). If equality would be violated, add a node first or abort.

3. Confirm receiver disk headroom. The load the leaving node carries redistributes across the survivors. Estimate the inbound per survivor as roughly leaving_load / (surviving_nodes_in_dc) and confirm each has that much free space plus compaction overhead.

   Copy

   # Per-node load; compare each survivor's free disk against its projected inbound.
   nodetool status | awk '/^UN|^DN/ {print $2, $3, $4}'
   df -h /var/lib/cassandra/data

   Safety gate: every surviving node must keep at least 30% free after projected inbound, so window-closing compactions during streaming do not trip OutOfSpaceException.

4. Ensure no other lifecycle op is in flight. Bootstrap (UJ), another decommission (UL), an active nodetool rebuild, or a running repair all contend for streaming bandwidth and ring stability.

   Copy

   nodetool status | grep -E "UJ|UL|DL"        # any node mid-lifecycle?
   nodetool netstats | grep -iE "repair|bootstrap|rebuild"

   Safety gate: output must be empty. Concurrent lifecycle operations are the single most common cause of a stuck or inconsistent decommission.

5. Pace outbound streaming, then trigger the decommission. Set a throughput ceiling appropriate to your disks, then start the drain. The command blocks until streaming completes.

   Copy

   nodetool setstreamthroughput 200      # megabits/sec — pace before draining
   nodetool decommission                 # blocks until the node has fully LEFT

   The node immediately transitions to UL and begins streaming. On Cassandra 5.0 you can add --force only to override a specific safety veto after you have manually confirmed the ring state; do not use it to bypass step 2.

6. Watch the drain progress from another node. Because the local process is busy streaming, observe from a peer.

   Copy

   # From a surviving node — the target should show UL, then disappear.
   watch -n 10 'nodetool status | grep -E "UL|UN|DN"'
   nodetool netstats -H          # on the leaving node: outbound stream progress

7. Confirm the node has left and clean up. When decommission returns, the node is gone from the ring. Stop the service and reclaim the host.

   Copy

   nodetool status | grep <target-host-id>   # should return nothing
   systemctl stop cassandra                   # only after it has LEFT

For a fully automated version of this sequence with polling and idempotency, see safely decommissioning a node with Python.

Verification & observability

The decommission is complete only when three things are true: the node is absent from the ring, ownership sums back to 100% across survivors, and no range is under-replicated. Verify each.

# 1. The retired node must not appear, and Owns must redistribute to survivors.
nodetool status

The target’s row is gone and the remaining nodes’ Owns percentages have grown to cover it. A row still showing UL means streaming did not finish — do not stop the process.

# 2. Confirm no streaming sessions linger and the ring agrees on membership.
nodetool netstats -H
nodetool gossipinfo | grep -E "STATUS|LOAD"

STATUS should read NORMAL for all survivors, with no LEFT/removing entry pinned in gossip. A stale LEFT entry that never clears is a known symptom of a partially propagated exit.

# 3. Prove no range lost a replica: run a bounded repair on a survivor.
nodetool repair -pr --in-local-dc

A clean primary-range repair that streams little or nothing confirms the survivors already hold everything the departed node did. Watch the compaction that follows — the inbound SSTables from streaming will trigger merges you should let settle; if the queue climbs, the techniques in compaction backlog analysis and alerting apply directly. For metrics, the org.apache.cassandra.metrics:type=Streaming MBeans expose active session and byte counts you can scrape while the drain runs; on Cassandra 5.0 the system_views virtual tables also surface streaming state without JMX.

Failure modes & rollback

Replication factor breached mid-drain (UnavailableException). You started a decommission in a DC that then held only RF nodes, so as ranges moved, some fell to RF−1 live replicas and LOCAL_QUORUM reads failed. Detection: application UnavailableException, and nodetool status shows the DC at exactly RF nodes with one UL. Rollback: decommission cannot be cleanly cancelled once streaming is well under way, but if you catch it early you can stop the leaving node’s service; because it has not yet released its tokens, restarting it returns it to UN and the ring reverts. If it has already streamed most ranges, the safer path is to let it finish, then bootstrap a replacement node to restore the count. Never leave a DC below RF.

Receiver disk exhaustion (OutOfSpaceException / FSWriteError). A survivor filled up absorbing inbound ranges and its own compaction of the new SSTables. Detection: FSWriteError or OutOfSpaceException on a survivor, streaming stalls at a fixed percentage. Rollback: pause the drain pressure by lowering nodetool setstreamthroughput on the leaving node, free space on the choking survivor (archive cold data, avoid nodetool cleanup mid-stream since it competes for I/O), and resume. If the survivor cannot be recovered, stop the leaving node before it releases tokens and re-plan capacity.

Stuck at a fixed streaming percentage (StreamException). The drain hangs — one receiver is unreachable, throttled to a crawl, or hit a transient network fault, and the stream neither completes nor errors out cleanly. Detection: nodetool netstats -H shows the same byte count across several polls; StreamException: Stream failed in system.log. Rollback: decommission does not resume from a checkpoint. Stop the leaving node’s service (it still holds its tokens, so the ring is intact), fix the network or receiver, confirm every node is UN, and re-issue nodetool decommission. Never removenode a node that is stuck in UL — it is still alive and owns its tokens.

FAQ

What is the difference between UL and UJ in nodetool status?

UL is Up / Leaving — a live node that is streaming its owned ranges out because a decommission is in progress. UJ is Up / Joining — a live node pulling ranges in during bootstrap. They are opposite directions of the same streaming machinery. A node should never be UL and UJ, and you should never run a decommission (producing UL) on one node while another is UJ on overlapping ranges, because the two operations compete for streaming bandwidth and can leave ownership inconsistent.

When should I use removenode instead of decommission?

Use nodetool decommission when the node is alive (UN) and can stream its own data out gracefully. Use nodetool removenode <host-id> only when the node is already dead (DN) and cannot participate — the surviving replicas then rebuild the missing data from their existing copies. Running removenode against a live node, or decommission against a dead one, corrupts ring membership. For a node that is DN and gossip will not let removenode proceed, nodetool assassinate is the forceful last resort.

How do I make sure decommission does not drop me below replication factor?

Count the live, UN nodes in the target’s datacenter and subtract one for the node you are removing. That remainder must be at least the largest keyspace replication_factor configured for that DC. If a three-node DC runs RF=3, removing a node leaves two live replicas and breaks LOCAL_QUORUM, so you must add a replacement node first. Validate this per DC, never cluster-wide, because replication factor is defined per datacenter in NetworkTopologyStrategy.

Can I cancel a decommission once it has started?

There is no clean cancel command. While the node is still UL and has not released its tokens, stopping its service and restarting it will return it to UN and revert the ring, but only if streaming has not substantially completed. Once most ranges have moved, the safe path is to let the operation finish and then bootstrap a replacement to restore capacity. Never assassinate or removenode a node that is actively UL.

Why is my decommission stuck at the same percentage?

Streaming has stalled — usually an unreachable or severely throttled receiver, or a transient network fault. Check nodetool netstats -H across a few polls; if the byte count is frozen, one session is wedged. Decommission has no checkpoint resume, so stop the leaving node (it still owns its tokens, so the ring stays consistent), fix the receiver or network, confirm all nodes are UN, and re-run nodetool decommission.

Related

• Node lifecycle automation — the parent guide covering how bootstrap, decommission, replacement, and rolling restarts fit together.
• Safely decommissioning a node with Python — an idempotent orchestrator that runs the pre-flight gates, triggers the drain, and polls the ring until the node leaves.
• Pre-flight checks before decommissioning a node — a read-only go/no-go validator you run before any decommission.
• Node gossip and failure-detection protocols — how the UL and LEFT states propagate across the ring.
• Read repair vs anti-entropy repair — the repair that confirms no range lost a replica after a node leaves.