Token Allocation & Cluster Rebalancing in Apache Cassandra 4.x/5.x

Uneven token ownership is a quiet tax on a Cassandra cluster: the node that owns 40% of the ring in a supposedly balanced four-node deployment absorbs more writes, compacts harder, and hits its disk ceiling first, dragging tail latency for everyone. This guide covers how token ranges are assigned, why the algorithmic allocator matters, how to read ownership skew from nodetool status, and how to rebalance with nodetool move and the mandatory nodetool cleanup that follows any topology change. It sits under Node Lifecycle Automation — read that first for the state-machine model of joins, leaves, and replacements, because rebalancing is what keeps every one of those transitions from silently distorting the ring. Everything below is validated against Apache Cassandra 4.0, 4.1, and 5.0, with version drift called out where it bites.

Concept: how tokens map to data, and why balance drifts

Cassandra distributes data by hashing each partition key with Murmur3Partitioner into a 64-bit token space and assigning ownership of contiguous token ranges to nodes. With virtual nodes, a single physical node owns many small, non-contiguous ranges rather than one large arc, which smooths distribution and shrinks the blast radius of any one range during repair or streaming. The full mechanics of how ranges map to replicas are laid out in the data partitioning and token ring basics; the short version is that the sum of a node’s range widths, relative to the whole ring, is its ownership share — and that share dictates its data volume, write load, and compaction pressure.

The number of ranges each node owns is set by num_tokens. Legacy clusters used num_tokens: 256, which statistically evens out ownership through sheer sample size but fragments the ring into thousands of tiny ranges, inflating repair overhead and gossip state. Cassandra 4.0 changed the recommended default to num_tokens: 16, a deliberate trade: far fewer ranges (cheaper repair, leaner gossip, tighter streaming) at the cost of needing help to stay balanced. That help is the algorithmic allocator. With only 16 ranges per node, purely random token selection produces visible ownership skew — one node can end up owning 30% more than another — so 16-token clusters rely on allocate_tokens_for_local_replication_factor to place tokens intelligently instead of at random.

Balance also drifts over time, not just at creation. Every time you add a node, the newcomer carves ranges out of existing owners; every decommission hands a departing node’s ranges to its successors. If those events are not spread evenly — say you repeatedly add nodes that all happen to relieve the same neighbor — ownership can creep toward imbalance even in a nominally healthy ring. Detecting that drift early, from the Owns column of nodetool status, is the operational heartbeat of rebalancing.

Configuration reference

Token behavior is fixed largely at a node’s first boot — num_tokens and the allocation strategy cannot be meaningfully changed on an existing node without a full data reload — so these settings must be right before you scale. The parameters below live in cassandra.yaml and apply per node, but they must be consistent cluster-wide.

A minimal balanced-scaling block for a replication-factor-3 keyspace looks like this:

# cassandra.yaml — Cassandra 4.0+/5.0, RF=3 keyspace
# Set identically on every node BEFORE first boot; these cannot be
# retro-fitted to a running node without a data reload.
num_tokens: 16
allocate_tokens_for_local_replication_factor: 3
auto_bootstrap: true
# initial_token: <only for num_tokens: 1 single-token topologies>

On Cassandra 5.0 you can confirm the allocator is active and inspect per-node token counts through the system_views virtual tables rather than parsing nodetool text, which is convenient inside restricted networks where JMX is closed off. On 4.x, nodetool ring and nodetool status remain the canonical surfaces.

Step-by-step: rebalancing after a topology change

The procedure below adds a node to an under-balanced ring, verifies even ownership, and reclaims freed space. Run it against a staging cluster with production-scale data before performing it live, and do exactly one node at a time — concurrent joins are how you turn a rebalance into a streaming storm, as the parent node lifecycle automation guide explains.

1. Confirm the ring is healthy and record the baseline. Every node must be Up/Normal before you change topology:

   Copy

   # Cassandra 4.x/5.x — ownership snapshot per keyspace
   nodetool status my_keyspace

   Expected output shows each node as UN with an Owns percentage; note the current spread so you can prove the rebalance improved it:

   Datacenter: dc1
   ==========
   --  Address     Load     Tokens  Owns    Host ID   Rack
   UN  10.0.0.11   412 GiB  16      41.2%   a1b2...   rack1
   UN  10.0.0.12   288 GiB  16      29.4%   c3d4...   rack1
   UN  10.0.0.13   301 GiB  16      29.4%   e5f6...   rack1
   

   A spread from 29% to 41% on three nodes is real skew worth correcting.

2. Provision the new node with matching token settings. In its cassandra.yaml, set num_tokens: 16 and allocate_tokens_for_local_replication_factor: 3 (matching the ring), leave auto_bootstrap: true, and point seeds at existing seed nodes. Do not set initial_token on a vnode cluster.

3. Bootstrap the node and watch streaming. Start Cassandra on the new host and follow the join through the bootstrapping new nodes safely procedure. Track streaming progress until it drains:

   Copy

   # On the joining node — should show UJ, then progress to UN
   nodetool netstats | grep -E "Mode|Receiving|Sending"

   The node reports Mode: JOINING while it streams the ranges the allocator assigned it, then flips to Mode: NORMAL.

4. Confirm the join completed and ownership evened out. From any node:

   Copy

   nodetool status my_keyspace

   You should now see four UN nodes each near 25%:

   UN  10.0.0.11   318 GiB  16      25.1%   a1b2...   rack1
   UN  10.0.0.12   295 GiB  16      24.8%   c3d4...   rack1
   UN  10.0.0.13   307 GiB  16      25.3%   e5f6...   rack1
   UN  10.0.0.14   301 GiB  16      24.8%   g7h8...   rack1
   

5. Run mandatory cleanup on the pre-existing nodes. Bootstrap streams data to the new node but does NOT delete the now-unowned copies from the old owners — they keep serving stale ranges and waste disk until you purge them. Run cleanup one node at a time to bound the I/O:

   Copy

   # Run on each ORIGINAL node (not the new one); one at a time.
   nodetool cleanup my_keyspace

   nodetool cleanup rewrites each SSTable, dropping partitions the node no longer owns. It is I/O-heavy and safe to defer to a low-traffic window, but the freed disk (the Load drop you saw above only fully materializes after cleanup) is often the whole point of scaling out.

6. For a single-token topology, use nodetool move instead. If the deployment runs num_tokens: 1 with initial_token, you rebalance by relocating a node’s single token rather than by adding vnodes:

   Copy

   # Single-token clusters ONLY. Moves this node to a new token position,
   # streaming ranges to/from neighbors. Not valid on vnode (num_tokens > 1) rings.
   nodetool move 3074457345618258602

   nodetool move is a single-token operation: it is meaningful only when a node owns exactly one token, and it streams the affected ranges as the node shifts. Follow every move with nodetool cleanup on the neighbors that ceded ranges.

Verification & observability

Prove the rebalance actually moved data and freed space rather than just reporting new percentages. The Owns column is the headline metric, but confirm it against real disk load:

# Ownership should now sit within a few points of the even share (100% / N nodes).
nodetool status my_keyspace
# Per-node data size — Load should fall on cleaned-up nodes, rise on the newcomer.
nodetool info | grep -E "Load|Token"

Confirm no ranges are orphaned or duplicated after the topology change by scanning the ring map:

# Each token maps to exactly one endpoint; no gaps, no endpoint that has left.
nodetool ring my_keyspace | awk '{print $NF}' | sort | uniq -c

For continuous observability, scrape the ownership and load MBeans through the Prometheus JMX exporter and alert when any node’s ownership exceeds roughly 1.3× the even share, or when per-node Load diverges by more than about 20% from the fleet median — either is an early signal that a future scale event, or a past one without cleanup, has left the ring lopsided. Pair that with a post-topology check that verifies cleanup ran everywhere; a common silent failure is a rebalance that looks even in the Owns column while old owners still carry the un-purged data on disk.

Failure modes & rollback

Skew persists after adding a node (random allocation). If a new node joins a 16-token ring without allocate_tokens_for_local_replication_factor set, its tokens are chosen randomly and ownership stays lumpy no matter how many nodes you add. Detection: nodetool status shows ownership variance well above 10% even at four or more nodes, and the new node’s tokens cluster rather than interleave. Rollback: you cannot re-allocate tokens on a live node; decommission the badly-placed node using the decommissioning nodes safely procedure, set the allocator option in its cassandra.yaml, wipe its data directory, and re-bootstrap it so the allocator places its tokens with balance in mind.

Disk fills because cleanup was skipped. After a scale-out, the original nodes still hold the ranges they ceded until nodetool cleanup rewrites their SSTables. Skip it and those nodes keep the stale data, so the disk savings from scaling never appear and the fullest node can still hit its ceiling. Detection: nodetool status shows even Owns percentages but Load on the old nodes has not dropped; disk usage stays flat post-bootstrap. Rollback: there is nothing to undo — run nodetool cleanup <keyspace> on each pre-existing node, one at a time, to reclaim the space. Cleanup is always safe to run late; it only removes data the node provably no longer owns.

nodetool move rejected or wedging a vnode cluster. nodetool move targets a single token and is invalid on a multi-token (vnode) ring; running it against num_tokens > 1 either errors out or produces a confusing partial reassignment. Detection: the command returns an error about token count, or on a misconfigured node it triggers unexpected streaming without converging. Rollback: do not force it — on vnode clusters, rebalance by adding or removing whole nodes and letting the allocator place tokens, and reserve nodetool move for genuine single-token deployments where initial_token is set.

FAQ

Should I use num_tokens 16 or 256 for a new cluster?

Use 16 for any new Cassandra 4.0+ or 5.0 cluster and set allocate_tokens_for_local_replication_factor to your keyspace’s replication factor. Sixteen tokens with the algorithmic allocator gives balanced ownership with far less repair and gossip overhead than 256. The old 256 default only made sense before the allocator existed, when large sample counts were the only way to even out random token placement; on modern versions it just multiplies repair ranges and streaming metadata for no balance benefit.

Can I change num_tokens on a node that is already running?

Not in place. num_tokens is fixed when a node first bootstraps and is baked into the token assignments it advertises through gossip. To change it you effectively rebuild the node: decommission it, update cassandra.yaml, wipe its data directory, and re-bootstrap so it claims a fresh set of tokens under the new count. Because this is disruptive, decide on 16 versus a legacy value before you scale, and migrate a whole cluster node-by-node if you must move off 256.

Do I always have to run nodetool cleanup after adding a node?

Yes, on every node that already existed before the new one joined. Bootstrap copies the newcomer’s ranges to it but never deletes the now-unowned copies from the previous owners, so until you run nodetool cleanup those nodes waste disk serving data they no longer own. Cleanup is I/O-intensive, so run it one node at a time during a low-traffic window — but do not skip it, or the disk savings from scaling out never materialize.

Why is one node still owning more of the ring than the others?

Persistent skew almost always means tokens were placed randomly rather than by the allocator, or the deployment mixes nodes configured with different num_tokens values. Check that allocate_tokens_for_local_replication_factor matches your RF on every node and that all nodes share the same token count via nodetool status (the Tokens column). If a single node was bootstrapped before the allocator was enabled, its tokens will stay lumpy until you re-bootstrap it with the option set.

Is nodetool move safe on a vnode cluster?

No. nodetool move relocates a single token and is only meaningful when num_tokens: 1. On a vnode ring (any num_tokens > 1) it is not the right tool and can produce confusing partial reassignments; rebalance those clusters by adding or removing whole nodes and letting algorithmic allocation position the tokens. Reserve move for single-token topologies that pin exact tokens with initial_token.

Related

• Node Lifecycle Automation — the parent guide covering the join/leave/replace state machine that every rebalance operates within.
• Data partitioning and token ring basics — how Murmur3 tokens, vnodes, and replica placement define ownership.
• Bootstrapping new nodes safely — the streaming-supervised join that redistributes tokens when you scale out.
• Decommissioning nodes safely — draining a node’s ranges back to the ring, the inverse of a scale-out rebalance.