Rolling Restarts & Safe Minor-Version Upgrades in Cassandra 4.x/5.x

A rolling restart is the workhorse of Cassandra operations: it is how you apply a cassandra.yaml change, rotate a JVM flag, pick up a new certificate, or move a deployment from one patch release to the next without ever taking the keyspace offline. Done well, it is invisible to clients; done carelessly, it drops writes, strands hints, or — during an upgrade — leaves a deployment stranded in a mixed-version state where repair and streaming are unsafe. This guide sits under node lifecycle automation and treats the restart as a serialized, health-gated loop rather than a for host in $hosts; do ssh $host restart one-liner. It is written for DBAs and SREs running Apache Cassandra 4.0, 4.1, and 5.0, and every command below distinguishes 4.x from 5.x behavior where it drifts. Two focused runbooks extend it: one that orchestrates rolling restarts with Python when no binary changes, and one that covers automating minor-version upgrades safely when the package on disk changes.

Concept: why the restart is a loop, not a broadcast

The safety of a rolling operation rests on a single invariant: at every instant, no more than one replica of any token range is unavailable, and the replicas that remain can still satisfy the deployment’s consistency level. With a replication factor of 3 and LOCAL_QUORUM reads, one node down per rack leaves two live replicas and quorum holds; two nodes down in the same replica set breaks it and clients start seeing UnavailableException. A rolling restart therefore proceeds strictly one node at a time, and — critically — it does not advance to the next node until the one just restarted has fully rejoined the ring and gossip has reconverged across every peer.

Three mechanics make this non-trivial. First, a running node holds unflushed data in memtables and un-acknowledged coordinator work in flight; killing the process without nodetool drain risks replaying a large commit log on restart and, worse, dropping mutations that were mid-flush. Draining flushes every memtable to SSTables and stops the node from accepting new writes, so the process exits with nothing in volatile memory. Second, gossip does not update instantly. When a node stops, its peers keep it in an UP state for up to phi_convict_threshold worth of missed heartbeats before the gossip failure detector marks it DOWN; when it comes back, peers must exchange gossip digests and reconverge before they route traffic to it again. Advancing before convergence means the next node’s restart can overlap with peers still believing the previous node is down. Third, while a node is down, its replica peers buffer hinted handoffs for it. Those hints must replay and drain before you add more downtime elsewhere, or hints pile up faster than they clear.

An upgrade adds one more layer. When you swap the binary on a node, that node speaks a newer SSTable format and internode-messaging version than its not-yet-upgraded peers. Cassandra is explicitly designed to run in this mixed-version state transiently, but it disables streaming between differing major/minor internode versions and forbids schema changes. That is why the cardinal rule of upgrades is: never run nodetool repair, bootstrap a new node, decommission a node, or ALTER a schema while the deployment spans two versions. After every node is on the new binary, you run nodetool upgradesstables to rewrite SSTables into the new format on each node, at your own pace.

Configuration reference

The knobs and commands below govern how a rolling operation behaves. The cassandra.yaml values are per-node and read at startup; the nodetool commands are runtime actions.

Command / setting	Scope	Purpose in a rolling operation
`nodetool drain`	runtime	Flushes all memtables and stops the node accepting writes; the mandatory pre-stop step so no mutation is lost and commit-log replay on restart is trivial
`nodetool flush`	runtime	Flushes memtables but leaves the node serving; a lighter pre-restart step only when you are not stopping the process
`nodetool upgradesstables [-a]`	runtime	Rewrites SSTables into the running binary’s format after a version change; `-a` forces a rewrite of already-current tables
`nodetool statusgossip`	runtime	Reports whether gossip is `running`; used to confirm a restarted node has rejoined the protocol
`nodetool netstats`	runtime	Shows hint and streaming activity; used to confirm hints have drained before advancing
`nodetool gossipinfo`	runtime	Dumps each peer’s gossip state and `STATUS`; used to confirm ring convergence
`phi_convict_threshold`	cassandra.yaml	Sensitivity of the failure detector; raising it (e.g. 8→12) reduces spurious `DOWN` flapping on a busy cluster during restarts
`auto_bootstrap`	cassandra.yaml	Must stay `true` for real joins; irrelevant to a restart because a restart is not a bootstrap and streams nothing
`hinted_handoff_enabled`	cassandra.yaml	When `true` (default), peers buffer hints for the down node; a rolling loop must let these drain between nodes
`max_hint_window` (5.0) / `max_hint_window_in_ms` (≤4.1)	cassandra.yaml	How long peers keep buffering hints for a down node before giving up; a slow rolling loop that exceeds this window forces a repair later

A minimal cassandra.yaml block tuned for stable rolling operations on a busy cluster:

# cassandra.yaml — Cassandra 4.1+ / 5.0
# Pre-4.1 clusters: max_hint_window_in_ms: 10800000
phi_convict_threshold: 12          # tolerate brief pauses during restarts without false DOWN
hinted_handoff_enabled: true
max_hint_window: 3h                # give a slow rolling loop room before hints expire

Nothing here changes the token layout or the compaction strategy; a restart is deliberately a no-op for data ownership. The only file you typically edit before a plain restart is the one whose change you are applying (a JVM option in jvm-server.options, a cassandra.yaml setting, a truststore). For an upgrade, you also change the installed package, which the minor-version upgrade runbook covers end to end.

Step-by-step: a rack-aware rolling restart

Run this loop for every node in the deployment, completing one node fully before touching the next. Choose the order to respect racks: within a datacenter, restart one rack at a time so you never take two replicas of the same range down together. Do the seed nodes last where practical, and never restart all seeds at once — the deployment needs at least one reachable seed for gossip bootstrap on the others.

Confirm the whole ring is healthy before you start. A rolling restart is only safe from a fully-UN baseline. If any node is already DN or UJ, stop and resolve it first.

# Every node must read UN. Any DN/UJ aborts the operation.
nodetool status

Expected output (abbreviated):

Datacenter: dc1
==========
--  Address       Load     Tokens  Owns   Host ID    Rack
UN  10.0.1.11     412 GiB  16      33.4%  a1b2...    rack1
UN  10.0.1.12     408 GiB  16      33.1%  b2c3...    rack2
UN  10.0.1.13     419 GiB  16      33.5%  c3d4...    rack3

Drain the target node. This is the discipline that separates a clean restart from a lossy one. Draining flushes memtables and stops write acceptance; the process can then be stopped with nothing to lose.
```
# On the target node only.
nodetool drain
```
Expect the command to return within seconds to a couple of minutes depending on memtable size. The last log line should read DRAINED.
Stop the service. Use your init system so the process shuts down cleanly rather than being killed.
```
sudo systemctl stop cassandra
```
Apply the change, then start the service. For a plain restart this is where your edited config or certificate is already on disk; for an upgrade this is where the new package is installed. Start the node and let it replay the (now trivial) commit log.
```
sudo systemctl start cassandra
```
Wait for UN on the restarted node itself. Poll its own view until it reports Normal/Up and gossip is running. Do not trust a single sample.
```
nodetool statusgossip        # expect: running
nodetool status | grep <this-node-ip>   # expect: UN
```
Wait for gossip to converge across peers. The restarted node showing UN locally is necessary but not sufficient; every peer must also agree. Check a couple of other nodes’ view of the ring, and confirm no peer still lists the restarted node as DN.
```
# Run from a peer, not the restarted node.
nodetool gossipinfo | grep -A2 <this-node-ip>   # STATUS should show NORMAL
```

Confirm hints have drained before advancing. Peers buffered hints while the node was down; let them replay so you do not stack downtime.

# On the peers: hint activity should fall to zero.
nodetool netstats | grep -i "hints"
nodetool tpstats | grep -i "HintsDispatcher"

After a version change only: rewrite SSTables. Once a node is on the new binary and back to UN, upgrade its SSTables so it stops carrying the old on-disk format. This is I/O-heavy; it can run in the background and does not block serving.
```
# Only after a binary upgrade — not for a plain config restart.
nodetool upgradesstables
```
Advance to the next node only when steps 5–7 (and 8 for upgrades) are all satisfied. Repeat until every node in every rack has cycled.

Throughout, the invariant is one node down at a time with full reconvergence between nodes. The loop is summarized below.

The rolling loop: one node drains, stops, restarts or upgrades, starts, and must fully rejoin — UN plus gossip convergence plus drained hints — before the loop advances.

Verification & observability

The gate between nodes is where most operators cut corners, so make it explicit and machine-checkable. Three signals together prove a node has genuinely rejoined:

Ring membership. nodetool status from at least two peers shows the node as UN. A node can report UN about itself while a partitioned peer still lists it DN; sampling multiple peers catches that.
Gossip convergence. nodetool gossipinfo shows the node’s STATUS as NORMAL with a stable generation/heartbeat on every peer. A rising heartbeat that a peer is not yet seeing means gossip has not propagated there.
Hint drain. nodetool netstats and the HintsDispatcher pool in nodetool tpstats fall to zero pending on the peers that buffered for the node. Residual hints mean the peers are still catching the node up; adding more downtime now compounds the lag.

For fleet-wide visibility, scrape the same state through the Prometheus JMX exporter: cassandra_gossip status per node, HintsDispatcher pending, and dropped-mutation counters. During and after an upgrade, also watch nodetool version on every node so a stalled node on the old binary is obvious, and track nodetool compactionstats because upgradesstables shows up there as regular compaction work. If you are running anti-entropy after the window closes, the read-repair versus anti-entropy repair guide explains why you wait until all nodes are on one version before scheduling it.

A healthy rolling operation has a recognizable signature: dropped-mutation counters stay flat, no UnavailableException reaches clients, hint dispatch spikes briefly per node and returns to zero, and read/write p99 rises only marginally as one replica cycles.

Failure modes & rollback

A node fails to come back to UN after restart. The most common causes are a bad config edit (a typo in cassandra.yaml, an unparseable JVM option) or a slow commit-log replay after a restart that skipped nodetool drain. Detection: the process is up but nodetool statusgossip reports not running, or nodetool status never leaves the node absent from the ring; system.log shows a startup exception or a long Replaying /var/lib/cassandra/commitlog sequence. Rollback: stop the node, revert the config file to the last-known-good version (keep a copy before editing), and restart. Because a restart streams no data, reverting is safe and fast — the node simply rejoins with the old settings. Do not advance the loop until this node is UN; taking a second node down now can break quorum.

Hints pile up faster than they drain. If the loop advances too quickly, or a node stays down long enough to exceed max_hint_window, peers accumulate — and then may expire — hints for the down node. Detection: HintsDispatcher pending in tpstats climbs across successive nodes instead of returning to zero, and debug.log shows hint-window-expiry warnings. Rollback: slow the loop down — wait for hint dispatch to reach zero before each advance. If hints already expired, the missed mutations require an anti-entropy repair after the whole cluster is on one version; the mechanics of the resulting tombstone and consistency drift are covered under tombstone management and garbage collection.

The deployment is stuck mid-upgrade in a mixed-version state and someone starts a repair. During an upgrade window, internode streaming between differing versions is disabled and schema changes are refused; a repair kicked off now either fails immediately with a streaming error or hangs. Detection: nodetool version differs across nodes while a repair or bootstrap is queued; the repair logs Cannot stream or the coordinator reports a validation that never completes. Rollback: stop the repair with nodetool stop VALIDATION on the affected nodes, freeze all lifecycle operations, and finish the upgrade — get every node onto the new binary and run upgradesstables — before re-attempting repair. If the upgrade itself is failing, roll the affected node back to the previous package from a snapshot as described in the minor-version upgrade runbook.

FAQ

Do I need `nodetool drain` for a plain restart, or only for upgrades?

Always drain before stopping the process, upgrade or not. Draining flushes memtables to SSTables and stops the node accepting writes, so the process exits with nothing in volatile memory and the commit-log replay on restart is trivial. Skipping it risks dropping in-flight mutations and forces a long, I/O-heavy commit-log replay that delays the node reaching UN. The only time you would use nodetool flush instead is when you are not actually stopping the process.

How do I know gossip has settled before moving to the next node?

Sample more than one peer. The restarted node reporting UN about itself is necessary but not sufficient, because a partitioned or lagging peer may still list it DOWN. Confirm nodetool gossipinfo shows the node’s STATUS as NORMAL with a stable heartbeat on at least two other nodes, and confirm HintsDispatcher pending has fallen to zero. Only advance when ring membership, gossip state, and hint drain all agree.

Can I restart or upgrade more than one node at once to go faster?

Only if the nodes hold no common token range — practically, that means at most one node per rack per replica set, and even then it narrows your fault tolerance to zero during the window. For upgrades it is strongly discouraged: the more nodes you flip to the new binary simultaneously, the larger and longer the mixed-version window during which streaming and schema changes are unsafe. The safe default is strictly one node at a time.

Why must I avoid repair, bootstrap, and decommission during an upgrade?

Because a deployment mid-upgrade spans two internode-messaging and SSTable-format versions, and Cassandra deliberately disables streaming between differing versions and refuses schema changes in that window. Repair, bootstrap, decommission, and node replacement all depend on streaming, so they either fail outright or hang. Complete the upgrade on every node first, run nodetool upgradesstables, and only then resume normal lifecycle operations.

When exactly do I run `nodetool upgradesstables`?

After a node is on the new binary and back to UN — per node, at your own pace. It rewrites that node’s SSTables from the old on-disk format into the new one, which is I/O-heavy and shows up as compaction work in nodetool compactionstats. It does not block serving, so you can stagger it across nodes to control disk load. It is unnecessary for a plain config restart, because a restart does not change the SSTable format.

Node Lifecycle Automation — the parent guide that frames every membership change, including restarts and upgrades, as a state-gated operation.
Orchestrating rolling restarts with Python — a resumable, health-gated orchestrator for restarts that change no binary.
Automating minor-version upgrades safely — the same loop with snapshots, binary swaps, upgradesstables, and rollback.
Node gossip & failure-detection protocols — how convergence and the failure detector decide when a restarted node is trusted again.
Read repair vs anti-entropy repair — why repair waits until the deployment is on a single version after an upgrade.

Related guides

Back to Node Lifecycle Automation