Orchestrating Rolling Restarts with Python for Cassandra 4.x/5.x

When a cassandra.yaml edit, a JVM flag change, or a certificate rotation needs to reach every node, a hand-run rolling restart is tedious and easy to botch under pressure — one operator advancing to the next node before gossip settles is all it takes to break quorum. This guide builds a Python 3.10+ orchestrator that drives the restart as a health-gated, resumable loop across the deployment, one node at a time, in rack-aware order. It is the automation counterpart to the rolling restarts and minor upgrades guide; read that first for the drain-before-stop discipline and the gossip-convergence rules this script enforces. Crucially, this orchestrator restarts nodes without changing the binary — no package swap, no upgradesstables. If you are changing versions, use the minor-version upgrade runbook instead, because a mixed-version window carries constraints this script does not handle.

Prerequisites: passwordless SSH to every node as a user that can run sudo systemctl and nodetool, Python 3.10+ on the host that runs the orchestrator, and a deployment that is fully UN before you begin.

Pre-conditions & safety gates

The orchestrator refuses to start unless the deployment is healthy, and refuses to advance unless the node it just cycled has genuinely rejoined. Verify the gates by hand once before trusting the automation.

# 1. Whole ring must be UN. Any DN/UJ/UL aborts.
nodetool status | awk 'NR>5 {print $1, $2}'

Expected output: every data row begins UN. A DN, UJ, or UL prefix means stop and fix first.

# 2. No active repair or streaming — a restart during streaming can abort a session.
nodetool netstats | grep -iE "repair|Receiving|Sending"

Expected output: empty, or only Not sending any streams / Not receiving any streams.

# 3. Hints already drained cluster-wide before you add any downtime.
nodetool tpstats | grep -i "HintsDispatcher"

Expected output: Active and Pending both 0 for HintsDispatcher.

If all three pass, the deployment is a safe baseline. The orchestrator re-checks gate 1 for every node and re-checks hints after each restart before advancing.

Implementation

The script below reads a node inventory (address plus rack), visits nodes in rack-aware order so it never cycles two nodes of the same rack back to back, and for each node runs drain → stop → start over SSH, then polls until the node is UN, gossip has settled, and hints have drained. It writes a JSON state file after every node so an interrupted run resumes where it left off instead of restarting nodes that are already done. Every SSH call is bounded by a timeout, and the health poll uses capped exponential backoff.

#!/usr/bin/env python3
# requirements: Python 3.10+, OpenSSH client on PATH, passwordless sudo for
# systemctl/nodetool on every target node. No third-party packages.
"""Resumable, rack-aware rolling RESTART orchestrator for Cassandra 4.x/5.x.

Restarts every node one at a time with no binary change. For each node:
drain -> stop -> start -> wait until UN + gossip settled + hints drained.
State is checkpointed to disk so an interrupted run resumes safely.
"""
from __future__ import annotations

import json
import subprocess
import sys
import time
from dataclasses import dataclass, field
from pathlib import Path

STATE_FILE = Path("rolling_restart_state.json")
SSH_TIMEOUT = 30           # seconds per remote command
STOP_SETTLE = 5            # grace after stop before start
POLL_MAX_ATTEMPTS = 40     # health-poll attempts per node
POLL_BASE_DELAY = 3.0      # seconds, grows with backoff
POLL_MAX_DELAY = 20.0      # backoff ceiling


@dataclass
class Node:
    address: str
    rack: str


@dataclass
class RunState:
    completed: set[str] = field(default_factory=set)

    @classmethod
    def load(cls) -> "RunState":
        if STATE_FILE.exists():
            data = json.loads(STATE_FILE.read_text())
            return cls(completed=set(data.get("completed", [])))
        return cls()

    def mark(self, address: str) -> None:
        self.completed.add(address)
        STATE_FILE.write_text(json.dumps({"completed": sorted(self.completed)}, indent=2))


def ssh(address: str, command: str) -> tuple[int, str]:
    """Run a command on a remote node; return (exit_code, combined_output)."""
    try:
        proc = subprocess.run(
            ["ssh", "-o", "BatchMode=yes", "-o", f"ConnectTimeout=10", address, command],
            capture_output=True, text=True, timeout=SSH_TIMEOUT,
        )
        return proc.returncode, (proc.stdout + proc.stderr).strip()
    except subprocess.TimeoutExpired:
        return -1, f"ssh timeout after {SSH_TIMEOUT}s"


def rack_aware_order(nodes: list[Node]) -> list[Node]:
    """Interleave racks so consecutive nodes never share a rack when avoidable."""
    by_rack: dict[str, list[Node]] = {}
    for n in nodes:
        by_rack.setdefault(n.rack, []).append(n)
    queues = [iter(v) for v in by_rack.values()]
    ordered: list[Node] = []
    exhausted = 0
    while exhausted < len(queues):
        exhausted = 0
        for q in queues:
            nxt = next(q, None)
            if nxt is None:
                exhausted += 1
            else:
                ordered.append(nxt)
    return ordered


def node_is_un(observer: str, target_ip: str) -> bool:
    """True only if a PEER (not the target) reports the target as UN."""
    rc, out = ssh(observer, "nodetool status")
    if rc != 0:
        return False
    for line in out.splitlines():
        parts = line.split()
        if len(parts) >= 2 and target_ip in parts[1]:
            return parts[0] == "UN"
    return False


def gossip_settled(observer: str, target_ip: str) -> bool:
    """True when a peer sees the target's gossip STATUS as NORMAL."""
    rc, out = ssh(observer, f"nodetool gossipinfo")
    if rc != 0:
        return False
    block = ""
    capture = False
    for line in out.splitlines():
        if line.startswith("/") and target_ip in line:
            capture = True
        elif line.startswith("/") and capture:
            break
        if capture:
            block += line + "\n"
    return "STATUS:" in block and "NORMAL" in block


def hints_drained(observer: str) -> bool:
    """True when the observer has no pending hint dispatch for peers."""
    rc, out = ssh(observer, "nodetool tpstats")
    if rc != 0:
        return False
    for line in out.splitlines():
        if line.startswith("HintsDispatcher"):
            cols = line.split()
            # HintsDispatcher  Active  Pending  Completed  Blocked ...
            return cols[1] == "0" and cols[2] == "0"
    return True  # pool absent => nothing pending


def wait_healthy(target: Node, observers: list[str]) -> bool:
    """Poll peers with capped exponential backoff until the target is fully back."""
    delay = POLL_BASE_DELAY
    for attempt in range(1, POLL_MAX_ATTEMPTS + 1):
        un = all(node_is_un(o, target.address) for o in observers)
        gossip = all(gossip_settled(o, target.address) for o in observers)
        hints = all(hints_drained(o) for o in observers)
        if un and gossip and hints:
            print(f"  [{target.address}] healthy (UN + gossip + hints) after {attempt} checks")
            return True
        print(f"  [{target.address}] not ready (un={un} gossip={gossip} hints={hints}); "
              f"retry in {delay:.0f}s")
        time.sleep(delay)
        delay = min(delay * 1.6, POLL_MAX_DELAY)
    return False


def restart_node(target: Node) -> bool:
    """drain -> stop -> start on a single node. Idempotent: safe to re-run."""
    print(f"  [{target.address}] nodetool drain")
    rc, out = ssh(target.address, "nodetool drain")
    if rc != 0:
        print(f"  [{target.address}] drain failed: {out}")
        return False
    ssh(target.address, "sudo systemctl stop cassandra")
    time.sleep(STOP_SETTLE)
    rc, out = ssh(target.address, "sudo systemctl start cassandra")
    if rc != 0:
        print(f"  [{target.address}] start failed: {out}")
        return False
    return True


def roll(nodes: list[Node]) -> int:
    state = RunState.load()
    ordered = rack_aware_order(nodes)
    all_ips = [n.address for n in ordered]

    for target in ordered:
        if target.address in state.completed:
            print(f"[{target.address}] already done — skipping (resumable)")
            continue

        # Observers = every OTHER node; a node cannot vouch for itself.
        observers = [ip for ip in all_ips if ip != target.address]

        # Gate: whole ring healthy before we take this node down.
        if not all(node_is_un(observers[0], ip) for ip in all_ips if ip != target.address):
            print(f"[{target.address}] ABORT: ring not fully UN before restart")
            return 1

        print(f"[{target.address}] rack={target.rack} restarting")
        if not restart_node(target):
            print(f"[{target.address}] ABORT: restart step failed")
            return 1

        if not wait_healthy(target, observers):
            print(f"[{target.address}] ABORT: node did not rejoin in time")
            return 1

        state.mark(target.address)
        print(f"[{target.address}] committed to state; advancing\n")

    print("Rolling restart complete for all nodes.")
    STATE_FILE.unlink(missing_ok=True)
    return 0


if __name__ == "__main__":
    # Inventory: (address, rack). Replace with your own or load from a file.
    inventory = [
        Node("10.0.1.11", "rack1"),
        Node("10.0.1.21", "rack2"),
        Node("10.0.1.31", "rack3"),
        Node("10.0.1.12", "rack1"),
        Node("10.0.1.22", "rack2"),
        Node("10.0.1.32", "rack3"),
    ]
    sys.exit(roll(inventory))

Two design choices carry the safety guarantee. First, health is judged only by peers, never by the node under restart — node_is_un and gossip_settled always run nodetool from another host, because a node reporting UN about itself does not prove the ring agrees. Second, the state file is written the instant a node is confirmed healthy, so a Ctrl+C, a dropped SSH session, or a crashed orchestrator resumes at the next unfinished node rather than re-restarting nodes that already cycled.

The backoff in wait_healthy is deliberately capped rather than unbounded: an early failure (say, gossip has not yet propagated) is retried quickly, but repeated failures widen the interval up to POLL_MAX_DELAY so the orchestrator does not hammer peers with nodetool calls while a slow node catches up. Because restart_node is idempotent — draining an already-drained node and starting an already-running service are both harmless — re-invoking the script after a mid-node interruption simply re-drives the same node to a healthy state before the state file advances it. Tune the inventory order to your topology: list one node per rack before returning to the first rack, exactly as the sample inventory does, so rack_aware_order never places two same-rack nodes consecutively even under an uneven rack count.

Verification

After the run reports completion, confirm the deployment converged and no client-visible damage occurred.

# All nodes UN from an arbitrary node's perspective.
nodetool status

Expected output: every row UN, and the rolling_restart_state.json file removed by the successful run.

# No dropped mutations accumulated during the roll.
nodetool tpstats | grep -iE "Dropped|MUTATION"

Expected output: dropped counters flat versus your pre-run baseline.

# Schema agreement across the deployment — one schema version only.
nodetool describecluster | grep -A5 "Schema versions"

Expected output: a single schema UUID mapping to all node addresses.

Troubleshooting

A node fails to rejoin and describecluster shows a schema disagreement. After restart the node is UN but nodetool describecluster lists two schema UUIDs, and the orchestrator’s wait_healthy may still pass on ring state while the node lags. Root cause: the restarted node came up before fully replaying a schema migration, or a config edit changed something schema-affecting. Fix: give gossip more time — the node usually reconciles within a minute or two as it pulls the schema from peers; if it persists past several minutes, restart that single node again (the orchestrator will skip the already-completed ones) and confirm describecluster collapses to one UUID before doing anything else. Never advance the loop while two schema versions exist.
Hints pile up on the node that was just down. hints_drained keeps returning False and the poll exhausts its attempts. Root cause: while the node was stopped, its replica peers buffered hints, and on a write-heavy cluster the HintsDispatcher needs time to replay them; advancing now would stack a second node’s downtime on top of an incomplete catch-up. Fix: the orchestrator already blocks on this — let it wait. If hints never drain, check that hinted_handoff_enabled is true and that the node is actually reachable for hint delivery; a network ACL that blocks the storage port lets gossip pass but stops hint replay, leaving the pool pending forever.
Gossip briefly marks the restarted node DOWN, then UP again. In the first seconds after systemctl start, peers may still be convicting the old process and flip the node DN before the new process’s heartbeats reconverge, causing node_is_un to fail intermittently. Root cause: the failure detector needs a few heartbeat intervals to trust the restarted node, especially on a busy cluster. Fix: this is exactly why the poll uses capped exponential backoff rather than a single check — transient flapping resolves on its own. If a node flaps persistently, raise phi_convict_threshold (for example 8 → 12) to make the detector more tolerant during restarts, per the guidance in the node gossip and failure-detection guide.

Rolling restarts & minor-version upgrades — the parent guide with the drain-before-stop discipline and gossip-convergence rules this orchestrator enforces.
Automating minor-version upgrades safely — the version-changing variant, with snapshots, binary swaps, and upgradesstables.
Node gossip & failure-detection protocols — how convergence and phi_convict_threshold govern the DOWN/UP flapping the poll rides out.
Node Lifecycle Automation — the top-level guide to state-gated membership operations.

Back to Rolling Restarts and Upgrades