Compaction Strategy Cassandra Ops & Automation

Automating Minor-Version Upgrades Safely in Cassandra 4.x/5.x

A minor-version (patch) upgrade — moving a deployment from 4.1.3 to 4.1.5, or 5.0.1 to 5.0.3 — looks routine, but it changes the binary on disk and therefore drags the deployment through a mixed-version window where streaming between differing versions is disabled and schema changes are refused. The two rules that keep it safe are simple and non-negotiable: snapshot before you touch a node, and take one node through the whole upgrade at a time. This guide automates that discipline. It extends the rolling restarts and minor upgrades guide, which covers the loop mechanics; here the loop additionally swaps the package, verifies the reported version, runs nodetool upgradesstables, and can roll a node back from its pre-upgrade snapshot. Unlike the pure restart orchestrator, which changes no binary, this procedure must guard against every other lifecycle operation running during the window: no repair, no bootstrap, no decommission until the last node is on the new version.

Prerequisites: the new package staged and reachable on every node, passwordless SSH with sudo, Python 3.10+ on the orchestration host, and a verified backup path.

Pre-conditions & safety gates

Upgrades are the one lifecycle operation where a bad step can lose data, so the gates are stricter than for a plain restart. Run all of them before starting, and re-take the snapshot per node inside the loop.

# 1. Confirm the source version and that every node is on it and UN.
nodetool version && nodetool status | awk 'NR>5 {print $1, $2}'

Expected output: one identical ReleaseVersion intended as the source, and every status row prefixed UN.

# 2. Verify a recoverable backup exists BEFORE any change.
#    Take a fresh snapshot and confirm it landed on disk.
nodetool snapshot -t pre_upgrade_$(date +%Y%m%d) ledger
ls -d /var/lib/cassandra/data/ledger/*/snapshots/pre_upgrade_* | head

Expected output: snapshot directories listed under each table of the ledger keyspace; an empty result means the snapshot failed and you must stop.

# 3. Check release compatibility: minor upgrades are only safe within a major line
#    and along the vendor's supported upgrade path (e.g. 4.1.x -> 4.1.y, or 4.1 -> 5.0).
#    Never skip a major (e.g. 3.11 -> 5.0) in one hop.
nodetool describecluster | grep -A5 "Schema versions"

Expected output: a single schema UUID for the whole cluster. A schema disagreement must be resolved before upgrading — you cannot start an upgrade from an inconsistent schema.

Only proceed when all three pass. The snapshot in gate 2 is the rollback anchor for the entire operation; SSTable snapshots are hard links, so they cost almost no disk until compaction diverges the live files, making a per-node snapshot cheap insurance.

Implementation

The upgrade is expressed as a per-node bash function (the node-local work) wrapped by a Python coordinator that enforces global ordering and refuses to run if any other lifecycle operation is active. The bash function does exactly six things in order — drain, snapshot, stop, swap package, start, verify version, then upgradesstables — and is idempotent enough to re-run on a partially-upgraded node.

#!/usr/bin/env bash
# upgrade_node.sh — run ON a single Cassandra node. Args: <new_pkg_version>
# Idempotent-ish: skips the package swap if already on the target version.
set -euo pipefail

TARGET="$1"                       # e.g. 4.1.5
KEYSPACES="ledger analytics"      # snapshot these; adjust to your keyspaces
SNAP="preupg_$(date +%Y%m%d_%H%M%S)"

current="$(nodetool version | awk -F': ' '/ReleaseVersion/ {print $2}')"
if [[ "$current" == "$TARGET" ]]; then
  echo "already on $TARGET — running upgradesstables only"
  nodetool upgradesstables
  exit 0
fi

echo "[1/6] draining"
nodetool drain

echo "[2/6] snapshot $SNAP"
# shellcheck disable=SC2086
nodetool snapshot -t "$SNAP" $KEYSPACES

echo "[3/6] stopping service"
sudo systemctl stop cassandra

echo "[4/6] swapping package to $TARGET"
# Debian/Ubuntu shown; use the matching yum/dnf line on RHEL-family hosts.
sudo apt-get install -y --allow-downgrades "cassandra=$TARGET"

echo "[5/6] starting service"
sudo systemctl start cassandra

# Wait for the process to answer nodetool before verifying.
for _ in $(seq 1 30); do
  if nodetool version >/dev/null 2>&1; then break; fi
  sleep 4
done

verified="$(nodetool version | awk -F': ' '/ReleaseVersion/ {print $2}')"
if [[ "$verified" != "$TARGET" ]]; then
  echo "VERSION MISMATCH: expected $TARGET, got $verified" >&2
  exit 3
fi
echo "version verified: $verified"

echo "[6/6] upgradesstables (I/O heavy; safe while serving)"
nodetool upgradesstables
echo "node upgrade complete"

The Python coordinator below drives that script across the deployment one node at a time, blocks until each node returns to UN on its peers, and — the upgrade-specific guard — aborts the whole run if it ever detects an active repair, bootstrap, or decommission anywhere in the deployment while the versions are mixed.

#!/usr/bin/env python3
# requirements: Python 3.10+, OpenSSH client, upgrade_node.sh staged on every node.
"""One-at-a-time minor-version upgrade coordinator for Cassandra 4.x/5.x.

Guards the mixed-version window: refuses to advance if any other lifecycle
operation (repair, bootstrap, decommission) is detected mid-upgrade.
"""
from __future__ import annotations
import subprocess, sys, time

TARGET_VERSION = "4.1.5"
NODES = ["10.0.2.11", "10.0.2.12", "10.0.2.13", "10.0.2.14"]  # upgrade order
SSH_TIMEOUT = 900  # upgradesstables can run long


def ssh(host: str, cmd: str, timeout: int = 60) -> tuple[int, str]:
    try:
        p = subprocess.run(["ssh", "-o", "BatchMode=yes", host, cmd],
                           capture_output=True, text=True, timeout=timeout)
        return p.returncode, (p.stdout + p.stderr).strip()
    except subprocess.TimeoutExpired:
        return -1, "ssh timeout"


def lifecycle_clear(observer: str) -> bool:
    """Abort guard: no repair/streaming/joining/leaving may run mid-upgrade."""
    rc, net = ssh(observer, "nodetool netstats")
    if rc != 0 or "Repair" in net or "Receiving" in net or "Sending" in net:
        return False
    rc, st = ssh(observer, "nodetool status")
    if rc != 0:
        return False
    # No node may be joining (UJ) or leaving (UL/DL) during the window.
    for line in st.splitlines():
        code = line.split()[0] if line.split() else ""
        if code in {"UJ", "UL", "DL", "DJ"}:
            return False
    return True


def peer_sees_un(observer: str, target: str) -> bool:
    rc, out = ssh(observer, "nodetool status")
    if rc != 0:
        return False
    for line in out.splitlines():
        parts = line.split()
        if len(parts) >= 2 and target in parts[1]:
            return parts[0] == "UN"
    return False


def wait_un(target: str, observers: list[str], attempts: int = 45) -> bool:
    delay = 4.0
    for _ in range(attempts):
        if all(peer_sees_un(o, target) for o in observers):
            return True
        time.sleep(delay)
        delay = min(delay * 1.5, 25.0)
    return False


def upgrade() -> int:
    for target in NODES:
        observers = [n for n in NODES if n != target]

        # Guard the mixed-version window before touching the node.
        if not all(lifecycle_clear(o) for o in observers):
            print(f"ABORT before {target}: another lifecycle op is active")
            return 1

        print(f"[{target}] upgrading to {TARGET_VERSION}")
        rc, out = ssh(target, f"sudo bash /opt/ops/upgrade_node.sh {TARGET_VERSION}",
                      timeout=SSH_TIMEOUT)
        print(out)
        if rc != 0:
            print(f"[{target}] upgrade_node.sh failed rc={rc} — halting; roll back this node")
            return 1

        if not wait_un(target, observers):
            print(f"[{target}] did not return to UN — halting")
            return 1
        print(f"[{target}] on {TARGET_VERSION} and UN\n")

    print("All nodes upgraded. Mixed-version window closed; repair is safe again.")
    return 0


if __name__ == "__main__":
    sys.exit(upgrade())

The coordinator never proceeds while the window is unsafe: lifecycle_clear runs before each node and treats any active repair stream, or any node in a joining/leaving state, as a hard abort. That is what enforces the “no repair, bootstrap, or decommission during a mixed-version window” rule in code rather than in a runbook someone might skip.

Two properties keep the run recoverable. The per-node bash script short-circuits when a node is already on the target version — it runs only upgradesstables and exits — so re-invoking the coordinator after an interruption never re-swaps a package that already moved, and the SSH_TIMEOUT of 900 seconds gives upgradesstables room to finish on nodes with large data volumes without the coordinator giving up prematurely. The snapshot taken inside step 2 of the bash script is per-node and timestamped, so every node carries its own rollback anchor; you are never dependent on a single cluster-wide snapshot that may have aged out. Stagger the actual upgradesstables I/O if disk pressure is a concern — it does not block reads or writes, so a node can serve traffic while its SSTables rewrite in the background, and you can even defer it to a quieter window as long as you complete it before the next major upgrade.

Verification

After the coordinator reports completion, confirm the whole cluster is homogeneous and the on-disk format has been rewritten.

# Every node reports the target ReleaseVersion.
for h in 10.0.2.11 10.0.2.12 10.0.2.13 10.0.2.14; do
  echo -n "$h "; ssh "$h" "nodetool version | awk -F': ' '/ReleaseVersion/{print \$2}'"
done

Expected output: the target version (e.g. 4.1.5) on every line.

# One schema version, and no lingering old-format SSTables.
nodetool describecluster | grep -A5 "Schema versions"
nodetool compactionstats     # upgradesstables work should be finished, not queued

Expected output: a single schema UUID; compactionstats showing no pending upgrade tasks. Only now is it safe to resume repair — see the read-repair versus anti-entropy repair guide for scheduling it after the window closes.

Troubleshooting

  • Cannot stream / streaming disabled between mixed versions. During the window, a node on the new binary refuses to stream to or from a node on the old one, and any operation that needs streaming (a repair someone kicked off, a replacement, a bootstrap) logs Cannot stream or hangs. Root cause: internode streaming is intentionally disabled across differing versions to prevent format corruption. Fix: do not stream during the window — the coordinator’s lifecycle_clear guard exists precisely to stop this. If a repair is already stuck, cancel it (nodetool stop VALIDATION), finish upgrading every node, then run repair once the deployment is homogeneous.

  • SSTable format incompatibility after start. A freshly-upgraded node starts but logs warnings about reading old-format SSTables, or reads on it are unusually slow because the data is still in the previous major’s format. Root cause: the new binary reads the old format for compatibility but has not yet rewritten it. Fix: this is expected until nodetool upgradesstables completes on that node — let it finish (it appears in nodetool compactionstats). Never downgrade a node once new-format SSTables have been written, because older binaries cannot read the newer format; that path is one-way and is why the snapshot is your only clean rollback.

  • A node fails the upgrade and must roll back from snapshot. The package swap succeeds but the node crash-loops on the new version, or the version verification fails. Root cause: a bad package, a config incompatibility, or a corrupted install. Fix: roll that single node back before touching any other node. Stop it, reinstall the previous package version, restore the pre-upgrade snapshot into the table directories, and restart:

    sudo systemctl stop cassandra
sudo apt-get install -y --allow-downgrades cassandra=4.1.3   # previous version
# For each table, clear live SSTables and hard-link the snapshot back in:
#   (do this per table dir under /var/lib/cassandra/data/ledger/<table>-<id>/)
nodetool refresh ledger transactions   # after copying snapshot files into place
sudo systemctl start cassandra
nodetool version   # confirm it is back on the source version and UN

    Because you only ever have one node in flight, a rollback affects a single replica and the deployment keeps serving. The mechanics of the tombstones and shadowed data that a snapshot restore can resurrect are covered under tombstone management and garbage collection.

Back to Rolling Restarts and Upgrades