Compaction Strategy Cassandra Ops & Automation

Draining a Compaction Backlog on a Single Node in Cassandra 4.x/5.x

Sometimes exactly one node is the problem. A replica came back from a long GC pause, a disk swap left it behind on merging, or a hot partition landed its tokens on a single machine — and now that node’s PendingCompactions is climbing while every peer sits idle. This is a targeted, hands-on-keyboard drain of that one node, not a fleet-wide rollout. This guide is for the operator logged into a specific box who needs its queue down now, using node-local levers with full awareness of which ones are safe and which can make things worse. It assumes Cassandra 4.0, 4.1, or 5.0 and a nodetool shell on the affected host. It sits under Compaction Backlog Analysis & Alerting; the thresholds there tell you a node has crossed into backlog. Where the fleet-wide sibling resolving high compaction backlog without downtime coordinates I/O across a fleet with automated rollback, this page is the single-node tactical playbook: fewer moving parts, more direct commands, and the specific dangers of nodetool compact and nodetool stop that a one-node drain runs into.

Pre-conditions & safety gates

Before touching anything, confirm the backlog is genuinely on this node and identify the table driving it. Draining the wrong table wastes I/O; draining a corruption stall wastes it worse.

# 1. Confirm this node's queue depth and see the active compactions.
nodetool compactionstats --human-readable

Safety Check: Confirm nodetool status shows this node as UN. If ActiveTasks is 0 while pending tasks stays high, suspect a corrupt SSTable pinning the queue, not a throughput shortfall — do not raise throughput, jump to Troubleshooting. Expected Output:

pending tasks: 148
id        compaction type  keyspace  table        completed  total     unit   progress
a1b2...   Compaction       events    raw_ingest   40 GiB     210 GiB   bytes  19.04%

# 2. Find the runaway table by history — which table is generating the work.
cqlsh -e "SELECT keyspace_name, columnfamily_name, bytes_in, bytes_out, compacted_at \
  FROM system.compaction_history LIMIT 20;"

Safety Check: A single table dominating recent bytes_in is your drain target. On Cassandra 5.0 the same live state is in system_views.sstable_tasks without parsing text. Expected Output: Rows clustered on one keyspace/table are the signal.

# 3. Read current node-local settings so every change is reversible.
nodetool getcompactionthroughput
nodetool getconcurrentcompactors

Safety Check: Write these two values down. Every lever below is reverted to exactly these at the end. Expected Output: e.g. Current compaction throughput: 64 MB/s and 4.

Implementation: node-local drain levers, in order of safety

Apply these from safest to most aggressive, checking the queue between each. Do not skip to nodetool compact.

1. Raise compaction throughput on this node

The first and safest lever gives the existing compactors more I/O budget. It is instantaneous and fully reversible.

# Double the throughput ceiling on THIS node only (MB/s). 4.1+ accepts the same
# nodetool verb even though the cassandra.yaml key was renamed to compaction_throughput.
nodetool setcompactionthroughput 128

Safety Check: Keep the ceiling at or below ~50% of the array’s sustained write bandwidth. On a shared SSD, going higher steals I/O from the write path and induces WriteTimeoutException. Rollback Path: nodetool setcompactionthroughput 64 (or your recorded baseline).

2. Add compactor threads

If throughput is raised but a single compactor thread is the bottleneck on a multi-core box, add threads. This takes effect immediately without restart on 4.x/5.x.

# Add threads up to a cap; never exceed min(cores, disks) on standard hardware.
nodetool setconcurrentcompactors 6

Safety Check: More threads means more concurrent I/O streams; past min(cores, disks) they contend rather than help. Watch tpstats for the read/write pools starving. Rollback Path: nodetool setconcurrentcompactors 4.

3. A guarded drain loop

The script below applies the two safe levers, watches the queue drain, and auto-reverts if the extra I/O is not actually reducing pending work — the same self-check the fleet-wide runbook uses, scoped to one node.

#!/usr/bin/env python3
# requirements: Python 3.10+, nodetool on PATH, node in UN state.
"""Single-node compaction drain with guarded, reversible throughput/concurrency
bumps. Reverts automatically if the queue is not draining."""
import logging
import re
import subprocess
import sys
import time

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")


def nodetool(args: list[str], timeout: int = 20) -> tuple[int, str]:
    try:
        r = subprocess.run(["nodetool", *args], capture_output=True,
                           text=True, timeout=timeout, check=True)
        return 0, r.stdout.strip()
    except subprocess.CalledProcessError as e:
        logging.error("nodetool %s failed: %s", args, e.stderr.strip())
        return e.returncode, ""
    except subprocess.TimeoutExpired:
        logging.error("nodetool %s timed out", args)
        return -1, ""


def pending_tasks() -> int | None:
    """Parse the 'pending tasks:' line from compactionstats."""
    rc, out = nodetool(["compactionstats"])
    if rc != 0:
        return None
    m = re.search(r"pending tasks:\s*(\d+)", out)
    return int(m.group(1)) if m else None


def drain(target_throughput: int, target_concurrency: int,
          base_throughput: int, base_concurrency: int,
          cycles: int = 4, cycle_seconds: int = 60) -> bool:
    rc, _ = nodetool(["status"])
    if rc != 0:
        logging.critical("Node not reachable / not UN. Aborting.")
        return False

    nodetool(["setcompactionthroughput", str(target_throughput)])
    nodetool(["setconcurrentcompactors", str(target_concurrency)])
    logging.info("Applied %d MB/s, %d compactors on this node.",
                 target_throughput, target_concurrency)

    first = pending_tasks()
    if first is None:
        logging.warning("Cannot read pending tasks; reverting.")
        _revert(base_throughput, base_concurrency)
        return False

    for cycle in range(1, cycles + 1):
        time.sleep(cycle_seconds)
        now = pending_tasks()
        if now is None:
            continue
        logging.info("Cycle %d: pending %d -> %d", cycle, first, now)
        # If active tasks are working but pending is flat, suspect a pinned
        # (corrupt) SSTable rather than a throughput shortfall.
        if now >= first and cycle >= 2:
            logging.warning("Queue not draining; possible pinned SSTable. Reverting.")
            _revert(base_throughput, base_concurrency)
            return False
        if now <= max(base_concurrency * 2, 8):
            logging.info("Backlog drained to baseline. Reverting scaled settings.")
            _revert(base_throughput, base_concurrency)
            return True
    _revert(base_throughput, base_concurrency)
    return False


def _revert(throughput: int, concurrency: int) -> None:
    nodetool(["setcompactionthroughput", str(throughput)])
    nodetool(["setconcurrentcompactors", str(concurrency)])
    logging.info("Reverted to baseline: %d MB/s, %d compactors.", throughput, concurrency)


if __name__ == "__main__":
    ok = drain(target_throughput=128, target_concurrency=6,
               base_throughput=64, base_concurrency=4)
    sys.exit(0 if ok else 1)

Safety Check: The loop reverts on a flat queue (the corrupt-SSTable fingerprint), on unreadable stats, and on success — so the node never lingers on scaled settings. It caps nothing on its own; you pass the ceilings, so keep them under the array’s bandwidth. Expected Output:

2026-07-17 09:12:03 [INFO] Applied 128 MB/s, 6 compactors on this node.
2026-07-17 09:13:03 [INFO] Cycle 1: pending 148 -> 121
2026-07-17 09:14:03 [INFO] Cycle 2: pending 148 -> 79
2026-07-17 09:15:03 [INFO] Backlog drained to baseline. Reverting scaled settings.

4. nodetool compact — last resort, understand the cost

A user-defined major compaction forces the runaway table’s SSTables to merge now. On STCS it collapses all SSTables into one enormous file, which needs transient free space roughly equal to the table’s on-disk size and destroys the size tiers — after it, nothing compacts again until enough new SSTables accumulate to reach min_threshold, so read amplification can actually rise for a while. On LCS and TWCS it is far less disruptive. Use --user-defined to target specific SSTables when you can, and never run a full-table major on a nearly-full disk.

# Scope the major compaction to the one runaway table; splits large output on 4.x/5.x.
nodetool compact --split-output events raw_ingest

Safety Check: Confirm free space exceeds the table’s size via nodetool tablestats events.raw_ingest. Do not run during peak; a major compaction monopolizes I/O. Rollback Path: There is none once started — a major compaction cannot be undone, only stopped mid-flight (below), leaving partially merged output.

A note on why the single-node case is different from the fleet case: on one node you own the whole I/O budget and can push it hard, but you also have no peer to shed reads onto automatically. Raising throughput on one machine in a replica set of three means that machine is briefly the slowest of the three for reads, so the coordinator’s speculative retries and the dynamic snitch will already be steering some traffic away from it — which is fine and even helpful during a drain, as long as the other two replicas can absorb the shifted load. Confirm that headroom exists on the peers before you push this node’s compaction throughput to the ceiling; a drain that saturates one node while its two replicas are also near capacity simply moves the latency problem around the ring instead of resolving it.

5. nodetool stop — halt a runaway compaction

If a compaction is itself the problem — starving reads or heading toward disk exhaustion — stop the server-side operation. Detaching the client with Ctrl+C does not stop it.

# Halt in-flight compactions on this node (interrupts, does not corrupt).
nodetool stop COMPACTION

Safety Check: This interrupts active merges cleanly; the partial output SSTables are discarded and the source SSTables remain, so no data is lost, but the queue does not shrink. Use it to reclaim I/O for reads, then re-drain with the levers above. Rollback Path: Compaction resumes on its own as the strategy re-evaluates; there is nothing to restart.

Verification steps

# 1. Confirm the queue is back to baseline and settings are reverted.
nodetool compactionstats --human-readable
nodetool getcompactionthroughput

Safety Check: pending tasks should be at or near this node’s healthy floor, and throughput/concurrency should read the baseline values you recorded.

# 2. Confirm reads recovered on this node.
nodetool tpstats | grep -E "Pool Name|ReadStage|CompactionExecutor"

Safety Check: ReadStage Pending and Blocked should sit near zero. If they stayed elevated through the drain, the throughput bump was starving reads — lower the ceiling next time.

Troubleshooting

  • OutOfSpaceException mid-major-compaction. A nodetool compact on an STCS table fails partway because the single merged output needs more transient space than the disk has. Root cause: a full-table major was launched above ~50% disk utilization, where the output plus the still-present sources cannot coexist. Fix: never launch a full major without free space exceeding the table size; use --split-output and --user-defined to merge a subset, or archive cold data first. Recover by letting nodetool stop COMPACTION discard the partial output, then drain with throughput/concurrency instead of a major.
  • A single corrupt SSTable pinning the queue. ActiveTasks sits at 0 or 1 while pending tasks stays flat-high, and debug.log shows CorruptSSTableException on the same file each cycle. Root cause: one unreadable SSTable fails its compaction repeatedly and blocks the task behind it — no amount of extra throughput helps. Fix: stop feeding it I/O, run nodetool scrub events raw_ingest (or nodetool stop COMPACTION then offline sstablescrub) to quarantine the bad file during a maintenance window, and categorize it via compaction error categorization & logging before resuming the drain.
  • Throughput bump starving reads (ReadTimeoutException). After raising throughput and compactors, application reads to this node start timing out. Root cause: compaction I/O oversubscribed the shared device, leaving nothing for the read path. Fix: drop the ceiling back toward 50% of sustained write bandwidth, reduce compactors toward min(cores, disks), and while the drain runs keep reads off this replica using the speculative_retry and routing techniques in fallback routing & read path optimization.
Back to Compaction Backlog Analysis Alerting