Python Script for Tracking Compaction Throughput in Cassandra 4.x/5.x

The configured compaction_throughput ceiling tells you what compaction is allowed to consume; it says nothing about what compaction is actually achieving. On a node whose disk is saturated by reads, repair streaming, or a slow controller, real throughput can sit far below the ceiling even while the queue backs up — and a ceiling raised in response does nothing because the limiter was never the bottleneck. This page delivers a complete Python 3.10+ script that reads the JMX BytesCompacted counter, differentiates it into an actual megabytes-per-second figure, and expresses that as a percentage of the configured ceiling so you can tell “throttle-bound” from “hardware-bound” at a glance. It sits under Python Monitoring for Cassandra Compaction; read that first for the JMX-versus-nodetool telemetry model, then use this script when your specific question is how close to the ceiling is compaction running right now. Prerequisites: Cassandra 4.0, 4.1, or 5.0, a reachable JMX port, Python 3.10+, and the jmxquery package.

Pre-conditions & safety gates

Every check below is read-only. Run them on one node before deploying, and stop if any gate fails.

1. JMX reachable and the counter exists

# Confirms the Compaction MBean and its BytesCompacted attribute are exposed.
timeout 10 nodetool sjk mx -b "org.apache.cassandra.metrics:type=Compaction,name=BytesCompacted" -f Count 2>/dev/null \
  && echo "PASS: BytesCompacted readable" || echo "CHECK: use jmxquery fallback below"

Safety Check: nodetool sjk ships with Cassandra 4.x/5.x and reads the MBean without extra tooling; the 10-second timeout stops the gate hanging on a saturated JMX thread pool. If sjk is unavailable the Python script reads the same attribute directly. Expected Output: PASS: BytesCompacted readable.

2. Read the configured ceiling

# The ceiling the derived rate is measured against. "0 MB/s" means UNLIMITED.
nodetool getcompactionthroughput

Safety Check: Read-only. On 4.1+/5.0 the YAML key is compaction_throughput; on 4.0 and earlier it is compaction_throughput_mb_per_sec, but nodetool getcompactionthroughput prints identically across versions. A value of 0 MB/s means unlimited, not zero — the script treats it as no ceiling. Expected Output: e.g. Current compaction throughput: 64 MB/s.

3. Python runtime and driver package

python3 -c "import sys; assert sys.version_info >= (3,10); import jmxquery; print('PASS: runtime + jmxquery ready')"

Safety Check: Confirms 3.10+ for the type-hint syntax and that jmxquery is importable (pip install jmxquery). No cluster state is touched. Expected Output: PASS: runtime + jmxquery ready.

Only proceed once all three pass. Because a throughput figure that pins the ceiling for hours is the signature of a throttle-bound node, this signal pairs directly with the depth-and-velocity thresholds in compaction backlog analysis & alerting.

Implementation

BytesCompacted is a monotonically increasing counter of total bytes written by compaction since the JVM started. Instantaneous throughput is its first derivative: sample the counter, wait a fixed interval, sample again, and divide the byte delta by the elapsed seconds. Dividing that rate by the configured ceiling yields utilization. The script keeps only the previous sample in memory, runs a bounded number of cycles, and guards against the two arithmetic traps unique to a monotonic counter — a negative delta (the JVM restarted and the counter reset) and a zero ceiling (unlimited, so utilization is undefined).

The reason this measurement is worth the effort is that utilization separates two failure modes that look identical from the queue alone. A node running at 95%+ of its ceiling with a growing backlog is throttle-bound: compaction wants to do more work than compaction_throughput permits, and raising the ceiling (or adding concurrent_compactors) will actually help. A node running at 40% of its ceiling with the same growing backlog is hardware-bound: the disk, the controller, or a competing workload such as repair streaming is the real limiter, and raising the throttle changes nothing because compaction never reaches it. Queue depth cannot tell these apart; the ratio of measured throughput to the ceiling can. That single distinction is what stops teams from repeatedly bumping compaction_throughput on a node whose bottleneck was never the throttle.

A note on units. Cassandra 4.1+ labels the throttle internally in mebibytes per second (MiB/s), while nodetool getcompactionthroughput historically printed the suffix MB/s; the numeric value is the same figure the limiter enforces, so the script parses the number and treats it as MiB/s to stay consistent with the byte-delta math (BytesCompacted is a raw byte count). Do not convert between MB and MiB in the comparison — both sides of the ratio must use the same base, and keeping everything in binary units against the raw counter avoids a silent 4.8% error.

Save the following as compaction_throughput_tracker.py.

#!/usr/bin/env python3
# requirements: Python 3.10+, jmxquery>=0.6.0, nodetool on PATH
"""Track actual compaction throughput on Cassandra 4.x/5.x.

Differentiates the JMX BytesCompacted counter into MB/s and expresses it as a
percentage of the configured compaction_throughput ceiling. Read-only, bounded,
in-memory only."""
from __future__ import annotations

import argparse
import subprocess
import sys
import time
from dataclasses import dataclass

from jmxquery import JMXConnection, JMXQuery

BYTES_COMPACTED_MBEAN = (
    "org.apache.cassandra.metrics:type=Compaction,name=BytesCompacted/Count"
)
MIB = 1024 * 1024


@dataclass
class Sample:
    """One reading of the monotonic BytesCompacted counter."""
    bytes_compacted: int
    monotonic_ts: float


def read_ceiling_mibps(nodetool: str = "nodetool") -> float | None:
    """Return the configured ceiling in MiB/s, or None when unlimited (0 MB/s)."""
    out = subprocess.run(
        [nodetool, "getcompactionthroughput"],
        capture_output=True, text=True, timeout=15, check=True,
    ).stdout
    # Line: "Current compaction throughput: 64 MB/s"
    for token in out.split():
        if token.replace(".", "", 1).isdigit():
            value = float(token)
            return None if value == 0 else value   # 0 == unlimited, not zero
    return None


def read_bytes_compacted(conn: JMXConnection) -> Sample:
    """Read the current BytesCompacted counter via JMX."""
    result = conn.query([JMXQuery(BYTES_COMPACTED_MBEAN)])
    if not result:
        raise RuntimeError("BytesCompacted MBean returned no value")
    return Sample(bytes_compacted=int(result[0].value), monotonic_ts=time.monotonic())


def compute_rate_mibps(prev: Sample, cur: Sample) -> float | None:
    """Derive MiB/s from two counter samples. None if the counter reset."""
    elapsed = cur.monotonic_ts - prev.monotonic_ts
    delta = cur.bytes_compacted - prev.bytes_compacted
    if delta < 0:
        # Counter went backwards => JVM restarted and BytesCompacted reset.
        return None
    if elapsed <= 0:
        return 0.0
    return (delta / elapsed) / MIB


def run(jmx_url: str, interval: float, cycles: int) -> int:
    conn = JMXConnection(jmx_url)
    ceiling = read_ceiling_mibps()
    prev: Sample | None = None
    for _ in range(cycles):                     # bounded: never an infinite loop
        try:
            cur = read_bytes_compacted(conn)
        except Exception as exc:                # keep running across a transient JMX blip
            print(f"WARN: JMX read failed, skipping cycle: {exc}", file=sys.stderr)
            time.sleep(interval)
            continue
        if prev is not None:
            rate = compute_rate_mibps(prev, cur)
            if rate is None:
                print("WARN: counter reset (JVM restart); re-baselining.")
            else:
                if ceiling is None:
                    pct = "n/a (unlimited)"
                else:
                    pct = f"{(rate / ceiling) * 100:.1f}%"
                print(
                    f"throughput={rate:.2f} MiB/s  ceiling="
                    f"{'unlimited' if ceiling is None else f'{ceiling:.0f} MiB/s'}  "
                    f"utilization={pct}"
                )
        prev = cur
        time.sleep(interval)
    return 0


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--jmx-url", default="service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi")
    ap.add_argument("--interval", type=float, default=10.0)
    ap.add_argument("--cycles", type=int, default=360)   # ~1h at 10s; bounded run
    args = ap.parse_args()
    return run(args.jmx_url, args.interval, args.cycles)


if __name__ == "__main__":
    sys.exit(main())

Safety Check: The loop is bounded by --cycles, so it cannot run forever; a negative counter delta is detected and re-baselined instead of reporting a nonsense negative rate; a zero ceiling is treated as unlimited rather than dividing by zero; every JMX read is wrapped so a transient failure skips one cycle instead of crashing. Expected Output: One line per cycle after the first, e.g. throughput=41.30 MiB/s ceiling=64 MiB/s utilization=64.5%.

The first cycle produces no output by design: with only one sample there is no delta to differentiate, so prev is None and the script simply records the baseline. This is why the default --cycles 360 at a 10-second interval gives roughly an hour of readings — the first is consumed establishing the baseline. Choose the interval deliberately: too short and the byte delta is dominated by the granularity of the counter update and reads as jitter; too long and a brief throttle-bound burst averages out into an innocuous-looking number. Ten to thirty seconds is a sound range for a node under steady write pressure, matching the scrape cadence you would use if you fed the same counter through a Prometheus exporter rather than reading JMX directly.

For continuous operation, wrap the bounded run in a service manager that restarts it on exit rather than converting the loop to while True. A bounded loop that a supervisor relaunches is easier to reason about than an unbounded one: each run has a defined lifetime, a restart cleanly re-establishes the JMX connection and re-reads the ceiling (which an operator may have changed with nodetool setcompactionthroughput), and there is no long-lived process to leak file descriptors or drift its baseline across a counter reset it failed to notice.

Verification steps

Confirm the derived figure agrees with Cassandra’s own accounting before trusting it.

# 1. The reported ceiling must match what the script prints.
nodetool getcompactionthroughput

Safety Check: The ceiling= field in the output must equal this value (converting MB/s to MiB/s is a no-op here since Cassandra labels the throttle in MiB/s internally on 4.1+). Expected Output: Matching ceiling; utilization at or below 100% during normal operation.

# 2. Cross-check that compaction is actually active while the rate is non-zero.
nodetool compactionstats -H

Safety Check: A non-zero throughput line should coincide with active compaction rows here. If the script shows MiB/s flowing but this shows no active tasks, your MBean path is wrong (see troubleshooting). Reading these columns is covered in interpreting nodetool compactionstats output. Expected Output: Active compaction rows present whenever throughput is non-zero.

# 3. Watch utilization trend over a few cycles under load.
python3 compaction_throughput_tracker.py --interval 5 --cycles 12

Safety Check: Sustained utilization near 100% means compaction is throttle-bound and raising compaction_throughput may help; utilization well below 100% while the queue grows means the node is hardware-bound and raising the ceiling will not. Expected Output: A short series of throughput/utilization lines that stabilize under steady load.

Troubleshooting

Throughput prints a negative or absurdly large value once, then recovers. The BytesCompacted counter reset because the Cassandra JVM restarted between samples, so the byte delta went negative (or the first post-restart sample straddled the reset). Root cause: BytesCompacted is monotonic only within a single JVM lifetime. Fix: the script already detects delta < 0 and re-baselines with a counter reset warning instead of emitting the bad value — if you adapted the math, always guard the delta for negativity rather than assuming monotonicity across restarts.
The script reports 0.00 MiB/s forever even though nodetool compactionstats shows active work. The MBean object name or attribute is not matching your exporter’s or JVM’s naming. Root cause: an exporter metric-name mismatch or a wrong BYTES_COMPACTED_MBEAN path — some setups expose the attribute as Count, others surface a pre-aggregated gauge, and Prometheus-normalized names (cassandra_compaction_bytescompacted) differ from the raw JMX object name used here. Fix: enumerate the actual MBeans with nodetool sjk mx -b "org.apache.cassandra.metrics:type=Compaction,*" and correct the object name and attribute suffix (/Count) to match exactly.
utilization reads n/a (unlimited) and you expected a percentage. nodetool getcompactionthroughput returned 0 MB/s, which in Cassandra means the throttle is disabled (unlimited), not that throughput is zero. Root cause: the classic 0 = unlimited confusion — a ceiling of zero cannot be a denominator. Fix: this is correct behaviour; if you want a utilization percentage, set a real ceiling with nodetool setcompactionthroughput 64 (or via cassandra.yaml), after which the script computes a meaningful percentage against it.

Python Monitoring for Cassandra Compaction — the parent guide covering the JMX/Prometheus telemetry model this script reads from.
Python script for real-time compaction latency tracking — the per-task latency companion to this throughput view.
Compaction backlog analysis & alerting — how throughput utilization feeds the depth-and-velocity alerting model.
Async compaction tracking & metrics — the MBean catalogue and velocity math behind these derivations.

Back to Python Monitoring for Cassandra Compaction