Compaction Strategy Cassandra Ops & Automation

Building a Tombstone-Rate Alerter in Python for Cassandra 4.x/5.x

By the time a query dies with TombstoneOverwhelmingException, the damage is already done — the read crossed tombstone_failure_threshold (default 100000 tombstones scanned in a single read) and Cassandra aborted it to protect the coordinator. The signal you actually want fires earlier, at tombstone_warn_threshold (default 1000), while there is still time to widen a partition, trigger compaction, or fix a delete-heavy access pattern. This page delivers a complete Python 3.10+ alerter that reads the TombstoneScannedHistogram MBean, extracts its p99, and compares that against the two configured thresholds so you get paged on the trend, not the crash. It sits under Python Monitoring for Cassandra Compaction; read that first for the telemetry model. Tombstones are produced and reclaimed by the deletion lifecycle described under tombstone management and garbage collection — this alerter watches the read-time cost of that lifecycle. Prerequisites: Cassandra 4.0, 4.1, or 5.0, a reachable JMX port, Python 3.10+, and the jmxquery package.

Pre-conditions & safety gates

Every check is read-only. Run them before deploying the alerter.

1. The histogram MBean is exposed per table

# TombstoneScannedHistogram is a per-table metric; confirm it exists for your table.
timeout 10 nodetool sjk mx \
  -b "org.apache.cassandra.metrics:type=Table,keyspace=telemetry,scope=sensor_readings,name=TombstoneScannedHistogram" \
  -f 99thPercentile 2>/dev/null && echo "PASS" || echo "CHECK MBean path"

Safety Check: The metric is scoped by keyspace and table (type=Table on 4.x/5.x; older builds used type=ColumnFamily). The timeout stops the gate hanging on a busy JMX pool. Expected Output: PASS and a numeric percentile.

2. Read the configured thresholds

# These live in cassandra.yaml; the alerter's bands are derived from them.
grep -E "tombstone_(warn|failure)_threshold" /etc/cassandra/cassandra.yaml

Safety Check: Read-only. Defaults are tombstone_warn_threshold: 1000 and tombstone_failure_threshold: 100000. These are counts of tombstones scanned per read, not bytes or a rate — the unit matters (see troubleshooting). Expected Output: The two configured integer thresholds.

3. Python runtime and driver

python3 -c "import sys; assert sys.version_info>=(3,10); import jmxquery; print('PASS')"

Safety Check: Confirms 3.10+ and that jmxquery (pip install jmxquery) is importable. Expected Output: PASS.

Only proceed once all three pass. A rising tombstone-scan p99 on a table often coincides with a compaction backlog letting expired data survive, so pair this alerter with compaction backlog analysis & alerting.

Implementation

TombstoneScannedHistogram is a per-table histogram of how many tombstones each read scanned. Its p99 answers the operational question directly: “how bad is the worst 1% of reads on this table right now?” The alerter reads that percentile for each watched table, places it in one of three bands relative to the thresholds — nominal, warning (approaching tombstone_warn_threshold), or critical (approaching tombstone_failure_threshold) — and emits an alert only when the band changes upward, so it pages on trend rather than spamming every cycle. It reads several tables per pass, runs a bounded number of cycles, and keeps only the last band per table in memory.

The reason to alert on the p99 rather than the mean is that tombstone damage is a tail phenomenon. A table can have a perfectly healthy average read that scans a handful of tombstones while a small fraction of reads — the ones hitting a partition where a delete-heavy access pattern has accumulated markers faster than compaction reclaims them — scan tens of thousands. The mean drowns those reads; the p99 surfaces them. Because tombstone_failure_threshold aborts an individual read the moment it crosses the line, it is precisely the worst reads, not the typical one, that determine whether the table is about to start throwing TombstoneOverwhelmingException. Watching the tail is the whole point.

The banding is intentionally set to lead the built-in log warnings rather than trail them. Cassandra emits its tombstone cells WARN line when a read crosses tombstone_warn_threshold; by entering the alerter’s WARNING band at 50% of that threshold and CRITICAL at 50% of the failure threshold, the alert fires while the p99 is still climbing toward the point where reads begin to warn, giving an operator time to act — run a targeted compaction, widen the partition, or fix the query — before any read aborts. The half-threshold factors are tunable per table; a table with a naturally spiky but harmless range-scan profile should have wider bands than a point-read table where any tombstone growth is a genuine warning sign.

Save the following as tombstone_rate_alerter.py.

#!/usr/bin/env python3
# requirements: Python 3.10+, jmxquery>=0.6.0
"""Alert before TombstoneOverwhelmingException on Cassandra 4.x/5.x.

Reads the per-table TombstoneScannedHistogram p99 via JMX and bands it against
tombstone_warn_threshold and tombstone_failure_threshold. Read-only, bounded."""
from __future__ import annotations

import argparse
import sys
import time
from enum import IntEnum

from jmxquery import JMXConnection, JMXQuery

MBEAN_TMPL = (
    "org.apache.cassandra.metrics:type=Table,keyspace={ks},scope={tbl},"
    "name=TombstoneScannedHistogram/99thPercentile"
)


class Band(IntEnum):
    NOMINAL = 0
    WARNING = 1
    CRITICAL = 2


def classify(p99: float, warn: int, fail: int) -> Band:
    """Band the p99 against the two thresholds. Warn is entered at 50% of the
    warn threshold so the alert leads the WARN log line, not trails it."""
    if p99 >= 0.5 * fail:
        return Band.CRITICAL          # halfway to a hard read failure
    if p99 >= 0.5 * warn:
        return Band.WARNING
    return Band.NOMINAL


def read_p99(conn: JMXConnection, ks: str, tbl: str) -> float | None:
    """Read one table's tombstone-scan p99. None if the MBean is absent."""
    mbean = MBEAN_TMPL.format(ks=ks, tbl=tbl)
    result = conn.query([JMXQuery(mbean)])
    if not result or result[0].value is None:
        return None
    return float(result[0].value)


def run(jmx_url: str, tables: list[tuple[str, str]], warn: int, fail: int,
        interval: float, cycles: int) -> int:
    conn = JMXConnection(jmx_url)
    last: dict[tuple[str, str], Band] = {}
    for _ in range(cycles):                       # bounded run, never infinite
        for ks, tbl in tables:
            try:
                p99 = read_p99(conn, ks, tbl)
            except Exception as exc:              # survive a transient JMX blip
                print(f"WARN: JMX read failed for {ks}.{tbl}: {exc}", file=sys.stderr)
                continue
            if p99 is None:
                continue                          # metric not yet populated
            band = classify(p99, warn, fail)
            prev = last.get((ks, tbl), Band.NOMINAL)
            # Alert only on an upward band change => trend, not per-cycle spam.
            if band > prev:
                label = band.name
                print(f"ALERT[{label}] {ks}.{tbl} tombstone p99={p99:.0f} "
                      f"(warn={warn}, fail={fail})")
            elif band < prev:
                print(f"RECOVER {ks}.{tbl} tombstone p99={p99:.0f} -> {band.name}")
            last[(ks, tbl)] = band
        time.sleep(interval)
    return 0


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--jmx-url",
                    default="service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi")
    ap.add_argument("--table", action="append", required=True,
                    metavar="KEYSPACE.TABLE", help="Repeatable; e.g. telemetry.sensor_readings")
    ap.add_argument("--warn-threshold", type=int, default=1000)
    ap.add_argument("--fail-threshold", type=int, default=100000)
    ap.add_argument("--interval", type=float, default=30.0)
    ap.add_argument("--cycles", type=int, default=120)   # ~1h at 30s
    args = ap.parse_args()

    tables: list[tuple[str, str]] = []
    for spec in args.table:
        if "." not in spec:
            print(f"FAIL: --table must be KEYSPACE.TABLE, got {spec!r}", file=sys.stderr)
            return 2
        ks, tbl = spec.split(".", 1)
        tables.append((ks, tbl))

    return run(args.jmx_url, tables, args.warn_threshold, args.fail_threshold,
               args.interval, args.cycles)


if __name__ == "__main__":
    sys.exit(main())

Safety Check: The loop is bounded by --cycles; every JMX read is wrapped so a transient failure skips one table rather than crashing; an absent MBean returns None and is skipped instead of raising; alerts fire only on an upward band transition, so a table sitting in WARNING does not re-page every cycle. Expected Output: Quiet during normal operation; on a delete-heavy table, e.g. ALERT[WARNING] telemetry.sensor_readings tombstone p99=612 (warn=1000, fail=100000).

The edge-triggered design — alerting only when band > prev — is what makes this safe to run continuously. A level-triggered alerter that fired every cycle a table sat above the warn line would bury the one transition that matters under hundreds of identical repeats, and operators would learn to ignore it. Emitting once on the way up and once on the way down (RECOVER) means each line in the log is a real state change worth reading. The trade-off is that the in-memory last map is the only state, so a restart re-arms every table to NOMINAL and will re-alert on the next reading if a table is still hot; that is acceptable — a restart-triggered re-alert on a genuinely hot table is a feature, not noise.

Deciding which tables to watch matters as much as the thresholds. Point this alerter at tables with a known delete or TTL-expiry access pattern — queues, session stores, time-windowed data whose old rows are removed — because those are where tombstone accumulation actually threatens reads. A pure append-only table rarely generates tombstones and does not need watching; spending JMX polls on it only dilutes the signal. Enumerate the candidate tables once with nodetool tablestats (look for a non-trivial Number of tombstones or a delete-heavy write pattern) and pass each as a repeated --table argument.

Verification steps

# 1. Read the same p99 by hand and compare to what the alerter bands.
nodetool sjk mx -b "org.apache.cassandra.metrics:type=Table,keyspace=telemetry,scope=sensor_readings,name=TombstoneScannedHistogram" -f 99thPercentile

Safety Check: The manual p99 must match the p99= value the alerter prints within one poll interval. Expected Output: Matching percentile figures.

# 2. Force an alert with a deliberately low warn threshold.
python3 tombstone_rate_alerter.py --table telemetry.sensor_readings --warn-threshold 10 --cycles 3

Safety Check: With a low threshold, any table with real tombstone scans should cross into WARNING and emit an ALERT line, proving the alert path works. Expected Output: At least one ALERT[WARNING] line.

# 3. Correlate with actual read-time warnings in the log.
grep -E "tombstone cells|TombstoneOverwhelmingException|Read .* tombstones" /var/log/cassandra/system.log | tail

Safety Check: An alerter CRITICAL band should coincide with Read N live rows and M tombstone cells WARN lines climbing toward the failure threshold. Reading a delete-heavy access pattern in depth is covered under tombstone management and garbage collection. Expected Output: Log warnings that track the alerter’s bands.

Troubleshooting

  • The p99 barely moves even though a big range scan clearly hit many tombstones. TombstoneScannedHistogram records tombstones scanned per read, and its percentiles are a decaying distribution over recent reads — it is not a cumulative counter you differentiate. Root cause: treating the histogram like a monotonic total. A single pathological range scan is one sample in the p99, so a rare offender can be masked by a flood of cheap point reads. Fix: alert on the p99 (and add p999/max for the tail), not on any notion of “total tombstones”, and remember the window decays — a spike fades from the percentile within minutes even if the underlying data problem persists.
  • Frequent WARNING alerts on a table that never throws TombstoneOverwhelmingException. The reads triggering the histogram are legitimate multi-row range scans, not single-partition point reads, so a high per-read tombstone count is expected and harmless up to a point. Root cause: false alarms from range-scan query models — a range query that sweeps many partitions naturally scans more tombstones than the thresholds assume for point reads. Fix: raise the warn band for that specific table, or better, tighten the query to a single partition; the thresholds were designed around per-partition point reads, so a range-scan workload needs table-specific banding rather than the deployment default.
  • The alerter’s numbers look off by orders of magnitude against tombstone_warn_threshold. You are comparing different units. Root cause: threshold-unit confusion — tombstone_warn_threshold and tombstone_failure_threshold count tombstone cells scanned in a single read, while some dashboards surface tombstone ratio, rate per second, or bytes. Fix: confirm the histogram attribute is the raw scanned-count percentile (not a rate or ratio), and band it against the raw integer thresholds from cassandra.yaml; never mix a per-second rate into a comparison with a per-read count threshold.
Back to Python Monitoring for Cassandra Compaction