Setting Compaction Backlog Alert Thresholds in Cassandra 4.x/5.x

A backlog monitor is only as good as the numbers you feed it, and most teams inherit a PendingCompactions > 100 rule that someone typed once and never revisited. That single static threshold pages constantly on a size-tiered table doing normal tier merges and stays silent while a leveled table quietly falls off a cliff. This guide is narrowly about choosing the threshold values themselves: what number to compare against, why it depends on concurrent_compactors and the compaction strategy in force, and how to derive it from your own telemetry instead of folklore. It assumes Cassandra 4.0, 4.1, or 5.0, a Prometheus scrape of the JMX exporter, and Python 3.10+ for the calibrator. It sits under Compaction Backlog Analysis & Alerting, which explains the velocity-aware daemon that consumes these thresholds; this page fills in the values that daemon fires on, and it stops where resolving high compaction backlog without downtime begins — that guide drains a breach, this one decides when to call it a breach.

Why a single absolute number never works

PendingCompactions is queue depth on the bounded CompactionExecutor pool, and its healthy ceiling scales with concurrent_compactors. A 2-compactor node saturates at a fundamentally lower depth than a 16-compactor node, so the only portable depth threshold is relative: alert when pending exceeds 2 × concurrent_compactors. On a 4-compactor node that is 8; on a 16-compactor node it is 32. Anchor every depth rule to that multiple rather than a fixed count.

Depth alone still cannot distinguish a busy-but-draining node from a stalled one. The complementary signal is velocity decay: the derivative of the monotonic BytesCompacted counter. Compute the instantaneous compaction rate as the delta between two scrapes, divide by its own rolling average, and alert when that ratio falls below a floor — a rate holding at less than 40% of its 1-hour average while depth is high is a stall, not a burst. Depth tells you the queue is deep; decay tells you it is not draining. You want both true before you page.

The third axis is disk. Backlog matters most when there is nowhere to put the merge output, so every escalation gate should pair a depth-or-decay condition with a disk-utilization gate — warn around 85% used, escalate hard past 90%, because that is where a stalled compaction turns into OutOfSpaceException and node eviction rather than mere latency.

Per-strategy baselines

The single most common miscalibration is applying one table’s baseline to a table running a different strategy. The four strategies produce structurally different PendingCompactions shapes:

The practical rule: LCS deserves a lower, faster-firing threshold because a growing queue means leveling is losing; STCS and TWCS deserve a duration gate because their queues are supposed to spike and recover. Wiring the same number into all four is how you end up simultaneously spammed and blind. The structural reasons behind these shapes are the trade-offs between STCS, LCS, and TWCS, and for time-ordered tables the rollover cadence you suppress around comes from strategy selection for time-series workloads.

Concrete alert rules

Below are Prometheus rules that encode the relative-depth, velocity-decay, and disk gates. concurrent_compactors is a per-node constant; expose it as a static label or recording rule so the comparison stays node-relative.

# prometheus/cassandra-backlog.rules.yml — Cassandra 4.x/5.x
groups:
  - name: cassandra-compaction-backlog
    rules:
      # Depth relative to the pool, sustained to filter STCS/TWCS spikes.
      - alert: CompactionBacklogDepthWarning
        expr: |
          cassandra_compaction_pendingtasks
            > 2 * cassandra_concurrent_compactors
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Pending compactions above 2x concurrent_compactors for 15m"

      # Velocity decay: current rate below 40% of its 1h average while deep.
      - alert: CompactionBacklogStalled
        expr: |
          (
            rate(cassandra_compaction_bytescompacted[5m])
            /
            avg_over_time(rate(cassandra_compaction_bytescompacted[5m])[1h:5m])
          ) < 0.40
          and cassandra_compaction_pendingtasks
            > 2 * cassandra_concurrent_compactors
        for: 10m
        labels: { severity: critical }
        annotations:
          summary: "Compaction rate decayed below 40% while queue is deep"

      # Disk gate turns a stall into a page.
      - alert: CompactionBacklogDiskCritical
        expr: |
          (1 - cassandra_storage_load_bytes / node_filesystem_size_bytes) < 0.10
          and cassandra_compaction_pendingtasks > cassandra_concurrent_compactors
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Disk under 10% free with a non-empty compaction queue"

The for: durations are the noise filter. A STCS tier merge clears well inside 15 minutes, so the warning never fires on it; a genuine stall persists and does. Keep the decay rule’s window (1h:5m) aligned to your scrape interval — differentiating a counter faster than you scrape it turns the rate into noise, the same sampling discipline used across async compaction tracking & metrics.

Pre-conditions & safety gates

Before you commit numbers, confirm you actually know each node’s compactor count and current baseline. These are read-only.

# 1. Read concurrent_compactors per node — the denominator of every depth rule.
nodetool getconcurrentcompactors

Safety Check: Heterogeneous hardware often means different values per node. A single cluster-wide threshold is wrong if the pool sizes differ; label per node. Expected Output: A single integer, e.g. 4.

# 2. Capture a healthy-baseline snapshot to size the warning floor against.
nodetool compactionstats --human-readable

Safety Check: Sample during normal load, not during a deploy or repair. The baseline you calibrate against must represent steady state. Expected Output: pending tasks: in single digits for LCS/TWCS, occasionally higher for STCS.

Implementation: a threshold calibrator

Rather than guess, mine your own history. The calibrator below pulls a window of PendingCompactions samples from Prometheus and derives strategy-aware warning and critical floors from the observed distribution, so the threshold reflects this deployment’s real behaviour.

#!/usr/bin/env python3
# requirements: Python 3.10+, requests>=2.31
"""Derive compaction backlog alert thresholds from observed history.
Reads a PromQL range query and emits warning/critical floors per strategy."""
import logging
import statistics
from dataclasses import dataclass
import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")

PROM_URL = "http://localhost:9090/api/v1/query_range"


@dataclass(frozen=True)
class Thresholds:
    strategy: str
    concurrent_compactors: int
    warning: int
    critical: int


def fetch_pending_series(query: str, start: float, end: float, step: str = "60s") -> list[float]:
    """Return a flat list of PendingCompactions samples over the range."""
    resp = requests.get(PROM_URL, params={"query": query, "start": start,
                                          "end": end, "step": step}, timeout=10)
    resp.raise_for_status()
    result = resp.json().get("data", {}).get("result", [])
    samples: list[float] = []
    for series in result:
        for _ts, val in series.get("values", []):
            try:
                samples.append(float(val))
            except ValueError:
                continue
    return samples


def calibrate(samples: list[float], strategy: str, concurrent_compactors: int) -> Thresholds:
    """Set the warning floor above normal spikes, critical above the tail.

    LCS is expected flat, so its floors sit lower and fire on any real rise;
    STCS/TWCS spike normally, so the floor is pushed to the p95 of history but
    never below the pool-relative 2x concurrent_compactors sanity minimum."""
    pool_min = 2 * concurrent_compactors
    if not samples:
        return Thresholds(strategy, concurrent_compactors, pool_min, 2 * pool_min)

    samples.sort()
    p95 = samples[int(len(samples) * 0.95)]
    p99 = samples[int(len(samples) * 0.99)]

    if strategy.upper() == "LCS":
        # Flat queue: alert just above the normal ceiling, not at a big spike.
        median = statistics.median(samples)
        warning = max(int(median + 4), pool_min // 2 or 4)
        critical = max(int(p95), pool_min)
    else:
        # Bursty strategies: absorb normal spikes, page above the tail.
        warning = max(int(p95), pool_min)
        critical = max(int(p99 * 1.25), 2 * pool_min)
    return Thresholds(strategy, concurrent_compactors, warning, critical)


if __name__ == "__main__":
    import time
    now = time.time()
    week_ago = now - 7 * 24 * 3600
    query = 'cassandra_compaction_pendingtasks{keyspace="telemetry"}'
    history = fetch_pending_series(query, week_ago, now)
    result = calibrate(history, strategy="STCS", concurrent_compactors=4)
    logging.info("Strategy=%s pool=%d -> warning=%d critical=%d",
                 result.strategy, result.concurrent_compactors,
                 result.warning, result.critical)

Safety Check: The calibrator never writes anything and floors every result at the pool-relative 2 × concurrent_compactors minimum so a quiet history cannot produce an absurdly low threshold. Re-run it after any concurrent_compactors change or strategy migration, because both invalidate the old distribution. Expected Output:

Strategy=STCS pool=4 -> warning=11 critical=41

Feed those numbers back into the Prometheus rules, replacing the raw literals with the calibrated values. The deeper telemetry-ingestion plumbing that exposes these metrics is in Python monitoring for Cassandra compaction.

Verification steps

# 1. Dry-run the rules against history before arming them.
promtool test rules prometheus/cassandra-backlog.tests.yml

Safety Check: Confirm the warning rule does not fire on a replayed normal-load week and does fire on a replayed incident. If it fires on steady state, the floor is too low.

# 2. Confirm the disk gate reads the right filesystem.
nodetool status | grep -E "^UN" && df -h /var/lib/cassandra/data

Safety Check: The PromQL node_filesystem_size_bytes must be scoped to the data mount, not root. A mis-scoped mountpoint silently disables the disk gate.

Troubleshooting

• Thresholds too noisy (constant warning pages). The warning fires several times a day with no real incident. Root cause: a static depth floor applied to a bursty STCS or TWCS table without a for: duration gate, so normal tier merges and window rollovers trip it. Fix: raise the floor to the calibrator’s p95 output, add a 15-minute for: clause, and suppress the TWCS rule around the known window-close cadence rather than lowering the number blindly.
• Threshold too slow to fire (incident found manually first). Disk filled and reads timed out before any alert. Root cause: a high absolute depth threshold copied from an STCS table onto an LCS table, where a shallow-but-growing queue never reaches the STCS-sized number. Fix: give LCS its own lower, decay-sensitive rule that fires on rate decay plus a modest depth rise, and always pair depth with the disk-utilization gate so a stall that is not deep in count still pages when free space runs low.
• Strategy-mismatched baseline after a migration. Alerts go silent (or scream) right after an ALTER TABLE ... WITH compaction. Root cause: the calibrated thresholds still reflect the pre-migration strategy’s distribution, which no longer matches the new queue shape. Fix: re-run the calibrator against a fresh post-migration window once the strategy rebuild settles, and never carry STCS-derived floors onto a table you have just moved to LCS or UCS. Separate any transient rebuild spikes from real stalls using compaction error categorization & logging.

Related

• Compaction Backlog Analysis & Alerting — the parent guide whose velocity-aware daemon consumes the thresholds this page derives.
• Resolving high compaction backlog without downtime — what to do after a threshold you set here actually fires.
• Async compaction tracking & metrics — the velocity-versus-depth math and sampling cadence behind the decay rule.
• Python monitoring for Cassandra compaction — exposing the JMX counters the PromQL rules and calibrator read.