Compaction Strategy Cassandra Ops & Automation

Tracking Repair Session Duration with Python in Cassandra 4.x/5.x

A repair schedule is only safe if each session finishes inside the window it was allotted. When sessions creep — a growing dataset, rising entropy, disk contention from compaction — the maintenance window silently overruns, the next scheduled tick collides with the last, and the cadence that was supposed to stay ahead of gc_grace_seconds starts slipping. This page delivers a complete Python 3.10+ script that queries system_distributed.repair_history through the Cassandra driver, computes the wall-clock duration of every recent session, and raises an alert when a session runs longer than its configured window. It sits under Python Monitoring for Cassandra Compaction; read that first for the telemetry model, then use this script to instrument the duration signal that a state-aware repair schedule depends on. Prerequisites: Cassandra 4.0, 4.1, or 5.0, the cassandra-driver Python package, a CQL login with SELECT on system_distributed, and Python 3.10+.

Pre-conditions & safety gates

Every check is read-only. Run them before deploying the tracker.

1. repair_history is populated

-- cqlsh: there should be recent rows if repairs have run.
SELECT keyspace_name, columnfamily_name, status, started_at, finished_at
FROM system_distributed.repair_history LIMIT 5;

Safety Check: Read-only. If this returns zero rows, no repair has completed on this deployment yet, or the rows have not flushed (see troubleshooting) — deploy the tracker only once real sessions appear. Expected Output: A handful of rows with started_at/finished_at timestamps and a status such as SUCCESS.

2. Driver connectivity with a read-only role

python3 -c "import sys; assert sys.version_info>=(3,10); import cassandra; print('PASS: cassandra-driver', cassandra.__version__)"

Safety Check: Confirms 3.10+ and that cassandra-driver (pip install cassandra-driver) is importable. Use a role granted only SELECT on system_distributed; the tracker never writes. Expected Output: PASS: cassandra-driver 3.x.

3. Node health

# Duration analysis is meaningless mid-outage; confirm the ring is healthy first.
nodetool status | awk 'NR>5 && $1!="" && $1!="UN"{print "NOT-UN:",$0}'

Safety Check: Read-only. An empty result means every node is UN. Interpreting durations while a node is DN mixes real overruns with sessions aborted by the outage. Expected Output: No NOT-UN: lines.

Only proceed once all three pass. Session duration is the leading indicator that a repair cadence is about to breach gc_grace_seconds, the deletion-grace window whose mechanics are detailed under tombstone management and garbage collection.

Implementation

system_distributed.repair_history records one row per repaired range per session, with started_at and finished_at timestamps and a status. Duration is finished_at - started_at. Because a single logical repair fans out into many range rows sharing one session id, the script groups rows by session, takes the earliest start and latest finish within each session as its true span, and flags any session whose span exceeds the configured window. It queries a bounded recent slice, computes durations in Python, and holds no state between runs.

Reading duration from repair_history rather than from a live JMX gauge has a deliberate advantage: it is the durable, after-the-fact record of what actually happened, so the tracker can run on any node with driver access and reconstruct the full history of the last day’s sessions even for repairs it never observed live. The JMX RepairSessionDuration-family metrics are the right tool for watching a session as it runs — they let a scheduler abort a session that has already blown its window — but they vanish when the JVM restarts and only reflect the local node. repair_history is replicated and survives restarts, which makes it the correct source for the retrospective question this page answers: did last night’s scheduled pass finish inside its window, and is the trend getting worse.

Two modelling choices in the script are worth calling out. First, grouping by session id and taking earliest-start/latest-finish means a session that repaired forty ranges is reported as one span from the first range’s start to the last range’s finish — the wall-clock time the maintenance window actually consumed, not the sum of per-range times (which would over-count because ranges repair with some overlap). Second, a session with any range still lacking a finished_at is reported as PENDING rather than assigned a misleadingly short duration; a session is only judged against its window once every range has reported a finish, which prevents a half-complete session from looking healthy.

Save the following as repair_duration_tracker.py.

#!/usr/bin/env python3
# requirements: Python 3.10+, cassandra-driver>=3.28
"""Track Cassandra 4.x/5.x repair-session duration from system_distributed.repair_history.

Groups range rows by session id, computes each session's wall-clock span, and
flags sessions that overrun the configured window. Read-only, bounded."""
from __future__ import annotations

import argparse
import sys
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone

from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider


@dataclass
class Session:
    """Aggregated span of one repair session across all its range rows."""
    session_id: str
    keyspace: str
    table: str
    started_at: datetime | None = None
    finished_at: datetime | None = None
    statuses: set[str] = field(default_factory=set)

    def observe(self, started: datetime | None, finished: datetime | None,
                status: str | None) -> None:
        if started and (self.started_at is None or started < self.started_at):
            self.started_at = started
        if finished and (self.finished_at is None or finished > self.finished_at):
            self.finished_at = finished
        if status:
            self.statuses.add(status)

    def duration(self) -> timedelta | None:
        if self.started_at and self.finished_at:
            return self.finished_at - self.started_at
        return None


def fetch_sessions(session, keyspace: str, lookback_h: int) -> list[Session]:
    """Read recent repair_history rows and aggregate them by session id."""
    since = datetime.now(timezone.utc) - timedelta(hours=lookback_h)
    rows = session.execute(
        """
        SELECT id, keyspace_name, columnfamily_name, status, started_at, finished_at
        FROM system_distributed.repair_history
        WHERE keyspace_name = %s ALLOW FILTERING
        """,
        (keyspace,),
    )
    grouped: dict[str, Session] = {}
    for r in rows:
        # Skip rows older than the lookback window (client-side; started_at may be null).
        if r.started_at and r.started_at.replace(tzinfo=timezone.utc) < since:
            continue
        sid = str(r.id)
        sess = grouped.setdefault(
            sid, Session(sid, r.keyspace_name, r.columnfamily_name)
        )
        started = r.started_at.replace(tzinfo=timezone.utc) if r.started_at else None
        finished = r.finished_at.replace(tzinfo=timezone.utc) if r.finished_at else None
        sess.observe(started, finished, r.status)
    return list(grouped.values())


def report(sessions: list[Session], window_min: int) -> int:
    """Print each session and flag overruns. Returns the count of overruns."""
    window = timedelta(minutes=window_min)
    overruns = 0
    for s in sorted(sessions, key=lambda x: x.started_at or datetime.max.replace(
            tzinfo=timezone.utc)):
        dur = s.duration()
        if dur is None:
            print(f"PENDING  {s.keyspace}.{s.table}  session={s.session_id[:8]}  "
                  f"(no finished_at yet; statuses={sorted(s.statuses)})")
            continue
        mins = dur.total_seconds() / 60.0
        flag = ""
        if dur > window:
            overruns += 1
            flag = f"  OVERRUN > {window_min}m"
        print(f"{'FAIL' if 'FAILED' in s.statuses else 'OK'}  "
              f"{s.keyspace}.{s.table}  session={s.session_id[:8]}  "
              f"duration={mins:.1f}m{flag}")
    return overruns


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--host", default="127.0.0.1")
    ap.add_argument("--keyspace", required=True)
    ap.add_argument("--lookback-hours", type=int, default=24)
    ap.add_argument("--window-minutes", type=int, default=120,
                    help="Alert if a session runs longer than this.")
    ap.add_argument("--username")
    ap.add_argument("--password")
    args = ap.parse_args()

    auth = (PlainTextAuthProvider(args.username, args.password)
            if args.username else None)
    cluster = Cluster([args.host], auth_provider=auth, connect_timeout=15)
    session = cluster.connect()
    try:
        sessions = fetch_sessions(session, args.keyspace, args.lookback_hours)
        overruns = report(sessions, args.window_minutes)
        print(f"\n{len(sessions)} sessions analysed, {overruns} overrun(s).")
        return 1 if overruns else 0        # non-zero exit lets cron/alerting react
    finally:
        cluster.shutdown()                 # always release the driver connection


if __name__ == "__main__":
    sys.exit(main())

Safety Check: The query is a bounded recent slice grouped client-side; the driver connection is always released in a finally; null finished_at is treated as a still-pending session rather than a zero duration; the process exits non-zero only when a real overrun is found, so a scheduler or alerting hook can react to the exit code. Expected Output: One line per session, e.g. OK telemetry.sensor_readings session=a1b2c3d4 duration=47.3m, and a summary line counting overruns.

The non-zero exit on overrun is the integration point. Run this from the same timer that wakes your repair schedule, before the next pass starts: if it exits non-zero, the previous window has not drained and starting another repair would stack streaming on streaming. A wrapper can read the exit code and defer, turning duration tracking from a passive dashboard into an active guard that keeps the schedule honest. The --window-minutes value should be set slightly below the interval between scheduled passes for a range, so an overrun is flagged before the collision rather than after it — if a range repairs every three hours, a two-hour window leaves an hour of margin and still catches a session that has doubled its normal runtime.

The ALLOW FILTERING in the query is deliberate and safe here because the result set is already narrowed to one keyspace and a short lookback, so the coordinator scans a small, bounded slice of a low-volume system table rather than a production data table. Do not generalize the pattern to filtering production tables by non-key columns; on repair_history, whose row count is proportional to sessions rather than to your data, it is an acceptable trade for not having to maintain a secondary index on a system-managed table.

Verification steps

# 1. Compare the tracker's session count to distinct ids in repair_history.
cqlsh -e "SELECT id FROM system_distributed.repair_history WHERE keyspace_name='telemetry' ALLOW FILTERING;" | sort -u | wc -l

Safety Check: The distinct-id count over the same lookback should be within one or two of the tracker’s sessions analysed figure (the tracker filters by started_at window). Expected Output: Comparable counts.

# 2. Force an overrun detection with a deliberately tiny window.
python3 repair_duration_tracker.py --keyspace telemetry --window-minutes 1

Safety Check: Any real session longer than one minute must print OVERRUN and the process must exit non-zero. This proves the alert path fires. Expected Output: At least one OVERRUN line and a non-zero exit status.

# 3. Confirm durations agree with the coordinator log.
grep -E "Repair .* finished in|Repair session .* finished" /var/log/cassandra/system.log | tail

Safety Check: A session’s computed duration should roughly match the elapsed time between its start and finish log lines on the coordinator. Expected Output: Log timings within a small margin of the tracker’s figures.

Troubleshooting

  • repair_history returns no rows even though repairs ran. The rows exist in the system_distributed keyspace but have not become visible on the node you queried, or they were written to a different coordinator. Root cause: repair_history is a distributed table; rows are inserted by the coordinator of each session and are subject to normal replication and flush timing, so a query moments after a session — or against a node that did not coordinate it — can miss them. Fix: query with CONSISTENCY QUORUM, allow a short delay after sessions complete, and remember that only the coordinating node writes the parent-session summary. If rows are truly absent long after, verify the session actually reached SUCCESS rather than aborting before it could record history.
  • A session’s duration is wildly inflated or slightly negative. The started_at and finished_at timestamps are stamped by different nodes whose clocks disagree. Root cause: clock skew across the deployment — finished_at recorded on a node running fast relative to the started_at node inflates the span, and the reverse can produce a small negative. Fix: enforce tight time synchronization (NTP/chrony) across all nodes, and treat any negative or implausibly large duration as a skew symptom rather than a real overrun; the script’s grouping takes the earliest start and latest finish, which bounds but does not eliminate skew, so monitor clock offset separately.
  • Every range shows as its own “session” and durations look tiny. The script is keying on the per-range row rather than the parent session, or the data genuinely holds per-range rows without a coordinator summary. Root cause: repair_history mixes coordinator-level session rows with per-range detail depending on version and repair type; grouping by the range id alone fragments one logical repair into many. Fix: this script already groups by the session id and aggregates earliest-start/latest-finish across a session’s range rows — confirm you are grouping on the session identifier, not the range, and cross-check against system_distributed.parent_repair_history where the coordinator-level summary lives.
Back to Python Monitoring for Cassandra Compaction