Automated Repair Scheduling in Cassandra 4.x/5.x

Anti-entropy repair is the only mechanism that guarantees eventual convergence of replicas for data that read-repair and hinted handoff never touch, yet it is also the single most destabilizing maintenance operation you can schedule badly. A repair job that fires from a fixed cron entry — the same minute, every node, regardless of cluster state — is one of the most common self-inflicted outages in production Apache Cassandra 4.x and 5.x. This guide is for DBAs and SREs who need repair to run continuously and unattended without saturating disk I/O, colliding with compaction, or resurrecting deleted data. It sits under Advanced Compaction Strategy Tuning & Monitoring; read that first if you have not yet aligned compaction throughput and backlog alerting, because repair validation competes for exactly the same I/O budget those pages govern. Everything below is a scheduler design pattern you implement yourself — no external product is named or required — and every command is validated against Cassandra 4.0, 4.1, and 5.0 with version drift called out inline.

Why fixed-cron repair is an anti-pattern

A naive crontab entry such as 0 2 * * 0 nodetool repair treats repair as a stateless batch job. It is not. Repair triggers validation compactions that build Merkle trees over every token range the node owns, streams mismatched partitions between replicas, and holds those sessions open for minutes to hours depending on data volume and entropy. Firing that work on a fixed clock has three failure modes that a state-aware scheduler exists to prevent.

First, cron ignores cluster health. If a node is DN (down) or UJ (joining) when the timer fires, the coordinator either aborts the session with an UnavailableException or streams against an under-replicated range, wasting hours and leaving the range no more consistent than before. Cron has no notion of nodetool status; it will happily start a full-cluster repair into a degraded ring.

Second, cron creates thundering herds. If the same entry is deployed by configuration management to every node, all of them start validating and streaming simultaneously. Every disk in the deployment is now saturated by Merkle-tree builds at once, read p99 latency doubles, and the compaction backlog climbs because validation compactions share the bounded CompactionExecutor with normal compaction. The deployment spends the maintenance window fighting itself.

Third, cron has no memory. It cannot tell whether last week’s repair actually completed, whether a range is overdue, or whether a session is still running when the next timer fires — producing overlapping repairs that pile streaming on top of streaming. The distinction between the two repair families this schedule must respect is covered in depth under read repair versus anti-entropy repair; a scheduler automates only the anti-entropy side.

A state-aware scheduler replaces the clock with a state machine: it repairs exactly one node (or one token range) at a time, only when the whole ring is healthy, tracks what has been repaired and when, and never lets the interval between successful repairs exceed the deletion grace window.

Concept & mechanics: gate, stagger, track

Three invariants define a safe automated repair schedule.

Gate every session on all-UN status. Before any repair starts, the scheduler runs nodetool status and parses it. If every node in the relevant datacenter is not in the UN (up/normal) state, the scheduler skips this tick and re-evaluates on the next one. Repair into a ring with a missing replica cannot converge the range that replica owns, and streaming toward a joining node competes with its bootstrap. The gate is non-negotiable and cheap — a single status parse costs milliseconds against hours of wasted validation.

Stagger so only one node or range is active at a time. Instead of every node repairing its data, the scheduler walks the ring node by node, and on each node repairs only that node’s primary ranges using nodetool repair -pr. The -pr (--partitioner-range) flag restricts the operation to the ranges for which the node is the primary replica. When you run -pr on every node in sequence, each token range is repaired exactly once from the perspective of its primary owner, with no redundant work — a full-cluster repair decomposed into non-overlapping, one-at-a-time slices. Because only one node validates and streams at any moment, aggregate cluster I/O never doubles up.

Keep gc_grace_seconds ≥ repair cadence. This is the hard correctness rule. Tombstones — the markers Cassandra writes to record deletions — are purgeable only after gc_grace_seconds has elapsed, and compaction physically removes them once purgeable. Anti-entropy repair is what propagates a tombstone to every replica before that happens. If the interval between successful repairs of a range exceeds gc_grace_seconds, a replica that missed the original delete can compact away its tombstone, then hand the old, un-deleted value back during the next repair — a zombie, or resurrected delete. The mechanics of that purge window are detailed under tombstone management and garbage collection. The scheduler must therefore complete a full pass of every range inside gc_grace_seconds (default 864000 seconds = 10 days). If a full pass takes longer than the grace window, you either raise gc_grace_seconds, shrink the ranges via subrange repair, or add capacity — you never let the schedule slip past the window.

For very large token ranges, whole-range -pr repair can build a Merkle tree so large that a single mismatched leaf forces streaming of a huge partition span. Subrange repair divides a node’s primary range into many smaller token intervals and repairs each with nodetool repair -st <start_token> -et <end_token>, so a diff streams only the narrow interval that actually diverged. Subrange trades more, shorter sessions for far less over-streaming, and it lets the scheduler checkpoint progress at fine granularity — if a pass is interrupted, it resumes at the next unrepaired interval rather than restarting the whole range.

Configuration reference

Repair behaviour is governed partly by table-level settings, partly by the flags the scheduler passes to nodetool repair, and partly by cassandra.yaml. The parameters that most directly shape a safe schedule are below.

The -pr flag is incompatible with restricting repair to a single datacenter in a way that leaves cross-DC ranges unrepaired, so in multi-DC clusters run the -pr ring walk in every datacenter. Set the deletion grace window explicitly per table rather than relying on the default:

-- Cassandra 4.x/5.x: widen the grace window on a table whose full repair
-- pass legitimately takes longer than the 10-day default.
ALTER TABLE telemetry.sensor_readings WITH gc_grace_seconds = 1209600;  -- 14 days

A scheduler is invoked on a timer, but the timer only wakes it — the scheduler itself decides whether to act. A minimal crontab entry that hands control to a state-aware wrapper rather than to nodetool directly looks like this:

# /etc/cron.d/cassandra-repair — wake the scheduler hourly; the wrapper decides
# whether the ring is healthy and whether this node's range is due. It does NOT
# run `nodetool repair` unconditionally.
0 * * * * cassandra /opt/repair/repair_scheduler.py --datacenter dc1 --keyspace telemetry >> /var/log/cassandra/repair_scheduler.log 2>&1

Step-by-step: stand up a state-aware scheduler

Run this on one node against a non-production keyspace first, confirm the gating and staggering behave, then roll it out node by node.

1. Confirm the ring is fully UN before doing anything. The scheduler’s first act every tick is to parse nodetool status and abort unless every node is UN:

   Copy

   # Any line whose two-letter state is not "UN" means: skip this tick.
   nodetool status | awk 'NR>5 && $1!="" && $1!="UN" {print "NOT-UN:", $0}'

   An empty result is the go signal. Any NOT-UN: line means the scheduler waits for the next tick.

2. Confirm no repair is already running on this node. Overlapping sessions are how staggering breaks. Check the active session count before starting:

   Copy

   # Expect "0" active repair sessions before starting a new one.
   nodetool netstats | grep -c "Repair session"

3. Establish the cadence budget. Divide gc_grace_seconds by the number of ranges the scheduler must cover and confirm one full pass fits with margin. If the pass would exceed the grace window, switch that table to subrange repair or raise gc_grace_seconds first — never proceed with a schedule that cannot complete in time.

4. Deploy the scheduler wrapper. The pattern below gates on all-UN, checks for an in-flight session, runs a single -pr repair for this node, and records the outcome so cadence can be verified. It is idempotent — a second invocation while a repair is in flight is a no-op.

   Copy

   #!/usr/bin/env python3
   # requirements: Python 3.10+, nodetool on PATH. No third-party deps.
   """State-aware anti-entropy repair scheduler for Cassandra 4.x/5.x.
   Gates on all-UN status, refuses to overlap sessions, and runs one -pr
   repair for this node's primary ranges. Records start/end for cadence audit."""
   import argparse
   import json
   import subprocess
   import sys
   import time
   from pathlib import Path
   
   STATE_FILE = Path("/var/lib/cassandra-repair/last_run.json")
   
   def run(cmd: list[str], timeout: int = 30) -> str:
       """Read-only helper with a hard timeout; raises on non-zero exit."""
       return subprocess.run(cmd, capture_output=True, text=True,
                             timeout=timeout, check=True).stdout
   
   def ring_all_un(datacenter: str | None) -> bool:
       """Gate: return True only if every node in scope reports the UN state."""
       out = run(["nodetool", "status"] + ([datacenter] if datacenter else []))
       for line in out.splitlines():
           parts = line.split()
           # Status/State token is the first field on node rows, e.g. "UN".
           if parts and parts[0] in {"UN", "DN", "UJ", "UL", "UM", "DL", "DJ", "DM"}:
               if parts[0] != "UN":
                   return False
       return True
   
   def repair_in_flight() -> bool:
       """Refuse to stagger on top of an existing session on this node."""
       out = run(["nodetool", "netstats"])
       return "Repair session" in out
   
   def record(keyspace: str, status: str, duration_s: float) -> None:
       STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
       STATE_FILE.write_text(json.dumps({
           "keyspace": keyspace, "status": status,
           "duration_s": round(duration_s, 1),
           "finished_epoch": int(time.time()),
       }))
   
   def main() -> int:
       ap = argparse.ArgumentParser()
       ap.add_argument("--datacenter")
       ap.add_argument("--keyspace", required=True)
       ap.add_argument("--full", action="store_true",
                       help="Full repair; omit for the incremental default.")
       args = ap.parse_args()
   
       if not ring_all_un(args.datacenter):
           print("SKIP: ring not all-UN; deferring to next tick.")
           return 0                      # not an error — this is the gate working
       if repair_in_flight():
           print("SKIP: a repair session is already active on this node.")
           return 0                      # idempotent: never overlap
   
       # -pr repairs only this node's primary ranges; run on every node in turn
       # and each range is covered exactly once across the ring.
       cmd = ["nodetool", "repair", "-pr", args.keyspace]
       if args.full:
           cmd.insert(2, "--full")
   
       start = time.monotonic()
       try:
           subprocess.run(cmd, check=True, timeout=6 * 3600)   # 6h hard ceiling
           record(args.keyspace, "success", time.monotonic() - start)
           print(f"OK: {args.keyspace} primary-range repair complete.")
           return 0
       except subprocess.TimeoutExpired:
           record(args.keyspace, "timeout", time.monotonic() - start)
           print("FAIL: repair exceeded the 6h ceiling; investigate before retry.")
           return 1
       except subprocess.CalledProcessError as e:
           record(args.keyspace, "error", time.monotonic() - start)
           print(f"FAIL: nodetool repair exited {e.returncode}.")
           return 1
   
   if __name__ == "__main__":
       sys.exit(main())

5. Sequence the ring. Deploy the wrapper to every node but offset the crontab minute per node (or elect a leader that advances a token cursor) so the nodes fire in ring order, one at a time. The all-UN gate plus the in-flight check together guarantee that even if two timers overlap, the second node’s session waits rather than piling on.

6. Alternate cadences. Schedule incremental repair as the frequent default (for example daily) and a --full repair as the weekly safety net. Incremental repair marks SSTables as repaired and skips them next time, which keeps daily passes cheap; a periodic full pass re-validates everything and catches anticompaction anomalies. The trade-offs between the two are examined under incremental versus full repair at scale.

Verification & observability

Prove the schedule is actually converging ranges and completing inside the grace window rather than silently skipping every tick.

On Cassandra 4.x/5.x the authoritative record of what repaired and when lives in system_distributed.repair_history:

-- Most recent repair sessions and their outcomes for one table.
SELECT keyspace_name, columnfamily_name, id, coordinator, status, started_at, finished_at
FROM system_distributed.repair_history
WHERE keyspace_name = 'telemetry' AND columnfamily_name = 'sensor_readings'
LIMIT 20 ALLOW FILTERING;

Every session should show status = 'SUCCESS' with a finished_at timestamp, and the newest started_at for each range must be more recent than now - gc_grace_seconds. A range whose most recent success is older than the grace window is overdue and at risk of resurrecting deletes.

Watch session duration through JMX. The RepairSessionDuration-family metrics under org.apache.cassandra.metrics (and the repair MBeans under org.apache.cassandra.db:type=StorageService) expose how long sessions run; a session whose duration trends toward your window ceiling is the early warning that the schedule is about to slip. Streaming volume during a session is visible live:

# During an active session, confirm exactly one node is streaming.
nodetool netstats -H | grep -A2 "Repair"

Grep the log for the anticompaction and session-completion lines that confirm real work retired:

grep -E "Repair session .* finished|Anticompaction|Completed .* repair" /var/log/cassandra/system.log

A healthy signature is: the status gate occasionally logging SKIP (proving it defends the ring), exactly one node streaming at a time, repair_history filling with SUCCESS rows, and the oldest per-range success comfortably inside gc_grace_seconds.

Failure modes & rollback

Schedule slips past gc_grace_seconds (zombie risk). A deployment grows, a full pass that used to finish in 4 days now takes 12, and the 10-day grace window is breached. Detection: the oldest finished_at per range in repair_history is older than now - gc_grace_seconds; deleted rows reappear on reads. Rollback: immediately raise gc_grace_seconds on affected tables to exceed the current real pass duration (ALTER TABLE ... WITH gc_grace_seconds = ...), then re-architect the schedule to subrange repair so passes shrink. Do not run a manual full-cluster repair to “catch up” — that reintroduces the thundering herd.

Overlapping sessions from a lost gate. If the in-flight check is removed or the state file is wiped, two nodes repair simultaneously and streaming saturates disk. Detection: nodetool netstats shows repair sessions on more than one node at once; read p99 spikes cluster-wide; validation compactions pile into the compaction backlog. Rollback: stop the scheduler on all nodes, let the active sessions drain (nodetool netstats returns to zero repair sessions), restore the in-flight guard, and restart the ring walk from the last recorded cursor.

Repair into a degraded ring. A node is DN when a tick fires and, if the gate is bypassed, the coordinator streams against an under-replicated range or aborts with UnavailableException. Detection: repair_history rows with status = 'FAILED' clustered around a node-down event; UnavailableException in the coordinator log. Rollback: none needed if the gate held (the tick simply skipped); if the gate was bypassed, wait for the node to return UN, then let the next scheduled tick repair the range cleanly.

FAQ

What happens if gc_grace_seconds is shorter than my repair cadence?

Deleted data resurrects. Tombstones become purgeable after gc_grace_seconds and compaction removes them; if a range is not repaired before that window elapses, a replica that missed the original delete can compact away its tombstone and then serve the stale, un-deleted value back during a later read or repair — a zombie. The rule is absolute: a full repair pass of every range must complete inside gc_grace_seconds. If it cannot, raise the grace window or shrink the work with subrange repair; never let the cadence exceed it.

Should I use primary-range (-pr) or full repair on every node?

Use -pr on every node in sequence for routine scheduled repair. Running plain nodetool repair on every node repairs each range once per replica — three times over on RF=3 — which triples the work for no extra consistency. With -pr, each node repairs only the ranges it owns as primary, so a ring walk covers every range exactly once. Reserve non--pr full repair for targeted recovery, such as after replacing a node.

Why gate on nodetool status instead of just trusting the timer?

Because the timer knows nothing about the deployment. A repair started into a ring with a DN or UJ node either fails with UnavailableException or wastes hours streaming against an under-replicated range. Parsing nodetool status for all-UN before every session costs milliseconds and is the single cheapest safeguard against wasted or harmful repair work. A tick that skips because the ring is degraded is the scheduler working correctly, not an error.

How do incremental and full repair fit into one schedule?

Run incremental repair as the frequent default and a full repair periodically as the safety net. Incremental repair marks SSTables as repaired and skips them on the next run, keeping daily passes cheap and fast. A weekly or biweekly full repair re-validates everything, which catches anticompaction edge cases and any data the incremental path missed. Both must respect the same all-UN gate and staggering.

Can I run repair on several nodes at once to finish faster?

No — that is the thundering-herd anti-pattern the whole design exists to prevent. Concurrent repairs saturate every disk with Merkle-tree validation and streaming at once, doubling read latency and starving compaction of the shared I/O budget. If passes are too slow, shrink each unit of work with subrange repair and raise gc_grace_seconds to buy headroom — do not add concurrency. One node or one range at a time is the invariant.

Related

• Advanced Compaction Strategy Tuning & Monitoring — the parent guide tying repair scheduling to compaction throughput and backlog control.
• Read repair vs anti-entropy repair — how the two repair families differ and why only anti-entropy needs scheduling.
• Incremental vs full repair tradeoffs at scale — choosing the cadence mix this scheduler alternates between.
• Tombstone management & garbage collection — the gc_grace_seconds purge window that bounds repair cadence.
• Compaction backlog analysis & alerting — the shared I/O budget repair validation competes for.
• Tracking repair session duration with Python — instrument the RepairSessionDuration signal this schedule depends on.