Compaction Strategy Cassandra Ops & Automation

Incremental vs Full Repair Trade-offs at Scale in Cassandra 4.x/5.x

On a small cluster the choice between incremental and full repair barely matters; at scale it decides whether a repair cycle finishes overnight or drags validation and streaming across the fleet for days. This guide is for operators running nodetool repair on large Cassandra 4.x/5.x clusters who need to choose deliberately between incremental repair — the default since 4.0, which separates SSTables into repaired and unrepaired sets via anticompaction — and full repair, including its subrange variant. It builds on the read repair versus anti-entropy repair overview, which frames why anti-entropy repair is the durable backstop that opportunistic read repair cannot replace; here the focus is the operational trade at scale: cost, streaming volume, anticompaction overhead, and when full or subrange is the safer instrument. Prerequisites: you know your cluster’s node count and replication factor, you can run nodetool on every node, and you have your current repair cadence and gc_grace_seconds to hand.

Pre-conditions: size the decision

Repair strategy is bounded by three cluster facts. Gather them before choosing.

# 1. Cluster size and per-DC replica topology — repair cost scales with both.
nodetool status

Expected output lists every node with its state (UN is healthy) and datacenter. Count the nodes per DC and note the replication factor: full repair’s validation and streaming cost grows with the number of replicas that must compare Merkle trees for each range.

# 2. gc_grace_seconds for every table you intend to repair.
#    Repair cadence MUST stay below this or deletes can resurrect.
SELECT keyspace_name, table_name, gc_grace_seconds
FROM system_schema.tables WHERE keyspace_name = 'app';

Gate: your full-cluster repair cadence must complete within gc_grace_seconds (default 864000 = 10 days) for every table. If a full-cluster repair cannot finish inside that window, that is itself a reason to prefer incremental or subrange repair, which finish faster.

# 3. Current repair footprint — is a repair already running?
nodetool compactionstats | grep -iE "validation|anticompaction"
nodetool netstats | grep -i repair

Expected when idle: no Validation or Anticompaction tasks and no repair streaming sessions. Do not start a new repair over an in-flight one — overlapping sessions on the same ranges waste I/O and inflate RepairSessionDuration.

Implementation: choosing and running the right repair

The decision reduces to how the two modes spend I/O. Incremental repair runs anticompaction the first time it touches a range, splitting SSTables into a repaired set and an unrepaired set; subsequent incremental repairs only validate and stream the unrepaired set, so steady-state cost is low. Full repair validates and reconciles the entire dataset every time — expensive, but it never performs anticompaction and never depends on the repaired/unrepaired boundary staying consistent, which makes it the safer choice when that boundary is suspect or has never been established.

The cost profiles diverge sharply as the deployment grows. Full repair’s work scales with the total dataset multiplied by the replicas that must exchange and compare Merkle trees, so a 50-node cluster with RF=3 pays validation and potential streaming across every range on every cycle — the reason a naive nightly full repair eventually cannot finish inside its window. Incremental repair front-loads that cost into a one-time anticompaction per range, then keeps steady-state cycles cheap by validating only the unrepaired set that has accumulated since the last run. The catch is that incremental repair’s cheapness depends on the repaired/unrepaired boundary staying trustworthy across every replica; anything that muddies that boundary — a restore, a mixed-mode run, an interrupted anticompaction — can turn a “cheap” incremental cycle into one that re-validates and re-streams data it should have skipped.

Streaming volume is the other axis that punishes large clusters. Both modes build Merkle trees at a fixed leaf granularity, so a single differing cell forces the whole leaf range to stream. On a big table with coarse per-range resolution, that over-streams badly under full repair; slicing the same table into subranges gives each session finer effective resolution and bounds how much a single mismatch can drag across the network. Primary-range repair with -pr is the orthogonal lever: it tells each node to repair only the ranges for which it is the primary replica, so running it once on every node covers the whole ring exactly once instead of every replica redundantly repairing the same ranges — a factor-of-RF reduction in total work that applies to both incremental and full modes.

Use this decision procedure:

  • Established cluster, incremental already in use, healthy repaired/unrepaired split → keep running incremental repair on a regular cadence. It is the 4.x/5.x default and the cheapest steady state.
  • First repair after a major event — a node replacement, a restore from backup, a version upgrade, or a deployment that has never been repaired incrementally → run a full repair to establish a clean baseline before resuming incremental.
  • Very large tables where a full repair cannot finish in one window → run subrange full repair, slicing the token range into chunks so each session is bounded and resumable by re-running the chunk.
  • Any cluster, to avoid redundant work across replicas → add -pr so each node repairs only its primary ranges; run it on every node so the whole ring is covered exactly once.

Incremental repair, primary-range, one datacenter at a time on a large cluster:

# Incremental (the 4.x/5.x default — no --full flag) limited to primary ranges.
# Run on every node in the DC so the ring is covered once with no replica overlap.
nodetool repair -pr --in-local-dc app

Full repair when establishing or re-establishing a baseline:

# Full repair validates the entire dataset; -pr still avoids replica-redundant work.
nodetool repair --full -pr app

Subrange full repair for a very large table, bounding each session to a token slice:

# Subrange full repair: repair one explicit token range so the session is bounded.
# Obtain ranges from `nodetool ring` / describering and iterate the whole ring.
nodetool repair --full -st -9223372036854775808 -et -4611686018427387904 app large_table

Wrap the ring walk in a bounded, idempotent loop so a large-table repair is resumable and never runs two overlapping sessions:

#!/usr/bin/env python3
# requirements: Python 3.10+, nodetool on PATH (Cassandra 4.x/5.x). Stdlib only.
"""Run subrange full repair over an explicit list of token ranges, one at a time."""
import subprocess
import sys

# (start_token, end_token) slices covering the ring; generate from nodetool ring.
TOKEN_RANGES: list[tuple[str, str]] = [
    ("-9223372036854775808", "-4611686018427387904"),
    ("-4611686018427387904", "0"),
    ("0", "4611686018427387905"),
    ("4611686018427387905", "9223372036854775807"),
]


def repair_range(keyspace: str, table: str, start: str, end: str, timeout: int = 21600) -> bool:
    """Full subrange repair of one token slice; returns True on clean exit."""
    cmd = ["nodetool", "repair", "--full", "-st", start, "-et", end, keyspace, table]
    try:
        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
    except subprocess.TimeoutExpired:
        print(f"[timeout] range {start}..{end} exceeded {timeout}s", file=sys.stderr)
        return False
    if proc.returncode != 0:
        print(f"[fail] range {start}..{end}: {proc.stderr.strip()}", file=sys.stderr)
        return False
    print(f"[ok] range {start}..{end}")
    return True


if __name__ == "__main__":
    ks, tbl = "app", "large_table"
    # Sequential, one range at a time: never overlap sessions on the same node.
    failures = [r for r in TOKEN_RANGES if not repair_range(ks, tbl, *r)]
    # Re-running only re-does completed ranges harmlessly; failed ranges can be retried.
    sys.exit(1 if failures else 0)

Whichever mode you run, confirm gc_grace_seconds on every table is greater than or equal to the time this repair cycle takes to cover the whole ring — the coupling detailed in the tombstone management and garbage collection material — or a delete can resurrect before every replica has reconciled it.

Verification

Confirm the repair actually reconciled the ranges and did not silently stall.

-- 1. Repair history: recent sessions, their ranges, and success/failure status.
SELECT keyspace_name, columnfamily_names, status, started_at, finished_at
FROM system_distributed.repair_history LIMIT 20;

Expect status = 'SUCCESS' rows with finished_at populated for the ranges you ran. A row stuck without finished_at is an incomplete session to investigate before the next cycle.

# 2. Confirm no validation or anticompaction work is still pending.
nodetool compactionstats | grep -iE "validation|anticompaction" || echo "repair I/O idle"

repair I/O idle confirms anticompaction (incremental) and validation have drained.

  • Watch RepairSessionDuration. Export the org.apache.cassandra.metrics repair MBeans through the Prometheus JMX exporter; a session duration that exceeds your cadence interval means repair cannot keep up and you should move toward incremental or subrange to shorten each cycle.

Troubleshooting

  • Anticompaction I/O overhead spikes on the first incremental repair. The first incremental repair over a range runs anticompaction to split SSTables into repaired and unrepaired sets, which rewrites a large fraction of the data and can saturate disk I/O on a big table — nodetool compactionstats shows sustained Anticompaction tasks and client latency climbs. Root cause: the one-time cost of establishing the repaired/unrepaired boundary. Fix: schedule the first incremental repair per table in an off-peak window, throttle with nodetool setcompactionthroughput, and expect the overhead to fall away on subsequent incremental runs that only touch the unrepaired set.
  • Over-streaming and validation timeouts on a large full repair. A full repair across many replicas builds Merkle trees at coarse granularity, so a single differing cell can trigger streaming of an entire large range; on a big cluster this over-streams and sessions time out with RepairException or validation failures. Root cause: full repair’s whole-dataset validation plus coarse Merkle resolution over large ranges. Fix: switch that table to subrange full repair so each session covers a bounded token slice with finer effective resolution, and cap stream_throughput_outbound_megabits_per_sec so a burst does not saturate the NIC.
  • Repaired/unrepaired mismatch after mixing full and incremental modes. Running a full repair without -pr or restoring SSTables can leave data marked unrepaired on some replicas and repaired on others; the next incremental repair then re-validates data it believes is unrepaired, redoing work and occasionally streaming already-consistent data. Root cause: an inconsistent repaired/unrepaired boundary across replicas after mixing modes. Fix: re-establish a clean baseline with a full repair, or use nodetool repair tooling to reset the repaired state, then resume a single consistent mode — do not alternate full and incremental on the same table without re-baselining.
Back to Read Repair vs Anti Entropy Repair