Compaction Strategy Cassandra Ops & Automation

STCS vs LCS for Mixed Read/Write Workloads in Cassandra 4.x/5.x

A table that both ingests steadily and serves point reads is the hardest compaction-strategy call in Cassandra, because the two classic strategies optimise for opposite costs: SizeTieredCompactionStrategy (STCS) minimises write amplification at the expense of read amplification, and LeveledCompactionStrategy (LCS) does the reverse. This guide is for operators who have a genuinely mixed table — a user-profile store taking constant updates while fielding lookups, an order table written and immediately queried — and need to decide which cost to pay. It builds directly on the STCS vs LCS vs TWCS overview, which explains how each strategy arranges SSTables; here the focus is narrower — measuring your own read:write ratio, disk headroom, and p99 read latency, then letting those numbers pick the strategy rather than picking on instinct. Prerequisite knowledge: you can run nodetool against the node, you know the table’s keyspace and name, and you have a recent read-latency baseline.

Pre-conditions: measure before you choose

The decision is data-driven, so gather three numbers first. All commands are read-only.

# 1. Read:write ratio and SSTables-per-read for the table.
#    On 4.x/5.x use tablestats (cfstats is the pre-4.0 alias).
nodetool tablestats app.profiles

Expected output includes Local read count, Local write count, and SSTables per read percentiles. Divide read count by write count for the ratio, and note the SSTable count — a high count on a read-heavy table is the signal that read amplification is hurting.

# 2. p99 read latency and the SSTables-per-read distribution.
nodetool tablehistograms app profiles

Expected output is a percentile table; the columns that matter are SSTables (how many SSTables a read touches at each percentile) and Read Latency. Record the 95th and 99th percentile SSTables-per-read as your baseline — this is the number LCS is designed to shrink.

# 3. Disk headroom on the data volume.
nodetool tablestats app.profiles | grep -E "Space used \(live\)"
df -h /var/lib/cassandra/data

Gate: STCS needs transient free space on the order of its largest tier to complete a big compaction (plan for roughly the table’s largest-tier size free, and conventionally up to ~50% of the volume for a worst-case merge); LCS needs far less headroom, realistically 1.5–2.5x the working set during re-leveling. Abort a switch to either if free space is below the strategy’s transient requirement — expand storage first.

Implementation: a decision procedure

With the three numbers in hand, apply the rule of thumb: the read:write ratio and the p99 SSTables-per-read together decide the strategy.

  • Read-dominant (ratio well above ~1:1) and SSTables-per-read climbing above ~2 at p95 → LCS. Predictable one-to-two-SSTable reads are worth the extra write amplification.
  • Write-dominant (ratio well below ~1:1) or reads already touching ~1 SSTable → STCS. You would pay LCS’s write amplification for a read improvement you do not need.
  • Genuinely balanced, or you cannot commit → on Cassandra 5.0, UnifiedCompactionStrategy (UCS) lets you interpolate between the two without a migration; on 4.x, lean LCS if reads are latency-sensitive and STCS if write throughput dominates.

Space amplification is the third axis and it usually breaks ties. STCS carries the heaviest space overhead: obsolete row fragments and tombstones linger across tiers until a large compaction sweeps them, and completing that sweep needs transient free space on the order of the largest tier — a real constraint on a node past 50% utilisation. LCS keeps space amplification low because its fixed-size, non-overlapping SSTables purge superseded cells on every promotion, so a mixed table that also updates rows frequently reclaims dead space far sooner under LCS. If your mixed table is both read-heavy and update-heavy, the space argument reinforces the read-latency argument and points the same way — toward LCS. If disk headroom is tight and reads are only mildly latency-sensitive, STCS’s read cost may still be the cheaper problem than the storage a large-tier merge demands.

Where the read:write ratio tips the decision is not a single crossover point but a band. Below roughly 1:1 (writes dominate) STCS almost always wins, because you would be paying LCS’s continuous re-leveling to improve reads you rarely issue. Above roughly 3:1 or 4:1 (reads dominate) with any measurable SSTable fan-out, LCS almost always wins. The ambiguous middle — a table that reads and writes in similar volume — is exactly where p99 SSTables-per-read becomes the deciding measurement: if reads already touch one SSTable, size-tiering is fine; if they fan across three or four, the read tax is real and leveling pays for itself. This is also the band where UCS earns its keep on 5.0, letting you sit between the two extremes and adjust the scaling_parameters knob as the ratio drifts rather than committing to a full strategy migration each time.

Switch a read-dominant mixed table to LCS, minimising read amplification:

-- Reads outnumber writes and p95 SSTables-per-read is above 2: level it.
ALTER TABLE app.profiles
  WITH compaction = {'class': 'LeveledCompactionStrategy', 'sstable_size_in_mb': 160};

Keep or set STCS on a write-dominant mixed table, capping merge size to smooth I/O:

-- Writes dominate and reads touch ~1 SSTable: size-tiering is cheaper.
ALTER TABLE app.profiles
  WITH compaction = {'class': 'SizeTieredCompactionStrategy', 'min_threshold': 4, 'max_threshold': 16};

On Cassandra 5.0, blend the two with UCS and a scaling parameter instead of committing to a fixed point:

-- 5.0 only: T4 behaves tiered (STCS-like); negative values lean leveled (LCS-like).
ALTER TABLE app.profiles
  WITH compaction = {'class': 'UnifiedCompactionStrategy', 'scaling_parameters': 'T4'};

The small helper below reads the p95 SSTables-per-read from nodetool tablehistograms and the read:write ratio from tablestats, then recommends a strategy. It only reads and prints — it never alters anything, so it is safe to run in a monitoring loop.

#!/usr/bin/env python3
# requirements: Python 3.10+, nodetool on PATH (Cassandra 4.x/5.x). Stdlib only.
"""Recommend STCS or LCS for a mixed-workload table from live nodetool metrics."""
import re
import subprocess


def _nodetool(args: list[str], timeout: int = 20) -> str:
    proc = subprocess.run(["nodetool", *args], capture_output=True, text=True, timeout=timeout)
    if proc.returncode != 0:
        raise RuntimeError(f"nodetool {' '.join(args)} failed: {proc.stderr.strip()}")
    return proc.stdout


def read_write_ratio(keyspace: str, table: str) -> float:
    """reads / writes from tablestats; returns inf when there are no writes yet."""
    out = _nodetool(["tablestats", f"{keyspace}.{table}"])
    reads = int(re.search(r"Local read count:\s*(\d+)", out).group(1))
    writes = int(re.search(r"Local write count:\s*(\d+)", out).group(1))
    return reads / writes if writes else float("inf")


def p95_sstables_per_read(keyspace: str, table: str) -> float:
    """95th-percentile SSTables touched per read from tablehistograms."""
    out = _nodetool(["tablehistograms", keyspace, table])
    for line in out.splitlines():
        if line.strip().startswith("95%"):
            # Columns: Percentile  SSTables  WriteLatency  ReadLatency ...
            return float(line.split()[1])
    return 0.0


def recommend(keyspace: str, table: str) -> str:
    ratio = read_write_ratio(keyspace, table)
    sstables = p95_sstables_per_read(keyspace, table)
    # Read-dominant AND reads fanning across SSTables -> leveled wins.
    if ratio >= 1.0 and sstables > 2.0:
        return "LCS"
    # Write-dominant or reads already tight -> size-tiered is cheaper.
    if ratio < 1.0 or sstables <= 1.5:
        return "STCS"
    return "BORDERLINE (consider UCS on 5.0, else LCS if latency-sensitive)"


if __name__ == "__main__":
    ks, tbl = "app", "profiles"
    print(f"ratio={read_write_ratio(ks, tbl):.2f} "
          f"p95_sstables={p95_sstables_per_read(ks, tbl):.1f} "
          f"=> {recommend(ks, tbl)}")

Apply the ALTER on one node’s schema (it propagates cluster-wide) and let the background recompaction drain before judging the result; the strategy switch is online but triggers a full-table rewrite that competes with client I/O, so throttle it with nodetool setcompactionthroughput 32 first. For the STCS→LCS direction specifically, the gated, rollback-capable step-by-step guide to switching from STCS to LCS walks the migration node by node.

Read amplification versus write amplification for STCS and LCS Two horizontal bars compare the two strategies. STCS shows low write amplification and high read amplification, so it favours write-dominant mixed tables. LCS shows low read amplification and high write amplification, so it favours read-dominant mixed tables. The read colon write ratio decides which cost is cheaper to pay. write amplification read amplification STCS low high favours write-dominant mixed tables LCS high low favours read-dominant mixed tables
STCS pays in reads to save writes; LCS pays in writes to save reads. The read:write ratio decides which cost is cheaper for a mixed table.

Verification

Confirm the switch achieved the trade you intended, not just that the schema changed.

# 1. Confirm the strategy took and watch the recompaction drain.
nodetool compactionstats -H

Expect a burst of Compaction tasks that trend back toward pending tasks: 0. A queue that only grows means throughput exceeds the disks — lower setcompactionthroughput.

# 2. For an LCS switch, confirm leveling: most data above L0.
nodetool tablestats app.profiles | grep -E "SSTables in each level"

A healthy LCS table shows only a few L0 SSTables with the bulk distributed across L1+.

# 3. Re-measure read amplification and compare to the pre-switch baseline.
nodetool tablehistograms app profiles

After moving to LCS, the p95/p99 SSTables-per-read should fall toward 1–2. If it did not drop, the table is still leveling or the workload was not actually read-bound.

Troubleshooting

  • Read latency got worse after switching to STCS. You moved a read-dominant table onto size-tiering, so a single partition’s data is now scattered across more tiers and each read fans out to more SSTables — nodetool tablehistograms shows p99 SSTables-per-read climbing. Root cause: the read:write ratio favoured LCS but STCS was chosen for its cheaper writes. Fix: revert to LCS (ALTER TABLE app.profiles WITH compaction = {'class': 'LeveledCompactionStrategy'}) and let it re-level; the write-amplification saving was never worth the read regression on this table.
  • LCS compaction never catches up and write amplification is punishing. After switching a write-dominant table to LCS, L0 grows unbounded, nodetool tablestats shows a large L0 count, and PendingCompactions climbs in lockstep with write volume because LCS re-levels overlapping ranges continuously. Root cause: the table ingests faster than leveled compaction can level it — the classic LCS-on-write-heavy anti-pattern, often worsened when repair streams a burst of SSTables into L0. Fix: raise concurrent_compactors and compaction_throughput to let L0 drain; if it still cannot keep up, the table is genuinely write-first and belongs on STCS.
  • Disk usage blows up mid-switch and threatens OutOfSpaceException. During the strategy change both the old and new SSTable layouts coexist while the recompaction runs, so live space spikes; a node already near capacity can hit OutOfSpaceException and wedge the compaction queue. Root cause: insufficient transient headroom for the full-table rewrite. Fix: abort further switches, add disk or archive cold partitions, and cap max_threshold (STCS) or throttle throughput so any single compaction stages less output at once; only resume once df -h shows the strategy’s transient requirement is comfortably free.
Back to Understanding STCS vs LCS vs TWCS