TWCS vs STCS for Time-Series With Late-Arriving Data in Cassandra 4.x/5.x

Not every time-series workload has clean, monotonically increasing timestamps. Mobile clients buffer readings offline and flush them hours later, upstream Kafka consumers replay partitions after an outage, and batch backfills rewrite yesterday’s data every night. Each of those patterns produces a write whose event timestamp lands in a time range Cassandra has already sealed, and that single fact decides whether TimeWindowCompactionStrategy is a good fit or an active liability. This guide is a decision aid for choosing between TWCS, SizeTieredCompactionStrategy (STCS), and Cassandra 5.0’s UnifiedCompactionStrategy (UCS) when out-of-order arrival is a permanent property of the pipeline rather than a transient anomaly. It assumes Cassandra 4.0, 4.1, or 5.0 and a nodetool/cqlsh path to a canary table. It sits under Strategy Selection for Time-Series Workloads; read that first, because the window-sizing rule it establishes — roughly 20–30 windows across the data lifetime — is the baseline this page stress-tests against late data. The clean-timestamp deployment path is covered separately in configuring TWCS for IoT sensor data streams; this page is about knowing when not to take that path.

How late writes land in old windows

TWCS assigns each SSTable to the window that contains the maximum timestamp of the data it holds. A memtable flush that mixes fresh readings with a handful of backfilled rows from last week does not land cleanly in the current window — the older cells drag the SSTable’s coverage backward, and once that SSTable ages, TWCS treats it as belonging to the newest timestamp present. The corrosive case is subtler: a write whose cells carry an old CQL WRITETIME (either an explicit USING TIMESTAMP or a skewed client clock) creates data that logically belongs in a closed window but physically sits in a new SSTable. When that new SSTable’s window and an old sealed window both cover overlapping token ranges, TWCS can be forced to compact across window boundaries to preserve read correctness, and the sealed-window-is-immutable invariant that makes TWCS cheap evaporates.

The immediate consequence is that whole-window expiry stalls. TWCS drops an expired window as a single file deletion only when every SSTable in that window is fully past default_time_to_live plus gc_grace_seconds and no live data overlaps. A late write that reopens or overlaps a window resets that clock: the file can no longer be dropped whole, so Cassandra falls back to the row-by-row tombstone management and garbage collection path that TWCS exists to avoid. Disk that should have been reclaimed by a unlink() instead requires a full tombstone-scanning compaction, and until that runs the expired data lingers.

How much lateness actually breaks the model is a question of degree, and that is what makes the decision subtle. A write that arrives a few seconds after its event time almost always lands in the same window it belongs to, because windows are hours or days wide — the skew is smaller than the window, so nothing reopens. The model degrades only when the skew is comparable to or larger than a window: a reading buffered for six hours on a phone, replayed against a six-hour window, has a coin-flip chance of landing one window late. This is why the same pipeline can run TWCS flawlessly with day-wide windows and fall apart with hour-wide ones — the failure is not “late data” in the abstract but late data measured against your specific compaction_window_size. Quantifying that ratio, rather than assuming it, is the whole point of the calibration below.

Pre-conditions & safety gates

Before committing a strategy, quantify how late your writes actually are. These checks are read-only; run them on a representative table over a full ingest cycle.

# 1. Measure write-time skew directly: compare the cell WRITETIME to now.
#    Large negative deltas mean data is arriving well after its event time.
cqlsh -e "SELECT sensor_id, reading_ts, WRITETIME(value) \
  FROM metrics.readings LIMIT 20;"

Safety Check: Convert WRITETIME (microseconds since epoch) and compare against the row’s logical reading_ts. If the spread routinely exceeds one compaction_window_size, TWCS window immutability is already broken. Expected Output: Rows where WRITETIME(value) trails reading_ts by minutes are fine; a trail of hours or days is the signal that matters.

# 2. Look for cross-window recompaction in the log — the fingerprint of late data.
grep -E "Compacting.*TimeWindowCompactionStrategy|dropping expired SSTable" \
  /var/log/cassandra/system.log | tail -40

Safety Check: Under healthy TWCS, only the active window recompacts. Repeated compactions naming SSTables whose generation timestamps span days mean sealed windows are being reopened. Expected Output: Ideally, dropping expired SSTable lines with no matching old-window Compacting lines.

# 3. Confirm the read-amplification cost of the overlap.
nodetool tablehistograms metrics.readings | grep -A2 "Percentile"

Safety Check: SSTablesPerRead at p99 should stay at 1–2 for correctly bucketed TWCS. A climbing value with a flat data volume confirms overlap-driven fan-out.

If any gate shows sustained skew, do not simply “tune TWCS harder” — the strategy choice itself is what is under review.

Implementation: a skew-aware strategy chooser

The script below samples write-time skew across a table’s partitions and recommends a strategy. It is read-only and idempotent — running it never mutates schema; it only reports. Wire its output into your migration decision rather than guessing.

#!/usr/bin/env python3
# requirements: Python 3.10+, cassandra-driver>=3.28
"""Recommend TWCS, STCS, or UCS for a time-series table based on measured
write-time skew. Read-only: computes a recommendation, never alters schema."""
import logging
import time
from dataclasses import dataclass
from cassandra.cluster import Cluster
from cassandra.query import SimpleStatement

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")


@dataclass(frozen=True)
class SkewReport:
    sampled_rows: int
    median_skew_s: float
    p95_skew_s: float
    window_size_s: int

    def recommend(self) -> str:
        """A window's worth of skew breaks TWCS immutability; grade against it."""
        if self.sampled_rows == 0:
            return "INSUFFICIENT DATA — widen the sample window"
        if self.p95_skew_s < 0.5 * self.window_size_s:
            return "TWCS — skew is well inside one window; whole-file expiry holds"
        if self.p95_skew_s < 3 * self.window_size_s:
            return "UCS (C* 5.0) or STCS — moderate skew reopens windows; prefer a strategy that tolerates out-of-order writes"
        return "STCS — heavy backfill; TWCS whole-window expiry will never fire cheaply"


def measure_skew(keyspace: str, table: str, ts_column: str, value_column: str,
                 window_size_s: int, sample_limit: int = 5000) -> SkewReport:
    cluster = Cluster(contact_points=["127.0.0.1"])
    session = cluster.connect(keyspace)
    try:
        # WRITETIME is in microseconds; the logical ts column is in ms or s.
        stmt = SimpleStatement(
            f"SELECT {ts_column}, WRITETIME({value_column}) AS wt "
            f"FROM {table} LIMIT %s",
            fetch_size=1000,
        )
        skews: list[float] = []
        for row in session.execute(stmt, (sample_limit,)):
            if row.wt is None or getattr(row, ts_column) is None:
                continue
            write_epoch_s = row.wt / 1_000_000.0
            event_epoch_s = getattr(row, ts_column).timestamp()
            # Positive skew = data written after its event time (late arrival).
            skews.append(max(write_epoch_s - event_epoch_s, 0.0))
        if not skews:
            return SkewReport(0, 0.0, 0.0, window_size_s)
        skews.sort()
        median = skews[len(skews) // 2]
        p95 = skews[int(len(skews) * 0.95)]
        return SkewReport(len(skews), median, p95, window_size_s)
    finally:
        cluster.shutdown()


if __name__ == "__main__":
    # A 6-hour window = 21600 seconds; adjust to your configured window.
    report = measure_skew("metrics", "readings", "reading_ts", "value",
                          window_size_s=21600)
    logging.info("Sampled %d rows | median skew %.0fs | p95 skew %.0fs",
                 report.sampled_rows, report.median_skew_s, report.p95_skew_s)
    logging.info("Recommendation: %s", report.recommend())

Safety Check: The query uses LIMIT and fetch_size so it never scans an unbounded range, and it closes the session in finally. It reads from a single coordinator — run it against a few nodes and take the worst skew, since backfill traffic is rarely uniform. Expected Output:

Sampled 4980 rows | median skew 40s | p95 skew 190s
Recommendation: TWCS — skew is well inside one window; whole-file expiry holds

When the report points away from TWCS, the two escape hatches differ sharply. On Cassandra 5.0, UCS absorbs out-of-order data gracefully because its scaling_parameters govern tiering by size rather than by time, so a late write is just another SSTable to merge — no window to violate. On 4.x, STCS is the conservative fall-back for the same reason: it never assumed temporal ordering to begin with. The trade-off is real and is worked through in the STCS, LCS, and TWCS comparison: STCS reclaims expired data only through tombstone-scanning compactions, so you trade TWCS’s cheap file-drop for tolerance of disorder.

Two TWCS knobs deserve explicit warnings. max_sstable_age_days is a legacy DateTieredCompactionStrategy option; it does not exist on TWCS and setting it does nothing on 4.x/5.x — reach for compaction_window_unit/compaction_window_size instead. And unsafe_aggressive_sstable_expiration (false by default) tells TWCS to drop fully expired SSTables without checking whether they overlap live data. With clean timestamps that is a large I/O win; with late-arriving data it is dangerous, because a backfill can write a live cell into a token range whose old SSTable is about to be dropped, and aggressive expiration will delete it out from under the still-unrepaired replicas, resurrecting deleted rows once a lagging replica returns.

Verification steps

Whichever strategy you land on, prove the expiry path is actually cheap.

# 1. Confirm expired data leaves by file deletion, not tombstone scan (TWCS).
nodetool compactionstats --human-readable

Safety Check: Only the active window should show compaction activity. Old windows appearing in the active-compaction table mean overlap is still forcing recompaction.

# 2. For STCS/UCS, confirm tombstone purging is keeping pace.
nodetool tablestats metrics.readings | grep -E "tombstone|SSTable count"

Safety Check: A rising Average tombstones per slice under STCS means gc_grace_seconds or repair cadence, not the strategy, is the bottleneck — validate against the anti-entropy repair schedule before re-tuning.

Troubleshooting

Windows never dropping (OutOfSpaceException as disk fills). Expired windows keep occupying disk and eventually a compaction cannot stage its output. Root cause: late writes overlap old windows, so TWCS refuses the whole-file drop and the tombstone-scan compaction never gets scheduled ahead of ingest. Fix: route the backfill stream to a separate STCS table, keep the live TWCS table for in-order data, and only re-merge at query time. Never enable unsafe_aggressive_sstable_expiration to force the drop — it resurrects data if repair is behind.
Cross-window compaction storms. compaction_history shows repeated compactions rewriting SSTables whose timestamps span many windows, and iowait spikes independent of the window cadence. Root cause: client clock skew or an USING TIMESTAMP backfill is minting new SSTables that overlap sealed windows. Fix: enforce server-authoritative timestamps at the coordinator, reject writes with implausibly old timestamps at the application layer, and if disorder is unavoidable, migrate the table to UCS on 5.0 or STCS on 4.x.
TombstoneOverwhelmingException on reads of old ranges. Reads of aged partitions scan thousands of tombstones instead of skipping dropped files. Root cause: out-of-order deletes (a backfill that both inserts and deletes historical rows) leave tombstones stranded in reopened windows past their expiry. Fix: keep gc_grace_seconds uniform and at or above the repair interval, run a targeted nodetool repair on the affected range, and consider isolating delete-heavy backfills onto STCS where single-SSTable tombstone compaction can reclaim them predictably.

Strategy Selection for Time-Series Workloads — the parent guide that establishes the window-to-TTL sizing this page stress-tests against late data.
Configuring TWCS for IoT Sensor Data Streams — the clean-timestamp deployment path to take once you have confirmed TWCS is the right fit.
Understanding STCS vs LCS vs TWCS — the strategy trade-offs behind choosing STCS or UCS when disorder is permanent.
Tombstone management and garbage collection — the row-by-row purge path that TWCS falls back to when whole-window expiry stalls.

Back to Strategy Selection for Time Series Workloads