Compaction Strategy Cassandra Ops & Automation

Configuring speculative_retry for Read Path Resilience in Cassandra 4.x/5.x

When one replica slows down — a compaction storm, a GC pause, a degrading disk — reads that happen to target it inherit its latency, and your p99.9 climbs even though the rest of the deployment is healthy. speculative_retry is the table-level lever that lets the coordinator stop waiting and fire a redundant read at a faster replica before the slow one times out. This guide is about configuring that option precisely: the full set of value forms, how to pick one from your measured latency distribution, and how to pair it with driver-side speculative execution so the protection spans both the coordinator and the client. It assumes Cassandra 4.0, 4.1, or 5.0, a cqlsh path to alter table options, and the DataStax Python driver for the client-side snippet. It sits under Fallback Routing & Read Path Optimization, which frames speculation as one layer of a compaction-aware read path; this page drills into the option’s exact syntax and calibration, while its sibling routing reads around a slow replica with Python handles keeping traffic off the slow node in the first place.

What the value forms mean

speculative_retry governs when the coordinator dispatches a second copy of an in-flight read. It is set per table and takes one of several forms:

  • 99PERCENTILE (default, also written p99 on 4.x/5.x) — the coordinator keeps a per-table latency histogram and speculates once elapsed time crosses that percentile of recent reads. Adaptive; it floats as load shifts.
  • <N>ms (for example 50ms) — a fixed threshold. Deterministic and independent of the histogram, which is what you want under a hard tail-latency SLA.
  • ALWAYS — speculate on every read immediately, doubling read load in exchange for the lowest possible tail latency. Reserve for tiny, latency-critical tables with ample I/O headroom.
  • NONE — never speculate; the coordinator waits out read_request_timeout_in_ms. Correct when peers have no spare capacity and redundant reads would only amplify load.
  • 4.x MIN(p99,50ms) / MAX(p99,50ms) — combine a percentile and a fixed value. MIN fires at whichever is sooner (more aggressive, bounds the window from above); MAX fires at whichever is later (more conservative, avoids speculating during a histogram dip). These composite forms are the precise tool when you want a percentile that also respects a floor or ceiling.

The value form is not cosmetic — it decides the trade-off the coordinator makes on every read, and the wrong one turns a resilience feature into a load multiplier.

Two properties of the percentile forms are worth understanding before you rely on them. First, the histogram is per table and per coordinator: each node maintains its own decaying latency histogram for each table, so a 99PERCENTILE window is computed from that node’s recent view, not a fleet-wide aggregate. A node that has only just started coordinating a table has too few samples to compute a meaningful percentile and effectively falls back to a conservative window until the histogram fills. Second, the histogram decays over time, so a table that swings between quiet and busy phases will see its percentile window widen and narrow with load — which is exactly the adaptivity you want for general traffic and exactly the unpredictability you do not want under a hard SLA. That tension is why the fixed ms and composite MIN/MAX forms exist: they let you pin or bound the window so it cannot drift outside a range you have validated.

Pre-conditions & safety gates

Never pick a threshold from intuition. Measure the real distribution first; these checks are read-only.

# 1. Capture per-table read percentiles. tablestats reports only the mean,
#    so use histograms for the p95/p99/p99.9 you will anchor the value to.
nodetool tablehistograms shop.orders

Safety Check: Record the p95 and p99 read-latency columns. Your fixed ms value must sit above p95, or you will speculate on nearly every read. Expected Output:

shop/orders histograms
Percentile  Read Latency (micros)
95%             1131.75
99%             4055.27

# 2. Confirm the current option before changing it (idempotency check).
cqlsh -e "SELECT speculative_retry, read_repair FROM system_schema.tables \
  WHERE keyspace_name='shop' AND table_name='orders';"

Safety Check: If the value already matches your target, skip the ALTER — it is a no-op that still bumps schema. Note that Cassandra 4.0 removed read_repair_chance/dclocal_read_repair_chance; read repair is now the independent read_repair option and does not interact with speculation. Expected Output: 99PERCENTILE | BLOCKING on a stock table.

# 3. Confirm the read timeout exceeds the widest speculative window.
grep read_request_timeout_in_ms /etc/cassandra/cassandra.yaml

Safety Check: read_request_timeout_in_ms (default 5000) must comfortably exceed your speculative window, or the read times out before the redundant copy can help.

Implementation: set the table option

Derive the value from step 1’s histogram, then apply it. Because a p95 of 1131 micros ≈ 1.1 ms with a p99 near 4 ms, a fixed window a little above p95 bounds the tail without speculating constantly.

-- Cassandra 4.x/5.x: bound tail latency with a fixed window just above p95,
-- and keep on-read convergence via BLOCKING read repair (independent option).
ALTER TABLE shop.orders WITH
  speculative_retry = '5ms'
  AND read_repair = 'BLOCKING';

For a table with a variable, bursty distribution where a fixed value would misfire during load swings, keep the adaptive percentile but bound it with a composite form so a histogram collapse cannot make the window absurdly tight:

-- 4.x/5.x composite: speculate at the p99, but never later than a 20ms ceiling.
ALTER TABLE shop.sessions WITH
  speculative_retry = 'MAX(p99,20ms)';

To turn speculation off on a table whose replicas have no spare I/O — for example a heavy analytical table where every replica is compaction-bound — use NONE so redundant reads do not pile onto saturated disks:

ALTER TABLE analytics.rollups WITH speculative_retry = 'NONE';

Table options change only through CQL — there is no nodetool verb for them — and the change takes effect on the next schema propagation with no restart. The reason a heavily-backlogged replica should be shielded rather than speculated at is the coupling explored in the compaction backlog analysis & alerting guide: speculation multiplies read load exactly when replicas are already struggling.

ALWAYS deserves a specific caveat because it is tempting and usually wrong. It sends a redundant read to a second replica on every single request, so it roughly doubles the table’s read load unconditionally — worthwhile only for a tiny reference table where the latency win justifies the cost and the extra I/O is negligible against the deployment’s total. On any table of meaningful size, ALWAYS converts a healthy cluster into one running at twice the intended read throughput, which manufactures the very saturation that then makes reads slow. Treat it as a diagnostic setting you might switch on briefly to confirm speculation helps at all, not a production default.

Driver-side speculative execution

Coordinator speculation protects the read once it reaches a coordinator; driver-side speculative execution protects the hop from client to coordinator, so a stalled coordinator does not sink the request. Configure both — they are complementary layers.

#!/usr/bin/env python3
# requirements: Python 3.10+, cassandra-driver>=3.28
"""Driver-side speculative execution to survive a slow coordinator, layered
on top of table-level speculative_retry which survives a slow replica."""
import logging
from cassandra import ConsistencyLevel
from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT
from cassandra.policies import (
    ConstantSpeculativeExecutionPolicy,
    TokenAwarePolicy,
    DCAwareRoundRobinPolicy,
)
from cassandra.query import SimpleStatement

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")


def build_session(contact_points: list[str], local_dc: str) -> "Cluster":
    """Wire token-aware routing plus constant speculative execution.

    Fire a redundant client-side request after 8ms of coordinator silence,
    up to 2 extra attempts. Keep this ABOVE the table's speculative_retry so
    the two layers do not both fire on the same normal read.
    """
    profile = ExecutionProfile(
        load_balancing_policy=TokenAwarePolicy(DCAwareRoundRobinPolicy(local_dc=local_dc)),
        # delay in seconds; 0.008 = 8ms, max_attempts counts the extra tries.
        speculative_execution_policy=ConstantSpeculativeExecutionPolicy(
            delay=0.008, max_attempts=2
        ),
        consistency_level=ConsistencyLevel.LOCAL_QUORUM,
        request_timeout=2.0,
    )
    cluster = Cluster(
        contact_points=contact_points,
        execution_profiles={EXEC_PROFILE_DEFAULT: profile},
    )
    return cluster.connect()


if __name__ == "__main__":
    session = build_session(["10.0.1.11", "10.0.1.12"], local_dc="dc1")
    # Speculative execution only applies to idempotent statements — mark it so.
    stmt = SimpleStatement(
        "SELECT * FROM shop.orders WHERE id = %s",
        is_idempotent=True,
    )
    rows = session.execute(stmt, (42,))
    logging.info("Fetched %d row(s) with speculative execution enabled.", len(rows.current_rows))
    session.cluster.shutdown()

Safety Check: Driver speculative execution runs only on statements marked is_idempotent=True — the driver refuses to duplicate a non-idempotent write, and forgetting the flag silently disables the protection. Keep the driver delay (8 ms) above the table’s window (5 ms) so the two layers stagger rather than both firing on a normal read. Expected Output:

2026-07-17 10:04:19 [INFO] Fetched 1 row(s) with speculative execution enabled.

Verification steps

# 1. Confirm the option committed on the live schema.
cqlsh -e "SELECT speculative_retry FROM system_schema.tables \
  WHERE keyspace_name='shop' AND table_name='orders';"

Safety Check: The value must read back exactly as set. If it shows the old value, the ALTER did not propagate — check for schema-disagreement warnings in system.log.

# 2. Confirm speculation is helping, not thrashing.
nodetool tablestats shop.orders | grep -iE "speculative"

Safety Check: Watch Speculative retries against total reads. A speculative-retry count approaching the read count means the window is too tight — every read is speculating and you have doubled load. Re-run proxyhistograms and confirm p99.9 actually dropped.

Troubleshooting

  • Speculative retries amplifying load (UnavailableException under load). Read load roughly doubles and reads start failing shortly after tightening speculative_retry. Root cause: the window sits below real p95, so nearly every read fires a redundant copy, and the redundant traffic lands on peers that are themselves busy — inducing the latency it was meant to prevent. Fix: ALTER TABLE ... WITH speculative_retry = '99PERCENTILE' to restore the adaptive default, re-derive a fixed value strictly above p95 from fresh tablehistograms, and confirm peers have I/O headroom before speculating at all.
  • PERCENTILE mis-set (tail latency unchanged). You set 99PERCENTILE expecting relief but p99.9 does not move. Root cause: on a table with a heavy-tailed, multimodal distribution the 99th percentile is already so high that the coordinator waits nearly the full slow-read duration before speculating, so the redundant read fires too late to win. Fix: switch to a composite MIN(p99,<N>ms) that caps the window from above, choosing <N> just over p95, so speculation fires soon enough to overtake the slow replica.
  • Interaction with read consistency level. Speculation appears to have no effect at QUORUM/LOCAL_QUORUM. Root cause: speculation only helps when there is a faster replica left to try — at a CL that already requires responses from most replicas, there is little slack for a redundant read to overtake, and at ALL there is none. Fix: keep read CL at LOCAL_QUORUM or lower where the SLA allows, so the coordinator has an unused faster replica to speculate toward; if you must run ALL, speculation cannot help and routing around the slow replica is the only lever — see routing reads around a slow replica with Python.
Back to Fallback Routing Read Path Optimization