Compaction Strategy Cassandra Ops & Automation

Routing Reads Around a Slow Replica With Python in Cassandra 4.x/5.x

Speculation reacts to a slow replica after the read is already in flight; routing avoids the slow replica before the read leaves the client. When a node is dragging — compaction-bound disks, a failing SSD, an overloaded coordinator — the durable fix is to stop sending it read traffic until it recovers, and Cassandra gives you two independent mechanisms to do exactly that: the server-side dynamic snitch and the client-side latency-aware load balancing policy. This guide wires both together in the DataStax Python driver and adds a programmatic detector so your application knows which replica to shun without a human reading dashboards. It assumes Cassandra 4.0, 4.1, or 5.0, the cassandra-driver on Python 3.10+, and a multi-node cluster where one replica can lag. It sits under Fallback Routing & Read Path Optimization, the guide that treats degraded-replica routing as the durable fix and speculation as insurance; where its sibling configuring speculative_retry for read path resilience tunes the reactive layer, this page builds the proactive one.

Two layers: snitch and driver policy

Read routing is decided in two places. On the server, the dynamic snitch wraps your configured snitch and continuously re-scores every replica on recent read latency and reported severity (compaction and I/O load). When a replica scores badly enough, the coordinator reorders it out of the preferred set and reads go elsewhere — no client change required. Its sensitivity is dynamic_snitch_badness_threshold: a replica must score at least that fraction worse than the best replica before the coordinator routes around it. The default is 0.1 on 4.0 and 1.0 on 4.1+, and the value trades stability for reactivity — lower reacts faster but flaps more, higher is steadier but tolerates a slow replica longer.

On the client, the driver’s load balancing policy decides which coordinator each query targets. Wrapping DCAwareRoundRobinPolicy in TokenAwarePolicy sends each read to a replica that actually owns the data in the local datacenter, and layering LatencyAwarePolicy on top penalizes coordinators whose recent latency exceeds a configurable multiple of the fastest peer’s. The two layers are complementary: the snitch fixes coordinator-to-replica ordering inside the deployment; the driver policy fixes client-to-coordinator selection. The failure detection that also feeds the snitch’s view of a down node is the phi-accrual mechanism behind node gossip and failure detection.

The order in which the policies wrap matters as much as which ones you use. TokenAwarePolicy must be the outermost data-routing decision so reads still land on an owning replica — a latency-aware policy that ignored token ownership would push work onto a coordinator that then has to proxy every read across the ring, adding a network hop to the very requests you are trying to speed up. LatencyAwarePolicy therefore sits inside the token-aware wrapper in effect: token-awareness narrows the candidate set to owners, and latency-awareness picks the fastest owner among them. Getting this wrong is a common cause of “routing made latency worse” — the policy was demoting owners in favour of fast-but-non-owning coordinators. The LatencyAwarePolicy knobs then govern how it discriminates: scale weights recent samples over old ones so a node that just recovered is not penalized forever, retry_period sets how long a penalized node stays out, and minimum_measurements prevents a cold node from being excluded before there is enough evidence.

Pre-conditions & safety gates

Confirm the slow replica is genuinely slow — not merely marked down — and read the snitch’s own view before changing anything. Read-only checks:

# 1. Confirm all replicas are UN; a DN node is a failure-detector case, not a routing one.
nodetool status shop

Safety Check: If the target is DN, gossip/failure-detection already excludes it — routing tuning is the wrong tool. Proceed only when the node is UN but slow. Expected Output: All rows UN, with the suspect node’s Load or ownership possibly skewed.

# 2. Read the dynamic snitch's per-replica scores from JMX (lower = better).
nodetool sjk mxdump -q "org.apache.cassandra.db:type=DynamicEndpointSnitch" 2>/dev/null \
  || echo "Use Jolokia/JMX to read DynamicEndpointSnitch Scores attribute"

Safety Check: A replica whose score is far above its peers is a routing candidate. If all scores are close, the latency you see is client-side or coordinator-side, not replica-side.

# 3. Confirm the current badness threshold before tuning it.
grep dynamic_snitch_badness_threshold /etc/cassandra/cassandra.yaml

Safety Check: Record the value. Lowering it makes routing more reactive but risks flapping; changing it needs a rolling restart, so plan the change deliberately.

Implementation: latency-aware routing with a detector

The program below builds a session with token-aware, DC-aware, latency-aware routing, and adds a detector that samples per-host read latency through the driver’s own metrics to name the slow replica programmatically. It degrades safely: if the detector cannot decide, it changes nothing.

#!/usr/bin/env python3
# requirements: Python 3.10+, cassandra-driver>=3.28
"""Route reads away from a slow replica using the Python driver's latency-aware
load balancing, and detect the slow replica from per-host driver latency."""
import logging
import time
from cassandra import ConsistencyLevel
from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT
from cassandra.policies import (
    LatencyAwarePolicy,
    TokenAwarePolicy,
    DCAwareRoundRobinPolicy,
)
from cassandra.query import SimpleStatement

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")


def build_session(contact_points: list[str], local_dc: str) -> "Cluster":
    """LatencyAwarePolicy penalizes a host whose latency exceeds
    exclusion_threshold x the fastest host's, retiring it from routing until
    it recovers. It wraps TokenAware(DCAware) so ownership and locality still win."""
    base = TokenAwarePolicy(DCAwareRoundRobinPolicy(local_dc=local_dc))
    lb = LatencyAwarePolicy(
        base,
        exclusion_threshold=2.0,   # exclude a host >2x slower than the fastest
        scale=100,                 # ms; weights recent samples over old ones
        retry_period=10,           # seconds a penalized host stays excluded
        minimum_measurements=50,   # samples before a host can be excluded
    )
    profile = ExecutionProfile(
        load_balancing_policy=lb,
        consistency_level=ConsistencyLevel.LOCAL_QUORUM,
        request_timeout=2.0,
    )
    cluster = Cluster(
        contact_points=contact_points,
        execution_profiles={EXEC_PROFILE_DEFAULT: profile},
    )
    return cluster.connect()


def detect_slow_replica(session, sample_query: str, rounds: int = 200,
                        slow_multiple: float = 2.0) -> str | None:
    """Attribute per-host coordinator latency and flag any host whose mean
    exceeds slow_multiple x the fleet-wide best. Returns an address or None."""
    per_host_ms: dict[str, list[float]] = {}
    stmt = SimpleStatement(sample_query, is_idempotent=True)
    for _ in range(rounds):
        t0 = time.perf_counter()
        result = session.execute(stmt)
        elapsed_ms = (time.perf_counter() - t0) * 1000.0
        host = result.response_future.coordinator_host  # the host that served it
        if host is not None:
            per_host_ms.setdefault(str(host.address), []).append(elapsed_ms)

    means = {h: sum(v) / len(v) for h, v in per_host_ms.items() if len(v) >= 20}
    if len(means) < 2:
        logging.info("Insufficient per-host samples to decide; changing nothing.")
        return None
    best = min(means.values())
    for host, mean_ms in means.items():
        if mean_ms > slow_multiple * best:
            logging.warning("Slow replica %s: mean %.1fms vs best %.1fms",
                           host, mean_ms, best)
            return host
    logging.info("No replica exceeds %.1fx the fastest; fleet balanced.", slow_multiple)
    return None


if __name__ == "__main__":
    session = build_session(["10.0.1.11", "10.0.1.12", "10.0.1.13"], local_dc="dc1")
    # A cheap, idempotent probe read against a small partition.
    slow = detect_slow_replica(session, "SELECT now() FROM system.local")
    if slow:
        logging.warning("Consider de-weighting %s and checking its compaction backlog.", slow)
    session.cluster.shutdown()

Safety Check: LatencyAwarePolicy will not exclude a host until it has minimum_measurements samples, so a cold start never mis-routes; and the detector returns None rather than guessing when per-host samples are thin. Both fail toward doing nothing, which is the safe default for routing changes. The probe statement is marked is_idempotent so it is safe to retry. Expected Output:

2026-07-17 11:20:44 [WARNING] Slow replica 10.0.1.13: mean 41.8ms vs best 6.2ms
2026-07-17 11:20:44 [WARNING] Consider de-weighting 10.0.1.13 and checking its compaction backlog.

Once the detector names a replica, the server-side lever is the badness threshold. Lowering it makes the coordinator route around a slow replica sooner:

# cassandra.yaml — Cassandra 4.x/5.x (requires rolling restart)
# Lower = route around a slow replica sooner, at the cost of more re-ordering.
dynamic_snitch_badness_threshold: 0.2
dynamic_snitch_update_interval_in_ms: 100   # how fast scores react to a lagging node

Almost always, a replica the detector flags as slow is slow because it is behind on merging, so pair the routing change with a look at its queue — the depth-versus-velocity math for deciding when a node counts as degraded lives in async compaction tracking & metrics, and wiring the detector into a real telemetry pipeline is covered in Python monitoring for Cassandra compaction.

Keep the server-side and client-side levers reconciled rather than fighting each other. If the dynamic snitch is already reordering a slow replica out of the coordinator’s preferred set, an aggressive client LatencyAwarePolicy that also excludes it is redundant but harmless; the danger is the opposite pull, where a low badness threshold routes hard away from a node that the client policy still considers acceptable, so different clients disagree about which replica is healthy and load sloshes unpredictably. The stable configuration is a moderately conservative snitch threshold that only reacts to genuine degradation, plus a client policy tuned to smooth over short spikes — server-side for durable exclusion, client-side for fast local reaction. Change one lever at a time and re-measure, because their effects compound and a simultaneous change makes attribution impossible.

Verification steps

# 1. Confirm reads shifted off the slow replica.
nodetool proxyhistograms

Safety Check: Coordinator p99/p99.9 should fall once the slow replica stops serving reads. Compare before and after; no improvement means routing did not actually exclude it.

# 2. Re-run the detector after the change; it should flag nothing.
#    (reuse detect_slow_replica from the module above)

Safety Check: A second run returning None confirms the fleet rebalanced. If the same host still flags, the snitch threshold is too high to react — lower it or de-weight the host explicitly.

Troubleshooting

  • Flapping between replicas (request_timeout churn). Routing oscillates — a replica is excluded, recovers, is re-included, slows, and is excluded again — and latency gets worse, not better. Root cause: exclusion_threshold or dynamic_snitch_badness_threshold set so tight that normal jitter crosses it repeatedly, so the routing set never stabilizes. Fix: raise exclusion_threshold toward 2.0, lengthen retry_period and scale so decisions weight sustained latency over spikes, and raise the snitch threshold back toward 1.0 — steadier routing beats twitchy routing.
  • Snitch badness too aggressive (uneven load, one node idle). After lowering dynamic_snitch_badness_threshold, one replica goes cold while the rest carry extra load and heat up. Root cause: the threshold is low enough that a marginally slower node is fully excluded, concentrating traffic and creating the next slow node. Fix: raise the threshold so only a genuinely degraded replica is routed around, and let LatencyAwarePolicy’s gradual weighting handle mild differences instead of the snitch’s binary reorder.
  • Slow replica still selected as coordinator. Reads keep landing on the slow node even though it is the one lagging. Root cause: TokenAwarePolicy routes to a replica that owns the data, and if the slow node is the only local-DC owner of a hot partition, token-awareness keeps choosing it. Fix: confirm replication placement gives the partition more than one local replica, rely on server-side speculation to overtake it via configuring speculative_retry, and drain the node’s compaction backlog so it stops being slow rather than only routing around it.
Back to Fallback Routing Read Path Optimization