System Design.

Reference · quick sheet

The Retry Playbook

Timeouts create retries, retries multiply, and backoff without jitter just gives the spike a rhythm.

Choosing a timeout

  1. Pick an acceptable rate of false timeouts — say 0.1%.
  2. Read the matching percentile off the downstream service's latency distribution — p99.9 for 0.1%.
  3. If p99.9 is close to p50 (tight latency bounds), pad it. Otherwise a small wobble becomes a flood of timeouts.
  4. Re-derive it when the downstream service changes. A timeout is a fact about someone else's distribution, not a constant of your own.

The jitter algorithms

no jitter      sleep = min(cap, base * 2**attempt)
full jitter    sleep = random(0, min(cap, base * 2**attempt))
equal jitter   temp  = min(cap, base * 2**attempt)
               sleep = temp/2 + random(0, temp/2)
decorrelated   sleep = min(cap, random(base, sleep * 3))

Equal Jitter is the loser — more work than Full Jitter and much slower. Full Jitter does the least work; Decorrelated finishes sooner. With 100 contending clients, jitter cut the call count by more than half. "It should be considered a standard approach for remote clients."

Vocabulary

Selfish retry
The correct mental model: a retry spends more of the server's resources to raise one client's chance of success. Cheap when failures are rare and uncorrelated; destructive when failures are caused by load.
Retry amplification
Retries at N layers multiply rather than add. Five layers × three tries = 243× the load at the bottom. Fix: retry at a single point in the stack, and let the rest fail fast.
Capped exponential backoff
Exponential waits with a ceiling. The ceiling creates its own problem — every client then retries forever at the capped rate — so also bound the number of attempts and fail earlier.
Jitter
Randomness added to a delay so that clients which failed together do not return together. Needed because backoff alone preserves correlation.
Retry budget (token bucket)
Retry freely while tokens remain, then at a fixed rate. Continuous and local, versus a circuit breaker's binary and global behaviour. In the AWS SDK since 2016. The client-side twin of admission control.
Modal behaviour
A state a system rarely enters, and therefore never exercises — a circuit breaker's "open" mode is reached for the first time during an incident. AWS's stated reason for preferring budgets: circuit breakers "introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery" (Timeouts, retries, and backoff with jitter, §"Retries and backoff", bullet "Load").

Before you retry anything

  1. Is the failure correlated? A dropped packet is worth retrying. An overload is everyone failing at once, and your retry makes the cause worse.
  2. Is the call idempotent? A timeout does not tell you whether the side effect happened. No idempotency, no safe retry — add a client token if you need one.
  3. How many layers already retry? Multiply, don't add. Then remove all but one.
  4. Is the backoff jittered? Plain exponential means synchronised clients.
  5. Is there a cap on attempts and on total retry traffic? Backoff slows a client down; only a budget stops the fleet.

Jitter is not only for retries

Alignment is the default: a schedule is an instant, not a window, and good NTP makes the fleet fire as one. Things that align without anyone intending it:

  • Cron and Kubernetes CronJob — the schedule is shared by every replica.
  • Rolling deploys and restarts — every host's periodic timer starts within seconds of the others.
  • Autoscaling — new hosts begin periodic work together, on a boundary.
  • Monitoring agents — scrape and flush on the minute, fleet-wide.
  • Token and session refresh — issued together, so they expire together.
  • Cache TTLs — keys warmed together expire together. A thundering herd on a timer, caused by nothing but the expiry being synchronised at birth.

And you will not see it: "These spikes of traffic can be very short, and are often hidden by aggregated metrics." A one-second spike is one sixtieth of a one-minute bar. Graph per-second before concluding it is not happening.

# stable per-host offset — spreads the fleet, keeps incidents reproducible
offset=$(( 0x$(hostname | md5sum | cut -c1-8) % 300 ))

# cache TTLs: plain randomness is right, there is nothing to debug per key
ttl = base + random(0, base * 0.1)

The rule: stable, identity-derived offset when the actor is something you will have to debug (a host, a worker, a shard). Plain randomness when it is not (a cache key). Both spread the load; only the first keeps the incident reproducible.