Lesson 04 · Timeouts, retries, backoff, jitter

Retries Are Selfish

A call fails, so you retry it. Obviously correct — and it is also the standard mechanism by which a struggling service becomes a dead one, a partial outage becomes a total one, and a system that would have recovered in seconds stays down for an hour.

The win in this lesson: you will be able to look at any client-side retry and say what it costs the server, where in a stack it belongs, and what must be true about the API before it is safe at all. And you will remember one number: 243.

1. Timeouts, and how to actually pick one

Before retries there is the timeout, because a timeout is what creates a retry. Most teams choose one by feel — a round number that sounds patient. Amazon's method is mechanical, and it is the third time in four lessons that a percentile has decided an engineering question:

"when we make one service call another service, we choose an acceptable rate of false timeouts (such as 0.1%). Then, we look at the corresponding latency percentile on the downstream service (p99.9 in this example)."

Timeouts, retries, and backoff with jitter, AWS Builders' Library · §"Timeouts"

Read it backwards and it is obvious: a timeout is a promise about how often you are willing to be wrong. Decide the error rate you can live with, then let the downstream service's own latency distribution tell you the number. You are not picking a duration; you are picking a point on a curve.

Both directions hurt. Too short and you abandon requests that were about to succeed — then retry them, adding load to a service that was merely slow. Too long and you hold connections, threads and memory while a dependency is already lost. And the method has a stated limit: it "doesn't work with services that have tight latency bounds, where p99.9 is close to p50", where you need to pad, or a tiny wobble converts into a flood of timeouts.

2. The reframe

"Retries are 'selfish.' In other words, when a client retries, it spends more of the server's time to get a higher chance of success."

Timeouts, retries, and backoff with jitter, AWS Builders' Library · §"Retries and backoff"

This is the sentence to keep. A retry is not a neutral act of robustness — it is a client asserting that its request matters enough to consume more of a shared resource. When failures are rare and random, that trade is excellent and nearly free. When failures are caused by overload, it is the worst possible response, and the paper is blunt about the consequence: retries "can even delay recovery by keeping the load high long after the original problem".

3. 243

"Consider a system where the customer's call causes a five-deep stack of service calls. It ends with a query to a database, and three retries at each layer. What happens when the database starts failing queries under load? If each layer retries independently, the load on the database will increase 243x, making it unlikely to ever recover."

Timeouts, retries, and backoff with jitter · §"Retries and backoff", bullet "Distributed systems often have multiple layers"

The remedy is a one-liner with real organisational teeth: "for low-cost control-plane and data-plane operations, our best practice is to retry at a single point in the stack." One layer owns retries; everyone else fails fast and propagates. The cost is named honestly too — retrying at the top "may waste work from previous calls, which reduces efficiency". You trade efficiency for a bounded blast radius.

4. Backoff, and the problem it creates

The first fix is to stop retrying immediately: wait, and wait longer each time — exponential backoff. Because exponentials grow fast, implementations cap the wait, giving capped exponential backoff. And that cap introduces the next problem, stated plainly:

"Now all of the clients are retrying constantly at the capped rate."

Timeouts, retries, and backoff with jitter · §"Retries and backoff"

Backoff slows a client down; it never makes one stop. So the other half of the answer is to limit the number of attempts and fail earlier — which costs nothing in practice, because "the client is going to give up on the call anyway, because it has its own timeouts".

5. Jitter: the cheapest idea in distributed systems

Now the subtle failure. Backoff should fix overload, and often it barely does — because of correlation:

"If all the failed calls back off to the same time, they cause contention or overload again when they are retried."

Timeouts, retries, and backoff with jitter · §"Jitter"

A thousand clients fail at the same instant, all wait exactly 100 ms, and all return at the same instant. You have not spread the load — you have moved a spike and given it a rhythm. The fix is to add randomness, and the four standard variants are worth knowing by name:

no jitter      sleep = min(cap, base * 2**attempt)
full jitter    sleep = random(0, min(cap, base * 2**attempt))
equal jitter   temp  = min(cap, base * 2**attempt)
               sleep = temp/2 + random(0, temp/2)
decorrelated   sleep = min(cap, random(base, sleep * 3))

The simulation results are unambiguous: "Of the jittered approaches, 'Equal Jitter' is the loser." Full Jitter and Decorrelated Jitter both win, trading a little differently — "The 'Full Jitter' approach uses less work, but slightly more time." And with 100 contending clients, jitter "reduced our call count by more than half". The author's verdict is as close to an unqualified recommendation as this field gets:

"The return on implementation complexity of using jittered backoff is huge, and it should be considered a standard approach for remote clients."

Exponential Backoff and Jitter — Marc Brooker, AWS Architecture Blog, 4 March 2015

6. Jitter everything else too

The under-appreciated half. Jitter is not a retry technique; it is a technique for anything that many machines do at the same time:

"When building systems, we consider adding some jitter to all timers, periodic jobs, and other delayed work."

Timeouts, retries, and backoff with jitter · §"Jitter"

Because clients with many servers "can line up and trigger their requests at the same time. This can be the first few seconds of a minute, or the first few seconds after midnight for daily jobs." Every cron at 0 0 * * * is a synchronised thundering herd with a schedule.

Why periodic work aligns by default

Alignment is not bad luck; it is the expected outcome. 0 0 * * * does not mean "some time around midnight", it means midnight — an instant, not a window. Run that crontab on four hundred hosts and you have scheduled four hundred simultaneous jobs.

And the thing that would naturally spread them out is clock drift, which we have spent decades engineering away. NTP makes this worse. The better your time synchronisation, the more precisely your fleet fires as one. It is a small irony worth holding onto: we work hard to make machines agree about the time, and then have to work hard to make them disagree about when to act on it.

Crontabs are only the obvious source. Anything that starts a timer at the same moment on many machines produces the same effect:

That last row is the one that will bite you first, and it links this lesson back to the previous one. Warm a cache with ten thousand keys at deploy time, all with ttl = 300, and in five minutes they all become misses at once. No traffic spike caused it and no code changed — the expiry was synchronised at birth.

Why you will not see it on a graph

"These spikes of traffic can be very short, and are often hidden by aggregated metrics."

§"Jitter"

A one-second spike inside a one-minute average is one sixtieth of the bar. Your dashboard is not being subtle about hiding it — at that resolution, the spike does not exist. Which is why the fix begins with looking differently, and the result is stated in capacity, not latency:

"By paying attention to per-second load, and working with clients to jitter their periodic workloads, we accomplished the same amount of work with less server capacity."

§"Jitter"

How to actually spread them

The instinct is to sleep for a random interval at the top of the job. Better is to derive a stable offset from the runner's identity, so the fleet spreads and each host keeps its own fixed slot:

# crude, and real: a fixed per-host offset inside the period
offset=$(( 0x$(hostname | md5sum | cut -c1-8) % 300 ))
sleep "$offset" && run-the-job

# better, if the scheduler allows it: shift the schedule, not the job.
# a sleeping process still holds a slot, and makes the job's own
# runtime graph a lie.

For cache TTLs the actor is a key, not a host, and there is nothing to debug per-key — so plain randomness is right there:

ttl = base + random(0, base * 0.1)   # 300s becomes 300–330s, spread

That is the distinction worth carrying: use a stable, identity-derived offset when the thing doing the work is something you will one day have to debug — a host, a worker, a shard. Use plain randomness when it is not, as with a cache key. Both spread the load; only the first keeps the incident reproducible.

7. Budgets, not breakers

The popular answer to retry storms is the circuit breaker — stop calling a dependency entirely once errors cross a threshold. Amazon's position is notably unfashionable:

"circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery."

("addition time" is the source's own wording, quoted as printed.)

Timeouts, retries, and backoff with jitter · §"Retries and backoff", bullet "Load"

A breaker adds a mode: your system now has an "open" state, reached rarely, exercised never, and entered for the first time during an incident. Their alternative is a token bucket retry budget: "This allows all calls to retry as long as there are tokens, and then retry at a fixed rate when the tokens are exhausted." Built into the AWS SDK since 2016.

Compare the two shapes. A breaker is binary and global — retries are on, then abruptly off for everyone. A budget is continuous and local — retries stay available, capped as a proportion of traffic, degrading smoothly with no new state to test. And you have seen this shape before: it is Lesson 03's admission control, moved to the client. Facebook capped who may enter the expensive path at one per key per ten seconds; AWS caps how much of your traffic may be retries. Same idea, opposite end of the wire.

8. The precondition everyone skips

"In general, our view is that APIs with side effects aren't safe to retry unless they provide idempotency. This guarantees that the side effects happen only once no matter how often you retry."

Timeouts, retries, and backoff with jitter · §"Retries and backoff", bullet "Deciding when to retry"

Read-only calls are usually idempotent for free. Anything that creates a resource is usually not — so APIs that want to be retryable provide an explicit mechanism, like EC2's RunInstances client token, which lets the server recognise a repeat and decline to do the work twice.

9. Check yourself

Two of these are from Lesson 01, deliberately un-revised and several days cold. Getting them wrong now is more useful information than getting them right the day you read them.

10. Back to your world

Almost every HTTP client library you use has retries on by default, with a count someone chose without knowing what would sit above or below it. That is the 243 diagram, already running in production, waiting for a dependency to slow down. Three questions worth asking of any codebase:

• How many layers of my stack retry? Count them, multiply the counts, and look at the number you get.
• Is the backoff jittered? If it is plain exponential, your clients are synchronised and the spike is intact.
• Are the retried calls idempotent? If not, you do not have a resilience feature — you have a duplicate-writes feature.