Lesson 38 · Cascading failure, feedback loops, shedding

One Slow Dependency

A backend gets slower. Not down — slower. Within minutes every caller is out of threads, half the fleet is marked unhealthy and removed, and the instances still standing are carrying more traffic than the whole fleet carried an hour ago. Nobody deployed anything.

The win in this lesson: you will be able to name the feedback loop out loud during an incident — which quantity is rising, and what that rise feeds — and you will know why just restart it is, more often than not, the thing keeping the outage alive.

1. The definition does all the work

"A cascading failure is a failure that grows over time as a result of positive feedback."

Addressing Cascading Failures, Google SRE Book ch. 22 · chapter opening

Three words carry it: positive feedback. Not "a big outage", not "a lot of errors" — a loop in which the system's response to failure produces more failure. The book's own first example is the whole shape in one sentence:

"a single replica for a service can fail due to overload, increasing load on remaining replicas and increasing their probability of failing, causing a domino effect that takes down all the replicas for a service."

Addressing Cascading Failures, ch. 22 · chapter opening

2. The loop, drawn

The trigger here is deliberately boring: one dependency's latency goes from 100 ms to 4 s. It never returns an error. It never goes down. Watch what the fleet does to itself.

The pivot is that a health check is a request. It needs a thread, a scheduler slot, and a free file descriptor like everything else, so it is starved by exactly the conditions it exists to detect:

"When a thread can’t make progress because it’s waiting for a lock, health checks may fail if the health check endpoint can’t be served in time."

Addressing Cascading Failures, ch. 22 · §"Resource Exhaustion" → "CPU"

"Running out of file descriptors can lead to the inability to initialize network connections, which in turn can cause health checks to fail."

Addressing Cascading Failures, ch. 22 · §"Resource Exhaustion" → "File descriptors"

3. Where the capacity actually goes

Overload does not arrive as one clean resource limit. It arrives as a chain, where each exhausted resource consumes the next:

"If there is insufficient capacity to handle all the requests at steady state, the server will saturate its queues."

Addressing Cascading Failures, ch. 22 · §"Resource Exhaustion" → "CPU"

A queue looks like patience and behaves like latency. The book's arithmetic is worth memorising because it is so unflattering:

"For example, if the queue size is 10x the number of threads, the time to handle the request on a thread is 100 milliseconds. If the queue is full, then a request will take 1.1 seconds to handle, most of which time is spent on the queue."

Addressing Cascading Failures, ch. 22 · §"Queue Management"

The deadline trap: 5% of requests, 80% of errors

The most counterintuitive number in the chapter. A frontend of 10 servers, 100 threads each — 1,000 threads of capacity, normally 100 in use. Then "Suppose an event causes 5% of the requests to never complete."

normal          1,000 QPS x 0.1 s        =   100 threads busy of 1,000
the 95% good      950 QPS x 0.1 s        =    95 threads
the 5% stuck       50 QPS x 100 s        = 5,000 threads  ← the deadline, not the failure
available                                  1,000 threads
served                1,000 / (5,000 + 95) = 19.6%

"the frontend will only be able to handle 19.6% of the requests (1,000 threads available / (5,000 + 95) threads’ worth of work), resulting in an 80.4% error rate."

Addressing Cascading Failures, ch. 22 · §"Latency and Deadlines" → "Bimodal latency"

A 5% fault became an 80% outage, and the multiplier was your own deadline. The thread is not spent on the broken thing; it is spent waiting for permission to give up. Which is also why work done past a client's deadline is not worth doing:

"you don’t get credit for late assignments with RPCs."

Addressing Cascading Failures, ch. 22 · §"Latency and Deadlines" → "Missing deadlines"

4. Retry amplification: the engine you have already met

Lesson 04 built this engine; here is what it does when bolted to a fleet. A backend caps out at 10,000 QPS and starts rejecting the excess. The callers retry. And then:

"The volume of retries grows: 100 QPS of retries in the first second leads to 200 QPS, then to 300 QPS, and so on."

Addressing Cascading Failures, ch. 22 · §"Retries"

An overload of 1% compounds into the whole load budget, and then the layers multiply:

"If the database can’t service requests because it’s overloaded, and the backend, frontend, and JavaScript layers all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4^3) on the database."

Addressing Cascading Failures, ch. 22 · §"Retries"

"requests should only be retried at the layer immediately above the layer that is rejecting them."

Handling Overload, Google SRE Book ch. 21 · §"Handling Overload Errors"

And a budget caps what is left. Google's is two numbers: at most three attempts per request, and at most 10% of a client's traffic may be retries.

"layering on the per-client retry budget (a 10% retry ratio) reduces the growth to just 1.1x in the general case"

Handling Overload, ch. 21 · §"Handling Overload Errors" → "Deciding to Retry"

3x worst case becomes 1.1x. That is the difference between an incident and an outage, bought with a counter.

5. Why "just restart it" so often fails

Here is the part that costs teams hours. A cascading failure has hysteresis: the load at which it starts is not the load at which it stops.

"if a service was healthy at 10,000 QPS, but started a cascading failure due to crashes at 11,000 QPS, dropping the load to 9,000 QPS will almost certainly not stop the crashes. […] In this example, if 10% of the servers are healthy enough to handle requests, the request rate would need to drop to about 1,000 QPS in order for the system to stabilize and recover."

Addressing Cascading Failures, ch. 22 · §"Service Unavailability"

Restarting does not add capacity. It removes capacity now in exchange for capacity later — and "later" arrives cold, into the same traffic that killed the process the first time:

"The problem tends to snowball and soon all servers begin to crash-loop. It’s often difficult to escape this scenario because as soon as servers come back online they’re bombarded with an extremely high rate of requests and fail almost immediately."

Addressing Cascading Failures, ch. 22 · §"Service Unavailability"

Worse, the machinery that restarts for you is itself part of the loop:

"This practice may create a failure mode in which health-checking itself makes the service unhealthy."

Addressing Cascading Failures, ch. 22 · §"Stop Health Check Failures/Deaths"

The distinction that defuses it is one most services never make:

"Process health checking is relevant to the cluster scheduler, whereas service health checking is relevant to the load balancer."

Addressing Cascading Failures, ch. 22 · §"Stop Health Check Failures/Deaths"

"Make sure that you identify the source of the cascading failure before you restart your servers. […] Canary this change, and make it slowly. Your actions may amplify an existing cascading failure if the outage is actually due to an issue like a cold cache."

Addressing Cascading Failures, ch. 22 · §"Restart Servers"

6. The escapes

Every escape does the same thing: it breaks one arrow in the loop. The order below is the order you reach for them in an incident.

Shedding: reject early, cheaply, and without apology

Lesson 34 covered which requests to drop. This is the reason the mechanism has to exist at all — and the threshold has to be low enough to fire before threads are gone, not after.

"one effective approach is to return an HTTP 503 (service unavailable) to any incoming request when there are more than a given number of client requests in flight."

Addressing Cascading Failures, ch. 22 · §"Load Shedding and Graceful Degradation"

Shedding is not giving up; it is the only way to keep a promise:

"a backend task provisioned to serve a certain traffic rate should continue to serve traffic at that rate without any significant impact on latency, regardless of how much excess traffic is thrown at the task."

Handling Overload, ch. 21 · §"Conclusions"

"It's a common mistake to assume that an overloaded backend should turn down and stop accepting all traffic. However, this assumption actually goes counter to the goal of robust load balancing. We actually want the backend to continue accepting as much traffic as possible, but to only accept that load as capacity frees up."

Handling Overload, ch. 21 · §"Conclusions"

The trigger should be a signal the task owns. Google's default is the number of runnable threads compared to the number of processors — "executor load average" — because it notices saturation without needing to know what a request costs:

"We smooth this value with exponential decay and begin rejecting requests as the number of active threads grows beyond the number of processors available to the task."

Handling Overload, ch. 21 · §"Utilization Signals"

Headroom, and what it does not buy

Capacity planning sets how far the trigger is from normal. It is necessary and it is not sufficient, and the book says so in the same breath:

"Capacity planning reduces the probability of triggering a cascading failure, but it is not sufficient to protect the service from cascading failures."

Addressing Cascading Failures, ch. 22 · §"Preventing Server Overload"

Headroom buys minutes, not immunity. Minutes are enough — if shedding exists to use them.

Drain before restart, and stagger the restarts

Draining is the cheap half. Put the instance in lame-duck state first: it stops receiving new work, the requests already in flight complete, and only then does the process exit. Nothing is dropped, and the fleet loses one instance's worth of capacity rather than one instance's worth of in-flight requests as well.

Staggering is the other half, and it is Lesson 04's jitter applied to restarts. A rolling restart is a synchronised event by construction — and a synchronised restart is a trigger in its own right:

"Pushing a new version of the binary or updating its configuration may initiate a cascading failure if a large number of tasks are affected simultaneously."

Addressing Cascading Failures, ch. 22 · §"Process Updates"

# restart offset from a stable per-host value, not from random()
# same host, same slot, every time — so the incident stays reproducible
offset=$(( 0x$(hostname | md5sum | cut -c1-8) % 60 ))
sleep "$offset" && drain-then-restart

Drop traffic, then ramp it back

"Reducing load enough so that the crashing stops. Consider being aggressive here—if the entire service is crash-looping, only allow, say, 1% of the traffic through."

Addressing Cascading Failures, ch. 22 · §"Drop Traffic"

"When adding load to a cluster, slowly increase the load. The initially small request rate warms up the cache; once the cache is warm, more traffic can be added."

Addressing Cascading Failures, ch. 22 · §"Slow Startup and Cold Caching"

7. Residual risk

Three things stay broken after you have done all of the above.

The escapes are code paths you never run. Degraded mode, the shedding threshold, the 1%-and-ramp runbook — all of them are exercised for the first time during the worst hour of the quarter:

"Remember that the code path you never use is the code path that (often) doesn’t work."

Addressing Cascading Failures, ch. 22 · §"Load Shedding and Graceful Degradation"

The stated remedy is uncomfortable and correct: "You can make sure that graceful degradation stays working by regularly running a small subset of servers near overload in order to exercise this code path."

Ramping back into an unfixed trigger just restarts the cascade. Shedding buys you a stable fleet, not a solved problem. If the cause was a capacity shortfall or a slow dependency, returning traffic returns the outage.

Every one of these mechanisms is also a way to fail. The chapter's closing remark is the sentence to leave with:

"Retrying on failures, shifting load around from unhealthy servers, killing unhealthy servers, adding caches to improve performance or reduce latency: all of these might be implemented to improve the normal case, but can improve the chance of causing a large-scale failure."

Addressing Cascading Failures, ch. 22 · §"Closing Remarks"

Read the list again. Retries, failover, health-check eviction, caching. That is not a list of mistakes — it is the standard resilience toolkit, and every item on it is a feedback path. Resilience features are how cascading failures travel.

8. Check yourself

9. Back to your world

Four questions that take an afternoon and change what your next incident looks like:

• Does your readiness probe call a dependency? If it does, one slow backend can mark your entire fleet unready at once. Check whether the liveness probe does too — that one kills.
• What happens at 2x your normal load? Not in theory. If you have never load-tested to breaking point, you do not know which resource runs out first, and you will find out at the worst moment.
• Is there a way to shed? A concurrency limit that returns 503 before the queue fills is a few lines in most frameworks, and it is the difference between degraded and dead.
• Does your deploy drain, and is it staggered? A rolling restart with no draining and no jitter is a synchronised capacity loss you perform on purpose, several times a week.