Lesson 34 · Load shedding, criticality, quotas

Choosing What to Drop

When demand exceeds capacity, every request cannot be served. That is arithmetic, not engineering. The only question left is which ones fail — and if you decline to answer it, the system answers for you, by slowing everything down until everything times out.

The win in this lesson: you will be able to say, for a service you own, what it drops first, second and third, where in the stack it drops them, and what a dropped request costs. And you will stop believing that a full queue is kinder than a fast rejection.

1. The only question is which requests fail

A service with capacity for 400 requests per second is offered 1,000. Six hundred of them are not going to be served. You may choose the six hundred deliberately, or you may refuse to choose — but refusing is not neutral. It is a decision to let contention pick, and contention picks the worst possible set: it fails all thousand, slowly.

That is not rhetoric. It falls out of how a busy machine behaves:

"Computers take on work even when they’re overloaded, but they spend increasing amounts of their time context switching and become too slow to be useful."

Using load shedding to avoid overload, AWS Builders' Library · §"The anatomy of overload"

A machine does not push back. It accepts everything and degrades, and the degradation is not linear — "services reach an inflection point where their performance starts degrading even more rapidly". Past that point, extra load buys you nothing and costs you everything.

2. What overload actually looks like

The word you need is goodput, and it is the measurement that makes the whole argument visible:

"Goodput is the subset of the throughput that is handled without errors and with low enough latency for the client to make use of the response."

Using load shedding to avoid overload · §"The anatomy of overload"

Throughput is what you are doing. Goodput is what anybody got. A server at 100% CPU, replying to every request three seconds after the client gave up, has magnificent throughput and zero goodput. And the crossover has a startlingly crisp definition:

"If the service's median latency is equal to the client timeout, half of the requests are timing out, so the availability is 50 percent."

Using load shedding to avoid overload · §"The anatomy of overload"

Your p50 crossing the caller's timeout is the moment a latency problem becomes an availability problem. There is no alarm for it by default, and it is one graph away from being obvious.

Here is the contrast, at the level of actual messages. Same fleet, same offered load, same client timeout. The only difference is what each server does at the front door.

That is the whole idea, stated by the source in one line:

"When a server approaches overload, it should start rejecting excess requests so that it can focus on the requests it decides to let in."

Using load shedding to avoid overload · §"Preventing work from going to waste"

And the payoff is not fewer errors — it is a different shape of error: "the server maintains high availability for the requests it accepts, and only the excess traffic’s availability is affected". You have converted a total brownout into a partial one, which is the only trade actually on offer.

3. Reject cheaply, and reject early

Shedding only works if a rejection is much cheaper than a response. If it is not, you have invented a new and slightly worse way to be overloaded. The SRE book states the failure mode with an exclamation mark, which for that book is shouting:

"the backend can become overloaded even though the vast majority of its CPU is spent just rejecting requests!"

Handling Overload, Site Reliability Engineering (Google) · §"Client-Side Throttling"

This is not hypothetical. The same section notes that "it's almost equally expensive to reject a request that requires a simple RAM lookup […] as it is to accept and run that request". If your served path is cheap, your shed path has to be nearly free, or the ratio does not save you.

Where the rejection happens decides what it costs

"Early rejection is important because it’s the cheapest place to drop excess traffic, but it comes at a cost to visibility. This is why we protect in layers: to let a server take on more than it can work on and drop the excess, and log enough information to know what traffic it is dropping."

Using load shedding to avoid overload · §"Protecting in layers"

The server sheds what it can see and afford to log; the layer in front absorbs the volumes the server cannot survive even rejecting. Two mechanisms, two purposes, deliberately.

One honest exception, worth knowing because it sounds wrong: "In rare cases, quickly dropping a request can be more expensive than holding on to the request." A rejection that returns in a microsecond invites the caller to try again immediately. Where that happens, the source deliberately slows the rejection "to match (at the minimum) the latency of successful responses" — but only "when the cost of holding on to requests is as low as possible; for example, when they’re not tying up an application thread".

4. What to drop first

Now the interesting part. You have decided to drop 600 requests per second. Which 600?

The first answer is not about business value at all, and almost everyone gets it wrong:

"The most important request that a server will receive is a ping request from a load balancer. If the server doesn't respond to ping requests in time, the load balancer will stop sending new requests to that server for a period of time, and the server will sit idle."

Using load shedding to avoid overload · §"Prioritizing requests"

Drop the health check and you do not lose one request — you lose the host, and its share of the load moves to hosts that are also struggling. Overload removing capacity from an overloaded fleet is the fastest route to a full outage, and "in a brownout scenario, the last thing we want to do is to reduce the size of our fleets".

Criticality as a first-class field

Beyond the ping, you need a label on the request that says how droppable it is. The SRE book's scheme is four values, carried by the RPC system itself:

Two design decisions inside that table are worth more than the table. First, capacity is planned against the top two tiers only: "Services are expected to provision enough capacity for all expected CRITICAL and CRITICAL_PLUS traffic." The lower tiers are explicitly using headroom you have not bought. Second, the tiers are a strict order, not a hint:

"When a task is itself overloaded, it will reject requests of lower criticalities sooner."

Handling Overload · §"Criticality"

Two practical notes the source is firm about. Set the tier at the edge — "Our practice is thus to set the criticality as close as possible to the browsers or mobile clients" — because only there do you know whether a human is waiting. And propagate it: a request issued while serving a SHEDDABLE request inherits SHEDDABLE, so a backend five hops down sheds correctly without knowing anything about the caller.

Per-customer quotas: one tenant must not take the fleet

Criticality decides which kind of request dies. Quotas decide whose. Without them, the tenant that ships a bad loop on Tuesday consumes the capacity of every other tenant, and your incident is about someone else's bug.

"When global overload does occur, it's vital that the service only delivers error responses to misbehaving customers, while other customers remain unaffected."

Handling Overload · §"Per-Customer Limits"

Two details make this work in practice. The first is the unit. Quotas are set in resources, not request counts, because the source is scathing about the alternative: modelling capacity as "queries per second" "often makes for a poor metric". A request's cost varies with the caller, the code and the hour, and "A moving target makes a poor metric for designing and implementing load balancing." So the limits are written in CPU seconds per second — "Gmail is allowed to consume up to 4,000 CPU seconds per second", "Every other user is allowed to consume up to 500 CPU seconds per second".

The second is that they are deliberately oversubscribed against a fleet of "10,000 CPUs allocated worldwide":

"Note that these numbers may add up to more than the 10,000 CPUs allocated to the backend service. The service owner is relying on the fact that it's unlikely for all of their customers to hit their resource limits simultaneously."

Handling Overload · §"Per-Customer Limits"

Quotas are therefore not a capacity guarantee. They are a blast-radius guarantee: they bound how much of the fleet any one tenant can take, which is a different and more achievable promise.

5. Shed, don't queue

The counter-intuitive one, and the hardest to accept: under overload, a queue is not a buffer. It is a machine for converting requests into expired requests.

A queue is a bet that the burst will end before the backlog outlives the client's patience. Above capacity that bet always loses, and the longer the queue, the more certainly it loses — the request at the back is guaranteed to be stale by the time anyone looks at it. Meanwhile the queue has hidden the overload from every signal you were watching.

"In addition to bounding the size of queues, we’ve found it’s extremely important to place an upper bound on the amount of time that an incoming request sits on a queue, and we throw it out if it's too old. This frees up the server to work on newer requests that have a greater chance of succeeding."

Using load shedding to avoid overload · §"Watching out for queues"

Note what that implies, and say it out loud because it sounds like heresy: under overload the newest request is the most valuable one, because it has the most time left. That is an argument for LIFO, and the source makes it: "we look for ways to use a last in, first out (LIFO) queue instead, if the protocol supports it."

The same conclusion was reached at the load balancer, and the product changed:

"A generally safe default is to use a spillover configuration, which fast-fails instead of queueing excess requests."

Using load shedding to avoid overload · §"Watching out for queues"

"The Classic Load Balancer used a surge queue, but the Application Load Balancer" rejects excess traffic instead. A generation of load balancer design turned on exactly this point.

Give the server a deadline to enforce

Dropping stale work requires knowing what stale means, and the server cannot infer it. The fix is for the client to say so: "One way to avoid this wasted work is for clients to include timeout hints in each request, which tell the server how long they’re willing to wait." Propagate the remaining budget at every hop — "we propagate the “remaining time” deadline between each hop" — so the service at the bottom of a five-deep call chain can tell that its answer is already worthless.

def admit(req, now):
    # 1. never shed the thing that keeps you in the fleet
    if req.op == "healthcheck":
        return ADMIT

    # 2. the client already gave up — the cheapest possible win
    if req.deadline <= now:
        return DROP_EXPIRED            # counted, not logged as an error

    # 3. it has been sitting in our own queue too long
    if now - req.enqueued_at > MAX_QUEUE_AGE:
        return DROP_STALE

    # 4. resources, not request counts
    if tenant_cpu_rate(req.tenant) > quota(req.tenant):
        return DROP_OVER_QUOTA

    # 5. utilisation vs a threshold that rises with criticality
    if load_average() > threshold[req.criticality]:
        return DROP_SHED

    return ADMIT

Every branch above the last is free or nearly free, and each one runs before a thread is committed. That ordering is the design: the checks that cost nothing come first, and the expensive judgement comes last.

6. Residual risk

Load shedding is a mechanism that lies to your other mechanisms. Four ways it bites back.

It can switch off your autoscaling. This is the one that turns a good afternoon into a long night:

"If misconfigured, load shedding can disable reactive automatic scaling. Consider the following example: a service is configured for CPU-based reactive scaling and also has load shedding configured to reject requests at a similar CPU target. In this case, the load shedding system will reduce the number of requests to keep the CPU load low, and reactive scaling will never receive or get a delayed signal to launch new instances."

Using load shedding to avoid overload · §"Load shedding effects on automatic scaling"

Your shedder holds CPU at 80%, your scaler scales at 80%, and the fleet sits perfectly stable while refusing half its traffic. Scale on shed rate, or on offered load, never solely on the signal your shedder is controlling.

It poisons your latency graphs. Rejections are fast, and they drag the median down:

"if a service is load shedding 60 percent of its traffic, the service's median latency might look pretty amazing even if its successful request latency is terrible, because it’s being under-reported as a result of fast-failing requests."

Using load shedding to avoid overload · §"Visibility"

Report latency for served requests only, and shed rate as its own series. A latency graph that improves during an incident is a broken graph, and it has fooled better teams than yours.

It hides how close you are to the edge. If you use CPU to estimate headroom, shedding makes CPU lie: "a fleet might run much closer to the point at which requests would be rejected than system metrics indicate". That matters most when you are sizing for the loss of an availability zone, and the answer is to test: "if they haven’t load tested their service to the point where it breaks, and far beyond the point where it breaks, they should assume that the service will fail in the least desirable way possible." The pass condition is stated plainly — "The ideal load test result is for goodput to plateau when the service is close to being fully utilized, and to remain flat even when more throughput is applied."

It can be wrong. A rejection issued while capacity remained is a false positive, and the target is uncompromising: "We strive to keep a service’s false positive rate at zero." A steady trickle of them means your thresholds are too tight or your load balancing is uneven — a different bug wearing a shedding costume.

7. Check yourself

8. Back to your world

Pick a service you own and answer these four. If you cannot, the answers exist anyway — they are just being chosen by your thread pool.

• What does it drop first? Name the actual request type. "We would drop the less important ones" is not an answer; the code has no idea which those are.
• What does a rejection cost? Measure it. If a 503 costs a third of a 200, your shedding buys you a third less than you think.
• Where are the queues? Socket buffers, executor queues, connection pools, the load balancer. Assume there is one you have not found — "I find that it’s helpful to assume there are queues somewhere that I don’t know about yet."
• Can one tenant take the whole fleet? If there is no per-caller limit in resources, the answer is yes, and you will find out on their schedule.