Lesson 33 · Asynchronous systems · Module 4
The Load You Defer
A queue does not remove work. It moves work in time — and it moves the pain from the side that measures to the side that does not. That trade is genuinely excellent, right up to the minute it becomes the outage.
The win in this lesson: you will be able to look at any queue in a proposed design and say what it is protecting, from what, and which single metric shows it failing. And you will stop watching depth.
1. The queue that looked like insurance
Someone puts a queue between two services and the design review ends. It is the most reflexively approved component in distributed systems, and for good reason:
"Queues are powerful tools for building reliable asynchronous systems. Queues allow one system to accept a message from another system, and persist the message until it is fully processed, even in the face of long outages, server failures, or problems with dependent systems."
Avoiding insurmountable queue backlogs, AWS Builders' Library · §"The duplicitous nature of queues"
All true. The same section names the price in the very next breath, and the price is not "complexity" — it is a unit of time:
"In the end, a queue increases a system’s durability and availability, at the price of occasional increased latency due to retries."
Avoiding insurmountable queue backlogs · §"The duplicitous nature of queues"
2. What a queue buys — and who it buys it for
The thing worth internalising is not that a queue costs latency. It is whose latency. The producer's request is finished the moment the broker acknowledges it. The consumer's request is finished when the work is actually done, and nothing bounds the gap between those two events.
The source states the measurement consequence flatly. Producer-side numbers are not a weak signal; they are a signal about a different system:
"Measuring error rates from the producer perspective gives us part of the availability picture, but not much of it. The producer availability is proportional to the queue availability of the system we’re using."
Avoiding insurmountable queue backlogs · §"Measuring availability and latency"
A queue gives the producer a bounded latency and gives the consumer an unbounded backlog. That is the whole trade in one line. Every good reason to add a queue — absorbing a burst, decoupling failure domains, smoothing a slow consumer — is a way of spending that unboundedness well. Every queue outage is a case of spending it badly.
3. What it defers: the arithmetic nobody does
Deferred work is still work, and it comes back at a rate you did not choose. The recovery cost is not proportional to the outage; it is proportional to the outage multiplied by how far behind you fell:
"Imagine a system that experiences an hour-long outage while processing messages. Regardless of the given rate and processing capacity, recovering from the outage requires double the system’s capacity for another hour after the recovery."
Avoiding insurmountable queue backlogs · §"Failures in asynchronous systems"
That is the benign case. The malignant one is a short spike, and the numbers are brutal:
"if the volume of queued messages was 10x the capacity that the consumer side was scaled to, this means it would take 300 minutes for the system to work through the backlog and recover. Even short load spikes can result in multi-hour recovery times, and therefore cause multi-hour outages."
Avoiding insurmountable queue backlogs · §"Backlogs"
Thirty minutes of unnoticed overload buys five hours of outage. Compare that with the synchronous system you replaced, which would simply have rejected the excess and been fine at minute thirty-one:
"Asynchronous services build up backlogs during outages, leading to long recovery times, unlike synchronous services, which drop requests during outages but have quicker recover times."
Avoiding insurmountable queue backlogs · §"Failures in asynchronous systems"
This is why the article calls queue-based systems bimodal. There is no gentle degradation between the two modes — there is a threshold, and it is crossed silently:
4. Depth and age — and why only one of them is the answer
Two numbers describe a queue. Depth is how many messages are waiting. Age is how long the oldest one has been waiting. They sound interchangeable. They are not, and the source is unambiguous about which it watches:
"Again, producer-observed latency mirrors the latency of our queue service itself. Therefore, we focus more on measuring the age of messages that are in the queue. This quickly catches cases where the systems are behind, or are frequently erroring out and causing retries."
Avoiding insurmountable queue backlogs · §"Measuring availability and latency"
| Depth | Age of oldest | |
|---|---|---|
| Units | Messages — meaningless without a drain rate | Seconds — directly comparable to a deadline |
| Threshold | Guessed. Is 50 000 bad? Depends on the day | Given by the business. "This is useless after 5 minutes" |
| Small slow drift | Invisible until it is enormous | Visible immediately, in the units that matter |
| Batch producer | Spikes hugely, harmlessly, every hour | Barely moves if the consumer keeps up |
| Answers | "How much work is left?" | "Is the work still worth doing?" |
Both are computable from the other only if you know the rates, and the relationship is worth memorising, because it explains why age moves earlier and more honestly than depth does:
# while arrivals exceed the drain rate, from the moment they start to:
age_of_oldest = elapsed * (1 - drain_rate / arrival_rate)
depth = elapsed * (arrival_rate - drain_rate)
drain_estimate = depth / drain_rate # if arrivals stopped this instant
# arrivals 1000/s, drain 200/s, 10 minutes in:
# age = 600 * (1 - 0.2) = 8 minutes → already past a 5 min deadline
# depth = 600 * 800 = 480 000 messages
# drain = 480000 / 200 = 40 minutes → of catching up, if the spike stops nowHere is the failure, drawn. Watch the two numbers move at different speeds, and watch which one crosses a line that anybody outside engineering could have told you about.
Depth is a capacity question; age is a correctness question. Depth tells you how much work is left. Age tells you whether the work is still worth doing — and only age can be compared against a promise you made to somebody. If you alarm on one number, alarm on age.
5. When the queue becomes the outage
Three mechanisms turn a helpful queue into the incident. They compound, which is why recovery is so slow.
5.1 The work being done is already worthless
A consumer that is far behind is not doing reduced-value work; past the deadline it is doing zero-value work, at full cost. The load-shedding article's image for this is exact — a late reply is a reply to nobody:
"If a service has gotten partway through serving a request and notices that the client has timed out, it can skip doing the rest of the work and fail the request at that point. Otherwise, the server keeps working on the request, and its late reply is like a tree falling in the forest."
Using load shedding to avoid overload, AWS Builders' Library · §"Keeping an eye on the clock"
Which yields the remedy, and it is a remedy about time, not size:
"In addition to bounding the size of queues, we’ve found it’s extremely important to place an upper bound on the amount of time that an incoming request sits on a queue, and we throw it out if it's too old. This frees up the server to work on newer requests that have a greater chance of succeeding."
Using load shedding to avoid overload · §"Watching out for queues"
5.2 The retries arrive
Consumers time out on a struggling dependency, so the messages go back on the queue. Clients time out on the producer, so they send again. You have met this loop before:
"Throwing away work creates a positive feedback loop that increases the overload, since clients often retry a request if a service doesn't respond in time. When that happens, one resource-consuming request turns into many resource-consuming requests, multiplying the load on the service."
Using load shedding to avoid overload · §"Finishing what was started"
Lesson 04 made this multiplicative across a call stack; a queue makes it durable. A synchronous retry storm ends when the clients give up. A queued one does not: the amplified load is now sitting on disk, waiting patiently for you to recover enough to be knocked over again.
5.3 The queues you did not know you had
Queue depth is at least visible when the queue is a broker you chose. Most of them are not:
"I’m often surprised to find in-memory queues where I didn’t intuitively think to look for them, in systems and libraries I depend on. When I’m digging into systems, I find that it’s helpful to assume there are queues somewhere that I don’t know about yet."
Using load shedding to avoid overload · §"Watching out for queues"
Thread-pool executors, socket buffers, connection pools, a load balancer's surge queue. Each of them defers load in exactly the way this lesson describes, with none of the instrumentation. The surge queue is the worst of them, because it destroys the one number you need:
"These queues can lead to brownout, because when a server finally gets a request, it has no idea how long the request was in the queue. A generally safe default is to use a spillover configuration, which fast-fails instead of queueing excess requests."
Using load shedding to avoid overload · §"Watching out for queues"
6. The levers, in the order you should reach for them
None of these makes the consumer faster. Each one decides, deliberately, which deferred work you are willing to abandon — which is the only decision available once arrivals exceed drain.
| Lever | What it does | What it costs |
|---|---|---|
| Backpressure | Scale an inbound throttle inversely with backlog size, so the producer is refused before the backlog is unrecoverable | Rejections become the producer's problem — and shared queues push back on innocent workloads too |
| Message TTL | Drop messages older than their useful lifetime at dequeue time | Only safe where a later sweep or full sync repairs the gap |
| Sidelining | Move old messages to a backlog queue and work the live one first — LIFO-ish behaviour, approximately | Ordering guarantees; the sidelined queue may never be drained |
| Delay queues | Re-enqueue with a delay so the system works fresher data first | Deferring the deferred work; easy to lose track of |
| Per-workload limits | Separate queues, separate thread pools, a semaphore per tenant | Lower utilisation, more moving parts to operate |
| Spillover | Fast-fail excess instead of queueing it at all | Visible errors now, in exchange for no recovery tail later |
Backpressure is the one to understand mechanically, because its implementation is embarrassingly simple — "occasionally measuring the queue depth of a workload (assuming a workload is on its own queue), and scaling an inbound throttle limit (inversely) proportionally to backlog size". That parenthesis is the catch: it needs per-workload queues to be fair.
And the preference for freshness is not an engineering aesthetic, it is what people asked for:
"Real time systems often are implemented with FIFO-ish queues, but prefer LIFO-ish behavior."
Avoiding insurmountable queue backlogs · §"Backlogs"
Lesson 03 put admission control at the front door of a synchronous service: decide who gets in, because the ones you let in you must serve. Asynchronous systems need it in every component, not just the front, precisely because they hoard rather than reject — "Because synchronous systems don’t tend to build up backlogs, we protect them with front-door throttling and admission control." A queue is a place where refused work goes to wait instead of dying, and that waiting is the thing you are on the hook for.
7. Residual risk
Do all of this and three risks remain, none of them fixable by configuration.
- Someone must choose the useful lifetime. Age only becomes an alarm once a human states the deadline a message is written against. No default is correct, and a wrong one either drops live work or hides a real backlog.
- Dropping is irreversible, and the drop happens under load. TTL and sidelining are load shedding by another name, executed by a system already in its worst state, on data whose value you inferred from its age alone.
- The consumer's capacity is still the real number. Every lever above buys time. If drain is chronically below arrivals, a queue is only converting an outage now into a larger outage later.
8. Check yourself
9. Back to your world
Take any queue in a system you work on and answer three questions in order. If the first has no answer, the queue is not a design decision, it is a habit.
- What is it protecting, and from what? Name the producer whose latency it bounds. "Decoupling" is not an answer; "so checkout returns in 30 ms even when the fulfilment service is redeploying" is.
- What is a message's useful lifetime? Ask someone outside engineering. If a message is worthless after five minutes, you now have an alarm threshold and a TTL, both for free.
- What is your drain rate, and what is age doing right now? Depth over drain rate is your recovery time. If you cannot produce either number in under a minute, you would not see the crossing — and the crossing is silent.
Then go looking for the queues you did not choose: the executor in front of your handler, the socket buffers, the load balancer's surge queue. Assume they are there, because they are.