Backpressure is the system saying no

Backpressure is what happens when a system says "not yet" instead of pretending it can say "yes" forever.

That sounds simple, but a surprising number of outages are just variations on ignoring it. A service gets slower. The caller keeps sending traffic. A queue grows. Retries multiply. Memory fills. Latency climbs. Dashboards show throughput staying flat for a while, which looks comforting if you do not look at the backlog. Then everything falls over at once and the incident channel discovers the queue was not a shock absorber. It was a fuse with no rating.

I like backpressure as a topic because it is both humble and everywhere. TCP has it. Reactive streams formalise it. Message queues need it. Thread pools need it. Humans have it too, though we call it a calendar.

A queue stores time

The most common mistake is treating a queue as spare capacity. It is not. A queue is stored waiting time. If work arrives faster than it completes, the excess has to go somewhere. You can reject it, slow the producer, shed lower-priority work, or let it wait. The queue is the waiting option.

Little's Law gives the tiny formula that makes this concrete:

$$L = \lambda W$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

Here $L$ is the average number of items in the system, $\lambda$ is the arrival rate, and $W$ is the average time an item spends in the system. If arrival rate goes up and service capacity does not, then either in-flight work grows, waiting time grows, or both. There is no configuration flag that exempts you from the equation.

Backpressure is feedback, not a queue

A queue by itself is not backpressure. It is just a buffer. Backpressure is the feedback signal that travels upstream when the buffer is full or getting dangerous. The producer learns that the consumer is saturated and changes behaviour.

That behaviour can take a few forms:

• Block: the producer waits until capacity is available.
• Drop: low-value work is discarded before it consumes more resources.
• Reject: the service returns a clear error, often with retry guidance.
• Slow down: the caller reduces rate, either by token bucket, adaptive concurrency, or stream demand.
• Batch: the system combines work so fixed overhead is paid less often.

The right choice depends on the domain. A telemetry pipeline can drop samples. A payment API should reject rather than silently discard. A video player can buffer. A control loop may need to skip stale commands because old work is worse than no work.

$$\rho = \frac{\lambda}{\mu}$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

When utilisation $\rho$ creeps toward 1, the system has no slack. Small bursts turn into long waits. That is the operational smell backpressure is supposed to catch early.

The retry trap

Retries are useful when the failure is transient and the downstream service has spare capacity. During overload, retries are often a small distributed denial-of-service attack launched by your own clients. The first request times out because the queue is long. The client sends another. Now the queue is longer. More clients time out. More retries arrive.

Backoff and jitter help because they turn a tight retry loop into a slower, spread-out signal. But they are still downstream of the main question: should this system accept more work right now?

This is why bounded queues feel harsh but are kinder. If a worker pool has 32 workers and a queue of 256, the 289th request gets a quick answer: try later, go elsewhere, or shed the optional path. With an unbounded queue, it gets accepted into a line that may already be seconds or minutes long. The caller thinks it has a promise. The service has only made a wish.

Streams make the signal explicit

The cleanest version of backpressure is a demand signal. In Reactive Streams, a subscriber asks for $n$ items. The publisher is not meant to send more than requested. That "request(n)" shape is a useful mental model even if you never use a reactive library. The consumer owns the pace because the consumer knows what it can absorb.

TCP does a related thing at another layer. Flow control stops a sender from overrunning the receiver's buffer, while congestion control tries not to overrun the network path. The details are very different from application queues, but the moral is the same: healthy protocols expose pressure upstream. They do not rely on infinite buffers and good luck.

Queue

waiting time

not spare capacity

Healthy limit

bounded

reject before memory becomes the bottleneck

Bad retry

amplifier

can multiply overload if not backed off

How I would design it

For a service boundary, I usually want four separate knobs, not one magic queue size.

First, set a concurrency limit around the expensive part. That might be database calls, external API calls, CPU-bound work, or a whole request handler. The limit should reflect the resource that actually saturates, not just the number of HTTP connections.

Second, make the queue bounded and small enough that waiting work still has a chance of being useful by the time it runs. A 10-minute queue for user-facing requests is just a slow error message.

Third, choose a rejection language. That might be HTTP 429 for rate limits, HTTP 503 with Retry-After for overload, a gRPC resource-exhausted status, or a domain-specific "come back later" response. The caller needs to know whether to retry, fail open, fail closed, or show the user a message.

Fourth, add load shedding before the expensive path. Optional enrichments, previews, analytics, recommendation calls, and best-effort side effects should be easier to turn off than the core transaction. Overload is not one thing; some work is more valuable than other work.

A small practical checklist

When a queue appears in a design, I think these are the useful questions:

• What is the maximum queue length, and why that number?
• What happens to item 1 past the limit?
• Does the producer learn about pressure immediately?
• Are retries capped, backed off, and jittered?
• Can stale work expire before it runs?
• Which work is dropped first when the system is hot?
• Is the dashboard showing queue length and age, not just throughput?

The "age" part matters. Queue length alone can hide trouble when item cost varies. A short queue of expensive items can be worse than a long queue of cheap ones. Oldest-item age is often the better panic gauge because users experience time, not list length.

Some food for thought: a lot of software reliability is just honesty about capacity. A system that says no early is not less available than one that says yes and times out. It is more honest about the work it can actually finish.

Reading further

• John D. C. Little, "Little's Law as Viewed on Its 50th Anniversary": the queueing result that makes backlog, throughput, and latency inseparable.
• Reactive Streams: the standard shape of asynchronous streams with non-blocking backpressure.
• IETF RFC 5681, TCP Congestion Control: the transport-layer version of pushing congestion information back into sender behaviour.
• Martin Thompson, "Applying Back Pressure When Overloaded": a practical systems note on bounded queues and upstream pressure.