Producer–consumer separates the code that generates work from the code that performs it, connected by a queue. It is the foundational concurrency pattern: thread pools, pipelines, event loops and stream processors are all built on it.
The pattern itself is three lines. Everything that matters is in two decisions people skip: how large the queue is, and what happens at shutdown.
The structure
Diagram, as text
- Producer leads to Bounded queue capacity N
- Producer leads to Bounded queue capacity N
- Bounded queue capacity N leads to Consumer
- Bounded queue capacity N leads to Consumer
- Bounded queue capacity N leads to Consumer
- Bounded queue capacity N leads to Producer (queue full: producer blocks)
Producers add; consumers take. Neither knows the other exists, neither needs to run at the same rate, and the queue absorbs the difference.
What that buys:
- Rate decoupling. A burst of work is absorbed rather than dropped or blocking the producer immediately.
- Independent scaling. Slow consumers get more consumers; the producer is untouched.
- A single synchronisation point. All the concurrency lives in the queue, which is written once and correct, rather than being spread across the participants. This is the pattern's real contribution: the mechanisms covered under concurrency and thread safety — locks, visibility, atomicity — are needed in one place instead of everywhere.
The bound is the whole design
An unbounded queue is the default in most standard libraries and it is almost always wrong.
Unbounded means the producer never waits. When consumers cannot keep up, the queue grows until memory is exhausted — and the failure is not a clean rejection but an out-of-memory error, often in an unrelated part of the process, long after the actual overload began.
Bounded means the producer blocks when the queue is full, and that blocking is backpressure. It propagates the slowdown upstream to whoever is generating work, which is exactly where the decision to shed load belongs. A web handler that cannot enqueue can return 503 immediately, which is a far better outcome than accepting a request that will time out.
Choosing the bound is not about memory alone. A queue of 10,000 items with a consumer processing 100 per second holds 100 seconds of latency — every item at the back is already too late to be useful. The queue depth is a latency budget, and sizing it by "how much memory can we spare" produces systems that are technically alive and functionally dead.
Shutdown is where work gets lost
The naive shutdown — set a flag, wait a moment, exit — loses whatever was in the queue. Real shutdown has an order, and getting it wrong is how deploys drop work silently.
- Stop accepting new work. The producer's entry point starts rejecting. This must happen first, or the queue never drains.
- Signal that no more will arrive. A sentinel value per consumer — the poison-pill idiom — or a closed-channel flag. Consumers blocked waiting for work must be woken; otherwise they wait forever and the process hangs.
- Let consumers drain the queue, with a deadline.
- On deadline expiry, decide explicitly. Discard the remainder and log the count, or persist it. Silently exiting is the failure mode.
A consumer blocked on take() will not notice a flag. This is the single most common shutdown bug: the flag is checked at the top of the loop, and the loop is parked inside the queue waiting for an item that will never come.
The consumer loop is where correctness lives
async function consume(queue) {
while (true) {
const item = await queue.take();
if (item === POISON) return;
try {
await handle(item);
} catch (err) {
logger.error({ err, item: item.id }); // one bad item must not kill the consumer
}
}
}An unhandled exception kills the consumer thread, and a pool that silently loses consumers one at a time degrades until nothing is processed — with no error, because the error was the thing that vanished. Catch inside the loop, log with enough identity to find the item, and monitor the live consumer count.
Poison messages — items that fail every time — need a retry limit and a dead-letter destination, or one bad item is retried forever while everything behind it waits.
Variations worth knowing
Work stealing. Each consumer has its own queue and takes from another's when idle. Removes contention on a single shared queue and balances uneven work. This is how most modern task schedulers are built.
Batching consumers. Take up to N items at once. Dramatically cheaper for work with per-operation overhead — database writes, network calls — at the cost of latency for the first item in a partial batch, which needs a timeout.
Pipelines. Several stages, each a producer–consumer pair. The slowest stage sets the throughput, and every intermediate queue must be bounded or the fastest stage will exhaust memory feeding the slowest.
Check your understanding
Common mistakes
An unbounded queue. Overload becomes memory exhaustion instead of backpressure.
No policy for a full queue. Block, reject, drop-oldest and drop-newest are different decisions with different correct uses.
Sizing by memory rather than latency. A deep queue holds work that has already expired.
A shutdown flag with no wake-up. Consumers blocked in take() never see it.
Exceptions escaping the consumer loop. Consumers die silently and throughput degrades with no error.
No dead-letter path. One poison message is retried forever while everything behind it waits.
Monitoring depth alone. An empty queue means healthy or dead, and depth cannot tell you which.
Unbounded intermediate queues in a pipeline. The fastest stage exhausts memory feeding the slowest.
Practice
An image-processing service accepts uploads over HTTP and processes them on a worker pool. Under load it crashes with out-of-memory errors, and during deploys some uploads are never processed. Diagnose both and design the fix.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Two symptoms, two classic causes — and they are related.
Show a strong answer
Both symptoms come from the same design gap: the queue was treated as an implementation detail rather than as the system's admission control.
The out-of-memory crash: an unbounded queue. The HTTP handler accepts every upload and enqueues it. When processing is slower than arrival — which is guaranteed during any traffic spike, since processing is CPU-bound and accepting is not — the queue grows without limit, holding image data. The process dies of memory exhaustion, and it typically dies somewhere unrelated, which is why this gets misdiagnosed as a leak.
The lost uploads: shutdown discards the queue. On deploy the process is signalled and exits, and everything enqueued but unprocessed vanishes. The client received a 202 and reasonably believes the work will happen. Nobody notices because nothing errors.
The fix, in order:
- Bound the queue, sized by latency. If processing takes 2 seconds per image with 8 workers, that is 4 images per second; a 60-second acceptable wait means a bound around 240. Deriving it this way rather than guessing means the number can be defended and revisited when the worker count changes.
- Reject when full, do not block. This is request-driven work, so a fast 503 with a
Retry-Afteris correct — the client can retry or back off. Blocking the HTTP handler would tie up connections and convert a queue problem into a connection-pool problem. - Do not hold image bytes in the queue. Write the upload to object storage and enqueue a reference. This alone removes most of the memory pressure and makes the bound about latency rather than about megabytes.
And the shutdown, which is the more serious defect because it loses accepted work:
- Stop accepting first — fail readiness so the load balancer drains, then reject new uploads.
- Wake the workers with a close or poison pill; a worker blocked on take will otherwise hang the shutdown.
- Drain with a deadline shorter than the orchestrator's grace period, so the process exits on its own terms rather than being killed.
- Persist or explicitly log anything left. With the reference-in-queue design above, the images are already durable in object storage, so the real fix is to make the queue itself durable — which is the honest answer here.
The design conclusion I would draw: an in-memory queue is the wrong mechanism for work that was acknowledged to a client. Once the service has returned 202, the work must survive a restart, and that means a durable queue with acknowledgement rather than a thread-pool queue. The bounded in-memory queue then protects the workers, and durability protects the promise made to the client — two different jobs that were being done by one unbounded list.
Design a batching consumer that writes events to a database, balancing throughput against latency. State every parameter and what happens at shutdown and on partial failure.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
A batch that never fills must still be written eventually.
Show a strong answer
Parameters, each with a reason:
- Maximum batch size, derived from what the database handles well in one statement — often a few hundred to a thousand rows. Beyond that, statement size and lock duration start to hurt.
- Maximum wait, the time a partially filled batch may sit before being written anyway. This is the latency guarantee, and without it a low-traffic period leaves events unwritten indefinitely — the failure people discover at 3 a.m. when the last event of the evening never appeared.
- Maximum batch bytes, because a hundred large events can exceed a statement limit that a thousand small ones would not. Size alone is not a sufficient bound.
- Concurrency — how many batches may be in flight. One is simplest and preserves ordering; more improves throughput and forfeits ordering unless partitioned.
The loop: accumulate until size, bytes or the timer fires, whichever comes first. The timer must start when the first item of the batch arrives, not when the previous batch was written — otherwise the wait is measured from the wrong point and a single late item can sit for two full intervals.
Partial failure is the hard part, and it needs a decision rather than a default:
- If the write is atomic, the whole batch succeeds or fails and retry is straightforward — provided the write is idempotent, which for event inserts means a unique key per event rather than an auto-increment.
- If some rows fail — a constraint violation on one event — retrying the whole batch will fail forever on the same row. The batch must be split: retry the remainder, and send the offending event to a dead-letter destination with its error. A naive implementation retries the batch indefinitely and blocks every event behind it, which is the poison-message failure at batch scale.
- Bound the retries and make the dead-letter path visible, or failures are silent.
Shutdown:
- Flush the partial batch. This is the step that gets missed, and it loses up to a full batch of events on every deploy — invisible unless someone counts.
- Stop accepting, wake the consumer, write what is held, then exit — with a deadline, and an explicit log of anything discarded.
What I would monitor: average batch fill ratio and flush reason. If most batches flush on the timer rather than on size, the batch size is too large for the traffic and is buying latency for no throughput. If most flush on size, the wait parameter is never binding and latency is set by arrival rate. Those two numbers tell you whether the parameters are tuned, and neither is visible from throughput alone.
Key takeaways
- Producer–consumer decouples work generation from processing through a queue, concentrating all the synchronisation in one place.
- The bound is the design. Unbounded queues convert overload into memory exhaustion; bounded queues convert it into backpressure.
- Size the queue from a latency budget, not from available memory — depth ÷ throughput is the wait for the last item.
- Choose the full-queue policy deliberately: block for internal pipelines, reject for request-driven work, drop-oldest for telemetry.
- Shutdown has an order: stop accepting, wake blocked consumers, drain with a deadline, then explicitly handle the remainder.
- A flag alone does not stop a consumer blocked in
take(). Use a poison pill or a closed queue. - Catch exceptions inside the consumer loop, or consumers die silently and throughput degrades with no error.
- Monitor depth, enqueue wait and live consumer count — depth alone cannot distinguish healthy from dead.
Further reading
- Brian Goetz and others, Java Concurrency in Practice (Addison-Wesley, 2006) — bounded queues, shutdown and the poison-pill idiom.
- Doug Lea, Concurrent Programming in Java, second edition (Addison-Wesley, 1999).
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — backpressure and shedding load.
- Robert Blumofe and Charles Leiserson, "Scheduling Multithreaded Computations by Work Stealing" (Journal of the ACM, 1999).
- Jeffrey Dean and Luiz André Barroso, "The Tail at Scale" (Communications of the ACM, 2013) — queueing and tail latency.