A user clicks "place order". Your handler does this:
async function placeOrder(order) {
await db.save(order); // 20ms
await paymentGateway.charge(order); // 800ms
await emailService.sendReceipt(order);// 400ms
await warehouse.notify(order); // 600ms
await analytics.track(order); // 200ms
return { ok: true }; // total: ~2 seconds
}Two problems, and the second is worse.
It is slow. The user waits two seconds for work they do not care about. They want confirmation the order was placed; they do not need to wait for an analytics event.
It is fragile. If the email provider is down, the whole request fails — and the customer is told their order failed, when in fact it was saved and paid for. Your order system's availability is now the product of five services' availability. Five dependencies at 99.9% each gives you 99.5%: roughly four hours of downtime a month, caused mostly by things that are not essential to placing an order.
The fix: hand work to a queue
Do the essential work synchronously. Publish an event for everything else.
async function placeOrder(order) {
await db.save(order);
await paymentGateway.charge(order);
await queue.publish('order.placed', order); // ~2ms
return { ok: true }; // total: ~820ms
}Separate workers consume order.placed and send the email, notify the warehouse, record analytics. The user gets a response in under a second, and if the email service is down the message simply waits until it recovers.
- The order service publishes an event to the queue and answers the request at once.
- A pool of workers consumes the queue on its own schedule.
Describe this diagram
Components
- client (Client)
- orders (Service)
- queue (SQS queue)
- workers (Service)
Connections
- client sends to orders
- orders sends to queue
- queue sends to workers
What a queue actually buys you
Decoupling. The order service does not know who consumes its events. Adding a fraud-check consumer requires no change to order code.
Load levelling. This one is underrated. A traffic spike produces a spike of messages, not a spike of work. The queue absorbs the burst and workers drain it at their own pace. Without it, a flash sale takes down your email provider.
Retries. A failed message goes back on the queue and is retried. Compare with a synchronous call, where a failure means the work is simply lost unless you wrote retry logic yourself.
Independent scaling. If image processing is slow, run 20 image workers and 2 email workers. You scale the bottleneck, not everything.
Queue or log: the distinction that matters
The word "queue" covers two genuinely different designs, and conflating them is a common interview error.
Message queue (RabbitMQ, SQS). A message is delivered to one consumer and then removed. The queue is a to-do list that shrinks as work is done. Ideal for task distribution — resize this image, send this email.
Event log (Kafka, Kinesis). Messages are appended to a durable, ordered log and retained for a configured period. Many independent consumer groups read the same stream, each tracking its own position. Nothing is removed when read.
| Message queue | Event log | |
|---|---|---|
| After consumption | Deleted | Retained |
| Consumers per message | One | Many independent groups |
| Replay history | No | Yes |
| Ordering | Usually per queue | Guaranteed per partition |
| Best for | Distributing tasks | Event streams, multiple consumers, replay |
The decisive question is whether more than one system cares about the event, and whether you might want to replay it. Adding a new analytics consumer that needs the last month of orders is trivial with a log and impossible with a queue.
Delivery guarantees
Three options, and only two are real.
At-most-once. Send and forget. Fast, and messages can be lost. Acceptable for metrics; never for orders.
At-least-once. Retry until acknowledged. Nothing is lost, but duplicates happen — a consumer may process a message, crash before acknowledging, and receive it again. This is the practical default.
Exactly-once. What everyone wants and almost nothing truly provides end-to-end. Systems advertising it usually mean exactly-once within their own boundary, which does not extend to the side effects your consumer performs — a charge sent to Stripe cannot be un-sent by the queue.
The workable approach is at-least-once delivery plus idempotent consumers:
async function handleOrderPlaced(msg) {
const done = await redis.set(`processed:${msg.id}`, '1', 'NX', 'EX', 86400);
if (!done) return; // already handled — skip
await sendReceipt(msg.order);
}Make processing the same message twice harmless, and duplicates stop mattering. This is the same idea as idempotency keys in the API lesson, applied to consumers.
Ordering
Most queues do not guarantee global order, because parallel consumers process messages concurrently — that parallelism is the point.
Usually this is fine. Sometimes it is not: account.created must be processed before account.updated.
The standard solution is partitioning by key. All messages sharing a key go to the same partition and are processed in order by one consumer, while different keys process in parallel. Key by user_id and each user's events stay ordered without serialising the whole system.
Note the trade-off, familiar from sharding: strict ordering means no parallelism within that key, so one very active key becomes a bottleneck.
Dead letter queues
A message that fails repeatedly is dangerous. Without a plan it retries forever, consuming capacity and often blocking the messages behind it. This is the poison message problem, and it takes down real systems.
A dead letter queue is where messages go after N failed attempts. They stop blocking the main queue, and they are preserved for inspection rather than silently discarded.
Two disciplines make this work: alert on dead letter depth — a filling DLQ means something is systematically broken — and make replay possible once you have fixed the cause.
Pair this with exponential backoff on retries. Retrying a struggling downstream service immediately and repeatedly is how you turn its bad minute into an outage.
Backpressure
Producers can outpace consumers. The queue grows. Left alone, it exhausts memory or disk and the broker fails, taking everything with it.
Your options are all uncomfortable, which is why you should choose deliberately rather than discover it:
- Scale consumers — the right answer when you can.
- Slow producers down by rejecting or throttling at the edge.
- Shed load by dropping low-value messages.
- Cap the queue and fail fast rather than accumulating unboundedly.
The queue depth trend is the metric that matters here. Depth rising steadily means consumers cannot keep up, and it is a leading indicator — you have until the queue fills to react.
Check your understanding
Common mistakes
Using a database table as a queue. Polling works at small scale and becomes a contention problem as you add workers. Use a real broker.
No dead letter queue. One malformed message retries forever and blocks the queue behind it.
Assuming ordering you were never promised. Verify what your broker actually guarantees, and partition by key when you need it.
Making the queue a hard dependency. If your API returns 500 when the broker is unavailable, you traded one fragile dependency for another. Decide what happens when publishing fails.
Putting essential work in the queue. Charging a card asynchronously means telling users their order succeeded before you know that it did. Async is for work whose failure the user does not need to hear about immediately.
Not monitoring queue depth and consumer lag. These are the two numbers that tell you whether the system is keeping up.
Practice
Orders are published to a queue and a consumer charges the customer. Design this so a redelivery never double-charges, and say exactly where the idempotency lives.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The broker cannot help you here. Something downstream has to recognise a repeat.
Show a strong answer
Accept that redelivery will happen — at-least-once is the guarantee, and building on the hope of exactly-once is how double charges occur.
Where idempotency lives: in the effect, not in the consumer.
- Give each order a stable idempotency key at the moment it is created — the order ID, not something generated by the consumer. A key generated when the message is handled is different on each redelivery and protects nothing.
- Make the charge itself conditional. Send the key to the payment provider, which records it and returns the original result for a repeat rather than charging again. If the provider supports this, most of the problem is solved at the only place that truly knows whether money moved.
- If it does not, record intent before acting. Insert a row keyed on the idempotency key inside a transaction, marked
pending, then charge, then markcharged. A redelivery finds the row: ifcharged, acknowledge and stop; ifpending, the previous attempt died mid-flight and you must reconcile with the provider rather than guess. - Acknowledge only after the effect is durable. Acknowledging first turns at-least-once into at-most-once and loses charges instead of duplicating them.
The residual risk worth naming: the window between charging and recording success. Nothing removes it — you can only make it small and make the reconciliation deterministic. That is why step 3 stores pending rather than nothing.
A queue's depth has been growing steadily for two hours. Give a diagnostic order and, for each cause, the correct response — including when adding consumers is wrong.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Depth growing means arrival rate exceeds completion rate. There are only a few reasons why.
Show a strong answer
Depth grows when arrivals exceed completions, so measure both before doing anything.
- Did arrivals rise? Compare publish rate with yesterday. If a campaign tripled traffic, the system is behaving correctly and under-provisioned. Response: add consumers — this is the one case where it is straightforwardly right.
- Did completion rate fall? If arrivals are flat and depth is growing, consumers slowed. Check consumer latency per message. Response: find what they wait on — usually a downstream dependency — because adding consumers now makes it worse.
- Are messages failing and being redelivered? A poison message that always fails is reprocessed forever, consuming capacity and never leaving. Depth may even look stable while nothing progresses. Response: a dead letter queue with a redelivery limit, so a message that fails N times leaves the main flow. Without one, a single bad message can occupy a fleet indefinitely.
- Is a downstream service the real constraint? If consumers are all blocked on one database at its connection limit, adding consumers is actively harmful: more consumers means more concurrent load on the thing that is already the bottleneck, so throughput falls and latency rises. The queue is doing its job — absorbing the mismatch — and the fix is downstream capacity or fewer, slower consumers.
The general rule: a queue converts an overload into a delay. Growing depth tells you the mismatch exists; it does not tell you which side of it to change, and adding consumers is only correct when the consumers themselves are the constraint.
Key takeaways
- A queue turns slow, fragile synchronous chains into fast responses plus background work — and stops non-essential dependencies determining your availability.
- Beyond decoupling, it provides load levelling, built-in retries and independent scaling of the actual bottleneck.
- Message queues delete on consumption; event logs retain and allow many consumers plus replay. Choose by whether several systems care, and whether you may want history.
- Exactly-once is largely a myth across external side effects. Use at-least-once plus idempotent consumers.
- Order is not guaranteed globally — partition by key when you need it, accepting no parallelism within a key.
- Dead letter queues stop poison messages blocking everything; alert on their depth and support replay.
- Watch queue depth and consumer lag as leading indicators of backpressure.
Next: everything so far has been individual components. The remaining lessons put them together on real products.
Further reading
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (Addison-Wesley, 2003) — the vocabulary of queues, channels and dead letters.
- Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (LinkedIn engineering, 2013) — the queue-versus-log distinction.
- Jay Kreps, Neha Narkhede and Jun Rao, "Kafka: a Distributed Messaging System for Log Processing" (NetDB 2011).
- Pat Helland, "Idempotence Is Not a Medical Condition" (ACM Queue, 2012) — why at-least-once forces idempotent handlers.
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 11 — stream processing and delivery guarantees.