Sending a notification looks like a one-line problem: call a provider's API.
It becomes a system design problem because of everything around that call. You must reach users across three unrelated channels, each with its own provider and failure modes. Those providers go down and rate limit you. Users have preferences, time zones and a limited tolerance for interruption. And a bug that sends one notification twice will send it twice to ten million people.
Requirements worth stating
Functional: send push, email and SMS; respect user preferences; support both immediate and scheduled sends; support templates.
Non-functional:
- High volume with bursts. A product announcement is millions of sends in minutes.
- Reliable but not synchronous. A notification arriving three seconds late is fine; losing it silently is not.
- At-least-once, deduplicated. Users must never receive duplicates, but the system must not drop messages either.
- Third-party dependent. You do not control the delivery providers, and you must assume they fail.
The architecture
Diagram, as text
- Services leads to Notification API
- Notification API leads to Queue
- Queue leads to Workers
- Workers leads to Channel
- Channel leads to Push: APNs/FCM
- Channel leads to Email provider
- Channel leads to SMS provider
- Workers leads to Log + status
Every part of this exists for a reason.
The API is thin. It validates, assigns an id, enqueues, and returns. It never calls a provider inline — that would make your checkout flow depend on an email vendor's availability.
The queue absorbs bursts. This is the load-levelling property from the message queue lesson, and it is the single most important design choice here. Ten million notifications become ten million messages, drained at whatever rate providers allow.
Workers are per-channel. Push, email and SMS have completely different throughput, latency and failure characteristics. Separate worker pools mean a backed-up SMS provider does not delay push notifications, and you scale each independently.
Everything is logged with its delivery state, so you can answer "did this user actually receive it?"
The three channels are not interchangeable
| Push | SMS | ||
|---|---|---|---|
| Cost | Free | Very cheap | Expensive per message |
| Latency | Seconds | Seconds to minutes | Seconds |
| Delivery proof | Weak | Opens, bounces | Carrier receipts |
| Fails when | App uninstalled, token stale | Spam filters | Wrong number, carrier blocks |
Cost asymmetry drives real design decisions. SMS is expensive enough that a bug sending duplicates is a financial incident, not just an annoyance — which is why deduplication matters more here than almost anywhere else.
Push has a subtlety worth knowing: device tokens expire. Users reinstall, switch devices, revoke permission. Providers return "invalid token" responses, and if you ignore them you accumulate dead tokens and your delivery rate quietly rots. Prune on failure.
Fan-out
One event often means many notifications. "Your team commented" may target 500 people; a marketing announcement targets everyone.
Do this in two stages rather than one:
- A fan-out worker reads the event, resolves the recipient list, applies preferences, and enqueues one message per recipient per channel.
- Delivery workers each handle one message.
The reason for splitting is failure granularity. If recipient 400,000 fails, you retry that one message — not the whole batch, which would resend to the 399,999 who already received it.
Deduplication
At-least-once delivery guarantees duplicates will happen: a worker sends the email, crashes before acknowledging, and the message is redelivered.
For notifications this is user-visible and, for SMS, expensive. So make the send idempotent:
async function deliver(msg) {
const first = await redis.set(`sent:${msg.id}`, '1', 'NX', 'EX', 604800);
if (!first) return; // already sent — skip
await provider.send(msg);
await db.markSent(msg.id);
}There is an unavoidable window: if the process dies after the provider call but before recording success, a retry sends twice. You choose which risk to take — record before sending and risk losing a message, or after and risk a duplicate. For SMS, most teams record first and accept rare losses; for push, duplicates are cheaper than misses.
Say this trade-off out loud in an interview. Claiming exactly-once across a third-party API is the wrong answer.
Handling provider failure
Providers fail, and your design must expect it.
Retry with exponential backoff. Immediate retries against a struggling provider make things worse.
Circuit breaker. After repeated failures, stop calling for a while. Without one, workers burn capacity on calls that will fail and the queue backs up behind them.
Fall back to another provider. Mature systems run two email vendors and switch on failure. This requires abstracting the provider behind an interface — the Adapter pattern from the LLD module.
Dead letter queue. Messages that fail repeatedly go aside for inspection rather than blocking the queue or being silently dropped.
Respect provider rate limits. Every provider caps your send rate. Exceeding it gets you throttled or blocked, so workers must limit themselves — the token bucket from the rate limiting lesson, applied outbound.
Not annoying users
A notification system that works perfectly and irritates people is a failed system. These are product requirements with architectural consequences.
Preferences per category and channel. Users want order updates but not marketing. Check before enqueuing, not at send time, so you do not do the work.
Quiet hours. Respect the user's time zone. An SMS at 3am is worse than no SMS.
Rate limit per user. Cap how many notifications any individual receives per hour, regardless of how many events fire. Ten separate "someone liked your post" pushes in a minute is why people disable notifications permanently.
Aggregate. "5 people liked your post" instead of five notifications. This is a batching window in the fan-out stage, and it is one of the highest-value features you can add.
Honour unsubscribes immediately, and note it is a legal requirement for marketing email in most jurisdictions.
Check your understanding
Common mistakes
Sending notifications synchronously. Your availability becomes the provider's availability.
No per-user rate limit. A retry loop somewhere sends a user forty pushes and they disable notifications forever.
Ignoring token invalidation. Dead device tokens accumulate and delivery rates decay without any visible error.
No delivery tracking. When support asks "did the customer get the email?", you need an answer.
Treating all channels alike. SMS costs real money and has stricter regulation; push is free but easily revoked.
Testing against production providers. Use sandbox modes. Sending real messages from a test suite is a memorable incident.
Practice
Design the fan-out for 'a post you follow was updated' where the average account has 200 followers and the largest has 40 million. State how each case is handled and why the strategies differ.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Compute the work for the largest account before choosing one strategy for everyone.
Show a strong answer
Do the arithmetic first. Writing one notification row per follower for the 40-million account means 40 million writes for a single update. At any reasonable write rate that takes minutes, occupies the notification pipeline entirely, and delays everyone else's notifications behind it.
So use two strategies, chosen by follower count.
- Ordinary accounts — fan-out on write. For 200 followers, write 200 notification records immediately. Reads are then trivial: each user's notification list is a simple query on their own rows. This is the right default because reads massively outnumber writes and this makes reads cheap.
- Very large accounts — fan-out on read. Do not write per follower. Record the update once, and when a follower opens their notifications, merge their fan-out-on-write rows with a query for updates from the large accounts they follow. The cost moves to read time, where it is bounded by the number of large accounts one person follows — a small number.
Why the strategies differ: fan-out on write optimises for many cheap reads and assumes writes are small. That assumption breaks exactly when one write becomes 40 million. Fan-out on read has the opposite profile. Neither is correct for the whole distribution, so the design follows the distribution.
The threshold is empirical, not principled — somewhere around where per-follower write cost exceeds the read-merge cost, often tens of thousands of followers. Make it configurable, because the right value changes as the platform grows.
Users complain about notification fatigue. Design the controls that reduce volume without users missing anything important, and say what you would measure.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Not all notifications are worth the same interruption.
Show a strong answer
Start by ranking notifications by what they are worth interrupting for, because every control below depends on that ranking.
- Critical — security alerts, payment failures. Never batched, never suppressed, delivered on the most reliable channel.
- Direct — someone messaged or mentioned this specific user. Delivered promptly, subject to quiet hours.
- Ambient — activity on things they follow. Always eligible for batching.
Then apply four controls:
- Digest the ambient tier. Instead of 40 separate notifications, one summary — "12 updates from people you follow". This is where most of the volume reduction comes from and it costs almost nothing in value.
- Coalesce within a short window. Five likes in two minutes become "5 people liked your post". Delay by a minute or two to allow the grouping.
- Respect quiet hours in the recipient's local timezone. Getting this wrong is the single most common cause of people disabling notifications entirely.
- Per-category preferences, not one global switch. A global switch means a user who dislikes one category turns off all of them, including the critical tier.
What to measure: not open rate. Measure the disable rate per category and time-to-disable after a volume change. Open rate rewards sending more; disable rate is the metric that captures the actual harm, and it is the one that tells you a change made things worse before the users are gone.
Key takeaways
- The API must be thin and asynchronous — never call a provider inside a request, or an email outage becomes an order outage.
- A queue absorbs bursts, turning ten million sends into ten million messages drained at the rate providers allow.
- Use separate worker pools per channel, because push, email and SMS have unrelated throughput and failure characteristics.
- Fan out in two stages so a single failed recipient is retried alone rather than resending to everyone.
- Deduplicate on message id; be honest that a crash between sending and recording leaves an unavoidable window, and choose which risk fits the channel.
- Expect provider failure: backoff, circuit breakers, a fallback provider, a dead letter queue, and self-imposed outbound rate limits.
- Preferences, quiet hours, per-user caps and aggregation are architectural requirements, not product polish.
Further reading
- RFC 8030, Generic Event Delivery Using HTTP Push — the web push protocol.
- Pat Helland, "Idempotence Is Not a Medical Condition" (ACM Queue, 2012) — deduplication keys that survive retries.
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (Addison-Wesley, 2003) — aggregator and resequencer patterns, which digesting relies on.
- Nir Eyal, Indistractable (BenBella, 2019) — the argument for measuring disable rate rather than engagement.
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapter 21 — shedding load when a downstream provider degrades.