Two questions about the same payment events. "How much revenue did we take last month?" And "is this transaction, arriving right now, fraudulent?"
The first has a correct answer that will not change, and you can afford to wait for it. The second must be answered before the transaction completes, on incomplete information, and being late is the same as being wrong.
That difference — is the data bounded, and can you afford to wait — is the whole distinction between batch and stream processing.
Batch: bounded input, complete answers
Batch processing runs over a finite dataset that has stopped changing: yesterday's logs, last month's transactions. The job starts, reads everything, produces a result, and exits.
Being bounded is what makes batch so pleasant. The job sees all the data, so aggregates are exact. Failure recovery is "run it again", because the input is still sitting there unchanged. And processing terabytes in one pass amortises overhead beautifully — the cost per record is as low as it goes.
It suits payroll, nightly reports, model training, warehouse loads. Anything where the answer is needed eventually, and needs to be right.
The cost is stated in its own name. Results are as old as the schedule. A nightly job means your worst-case answer is a day out of date, and no amount of hardware fixes that.
Stream: unbounded input, answers that are never final
A stream never ends. Events arrive continuously and are processed as they land, in milliseconds rather than hours.
Because there is no end, there is no moment at which a stream computation is complete. Every result is a current best answer that the next event may change. That is not a defect — for fraud detection, live dashboards, alerting or vehicle tracking, an approximate answer now beats an exact one after the moment has passed.
It costs you almost everything batch made easy. State must be held across events and survive restarts. Failure recovery means resuming from a checkpoint rather than starting over. And the job runs continuously, so you pay for the infrastructure whether or not anything is happening.
| Batch | Stream | |
|---|---|---|
| Input | Bounded, at rest | Unbounded, in motion |
| Latency | Minutes to hours | Milliseconds to seconds |
| Completeness | Sees all the data | Sees what has arrived so far |
| Recovery | Rerun the job | Resume from checkpoint |
| State | Lives inside one run | Long-lived, must be durable |
| Cost profile | Pay per run | Pay continuously |
| Typical tools | Spark, MapReduce | Kafka Streams, Flink, Kinesis |
Windows: giving an unbounded stream a boundary
"Count the events" is meaningless on an infinite stream. Every stream aggregation therefore happens inside a window — a bounded slice you can actually close and emit.
Tumbling windows are fixed and non-overlapping: 0–5 minutes, 5–10, 10–15. Each event belongs to exactly one. Use them for periodic reporting, where you want one clean figure per interval.
Sliding windows overlap: a 15-minute window advancing every minute. Each event belongs to several windows at once, and you get a continuously refreshed view of a fixed lookback. Use them for rolling metrics — "requests in the last five minutes", updated every ten seconds.
Session windows have no fixed size. They group events by activity and close after a gap of inactivity. Use them when the natural unit is a burst of user behaviour rather than a clock interval.
Tumbling |--A--|--B--|--C--| fixed, no overlap
Sliding |--A--|
|--B--| fixed size, advances
|--C--|
Session |-AAA-| |-BB-| bounded by idle gapsThe hard part: event time versus processing time
Every event has two timestamps: when it happened, and when your processor saw it. On a healthy network they are milliseconds apart. On a phone that went through a tunnel they are twenty minutes apart.
Windowing by processing time is easy and quietly wrong: the tunnel events land in the wrong window, so your 10:00 figure is understated and your 10:20 figure is inflated. Windowing by event time is correct, and immediately raises the question that defines stream processing.
There is no configuration that makes this go away. Completeness and latency are genuinely opposed on an unbounded stream, and you are choosing where to sit between them.
Lambda architecture, and what replaced it
The classic answer to "I need both fast and exact" is the lambda architecture: run the same logic twice.
Diagram, as text
- Events leads to Speed layer stream, approximate
- Events leads to Batch layer nightly, exact
- Speed layer stream, approximate leads to Serving layer
- Batch layer nightly, exact leads to Serving layer
- Serving layer leads to Queries
The speed layer gives an immediate approximation — "about 100 likes". The batch layer recomputes everything from scratch overnight and produces the authoritative figure — "exactly 102". The serving layer merges them, using batch results for the period they cover and the stream for everything since.
It works, and its cost is severe: the same business logic exists twice, in two frameworks, and must be kept identical. Every change is two changes, and every discrepancy is a bug hunt across two systems.
The kappa architecture is the response. Keep only the stream pipeline, and treat reprocessing as replaying the retained log from the beginning — which a durable event log makes possible. One codebase, one set of semantics. It is why so much of this ecosystem is built on Kafka: the log's retention is the batch layer.
Worked example: trending hashtags
Twitter wants the top hashtags of the last fifteen minutes, refreshed continuously.
Batch cannot do this. An hourly job detects the trend an hour after it started, by which time it is over — for this problem, late really is the same as wrong.
The pipeline is a sliding window with a serving cache in front:
tweets → Kafka topic → stream processor
├─ extract hashtags
├─ count per tag in a 15-minute sliding window
└─ emit tags above threshold
→ Redis sorted set → APIThe choices worth defending are the ones a window forces you to make. Sliding, not tumbling, because a tumbling window would reset the count on the quarter-hour and a trend building at 10:14 would vanish at 10:15. Event time, not processing time, so a tweet delayed by a slow client still counts towards the minute it was written. And Redis in front, because the read path serves every user on the site and must not touch the stream processor's state.
Check your understanding
Common mistakes
Windowing by processing time without meaning to. It is usually the default and it is quietly wrong the moment the network is slow.
No policy for late events. They are silently dropped, so your numbers are slightly low in a way nobody notices until someone reconciles against batch.
Streaming something nobody needs quickly. A monthly report does not benefit from a pipeline that runs continuously and costs continuously.
Assuming stream results are final. They are the best answer so far. Anything downstream that treats them as authoritative will disagree with the batch figures.
Maintaining lambda's two codebases by hand. If the logic diverges, the two layers give different answers and you will not know which is right.
Forgetting that streaming state is a database. Windows, counters and joins hold state that must be checkpointed, sized and recovered, or a restart silently loses it.
Practice
You must report hourly revenue with figures the finance team can rely on, from a stream where 0.1% of events arrive more than an hour late. Design the windowing, watermark and correction strategy.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Finance will accept a late correction. They will not accept a number that quietly changes.
Show a strong answer
Window on event time, hourly, tumbling. Processing time is not defensible in a financial figure.
Watermark: allow a modest delay, then close. Something like 15 minutes past the hour covers the overwhelming majority of stragglers without holding the result all day. Emit the hour's figure then.
Handle the 0.1% with explicit corrections, not silent updates. This is the important part:
- Keep the window's state for a longer grace period — say 24 hours — so late events can still be applied.
- When one arrives, do not mutate the published figure in place. Emit a correction record that says which hour changed, by how much, and why.
- Downstream consumers apply corrections; the finance report shows the original and the adjustment.
Why not just update the number? Because a report that silently changes is worse than one that is late. Someone screenshots the 2pm figure, and next week it reads differently with no record of what happened — which destroys trust in the whole pipeline. An explicit correction is auditable; a mutation is not.
Also required: mark each published figure as provisional until the grace period expires and final afterwards, so a consumer knows which they are holding.
Explain why the lambda architecture existed, and what changed to make it largely unnecessary.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask what a stream engine could not originally do that a batch job could.
Show a strong answer
Why it existed. Early stream processors were fast but approximate: they could not reprocess history, they lost state on failure, and they gave no exactly-once guarantee. Batch systems were the opposite — exact, replayable, and hours behind. Neither alone was acceptable for something like a usage dashboard that must be both current and correct.
So lambda ran both: a speed layer giving approximate answers within seconds, and a batch layer recomputing the same figures nightly from raw data and overwriting them. The serving layer merged the two.
The cost was the reason it was abandoned. Every metric had to be implemented twice, in two systems with different languages and semantics, and kept in agreement forever. When the two disagreed — which they did — nobody could tell which was wrong. The maintenance burden was roughly double for a result that was correct only after the nightly run.
What changed: stream engines gained the properties that batch had.
- Durable, replayable logs meant history could be reprocessed by rewinding a consumer, so the batch layer's unique advantage disappeared.
- Checkpointed state and exactly-once processing made streaming results exact rather than approximate.
- Event-time windowing with watermarks made out-of-order data a solved problem rather than a reason to recompute later.
The replacement — running one streaming pipeline, and reprocessing by replaying the log through a second instance when logic changes — gives the same correctness with one implementation. The insight is that batch is just streaming over a bounded range, so a system that handles the unbounded case handles both.
Key takeaways
- The choice is set by whether the input is bounded and whether you can afford to wait, not by which technology is more modern.
- Batch sees all the data, so its answers are complete and recovery is a rerun; its results are as stale as its schedule.
- Stream answers in milliseconds on partial data, so every result is a current best answer that a later event may change.
- Unbounded streams need windows: tumbling for periodic reports, sliding for rolling metrics, session for bursts of activity.
- Event time versus processing time is the real difficulty; watermarks decide when a window closes, trading completeness against latency, and late events need an explicit policy.
- Lambda runs both layers to get speed and exactness, at the cost of the same logic maintained twice; kappa keeps one stream pipeline and reprocesses by replaying the log.
- Use batch for reports, warehousing and training; use stream for fraud, alerting and live metrics.
Further reading
- Tyler Akidau et al., "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing" (VLDB 2015) — event time, windows and watermarks.
- Tyler Akidau, "Streaming 101" and "Streaming 102" (O'Reilly, 2015) — the accessible version of the same material.
- Jay Kreps, "Questioning the Lambda Architecture" (O'Reilly Radar, 2014) — the argument that retired it.
- Paris Carbone et al., "Lightweight Asynchronous Snapshots for Distributed Dataflows" (arXiv, 2015) — how checkpointing makes streaming exact.
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), chapter 11.