You Probably Do Not Need Kafka

A design document lands in review. The feature is welcome emails: a user signs up, and some time later they receive a message. The diagram has a broker cluster, a schema registry, a topic naming convention, a consumer group, and an appendix on partition counts. The review thread is forty comments long and every one of them is about the partitioning strategy.

Nobody has asked how many signups there are per day. If someone had, the answer would have made the whole appendix unnecessary.

The three things that are not the same

There are really three tools here, and they get discussed as if they were one.

A table with a status column is a work queue you already own. You insert a row in the same transaction as the business change, a worker selects unclaimed rows, marks them in progress, does the work, marks them done. It is boring, it is transactional with your actual data, and it fails in ways your team already knows how to debug.

A queue is a to-do list that gets burned as it's consumed. A message is delivered, acknowledged, and gone. Multiple workers compete for messages; adding workers adds throughput. Retries and dead-letter handling are built in because that's the entire job.

A log is an ordered, retained sequence of records that consumers read at their own position. Nothing is consumed away. Each consumer group has its own cursor, and the same record can be read by five different systems, or by the same system again next week.

The third one is a different category of thing, and it is the only one people put in the architecture diagram to look serious.

What the log actually costs

The cost of a distributed log is not the cluster. It's the set of decisions it forces on you permanently, and the ones it quietly declines to make for you.

Partition count is close to irreversible. The mapping from key to partition depends on how many partitions there are, so adding partitions later reshuffles which key lands where — and ordering guarantees are per partition, which means the ordering you promised your users can break the day you scale out.

Ordering itself is narrower than people assume. A log gives you order within a partition, never across the topic. If your ordering requirement is "events for one account must be applied in sequence" then you need account to be the partition key, and now one enormous account is a hot partition you cannot split without breaking the guarantee.

Consumer groups rebalance. A deploy, a slow consumer, a network blip: partitions get reassigned, in-flight work is reprocessed, and every handler you wrote must be idempotent whether or not you thought about it. Exactly-once is a property of a carefully constructed pipeline, not a checkbox, and the moment your consumer writes to an external system that doesn't participate in the protocol, you are back to at-least-once and idempotency keys.

A poison message stops a partition. There's no built-in retry-with-backoff-then-park behaviour of the sort a queue gives you for free; you build it, and until you do, one malformed record blocks everything behind it.

Then there's the ongoing surface: retention and disk sizing, lag monitoring per group per partition, schema evolution and compatibility rules, broker upgrades, and a new class of on-call incident that only one or two people on the team understand.

None of that is an argument that logs are bad. It's an argument that they're a commitment, and commitments should buy something.

The properties worth paying for

There are four, and they're real.

Replay. You can rebuild derived state by reading history again from an offset. When your search index is corrupt, your materialised view has a bug, or a new cache needs warming, you re-read rather than reconstruct. A queue cannot do this at all — the messages are gone. This is the single strongest reason to choose a log, and it's the one least often cited.

Multiple independent consumers. One producer, several unrelated systems reading the same stream at their own pace, none of them coordinating. Analytics, search indexing, notifications and fraud scoring all reading the same order events. With a queue you'd fan out to one queue per consumer and the producer would have to know about all of them.

Ordered partitions. Per-key ordering at volume, with parallelism across keys. If you're maintaining state machines per entity and applying out-of-order updates would corrupt them, this is exactly the guarantee you want.

Retention as a feature. The log is the record. A consumer that didn't exist last month can start from the beginning and catch up. This is what makes a log an integration point between teams rather than a pipe between two services.

The test

Answer three questions honestly, before anyone opens a broker configuration.

Can you name the second consumer? Not a hypothetical future one — a system that exists, or is on a roadmap someone has committed to. One consumer is a queue.

Will you ever need to read a message twice? If the answer is "we'd just re-derive it from the database", you don't need retention.

Does ordering matter per entity, at a volume where a single ordered consumer isn't enough? If a single worker keeps up, ordering is free.

Three noes and a table with a status column will serve you for years — and it participates in your transactions, which no broker does. Two or three yeses and you should stop apologising for the cluster; you have the workload logs were designed for, and the operational cost is worth it.

The mistake isn't choosing a log. It's choosing one for the emails.