A domain event is a record that something meaningful happened: OrderPlaced, PaymentCaptured, SubscriptionCancelled. Publishing them lets other parts of the system react without the originator knowing who they are — Observer, applied to the domain.
The pattern is easy. The reason this topic exists is the dual-write problem: saving to a database and publishing to a message broker are two separate writes, and there is no way to make them atomic. Every system that ignores this loses events, and the loss is silent.
The dual write
await db.orders.insert(order); // write 1: succeeds
await broker.publish('order.placed', event); // write 2: process crashes hereThe order exists and nobody was told. Inventory is not reserved, no confirmation is sent, the warehouse has no record — and the database looks perfectly healthy.
Reversing the order is no better:
await broker.publish('order.placed', event); // write 1: succeeds
await db.orders.insert(order); // write 2: failsNow subscribers react to an order that does not exist. Downstream systems have state derived from a fact that never became true.
There is no ordering of two writes to two systems that is safe, and wrapping them in a try/catch does not help: the crash can occur between the commit and the catch. This is not an edge case — a process restart during a deploy is enough.
Diagram, as text
- Business operation leads to ONE transaction
- ONE transaction leads to Write the order
- ONE transaction leads to Write the event row to the outbox table
- ONE transaction leads to Commit
- Commit leads to Both exist, or neither does (success)
- Commit leads to Neither exists (failure)
- Both exist, or neither does leads to Relay process reads unpublished outbox rows
- Relay process reads unpublished outbox rows leads to Publish to broker
- Publish to broker leads to Mark row published
The transactional outbox
Write the event to a table in the same transaction as the state change. Now there is one write, and atomicity is the database's problem — which it is good at.
BEGIN;
INSERT INTO orders (...) VALUES (...);
INSERT INTO outbox (id, type, payload, created_at) VALUES (...);
COMMIT;A separate relay reads unpublished rows and publishes them, marking each as sent. If the relay crashes after publishing and before marking, the event is published twice — so delivery is at-least-once, and consumers must be idempotent. That is the trade: you have exchanged "may be lost" for "may be duplicated", and duplication is the one you can defend against.
Two ways to run the relay:
- Poll the table for unpublished rows. Simple, works everywhere, adds latency equal to the poll interval and load equal to the poll rate.
- Tail the database's replication log. Lower latency, no polling load, and no application code in the publish path — at the cost of infrastructure and a tighter coupling to the database. This is change data capture, covered in its own topic.
Operationally, the outbox needs three things that are routinely forgotten: a retention policy so the table does not grow forever, an index supporting the "unpublished, oldest first" query, and an alert on the age of the oldest unpublished row. That last one is the single most valuable signal — it catches a stopped relay, which otherwise looks like silence.
Event notification versus event-carried state
Two shapes, and choosing wrongly causes either chatter or staleness.
Event notification carries an identifier: "order 42 was placed". Consumers call back for details. Small messages, always-current data, and a synchronous dependency on the producer at exactly the moment it is busy — plus the risk described under Observer of reading a later state than the event described.
Event-carried state transfer carries the data: the order's fields at the moment of the event. Consumers need nothing else, the producer is not called back, and the payload is a snapshot that is correct for that event forever. The costs are larger messages and a payload that becomes a contract — adding a field is easy, changing or removing one is a breaking change for every consumer.
Default to carrying state, precisely because it removes the callback and the staleness. Use notification when payloads would be genuinely large or contain data most consumers must not see.
Designing the event itself
Name it as a past-tense fact. OrderPlaced, not PlaceOrder and not SendConfirmation. An event states what happened; a command states what to do. Once the name reads as an instruction, subscribers have become the producer's to-do list.
Version it from the start. Events outlive deploys and are frequently stored. A schema version in the payload costs nothing now and is the difference between a smooth change and a coordinated one later.
Include a stable event identifier. Consumers need it to deduplicate under at-least-once delivery, and without one they must invent a key from the payload — which breaks the moment the payload changes.
Carry a correlation identifier from the originating request, or tracing a cascade of reactions across services means reading unlinked logs.
Do not include what you would not publish to a stranger. Events flow to consumers you do not control, are frequently retained, and are hard to redact after the fact.
Check your understanding
Common mistakes
Writing to the database and publishing separately. Silent event loss on every crash between them.
Publishing inside the transaction. Subscribers react to uncommitted or rolled-back state.
No idempotency in consumers. At-least-once delivery makes duplicates certain.
No alert on the oldest unpublished outbox row. A stopped relay looks exactly like quiet.
No retention policy for the outbox table. It grows forever and eventually degrades the transaction it is part of.
Command-shaped event names. Subscribers become the producer's to-do list, restoring the coupling.
No schema version. Events outlive deploys and are frequently stored.
Sensitive data in payloads. Events reach consumers you do not control and are hard to redact.
Practice
A service publishes to a message broker immediately after saving to its database. An incident review finds that 0.3% of orders in the last month have no corresponding downstream records. Explain the cause and design the fix, including how you would recover the missing events.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Three parts: why it happens, how to stop it, and what to do about the orders already affected.
Show a strong answer
The cause is the dual write, and 0.3% is exactly the right order of magnitude for it. The service commits the order, then publishes. Anything that ends the process between those two points — a deploy, an out-of-memory kill, a node eviction, a broker timeout that exhausted retries — leaves an order with no event. It is invisible because the database is consistent and the application logs show a successful order.
A detail worth confirming before designing anything: whether the failures cluster around deploy times. If they do, that is conclusive and it also tells you the rate is a function of deploy frequency, not of load — which changes how urgent the fix looks to whoever is prioritising it.
The fix: a transactional outbox.
- Write the event row in the same transaction as the order. One write, atomic by construction.
- A relay publishes unpublished rows and marks them sent. Start with polling; it is simple, has no infrastructure dependency, and a one-second interval is well inside what downstream consumers need. Change data capture is the better long-term answer and is not worth blocking this fix on.
- Accept at-least-once and make consumers idempotent. The relay can publish and die before marking, so duplicates will happen. Each event carries a stable identifier; each consumer records handled identifiers before acknowledging.
- Index the outbox on (published, created_at) so the relay's query stays fast as the table grows, and delete or archive published rows on a schedule.
Recovering the events already missing — the part that is easy to skip and is the actual incident:
- Identify them by comparing orders against downstream records over the affected period. This is a reconciliation query, and it is worth keeping permanently rather than writing once.
- Reconstruct the events from the current order state. This is imperfect and I would say so explicitly: an order that has changed since it was placed will produce an event describing its state now, not then. For most consumers that is acceptable; for anything that computes a delta from the event, it is not, and those consumers need handling individually.
- Republish in small batches, with monitoring, since a month of backfilled events arriving at once can overwhelm consumers that were sized for live traffic.
What I would add regardless of the fix: an alert on the age of the oldest unpublished outbox row. It is one query, and it converts "the relay stopped" from a silent failure into a page. And the reconciliation check should stay in place afterwards — it is the control that would have caught this in week one rather than at a monthly review.
Design the event schema and contract for `OrderPlaced` in a system with five consuming services, and explain how you would evolve it when the order model changes.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The payload is an interface with five consumers you do not control the release schedule of.
Show a strong answer
Treat the payload as a published interface, because that is what it is. Five consumers deploy on their own schedules, so any change must work with old and new consumers running simultaneously — the same constraint as an API, and it is routinely underestimated because events feel internal.
The envelope, separate from the payload:
- Event id — stable, generated once at production, used by consumers to deduplicate under at-least-once delivery.
- Type and schema version, so a consumer can dispatch and can reject what it does not understand rather than misreading it.
- Occurred-at timestamp, distinct from published-at. Consumers frequently need to know when the thing happened, not when it was relayed, and with an outbox those differ.
- Correlation and causation identifiers, so a cascade of reactions across five services can be traced from the originating request.
The payload: event-carried state transfer.
- Include the order's fields as at the moment of placement — id, customer reference, line items, totals, currency, shipping address. Consumers need no callback and cannot read a later state.
- Include only what consumers legitimately need. Payment card details do not go in an event that reaches five services and is retained.
- Use identifiers for entities that consumers own — customer id rather than the full customer record — since duplicating another service's model into the payload makes it a contract too.
Evolution rules, which are the substance of the question:
- Adding an optional field is safe and needs no version bump. Consumers ignore what they do not know.
- Removing or renaming a field is breaking. The path is: add the new field, publish both for a period long enough that every consumer has migrated, verify with consumer-side metrics rather than by asking, then remove the old one. The verification step is the one people skip and it is the one that prevents the outage.
- Changing a field's meaning is the dangerous change, worse than removing it, because nothing fails — consumers keep reading it and quietly do the wrong thing. Never change meaning in place; add a new field with a new name.
- Bump the schema version for anything breaking, and publish both versions during migration if the change cannot be made additive.
How I would keep this honest rather than aspirational:
- A schema registry or a checked-in schema with a compatibility test in CI, so a breaking change fails the build rather than failing in production. This is the control that makes the rules above real.
- Consumer-driven contract tests, so the producer learns which fields are actually depended upon. Frequently a field everyone assumed was load-bearing has no consumers, and one nobody thought about has three.
- Never reuse a field name for a different purpose, even years later. Stored events from the old meaning still exist, and a replay will misread them.
Key takeaways
- A domain event is a past-tense fact; publishing one lets others react without the producer knowing them.
- The dual-write problem has no safe ordering: state-then-publish loses events, publish-then-state announces things that never happened.
- The transactional outbox writes the event in the same transaction as the state, then relays it — one atomic write instead of two.
- It trades loss for duplication, so consumers must be idempotent and events need a stable identifier.
- Operationally the outbox needs retention, an index, and an alert on the oldest unpublished row — a stopped relay looks like silence.
- Default to event-carried state transfer: no callback to the producer, and a snapshot that stays correct for that event.
- Publish after commit, never inside the transaction.
- Treat the payload as a published interface: version it, evolve additively, and never change a field's meaning in place.
Further reading
- Martin Fowler, "What do you mean by 'Event-Driven'?" (2017) — event notification, event-carried state transfer and event sourcing distinguished.
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (Addison-Wesley, 2003) — guaranteed delivery and message construction.
- Vaughn Vernon, Implementing Domain-Driven Design (Addison-Wesley, 2013) — domain events and the outbox.
- Chris Richardson, Microservices Patterns (Manning, 2018) — transactional outbox and polling publisher.
- Pat Helland, "Life Beyond Distributed Transactions" (CIDR, 2007).
- Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — logs, idempotence and exactly-once semantics.