In a monolith, placing an order is one transaction:
BEGIN TRANSACTION;
INSERT INTO orders ...;
INSERT INTO payments ...;
UPDATE inventory ...;
COMMIT; -- all of it, or none of itThe database guarantees atomicity. If the inventory update fails, the order and the payment vanish with it, and you never wrote the code that made that happen.
Split those three tables across three services with three databases and that guarantee is gone. There is no BEGIN TRANSACTION spanning separate machines with separate storage engines. You now own the problem the database used to solve.
What you actually lost
It is worth naming precisely what disappeared, because the replacement only gives some of it back.
Atomicity — all-or-nothing. You can approximate this, and the rest of this lesson is how.
Isolation — nobody sees the half-finished state. You cannot get this back. In a distributed transaction the order exists in the order service before payment has been taken, and anything reading it will see that.
That second loss is the one people forget, and it drives real product decisions: the order needs a PENDING status because pending is now a state users can observe.
Sagas: local transactions plus undo
A saga replaces one distributed transaction with a sequence of local transactions, each committed independently in its own service. If a later step fails, you do not roll back — you run a compensating transaction for each completed step, in reverse order.
Trip booking makes the shape obvious:
| Forward step | Compensating step |
|---|---|
| Book flight | Cancel flight and refund |
| Book hotel | Cancel hotel and refund |
| Charge card | Issue refund |
| Reduce stock | Restore stock |
| Create order | Cancel order |
Book the flight, book the hotel, then find no cars are available: you cancel the hotel, cancel the flight, and tell the user the trip failed.
Choreography: services react to events
There is no coordinator. Each service listens for events and emits its own.
- Order service creates the order as
PENDINGand emitsOrderCreated. - Payment service hears
OrderCreated, charges the card, emitsPaymentSucceededorPaymentFailed. - Order service hears the result and moves the order to
CONFIRMEDorCANCELLED.
Diagram, as text
- Order service leads to Event bus (OrderCreated)
- Event bus leads to Payment service
- Payment service leads to Event bus (PaymentSucceeded)
- Event bus leads to Order service
Nothing is centrally in charge, so there is no coordinator to fail and no coordinator to change when you add a fraud check — the new service just subscribes.
The price is that the workflow exists nowhere in the code. To answer "what happens when an order is placed?" you must read every service and reconstruct the graph in your head. Debugging is tracing an event across five logs, and it is easy to create accidental cycles where two services trigger each other.
Orchestration: one service drives
A dedicated orchestrator calls each service in turn, tracks which steps succeeded, and issues the compensations when a step fails.
Diagram, as text
- Saga orchestrator leads to Order service
- Saga orchestrator leads to Payment service
- Saga orchestrator leads to Inventory service
Now the workflow is a single readable piece of code, the failure handling is in one place, and the orchestrator can persist its progress so a crash mid-saga can be resumed rather than left dangling.
The price is coupling and concentration: the orchestrator knows about every participant, it grows as the process grows, and it is a component that can itself fail. That last point is manageable — persist saga state and run more than one instance — but it must be designed for.
| Choreography | Orchestration | |
|---|---|---|
| Control | Distributed across services | One coordinator |
| Coupling | Low, services know only events | Coordinator knows all participants |
| Visibility of the flow | Scattered, hard to trace | Explicit and readable |
| Adding a participant | Subscribe to an event | Change the orchestrator |
| Failure handling | Each service handles its own | Centralised |
| Suits | Short flows, 2–4 steps | Longer flows with real business logic |
The rule of thumb: choreography for short flows, orchestration once the process has enough steps that nobody can hold it in their head.
Two-phase commit, and why it faded
The older answer keeps atomicity properly. A coordinator asks every participant "can you commit?"; each one locks the rows it needs and answers. If all say yes, the coordinator tells everyone to commit.
It genuinely provides atomicity, and it is still reasonable inside one datacentre, for low-volume internal systems where correctness beats throughput.
It fades at scale because locks are held across the network for the whole protocol. Every participant blocks on the slowest one, throughput collapses, and if the coordinator dies between the two phases, participants sit holding their locks until it recovers — they cannot safely commit or abort on their own. Sagas give up isolation precisely to avoid that blocking.
Making a saga survivable
Four disciplines separate a saga that works from one that leaves data wrong in ways nobody notices for a month.
Compensations must be idempotent. They are triggered by retries and redeliveries, so refunding twice must refund once.
Persist saga state. If the process dies halfway, something must know which steps completed so the compensations can still run.
Set a timeout. A step that never responds must eventually be treated as failed, or the saga hangs forever holding a PENDING order.
Handle failed compensations. A refund can fail too. That case needs an alert and a human, not a swallowed exception — this is the distributed-systems equivalent of a dead letter queue.
Check your understanding
Common mistakes
Assuming a saga gives you ACID. You get atomicity approximately and no isolation at all. Intermediate states are visible and your product must account for them.
Writing the forward path and leaving the undo for later. Every step needs its compensation designed at the same time, or the first failure in production is a manual database repair.
Putting an irreversible step in the middle. Send the confirmation email after everything reversible has committed, not before.
Non-idempotent compensations. Retries are guaranteed. A refund that fires twice is as bad as the double charge you were avoiding.
No saga state and no timeout. A crashed or stalled saga leaves records PENDING forever, and nobody finds out until a customer complains.
Reaching for two-phase commit at scale. It blocks every participant on the slowest, and a coordinator failure leaves locks held until recovery.
Practice
Design a saga for booking a holiday: flight, hotel, car. Specify the compensations, the order of steps, and what you would do about the step that cannot be compensated.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Order the steps so the least reversible one happens last.
Show a strong answer
Order the steps by how reversible they are, least reversible last. That single decision removes most of the difficulty.
- Reserve the car — compensation: cancel the reservation. Usually free and instant.
- Reserve the hotel — compensation: cancel, typically free within a window.
- Book the flight, last — compensation: cancel, often with a fee, sometimes non-refundable.
If the flight fails, cancelling a hotel and a car costs nothing. If the flight were first and the hotel failed, you would be cancelling a flight and paying a penalty for a booking the customer never wanted.
The step that cannot be cleanly compensated is the non-refundable flight. Three honest options:
- Use a two-phase reservation where the provider supports it. Hold the seat without purchasing, complete the rest, then confirm. This converts an irreversible action into a reversible one and is by far the best answer when available.
- Make it the final commit point. Everything before it is reversible, so the flight purchase becomes the moment the booking becomes real. Failures after it are handled by retrying forward, not by compensating backwards.
- Escalate to a human. If it genuinely cannot be undone and the saga fails afterwards, record the state and raise it for manual resolution. This is not a failure of design — some compensations are business decisions, and pretending code can make them silently is worse.
Two things the saga needs regardless: every step idempotent, because steps will be retried; and a persisted saga state so a crashed orchestrator resumes rather than restarting from the beginning.
Compare choreography and orchestration for the booking saga above. Which would you choose and what would change your mind?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask where the sequence is written down, and who can answer 'why is this booking stuck?'
Show a strong answer
Choreography: each service listens for events and reacts. The car service hears BookingStarted and emits CarReserved; the hotel service hears that and proceeds. There is no central component.
- For: no single point of failure, services stay decoupled, and adding a step means adding a listener without touching anything else.
- Against: the sequence exists nowhere. To understand the flow you read five services and infer it. Debugging "why is this booking stuck?" means correlating events across services with no authoritative view of where it stopped. Cyclic dependencies creep in unnoticed.
Orchestration: one saga coordinator calls each service in turn and records progress.
- For: the sequence is explicit, in one place, readable. State is queryable — you can ask exactly which step a booking is on. Compensation logic lives together and is testable. Timeouts and retries are uniform.
- Against: the orchestrator is a component that must be available and becomes a place logic accumulates; it can drift towards knowing too much about each service's internals.
I would choose orchestration here, because a holiday booking is a business process with a defined order, real money and compensations that must be reasoned about together — and because the operational question "where is this booking?" must have an answer. The coupling an orchestrator introduces is worth an explicit, inspectable process.
What would change my mind: if the steps were genuinely independent and unordered (send a receipt, update analytics, warm a cache), choreography is better — those are reactions to an event, not a process, and forcing them through an orchestrator creates coupling for no benefit.
Key takeaways
- Across separate services with separate databases there is no distributed
BEGIN TRANSACTION— the atomicity the database gave you is now your problem. - You lose isolation permanently: intermediate states are visible, which is why
PENDINGbecomes a real product state. - A saga is a chain of local transactions, each with a compensating transaction that undoes it semantically, run in reverse on failure.
- Compensations are not rollbacks. They write offsetting history, some actions cannot be undone at all, so put irreversible steps last.
- Choreography (services react to events) is loosely coupled but scatters the workflow; orchestration (one coordinator) is readable and recoverable but couples to every participant.
- Make compensations idempotent, persist saga state, set timeouts, and alert when a compensation itself fails.
- Two-phase commit does preserve atomicity but holds locks across the network and blocks on coordinator failure — acceptable at low scale, not at high.
Further reading
- Hector Garcia-Molina and Kenneth Salem, "Sagas" (SIGMOD 1987) — the original paper.
- Pat Helland, "Life Beyond Distributed Transactions: An Apostate's Opinion" (CIDR 2007).
- Jim Gray and Leslie Lamport, "Consensus on Transaction Commit" (ACM TODS, 2006) — why 2PC blocks and what fixes it.
- Chris Richardson, Microservices Patterns (Manning, 2018), chapter 4 — sagas, choreography and orchestration.
- Caitie McCaffrey, "Applying the Saga Pattern" (GOTO 2015) — operational experience with long-running sagas.