A business process that spans several services cannot be a transaction. High-level design covers what that costs and how a saga recovers it — local transactions plus compensations, choreography against orchestration, and the disciplines that make one survivable.
This topic is about the orchestrator itself. A process manager is an object, and treating it as one — with identity, state, a lifecycle and a test strategy — is what separates a workflow you can reason about from a distributed god object.
It is a State machine that receives messages
The structure is two patterns you already have, arranged in a way that has one surprising consequence.
Diagram, as text
- OrderPlaced leads to OrderFulfilment process manager (id = orderId)
- PaymentCaptured leads to OrderFulfilment process manager (id = orderId)
- StockReserved leads to OrderFulfilment process manager (id = orderId)
- Timeout fired leads to OrderFulfilment process manager (id = orderId)
- OrderFulfilment process manager (id = orderId) leads to State + event decide the next step
- State + event decide the next step leads to ReserveStock
- State + event decide the next step leads to DispatchOrder
- State + event decide the next step leads to RefundPayment (compensation)
- OrderFulfilment process manager (id = orderId) leads to Saga state (persisted after every decision)
The consequence: its method is handle(message), not run().
That is the whole structural difference from a sequence of calls, and everything else follows from it. A function that calls payment, waits, calls inventory, waits, and dispatches holds its progress in the call stack — and a call stack does not survive a deploy, a crash or a node eviction. A process manager holds its progress in persisted state, so the process can be interrupted between any two steps and resumed by whichever instance receives the next message.
A saga is not a long function. It is a state machine that happens to take days.
The rule that keeps it maintainable
It decides what happens next. It does not decide anything else.
Whether a customer is eligible for express dispatch, how a refund is calculated, whether stock may be oversold — those belong to the services that own them. The process manager's entire job is sequencing: given where we are and what just arrived, which command goes out next.
Break this and it becomes the distributed form of the god object described under Mediator. Every feature edits it, it needs every service's data to make its decisions, and it can no longer be tested without constructing the world. The Mediator topic's warning applies exactly: it coordinates, it does not compute.
Correlation: how a reply finds its instance
A running process manager is one instance among thousands, and a reply arriving from the payment service has to reach the right one.
Every message carries a correlation identifier, and it is the process manager's key. For order fulfilment that is the order id. The infrastructure loads the instance by that key, hands it the message, and persists whatever changed.
Two consequences worth designing for rather than discovering:
- Every participant must echo the correlation id back. A service that replies without it has produced a message nobody can route. This is a contract requirement on participants, not an implementation detail of the coordinator.
- The key must be chosen so that one instance owns one process. Keying on customer id rather than order id would put every order that customer ever places into a single instance — the same granularity mistake as one actor for all orders, or a shard key with no spread.
Timeouts are messages, not timers
A step that never answers must eventually be treated as failed, or the process hangs forever holding a pending order. HLD says that; the structural question is how.
The process manager schedules a message addressed to itself. "In thirty minutes, deliver PaymentTimedOut to instance order-123." When it arrives, it is handled like any other message: if payment has since succeeded, the state says so and the timeout is ignored; if not, compensations begin.
An in-process timer is the wrong shape and fails for the reason everything else here does — it dies with the process. A scheduled message survives, is delivered to whichever instance is alive, and is testable by simply delivering it.
Ignoring a timeout that has been overtaken by events is normal, not an error case. It is the same at-least-once tolerance every message handler needs.
Compensation is a first-class command
A compensation is a business action with a name — RefundPayment, ReleaseStock — not an undo() on a step object. The Command topic makes the general argument; here it has three specific consequences:
- It can fail. A refund can be declined. That path needs an alert and a human, and the process manager needs a state to sit in while it waits — which means "compensation failed" is a state in the machine, not an exception.
- It must be idempotent, because it is triggered by retries and redeliveries.
- It is not always the inverse. Releasing stock after a customer has already been told their order shipped is not a reversal — it is a different business event with its own consequences. Some steps have no compensation at all, which is a design constraint on step order: put the irreversible ones last, so there is less to unwind.
Testing one
The payoff for handle(message) over run(): a process manager is a pure function of its history, so it can be tested without a broker, a database or a clock.
The shape is given → when → expect:
given([OrderPlaced(order123), PaymentCaptured(order123)])
.when(StockReservationFailed(order123))
.expect([RefundPayment(order123), NotifyCustomer(order123, 'out_of_stock')]);Feed it prior messages to reach a state, deliver one more, and assert on the commands it emitted — not on what any service did. That last part is what keeps the test honest: it verifies the coordination, which is the only thing this object is responsible for.
Cases worth writing that people skip:
- A duplicate message. Delivering
PaymentCapturedtwice must emit the next command once. - A timeout that lost the race.
PaymentTimedOutarriving afterPaymentCapturedmust emit nothing. - A reply for a step already compensated. Late messages arrive; the state must absorb them.
Check your understanding
Common mistakes
Writing it as a sequential function. Progress lives in the call stack and does not survive a deploy.
Business logic in the coordinator. It becomes the distributed god object: every feature edits it, nothing can test it alone.
Querying participants to decide. The information should arrive in messages; reaching for it is how the rules migrate in.
An in-process timer for step timeouts. It dies with the process, and the saga hangs holding a pending record.
Keying instances too coarsely. Correlating on customer rather than order puts every order into one instance.
Treating compensation as undo(). It is a named business action that can itself fail, so "compensation failed" is a state, not an exception.
Irreversible steps early in the sequence. No compensation machinery can unwind a dispatched parcel; order steps by reversibility.
Testing through the broker. Slow, flaky, and it verifies the transport rather than the coordination.
Practice
Design the order-fulfilment process manager for: reserve stock, capture payment, dispatch, notify. State the step order and why, the state machine, what happens when payment fails after reservation, and what happens when dispatch fails after payment.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Order the steps before designing anything else — it determines which failures you must handle at all.
Show a strong answer
Step order first, because it decides how much compensation machinery is needed at all: reserve stock → capture payment → dispatch → notify.
- Reserve before capture. A reservation is a soft hold that expires on its own; releasing it is cheap and invisible to the customer. Capturing money first and then discovering there is no stock means a refund, a support contact and a worse experience for the same outcome.
- Dispatch last among the irreversible steps. A parcel that has left cannot be unsent. Everything reversible happens before it, so a failure at any earlier point unwinds cleanly.
- Notify after dispatch, and treat it as optional — a failed notification must not fail the order, because the order genuinely happened.
States: AwaitingStock → AwaitingPayment → AwaitingDispatch → Completed, plus the failure branch Compensating → Cancelled, and CompensationFailed as a terminal state that alerts a human.
CompensationFailed being a state rather than a thrown exception is the part worth arguing for: a refund can be declined, and the process manager needs somewhere to sit while a person deals with it. An exception here just loses the workflow.
Payment fails after reservation — the easy case, and the reason for the ordering:
- Emit
ReleaseStock. The customer has been charged nothing. - Move to
Cancelled, notify with a reason they can act on ("payment declined" is actionable; "something went wrong" is not). - If
ReleaseStockitself fails, go toCompensationFailedand alert. Stock held against a cancelled order is a real cost and someone must resolve it — silently swallowing this is how phantom out-of-stock builds up over months.
Dispatch fails after payment — the genuinely hard case:
- Retry first, several times, over a longer window. Dispatch failures are frequently transient — a warehouse system down for ten minutes — and refunding a customer whose order could have shipped an hour later is a worse outcome than waiting.
- Only then compensate:
RefundPayment,ReleaseStock, notify. - The order of compensations matters. Refund before releasing stock: if the release succeeds and the refund fails, the customer has neither goods nor money and the stock has already gone to someone else. Doing the money first means the worst case is a released order with held stock, which is recoverable.
What I would add regardless: an alert on any instance sitting in a non-terminal state beyond a threshold. Every failure mode here ends with a workflow that is stuck rather than wrong, and stuck workflows are silent.
A team has a saga implemented as an async function that awaits each service in turn, with try/catch running compensations. It works in testing and loses orders in production. Explain what is happening and describe the migration.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask what happens to that function's local variables during a deploy.
Show a strong answer
What is happening: the workflow lives in the call stack, and the call stack does not survive anything.
- A deploy mid-saga kills it. The pod is terminated while awaiting the payment service. The
tryblock never completes, thecatchnever runs, and the compensations that would have released stock never fire. The order sitsPENDINGforever, and nobody is told, because from the process's point of view nothing failed — it simply ceased to exist. - The same applies to a crash, an out-of-memory kill or a node eviction. Testing never reproduces it because nobody deploys mid-test.
- A long await makes it likelier. If payment can take thirty seconds and the team deploys several times a day, a steady trickle of orders is lost — small enough to look like noise, persistent enough to matter.
Two smaller faults that usually travel with it: retries re-run steps that are not idempotent, and there is no timeout, so a step that never answers hangs the function until something kills it — producing the same lost workflow.
The migration, in an order that delivers safety early:
- First, make it visible. Before changing any structure, alert on orders in a non-terminal state past a threshold. That converts silent loss into a known number, tells the team whether this is one order a week or fifty, and is worth having permanently regardless of the redesign.
- Persist progress, even inside the existing shape. Write a saga-state record after each step. It does not fix resumption on its own, but it means a recovery job can find the stranded workflows and finish or compensate them — which stops the bleeding within days rather than at the end of a rewrite.
- Then invert the shape. Turn each
awaitinto: emit a command, and return. The reply arrives as a message that loads the state by correlation id and decides the next step. This is the real change, and it can be done one step at a time, from the last step backwards, so each increment is small and the earlier steps keep working. - Add timeouts as scheduled messages once the machinery exists, since there is finally something durable to deliver them to.
- Make every step and compensation idempotent, with the correlation id as the deduplication key.
What I would not do: keep the sequential shape and wrap it in a retry at the top. Re-running a saga from the beginning re-executes steps that already succeeded, which for payment capture means charging twice — replacing lost orders with duplicate charges, which is worse.
Key takeaways
- A process manager is a State machine fed by messages, not a long function —
handle(message), neverrun(). - Progress lives in persisted state, which is what lets a workflow survive a crash or a deploy and resume on any instance.
- It sequences; it does not decide. Business rules in the coordinator make it the distributed god object.
- A correlation identifier is its key. Every participant must echo it, and the key must give one instance per process.
- Timeouts are scheduled messages to itself, because an in-process timer dies with the process. A timeout overtaken by events is ignored, not an error.
- Compensation is a named business action that can itself fail — so "compensation failed" is a state with an alert, not an exception.
- Order steps by reversibility, irreversible ones last; that decision does more than any compensation machinery.
- Test it as given(history) → when(message) → expect(commands), with no broker and no clock.
Further reading
- Hector Garcia-Molina and Kenneth Salem, "Sagas" (ACM SIGMOD, 1987) — the original paper.
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (Addison-Wesley, 2003) — the process manager, correlation identifiers and message routing.
- Chris Richardson, Microservices Patterns (Manning, 2018) — saga orchestration and compensating transactions.
- Vaughn Vernon, Implementing Domain-Driven Design (Addison-Wesley, 2013) — long-running processes and their state.
- Pat Helland, "Life Beyond Distributed Transactions" (CIDR, 2007).
- Bernd Rücker, Practical Process Automation (O'Reilly, 2021) — workflow engines and when they beat hand-rolled coordinators.