Skip to content

Eventual Consistency Is a Product Decision

Engineers argue consistency via CAP. The answerable question is what a user will forgive seeing, and how to get that from a stakeholder instead of guessing.

By ArchiDepth Team

5 min read

On this page

An engineer asks a product manager whether the feature needs strong consistency. The product manager, who has never seen a quorum read and reasonably assumes that strong is the opposite of weak, says yes. The engineer writes it down, and six weeks of cross-region coordination gets designed into a page that shows how many people liked a photo.

The question was answered honestly. It was just unanswerable in the form it was asked.

The engineering framing is correct and unhelpful

The technical account of this is not wrong. Networks partition. A write acknowledged by a quorum spanning regions pays a round trip that no amount of engineering removes. Synchronous replication converts one machine's bad afternoon into a global outage. Consistency, availability and partition tolerance really do trade against each other, and an engineer who insists on that is being rigorous rather than pedantic.

The problem is that the framing produces a vocabulary nobody outside the team can use. Ask a stakeholder to choose between strong and eventual and you have asked them to pick an adjective. They will pick the reassuring one every time, because the cost of the other is invisible to them and the risk of being the person who approved "weak" is not.

Meanwhile the actual decision — the one that determines the architecture — is a question about human tolerance that the stakeholder is uniquely qualified to answer, and it was never put to them.

What the decision really is

Nobody experiences consistency. They experience a number on a screen and a belief about whether it's true.

A like count that lags by a few seconds costs nothing, because no user has a mental model precise enough to notice, and nothing depends on the value. A bank balance that lags costs trust immediately and permanently, because the user is doing arithmetic with it and the arithmetic will be wrong. Both are just a counter behind a replica. The difference is entirely in what the reader is going to do next.

The interesting cases are between those poles, and they're where the design work is:

  • Seat or inventory counts. Staleness is tolerable right up until it isn't, and the failure is asymmetric: showing fewer seats than exist loses a sale, showing more than exist creates an obligation you can't meet.
  • Permission revocation. Nobody minds if a new permission takes a moment to appear. A revoked permission that takes a moment to disappear is a security incident with a duration.
  • Moderation state. The lag between removing content and the removal taking effect is not a technical detail; it's the whole product promise.
  • A profile edit. Almost no one cares whether other people see the new name promptly. Everyone cares that they see it on the next page load.

That last one is the most common real requirement in any product, and it isn't strong consistency at all — it's read-your-own-writes, which is a session guarantee you can implement without global coordination. Teams routinely build the expensive thing because they never separated "the user must not be confused" from "all observers must agree".

Eliciting the tolerance

You will not get a useful answer by asking about consistency. You get it by asking about consequences, in the stakeholder's own units.

Who sees the stale value? The person who made the change, or somebody else? These have completely different costs, and completely different implementations.

How long is acceptable, in words you'd use with a customer? Not milliseconds. "By the time they refresh." "Within a minute." "Before the end of the working day." If the honest answer is "before the end of the working day", say so in the document, because you have just been handed an enormous amount of design freedom.

What does the user do with the wrong value? Nothing, and they never notice? Click something that fails and needs retrying? Take an irreversible action? Move money? The staleness budget shrinks with each step and it shrinks sharply.

Is the divergence visible? A wrong number the user can check against another screen is much worse than a wrong number they cannot, because the first one destroys confidence in every other number you show.

Can you compensate afterwards? A great many businesses run on apology rather than prevention: an oversold seat gets rebooked, a duplicate charge gets refunded, a race gets resolved by whoever's timestamp is earlier. If a compensating action exists and is acceptable, you have converted a consistency requirement into an operational one, which is far cheaper.

Write the answers next to the latency budget in the same document. "Visible to other users within thirty seconds; visible to the author immediately" is a specification an engineer can build against and a stakeholder can be held to. "Strongly consistent" is neither.

Where the engineers are right

Some invariants are not a matter of opinion and no amount of stakeholder conversation softens them. Uniqueness of an identifier, a balance that must not go negative, an idempotency key that must be honoured once, a double-spend that has no compensating action — those need real coordination, and a product manager saying "a few seconds is fine" doesn't change what the database has to do.

Engineers are also right to resist strong-everywhere as a default. It's not a safe choice; it's a choice that trades availability for a property most of the screen doesn't need, and the outage it eventually causes is also a product decision, just one nobody made deliberately.

The reframing

Consistency isn't a property you select in the data layer and then explain to product. It's a product requirement that happens to be implemented in the data layer — and like every other product requirement, it has an owner, a stated tolerance, and a cost that someone outside engineering should be told about before it's paid.

The next time the question comes up, don't ask which consistency model they want. Ask what they're willing to have a user see, and for how long. You'll get an answer you can build.