You have a new model version, or a new prompt, or a reranker you believe is better. Offline evaluation says it improves things.
Deploying it is riskier than deploying code, for a reason worth stating precisely: you cannot fully test it beforehand. Code either satisfies its tests or does not. A model change alters behaviour across an input space you have only sampled, and the cases you did not sample are exactly where it may have regressed.
So model rollout is built around exposing the change to reality gradually, with a fast way back.
What counts as a deployment
Everything that changes behaviour is a release, and teams routinely under-appreciate this:
- A new model version, or a provider silently updating one behind an unchanged name
- A prompt edit — a one-word change can shift behaviour measurably
- Retrieval configuration: chunking, top-k, reranking, thresholds
- Tool definitions and descriptions, which alter selection
- Guardrail thresholds
- Temperature or other decoding parameters
Version all of it together. A prompt change deployed independently of its evaluation results is an untracked behavioural change, and reconstructing what was live during an incident becomes guesswork.
Treat the whole configuration — model, prompt, retrieval settings, tool schemas — as one versioned artefact.
The rollout ladder
Each rung buys more confidence at more cost.
Offline evaluation
The gate from the evaluation lesson. Run the eval set, compare per-case against the current production configuration, block on regressions.
Necessary and insufficient: your set does not cover everything, and offline scores do not always predict real outcomes.
Shadow deployment
Send production traffic to the new version without serving its output. Log both responses and compare.
This is the strongest tool available here, and it is underused. You see real traffic — including the messy inputs no eval set contains — with zero user risk. Nobody receives the new output.
Diagram, as text
- Request leads to Current version
- Request leads to New version (copy)
- Current version leads to User
- New version leads to Log for comparison
Compare on quality (judge-scored), cost, latency, and disagreement rate — the fraction of requests where the two differ meaningfully. High disagreement is not automatically bad, but it tells you how large the change is and where to look.
The cost is running inference twice. For an expensive model, shadow a sample rather than everything.
Canary
Serve the new version to a small share of real traffic. Now users are exposed, so you need automated rollback on regression.
Ramp gradually — 1%, then 5%, then 25% — and watch error rate, latency, cost per request, quality on sampled traffic, and user signals such as rephrasing or escalation.
Route deterministically by user, not randomly per request. A user who gets the new version on one turn and the old on the next experiences an incoherent assistant, and conversation history may not even be compatible.
Online experiment
For changes where the goal is a business outcome rather than a quality score, split traffic and measure the outcome directly — task completion, escalation to a human, retention, revenue.
This is where the previous lesson's warning bites: a model with better offline metrics can produce worse business results. Only an experiment settles it.
Run long enough for significance, and beware novelty effects in the first days.
Rollback
Because you cannot fully test beforehand, the ability to revert quickly matters more than confidence in the change.
Keep the previous version warm. For a self-hosted model, loading weights takes minutes you do not have during an incident.
Make rollback configuration, not deployment. Flipping a version pointer is seconds; redeploying is not. This is the feature-flag argument from the configuration lesson.
Decide the trigger in advance. Quality below a floor, cost per request above a ceiling, error rate up — automated where possible, because judgement under pressure is poor.
Know what rollback does not undo. A model that took actions during the canary window has already sent emails and issued refunds. For agents, rollback restores behaviour; it does not restore the world.
Fallbacks
Rollout handles your changes. Fallbacks handle the provider's failures.
Provider outage. Route to an alternative provider or a self-hosted model. This requires abstracting the provider behind an interface — the AI gateway lesson covers doing it properly.
Rate limiting. Back off, queue, or fall back to a smaller model.
Timeouts. Decide what the user sees. A degraded answer from a faster model often beats an error; sometimes an honest failure is better.
Total failure. Have a non-AI path. A support assistant that cannot answer should hand off to a human, not present an error page.
Design the degradation explicitly, and say so in interviews — an AI feature with no plan for provider failure is an availability risk sitting on your critical path.
Check your understanding
Common mistakes
Prompt changes deployed untracked. They are behavioural releases; version them with everything else.
Trusting offline scores alone. They do not capture cost, latency or business outcome.
Skipping shadow. The cheapest real-traffic signal available, and commonly skipped.
Random per-request canary routing. Produces incoherent conversations; route by user.
No pre-agreed rollback trigger. Judgement under incident pressure is poor.
Cold standby. Minutes of load time during an outage.
No provider fallback. Their outage is your outage.
Assuming rollback undoes agent actions. It restores behaviour, not effects.
Practice
Your provider announces the model version you pin will be retired in 60 days. Plan the migration.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Sixty days is generous only if you start immediately, because the outcome may be that the new model is worse.
Show a strong answer
Day 1 — establish the baseline. Run the full evaluation set against the current pinned version and record it. Without this, "the new model is worse" is an impression. Also record cost and latency, because those change too and are often the thing that actually blocks a migration.
Week 1 — evaluate the candidate offline. Run the evaluation set against the new version. Expect differences in three places: refusal behaviour, output length and format compliance. Prompts tuned against one model are partly tuned to its quirks, so some regression is normal and is a prompt problem rather than a model problem.
Weeks 2–3 — shadow on live traffic. Serve the old version, run the new one alongside, log both. Sample the disagreements and score them. This finds the failures the evaluation set does not cover, and every disagreement worth arguing about becomes a new case.
Weeks 3–5 — adjust the prompt for the new model, re-running the evaluation set after each change. This is usually where most of the work is, and it is why 60 days is not as long as it sounds.
Weeks 5–7 — canary. 1%, then 10%, then 50%, watching quality signals, refusal rate, cost and latency at each step. Keep the old version routable throughout.
Week 8 — complete, with the old path still available until the retirement date actually passes.
What I would decide in advance: the criteria for proceeding at each step, and what happens if the new model is simply worse. That last case is real, and the options — accept a quality reduction, spend more on a larger model, or move provider — take longer than 60 days to arrange. Discovering it in week 7 is the failure mode this plan exists to avoid.
Design the fallback behaviour for an LLM feature when the provider is degraded or down.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Rank by what the feature can still usefully do without a model.
Show a strong answer
Design a ladder, and choose where the feature sits on it by consequence.
- Retry with backoff and jitter for transient failures, once or twice. Anything more amplifies load on a struggling provider.
- Fail over to a second provider or a second region. This requires prompts that work on both — which is a design constraint to adopt early, not a switch to flip during an incident. Expect quality to differ, and know in which direction.
- Fall back to a smaller or older model that is still available. Degraded quality beats no service for most features.
- Serve from cache. A semantic cache hit that would normally be refused for freshness may be acceptable during an outage — but this must be a deliberate mode, with the answer marked as possibly stale, not a silent widening of the threshold.
- Degrade to a non-model path. Keyword search instead of semantic search; a templated response instead of a generated one; the previous answer instead of a new one.
- Fail honestly. For anything where a wrong answer causes harm — a medical or financial question — refusing is correct. "This feature is temporarily unavailable" is a better outcome than a fallback that is quietly worse and unlabelled.
Two things that decide whether this works:
- The fallback must be exercised regularly, or it is untested code that runs for the first time during an incident. Force it on a small share of traffic continuously.
- Users must be able to tell. A degraded answer presented identically to a normal one destroys trust when it is noticed — and it will be. Mark it.
Key takeaways
- A model change cannot be fully tested in advance, so rollout is about gradual exposure with fast reversal.
- Prompts, retrieval settings, tool schemas and guardrail thresholds are all deployments — version them as one artefact.
- The ladder is offline evaluation → shadow → canary → online experiment, each buying confidence at more cost.
- Shadow deployment is the strongest underused tool: real traffic, zero user risk, and it surfaces cost and latency effects.
- Route canaries deterministically by user, or conversations become incoherent.
- Make rollback configuration rather than deployment, keep the previous version warm, and agree triggers in advance.
- Provider updates are deployments you did not make — pin versions and run evals on a schedule.
- Plan explicit degradation for provider failure, including a non-AI path.
Further reading
- Google SRE, Site Reliability Engineering (O'Reilly, 2016), chapters 17 and 27 — testing reliability and progressive rollout.
- Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022), chapter 9 — shadow deployment, canary and A/B testing for models.
- Ronny Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — fallbacks and graceful degradation.
- Jez Humble and David Farley, Continuous Delivery (Addison-Wesley, 2010) — decoupling deployment from release.