A feature toggle is a conditional that decides at runtime whether a code path is active. It is the simplest pattern in this module and the one that most reliably becomes a maintenance problem, because every toggle is a permanent branch until someone deletes it.
Its purpose is to separate deployment from release. Code ships to production inactive; it is turned on later, for some users, and turned off in seconds if it misbehaves. That separation is what makes trunk-based development and continuous deployment possible, and it is what makes a bad release a configuration change rather than a rollback.
Four kinds, with different lifetimes
Treating all toggles the same is the root of most toggle problems. They differ in how long they live and who changes them.
Diagram, as text
- Why does this toggle exist? leads to Release toggle days-weeks · engineers · DELETE (hide unfinished work)
- Why does this toggle exist? leads to Experiment toggle weeks · product · DELETE (compare variants)
- Why does this toggle exist? leads to Operational toggle long-lived · operators · KEEP (shed load in an incident)
- Why does this toggle exist? leads to Permission toggle permanent · product · this is a FEATURE (differ by plan or customer)
Release toggles hide work in progress so incomplete code can ship. They should live days or weeks and must be deleted once the feature is fully on.
Experiment toggles split traffic to compare variants. They live as long as the experiment and are then deleted along with the losing branch.
Operational toggles let operators disable an expensive feature under load or turn off a misbehaving integration. These are legitimately long-lived and are part of the system's controls.
Permission toggles vary behaviour by plan, tenant or role. These are not really toggles at all — they are a product feature, and they belong in the domain with the entitlement logic, not in a flag system.
The most valuable thing you can do with toggles is classify them on creation, because the classification determines whether it has an expiry date.
The cost, which compounds
Each toggle doubles the paths through the code, at least in principle. Ten independent toggles is 1,024 possible configurations, and you test perhaps three of them. Combinations nobody tried are reachable in production.
Toggle logic scatters. A check in the handler, another in the service, another in the client. Removing the toggle then means finding all of them, and the one that is missed is the one that matters.
Stale toggles are dangerous, not merely untidy. A flag that has been on for a year still has an off branch — unexercised, untested and reachable if someone flips it or if the flag system fails open in the wrong direction.
Testing gets harder. Which combination is the test running against? A suite that always runs with defaults tests one path and gives confidence about all of them.
Managing the lifecycle
The single most effective practice is an expiry date on every release and experiment toggle, set when it is created, with a build warning or failure when it passes. Without a forcing function, toggles are never removed — removing one is nobody's priority, delivers no visible value, and carries a small risk.
Supporting practices that work:
- Cap the number of active toggles, so adding one means removing one. Crude, and effective.
- An owner per toggle, named. Unowned toggles are never deleted.
- Delete the branch with the toggle. Removing the flag while leaving both code paths achieves nothing.
- A regular review of toggles older than their expiry, with the default action being removal rather than extension.
Progressive delivery
Toggles enable a family of techniques for limiting the blast radius of a change.
Canary release. Send a small percentage of traffic to the new version, watch error rates and latency, and increase gradually. The essential part is the automated rollback criterion — a canary nobody is watching is just a slow deploy, and the metric must be decided before the release, not interpreted after it.
Dark launch. Run the new code path without using its result: call the new service, discard the response, compare it with the old one. This is run-both-and-compare, applied to a release rather than a migration, and it validates against real production traffic before anything depends on it.
Ring deployment. Roll out in waves — internal users, then a beta group, then everyone — so problems are found by people who can report them well.
Blue–green. Two full environments, traffic switched between them. Simple and fast to roll back; it costs double the infrastructure and it does not limit blast radius the way a canary does, because the switch is all-or-nothing.
Check your understanding
Common mistakes
No expiry date. Toggles are never removed without a forcing function.
All toggles treated the same. Release and operational toggles have completely different lifetimes.
Permission logic in the flag system. Entitlements belong in the domain where they can be tested.
No documented default when the flag service is unreachable. The correct default differs per toggle.
The flag service as a hard dependency. An unreachable flag service must not take down the application.
Deleting the flag but keeping both branches. Nothing has been simplified.
Toggle checks scattered across layers. Removal misses one, and it is the one that matters.
A canary with no automated rollback criterion. It detects problems that nobody acts on.
Practice
A codebase has 200 feature flags, the oldest four years old, and nobody knows which are safe to remove. Design a plan to get it under control.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Do not start by removing flags. Start by finding out which ones are actually load-bearing.
Show a strong answer
Do not start by removing anything. With 200 flags of unknown status, the first removal that breaks something ends the initiative — the team will conclude the flags are too risky to touch and the situation becomes permanent. Start by making the state visible.
Step one: instrument evaluation. Log every flag evaluation with its result, for a month. This produces the facts nobody has:
- Flags never evaluated are dead code paths whose conditionals are unreachable. Safest to remove, and there will be many.
- Flags always returning the same value are constants. Removing them means deleting the unused branch — mechanical, and the largest group.
- Flags with varying results are live and need a decision from someone.
A month rather than a week, so anything on a monthly cycle — billing, reporting — is captured.
Step two: assign owners by archaeology. For each flag, find who created it from version control. That person or their team owns the decision. Unowned flags need an owner assigned by whoever runs the area, because unowned flags are never deleted — this is the step that determines whether the cleanup finishes.
Step three: classify each survivor as release, experiment, operational or permission. This immediately identifies flags that should not be in the system at all — the permission ones belong in entitlement logic — and it tells you which have any business being long-lived.
Step four: remove in waves, easiest first, one flag per change so anything that breaks is attributable and revertible. Never-evaluated first, then always-same-value, then the ones with owners who confirm they are done.
Step five: stop the growth, and this should happen in parallel with step one rather than at the end. Every new flag requires a classification, an owner and an expiry date. A build warning once a flag passes its expiry. Without this, the cleanup races flag creation and never converges — the same lesson as every other migration in this module.
How I would track it: flag count, on a dashboard, weekly. One number that everyone can see going down. Also the count past expiry, which will be zero at first and is the leading indicator of a relapse.
What I would expect: of 200 flags, perhaps 120 are constants or dead, 50 are permission logic misfiled, and 30 are genuinely live. The problem will look far smaller after the first month of data than it does now, which is itself a good reason to start with measurement rather than with removal.
Design the rollout for a rewritten payment processing path — the most business-critical code in the system. Cover the release mechanism, validation and the abort criteria.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Money makes both directions dangerous: shipping a bug and being unable to fall back.
Show a strong answer
The constraint that shapes everything: a payment bug is not recoverable by rolling back. A duplicate charge or a lost payment has already happened and needs refunds, reconciliation and customer contact. So the design must weight detection before impact far more heavily than speed of rollback.
Phase one: dark launch, and this is where most of the confidence comes from. Run the new path alongside the old for real traffic, using the old result and discarding the new one. Compare every outcome and log disagreements.
- Real traffic is the only adequate test for payment code, because the interesting cases — declines, partial captures, currency edge cases, retries, the specific behaviours of each provider — are not in any test suite anyone would write.
- Run for a full business cycle, including a month-end and whatever the peak is. Volume alone is not coverage.
- Every disagreement gets a named explanation, not a rate. "0.02% differ" is not a result; each one is either a bug in the new path or an undocumented behaviour of the old one, and both need a decision from someone who owns payments.
- Critically: the new path must not call the payment provider during this phase. Comparing decisions is safe; making duplicate charges is not. This means dark-launching the logic against a provider sandbox or against recorded responses, and being explicit that the provider interaction itself is validated separately.
Phase two: canary by cohort, not by percentage. Start with internal employees, then a small set of low-value transactions, then progressively higher values.
- Value-based progression is the right axis for payments, because a 0.1% traffic sample includes a large transaction eventually, and the cost of being wrong scales with the amount rather than with the count.
- Hold at each stage for long enough to observe settlement, not just the immediate response. Many payment problems only appear at capture, at settlement, or in the reconciliation the next day — so a stage that lasts an hour proves very little.
Abort criteria, agreed in writing before starting and automated:
- Any increase in decline rate beyond normal variance — this is the earliest signal and the one most likely to move first.
- Any duplicate charge at all. Not a threshold; one is an abort.
- Any reconciliation mismatch between what the system believes and what the provider reports.
- p99 latency beyond the checkout budget, since a slow payment path abandons carts even when it is correct.
The kill switch: a single operational toggle reverting all traffic to the old path, tested before the rollout begins rather than assumed to work. And the old path must remain deployed and exercised — a fallback that has not run for a month is not a fallback.
The deletion step: the old path and the toggle are removed only after a full reconciliation cycle at 100%, with an owner and a date. Leaving them means maintaining two payment implementations indefinitely, which is its own risk.
Key takeaways
- A toggle separates deployment from release: ship inactive, turn on later, turn off in seconds.
- Classify on creation. Release and experiment toggles must be deleted; operational ones are legitimate controls; permission toggles belong in the domain as entitlements.
- Every toggle is a permanent branch until deleted, and a stale one has an unexercised path that is still reachable.
- Set an expiry date with a build warning. Without a forcing function, toggles are never removed.
- Document the default when the flag service is unreachable, per toggle, and never let it be a hard dependency.
- Delete the branch, not just the flag.
- A canary without an automated rollback criterion is a slow deploy.
- Dark launch — run both and compare on real traffic — is the strongest validation available for high-risk changes.
Further reading
- Pete Hodgson, "Feature Toggles (aka Feature Flags)" (martinfowler.com, 2017) — the four categories and their lifetimes.
- Jez Humble and David Farley, Continuous Delivery (Addison-Wesley, 2010).
- Nicole Forsgren, Jez Humble and Gene Kim, Accelerate (IT Revolution, 2018) — deployment practices and their measured effects.
- Betsy Beyer and others, Site Reliability Engineering (O'Reilly, 2016) — canarying and error budgets.
- Michael Nygard, Release It!, second edition (Pragmatic Bookshelf, 2018) — operational controls and kill switches.