Your classification pipeline works. It uses a frontier model, handles millions of requests a month, and the bill is now a line item somebody asks about.
The task itself is narrow: read a support ticket, assign a category and a priority. You are paying frontier-model prices for something far simpler than what that model can do.
Distillation is the answer: use the large model to teach a small one your specific task, then serve the small one.
The idea
The large model becomes a teacher. You run it over your inputs, collect its outputs, and fine-tune a small model to reproduce them.
Diagram, as text
- Your inputs leads to Large model teacher
- Large model teacher leads to Input-output pairs
- Input-output pairs leads to Fine-tune small model
- Fine-tune small model leads to Small model serves production
The result is a model that is dramatically cheaper and faster, and that performs comparably on that one task. It is far worse at everything else, which is exactly the trade — you did not need everything else.
Why the economics are so favourable
Three effects compound.
Price per token falls by a large multiple. Capable small models cost a fraction of frontier models.
Latency falls. Fewer parameters means less memory traffic per decode step — the memory-bound behaviour from the serving lesson — so generation is quicker.
Throughput rises. A smaller model leaves more room for KV cache, so more requests fit per GPU.
These multiply rather than add, which is why distillation is frequently the largest single cost reduction available to a mature AI product.
Where the training data comes from
This is the reason distillation is so much more practical than ordinary fine-tuning: you do not have to label anything by hand.
The teacher generates the labels. Run it over your real inputs, keep the outputs, and you have a dataset that reflects your genuine traffic distribution rather than a curated approximation.
Three refinements that matter:
Use real production inputs. Synthetic inputs miss the messy, ambiguous cases that dominate real traffic.
Filter the teacher's output. The teacher is not perfect. Where you have ground truth — a resolved ticket, a human correction — drop or fix the examples where it was wrong. Training on the teacher's errors teaches them.
Cover the tail. Naive sampling produces mostly easy cases. Deliberately oversample hard, rare and ambiguous inputs, or the student will be excellent on the common path and useless at the edges.
Volume needed varies with task difficulty. Narrow classification and extraction often need surprisingly few well-chosen examples; open-ended generation needs considerably more.
When it works, and when it does not
Works well:
- Classification, routing and tagging
- Structured extraction from documents
- A consistent transformation — rewriting, summarising to a fixed style
- Anything narrow, repetitive and high-volume
Works poorly:
- Open-ended reasoning across varied problems
- Tasks needing broad world knowledge
- Anything where the input distribution shifts constantly
- Low-volume work where the effort will never pay back
The test is narrowness plus volume. A well-defined task at high volume distils beautifully. A varied assistant handling anything a user types does not.
The hybrid that usually wins
Rather than replacing the large model, put the small one in front and escalate.
request -> small model
├── confident -> return
└── uncertain -> large model -> returnMost traffic is handled cheaply; hard cases get the expensive model. You keep frontier quality on the requests that need it and pay small-model prices for the rest.
This is the model routing pattern from the serving lesson, with a distilled model as the cheap tier — and it is usually a better first step than a full replacement, because it degrades gracefully. If the student is wrong on a case, escalation catches it.
Getting the escalation signal right is the real work. Options include the model's own confidence, a schema escape value ("unknown" from the structured outputs lesson), an explicit classifier, or simple rules on input characteristics.
Operating a distilled model
You now own a model. Versioning, deployment, rollback, monitoring — the deployment lesson covers doing this safely.
Drift is the main risk. The student learned one input distribution. As real inputs shift — new products, new phrasing, new customer segments — accuracy decays silently. Nothing errors; quality just erodes.
Two defences: monitor the escalation rate, which rises before accuracy visibly falls and is a good leading indicator, and sample production traffic for evaluation on a schedule rather than trusting the original test set forever.
Keep the pipeline reproducible. Data generation, filtering, training and evaluation should be repeatable, because you will run it again — on a better teacher, a better base, or a shifted distribution.
Check your understanding
Common mistakes
Evaluating against the teacher. Measures imitation, not correctness.
Training on unfiltered teacher output. Bakes in its errors.
Sampling only easy cases. Excellent on the common path, useless on the tail.
Full replacement instead of routing. Escalation degrades gracefully; replacement does not.
No drift monitoring. Accuracy erodes silently as inputs change.
Distilling a broad task. Narrowness is the precondition.
Distilling before volume justifies it. The effort must pay back.
Practice
Your classification step costs a large fraction of your bill using a frontier model. Plan a distillation project, including how you would know it is safe to switch.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
You need labelled data, a comparison, and a plan for what the small model cannot do.
Show a strong answer
Collect the data from your own traffic. Log inputs and the frontier model's outputs for a period covering seasonal and edge cases — a few weeks rather than a day. Tens of thousands of examples is typically ample for classification, and they are already labelled by construction.
Clean it. The teacher is not always right, and training on its mistakes teaches them. Sample and hand-check, and for classification specifically, look at the class distribution: rare classes will be under-represented in natural traffic and the student will simply never predict them unless you oversample.
Train and compare on a held-out set drawn from the same traffic, measuring per class rather than overall accuracy. An aggregate that looks fine can hide a rare class collapsing to zero, and rare classes are frequently the important ones.
How I would know it is safe to switch:
- Shadow it first. Run the student alongside the teacher in production, serving the teacher's answer and logging both. This gives real disagreement rates on live traffic without any risk, and it is the step that turns a benchmark into evidence.
- Examine the disagreements, not just the rate. If they cluster on one class or one input type, that is a routing rule rather than a blocker.
- Set the switch criterion in advance — for example, agreement above 98% overall with no class below 95% — so the decision is not made by whoever is keenest.
Plan for what the student cannot do: route low-confidence inputs to the teacher. This hybrid is usually the design that wins — the small model handles the overwhelming majority cheaply, and the expensive model handles the tail. The saving comes from volume, not from the small model being universally sufficient.
Then keep measuring. Traffic drifts, and a student trained on last quarter's distribution degrades quietly. Re-run the comparison periodically and re-distil when it moves.
A colleague proposes distilling your entire agent — reasoning, tool selection and all — into a small model. Assess this honestly.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Ask which parts of an agent's behaviour are a narrow distribution and which are not.
Show a strong answer
Separate the agent into its parts, because they distil very differently.
- Tool selection — a good candidate. Given a request and a set of tools, choose one and fill its arguments. This is close to classification with structured output, the distribution is narrow, and there is abundant training data from logged trajectories. A small model can do this well and it is often called on every loop iteration, so the saving is large.
- Argument extraction — also good, for the same reasons, and it benefits from constrained decoding regardless of model size.
- Multi-step reasoning and recovery from unexpected errors — a poor candidate. This is precisely where capability is broad rather than narrow: handling a situation not in the training distribution is the definition of what a small model loses. An agent that plans well on the cases it saw and flails on novel ones is worse than one that is uniformly competent, because the failure is unpredictable.
- Deciding when to stop or escalate — poor, and consequential. Getting this wrong means either burning budget or abandoning solvable tasks.
What I would propose instead: distil the hot path, keep the frontier model for judgement.
Route the high-volume, narrow decisions — tool selection, extraction, routine classification — to the small model, and call the large one for planning and for anything the small model flags as uncertain. That captures most of the cost saving, because the narrow decisions are the frequent ones.
What I would ask the colleague for before agreeing to anything: the breakdown of tokens by call type. If 80% of spend is planning rather than tool selection, the proposal saves little and risks the part that matters. The distribution of the bill decides which part is worth distilling, and it is usually not the part people assume.
Key takeaways
- Distillation trains a small model on a large model's outputs, and the cost, latency and throughput gains compound.
- A distilled model can beat a frontier model on its narrow task, because narrow and trained beats broad and prompted.
- The teacher generates the labels, so no manual labelling is required — but filter its output, since errors are learned.
- It suits narrow, high-volume tasks: classification, extraction, fixed transformations. It does not suit open-ended reasoning.
- Prefer the hybrid: small model first, escalate to the large one when uncertain. It degrades gracefully.
- Evaluate against independent ground truth, never against the teacher — agreement measures imitation.
- Monitor drift, using escalation rate as a leading indicator, and keep the pipeline reproducible for retraining.
Further reading
- Geoffrey Hinton, Oriol Vinyals and Jeff Dean, "Distilling the Knowledge in a Neural Network" (NIPS Deep Learning Workshop, 2015).
- Victor Sanh et al., "DistilBERT, a distilled version of BERT" (arXiv, 2019).
- Cheng-Yu Hsieh et al., "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes" (ACL 2023).
- Jason Wei et al., "Emergent Abilities of Large Language Models" (TMLR, 2022) — which capabilities do not survive scaling down.
- Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022) — shadow deployment and drift.