"The model does not know our product, so we should fine-tune it on our documentation."
This is the most common expensive mistake in applied AI. It sounds obviously right, it is usually wrong, and the reason why clarifies what each technique actually does.
The ladder
Four rungs, in order of cost and commitment. Climb only when the rung below has demonstrably failed.
| Rung | Changes | Cost | Iteration speed |
|---|---|---|---|
| Prompting | Instructions and examples | Nearly nothing | Minutes |
| RAG | Facts available at inference | Low | Hours |
| Fine-tuning | Behaviour, style, format | Moderate to high | Days |
| Distillation | A smaller model's capability | High | Weeks |
Teams skip rungs because higher ones feel more serious. In practice the top of the ladder is slower to iterate on, harder to debug, and locked to a base model that will be superseded.
The distinction that decides it
Diagram, as text
- Problem with model output leads to Is it what it says or how it says it?
- Is it what it says or how it says it? leads to Retrieval: reindex, cite, no retraining (missing or stale facts)
- Is it what it says or how it says it? leads to Does prompting get close? (wrong form, tone, structure)
- Does prompting get close? leads to Keep prompting: cheaper, instantly changeable (yes)
- Does prompting get close? leads to Fine-tune on a few hundred ideal examples (no, or prompt is huge)
- Is it what it says or how it says it? leads to Distil into a smaller model (too slow or costly)
RAG changes what the model knows. Fine-tuning changes how the model behaves.
Knowledge is facts: your policies, your inventory, yesterday's incident. Behaviour is style, tone, format, and the shape of reasoning applied to a task.
Almost every fine-tuning decision becomes clear once you classify the problem this way.
Two reasons people fine-tune that are wrong
"To teach it our knowledge"
The instinct is that training on documentation embeds the facts. Four problems:
Knowledge changes. A policy updated tomorrow means retraining. RAG means editing a document.
You cannot cite. Retrieved passages support attribution; weights do not. Users cannot verify, and you cannot detect ungrounded claims.
It absorbs facts unreliably. Fine-tuning teaches patterns strongly and specific facts weakly. Ask about a detail seen twice in training and you often get a fluent invention.
Retrieval is cheaper and better at exactly this. It was designed for it.
Use RAG for knowledge. This is the single most useful rule in the lesson.
"To make it output our format"
Also wrong, and now unnecessary. Format is a decoding constraint, not learned behaviour — the structured outputs lesson covers making invalid output impossible to generate.
Fine-tuning to enforce JSON shape is paying training cost for something a schema gives you with certainty.
When fine-tuning is right
Two genuine cases.
Behaviour prompting cannot hold. A specific tone, a domain style, a consistent reasoning pattern. Symptoms: your system prompt has grown to two pages, still drifts over long conversations, and adding instructions now breaks other behaviour. That is prompting exhausted, and it is a legitimate trigger.
Cost and latency, via distillation. You have a working system on a large model. Fine-tune a small one on its outputs and serve the same task far more cheaply. This is often the strongest business case — covered in its own lesson.
A third, narrower case: niche domains whose vocabulary and conventions are thinly represented in general training — specialist medical, legal or scientific text.
Diagnose before choosing
Look at what is actually failing:
| Symptom | Cause | Fix |
|---|---|---|
| Invents facts about your domain | No access to the facts | RAG |
| Right facts, wrong tone | Behaviour | Prompting, then fine-tuning |
| Right facts, inconsistent format | Decoding | Structured outputs |
| Correct but too slow or costly | Model size | Routing, then distillation |
| Cannot follow multi-step instructions | Task complexity | Better decomposition, or a stronger model |
| Ignores retrieved context | Prompt or retrieval quality | Grounding instructions, reranking |
Only two rows point at fine-tuning, and one of them says try prompting first.
What fine-tuning actually costs
Beyond compute, which is often the smallest part.
Data is the real cost. You need consistent, high-quality examples of the behaviour you want — typically hundreds for narrow tasks. Curating them is the work, and quality dominates quantity: a few hundred clean examples beat thousands of noisy ones.
You now own a model. It must be versioned, evaluated, deployed and rolled back. Every base model upgrade means deciding whether to retrain — and base models improve quickly enough that yesterday's fine-tune can be beaten by today's general model with a good prompt.
Iteration slows dramatically. A prompt change is a minute. A fine-tune is a training run, an evaluation cycle and a deploy.
Evaluation becomes mandatory. Without a solid eval set you cannot tell whether the fine-tune improved anything or quietly degraded capabilities it was not trained on.
A sensible sequence
- Prompt well. Clear instructions, a few examples, structured outputs. Most projects stop here.
- Add retrieval when it needs your facts. Most of the rest stop here.
- Build an evaluation set. Non-negotiable before any training — without it you cannot measure improvement.
- Exhaust prompting. Try examples, decomposition, a stronger model. Cheaper than training.
- Fine-tune for behaviour, measured against the eval set.
- Distil for cost once quality is proven.
Note that step 3 comes before any training. Teams that fine-tune first end up unable to answer whether it helped.
Check your understanding
Common mistakes
Fine-tuning for knowledge. The defining error of the field.
Fine-tuning for format. Constrained decoding is cheaper and gives certainty.
Fine-tuning before an eval set exists. You cannot tell whether it worked.
Assuming it removes the need for retrieval. They are complementary.
Underestimating data curation. The examples are the project.
Ignoring base model improvements. A general model plus a good prompt often overtakes an older fine-tune.
Practice
A support assistant gives correct answers in an inconsistent voice, sometimes too technical, and occasionally cites internal ticket numbers customers cannot see. Diagnose which problems fine-tuning would fix.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Separate what the model says from how it says it, and check what is in the context.
Show a strong answer
Three distinct problems, and only one is a fine-tuning candidate.
- Inconsistent voice and register — a genuine fine-tuning case. This is behaviour, not knowledge. It can be attempted with prompting and few-shot examples first, and if that gets close but drifts over long conversations or costs a very long prompt, fine-tuning on a few hundred well-written examples is the right tool. It is also cheaper at serving time, since the style no longer needs to be re-explained in every request.
- Too technical for the audience — probably not fine-tuning. This is usually a missing input rather than a behaviour: the model does not know who it is talking to. If the assistant were told "this customer is on the basic plan and has not used the API", it could adjust. Fine-tuning to be uniformly simpler would then degrade answers for technical users. Fix the input before changing the weights.
- Leaking internal ticket numbers — definitely not fine-tuning. This is a data problem: internal references are in the retrieved context and nothing stops them being repeated. Fine-tuning would make it less likely, not impossible, and a probabilistic fix for a data-exposure problem is not a fix. Strip internal identifiers from documents at index time, and add an output filter that refuses responses containing them.
The order I would work in: filter the leak today, because it is a live exposure. Add audience context to the prompt, and measure. Only then consider fine-tuning for voice, with an evaluation set that includes both audiences so the improvement can be shown rather than assumed.
Your team has decided fine-tuning is justified for output style. Plan the work honestly — data, evaluation, deployment and what happens six months later.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The training run is the smallest part of the cost.
Show a strong answer
Data — where most of the effort goes.
- Several hundred to a few thousand examples of input and ideal output. Quality dominates quantity: 300 carefully written examples beat 3,000 scraped from historical tickets, because the historical answers include all the inconsistency you are trying to remove.
- Cover the distribution deliberately, including the awkward cases — refusals, ambiguous questions, angry customers — or the model will be well-styled on easy inputs and unchanged on hard ones.
- This is weeks of human work, and it is the part that gets underestimated.
Evaluation — before training, not after.
- A held-out set the model never sees, scored on the thing you are trying to change. If "consistent voice" cannot be measured, you cannot tell whether it worked.
- Also measure what you are not trying to change — factual accuracy, refusal behaviour, instruction-following. Fine-tuning frequently degrades capabilities nobody was watching, and that regression is the main risk.
Deployment.
- Serve alongside the base model behind a flag, roll out by percentage, compare on live metrics.
- Keep the base model path available permanently as a fallback.
Six months later — the part that decides whether this was wise:
- The base model will have been superseded. Your fine-tune is anchored to an older checkpoint, and adopting the newer, better base model means redoing the work. Budget for it, and keep the training data in version control so a re-run is a day rather than a project.
- The style guide will have changed, so the data needs revision.
- Someone must own it. A fine-tune with no owner becomes an unexplained artefact nobody dares to change.
The honest summary: the training run is an afternoon. The dataset, the evaluation and the recurring re-training are the actual commitment, and they are what should be weighed against a longer prompt.
Key takeaways
- The ladder is prompt → RAG → fine-tune → distil, and you climb only when the rung below has failed.
- RAG changes what the model knows; fine-tuning changes how it behaves. Classify the problem and the answer usually follows.
- Do not fine-tune for knowledge — facts change, cannot be cited, and are learned unreliably.
- Do not fine-tune for format — constrained decoding guarantees it.
- Fine-tune for behaviour prompting cannot hold, for niche domain conventions, or to distil a working system into a cheaper model.
- Fine-tuning and RAG are complementary, not alternatives.
- The real costs are data curation, owning a model, slower iteration and mandatory evaluation — build the eval set first.
Further reading
- Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020).
- Long Ouyang et al., "Training language models to follow instructions with human feedback" (NeurIPS 2022).
- Chunting Zhou et al., "LIMA: Less Is More for Alignment" (NeurIPS 2023) — why a small, high-quality dataset outperforms a large noisy one.
- Angels Balaguer et al., "RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture" (arXiv, 2024).
- Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022) — the recurring cost of owning a trained artefact.