Key takeaways
- Fine-tuning teaches a model how to behave — format, tone, task structure. It is a poor way to teach it facts.
- It does not fix hallucination. A model fine-tuned on your documents learns to sound like them, which can make wrong answers more convincing rather than less.
- LoRA and QLoRA changed the economics: instead of updating every weight, you train a small adapter. A job that once needed a cluster now fits on a single GPU.
- You need fewer examples than people expect — often hundreds to a few thousand, if they're consistent. Quality beats volume, and inconsistent labels are worse than fewer labels.
- Try prompting first, then retrieval. Fine-tuning is the third thing to reach for, not the first.
Fine-tuning has a reputation problem. It gets recommended as the answer to "the model doesn't know our stuff," which is precisely the problem it's worst at solving.
It's a genuinely useful technique. But it's useful for a narrower set of things than the pitch suggests, and getting that boundary wrong is expensive — in GPU time, in labelling effort, and occasionally in credibility when the thing ships and behaves oddly.
What fine-tuning changes
You start with a pre-trained model and continue training it on your own examples. The weights shift toward the patterns in your data.
The critical word is patterns. The model is learning the shape of your examples — vocabulary, structure, how a response to this kind of input tends to look. It is not building a lookup table of the facts in your training set.
That's why fine-tuning is strong at:
- Output format. If you need every response as JSON with specific fields, fine-tuning enforces that far more reliably than asking politely in a prompt.
- Domain register. Getting a model to write like a compliance analyst rather than a helpful assistant.
- Narrow classification. Given enough labelled examples, a small fine-tuned model can match a much larger general one on a single task, at a fraction of the inference cost.
- Dropping the prompt overhead. Behaviour baked into weights doesn't need re-explaining in every request, which matters at volume.
The hallucination misconception
The claim you'll hear — and one I've made too casually myself in the past — is that fine-tuning on accurate domain data stops the model inventing things.
It doesn't, and the mechanism is worth being precise about.
Fine-tuning adjusts the probability distribution over what the model says. Train it on a thousand well-written internal policy documents and it gets very good at producing text that looks like a well-written internal policy document. What it hasn't acquired is a reliable index of which specific facts are true.
The practical result is that hallucinations get harder to spot, not rarer. The model now produces confident, correctly-formatted, on-brand answers — including when it's wrong. You've improved the packaging without improving the contents. In a domain where someone downstream acts on the output, that's a worse failure than an obviously clumsy one.
If the actual problem is "the model states things that aren't true," the tool is retrieval — put the source documents in the context at question time so the answer is grounded in something you can point at. Fine-tuning and retrieval solve different problems and pair well; substituting one for the other doesn't work.
What it costs now
The economics changed substantially with parameter-efficient methods, and a lot of received wisdom about cost predates them.
Full fine-tuning updates every weight. For anything modern that means serious multi-GPU infrastructure and a full model copy per task.
LoRA (Low-Rank Adaptation) freezes the base model and trains small adapter matrices injected into its layers. You're updating a tiny fraction of the parameters, so memory drops sharply and the artefact is megabytes rather than gigabytes — meaning you can keep many task-specific adapters against one base model.
QLoRA adds quantisation: hold the frozen base in 4-bit precision, train the adapter on top. This is what makes single-GPU fine-tuning of a sizeable model practical at all.
For most applied work LoRA or QLoRA is the sensible default. Full fine-tuning is for when you're substantially changing what the model is, not tuning how it responds.
How much data
Less than people assume, and the shape matters more than the volume.
For a well-defined behavioural task — a consistent output format, a specific tone — several hundred to a few thousand good examples will typically get you a long way. Consistency is what's being learned, so:
- Consistent beats numerous. A thousand examples following the same convention teach more than ten thousand that disagree with each other. Contradictory labels actively teach the model to be inconsistent.
- Coverage matters more than count. Include the awkward cases. If every training example is a clean, typical input, the model will handle clean typical inputs and flounder on the rest.
- Hold out a real evaluation set before you start, and write down what "better" means. Without that you cannot tell whether the fine-tune helped or just changed the flavour of the output.
That last one is where most projects quietly fail. Fine-tuned output usually feels better because it looks more like what you expected. Feeling better and being better are different, and only one of them survives contact with production.
When not to fine-tune
- The facts change. Anything time-sensitive or frequently updated. Retrain-on-every-change isn't a workable operating model — use retrieval.
- You haven't seriously tried prompting. A well-constructed prompt with a few good examples solves a large share of what people reach for fine-tuning to fix, at approximately zero cost and with an iteration loop measured in seconds.
- You have under a hundred examples. You'll likely overfit and lose general capability without gaining reliable task performance.
- You can't articulate the improvement. If there's no metric, there's no way to know it worked, and you'll be maintaining a model artefact on the strength of a hunch.
- Traceability is a requirement. In regulated settings you often need to show why a system produced an answer. Retrieval gives you a document to point at. A fine-tuned model gives you weights, and no auditor has ever accepted weights.
The order to try things
- Prompt engineering. Cheapest, fastest to iterate, and more capable than its reputation.
- Retrieval. When the model needs access to facts it can't have — yours, or anything recent.
- Fine-tuning. When behaviour is the problem: format, tone, a narrow repeated task, or inference cost at volume.
Most projects that jump to step three end up back at step one having spent a fortnight finding out their prompt was the issue.
Fine-tuning is a good tool. It's just a tool for shaping how a model responds — and knowing that is what stops you spending a month teaching a model to be confidently wrong in your house style.
An earlier version of this post was published on Medium.
