Key takeaways
- A base LLM is frozen at its training cutoff and knows nothing private. Those are two different limitations, and they need different fixes.
- RAG — retrieval-augmented generation — fetches relevant documents at question time and puts them in the prompt. It solves staleness and private knowledge. It does not make the model reason better.
- Retrieval quality is the whole game. A RAG system that retrieves the wrong three paragraphs produces a confident, well-written wrong answer.
- If your answer changes when the underlying data changes, you want retrieval. If it changes when the task changes, you want fine-tuning — or, more often than people admit, a better prompt.
- In regulated work the deciding factor usually isn't accuracy at all. It's whether you can show an auditor which source produced the answer.
Ask a base model who your highest-risk customer is and it will give you a fluent, structured, completely invented answer. Not because the model is bad — because you asked it something it has no way to know.
That failure is worth understanding properly, because the fix depends on why it happened, and there are two very different reasons that look identical from the outside.
Two limitations that get confused
A base LLM — GPT-4, Claude, Gemini — is a set of frozen weights. Everything it knows was fixed when training stopped. That produces two separate gaps:
It doesn't know anything recent. Ask about a regulation published after the cutoff and it either tells you so or guesses.
It doesn't know anything private. Your transaction monitoring rules, your case history, your internal policy documents were never in its training data and never will be.
These get lumped together as "the model doesn't know things," but they're distinct. The first is about time. The second is about access. Retrieval happens to fix both, which is why it gets recommended so freely — but knowing which one you have tells you how much retrieval infrastructure you actually need.
What RAG actually does
Retrieval-augmented generation is less exotic than the name suggests. Three steps:
- Index your documents — split them into chunks, embed each chunk as a vector, store it.
- Retrieve at question time — embed the question, find the closest chunks.
- Generate — paste those chunks into the prompt and ask the model to answer using them.
That's it. The model isn't modified at all. You're doing research on its behalf and handing it the findings, then asking it to read and summarise.
This is worth internalising, because it sets the ceiling on what RAG can do. It gives the model material. It does not make the model smarter, and it does not make the model careful.
Where RAG quietly fails
The failure people expect is "it couldn't find the answer." The one that actually causes problems is subtler: retrieval returns something plausible but wrong, and the model writes it up beautifully.
The output has correct grammar, a confident tone, and a citation pointing at a real document. It's just answering a different question than the one you asked. That's worse than a blank refusal, because a refusal is obviously a failure and this isn't.
A few specific ways it happens:
- Chunking splits the answer. A policy states its condition in one paragraph and its exception in the next. Retrieval returns the first. The answer is now confidently missing the exception.
- Semantic similarity isn't relevance. "What is our threshold for reporting?" retrieves everything that talks about thresholds and reporting — including the deprecated 2019 version.
- The model fills gaps anyway. Given partial context, most models smooth over what's missing rather than say the retrieved material was insufficient.
None of these are fixed by a better model. They're fixed by better retrieval — chunking strategy, metadata filters, reranking, and being deliberate about what "we don't have this" should look like.
So when do you fine-tune instead?
Fine-tuning changes the weights. It's the right tool when you want to change how the model behaves rather than what it knows:
- A consistent output format you can parse reliably
- A domain register — the model should write like a compliance analyst, not a chatbot
- A narrow classification task where you have thousands of labelled examples and want a smaller, cheaper model to do it
Notice none of those are about facts. Fine-tuning on a corpus of documents does not reliably let the model recall those documents — it teaches it to sound like them. That distinction gets missed constantly, and it's how you end up with a model that has learned the house style of your policy library and hallucinates in it with total conviction.
Rough rule: if the right answer changes when your data changes, you need retrieval. If it changes when the task changes, you might need fine-tuning. A good share of the time you need a better prompt and haven't seriously tried that yet.
The two paired together
They aren't alternatives, and the strongest systems use both — fine-tuning to fix the shape of the output, retrieval to supply the facts.
A realistic pattern for a compliance assistant: retrieval pulls the relevant policy sections and the customer's case history, while a fine-tuned model formats the response as a structured assessment with mandatory fields, because the downstream case management system has to parse it.
Retrieval handles knowledge. Fine-tuning handles form. Trying to make either do the other's job is where the expensive mistakes live.
The part that decides it in regulated work
Everything above is the engineering answer. In banking it's frequently not the deciding one.
With RAG you can point at the retrieved chunk and say: this paragraph, of this document, version 3, effective this date, produced this answer. With a fine-tuned model the knowledge is smeared across billions of weights and there is no such answer. You cannot show an auditor a weight.
That single property — traceability — settles the architecture more often than any accuracy benchmark, and it's why so many financial-crime deployments end up retrieval-first even where fine-tuning would perform perfectly well. When a regulator asks why a customer was flagged, "the model learned it during training" is not a response anyone wants to give.
The short version
Start with the plain model and a good prompt. Genuinely try this — a surprising number of "we need RAG" problems are prompt problems.
Add retrieval when the answer depends on information the model cannot have: your documents, your data, anything after the cutoff. Expect to spend most of your effort on retrieval quality rather than on the model itself.
Consider fine-tuning when the shape of the output is the problem, and you have the labelled examples to do it properly.
And whatever you build, ask what happens when it's confidently wrong — because it will be, and in this domain the answer to that question matters more than the demo.
An earlier version of this post was published on Medium.
