The short version
Retrieval decides most of it: if the right passage was never fetched, no amount of prompting saves the answer. Then force citation, so an unsupported claim is visibly unsupported. Then make “I don't know, here's who does” a first-class outcome rather than a failure. Then measure — because without a test set you are managing a feeling.
Why prompting alone does not fix it
A model generates the most likely next text given everything in front of it. When the context contains the answer, the likely continuation is the correct one. When it does not, the likely continuation is something that reads like the correct one — same shape, same confidence, wrong content.
That is why the failure is so convincing. It is not a bug producing garbage; it is the system working exactly as designed on insufficient input. Which tells you where to intervene: the input.
Fix one: the retrieval is the real problem
In most deployments that invent things, the model never saw the answer. Chunking split the paragraph in half. The search matched on wording rather than meaning. The document was out of date. The passage that mattered ranked fourth and only three were passed in.
Before touching the prompt, take twenty questions the system got wrong and check whether the correct passage was retrieved at all. In practice a large share of hallucinations are retrieval failures wearing a costume, and they are fixed by chunking, ranking and freshness work rather than by instructions.
Fix two: make every claim carry a citation
Require the answer to quote or reference the source passage for each substantive statement, and show that citation to the user. This does two things at once. It makes an unsupported claim visible rather than indistinguishable, and it gives the reader a cheap way to verify without trusting the summary.
It also changes the failure mode from “confidently wrong” to “obviously unsupported”, which is the difference between a system people stop trusting and one they learn to use.
Fix three: design the refusal properly
Most systems treat “I don’t know” as failure, so they are tuned — implicitly or explicitly — to always produce something. Invert it. A good refusal states what it could not find, what it did find that was close, and where to go next. That is a genuinely useful answer, and it costs nothing in trust.
A deployment where the model declines eight percent of questions and is right on the rest is far more valuable than one that answers everything and is wrong three percent of the time, because in the second case every answer needs checking — which was the work you were removing.
How do you measure it?
Build a test set before you tune anything. Fifty to a hundred real questions, with correct answers agreed by someone who knows the domain, including questions the system should refuse. Run it after every change.
Track three numbers separately, because they move independently: how often the right source was retrieved, how often the answer was correct given what was retrieved, and how often it refused when it should have. Averaging them into a single accuracy figure hides which part is broken and is the most common reason teams tune the wrong thing for weeks.
What if it still gets things wrong?
It will, so design for it. Route consequential answers through a human. Show sources so errors are catchable by the reader. Log every question and answer so a complaint can be reconstructed. Cap what the system is allowed to do without approval.
Every deployment that survives contact with real users has a plan for being wrong. The ones that fail were built on the assumption that enough tuning would make the plan unnecessary.
Will a different model fix it?
Marginally, and never completely. Newer models are better at declining and better at staying within supplied context, so the rate falls. But the failure is structural rather than a defect: given insufficient grounds, the most probable continuation is still a plausible-sounding invention.
Which means model choice is worth perhaps a modest improvement, while retrieval quality, citation and refusal design are worth far more. Teams that respond to a hallucination problem by upgrading the model usually find the rate drops slightly and then plateaus, because they have not touched the cause.
What about fine-tuning on our own data?
Usually the wrong tool for this problem. Fine-tuning shapes style, format and task behaviour well. It is a poor way to install facts, because the model learns the shape of your content rather than acquiring a reliable lookup — and facts that change require retraining rather than an edit.
For grounding answers in current, correct information, retrieval is the appropriate mechanism: the document is fetched at question time, it can be updated the moment it changes, and it can be cited. Fine-tuning and retrieval solve different problems and are often confused in proposals.
How much should we expect it to refuse?
More than feels comfortable, and the number is worth agreeing in advance. A system covering a well-documented domain might decline a small percentage of questions; one covering a patchy knowledge base will decline considerably more, and that is the knowledge base telling you something useful.
Track refusals as a metric rather than a fault. A rising refusal rate is usually a documentation problem surfacing, which is valuable information — the questions people ask that the system cannot answer are a ranked list of what to write next.
Who should own this after launch?
Someone named, with time allocated. The failure rate on a retrieval system is not static — the documents change, people ask new kinds of question, and a model version update shifts behaviour slightly. Without an owner running the test set periodically, degradation is discovered by a customer complaint.
The job is small but real: run the evaluation monthly, review the questions that were refused, and feed the gaps back into the knowledge base. An hour a month protects the thing you paid to build.
What does good look like in practice?
A deployment where the answer arrives with its source attached, the reader can check it in one click, the system says so when it does not know, consequential answers reach a person before they reach a customer, and someone can tell you this month’s numbers for retrieval accuracy, answer accuracy and refusal rate.
None of that is exotic and none of it depends on which model you chose. It is the difference between a system people come to rely on and one that quietly stops being used after the first embarrassing answer.