What happens, technically
A language model does not store facts as records that can be checked. It predicts how text continues: at each step it chooses the next fragment based on statistical patterns learned in training. For this process, plausibility and truth are not the same thing, and the model optimises plausibility.
So for a model, “I don’t know” is a statistically rare continuation. In the texts it learned from, a question is almost always followed by an answer. The model reproduces this pattern and confidently fills in whatever is missing, the way it completes the end of a word.
Why it is not called a bug
The same mechanism gives useful properties: the model paraphrases, generalises, handles tasks that were not in its training data and fills in meaning from an incomplete description. A model unable to say anything beyond the texts it absorbed would be useless as an assistant.
Accepting this does not mean doing nothing. You stop looking for the one right setting and design a system in which an invented answer has a hard time reaching the user.
Five types of hallucination that behave differently
- An invented fact: a policy clause, an article of law or a version number that does not exist. The most dangerous type in corporate tasks.
- An invented source: a link or a document title that does not exist. This is caught automatically, because whether a source exists is easy to check.
- Mixed sources: two real documents glued into one answer, so that a condition from one is applied to the subject of the other.
- Outdated knowledge presented as current: the model answers as things stood when it was trained.
- Invented reasoning: a correct answer with a wrong justification. It happens more often than it seems and is almost never caught by checking the final answer.
What really reduces invented answers
Give the model a source and require a citation
The baseline measure: the answer is assembled from retrieved fragments and must refer to them. This does not remove the problem, but it makes it checkable. If there is no citation, or the citation does not support the claim, the answer is rejected before a person sees it.
Check that what the model named exists
Document numbers, part numbers, identifiers, links and table names can be checked against real systems in code. Such a check catches a whole class of hallucinations with no person involved, and it costs next to nothing compared with the consequences.
Allow refusal and reward it
The system needs an explicit path for “this is not in the sources”. It is tested on purpose: the set of test questions must include ones where the correct answer is a refusal. Without them, the model will get good scores for confident invention.
Narrow the task
The broader the question, the more freedom the model has to fill in. If you split the work into narrow steps with a checkable result, there will be less invention than any instruction in the prompt could achieve.
Deterministic constraints on the output
Answer format, allowed values, ranges and required fields are checked by code, not by a request to the model. Anything critical must be impossible to violate. A wish written in the prompt is something the model may simply not follow.
What only creates a feeling of control
- “Only tell the truth” in the prompt. The instruction changes the style, not the prediction mechanism.
- Asking for confidence as a percentage. The model makes that up too: the numbers are not linked to the real probability of error.
- Checking the answer with the same model on the same data. It tends to agree. You need an independent source of truth: a document, a calculation, a query to a system.
- A larger model. It is wrong less often but more convincingly, and checking it becomes harder.
- Lowering the temperature. The output becomes more stable, but a consistently wrong answer is still wrong.
How to measure
Without measurement, any measure rests on faith. A minimal set of metrics on a fixed set of questions with reference answers:
- the share of answers fully supported by the sources they cite;
- the share of answers that cite a non-existent or irrelevant source;
- the share of correct refusals on questions whose answer is not in the data;
- the share of wrong refusals. This one needs watching too, or the system becomes uselessly cautious;
- the share of correct answers with a wrong justification. Only spot manual checks of the reasoning catch it.
These figures are taken again at every change of model, prompt or retrieval rules. Model behaviour changes from version to version, and a system that worked before an update can quietly degrade after it.
What it looks like in a project
- Before development: collect 30–50 real questions with reference answers, including ones where the correct answer is a refusal.
- In the architecture: mandatory citation, checks of identifiers in code, deterministic format constraints, a decision log.
- In the pilot: confirmation by a person where an error is costly, and metrics measured on the real flow.
- In operation: a regression run of the set at every change and a watch on the refusal rate. A rise in it is usually the first sign that something has broken in retrieval.