AI engineeringLLM · Quality · Evaluation

LLM hallucinations: the mechanism and the engineering measures against them

LLM hallucinations are not a model failure. They are the model’s normal mode of operation arriving at a result that is inconvenient for us. The mechanism matters more than any list of “prompts against hallucinations”: it shows which measures work and which only create a feeling of control.

What happens, technically

A language model does not store facts as records that can be checked. It predicts how text continues: at each step it chooses the next fragment based on statistical patterns learned in training. For this process, plausibility and truth are not the same thing, and the model optimises plausibility.

So for a model, “I don’t know” is a statistically rare continuation. In the texts it learned from, a question is almost always followed by an answer. The model reproduces this pattern and confidently fills in whatever is missing, the way it completes the end of a word.

Why it is not called a bug

The same mechanism gives useful properties: the model paraphrases, generalises, handles tasks that were not in its training data and fills in meaning from an incomplete description. A model unable to say anything beyond the texts it absorbed would be useless as an assistant.

Accepting this does not mean doing nothing. You stop looking for the one right setting and design a system in which an invented answer has a hard time reaching the user.

Five types of hallucination that behave differently

What really reduces invented answers

Give the model a source and require a citation

The baseline measure: the answer is assembled from retrieved fragments and must refer to them. This does not remove the problem, but it makes it checkable. If there is no citation, or the citation does not support the claim, the answer is rejected before a person sees it.

Check that what the model named exists

Document numbers, part numbers, identifiers, links and table names can be checked against real systems in code. Such a check catches a whole class of hallucinations with no person involved, and it costs next to nothing compared with the consequences.

Allow refusal and reward it

The system needs an explicit path for “this is not in the sources”. It is tested on purpose: the set of test questions must include ones where the correct answer is a refusal. Without them, the model will get good scores for confident invention.

Narrow the task

The broader the question, the more freedom the model has to fill in. If you split the work into narrow steps with a checkable result, there will be less invention than any instruction in the prompt could achieve.

Deterministic constraints on the output

Answer format, allowed values, ranges and required fields are checked by code, not by a request to the model. Anything critical must be impossible to violate. A wish written in the prompt is something the model may simply not follow.

What only creates a feeling of control

How to measure

Without measurement, any measure rests on faith. A minimal set of metrics on a fixed set of questions with reference answers:

These figures are taken again at every change of model, prompt or retrieval rules. Model behaviour changes from version to version, and a system that worked before an update can quietly degrade after it.

What it looks like in a project

  1. Before development: collect 30–50 real questions with reference answers, including ones where the correct answer is a refusal.
  2. In the architecture: mandatory citation, checks of identifiers in code, deterministic format constraints, a decision log.
  3. In the pilot: confirmation by a person where an error is costly, and metrics measured on the real flow.
  4. In operation: a regression run of the set at every change and a watch on the refusal rate. A rise in it is usually the first sign that something has broken in retrieval.

Frequently asked questions

Can hallucinations be eliminated completely?

No. They come from the same mechanism that makes the model useful. A realistic goal is to bring their frequency down to a level the process can accept, and to make the remaining errors visible and reversible.

Does RAG solve the hallucination problem?

Partly. RAG gives the model something to rely on and the ability to cite. But if retrieval brought the wrong fragment, or a fragment cut off in the middle of a condition, the model will confidently fill in the missing part. That is why retrieval quality is measured separately from answer quality.

Which data quality metric has the strongest effect on hallucinations?

The completeness and currency of the corpus. If the answer is not in the data or is out of date, the system has two paths: refuse or invent. Next in influence comes correct chunking of documents: a fragment without the condition under which it applies pushes the model to fill in.

Does multi-agent checking help?

Only when the checker has an independent source of truth: a test run, a comparison against a document, a recalculation. Checking with the same model on the same data produces agreement; it does not give control.

How we help with this

Read next

Tell us about your problem

If you’re dealing with something similar, describe it in your own words in the form on the homepage. If it has a simpler solution than it seems, we’ll say so.

New write-ups on TelegramEmail hello@xteam.pro