AI engineeringRAG · Document search · LLM engineering

RAG system: what it is, when you need it and where it breaks

A RAG system makes a language model answer from your documents instead of from what it remembers of the internet. The idea is simple, and its failures are predictable. Almost all of them happen before the model: where the documents are cut into chunks, searched and passed to it.

What RAG is, in plain words

RAG (retrieval-augmented generation) connects search with a language model. Before answering, the system finds the fragments of your knowledge base that relate to the question and puts them into the model’s context. The model answers from what was found.

Think of an employee asked a question about an internal policy. They open the right pages and answer from them, citing the clause. RAG works the same way, which is why its answers can be checked.

When you need RAG and when you do not

RAG makes sense when knowledge lives in documents and changes: policies, contracts, technical documentation, a support knowledge base, project history. Update a document, and the system answers from the new version straight away, with no retraining.

RAG is not needed, and only adds complexity, if:

How a RAG system works

A minimal working pipeline has six parts, and any of them can fail.

  1. Collecting documents: exports from storage, email, wikis, file shares. This is also the step where you decide what to do with scans and tables.
  2. Chunking: each document is cut into pieces that the model will receive whole. Chunks that are too small lose their meaning; chunks that are too large blur the search.
  3. Vectorisation: each chunk is turned into a set of numbers that reflects its meaning and goes into a vector database.
  4. Retrieval: the question is turned into the same kind of vector, and the database returns the closest chunks. Good systems add ordinary full-text search over terms.
  5. Context assembly: the selected chunks are filtered, reordered and put into the prompt together with the instruction.
  6. Generation with citations: the model answers and states which chunks the answer is based on.

Where RAG breaks on real documents

Chunking

The most common cause of bad answers. If a policy is cut in the middle of a clause, the chunk loses the condition under which the clause applies, and the model answers confidently and wrongly. Chunking has to follow the structure of the document: sections, clauses, tables. Cutting by a fixed number of characters does not work.

Tables, scans and appendices

A table flattened into plain text loses the link between header and value. A scan without text recognition never reaches the search at all. Corporate archives usually hold more of this material than anyone expects at the start, so handling it is planned as a separate line of work.

Versions and duplicates

A contract, its amended version and a draft all sit in the base at once. Search by meaning will find any of them just as readily. If a document has no date, status or version priority, the system will answer from a revoked version and, formally, will not be wrong.

Meaning versus terms

Vector search handles rephrasing well and exact identifiers badly: part numbers, GOST standard numbers, version names. Hybrid search helps: vector and full-text search work together, and their results are merged.

Access rights

If filtering by access rights is not built into the search itself, sooner or later the system will quote to an employee a document they are not allowed to see. Access rights have to be part of the index. Filtering a finished answer is too late.

How to tell that the system works

A feeling that “it answers reasonably well” does not count as acceptance. You need a set of questions with known correct answers and references to the source. It is put together with domain experts before launch, and against it you measure what really matters:

Retrieval and generation are measured separately because they are fixed in different ways. If the error is in retrieval, changing the prompt will do nothing, and the other way round. Without that split, improving the system becomes guesswork.

RAG, fine-tuning or a knowledge graph

The three approaches solve different problems, and they are often confused.

Strong systems usually combine the approaches: the graph handles structure and relations, vector search handles phrasing, fine-tuning handles style and format. How to choose between a graph and vector search is covered in a separate article.

Where to start

  1. Take one process where questions repeat and an answer has a measurable cost: support, product selection from documentation, contract review.
  2. Collect 30–50 real questions with reference answers. This is the most valuable work of the stage, and you cannot hand it over to a contractor entirely.
  3. Build a prototype on a limited set of documents and test it on your questions.
  4. Expand the corpus only after that. A bad system on a large corpus costs many times more to debug.

Frequently asked questions

How is RAG different from ordinary document search?

Ordinary search returns a list of documents and leaves the reading to a person. RAG finds fragments and formulates an answer, stating where it came from. Search still sits inside it: if search works badly, the answer will be bad whatever the model.

Can a RAG system be built on-premise?

Yes. Both the vector database and the model can run inside your perimeter, with no calls to external providers. Local models fall behind the best cloud models on complex reasoning, so the scenario is matched to what the model can do. This is discussed before the start, so that nobody finds out at the end.

How many documents does it take for this to make sense?

What makes it worthwhile is how often the questions repeat; the number of documents matters less. If people turn to the same policy every day and an error has a noticeable cost, a hundred documents is enough. If the requests are one-offs, it is cheaper to keep ordinary search.

Why does the system sometimes answer confidently and wrongly?

A language model formulates its answer from what it was given. If retrieval brought the wrong chunk, or a chunk cut off in the middle of a condition, the model fills in the missing part itself. Good chunking and retrieval help, along with a required reference to the source and the ability to decline to answer.

How we help with this

Read next

Tell us about your problem

If you’re dealing with something similar, describe it in your own words in the form on the homepage. If it has a simpler solution than it seems, we’ll say so.

New write-ups on TelegramEmail hello@xteam.pro