What RAG is, in plain words
RAG (retrieval-augmented generation) connects search with a language model. Before answering, the system finds the fragments of your knowledge base that relate to the question and puts them into the model’s context. The model answers from what was found.
Think of an employee asked a question about an internal policy. They open the right pages and answer from them, citing the clause. RAG works the same way, which is why its answers can be checked.
When you need RAG and when you do not
RAG makes sense when knowledge lives in documents and changes: policies, contracts, technical documentation, a support knowledge base, project history. Update a document, and the system answers from the new version straight away, with no retraining.
RAG is not needed, and only adds complexity, if:
- the answer is computed from structured data: that needs a database query, and search by meaning will not help;
- there are few documents and they fit into the model’s context in full;
- the task calls for actions in systems rather than knowledge: that is an agent’s job, and RAG would be only one of its tools;
- you need a strictly deterministic answer by formula or policy: a rule in code is more reliable.
How a RAG system works
A minimal working pipeline has six parts, and any of them can fail.
- Collecting documents: exports from storage, email, wikis, file shares. This is also the step where you decide what to do with scans and tables.
- Chunking: each document is cut into pieces that the model will receive whole. Chunks that are too small lose their meaning; chunks that are too large blur the search.
- Vectorisation: each chunk is turned into a set of numbers that reflects its meaning and goes into a vector database.
- Retrieval: the question is turned into the same kind of vector, and the database returns the closest chunks. Good systems add ordinary full-text search over terms.
- Context assembly: the selected chunks are filtered, reordered and put into the prompt together with the instruction.
- Generation with citations: the model answers and states which chunks the answer is based on.
Where RAG breaks on real documents
Chunking
The most common cause of bad answers. If a policy is cut in the middle of a clause, the chunk loses the condition under which the clause applies, and the model answers confidently and wrongly. Chunking has to follow the structure of the document: sections, clauses, tables. Cutting by a fixed number of characters does not work.
Tables, scans and appendices
A table flattened into plain text loses the link between header and value. A scan without text recognition never reaches the search at all. Corporate archives usually hold more of this material than anyone expects at the start, so handling it is planned as a separate line of work.
Versions and duplicates
A contract, its amended version and a draft all sit in the base at once. Search by meaning will find any of them just as readily. If a document has no date, status or version priority, the system will answer from a revoked version and, formally, will not be wrong.
Meaning versus terms
Vector search handles rephrasing well and exact identifiers badly: part numbers, GOST standard numbers, version names. Hybrid search helps: vector and full-text search work together, and their results are merged.
Access rights
If filtering by access rights is not built into the search itself, sooner or later the system will quote to an employee a document they are not allowed to see. Access rights have to be part of the index. Filtering a finished answer is too late.
How to tell that the system works
A feeling that “it answers reasonably well” does not count as acceptance. You need a set of questions with known correct answers and references to the source. It is put together with domain experts before launch, and against it you measure what really matters:
- whether the right chunks were found (retrieval quality is measured separately from answer quality);
- whether the answer rests on what was found or the model added something of its own;
- whether the answer carries a reference to the source and whether it points to the right place;
- whether the system can decline when the documents hold no answer. This is a separate skill, and it is tested on purpose.
Retrieval and generation are measured separately because they are fixed in different ways. If the error is in retrieval, changing the prompt will do nothing, and the other way round. Without that split, improving the system becomes guesswork.
RAG, fine-tuning or a knowledge graph
The three approaches solve different problems, and they are often confused.
- RAG fits when knowledge changes and has to be cited. Updating the data costs next to nothing.
- Fine-tuning fits when you need to change the model’s behaviour and style or teach it a narrow format. It is poor at giving the model knowledge of facts, and it goes stale along with the data.
- A knowledge graph fits when the relations between entities matter and the answer takes several steps of reasoning: who is linked to whom, what depends on what, which part sits in which assembly.
Strong systems usually combine the approaches: the graph handles structure and relations, vector search handles phrasing, fine-tuning handles style and format. How to choose between a graph and vector search is covered in a separate article.
Where to start
- Take one process where questions repeat and an answer has a measurable cost: support, product selection from documentation, contract review.
- Collect 30–50 real questions with reference answers. This is the most valuable work of the stage, and you cannot hand it over to a contractor entirely.
- Build a prototype on a limited set of documents and test it on your questions.
- Expand the corpus only after that. A bad system on a large corpus costs many times more to debug.