What is Retrieval Augmented Generation?
Also called RAG.
Retrieval augmented generation is a technique that improves language model output by fetching relevant documents from an external store at query time and placing them in the model's prompt before it answers. The model composes its response from that supplied evidence rather than from its parameters alone, which allows answers to reflect private or recently updated information.
A standard language model answers only from patterns absorbed during training, so it cannot cite a document it never saw and cannot know what changed after training ended. Retrieval augmented generation adds a lookup step in front of generation. A retriever searches an indexed collection for passages related to the question, and those passages are concatenated into the prompt as reference material.
The pattern was named in a 2020 research paper and has since become the common way to connect a general model to a specific corpus. It is usually cheaper and faster to change than retraining, because updating an answer means updating a document rather than adjusting model weights. Removing a document from the index removes it from future answers.
Quality depends more on retrieval than on generation. If the retriever returns passages that do not contain the answer, the model must either say so or invent something, so precision at the top of the result list matters greatly. Common failure modes include documents split at awkward boundaries, queries whose wording differs from the source text, and stale entries left in the index.
Practitioners disagree about how much retrieval remains necessary as context windows grow, since a small corpus can sometimes be supplied in whole. Retrieval still helps when the corpus is large, when permissions vary by reader, or when the cost of processing every document on every request would be prohibitive.
Key points
- Fetches relevant documents at query time, then generates from them.
- Updates by changing documents, not by retraining the model.
- Answer quality is bounded by retrieval quality.
- Enables citations by tracking which passage supported a claim.
- Common alternative to fine tuning for private or changing data.
In practice
A support assistant receives the question, what is the refund window for annual plans. The retriever embeds the question, searches the policy index, and returns three passages from a billing document updated last week. Those passages are inserted into the prompt, and the model answers thirty days, quoting the clause. Because the passage came from the index, the reply links back to its source document.