RAG (retrieval-augmented generation) is a technique that fetches relevant passages from your own documents and gives them to a language model before it answers. The model then responds from that retrieved material rather than from memory, which makes answers current, grounded in your data, and traceable to a source.
That is the whole idea. The rest of this piece is about how it works in practice, what it costs, and why a technique that was declared dead in January 2026 is still running underneath most production AI systems.
How RAG actually works
A RAG system does four things between your question and the answer:
- Chunk and index. Your documents are split into passages and converted into vectors that capture meaning, then stored in a search index.
- Retrieve. Your question is converted the same way, and the index returns the passages closest in meaning, usually blended with old-fashioned keyword matching because exact terms like part numbers still matter.
- Rerank. A second, more expensive model reorders the candidates so the genuinely relevant passages sit at the top.
- Generate. The top passages are handed to the language model with your question and an instruction to answer only from what it was given.
The fourth step is where grounding happens. A model told to answer from supplied passages, and to say when they do not contain the answer, behaves very differently from one asked the same question cold.
Naive RAG vs what production actually needs
| Stage | Naive version | What production needs |
|---|---|---|
| Chunking | Fixed 500-token blocks | Structure-aware splits that keep a table or clause intact |
| Retrieval | Vector similarity only | Hybrid vector plus keyword, so exact identifiers still hit |
| Filtering | None | Metadata filters for date, product, region, and per-user permissions |
| Ranking | Top 5 by distance | Overfetch, then rerank with a cross-encoder |
| Answering | Stuff and hope | Citations back to source, plus a real "I do not know" path |
Almost every disappointing RAG pilot we are asked to rescue is the left column deployed to real users. The gap between the two columns is most of the engineering, and most of the budget.
Is RAG dead?
The argument that went around in January 2026 was that context windows are now large enough to paste in everything, so retrieval is unnecessary. It is wrong for two measurable reasons.
The first is cost. Retrieving a handful of relevant passages costs a fraction of sending an entire corpus with every question. Published comparisons put the gap at roughly three orders of magnitude per query, around 1,250x in one analysis (byteiota). At any real query volume that difference decides whether the product has a viable margin.
The second is accuracy. Long-context models degrade when the relevant passage sits in the middle of a very long input, with reported accuracy drops above 30% in that position. Filling the window with everything makes the needle harder to find, not easier.
What is genuinely true is that the shape changed. The 2026 pattern is hybrid: retrieve to narrow the field, then use a generous context window to reason over what came back. Retrieval decides what the model sees; the long window decides how well it can think about it. Industry commentary has taken to calling the surrounding discipline context engineering (The New Stack), which is a better name for what the work actually is.
When you do not need RAG
Worth saying plainly, because we talk clients out of it regularly:
- Your corpus is genuinely small. Under roughly fifty pages that change rarely, put them in the prompt and skip the infrastructure.
- The answer is a database query. "How many orders shipped last week" is SQL. Retrieval over prose is the wrong tool for structured facts.
- The knowledge is general. If the model already knows it and being slightly out of date is fine, retrieval adds cost and latency for nothing.
- Your documents are wrong. Retrieval faithfully surfaces bad content. Fix the source first, or you have built a fast way to distribute errors.
What it costs
RAG systems run $30k–$300k to build. The corpus drives the number: a thousand clean pages and a million scanned PDFs with handwritten annotations are different projects wearing the same acronym. Per-user permissions, where each person may only retrieve what they are allowed to see, is the single most underestimated requirement, and it has to be designed in rather than bolted on. Our RAG development page covers how we scope it, and the 2026 cost guide puts it next to the other solution types.
How to tell whether yours is working
Measure retrieval separately from generation. Most failures are retrieval failures wearing a generation costume: if the right passage never came back, no amount of prompt tuning will fix the answer. Build a fixed set of real questions with known correct sources, then track how often the right passage appears in the top results at all. That single number tells you where to spend.
Common questions
What is RAG in AI?
RAG stands for retrieval-augmented generation. It is a technique that searches your own documents for passages relevant to a question, then gives those passages to a language model and asks it to answer from them. The result is grounded in your data, current, and traceable back to a source document.
What is the difference between RAG and fine-tuning?
RAG changes what the model knows at question time by supplying relevant documents. Fine-tuning changes how the model behaves by training it on examples. Use RAG for facts that change or must be cited; use fine-tuning for tone, format and task-specific behaviour. Most production systems need retrieval, and only some need fine-tuning.
Is RAG dead in 2026?
No. Long context windows have not replaced retrieval, for two reasons: retrieving a few relevant passages is roughly three orders of magnitude cheaper per query than sending an entire corpus, and long-context accuracy degrades by over 30% when the relevant passage sits mid-window. The 2026 pattern is hybrid: retrieve to narrow, then use a long window to reason.
How much does a RAG system cost to build?
RAG systems cost $30,000–$300,000 depending on corpus size and messiness, retrieval quality targets, per-user permission handling, and evaluation depth. A single-source knowledge assistant sits near the bottom of that range; an enterprise deployment over millions of documents with per-user permissions sits at the top.