AI glossary
Retrieval-Augmented Generation (RAG): How AI Answers From Your Documents
What RAG is, where the term came from, how it works and when a business needs one.
By Chad Westleigh · Updated
What is retrieval-augmented generation?
Retrieval-augmented generation pairs a search step with a writing step. A language model on its own answers from what it absorbed in training, which is general, dated and silent about your business. RAG fetches the relevant pieces of your own material at question time and places them in the prompt, so the model writes from evidence instead of memory.
The term comes from Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and 11 co-authors, posted in May 2020 and presented at NeurIPS 2020. The paper combined a pre-trained sequence-to-sequence model (parametric memory) with a dense vector index of Wikipedia searched by a neural retriever (non-parametric memory). Modern RAG systems keep the same split, with far larger models and business documents in place of Wikipedia. AI search engines work the same way, which is why generative engine optimization focuses on quotable passages.
How RAG works, step by step
A RAG pipeline has two halves: prepare the documents once, then retrieve and answer on every question.
1. Collect and clean the sources
Choose the documents answers should come from and remove outdated or duplicate versions. A RAG system repeats whatever it retrieves, mistakes included.
2. Chunk and index
Split documents into passages and index them, usually as embeddings (numeric representations of meaning) in a vector database, often alongside keyword search.
3. Retrieve
Turn the question into a search and pull the most relevant passages. Retrieval quality sets the ceiling: Anthropic reported in September 2024 that its contextual retrieval method cut top-20 retrieval failures by 49% in its tests, and by 67% with reranking.
4. Augment the prompt
Insert the retrieved passages into the prompt, with instructions to answer only from them and to say so when the answer isn't there.
5. Generate and cite
The model writes the answer and points to the passages it used, so a person can check the source.
6. Evaluate
Test on real questions with known answers, scoring retrieval and answers separately. Most failures start with the wrong passage.
RAG vs fine-tuning vs long context vs MCP
| Approach | What it does | Suits | Watch out for |
|---|---|---|---|
| Retrieval-augmented generation | Searches your documents at question time and feeds the relevant passages to the model | Large or changing knowledge: policies, help centers, catalogs, contracts | Wrong or stale passages produce confident wrong answers |
| Fine-tuning | Further trains a model on examples to change its behavior | Consistent format, tone or classification at volume | Less reliable for adding facts; retrain when information changes |
| Long context | Places whole documents directly in the prompt | A handful of documents per task | Cost and latency grow with every page |
| MCP tool calls | Lets the model query live systems such as a CRM or store | Current records: orders, inventory, account status | Needs scoped permissions and logging |
How Operator Intelligence uses RAG
Operator Intelligence uses retrieval-augmented generation when an AI agent or assistant must answer from company knowledge that is too large, or changes too often, to paste into a prompt: policies, SOPs, product specifications, support history or the course material behind a coaching program assistant. Often a lighter option is enough. A Claude Project or ChatGPT project with curated knowledge files covers many team use cases without a custom pipeline, and our Claude training sets those up in your own account.
When live records matter more than documents, we connect the model to the system itself through the Model Context Protocol rather than copying data into an index. Repeatable procedures, such as how to format a report, belong in Claude Skills. For custom AI agents we often combine all three, with evaluation sets built from your real past questions.
Frequently asked questions
Does RAG stop AI hallucinations?
RAG reduces them but doesn't eliminate them. A model can still misread a passage, blend two sources or answer when retrieval found nothing useful. Instructions to answer only from sources, visible citations and evaluation on real questions keep errors in check.
Do we need a vector database for RAG?
Not always. Small collections can sit in a Claude Project or ChatGPT project. A dedicated vector database, often paired with keyword search, makes sense with thousands of documents, frequent updates or strict access rules.
Is RAG better than fine-tuning?
For facts, usually. RAG updates when the documents do and can cite its sources; fine-tuning is better at changing a model's style, format or classification behavior.
Is our data safe in a RAG system?
That depends on where the index lives and who can query it. We keep indexes in your accounts, mirror existing document permissions so people retrieve only what they're allowed to see, and use business or API tiers of model providers, not consumer apps.
Put your company knowledge to work
Tell us which questions your team answers over and over. We'll tell you whether that needs RAG, a configured AI workspace or something simpler. Or start with the AI readiness assessment.
Book a scoping callSources
- arXiv / NeurIPS 2020 (2020): 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' by Patrick Lewis and 11 co-authors (12 authors total); arXiv v1 22 May 2020; accepted at NeurIPS 2020; combines a pre-trained seq2seq model (parametric memory) with a dense vector index of Wikipedia accessed by a pre-trained neural retriever (non-parametric memory).
- Anthropic (2024): Contextual Embeddings plus Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49%; with reranking, by 67% (published Sep 19, 2024).
- OpenAI (announcement, September 25, 2025) and OpenAI Help Center, Projects in ChatGPT (2025): Shared projects available on ChatGPT Business, Enterprise and Edu; projects group chats, reference files and custom instructions
Get the Operator Intelligence briefing
A short, practical note every two weeks on what is working with AI in real businesses. No spam; unsubscribe anytime.
Related
Model Context Protocol (MCP)
The open standard for connecting AI to live tools and data.
Claude Skills
Packaged procedures Claude loads when a task needs them.
Custom AI agents
Agents grounded in your documents and systems.
Claude training for business teams
Projects, Skills and connectors set up in your account.
AI for law firms
Document workflows that keep client information in the firm's systems.
AI glossary
Sixteen AI terms defined for business leaders.