Short answer
Retrieval augmented generation (RAG) is a way of making an AI model answer from your documents: a search step finds the passages most relevant to a question, those passages are placed in the prompt, and the model writes an answer from them, ideally with citations. Most RAG failures are retrieval failures, where the right passage was never found, so test retrieval separately from the answer.
- RAG has four steps: chunk the documents, embed them, retrieve passages, generate an answer.
- If retrieval misses the right passage, even the best model answers wrongly.
- Old and new versions of a policy in the same index cause confident contradictions.
- Small, stable document sets can often be pasted in whole instead.
- A few poisoned documents can steer a RAG system, so control who can add content.
- Goal
- Understand how retrieval augmented generation works by doing the retrieval step yourself, and learn to tell a retrieval failure from a generation failure.
- Who it is for
- Managers, analysts, knowledge managers and anyone about to buy, build or own an AI system that answers from company documents.
- Time
- 30 minutes.
- You need
- ChatGPT, Claude, Copilot or Gemini and the three fictional policy documents below.
- You will produce
- Four test questions with results, and a short note on which failures came from retrieval.
What is retrieval augmented generation?
Retrieval augmented generation (RAG) is a pattern where an AI system looks things up before it answers. When someone asks a question, a search step finds the most relevant passages in your documents, places them in the prompt, and the model writes an answer from them. Done well, the answer cites the passages it used. The model itself is not retrained; your documents are consulted fresh each time.
How does each step work?
RAG has four steps, and each has its own way of failing:
| Step | What happens | What goes wrong |
|---|---|---|
| Chunk | Documents are split into passages of a few hundred words | A table or rule split across two chunks loses its meaning |
| Embed | Each chunk becomes an embedding, a list of numbers representing meaning | Jargon, codes and te reo terms may embed poorly if the model rarely saw them |
| Retrieve | The question is embedded and the closest chunks are fetched | The right passage is not in the top results, or an outdated one is |
| Generate | The model answers using the retrieved chunks | It blends in general knowledge or cites a chunk that does not support the claim |
OpenAI’s embeddings guide puts the core idea simply: an embedding is a list of numbers, and “the distance between two vectors measures their relatedness.” Anthropic’s documentation notes it “does not offer its own embedding model”, so Claude-based systems pair it with another provider’s embeddings.
Where do you already meet RAG?
You may be using RAG already. Microsoft says Copilot accesses content through Microsoft Graph and can anchor its responses in your organisation’s documents, emails, calendar and chats. Google renamed NotebookLM to Gemini Notebook on 16 July 2026; it answers from sources you upload. Custom builds add control over which sources count, who can see what, and how answers are tested.
Exercise: be the retriever
You will play the search step by hand. The company and documents are fictional.
Illustrative documents. Fictional company: TÅtara Freight Ltd.
D1 Leave policy (2025)
1. Annual leave requests need 2 weeks' notice via the HR portal.
2. Requests over 10 days need the operations manager's approval.
D2 Vehicle use policy (2025)
1. Fleet vehicles may be taken home only by on-call staff.
2. Personal use of fuel cards is not permitted.
D3 Expenses policy (2023)
1. Meal allowance on overnight trips: $45 per day.
2. Receipts are required for all claims over $20.
D3b Expenses policy update (March 2026)
1. Replaces D3 section 1. Meal allowance on overnight trips:
$60 per day.
Questions
Q1 How much notice do I need for annual leave?
Q2 I'm on call this weekend. Can I take the van home and use the
fuel card to fill my own car?
Q3 What is the policy on working from home?
Q4 What is the meal allowance for an overnight trip?
- Give it everything. Paste all four documents and the four questions with the worked example prompt. Record the answers. With everything in view, a good tool should get all four right and say Q3 is not covered.
- Retrieve badly on purpose. In a new chat, paste only D3 (the 2023 policy) and ask Q4. This is what happens when retrieval finds the old version. Note how confident the answer is.
- Retrieve only half. In a new chat, paste only D2 section 1 and ask Q2. The answer will miss the fuel card rule because that chunk was never retrieved.
- Retrieve nothing relevant. Paste only D1 and ask Q3. Check whether the tool says the documents do not cover it or invents a policy.
- Label each failure. For steps 2 to 4, write whether the wrong answer came from retrieval (the right text was missing) or generation (the right text was there but misused).
- Write one test question for your own documents where the answer depends on the newest version of a policy.
Worked example: a grounded answering prompt
Answer the questions using ONLY the documents below.
- Cite the document and section for every statement, e.g. (D2 s1).
- If two documents conflict, use the newer one and say which one
it replaces.
- If the documents do not answer a question, reply "Not covered
by these documents" and stop.
[paste documents and questions]
Example output (illustrative and abridged; your tool’s wording will differ):
Q2 Yes, on-call staff may take a fleet vehicle home (D2 s1). No,
you may not use the fuel card for your own car; personal use
is not permitted (D2 s2).
Q3 Not covered by these documents.
Q4 $60 per day (D3b s1). This replaces the $45 rate in the 2023
policy (D3 s1).
Compare that with step 2, where the model saw only D3 and answered $45 with the same confidence. The model did nothing wrong; retrieval handed it the wrong page.
When is RAG the right choice?
RAG earns its complexity when the document set is large, changes often, serves many users or needs permission controls. If you have a handful of stable documents that fit in the model’s context, pasting or uploading them whole is simpler and often more accurate; our comparison of RAG versus long context sets out the trade-offs, and RAG versus fine-tuning explains why retraining rarely suits factual answering.
Where this goes wrong
- Retrieval misses. Most bad answers trace back to the wrong chunks. Test retrieval on its own, as in RAG retrieval evaluation.
- Old versions left in the index. Superseded policies produce confident contradictions. Archive them or label them clearly.
- Permissions ignored. An index that does not respect source permissions can surface HR or client files to anyone.
- Poisoned content. The OWASP Top 10 for LLM Applications 2026 cites research where as few as five poisoned documents reached roughly 90% attack success against a knowledge base of millions of texts. Control who can add documents.
- Citations taken on trust. A cited section can still fail to support the sentence. Spot-check them, as in summarising documents with citations.
Check your work
- I can name the four steps of RAG and one failure for each.
- In the exercise I saw a wrong answer caused by retrieving the old policy.
- I can tell whether a wrong answer came from retrieval or generation.
- My test questions include one the documents cannot answer.
- I know whether our document set is small enough to skip RAG entirely.
Where to go next
Next in the pathway: choosing between an API and a subscription, the decision you face once a RAG idea moves toward a build. To organise the documents first, see building a searchable business knowledge base, and check outputs with the AI quality review checklist. Levy Street’s Lucent research platform is built around outputs grounded in citations. When you are ready for a production system with retrieval testing and permissions, see our RAG development service.
Sources
- Vector embeddings, OpenAI, Accessed 19 October 2026
- Embeddings, Anthropic, Accessed 19 October 2026
- Data, privacy, and security for Microsoft Copilot, Microsoft Learn, Accessed 19 October 2026
- NotebookLM is now Gemini Notebook, Google, 16 July 2026
- OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Project, August 2026
Written by the Levy Street team in Wellington. Last updated 19 October 2026.