Short answer
RAG (retrieval-augmented generation) development is building AI systems that answer from your own documents: the system searches an index for the relevant passages, then a language model writes an answer that cites them. Levy Street builds RAG systems for New Zealand organisations with permission-aware retrieval, a relevance benchmark built from your staff’s real questions, and citations a person can check in one click.
- Most RAG failures are retrieval failures: the right passage never came back.
- Retrieval must enforce the same permissions as the source system.
- Measure retrieval and answers separately, on real questions from your staff.
- Questions that need counting or totals belong in a database query, not RAG.
What RAG is, and when it is the right tool
RAG (retrieval-augmented generation) is a design in which an AI system first searches your documents for the passages relevant to a question, then asks a language model to answer using only those passages, with citations. Knowledge stays in an index you control, so updating what the system knows means updating documents, not retraining a model.
It is the right tool when the collection is too large or too changeable to paste into every request, when different people may see different documents, or when answers must point to a source someone can check. It is the wrong tool when the question needs counting or totals across many records (“how many leases expire next year?”), which belongs in a database query, or when a small, shared document set fits comfortably into a long-context model. Our comparison of RAG and long context works through the choice.
Who it is for
Teams that spend real time finding answers in documents they already own: policy and procedure libraries, contract repositories, maintenance manuals, tenancy files, board papers and support knowledge bases. Typical first projects include internal policy retrieval in financial services and maintenance knowledge search in manufacturing.
What we deliver
- Source inventory
- Every document source, its format, owner, update frequency and permission model.
- Ingestion pipeline
- Extraction from PDFs, Word, SharePoint, Google Drive and scanned files, split into passages with titles, dates and section headings attached.
- Search index
- Keyword and semantic search combined, tuned on your questions rather than defaults.
- Permission model
- Each passage carries the access rules of its source document, enforced before retrieval.
- Answer layer
- Answers that cite passages, quote where precision matters, and say so when the documents do not cover the question.
- Relevance benchmark
- Real staff questions with the passages that answer them, used to score every change.
- Freshness and monitoring
- Scheduled syncs, deletion handling, usage logs and a feedback button on every answer.
Access control is the hard part
Most RAG demos skip permissions, and most RAG security problems start there. The OWASP Top 10 for LLM Applications 2026 lists both sensitive information disclosure and vector and embedding weaknesses. We choose one of three designs per collection:
| Design | How it works | When it fits |
|---|---|---|
| Separate indexes per audience | Each group gets its own index built only from documents it may see | A few clearly separated audiences, such as staff and board |
| Access lists copied into the index | Each passage stores who may read it; results are filtered by the user’s identity at query time | Large collections with SharePoint or Drive permissions |
| Live check against the source | Candidate passages are re-checked with the source system before use | Sensitive collections where permissions change often |
Two rules apply to all three: filter before the model sees anything, and make deletions in the source flow through to the index on a schedule you can state.
How the work runs
- Discover: collect real questions. We gather a hundred or so questions your staff actually ask, with where each answer lives, and inventory the sources and their permissions. Questions with no written answer go on a gap list.
- Prove: a retrieval demo on your documents. We index a representative slice and score retrieval first: does the right passage appear in the top five results? Then we score answers for faithfulness to their citations. You see the numbers before anything is scaled.
- Scale: full ingestion and upkeep. All sources are connected with permissions enforced, syncs scheduled, and feedback flowing back into the benchmark. Ongoing care can sit with your team or our managed AI operations.
The relevance benchmark is the deliverable that lasts. When someone proposes a new embedding model, chunk size or search setting, the benchmark decides. Our guide to RAG retrieval evaluation explains the metrics.
New Zealand specifics
- Where the index lives. The search index and document copies can sit in New Zealand: AWS opened its Auckland region in September 2025, and Azure lists New Zealand North in Auckland.
- Where the model runs. No major provider offers in-New Zealand inference. Amazon Bedrock’s AU profile keeps Claude processing within Australia and New Zealand. See data residency for AI systems.
- Privacy. An index is a copy of personal information. The Privacy Act 2020 principles on access, correction and retention apply to it, so corrected or deleted records must not live on in the index.
What it costs to build and run
Build cost is driven by the number of sources and connectors, how messy the documents are (scanned forms and complex tables need extra extraction work), the permission design, and how many question types the benchmark must cover. Running costs are usually modest. Worked examples at prices published on 19 October 2026: embedding 10,000 documents of 2,000 tokens each (20 million tokens) with OpenAI’s text-embedding-3-small at US$0.02 per million tokens costs about US$0.40 (OpenAI). An answer that reads 4,000 tokens of passages and writes 400 tokens on Claude Sonnet 5.5 (US$2 input, US$10 output per million; Anthropic) costs about US$0.012. Hosting the index and the engineering time to keep sources healthy usually outweigh both. See what custom AI projects cost.
Where RAG goes wrong
- The right passage never comes back. Poor extraction, chunks that split a clause from its exceptions, or acronyms the search does not recognise. Measure retrieval on its own.
- Contradictory or stale documents. If three versions of a policy exist, the system will find all three. Source hygiene comes first.
- Tables and forms. Rates tables and scanned forms lose structure in extraction unless handled deliberately.
- Planted instructions. Any document in the index can carry text aimed at the model. Retrieved text is treated as data, never as instructions.
Work this draws on
- Lucent: a Levy Street venture, an agentic investment research platform covering NZX and ASX that runs several frontier models together and grounds its outputs in citations.
- PainPal: Levy Street worked with BetterWorx on this AI workmate for aches and strains, which answers from expert-reviewed guidance rather than the model’s general knowledge.
Sources
- API pricing, OpenAI, accessed 2026-10-19
- Pricing (Claude API), Anthropic, accessed 2026-10-19
- Models overview (Claude), Anthropic, accessed 2026-10-19
- Run generative AI inference with Amazon Bedrock in Asia Pacific (New Zealand), AWS Machine Learning Blog, 2026-03-26
- Now open: AWS Asia Pacific (New Zealand) Region, AWS News Blog, 2025-09-01
- Azure regions list, Microsoft Learn, accessed 2026-10-19
- OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Project, 2026-08-03
- Privacy Act 2020, New Zealand Legislation, version as at 2026-10-01
Written by the Levy Street team in Wellington. Last updated 19 October 2026.