RAG vs long context for New Zealand businesses.

How do RAG and long context compare in 2026?

Long context means putting the whole document set into the prompt and letting the model read it. Retrieval-augmented generation (RAG) means indexing the documents, retrieving the passages most relevant to each question, and sending only those. In 2026 the window is rarely the constraint: every current Claude model takes one million tokens, and OpenAI lists a 1,050,000-token window for its GPT-6 models.

Many production systems combine them: retrieve a generous candidate set, then let a long context model read it all. Our guide to evaluating long context models explains why the advertised window and the usable window differ.

How do you test both on your own documents?

Build both versions over the same documents and ask the same questions, with answers you have checked by hand.

1. A 300-page procedures manual. Ask 40 questions with known answers, plus five whose answers are not in the manual. Score correct answers, accurate citations and whether each version says “not covered”.
2. Twelve supplier contracts. Ask “which contracts allow termination for convenience, and on what notice?” Score completeness: every contract must be listed correctly.
3. Two thousand support tickets. Ask for every ticket in the last quarter mentioning a refund over $500. Score recall against a list you built with a plain search.

Record cost and response time per query for each. Our guide to RAG retrieval evaluation covers the retrieval-side measures.

What does each cost per query?

Worked example: a 600,000-token document set, 200 questions a month, about 1,000 output tokens per answer, at list prices on 19 October 2026 in US dollars.

The rates behind it: Anthropic bills the full million-token window at standard rates on Claude 4.6 and later models, except Haiku 5.5, which charges more for prompts over 100,000 tokens. OpenAI bills GPT-6 prompts over 272,000 input tokens at twice the input rate and 1.5 times the output rate for the whole request. A Claude Sonnet 5.5 cache read costs $0.10 per million tokens, against $4 for a one-hour cache write.

RAG adds costs the table does not show. Embedding the set is trivial: 600,000 tokens at OpenAI’s $0.02 per million is about one cent. The real costs are the index, its hosting and the engineering to build and maintain retrieval. With a build cost B and monthly running cost V, RAG pays back in B ÷ (long context monthly cost − RAG monthly cost − V) months. At 20 questions a month, long context on Sonnet 5.5 costs about $24 uncached, and RAG rarely pays back.

Measure token counts with the model you will use. Anthropic says its newer tokenizer produces about 30% more tokens than older Claude models for the same text, so counts from another tokenizer will undercount.

Which should you choose?

Choose long context if

• The document set is bounded and fits the window with room to spare.
• Questions need the whole set: comparisons, completeness checks, summaries.
• Query volume is low, or questions arrive in bursts that caching can absorb.
• Every user may see every document in the set.

Choose RAG if

• The corpus is large or grows beyond a single window.
• Documents change daily and freshness matters.
• Users must only see documents they are permitted to see.
• Volume is high enough that per-query cost dominates; see our RAG development service.

Where do both fall short?

• Superseded versions. Both answer from an old policy as readily as the current one. Remove or label outdated documents first.
• Confident gaps. Long context models can skip material deep in a long input; RAG systems answer from whatever was retrieved. Absence questions in your test set catch both.
• Privacy obligations. The Privacy Commissioner warns that generative AI tools may not always be compatible with rights to access and correct personal information (IPPs 6 and 7). An index you control makes finding and removing a person’s records easier than a long prompt rebuilt from scattered files.
• Offshore processing. Both approaches send documents to a model. For Claude, Amazon Bedrock’s Australian profile keeps inference within Australia and New Zealand; see data residency for AI systems for the options, and choosing a vector database for where an index can live.

For the basics, start with our lesson on understanding retrieval augmented generation.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.