Short answer
A long context LLM accepts very large inputs, but its advertised window is not the length at which it reliably uses what it reads. Evaluate it on your own documents: plant facts at controlled depths, test multi-hop, aggregation and absence questions, add realistic distractors such as superseded versions, and plot accuracy against length. The effective context is the longest length where accuracy stays within your tolerance.
- Advertised context length and effective context length are different numbers; measure the second.
- Literal needle tests flatter models; use questions that require paraphrase, multi-hop reasoning and aggregation.
- Superseded document versions are the most realistic distractor in business corpora.
- Include absence questions; a model that invents answers for missing facts fails in production.
- Report latency and cost by length, because both grow with every token you send.
Why the advertised window is not the usable window
A model’s context window is how much text it will accept; its effective context is how much it reliably uses. Published research has shown the gap repeatedly. Lost in the Middle found that models often perform best when the relevant information sits at the start or end of the input and worse when it sits in the middle. RULER, which adds multi-hop tracing and aggregation tasks to needle retrieval, found that of models claiming 32K-token windows or more, only half kept satisfactory performance at 32K. NoLiMa removed the literal word overlap between question and planted fact, and at 32K tokens 11 of the 13 models tested fell below half of their own short-context scores.
Simple sweeps such as the open-source Needle In A Haystack test, which scores retrieval across a grid of context lengths and needle depths, are a good smoke test but share these weaknesses. Models improve every few months, so the papers tell you what to test rather than how today’s models score. The only number that matters for a build decision is the effective context of the model you plan to use, on documents like yours.
What to measure
Define the metrics before building the test set, so the results answer a decision rather than decorate a slide.
| Metric | Definition | Why it matters |
|---|---|---|
| Accuracy at (length, depth) | Share of questions answered correctly when the evidence sits at a given position in a context of a given size | Shows where the model stops reading reliably |
| Effective context | Longest tested length at which accuracy stays within a set tolerance (say 5 points) of short-context accuracy | The number to design around |
| Distractor sensitivity | Accuracy without distractors minus accuracy with them | Business documents are full of near-duplicates |
| Absence precision | Share of questions with no answer in the context that the model correctly says it cannot answer | Catches invented answers |
| Citation accuracy | Share of answers whose cited passage actually contains the supporting evidence | An answer is only as good as its source |
| Latency and cost by length | Time to first token, total time and spend per call at each length | Long inputs are slow and paid for on every call |
Build the test set from your own documents
Use the documents the system will read in production: contracts, policy manuals, board papers, technical standards. Public documents risk testing what the model memorised during training rather than what it read in your prompt.
Write questions in five types, because each fails differently:
| Question type | Example (contract set) | Typical failure |
|---|---|---|
| Single fact, paraphrased | “How long does the supplier have to fix a defect?” when the text says “remediation period” | Missed because the words differ |
| Multi-hop | “Which clause overrides the termination notice period in schedule 3?” | Finds one hop, guesses the other |
| Aggregation | “List every obligation that falls on the tenant” | Stops early, misses items deep in the input |
| Conflict | Two versions of a policy with different limits; “What is the current limit?” | Uses the older or first-seen version |
| Absence | A question whose answer is not in the documents | Invents a plausible answer |
Then place the evidence at controlled depths (0%, 25%, 50%, 75%, 100% of the input) and test at several lengths, for example 8,000, 32,000, 128,000 and 256,000 tokens, up to the model’s limit. Count length in the model’s own tokens, not words.
def build_context(evidence: str, filler_docs: list[str], depth: float,
target_tokens: int, count_tokens) -> str:
"""Insert the evidence at a relative depth inside filler made from real documents."""
filler, used = [], 0
for doc in filler_docs:
n = count_tokens(doc)
if used + n > target_tokens - count_tokens(evidence):
break
filler.append(doc)
used += n
cut = round(depth * len(filler))
return "\n\n".join(filler[:cut] + [evidence] + filler[cut:])
def run_grid(cases, lengths, depths, ask, grade, repeats=3):
results = []
for case in cases:
for length in lengths:
for depth in depths:
ctx = build_context(case.evidence, case.filler, depth, length, case.count_tokens)
for r in range(repeats):
answer = ask(ctx, case.question)
results.append((case.id, case.kind, length, depth, r, grade(case, answer)))
return results
Filler should be real documents from the same domain, not random text. A model can ignore Lorem Ipsum; it cannot ignore forty other supplier agreements.
Stress the model with distractors
Distractor tests are where long context models fail on business documents, so give them their own section of the suite. Four kinds are worth building:
- Superseded versions. The 2024 and 2026 versions of the same policy, both in context. Score whether the model uses the effective one and says so.
- Near-duplicate entities. Two customers with similar names, two sites with the same street name in different towns.
- Lexical decoys. Passages that share keywords with the question but answer a different question.
- Embedded instructions. A document that contains “ignore previous instructions” text. Long inputs give injected text more places to hide; see prompt injection testing.
Run each distractor case with and without the distractor present. The difference, not the raw score, is the measure.
Reading the results and choosing a design
Plot accuracy against length for each question type, one line per depth, and mark the effective context. Then make the design call with the numbers in front of you:
| What the results show | Design choice |
|---|---|
| Corpus fits well inside the effective context and changes rarely | Send it whole; use prompt caching for repeated questions |
| Corpus is larger than the effective context, or changes daily | Retrieve first, then let the model read a generous candidate set |
| Aggregation questions fail at length | Split the input, answer per section, then combine (map then reduce) |
| Conflict questions fail | Resolve versions before the model sees them: keep only effective versions, or label each with its date |
| Absence precision is low | Require a citation for every claim and reject answers without one |
Cost belongs in the same table. Illustrative arithmetic, with assumed numbers you should replace with your provider’s current list prices: 200,000 input tokens per question at an assumed US$2 per million input tokens is US$0.40 per question before output, so 1,000 questions a day is about US$400 a day. Retrieval that sends 20,000 well-chosen tokens costs a tenth of that. Our comparison of RAG and long context and the guide to managing LLM costs work through the trade in more detail, reducing AI latency covers the time-to-first-token side, and RAG retrieval evaluation shows how to measure the retrieval half of a hybrid design.
Where long context evaluation goes wrong
| Failure | Why it misleads | Control |
|---|---|---|
| Literal needles only | Models match strings instead of reading | Paraphrase every question; include multi-hop and aggregation |
| Memorised content | Public documents test training data, not context use | Run each question closed-book; drop items the model answers without context |
| Words counted instead of tokens | Lengths are wrong by a large margin, more for te reo Māori and code | Count with the target model’s tokenizer |
| Single run per cell | Sampling variance looks like a depth effect | Repeat runs and report spread |
| Lenient grading | An LLM judge accepts vague answers | Grade against exact evidence spans; spot-check the judge by hand |
| Hidden truncation or compaction | A platform silently trims long inputs | Plant a check fact at the very end and assert it is answerable |
| Testing once | A model update changes behaviour | Rerun the grid on every model change, as part of model migration |
Citation-grounded products such as Lucent, Levy Street’s investment research venture, show why the citation check matters: an answer that cannot point to its source is not usable in professional work, however fluent it is. If you want this testing done independently on your documents, it is part of our LLM evaluation services.
- Test documents come from your own domain, and filler is real documents rather than random text
- Questions cover single fact (paraphrased), multi-hop, aggregation, conflict and absence
- Evidence is placed at five or more depths across four or more lengths, counted in model tokens
- Each cell has several questions and repeated runs, with variance reported
- Superseded versions, near-duplicates, lexical decoys and embedded instructions are tested with and without the distractor
- Closed-book runs remove questions the model can answer from memory
- Effective context is defined with an explicit tolerance and reported per question type
- Latency and cost are recorded for every length tested
- The grid is rerun on every model or prompt change
Sources
- Lost in the Middle: How Language Models Use Long Contexts, arXiv (Liu et al.), 2023
- RULER: What’s the Real Context Size of Your Long-Context Language Models?, arXiv (Hsieh et al.), 2024
- NoLiMa: Long-Context Evaluation Beyond Literal Matching, arXiv (Modarressi et al.), 2025
- Needle In A Haystack, GitHub (Greg Kamradt), 2023
Written by the Levy Street team in Wellington. Last updated 19 October 2026.