Reducing AI latency.

Where does LLM latency come from?

For a single model call, latency is roughly network time, plus queueing, plus time to first token, plus the number of output tokens divided by output speed. For a real system, add every sequential model call, retrieval step and tool round trip on the critical path. Output generation usually dominates: OpenAI’s latency optimisation guide says cutting 50% of output tokens may cut about 50% of latency, while cutting 50% of the prompt may improve it by only 1 to 5%.

Reasoning or thinking tokens count here too. Providers bill them as output (Google’s pricing tables, for example, list output prices “including thinking tokens”), and they take time to generate even when you never see them.

How do you set a latency budget?

Write the budget as p95 targets per stage before optimising anything, so every change has a number to beat. Illustrative example: a support assistant that rewrites the query, retrieves policy passages, looks up the customer in the CRM and streams a reply, with targets of 2 seconds to first visible text and 8 seconds to complete.

The useful output of this exercise is the critical path. Parallel stages cost only their slowest member, and every stage on the path either needs a reason to exist or a cheaper way to run.

Which optimisations actually reduce latency?

Work down this list in order: the first items are free and usually larger than the later ones.

Streaming is the single biggest perceived improvement: OpenAI’s guide calls it the most effective way to make users wait less. Anthropic’s latency guide makes the same three core points: choose a faster model where it is good enough, trim prompt and output length, and stream. Anthropic’s pricing documentation also notes that prompt caching reduces latency as well as cost, which matters most for long system prompts and documents reused across requests.

When are fast tiers worth paying for?

Fast tiers are worth it for user-facing steps with long outputs, where generation speed is the bottleneck and someone is waiting. As listed on 19 October 2026:

• Anthropic fast mode gives up to 2.5 times higher output tokens per second on Claude Opus 5.5 (and, in research preview, Opus 5 and Opus 4.8). The fast mode documentation says the benefit is in output speed, not time to first token, and the model is the same. Opus 5.5 fast mode costs US$8 per million input tokens and US$40 per million output tokens, against US$4 and US$20 at standard speed, and is not available through the Batch API.
• OpenAI fast mode, the renamed priority processing tier, promises up to 2.5 times faster speeds on supported models. Its documentation says GPT-5.6 Sol costs twice the standard rate in fast mode, and that traffic ramping too quickly can be downgraded to standard speed, shown as service_tier: "default" in the response.
• Google’s priority inference for the Gemini API is in preview and is prioritised above standard and Flex traffic at a premium price.

Log the service tier actually returned on every response. Paying for a fast tier while silently receiving standard speed is a common and invisible waste; see managing LLM costs for the cost side.

Does running AI from New Zealand add latency?

Distance adds network time to every call, and agents make many calls, so it is worth measuring. Amazon Bedrock is available in the Asia Pacific (New Zealand) region (ap-southeast-6), using cross-region inference; its Australia profile routes requests from Auckland across Auckland, Sydney and Melbourne. Anthropic’s own API offers inference_geo values of global or US-only, according to its data residency documentation. Measure a minimal request from your own servers to each endpoint to separate network time from model time, and keep the tool servers your agent calls close to the orchestrator. Location also affects where data is processed; see data residency for AI systems.

How do you measure a latency change properly?

Use a fixed, representative request set, interleave the variants so both see the same provider conditions, and compare percentiles, not averages.

1. Freeze a request set. 100 to 300 real requests covering short and long outputs, with tools stubbed or pointed at stable test systems.
2. Interleave variants. Alternate baseline and candidate in random order at production concurrency, rather than running one after the other.
3. Record per request. TTFT, end-to-end time, input and output tokens, cached tokens, and the service tier returned.
4. Compare p50 and p95. Look at the distribution, especially the tail, and the token counts that explain it.
5. Check quality. Run the evaluation suite on the candidate. A faster wrong answer is not an improvement.

import asyncio
import random
import time

async def timed(stream, request):
    """stream(request) yields chunks; adapt the attribute names to your SDK."""
    start = time.perf_counter()
    first = None
    out_tokens = 0
    async for chunk in stream(request):
        if first is None and chunk.text:
            first = time.perf_counter() - start
        out_tokens += chunk.output_tokens
    return {"ttft": first, "total": time.perf_counter() - start, "out": out_tokens}

def percentile(values, q):
    ordered = sorted(values)
    return ordered[min(len(ordered) - 1, int(q * len(ordered)))]

async def compare(variants: dict, requests: list, concurrency: int = 8):
    jobs = [(name, r) for r in requests for name in variants]
    random.shuffle(jobs)                                   # interleave variants
    gate = asyncio.Semaphore(concurrency)
    async def run(name, r):
        async with gate:
            return name, await timed(variants[name], r)
    results = await asyncio.gather(*(run(n, r) for n, r in jobs))
    for name in variants:
        rows = [m for n, m in results if n == name]
        totals = [m["total"] for m in rows]
        ttfts = [m["ttft"] for m in rows if m["ttft"] is not None]
        print(name, "p50", round(percentile(totals, 0.5), 2), "p95", round(percentile(totals, 0.95), 2),
              "ttft p95", round(percentile(ttfts, 0.95), 2),
              "mean out tokens", round(sum(m["out"] for m in rows) / len(rows)))

Keep the same panels running in production, as described in monitoring AI quality, so you see regressions after provider changes too.

Where do latency optimisations go wrong?

The usual mistakes trade a visible improvement for a hidden cost.

• Optimising the average. Users remember the slow tail. Retries, tool timeouts and unusually long outputs drive p95.
• Breaking the cache. A timestamp or user name at the top of the prompt changes the prefix on every request, so the cache never hits and TTFT stays high.
• Schema churn. Structured outputs compile a grammar for each new schema; Anthropic’s structured outputs documentation notes extra latency on first use and a 24-hour cache. Generating schemas per request pays that cost repeatedly.
• Smaller model, worse answers. Routing to a faster model without evaluation moves the cost to the people fixing its mistakes.
• Agent loops. Latency grows with steps. Cap steps, and see tool calling in production for parallel tool calls and timeouts.
• Voice treated like chat. Phone conversations need far tighter budgets; our AI voice agents work starts from the latency budget.

If an existing system is slow and nobody knows why, a measured latency review is a standard part of our AI system rescue work. Where long documents are involved, also weigh RAG versus long context, since sending a whole document every time trades retrieval latency for prompt processing.

• p95 targets exist for time to first token and complete response, per use case
• The critical path is mapped and every serial stage has a reason to exist
• Prompts ask only for output fields that are used, with output caps set
• Responses stream to users, and post-processing works on chunks
• Independent steps and tool calls run in parallel
• Stable prompt prefixes come first and are cached
• Smaller models handle easy steps, proven by evaluation
• Fast tiers are used only where people wait, and the returned tier is logged
• Network time to each provider endpoint is measured separately
• Every latency change is measured on a fixed request set with quality rechecked

Written by the Levy Street team in Wellington. Last updated 19 October 2026.