Managing LLM costs.

What should you measure before cutting LLM costs?

Measure cost per successful task: total spend on model calls, embeddings, tools and human rework for a period, divided by the number of runs that met their success criteria. Price per token ignores failures, retries and the people who fix bad outputs, which is where most of the money goes.

Illustrative example, with stated assumptions: model A costs US$0.020 per run and succeeds 95% of the time; model B costs US$0.008 per run and succeeds 70% of the time. Assume each failure costs US$2.00 of staff time to catch and fix. Per run, A costs 0.020 + 0.05 × 2.00 = US$0.12 and B costs 0.008 + 0.30 × 2.00 = US$0.61. The cheaper model is five times more expensive. Success rates come from your evaluation suite and from quality monitoring, not from guesses.

Which tokens are you actually paying for?

You pay for several kinds of tokens at different rates, and providers report them differently, so normalise usage before you build dashboards.

Two accounting traps catch most teams. Anthropic’s input_tokens counts only tokens after the last cache breakpoint, so total input is cache reads plus cache writes plus input_tokens (prompt caching docs). OpenAI’s input_tokens includes cached and cache-write tokens, which are broken out in input_tokens_details (OpenAI prompt caching). Summing them naively double-counts one provider and under-counts the other. And per Anthropic’s pricing page, Claude 4.7 and later models use a tokenizer that produces about 30% more tokens for the same text, so a per-token comparison across model generations can mislead.

What do the main models cost today?

These are list prices in USD per million tokens, short-context rates, as listed on 19 October 2026 on the Anthropic, OpenAI and Google pricing pages. Check them before budgeting; they change often.

Google lists Gemini 3.8 Flash rising to US$1.50 input and US$7.50 output from 1 January 2027. Budgets built on today’s introductory price will be wrong in ten weeks. Our AI cost calculator lets you plug in your own volumes.

When does prompt caching pay off?

Caching pays off when the same prefix is read enough times within its lifetime to recover the write premium. With a write multiplier w, a read multiplier r and N requests sharing a prefix within the cache window, caching is cheaper than not caching when N is greater than (w minus r) divided by (1 minus r).

For Claude’s 5-minute cache (w = 1.25, r = 0.1), that is N greater than about 1.3, so a single read after the write already saves money; for the 1-hour cache (w = 2), N must exceed about 2.1. Anthropic’s pricing page states the same result: one read for the 5-minute cache, two for the 1-hour cache.

Worked example: a 10,000-token system prompt and tool definitions on Claude Sonnet 5.5, 50,000 requests a month. Uncached, that prefix costs 500 million tokens at US$2.00, or US$1,000. Assume 2% of requests land after the cache has expired and pay a 5-minute write at US$2.50: 1,000 writes cost US$25. The other 49,000 read at US$0.10: US$49. The prefix now costs about US$74 a month.

Minimum lengths matter. Claude’s minimum cacheable prompt is 512 tokens on Opus 5.5, Sonnet 5.5, Haiku 5.5 and the Fable and Mythos models, and 1,024 tokens or more on several older ones; shorter prompts are processed uncached with no error. OpenAI’s minimum is 1,024 tokens on GPT-5.6 and later, and its guide shows that padding a shared prefix up to the minimum with useful, stable content can be cheaper than leaving it short. Google’s implicit caching needs 4,096 tokens on Gemini 4 Argon and 3.8 Flash.

Things that silently break caches: a timestamp or user name at the top of the prompt, tool definitions in a different order on each request, and changing the structured output schema, which Anthropic notes invalidates the prompt cache for that thread. Put stable content first and variable content last.

When should you use batch or flex processing?

Use them for any work where nobody is waiting. As listed on 19 October 2026, Anthropic’s Batch API charges 50% of standard input and output prices; OpenAI’s Batch API offers a 50% discount with a 24-hour completion window; and Google’s batch mode is priced at 50% of standard with a 24-hour target turnaround. OpenAI’s flex processing charges batch rates for ordinary requests with slower responses, and may return a 429 resource-unavailable error, which is not charged; fall back to standard processing if the job is time-bound.

Good candidates: nightly document extraction, CRM enrichment, evaluation suite runs, back-catalogue processing and scheduled reports. Discounts stack with caching on the providers that document it, so a batch job over a shared prefix can cost a small fraction of the interactive price.

How do routing and cascades cut cost?

Route each step to the cheapest model that passes its evaluation, and escalate only when needed. A cascade tries a small model first and escalates to a larger one when validation fails or confidence is low. Expected cost per request is the small model’s cost plus the escalation rate times the large model’s cost. With a small model at one-twentieth of the large model’s price, the cascade wins whenever fewer than about 95% of requests escalate, but it only saves money if the outputs the small model keeps are actually good, which your evaluation must show. Choosing models step by step is the core of our AI model selection work.

How do you enforce budgets in code?

Put limits where the calls are made: per run, per customer or feature per day, with an alert before the hard stop.

PRICES = {  # USD per million tokens, as listed 19 October 2026
    "claude-sonnet-5-5": {"input": 2.00, "cache_read": 0.10, "cache_write": 2.50, "output": 10.00},
    "gpt-6.1-sol":       {"input": 2.00, "cache_read": 0.10, "cache_write": 2.50, "output": 10.00},
}

class BudgetExceeded(Exception):
    pass

def call_cost(model: str, u: dict) -> float:
    """u holds normalised counts: uncached input, cache_read, cache_write, output (reasoning included)."""
    p = PRICES[model]
    return (u["input"] * p["input"] + u["cache_read"] * p["cache_read"]
            + u["cache_write"] * p["cache_write"] + u["output"] * p["output"]) / 1_000_000

class Budget:
    def __init__(self, per_run_usd: float, per_day_usd: float, spent_today_usd: float):
        self.per_run, self.per_day = per_run_usd, per_day_usd
        self.run_spend, self.day_spend = 0.0, spent_today_usd

    def check(self, estimated_next_usd: float) -> None:
        if self.run_spend + estimated_next_usd > self.per_run:
            raise BudgetExceeded("run limit")
        if self.day_spend + estimated_next_usd > self.per_day:
            raise BudgetExceeded("daily limit")

    def charge(self, model: str, usage: dict) -> float:
        cost = call_cost(model, usage)
        self.run_spend += cost
        self.day_spend += cost
        return cost

Normalise usage into uncached, cache-read, cache-write and output counts per provider before calling charge, using the accounting rules above. When BudgetExceeded is raised, stop the run cleanly and escalate it, rather than retrying.

Where does LLM cost control go wrong?

Most overruns come from behaviour, not prices.

• Retries without limits. A failing step retried five times costs five times as much and usually fails again.
• Agent loops. Costs grow with steps; cap them, as in designing reliable AI agents.
• Verbose outputs. Output tokens cost several times more than input. Ask for only the fields you use.
• Caches that never hit. Track cache hit rate per prompt; a write-only cache is a surcharge.
• Cutting quality to cut cost. A cheaper model that raises rework or complaints moves the cost to staff time.
• Ignoring scheduled price changes. Introductory prices end; diarise them.

Latency and cost often improve together, since shorter outputs and fewer calls help both; see reducing AI latency. For the full cost of building and running a system, not just tokens, our guide to what custom AI projects cost covers the arithmetic, and the AI budget lesson helps non-engineers plan. Ongoing cost reviews are part of managed AI operations.

• Cost per successful task is reported, including estimated rework
• Usage is normalised across providers into uncached, cached, write and output tokens
• Prices and their dates are stored in config, with scheduled changes diarised
• Stable prompt content comes first, and cache hit rate is tracked per prompt
• Prefixes meet each provider’s minimum cacheable length
• Work nobody waits for runs through batch or flex processing
• Each step uses the cheapest model that passes its evaluation
• Output length is capped and prompts ask only for used fields
• Per-run and per-day budgets are enforced in code with early alerts
• Retries and agent steps are capped and logged

Written by the Levy Street team in Wellington. Last updated 19 October 2026.