AI model selection.

What model selection decides

AI model selection is deciding which model runs each step of a system, through which provider or cloud, with what fallback, and when the choice gets rechecked. A document workflow might use a small, fast model to classify incoming files, a mid-tier model to extract fields and a frontier model only for the hard exceptions. Picking one model for everything usually means paying frontier prices for simple work, or getting cheap-model errors on the hard parts.

The decision has four inputs: accuracy on your tasks, cost per completed task, speed, and where your data is processed. Leaderboards answer none of them for your workflow.

Illustrative example: classifying 20,000 inbound documents a month at 1,500 input and 50 output tokens each uses 30 million input and 1 million output tokens. At the prices listed below, GPT-6 Luna costs US$3.00 plus US$0.50, about US$3.50 a month; GPT-6 Astra costs US$300 plus US$50, about US$350. If both classify equally well on your test set, the choice is easy. If the cheaper model misroutes 4% more documents, the real question is what 800 misrouted documents a month cost your team.

Why leaderboards are not enough

Bigger is not reliably better on a specific job. In GTM-Bench, a 72-task benchmark for go-to-market research agents published by Blackpearl Group in June 2026, Claude Sonnet 4.6 finished with a positive net score while the larger Claude Opus 4.7 finished below zero, because weak results were penalised. A purpose-built system led by a wide margin.

Price per token is not comparable either. Anthropic notes that its Claude 4.7 and later models use a tokenizer that “produces approximately 30% more tokens for the same text” (Anthropic), so the honest comparison is cost per completed task, measured on your own inputs.

The current field, with listed prices

Listed standard-tier API prices in US dollars per million tokens, as published on 19 October 2026. Prices change often; recheck before budgeting.

Sources: OpenAI, Anthropic, Google. All three offer roughly 50% discounts for batch processing that can wait.

What we deliver

Task inventory

Each step of the workflow, what it needs (reading, reasoning, writing, tool use) and its volume.

Candidate shortlist

Two to four models per step, including an open-weight option where data control or volume justifies it.

Test set

Real cases with known good outcomes, reusable for every future comparison through our LLM evaluation method.

Comparison report

Accuracy by category, cost per completed task, typical and slow-case response times, and failure examples for each candidate.

Routing design

Which model runs which step, the fallback when a provider is down, and the rules for escalating to a stronger model.

Hosting and data note

Provider, route (direct, Bedrock, Azure or Google Cloud) and processing location for each choice.

How the work runs

1. Discover: break the work into steps. We map the workflow, collect real inputs for each step and agree what a correct output looks like with the people who do the work.
2. Prove: run the comparison. Every candidate runs the same test set under the same prompts and settings, with results logged so anyone can reproduce them. We tune prompts per model only where the difference is material, and say so in the report.
3. Scale: choose, route and recheck. You approve the routing, we wire it in, and the saved test reruns whenever a provider ships or retires a model. Our guide to AI model migration covers the switch itself.

Where your data is processed

Processing location often narrows the field before accuracy does. As of October 2026, no major provider offers AI processing inside New Zealand. OpenAI’s Australian data-residency region stores data but does not process it there (OpenAI). Anthropic’s own API offers only US or global processing (Anthropic). Amazon Bedrock’s AU profile keeps Claude processing within Australia and New Zealand, and Anthropic lists a 10% premium for Claude on regional rather than global endpoints of Bedrock and Google Cloud. Under the Privacy Act a provider that only processes data on your behalf leaves you fully responsible for it, so we record the location for every step.

What drives the cost

The number of distinct tasks, the effort to build a good test set, how many candidates are compared, and whether open-weight hosting is tested (it needs infrastructure set up) drive the cost of a selection project. Running the comparison itself is cheap in model fees. The bigger financial effect is downstream: routing simple steps to a model at a tenth of the price can matter more than any other optimisation. See managing LLM costs.

Limits of model selection

Results expire. A comparison is only as good as its test set and only as current as the models in it, which is why the deliverable is a rerunnable test rather than a recommendation. For low-volume work, elaborate routing is not worth it; pick one capable model and move on. And no model choice fixes a bad prompt, missing context or messy source documents.

Work this draws on

• GTM-Bench: co-authored by Levy Street team members and published by Blackpearl Group, it compared agents built on OpenAI, Anthropic, Google, DeepSeek and Kimi models on the same 72 tasks with the same scoring.
• Lucent: a Levy Street venture that runs several frontier models together for investment research and grounds outputs in citations, so choosing and combining models is a production concern for us, not a theory.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.