AI model migration.

Why model migrations keep happening

Every production AI system will change models, usually more than once a year. Providers retire older models, new ones are cheaper or better, and occasionally a model’s behaviour changes enough that staying put is the risk.

Retirement notice periods are published. As listed on 19 October 2026, OpenAI’s deprecations page states minimum notice of at least six months for generally available models and at least three months for specialised variants, unless safety or compliance require faster action; preview models can go with much shorter notice. Anthropic’s model deprecations page states at least 60 days’ notice for publicly released models. Sixty days is not long for a regulated workflow, so the migration process has to be ready before the notice arrives.

Pin dated model snapshots in configuration. Aliases that point to “the latest” version change behaviour without a deploy, which turns a planned migration into an unplanned one.

Inventory every call site

You cannot migrate what you have not listed. Keep a registry with one row per place the system calls a model.

The prompt and evaluation versioning that makes this registry possible is covered in versioning prompts and evaluations.

What changes between models

A new model is a new dependency, even from the same provider. These are the differences that break production systems.

Regional availability catches New Zealand teams out. For example, when Amazon Bedrock launched in the Auckland region in March 2026, its Australia and New Zealand geographic routing covered Anthropic Claude models, while a broader model set needed global routing. A model you can call is not necessarily a model you can call in-region; see data residency for AI systems.

Build the regression suite and gates

Freeze a regression suite before you touch the prompt. Draw it from four places: the existing golden set, a stratified sample of recent production traffic, cases from your human review queue where staff corrected the model, and every past incident. Then set gates per call site.

GATES = {
    "task_accuracy":     {"max_drop": 0.01},
    "schema_valid_rate": {"max_drop": 0.0},
    "refusal_rate":      {"max_rise": 0.005},
    "p95_latency_s":     {"max_rise": 0.5},
    "cost_per_task_usd": {"max_rise_pct": 0.10},
}

def check_gates(old: dict, new: dict) -> list[str]:
    failures = []
    for metric, rule in GATES.items():
        delta = new[metric] - old[metric]
        if "max_drop" in rule and -delta > rule["max_drop"]:
            failures.append(f"{metric} fell by {-delta:.3f}")
        if "max_rise" in rule and delta > rule["max_rise"]:
            failures.append(f"{metric} rose by {delta:.3f}")
        if "max_rise_pct" in rule and delta > rule["max_rise_pct"] * old[metric]:
            failures.append(f"{metric} rose by {delta / old[metric]:.0%}")
    return failures

Aggregate gates are necessary but not sufficient. List every case that flipped from pass to fail and have someone read them; a 1% accuracy change can hide a whole category of failures, such as every credit note now coded as an invoice. The method for building suites is in AI agent evaluations.

Keep the judge fixed. If you score with a model-based judge, do not change the judge model during the migration, or you are measuring two changes at once. Research on LLM-as-a-judge documented position and verbosity biases in judges, and a more verbose candidate model can score higher for that reason alone. Calibrate the judge against human grades on a sample.

Adapt the prompt, then roll out in stages

Do not port prompts verbatim. Read the provider’s migration notes, remove workarounds for the old model’s quirks, and tune against the regression suite. Re-check output token caps and any reasoning or effort settings, which change both quality and cost. Version the adapted prompt separately so either combination can be restored, and tell the people who review outputs that a change is under way, so they report anything odd during the ramp rather than quietly fixing it.

1. Offline regression. The candidate passes every gate on the frozen suite, and someone has read all flipped cases.
2. Shadow traffic. A copy of live requests goes to the candidate; users still get the current model. Compare format validity, agreement, cost and latency for several days of normal traffic.
3. Canary. Route a small share of real traffic, say 5%, to the candidate, bucketed by tenant or user so each person gets consistent behaviour. Automatic rollback triggers on schema failures, error rate, review-queue override rate or latency.
4. Ramp. Increase to 25%, 50% and 100% over days, checking the same signals at each step.
5. Retire the old path. Keep the previous model configured until the provider’s retirement date, then remove it.

import hashlib

ROUTES = {
    "invoice_coding": {"stable": "provider-model-2026-03-05",
                       "candidate": "provider-model-2026-09-08",
                       "canary_pct": 5},
}

def pick_model(call_site: str, tenant_id: str) -> str:
    route = ROUTES[call_site]
    bucket = int(hashlib.sha256(f"{call_site}:{tenant_id}".encode()).hexdigest(), 16) % 100
    return route["candidate"] if bucket < route["canary_pct"] else route["stable"]

Rollback is a configuration change: set canary_pct to zero. Practise it before you need it. During the ramp, watch the production signals described in monitoring AI quality.

Embedding and retrieval migrations

Embedding model changes are migrations of the whole index, not a configuration switch. Vectors from different models are not comparable, and OWASP’s 2026 Top 10 for LLM applications recommends re-embedding the corpus rather than mixing old and new vectors. Build the new index alongside the old one, run your retrieval evaluation on both, switch the application to the new index by alias, and keep the old index until the new one has run cleanly in production. Index operations differ by database; see choosing a vector database.

Where migrations go wrong

Migrations are routine work in managed AI operations, and choosing the candidate in the first place is part of AI model selection.

• Every call site is in a registry with model snapshot, prompt version, parameters, region, owner and retirement date
• Deprecation notices from each provider reach a named owner
• Model IDs are dated snapshots; no production call uses a moving alias
• The regression suite includes production samples, review-queue corrections and past incidents
• Gates cover accuracy, schema validity, refusals, latency and cost, per category
• Every case that flips from pass to fail has been read by a person
• The evaluation judge is pinned and calibrated against human grades
• The candidate is confirmed available in the required region and approved for the data
• Rollout runs offline, shadow, canary and ramp, with automatic rollback triggers
• Embedding changes rebuild the full index in parallel before switching

Written by the Levy Street team in Wellington. Last updated 19 October 2026.