AI system rescue and optimisation.

When a system needs rescue

An AI system needs rescue when it no longer earns its place: people route around it, nobody can say how often it is wrong, or it costs more each month for the same work. The common signs:

• Staff check every output by hand, so the system saves no time.
• The pilot impressed in a demo and stalled before production.
• Running costs have climbed with no matching growth in use.
• Behaviour changed after a provider updated or retired a model.
• The original builder has moved on and nobody understands the prompts.

Platform changes force the issue. OpenAI shut down its Assistants API on 26 August 2026 (OpenAI), and Anthropic retires Claude Sonnet 4.5 from its API on 30 November 2026 (Anthropic). Systems built on either need work whether they were performing or not.

The diagnostic checklist

We work through the system layer by layer, using its logs and a labelled sample of real failures:

• Inputs: are real inputs like the ones the system was tested on, including scans, attachments and odd formats?
• Retrieval: for wrong answers, did the right document or passage come back at all?
• Context: is the model given what it needs, and only that, or a growing pile of history and boilerplate?
• Prompt and model: is the instruction clear, and is the model tier right for each step?
• Tools: does each tool do one clear thing, with the narrowest permissions that work?
• Output handling: is output validated before it reaches another system or a customer?
• Human workflow: does reviewing an output take less time than doing the task by hand?
• Measurement: is there a test set, and does anyone look at quality and cost weekly?
• Ownership: is there a named owner who can approve changes?

Where failures usually come from

The long-context rule is from OpenAI’s GPT-6.1 Sol model page.

A failed-pilot recovery, step by step

Illustrative example: a 60-person Christchurch manufacturer built an assistant over its maintenance manuals. Three months in, technicians had stopped using it. We label 150 logged questions with the outcome a technician would have wanted. Two causes dominate. First, retrieval: the manuals are scanned PDFs whose torque and clearance tables were flattened into unreadable text, so the right passage rarely came back. Second, scope: a large share of questions asked for part numbers and stock levels, which live in the ERP system, not in any manual. The fixes follow the evidence. Tables are re-extracted and checked, part-number questions are answered by a direct lookup in the ERP rather than by the language model, and the assistant says it does not know when nothing relevant is found. The same 150 questions are rerun after each change, and the system goes back to a small group of technicians before anyone else. See maintenance knowledge search for the workflow.

What we deliver

Diagnostic report

Failure causes ranked by how many real cases each explains, with examples.

Labelled failure set

The sample used for diagnosis, kept as a regression test.

Fix plan

Changes in order of impact, each with an expected effect and the check that proves it.

Cost review

Tokens and spend per task by step, with savings from caching, batching and model tiering quantified.

Implemented fixes

Built with your team or for them, one change at a time.

Keep, fix or retire

A plain recommendation, including buying a product if one now does the job.

How the work runs

1. Discover: diagnose from evidence. Logs, configuration and a labelled failure sample show where errors come from. No changes are made until the causes are ranked.
2. Prove: fix the biggest cause first. One change at a time, each checked against the failure set and the existing tests, so improvements are real and nothing else regresses.
3. Scale: hand back a system that can be run. Monitoring, an owner and a regression suite are in place before we step away, or we keep running it through managed AI operations.

Cutting running costs

Optimisation is often the quickest win. Current price lists reward four habits: route simple steps to cheaper tiers; keep context lean; cache the stable start of long prompts (OpenAI bills cached input on GPT-6.1 Sol at 5% of the normal rate, and Anthropic bills cache reads on Opus 5.5 and Sonnet 5.5 at 0.05 times base input; OpenAI, Anthropic); and send work that can wait to batch processing, which costs half the standard price (OpenAI pricing). Each change is checked for accuracy, because a cheaper wrong answer is not a saving. Our guide to managing LLM costs goes deeper. Engagement cost depends on the system’s size, access to code and logs, and how many fixes you want implemented rather than specified.

When not to rescue

Some systems should be switched off. If the workflow was the wrong target, if review will always take longer than the work, or if an off-the-shelf product now does the job well enough, more engineering is sunk cost. We will say so in the diagnostic.

Work this draws on

• GTM-Bench: co-authored by Levy Street team members, it scores agents so that weak results count against them. In the paper, one agent returned 6,342 rows on a single task: busy output that looks productive and creates work.
• World of ClaudeCraft: our open-source browser MMO, built fast with AI coding tools and since extended by roughly 60 contributors. An automated architecture test scans every simulation file for forbidden imports and stray clock or random calls (repository), the kind of guardrail that keeps a fast-moving codebase from rotting.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.