Short answer
Managed AI operations means a specialist team runs your AI systems after launch: monitoring quality and cost, owning incidents, testing and releasing prompt and model changes, and keeping integrations working as vendors change. Levy Street runs production AI for New Zealand organisations under a written support scope that states what we watch, how we respond, what we may change and what always needs your approval.
- AI systems degrade quietly as models, documents and inputs change.
- Every change passes the regression suite before it reaches users.
- Model switches, new tools and new data sources always need your approval.
- We run our own AI products in production, so these are runbooks we use.
What managed AI operations covers
Managed AI operations is running an AI system day to day once real people depend on it: watching quality and cost, owning incidents, releasing changes safely and keeping up with vendors. It is the Scale stage of our Discover, Prove, Scale model, offered on its own for systems we built and for systems others built.
- Monitoring
- Quality, cost, speed, errors and usage tracked against baselines, with alerts tuned to your system.
- Quality reviews
- A weekly sample of real outputs scored by a person or a calibrated judge, by category.
- Incident ownership
- Detection, containment, investigation and a written review for every significant incident.
- Release management
- Prompt, model, retrieval and code changes tested against the regression suite and rolled out in stages.
- Vendor watch
- Model retirements, price changes and API changes tracked, tested and planned for.
- Runbook
- How the system works, how to switch it off, and who to call, kept current.
- Monthly report
- Quality, cost, incidents, changes and what is coming.
Why AI systems need their own operations
Conventional monitoring tells you a system is up. It cannot tell you the answers got worse. AI systems drift as documents change and inputs shift, and the platforms underneath move constantly. Between September and October 2026 alone, OpenAI, Anthropic and Google all released new flagship models. OpenAI shut down its Assistants API on 26 August 2026 and will shut down Whisper and its gpt-4o transcription models on 26 February 2027 (OpenAI). Anthropic retires Claude Sonnet 4.5 from its API on 30 November 2026 (Anthropic). Google’s Gemini 4 Argon introductory price doubles on 1 December 2026 (Google). Business platforms change too: Xero’s developer terms now prohibit using data from its APIs to train AI models (Xero). Someone has to read those notices and act on them.
What we monitor
| Signal | How it is measured | Typical response |
|---|---|---|
| Output quality | Weekly scored sample, by category | Investigate any category that drops |
| Retrieval health | Rate of “no relevant document” answers | Check sources, syncs and new question types |
| Overrides and escalations | Share of outputs staff edit or reject | Find the pattern, add cases to the test set |
| Cost per task | Tokens and fees per completed task, by step | Look for context growth, loops or tier creep |
| Speed | Typical and slowest response times | Trace slow steps; adjust model or design |
| Errors and retries | Failed calls, timeouts, tool errors | Fix integration faults; check provider status |
| Abuse signals | Suspected prompt injection, unusual volumes | Contain, then review permissions |
Monitoring needs logs, and AI logs are full of personal information: prompts, documents and replies. We agree retention periods with you, keep logs in a New Zealand or Australian region where the stack allows, and mask fields monitoring does not need, because the Privacy Act’s security and retention principles apply to logs as much as to the source systems. Our guide to AI audit logs covers what to keep.
Incidents and releases
Incidents are graded by harm. The worst grades are exposure of personal or confidential information and actions taken that should not have been. Wrong outputs reaching customers come next, then degraded quality, then cosmetic issues. For serious incidents the first step is containment: a switch that pauses the system or one capability without taking down everything else. If personal information may have been exposed, your organisation decides on notification. Under the Privacy Act 2020 a notifiable breach must be reported to the Privacy Commissioner as soon as practicable, and the Commissioner’s guidance asks for within 72 hours of becoming aware. We supply the facts quickly so you can make that call. Our AI incident log template shows what gets recorded.
Every release passes the regression suite, goes to a small share of traffic first, and can be rolled back. Bug fixes and security patches within approved behaviour can ship without a meeting. Model switches, new tools, new data sources and any change in what the system is allowed to do need your written approval.
Support boundaries
| Area | We own | You own |
|---|---|---|
| AI behaviour | Monitoring, testing, tuning within approved scope | What the system is allowed to do |
| Incidents | Detection, containment, investigation, fixes | Customer and regulator communication |
| Vendors | Tracking changes, testing alternatives | Contracts and approving switches |
| Data | Pipelines, syncs, index health | Source content accuracy and retention rules |
| Infrastructure | The AI services and their integrations | Your wider IT, identity and networks |
How the work runs
- Discover: handover review. We read the system end to end, confirm or build the regression suite, record baselines for quality and cost, and write the runbook. Systems we did not build get this review before we accept them.
- Prove: a supervised first month. Alerts are tuned against real traffic, a mock incident tests the switch-off and the escalation path, and the first monthly report sets the format.
- Scale: steady operations. Monitoring, weekly quality samples, staged releases, vendor watch and a quarterly review of whether the system still earns its place.
What drives the cost
The number of systems and how critical they are, the hours of cover and response times you need, how often the system changes, and how much human quality review is included. Model and hosting fees are separate and usually passed through at cost. Our guides to AI automation pricing and what custom AI projects cost show how running costs are built up.
When you do not need us
A stable, low-risk system with a capable in-house engineer often needs only a good runbook and a regression suite, which we can hand over in a short engagement. We do not take over general IT support, and we do not replace your privacy officer or your legal advice. Sampled reviews also have limits: a rare failure can slip between samples, which is why consequential actions keep a person in the loop.
Work this draws on
We run production AI products of our own, so the runbooks are ones we use:
- Singing Coach AI: an AI vocal coach on iPhone, Android and the web that scores recordings across 26 vocal dimensions and returns coaching in a minute or two.
- Idyllic: a consumer AI image generation and editing platform on web, iOS and Android.
- World of ClaudeCraft: our free, open-source browser MMO, live online since June 2026 with regular releases built from community contributions.
Sources
- API changelog, OpenAI, accessed 2026-10-19
- Model deprecations, Anthropic, accessed 2026-10-19
- Gemini Developer API pricing, Google, updated 2026-10-16
- Pricing and policy updates FAQ, Xero Developer, accessed 2026-10-19
- Notify us of a privacy breach, Office of the Privacy Commissioner, accessed 2026-10-19
- Privacy Act 2020, New Zealand Legislation, version as at 2026-10-01
Written by the Levy Street team in Wellington. Last updated 19 October 2026.