AI incident response.

What counts as an AI incident

An AI incident is any event where an AI system causes, or nearly causes, harm or departs from its intended behaviour. Most happen while the service is up and returning 200s, which is why uptime monitoring alone misses them.

Generic incident guidance still applies. NIST’s SP 800-61 Rev. 3, published April 2025, frames incident response within the Cybersecurity Framework 2.0. This guide adds what is specific to AI systems.

Severity levels with AI criteria

Define severity before the incident, in terms an on-call engineer can apply at 2am. When unsure between two levels, PagerDuty’s incident response guidance says to treat it as the higher one, and that holds here.

Prepare before anything breaks

Three things must exist before the first incident: graded kill switches, retained evidence and named owners.

Kill switches let on-call staff reduce what the system can do in seconds, without a deploy. Read the setting at run time, cache it briefly, and fail to the safe mode if it is missing.

import time

class KillSwitch:
    """Modes: 'normal', 'drafts_only' (no external effects), 'tools_off', 'off'."""

    def __init__(self, store, ttl_s: float = 10.0):
        self.store, self.ttl_s, self.cache = store, ttl_s, {}

    def mode(self, feature: str) -> str:
        value, fetched_at = self.cache.get(feature, (None, 0.0))
        if value is None or time.monotonic() - fetched_at > self.ttl_s:
            value = self.store.get(f"ai_mode:{feature}", "drafts_only")  # safe default
            self.cache[feature] = (value, time.monotonic())
        return value

def run_action(action, switch: KillSwitch):
    mode = switch.mode(action.feature)
    if mode == "off" or (mode == "tools_off" and action.is_tool_call):
        raise RuntimeError(f"{action.feature} is disabled by kill switch")
    if mode == "drafts_only" and action.has_external_effect:
        return action.save_as_draft()   # a person approves it later
    return action.run()

Evidence means traces that hold the input, retrieved documents, model and prompt versions, configuration hash, tool calls and output for every production call, retained long enough to scope an incident found weeks later. The trace design is in AI audit logs. Owners means an on-call rota, a privacy officer who can be reached, and a record template such as our AI incident log template. The business side, including how staff report AI mistakes, is covered in our lesson on managing AI mistakes and escalation.

The runbook

1. Detect and declare. Anyone can declare. Name an incident lead, set a severity and open a timeline.
2. Contain. Use the smallest kill switch that stops the harm: drafts only, tools off or feature off. Revoke credentials if an agent or key is compromised.
3. Preserve evidence. Snapshot traces, retrieval logs, configuration hashes, prompt and model versions and provider status for the window before changing anything.
4. Scope. Query traces to list every affected output, user and record. Do not estimate from a sample.
5. Fix the cause. Correct the prompt, filter, permission or data, and add a regression test that fails on the old behaviour.
6. Recover in stages. Re-enable through the same gates as a release: offline evaluation, then a small share of traffic, then full.
7. Notify. Customers, partners and regulators as required. For personal information, assess serious harm and notify the Privacy Commissioner where required.
8. Review. Hold a blameless review within a week and track its actions to completion.

Scoping is where AI incidents differ most. For a retrieval leak, the question is every answer that used a document its reader could not access, judged against permissions as they stood at the time:

SELECT t.trace_id, t.user_id, r.doc_id, t.created_at
FROM traces AS t
JOIN retrievals AS r ON r.trace_id = t.trace_id
LEFT JOIN access_snapshot AS a
  ON a.doc_id = r.doc_id AND a.principal_id = t.user_id
WHERE t.created_at BETWEEN '2026-10-14 09:40' AND '2026-10-14 15:05'
  AND a.doc_id IS NULL
ORDER BY t.created_at;

An illustrative scenario

Illustrative scenario, not a real incident: a customer support assistant answers questions about each customer’s own account.

The permission design that would have prevented this is covered in building private AI search.

Notification and New Zealand obligations

Under the Privacy Act 2020, a privacy breach that has caused or is likely to cause serious harm must be notified to the Office of the Privacy Commissioner and affected individuals. The Commissioner’s NotifyUs guidance says to do so ideally within 72 hours of becoming aware of a notifiable breach, even while still investigating. If a vendor processes the data for you, the clock starts when they know, so contracts should require prompt notice of every breach.

Regulated entities have more to consider. In October 2026 the Reserve Bank and the Financial Markets Authority set out supervisory expectations on cyber risks from frontier AI, stressing that boards and senior management need to know where AI is used and who is accountable. Take legal advice on what a specific incident triggers.

The review: blameless, specific, tested

Google’s SRE book describes blameless postmortems as focusing on contributing causes without indicting individuals or teams. For AI incidents, the review should also record:

• What changed: configuration hash before and after, model snapshot, prompt version, index build, provider status. This is only possible with versioned prompts and evaluations.
• How it was detected: monitoring, review queue, test or customer. Customer-detected incidents get a detection action.
• Times: to detect, to contain, to scope and to recover.
• Which evaluation would have caught it, and the regression case now added to the suite.
• Near misses that came before it. The AI Incident Database is also worth searching for incidents in your domain.

Where incident response goes wrong

Monitoring that detects these incidents early is covered in monitoring AI quality, and injection-driven incidents in prompt injection testing. On-call and incident response are part of managed AI operations for the systems we run.

• AI incident types and SEV1 to SEV4 criteria are written down and known to on-call staff
• Graded kill switches exist per feature, are read at run time and default to a safe mode
• Traces hold inputs, retrieved documents, versions, configuration hash, tool calls and outputs
• Trace retention covers the time it might take to discover an incident
• A scoping query exists for each major failure type, including permission-at-the-time checks
• The privacy officer can be reached, and vendor contracts require prompt breach notice
• Notifiable breach assessment and the 72-hour expectation are in the runbook
• Recovery uses the same staged gates as a release
• Every review adds a regression test and records detection source and times
• Near misses from review queues are reviewed weekly

Written by the Levy Street team in Wellington. Last updated 19 October 2026.