Monitoring AI quality.

What does AI quality monitoring cover that ordinary monitoring misses?

AI quality monitoring measures whether the outputs are still good, which infrastructure monitoring cannot see. A model that returns a fluent, wrong answer in 800 milliseconds looks perfect on an uptime dashboard. The NIST AI RMF core makes the requirement explicit: the functionality and behaviour of an AI system should be monitored in production (Measure 2.4), with post-deployment monitoring plans that capture input from users, appeals and overrides, and incident response (Manage 4.1).

In practice that means three layers on top of normal service monitoring: grading a sample of real outputs, tracking behavioural signals that correlate with quality, and attributing every change to a version or a segment.

Which panels belong on a quality drift dashboard?

Each panel needs a written definition, a denominator and a version slice, or nobody will trust it during an incident. These are the panels we put on the dashboards for systems we run.

Use consistent field names so the dashboard can be built from your traces; the OpenTelemetry GenAI conventions cover model, token and tool attributes. The trace design itself is in AI audit logs, and latency and cost panels connect to reducing AI latency and managing LLM costs.

How do you grade production outputs?

Grade a stratified sample every day with the same rubrics you use before release, and send a smaller subset to people.

1. Sample on purpose. Take a random sample for the unbiased rate, plus every run a user flagged, every escalation and every run a guard rail touched. Report the random sample and the targeted samples separately.
2. Check what code can check. Schema validity, citation existence, policy rules and tool outcomes need no judge.
3. Judge the rest with a rubric. Use the rubric and judge prompt from your evaluation suite, pinned to a judge model version.
4. Double-grade a subset. People grade a fixed share of the judged runs blind. Track agreement weekly.
5. Feed failures back. Confirmed failures become regression cases in the evaluation suite, with the trace attached.

Calibrate the judge before relying on it and keep calibrating. Zheng et al. documented position, verbosity and self-enhancement biases in LLM judges. If agreement with people falls, stop trusting the pass rate until you know why.

How should AI quality alerts be designed?

Alert on statistically meaningful changes, with a minimum sample size, and route by severity. A raw daily pass rate on 30 graded runs will swing by ten points on noise alone, and alerts that fire on noise get muted.

Two patterns work well together. For gradual drift, compare a confidence bound with your target: if even the optimistic end of the interval is below target, quality has really dropped. For example, 168 passes from 200 graded runs is 84%, with a 95% Wilson interval of roughly 78% to 88%, so a 90% target is breached with confidence. For error budgets, borrow the burn-rate approach from the Google SRE workbook: define a quality objective over a period, and alert when the rate of failures would exhaust the budget early, using a short and a long window together to catch both spikes and slow leaks.

from collections import deque
from math import sqrt

def wilson_bounds(passes: int, n: int, z: float = 1.96):
    if n == 0:
        return 0.0, 1.0
    p = passes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return centre - half, centre + half

class QualityMonitor:
    def __init__(self, target: float = 0.90, window: int = 200, min_n: int = 50):
        self.target, self.min_n = target, min_n
        self.results = deque(maxlen=window)        # rolling window of graded runs

    def add(self, passed: bool) -> None:
        self.results.append(passed)

    def status(self) -> str:
        n = len(self.results)
        if n < self.min_n:
            return "insufficient_data"
        low, high = wilson_bounds(sum(self.results), n)
        if high < self.target:
            return "alert"                         # confidently below target
        if low < self.target:
            return "watch"                         # cannot rule out a breach yet
        return "ok"

Run one monitor per important segment and version, not one for the whole system: an overall rate can stay flat while one customer’s quality collapses.

Why does quality drift when nothing was deployed?

Most drift comes from dependencies outside your release process. The table lists the causes we check first.

Planned changes deserve the same treatment: run the full evaluation suite before switching models, as covered in AI model migration, and watch the dashboard closely for the first days after.

Where does AI quality monitoring go wrong?

The common failures are organisational as much as technical.

• Monitoring only latency and errors. The system degrades in silence until a customer complains.
• Trusting an uncalibrated judge. The pass rate becomes a number about the judge, not the system.
• One global metric. Segment-level failures disappear into an average.
• No owner. Dashboards nobody reviews weekly are decoration. Name a person and a meeting.
• No path to action. An alert should link to the runbook, the traces and the incident response process.

A weekly review against the AI quality review checklist keeps the habit going. If you would rather not staff this yourself, quality monitoring, judge calibration and incident handling are the core of our managed AI operations service.

• Every panel has a written definition, denominator and version slice
• A random sample of production runs is graded every day
• Flagged runs, escalations and guard-rail events are graded separately
• The judge model is pinned and judge-human agreement is tracked weekly
• Alerts use confidence bounds and a minimum sample size
• Monitors run per important segment, not just for the whole system
• Harm signals page someone; drift opens a ticket with evidence
• A fixed canary set runs daily to separate provider changes from traffic changes
• Index freshness and input mix are on the dashboard
• Confirmed production failures are added to the evaluation suite

Written by the Levy Street team in Wellington. Last updated 19 October 2026.