# Monitoring AI quality.

URL: https://levystreet.com/engineering/monitoring-ai-quality/
Updated: 2026-10-19

AI observability beyond uptime: a quality drift dashboard with metric definitions, sampled grading, judge calibration and alerts that respect sample size.

## Short answer

AI observability means tracking whether an AI system's outputs stay good in production, not only whether it is up and fast. Sample live runs for automated and human grading, track groundedness, refusals, escalations, tool errors and cost per successful task alongside latency, alert on sustained changes only when the sample is large enough, and slice every metric by the prompt, model and index version that produced it.

- A 200 OK says nothing about whether the answer was right. Grade samples of real runs.
- Slice every metric by prompt, model and index version, and by customer segment.
- Calibrate LLM judges against human grades every week and pin the judge model.
- Alert on confidence bounds with a minimum sample size, not on raw daily rates.
- Page people for harm signals; open tickets for gradual drift.

## What does AI quality monitoring cover that ordinary monitoring misses?

AI quality monitoring measures whether the outputs are still good, which infrastructure monitoring cannot see. A model that returns a fluent, wrong answer in 800 milliseconds looks perfect on an uptime dashboard. The NIST [AI RMF core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) makes the requirement explicit: the functionality and behaviour of an AI system should be monitored in production (Measure 2.4), with post-deployment monitoring plans that capture input from users, appeals and overrides, and incident response (Manage 4.1).

In practice that means three layers on top of normal service monitoring: grading a sample of real outputs, tracking behavioural signals that correlate with quality, and attributing every change to a version or a segment.

## Which panels belong on a quality drift dashboard?

Each panel needs a written definition, a denominator and a version slice, or nobody will trust it during an incident. These are the panels we put on the dashboards for systems we run.

| Panel | Definition | What a change usually means |
|---|---|---|
| Graded pass rate | Sampled runs passing the rubric, divided by sampled runs graded | Overall quality shift; drill into segments |
| Judge agreement | Double-graded runs where the automated judge and a person agree, divided by double-graded runs | Whether the pass rate can be trusted |
| Citation validity | Citations that support the sentence they are attached to, divided by citations checked | Retrieval or generation drift in grounded answers |
| Refusal and fallback rate | Runs ending in refusal, "I don't know" or a fallback path, divided by all runs | Over-cautious prompt, missing documents, or new question types |
| Escalation and override rate | Runs escalated to a person, and approvals rejected or edited, divided by runs | Agent struggling, or reviewers losing trust |
| Tool error rate | Failed tool calls divided by tool calls, per tool | Upstream API change or argument drift |
| Cost per successful task | Total model and tool spend divided by runs that passed | Longer outputs, retries, or more loops per task |
| Latency p50 and p95 | End-to-end and time to first token | Provider load, longer prompts, extra steps |
| Input mix | Share of traffic per intent or document type against a baseline period | New use, new customers, seasonal change |
| Index freshness | Time between a source document changing and its re-indexing | Answers grounded in outdated content |

Use consistent field names so the dashboard can be built from your traces; the [OpenTelemetry GenAI conventions](https://github.com/open-telemetry/semantic-conventions-genai) cover model, token and tool attributes. The trace design itself is in [AI audit logs](/engineering/ai-audit-logs/), and latency and cost panels connect to [reducing AI latency](/engineering/reducing-ai-latency/) and [managing LLM costs](/engineering/managing-llm-costs/).

## How do you grade production outputs?

Grade a stratified sample every day with the same rubrics you use before release, and send a smaller subset to people.


1. **Sample on purpose.** Take a random sample for the unbiased rate, plus every run a user flagged, every escalation and every run a guard rail touched. Report the random sample and the targeted samples separately.
2. **Check what code can check.** Schema validity, citation existence, policy rules and tool outcomes need no judge.
3. **Judge the rest with a rubric.** Use the rubric and judge prompt from your [evaluation suite](/engineering/ai-agent-evaluations/), pinned to a judge model version.
4. **Double-grade a subset.** People grade a fixed share of the judged runs blind. Track agreement weekly.
5. **Feed failures back.** Confirmed failures become regression cases in the evaluation suite, with the trace attached.


Calibrate the judge before relying on it and keep calibrating. [Zheng et al.](https://arxiv.org/abs/2306.05685) documented position, verbosity and self-enhancement biases in LLM judges. If agreement with people falls, stop trusting the pass rate until you know why.

## How should AI quality alerts be designed?

Alert on statistically meaningful changes, with a minimum sample size, and route by severity. A raw daily pass rate on 30 graded runs will swing by ten points on noise alone, and alerts that fire on noise get muted.

Two patterns work well together. For gradual drift, compare a confidence bound with your target: if even the optimistic end of the interval is below target, quality has really dropped. For example, 168 passes from 200 graded runs is 84%, with a 95% Wilson interval of roughly 78% to 88%, so a 90% target is breached with confidence. For error budgets, borrow the burn-rate approach from the Google [SRE workbook](https://sre.google/workbook/alerting-on-slos/): define a quality objective over a period, and alert when the rate of failures would exhaust the budget early, using a short and a long window together to catch both spikes and slow leaks.

```python
from collections import deque
from math import sqrt

def wilson_bounds(passes: int, n: int, z: float = 1.96):
    if n == 0:
        return 0.0, 1.0
    p = passes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return centre - half, centre + half

class QualityMonitor:
    def __init__(self, target: float = 0.90, window: int = 200, min_n: int = 50):
        self.target, self.min_n = target, min_n
        self.results = deque(maxlen=window)        # rolling window of graded runs

    def add(self, passed: bool) -> None:
        self.results.append(passed)

    def status(self) -> str:
        n = len(self.results)
        if n < self.min_n:
            return "insufficient_data"
        low, high = wilson_bounds(sum(self.results), n)
        if high < self.target:
            return "alert"                         # confidently below target
        if low < self.target:
            return "watch"                         # cannot rule out a breach yet
        return "ok"
```

Run one monitor per important segment and version, not one for the whole system: an overall rate can stay flat while one customer's quality collapses.

| Signal | Route | Response time |
|---|---|---|
| Canary or personal information in output, unapproved side effect | Page on call | Immediately; consider disabling the tool |
| Grounding failure across a segment, judge agreement collapse | Page in working hours | Same day |
| Pass rate confidently below target, cost per task rising | Ticket with evidence | Next working day |
| Input mix shift, index lag growing | Weekly review | Planned work |

## Why does quality drift when nothing was deployed?

Most drift comes from dependencies outside your release process. The table lists the causes we check first.

| Cause | Symptom | How to detect it |
|---|---|---|
| Provider changes a model behind an alias | Step change with no deploy | Pin versions; run a fixed canary set daily and compare |
| Model retirement forces a switch | Behaviour change on a known date | Track provider deprecation pages; OpenAI lists several legacy models shutting down on 23 October 2026, as listed on 19 October 2026 |
| Stale retrieval index | Faithful answers citing superseded documents | Index freshness panel; spot checks against source systems |
| Prompt edited outside the release process | Step change tied to an edit | Version every prompt and show the version on every panel |
| Upstream format change | Rising nulls or tool errors on one field | Per-field extraction completeness and tool error panels |
| New kinds of request | Refusals and escalations rise in one intent | Input mix against baseline; review unclassified inputs |

Planned changes deserve the same treatment: run the full evaluation suite before switching models, as covered in [AI model migration](/engineering/ai-model-migration/), and watch the dashboard closely for the first days after.

## Where does AI quality monitoring go wrong?

The common failures are organisational as much as technical.

- **Monitoring only latency and errors.** The system degrades in silence until a customer complains.
- **Trusting an uncalibrated judge.** The pass rate becomes a number about the judge, not the system.
- **One global metric.** Segment-level failures disappear into an average.
- **No owner.** Dashboards nobody reviews weekly are decoration. Name a person and a meeting.
- **No path to action.** An alert should link to the runbook, the traces and the [incident response](/engineering/ai-incident-response/) process.

A weekly review against the [AI quality review checklist](/resources/ai-quality-review-checklist/) keeps the habit going. If you would rather not staff this yourself, quality monitoring, judge calibration and incident handling are the core of our [managed AI operations](/services/managed-ai-operations/) service.


- Every panel has a written definition, denominator and version slice
- A random sample of production runs is graded every day
- Flagged runs, escalations and guard-rail events are graded separately
- The judge model is pinned and judge-human agreement is tracked weekly
- Alerts use confidence bounds and a minimum sample size
- Monitors run per important segment, not just for the whole system
- Harm signals page someone; drift opens a ticket with evidence
- A fixed canary set runs daily to separate provider changes from traffic changes
- Index freshness and input mix are on the dashboard
- Confirmed production failures are added to the evaluation suite

## Questions people ask

### What is the difference between AI observability and AI evaluation?

Evaluation runs a fixed set of test cases before release to decide whether a change is good enough to ship. Observability watches real production traffic afterwards to see whether quality holds as inputs, data and providers change. You need both, and they feed each other: failures found in production become new evaluation cases, and evaluation rubrics are reused to grade production samples.

### How many production runs should we grade?

Enough for the confidence interval to be narrower than the change you care about. Fifty graded runs a day can only detect large shifts; a few hundred a week per important segment detects moderate ones. Grade every run that users flag, every escalation and a random sample on top, and have people grade a smaller subset so you can check the automated judge.

### Can we rely on an LLM judge to monitor quality?

Only after you have measured how often it agrees with people on your task, and only while you keep measuring. Judges have known biases, such as favouring longer answers, and they change when their model is updated. Double-grade a sample every week, investigate disagreements, pin the judge model version and recalibrate whenever the rubric or judge changes.

### Which AI quality signals should wake someone up at night?

Only signals that indicate harm in progress: a canary string or personal information appearing in outputs, a tool performing an action that should have needed approval, a sudden spike in a safety classifier, or a complete failure of grounding. Gradual drift in pass rates, costs or refusal rates should open a ticket for the next working day, with the evidence attached.

### Why does AI quality change when we have not changed anything?

Because the system depends on things outside your code. Providers update or retire models, documents in the retrieval index go stale, upstream systems change formats, and users start asking about new things. Pin model versions where you can, monitor index freshness and input mix, and keep a fixed daily canary set of known questions to separate provider changes from traffic changes.


## Sources

- [AI RMF Core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/)
- [Alerting on SLOs](https://sre.google/workbook/alerting-on-slos/)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)
- [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai)
- [Deprecations](https://developers.openai.com/api/docs/deprecations)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.