Human review queues.

What a human review queue is for

A human review queue has three jobs: stop costly errors before they take effect, measure the error rate of the work that skips review, and produce labelled examples for evaluation. Most teams build only the first and then cannot say whether the system is getting better or worse.

Every item an AI system produces should land in exactly one of three lanes:

• Auto: applied without a person, but eligible for random audit.
• Review: held until a named role approves, edits or rejects it.
• Block: refused outright, with the reason logged (failed validation, policy topic, missing data).

The lanes are a design decision, written down before launch, not something the model decides at run time. Our lesson on planning human review for AI output covers the business side; this guide covers the mechanics.

How to decide what goes to review

Route on consequence first and confidence second. An item that sends money, contacts a customer or deletes data goes to review regardless of how confident the model is, until you have audited evidence that the automatic path is safe for that action.

The same split applies to agents. Tool calls that write to systems of record belong behind an approval step, as described in our guide to agent permissions and approvals.

How to set confidence thresholds

Set thresholds on a labelled holdout set, not on the model’s self-reported certainty. Language models are often confident and wrong, and a “0.9” from a prompt that asks for a confidence score is not a probability. Use a score you can calibrate: a classifier probability, agreement between two independent extractions, a validator pass count, or log probabilities where the API exposes them.

Calibration means that items scored 0.9 are right about 90% of the time. Expected calibration error (ECE) summarises the gap: bucket predictions by score, then take the weighted average of |accuracy − mean score| across buckets. A 2020 paper co-authored by Levy Street’s Max Polaczuk on temporal probability calibration deals with the harder case where predictions are made on partial sequences.

Pick the threshold from the precision you need on the auto lane, with a minimum number of supporting examples so a lucky handful does not set policy:

def choose_threshold(scores, labels, target_precision=0.98, min_support=200):
    """Return the lowest score threshold whose auto-lane precision meets the target.

    scores: model scores for a labelled holdout set
    labels: 1 if the output was correct, else 0
    """
    pairs = sorted(zip(scores, labels), reverse=True)
    best = None
    correct = 0
    for n, (score, label) in enumerate(pairs, start=1):
        correct += label
        if n >= min_support and correct / n >= target_precision:
            best = score  # everything at or above this score goes to auto
    return best  # None means: no safe auto lane yet, review everything

Rerun this whenever the model, prompt or input mix changes. A threshold is only valid for the system it was measured on.

How large the audit sample needs to be

The audit sample size follows from how precisely you need to know the auto lane’s error rate. For an expected error rate p and margin of error e at 95% confidence, n ≈ 1.96² × p(1 − p) / e².

Worked example: you expect about 2% errors in auto-approved invoice codings and want to know the rate within ±1 percentage point. n = 3.8416 × 0.02 × 0.98 / 0.0001 ≈ 753 audited items. At 150 auto-approved invoices a week that is most of a quarter, so either accept a wider margin or audit a higher share early on.

Two shortcuts help. If you audit n items and find zero errors, the 95% upper bound on the error rate is roughly 3/n (the rule of three), so 300 clean audits supports “below 1%”. And stratify: sample more heavily from risky segments, such as new suppliers or high values, then weight results back by segment volume so the overall estimate stays honest.

Designing the reviewer’s screen

A reviewer can only be as good as the evidence in front of them. The screen should show the AI output next to its source, with the supporting span highlighted, so checking takes seconds rather than a search.

• Show evidence, not just answers. For extraction, the page region; for drafted replies, the customer’s message and the policy paragraph used.
• Make the decision explicit. No pre-ticked approve buttons. Approve, edit and reject are separate actions.
• Capture a reason code on every edit and rejection. A short fixed taxonomy (wrong field, wrong value, missing information, tone, policy) beats free text for analysis.
• Record time on item. It is the main signal for rubber-stamping and for estimating review cost.
• Double-review a sample blind. Two reviewers decide independently, and you report agreement as Cohen’s kappa, which corrects raw agreement for chance.

The Office of the Privacy Commissioner’s guidance on AI and the information privacy principles makes the same point from a New Zealand legal angle: where AI affects outcomes for people, simply having a “human in the loop” may not be enough to uphold the accuracy principle (IPP8 in the Privacy Act 2020), because of automation blindness.

Queue mechanics that hold up in production

Treat each review item as a small state machine with leases, so two reviewers never decide the same item and a closed browser tab never strands work.

from dataclasses import dataclass
from datetime import datetime, timedelta

STATES = {"pending", "claimed", "approved", "edited", "rejected", "applied", "expired"}

@dataclass
class ReviewItem:
    id: str
    lane_reason: str          # e.g. "low_confidence", "audit_sample", "value_limit"
    state: str = "pending"
    claimed_by: str | None = None
    lease_until: datetime | None = None

    def claim(self, reviewer: str, now: datetime, lease=timedelta(minutes=15)):
        expired = self.state == "claimed" and self.lease_until and self.lease_until < now
        if self.state != "pending" and not expired:
            raise ValueError("item is not claimable")
        self.state, self.claimed_by, self.lease_until = "claimed", reviewer, now + lease

Three rules matter more than the code. Decisions are written once and applied idempotently, so a retry cannot pay an invoice twice. Every item has an ageing deadline with an escalation owner. And when the queue backs up, items wait or expire to a safe default; they never fall through to auto-approval. A system that quietly approves its backlog on a busy Friday has no review at all.

Where review queues go wrong

Most failures are invisible until someone measures them. These are the ones we design against when we run systems under managed AI operations.

The override rate (edits plus rejections divided by items reviewed) is the single most useful weekly number. Rising overrides mean the model got worse or the inputs changed; overrides near zero on a busy queue usually mean nobody is really looking.

Turning decisions into better evaluations

Every approve, edit and reject is a labelled example. Store the input, the AI output, the final human version, the reason code and the system version together, and you have a growing evaluation set drawn from real traffic. Feed disagreements into your evaluation suite, retune thresholds from the audit sample, and watch the trend on a quality dashboard as described in monitoring AI quality. The wider loop, including end-user feedback, is covered in designing AI product feedback loops.

This is how the property management agent for APL Property is framed: the agent handles routine administration and staff keep the decisions. The queue is where that division of labour becomes enforceable.

• Every output type is assigned to auto, review or block, with the rule written down
• Irreversible and external actions go to review until the auto lane has an audited error rate
• Thresholds are set on a labelled holdout with a minimum support count, and recomputed on every release
• A random audit sample of auto-approved work is reviewed, sized from the error margin you need
• Reviewers see source evidence beside the output and cannot approve with one pre-filled click
• Edits and rejections carry a reason code from a fixed taxonomy
• Canary items and blind double reviews measure reviewer catch rate and agreement
• Items claimed by a reviewer have leases; decisions apply idempotently
• A full queue holds or expires work to a safe default and pages an owner
• Override rate, audited error rate and queue age are reported weekly

Written by the Levy Street team in Wellington. Last updated 19 October 2026.