Testing AI with synthetic data.

Where synthetic test data earns its place

Synthetic test data is worth generating when real examples are too rare, too sensitive or do not exist yet. It is not worth trusting as an estimate of production accuracy. Keep those two uses apart and it is one of the most useful tools in an evaluation programme.

Attack generation has its own techniques, covered in prompt injection testing.

Why synthetic results flatter your system

Generated inputs differ from real ones in systematic ways, and almost all of them make the system look better than it is.

• Too clean. Real customer emails arrive with forwarded chains, signatures, disclaimers, typos, photos of documents and two requests in one message. Generated ones usually do not.
• Answers written into the question. A generator asked for “a customer who wants to cancel” writes “I want to cancel my booking”. Real customers write “can’t make it Friday, sorry”.
• Shared blind spots. A model tends to find text from its own family easy to read. Research on LLM judges reported models favouring their own outputs, though the authors could not confirm a self-enhancement bias. Generating tests with the same model family you are testing invites the same effect.
• Missing tails. Generators produce typical examples. The strange, rare inputs that cause production incidents are the ones they leave out. Training on generated data shows the extreme version: the Nature paper on model collapse found that indiscriminate training on model-generated content makes the tails of the original distribution disappear.
• Wrong mix. The proportions of intents, channels and difficulty in a synthetic set reflect the generator’s spec, not your traffic.

Generate along explicit dimensions

Specify what varies, then generate to fill every combination. A spec makes coverage measurable and the set reproducible.

import itertools
import random

DIMENSIONS = {
    "intent": ["change_booking", "cancel", "complaint", "invoice_query"],
    "difficulty": ["plain", "ambiguous", "two_requests_in_one"],
    "channel": ["email", "web_form", "sms"],
    "noise": ["none", "typos", "forwarded_chain", "signature_and_disclaimer"],
}

def case_specs(per_cell: int = 2, seed: int = 42):
    rng = random.Random(seed)
    for combo in itertools.product(*DIMENSIONS.values()):
        for i in range(per_cell):
            yield {**dict(zip(DIMENSIONS, combo)),
                   "case_id": "-".join(combo) + f"-{i}",
                   "seed": rng.randrange(2**31)}

def add_typos(text: str, rate: float, rng: random.Random) -> str:
    chars = list(text)
    for i in range(len(chars) - 1):
        if chars[i].isalpha() and rng.random() < rate:
            chars[i], chars[i + 1] = chars[i + 1], chars[i]   # swap neighbours
    return "".join(chars)

Four intents, three difficulties, three channels and four noise types give 144 cells; two cases per cell gives 288 tests. Include near-miss negatives in the spec as well, such as a message that mentions cancelling but is really a change request, because those are where classifiers and routers fail. Then make them realistic:

1. Start from real structure. Base the taxonomy and templates on a reviewed sample of real, anonymised cases, not on what you imagine customers write.
2. Add local texture. New Zealand specifics belong in the spec: day-first dates, GST-inclusive and exclusive amounts, te reo Māori place names with and without macrons, local phone formats.
3. Use a different generator. Generate with a different model family from the one under test.
4. Keep labels separate. Record the intended label in the spec, never only in the generated text, and check that the text does not state the label outright.
5. Review a sample. A person reads a random share of generated cases for realism and label correctness before the set is published.
6. Version the set. Store the spec, seeds and generator version, so the set can be regenerated exactly; see versioning prompts and evaluations.

Measure the gap against a real holdout

Every synthetic suite needs a real held-out set beside it, drawn from production or a pilot, anonymised where it can be and access-controlled where it cannot. Score both with the same scorers and compare per category.

def realism_gap(real_acc: dict[str, float], synthetic_acc: dict[str, float]) -> dict[str, float]:
    """Positive values mean the synthetic cases are easier than real ones."""
    return {cat: round(synthetic_acc[cat] - real_acc[cat], 3)
            for cat in real_acc if cat in synthetic_acc}

Use the result to decide what synthetic data is allowed to tell you. Where the gap is small, the synthetic suite is a reasonable regression signal for that category. Where it is large, the synthetic cases are too easy: rewrite them using real failures as templates, and never quote their scores as expected accuracy.

A second check is blind discrimination. Mix real and synthetic cases and ask reviewers which is which. If they can tell easily, the realism problem is visible before you have scored anything.

The real holdout has to come from somewhere. Our guides to AI agent evaluations and AI data preparation cover building and splitting it, and voice systems need special care because generated speech understates accents and background noise; see voice agent quality evaluation.

Test against the real system where you can

Sometimes the better move is to avoid simulating at all. World of ClaudeCraft, Levy Street’s open-source browser MMO, runs the same deterministic game core as a Gymnasium environment, so reinforcement-learning agents train against the actual game rather than a reimplementation of it. The business equivalent is a sandbox copy of the real system (a test Xero organisation, a staging CRM) with synthetic records inside it. Your inputs are invented, but the system’s behaviour is real, which removes a whole class of simulation gaps.

Privacy: synthetic is not automatically anonymous

Data invented from scratch carries little privacy risk. Data generated from real records is different. Generators, including tabular tools such as the Synthetic Data Vault (check its Business Source License before commercial use), can reproduce real records or rare combinations that identify someone. The Privacy Act 2020 applies to information about identifiable individuals, whatever label is on the file.

Before treating a generated set as anonymous, compare each synthetic record with its nearest real record and remove exact and near copies. And do not paste real customer records into a hosted generator under weaker controls than the production system that normally holds them.

Where synthetic testing goes wrong

For a non-technical starting point, see our lesson on testing a business AI assistant. Independent evaluation design is part of our LLM evaluation services.

• Synthetic data is used for coverage, attacks, privacy-safe development and regression, never for launch estimates
• The generation spec lists dimensions, values and cases per cell, with seeds and generator version recorded
• The taxonomy and templates come from reviewed real cases, with New Zealand formats and names included
• Tests are generated by a different model family from the system under test
• Labels live in the spec and are checked for leakage into the generated text
• A person reviews a random sample of generated cases before publication
• A real, access-controlled holdout set is scored alongside every synthetic run
• The synthetic-to-real gap is reported per category, and large-gap categories are rewritten
• Synthetic records derived from real data are checked for copies before being treated as anonymous

Written by the Levy Street team in Wellington. Last updated 19 October 2026.