Versioning prompts and evaluations.

What has to be versioned

Version everything that changes the system’s behaviour, not just the prompt text. Two runs with the same prompt and a different model snapshot, temperature or retrieval index are different systems.

One manifest, one hash, logged on every call

Describe each call site’s configuration in a manifest kept in the repository, and compute a hash over the manifest and every file it references.

call_site: invoice_coding.classify_line
prompt: prompts/invoice_coding/classify_line.md
prompt_version: 3.2.0
model: provider-model-2026-09-08
params: {temperature: 0, max_output_tokens: 400}
tools: schemas/coding_tools.json
retrieval: {embedding_model: embed-2026-05, index_build: coa-index-2026-10-01, top_k: 8}
evaluation: {dataset: invoice-coding-eval-v7, scorers: coding-scorers-2.1.0}

import hashlib
import json
from pathlib import Path

def config_hash(manifest: dict, referenced_files: list[str]) -> str:
    digest = hashlib.sha256(json.dumps(manifest, sort_keys=True).encode("utf-8"))
    for path in sorted(referenced_files):
        digest.update(Path(path).read_bytes())
    return digest.hexdigest()[:12]

Write the hash into the trace of every production call. When someone asks why an answer changed between Tuesday and Thursday, you compare two hashes instead of reconstructing history from chat messages. Trace design is covered in AI audit logs.

Tracing a changed answer

Illustrative example: an accounts team reports that from Wednesday the system began coding courier invoices to Freight instead of Postage. With versioning in place, the investigation is a set of lookups rather than a debate.

1. Pull traces for the affected invoices and compare configuration hashes before and after Wednesday. The hash changed, and the manifest diff shows the index build moved from coa-index-2026-10-01 to coa-index-2026-10-08.
2. Diff the two index builds. A newly added chart-of-accounts note describes Freight in words that now outrank Postage for courier lines.
3. Run the regression suite against both builds. The courier cases flip, which confirms the cause.
4. Fix the source note, rebuild the index, add the courier cases to a new dataset version, and move the alias once the suite passes.

Had the hash not changed, the cause would lie outside your configuration: different inputs, or a change on the provider’s side. Knowing which it is narrows the search immediately.

Semantic versions for prompts

Give prompts version numbers that say what kind of change was made. Semantic Versioning defines MAJOR.MINOR.PATCH: increment MAJOR for incompatible API changes, MINOR for backward-compatible functionality, PATCH for backward-compatible fixes. Mapped to prompts:

The last row matters. With language models there is no such thing as a change that cannot alter behaviour, so even a PATCH runs the suite.

Where prompts should live

Keep the repository as the source of truth for engineering teams: prompts change through pull requests, evaluations run in CI, and deploys are tied to commits. Use a registry when you need runtime switching or non-engineer editing. MLflow’s Prompt Registry, for example, makes prompt versions immutable and lets an alias such as production point at a specific version.

The pattern that works in either case: immutable versions, environment aliases (staging, production) that point at versions, and a rule that an alias only moves after the evaluation run for that version has passed. Our lesson on building a reusable prompt library covers the same idea for teams using AI chat tools.

Versioning evaluations so results stay comparable

An evaluation score is only meaningful with three identifiers attached: the system configuration hash, the dataset version and the scorer version. Change any one and the score is a different measurement.

• Append-only datasets. A wrong gold label is fixed by publishing a new version with a changelog entry, never by editing in place.
• Scorer bundles. Scoring code, judge prompt and judge model are versioned together.
• Like with like. Dashboards compare runs only within the same dataset and scorer versions; a scorer upgrade means re-scoring old outputs, which is why raw outputs must be stored.
• No leakage. Few-shot examples in a prompt must not appear in the evaluation set. Check by content hash on every run.
• Protected holdouts. Whoever, or whatever, tunes the prompt must not see the test set.

That last rule matters more as optimisation is automated. In Tiny Shakespeare CLM, a research repository by Levy Street’s Jeremy Baron described on our code policy models page, an autonomous Claude agent improved a model by editing code, with improvements committed alongside their metric changes. A hook blocked the agent from reading the development and validation splits, and those evaluations were operator-only commands. The repository’s published bits-per-character figures are training-set numbers; held-out scores stay with the operator.

A reproducible experiment record

Every experiment that informs a release decision should be reproducible from its record.

from dataclasses import dataclass, field
from datetime import datetime, timezone

@dataclass(frozen=True)
class ExperimentRun:
    config_hash: str        # system under test
    dataset_version: str    # e.g. "invoice-coding-eval-v7"
    scorer_version: str     # e.g. "coding-scorers-2.1.0"
    repeats: int            # hosted models vary between runs, so repeat
    provider_region: str    # where inference ran
    outputs_uri: str        # raw outputs, kept for re-scoring
    started_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())

Run each configuration several times and report the spread, because hosted models do not always return identical outputs for identical inputs. Methods for building the suite itself are in AI agent evaluations.

Rollback that takes minutes

Rollback should be a configuration change: move the production alias back to the previous version. Three details decide whether that works under pressure.

1. Runtime resolution. The application resolves the alias at run time with a short cache, so a rollback does not need a deploy.
2. State compatibility. Conversations, memory and stored outputs created under the new version must still work under the old one. MAJOR changes need a data migration plan before release.
3. Rehearsal. Roll back in staging on a schedule, and time it. A rollback nobody has practised fails during the incident it was built for; see AI incident response.

Model changes are the largest kind of version change, covered step by step in AI model migration.

Where versioning goes wrong

If you want an independent check of how your prompts and evaluations are versioned and run, it is part of our LLM evaluation services.

• Prompts, model snapshots, parameters, tools, retrieval settings and scorers are all versioned
• Each call site has a manifest, and its configuration hash is logged on every production call
• Prompt versions follow MAJOR.MINOR.PATCH rules, and every change runs the regression suite
• Prompt versions are immutable; environment aliases move only after evaluations pass
• Evaluation datasets are append-only with changelogs; raw outputs are stored for re-scoring
• Results are compared only within the same dataset and scorer versions
• Few-shot examples are checked for overlap with evaluation cases
• Test sets are hidden from whoever, or whatever, tunes the prompt
• Rollback is an alias change resolved at run time, rehearsed on a schedule

Written by the Levy Street team in Wellington. Last updated 19 October 2026.