Code Policy Models.

What a Code Policy Model is

A Code Policy Model is a policy, meaning the function that maps a state to an action or a probability distribution, written as ordinary source code and improved by editing that code. Nothing is learned in a matrix of weights. The program is the model.

The idea comes from a simple observation. Modern AI coding models can read a codebase, form a hypothesis about why it behaves badly, and write a targeted change. If you pair that ability with an automatic scorer, you get an optimiser whose steps are code commits instead of gradient updates. The Tiny Shakespeare CLM repository, written by Levy Street’s Jeremy Baron, puts it directly: “Optimization happens by changing code instead of updating learned weights”, and “the codebase itself is the parameter space.”

The repository calls the technique Code Policy Optimization. We use “Code Policy Models” for the research direction as a whole.

How the optimisation loop works

The loop is evolutionary: propose, evaluate, select, repeat. Each turn of it produces either a kept improvement or a discarded attempt.

1. Fix the interface and the scorer. Define the policy’s inputs and outputs, and an evaluation that returns a number: game rollouts, a prediction loss, or a task success rate.
2. Propose an edit. A language model reads the current policy, its scores and sample behaviour, then writes a change: a new rule, a new state variable, a reweighted heuristic.
3. Evaluate the candidate. Run the scorer on the edited policy. Held-out data stays out of the optimiser’s reach.
4. Select. Keep the candidate only if it beats the incumbent, and commit it with its score in the message. Otherwise discard it.
5. Repeat with pressure. Feed results back into the next proposal, and occasionally nudge the optimiser towards structural changes rather than small tweaks.

In pseudo-code, the simplest version is a hill climb with one candidate at a time:

def optimise(repo, propose_edit, evaluate, budget):
    best = evaluate(repo.head())
    for _ in range(budget):
        candidate = propose_edit(repo.head())  # an LLM writes a code change
        score = evaluate(candidate)            # training data or rollouts only
        if score > best:                       # higher is better here
            repo.commit(candidate, score)      # every gain is a readable diff
            best = score
    return repo.head()

Population-based variants keep several candidates alive and recombine them, which helps escape local optima at the cost of more evaluation.

Where the idea comes from

Code Policy Models build on a clear line of published work; we did not invent the idea that language models can write policies or that code can be evolved.

The question we are testing is narrower: what happens when the whole policy, not one function or one skill, is code that an agent keeps rewriting for hundreds of commits?

Why readable policies matter for production systems

The appeal is not raw performance. It is that a code policy can be governed like software.

For regulated or high-stakes decisions, “we can show you the exact rule that fired” is a stronger position than “the model learned it”. The same reasoning drives our advice on versioning prompts and evaluations: if you cannot diff it, you cannot govern it.

Public evidence: Tiny Shakespeare CLM

Our public evidence is one repository. Tiny Shakespeare CLM is a character-level language model of Shakespeare with no neural network. An autonomous Claude Opus 4.7 session, run at maximum effort through the Claude Agent SDK, edited the model’s Python and committed each gain.

Two details matter for anyone copying the method. First, a PreToolUse hook blocked the optimiser from reading the development and validation splits, so it could not tune against them. Second, the published figures are training-corpus scores only. No held-out score has been released, so the repository shows that the loop makes steady progress on its objective, not how well the result generalises.

Game-playing agents: ongoing internal work

Levy Street is also applying the loop to game-playing agents: a policy written as code that plays Pokémon Blue on a headless Game Boy emulator, with rollouts in the emulator taking the place of a text corpus as the evaluator. This is ongoing internal work. It has not been published, and we claim no results from it.

Games suit the method because they offer long horizons, unambiguous scores and exact replays. Deterministic simulators help most, because a candidate can be re-run with the same seed and compared fairly. We built World of ClaudeCraft with that property: its game core runs as a Gymnasium environment that is, in the README’s words, “Deterministic by construction”.

Trade-offs and where it goes wrong

Code Policy Models fail in predictable ways. Most of them are evaluation problems, not coding problems.

• Overfitting to the scorer. The optimiser will exploit whatever the score rewards. A 60,878-line model tuned on training text can encode corpus-specific patterns that would not transfer. Guard held-out data, as the repository’s hook does, and publish held-out results before claiming generalisation.
• Evaluation cost. Every candidate needs a full scoring run. Slow rollouts limit how many ideas can be tested.
• Code bloat. The Shakespeare model grew 219 times larger. Readable in principle is not the same as readable in practice; plan pruning and refactoring passes.
• Local optima. A single-candidate hill climb accepts only immediate gains, so improvements that need a temporary step backwards are missed.
• Tampering with the judge. An agent with write access could edit the evaluation instead of the policy. Keep the scorer outside the editable directory and check diffs.
• Noisy scores. If evaluation is not deterministic, small “gains” can be luck. Fix seeds and repeat runs.

• The evaluation returns one number and runs fast enough for hundreds of trials
• Held-out data is technically unreachable by the optimiser, not just discouraged
• The scorer lives outside the directory the agent can edit
• Every commit records its score so regressions can be traced
• A person reviews the policy’s size and structure at regular intervals

When it makes sense in business systems

Use the idea when a decision is narrow, a trustworthy score exists, and someone must be able to explain each outcome: routing rules, eligibility checks, scheduling heuristics, game or simulation agents. Avoid it when the input is raw perception, when there is no agreed measure of “better”, or when a conventional model plus good agent evaluations would do the job with less risk.

For most organisations the transferable lesson is the discipline, not the technique: score every change, protect your test data, and keep every decision traceable to a version. That is how we approach custom AI development for clients, and it is why our coding-agent work leans on tools like Claude Code inside strict guardrails.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.