Tiny Shakespeare CLM.

What Tiny Shakespeare CLM is

Tiny Shakespeare CLM is a language model of Shakespeare-style text with no neural network inside it. In the repository’s words: “This project does not train a neural network. The model’s policy, the function that maps a rolling character state to a next-character probability distribution, is written directly in Python.”

The model exposes two functions. predict(state) returns natural log-probabilities for every character in the vocabulary, and advance(state, token_id) returns a new, immutable state after reading one character. The state is a frozen Pydantic object, and updates run through a pipeline of small stages: word-shape rules, syntactic heuristics, semantic trackers and probability-bias layers. By the end, the model package held 291 files.

The project was written by Levy Street’s Jeremy Baron and lives in the Levy Street GitHub organisation. CLM in the name stands for character-level language model; we file it under code language models because the entire model is code. It is the public evidence for our wider research on Code Policy Models.

Bits per character, explained simply

Bits per character (BPC) measures how surprised a language model is by real text. For each character, the model gives a probability to what actually comes next; the cost is the base-2 logarithm of one over that probability. BPC is the average cost across the whole text. Lower is better.

A useful way to read a score is to raise 2 to its power, which gives the number of equally likely options the model is effectively choosing between. At 4.7744 bits, the starting model behaved as if picking from about 27 equally likely characters. At 2.3100 bits, it behaves as if picking from about 5. That is a large gain in how well it anticipates Shakespeare, measured on text it was optimised against.

The verified results

Every figure below comes from the repository’s README, its training chart, its optimiser script or the GitHub API, read on 19 October 2026.

The optimiser is a single script. Its docstring says it “Spawns one Claude Opus 4.7 session (max effort, unlimited turns)” and that “A PreToolUse hook blocks access to dev/val splits.” An excerpt of its configuration, with some lines omitted:

options = ClaudeAgentOptions(
    model="claude-opus-4-7",
    system_prompt=system_prompt,
    allowed_tools=["Read", "Write", "Edit", "Bash", "Grep", "Glob"],
    permission_mode="bypassPermissions",
    max_turns=None,  # unlimited
    effort="max",
)

The optimiser prompt asks the agent to improve both training BPC and sample quality, then commit successful improvements. Most commit messages record the change in score, for example “mid_sentence_cap_penalty: 2.3114 → 2.3100 BPC”. A separate file of nudges can be edited while the agent runs, to push it towards structural changes.

What the results show

The repository shows three things clearly.

1. An AI agent can optimise a model through code alone, for hundreds of steps. BPC fell steadily across 463 commits with no gradient descent anywhere in the system.
2. The process is fully inspectable. Every gain is a diff with a score attached. You can find the exact commit that introduced a behaviour and read the rule responsible.
3. Evaluation hygiene can be enforced in tooling. Blocking the dev and validation splits with a hook, rather than relying on the prompt, is a pattern any team running autonomous agents should copy. Our guide to agent evaluations covers the same principle for business systems.

What the results do not show

Be precise about the limits, because the headline numbers are easy to over-read.

• No generalisation result. Every published BPC figure is on the training corpus. The README says the dev and validation commands are “operator-only”, and no held-out score has been released.
• No comparison with neural models. For orientation only: the nanoGPT README reports a best validation loss of 1.4697 for its small character-level Shakespeare model, which is about 2.12 bits per character. That is a held-out score from a neural network on its own data split; Tiny Shakespeare CLM’s 2.3100 is a training score. The two cannot be compared, and we make no claim either way.
• Memorisation risk. A 60,878-line program has room to encode patterns specific to its training text. Without a held-out score, some of the gain could be fitting rather than learning.
• Sample quality is uneven. The README’s own fixed-seed samples mix plausible verse with invented words. At temperature 0.65:

KING HENRY:
Now is the winter of the court stretch
We sit test back to the time and mention, so be dress.

• One run, one author. There is no replication across seeds or optimiser settings, and no write-up beyond the README.
• No licence. The code is visible, but default copyright applies to reuse.

Why this matters beyond Shakespeare

Most organisations will never need a language model written as rules. The pattern still matters for two reasons.

First, it is a working example of a fully autonomous coding agent operating for days against a numeric objective, with git as its memory. The safeguards it needed (protected test data, a fixed scorer, score-tagged commits) are the same ones we put around coding agents such as Claude Code in client work.

Second, it points at a class of models that can be audited line by line. For decisions that must be explained, that property can matter more than the last few points of accuracy. We test claims like these the same way in our LLM evaluation service: agree the metric, protect the test set, and report what was and was not measured.

How to inspect it yourself

1. Open the repository. It is at github.com/levy-street/tiny-shakespeare-clm; the model lives under model_repo/model/.
2. Install the one dependency. The README asks for Python 3.10 or later and pydantic; the Claude Agent SDK is needed only to run the optimiser.
3. Run the contract check and full-train evaluation. The README lists commands that check the probability distribution sums to one and score the whole training corpus.
4. Read the history. Walk the commit log to watch individual rules arrive with their score changes.

The project sits alongside our other public code on the open-source page.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.