Temporal probability calibration.

The problem: the right confidence at the wrong time

A probabilistic model is calibrated when its confidence matches reality. The paper puts it plainly: “for those examples that are assigned a confidence of 75% by a perfectly calibrated classifier, 75% of them should actually be classified correctly.” Many modern models classify well but report poor probabilities, so practitioners add a calibration step, such as temperature scaling, fitted on held-out data.

Those methods assume a prediction is made once, on a complete input. Live systems do not work that way. A betting model prices a match while it is being played; a support system guesses intent while a customer is still typing. The authors’ example: “if the score in a football game is 1-0 with one minute remaining, we should be much more confident in our prediction of the winner than if the score is 1-0 at half-time.”

The paper’s first figure shows the problem directly. On the Large Movie Review dataset, the calibration error of common architectures changed substantially with input length. Deep averaging networks and transformers were worse calibrated on short sequences, while a recurrent network showed the reverse. In the authors’ words, “a global calibration strategy that applies the same calibration to sequences of any length will not be suitable for these models.”

The idea: calibrate by stage, not globally

Temporal probability calibration lets the correction itself depend on how far through the sequence a prediction is made. It builds on temperature scaling, which learns one parameter that makes predictions more or less extreme without changing which class wins, so accuracy is untouched.

The third row is the subtle one. In a game played to five round wins, reaching round nine means the players are evenly matched, so the outcome is harder to call, not easier. The paper suggests grouping by score difference instead and leaves the choice of progress measure “to the practitioner to decide the best approach for their specific situation.”

The continuous variant uses a function of the form below, where the output acts as an inverse temperature applied to the model’s logits:

g(t) = γ − α · exp(−β·t − s)        t = sequence position, normalised to [0, 1]
calibrated probability = sigmoid( g(t) · logit )

What the experiments showed

The authors tested three settings, reporting the mean of ten runs with different random seeds and checking significance with Friedman and Nemenyi tests at p = 0.05.

Large Movie Review

20,000 training, 5,000 calibration and 25,000 test reviews, averaging 233 words. Positive or negative sentiment.

Amazon Fine Food Review

450,000 training, 50,000 calibration and 68,484 test reviews, averaging 85 words, reduced to a binary task.

CS:GO match outcomes

a new public dataset of 12,362 games (8,751 training, 2,907 calibration, 704 test), which the authors describe as the first public dataset for Counter-Strike winner prediction.

Models

a deep averaging network, a recurrent network (GRU) and a BERT-based classifier for text; a GRU for CS:GO.

Three findings stand out. For every dataset and model pair, temporal calibration gave “the lowest (or tied lowest)” expected calibration error for most sequence lengths. Global calibration “often resulted in degraded overall ECE compared to the baseline, despite an improvement in NLL”, a warning for anyone who checks only one metric. And on CS:GO, calibrating by round number gave the best results. The authors conclude the gains are largest for deep averaging networks on text, or when training data is limited.

Why partial-sequence calibration matters

The paper was written at Sportsflare, which supplied in-play esports odds to sportsbooks: prices set from probabilities while a match is under way. In that setting a badly calibrated model is not a statistics problem; it is a pricing error.

The same structure appears wherever a probability is acted on before the outcome is known:

• Streaming classification. Intent detection while a customer types, or escalation risk as a call transcript grows.
• Risk scoring during a session. Fraud or abandonment scores that update with every event.
• AI agents mid-task. A 2026 Technion paper applies calibration to vision-language-action models over a task’s steps, describing this paper’s approach as one where “predictions at different time steps are treated independently”.

If a downstream rule says “escalate above 0.8”, the 0.8 must mean the same thing at minute one and minute ten. That is the practical case for checking calibration by stage, part of the wider discipline in our guide to monitoring AI quality.

Who has cited it

OpenAlex records 6 citations. Levy Street has verified at least 7 citing works, each confirmed from the citing paper’s own text or from OpenAlex’s reference data.

We list only citations we could confirm. Other institutions have been mentioned in the past without evidence, and we have dropped them.

Status and limits

The paper is an arXiv preprint (arXiv:2002.02644), submitted on 7 February 2020 and revised on 15 February 2020. It has not been through peer review or appeared at a venue. Its limits are worth stating:

• Simple methods by design. It extends temperature scaling only; richer calibrators were not tested in temporal form.
• Synthetic truncation. The text experiments create partial sequences by cutting complete reviews at random points, which approximates but is not the same as live streams.
• Small test set for CS:GO. 704 test games limits how finely per-round results can be read.
• Independent steps. Each time step is calibrated separately, a limitation that the 2026 Technion work sets out to address.

The authors were Max Polaczuk, then CEO of Sportsflare and now Levy Street’s chief executive, and Tim Leathart, Sportsflare’s Chief Scientist. The acknowledgements thank Levy Street co-founder Chris Herrmann for useful discussions.

How to apply it in practice

1. Hold out a calibration set. Keep it separate from training and test data, with complete sequences and known outcomes.
2. Create partial sequences. Cut each held-out sequence at several random points, or at every discrete step, so the set covers early, middle and late predictions.
3. Choose the measure of progress. Time, position, round number or a domain signal such as score difference.
4. Fit and compare. Fit a global temperature and a stage-dependent one; compare negative log-likelihood and expected calibration error separately for early, middle and late bins.
5. Monitor in production. Track calibration by stage over time, because drift often shows up first in early predictions.

We use the same staged evaluation when testing forecasting and agent systems; see AI agent evaluations and our LLM evaluation service.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.