Agent retry and failure recovery.

Why agents need a recovery design

An agent’s reliability is the product of its steps. If each step succeeds 99% of the time, a 20-step task completes 0.99²⁰ ≈ 82% of the time; at 95% per step, a 10-step task completes about 60% of the time. Recovery logic is what closes that gap without hiding real failures.

Agents fail at three layers, and each needs different handling:

• Infrastructure: timeouts, rate limits (HTTP 429), provider overload, server errors, dropped connections.
• Model: malformed structured output, a call to the wrong tool, invalid arguments, a refusal, or a loop that repeats the same action.
• World: the tool works but the business says no. The record is locked, the customer does not exist, the user lacks permission, or the data changed since the agent read it.

The architecture around these steps is covered in designing reliable AI agents. This guide covers what happens when a step fails, and it reflects the defaults we use in AI agent development.

Classify the failure before you retry

Retry only what a retry can fix. A retried permission error just fails again more slowly, and a retried write after a timeout can do the action twice.

Validation of model output is a topic of its own; see structured outputs and validation.

Backoff with jitter, at one layer only

Use capped exponential backoff with random jitter, bounded by a deadline. Exponential growth gives a struggling service room to recover, and jitter stops a fleet of clients retrying in lockstep. The Amazon Builders’ Library article on timeouts, retries and backoff with jitter is the clearest reference on why both matter, and it recommends retrying at a single point in the stack because retries at each layer multiply.

import random
import time

class TransientError(Exception):
    def __init__(self, retry_after: float | None = None):
        super().__init__("transient failure")
        self.retry_after = retry_after

def call_with_retry(fn, *, max_attempts=3, base=0.5, cap=20.0, deadline_s=60.0):
    start = time.monotonic()
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except TransientError as err:
            if attempt == max_attempts:
                raise
            backoff = random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter
            delay = max(backoff, err.retry_after or 0)
            if time.monotonic() - start + delay > deadline_s:
                raise
            time.sleep(delay)

Retry at one layer. If the HTTP client, the model SDK and the agent loop each try three times, one failing dependency receives 3 × 3 × 3 = 27 calls per user action, which turns a brownout into an outage. Pick the layer with the most context (usually the agent’s step runner), and switch the others off or set them to a single attempt.

Make every side effect idempotent

An idempotent operation has the same effect whether it runs once or many times. RFC 9110 defines PUT, DELETE and safe methods such as GET as idempotent, and says a client should not automatically retry a non-idempotent request unless it can tell the original was never applied. POST is not idempotent, and most agent tools that create things are POSTs underneath. The fix is an idempotency key: a unique value sent with the write so the receiver does it at most once. Stripe’s idempotent requests are a good model: the first response for a key is saved and replayed to retries, and a retry with the same key but different parameters is rejected. For systems that do not support keys, look up by a natural business key before you create.

import hashlib
import json

def idempotency_key(run_id: str, step_id: str, tool_name: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(f"{run_id}:{step_id}:{tool_name}:{canonical}".encode()).hexdigest()

def run_write_tool(store, run_id, step_id, tool, args):
    key = idempotency_key(run_id, step_id, tool.name, args)
    prior = store.get(key)
    if prior and prior["status"] == "done":
        return prior["result"]                    # replay, never repeat
    if prior and prior["status"] == "in_flight":
        found = tool.find_existing(args)          # outcome unknown: look first
        if found is not None:
            store.put(key, {"status": "done", "result": found})
            return found
    store.put(key, {"status": "in_flight"})
    result = tool.execute(args, idempotency_key=key)
    store.put(key, {"status": "done", "result": result})
    return result

The run and step IDs stop a retry of the same step from repeating its effect. They do not stop the model from deciding, three steps later, to create the same invoice again. For that, the tool itself must check a business key such as supplier plus invoice number. Our guide to AI tool calling in production covers tool contracts in more detail.

Checkpoint so a crash resumes, not restarts

Persist each step’s input, tool call, result and status before starting the next step. When the worker dies, a new worker reads the run log, skips completed steps (returning their stored results), reconciles any step left in flight, and continues. Without checkpoints, a crash at step 14 either repeats steps 1 to 13 or abandons the run.

Practical rules:

• Write the log first. Record “about to call tool X with key K” before the call, and the result after it.
• Keep runs replayable. Store model outputs, so a resumed run does not re-sample a different plan halfway through.
• Version the run. Record the prompt, model and tool versions; resuming a run under a new prompt is a different run.
• Bound everything. Maximum steps, tokens, wall-clock time and spend per run. Budget exhaustion is a normal outcome with a handler, not a crash.

The same idea runs through SPAWN D, Levy Street’s open-source control plane for CLI coding agents, where each session’s worker is designed to survive a restart of the daemon that manages it.

Fallbacks and graceful degradation

When retries run out, step down a ladder of fallbacks that you chose and tested in advance:

1. Repair. For malformed output, re-ask once with the exact validation error.
2. Alternate route. Call a second provider or model, but only one that has passed the same evaluation suite on these tasks.
3. Reduced scope. Return a partial result that names what is missing, rather than a confident complete one.
4. Human queue. Park the run with its full state in a human review queue so a person can finish it.
5. Fail closed. Stop, record why, and notify the owner. No silent success.

Add a circuit breaker per dependency: when the error rate over a short window crosses a threshold, stop calling it for a cool-down period and go straight to the fallback. This protects both the provider and your token budget.

Test recovery with injected faults

Recovery code that has never run will not work on the day it is needed. Build a fault-injection mode into the tool layer and run it in CI and in staging.

def test_create_invoice_runs_once_under_retry(fake_accounts, store):
    args = {"supplier": "ACME", "number": "INV-1042", "total": "1150.00"}
    fake_accounts.fail_next(times=1, mode="timeout_after_commit")
    first = run_write_tool(store, "run-7", "step-3", create_invoice, args)
    second = run_write_tool(store, "run-7", "step-3", create_invoice, args)
    assert first == second
    assert fake_accounts.count(number="INV-1042") == 1

The cases worth covering: process killed mid-run, 429 and 503 responses, a write that commits and then times out, malformed model output twice in a row, duplicate webhook delivery, and a tool that returns success with an empty body.

Where retry logic goes wrong

Each of these deserves an alert and a runbook entry, as described in AI incident response. If you are inheriting an agent that already shows these symptoms, AI system rescue is usually where we start.

• Every failure type the agent can hit is classified as retry, repair, escalate or stop
• Retries use capped exponential backoff with jitter, honour Retry-After and sit at one layer
• Every write tool takes an idempotency key and checks a business key before creating
• Timed-out writes are reconciled before any retry
• Each step is persisted before the next starts; runs resume from the log after a crash
• Runs have budgets for steps, tokens, time and spend, with a handler for exhaustion
• Fallback models have passed the same evaluation suite as the primary
• Circuit breakers protect each external dependency
• Fault-injection tests cover crashes, timeouts after commit, malformed output and duplicate events
• Retry rate, fallback rate, loop stops and duplicate reconciliations are monitored

Written by the Levy Street team in Wellington. Last updated 19 October 2026.