# Agent retry and failure recovery.

URL: https://levystreet.com/engineering/agent-retry-and-failure-recovery/
Updated: 2026-10-19

AI agent reliability in practice: classify failures, retry with backoff and jitter, make tool calls idempotent, checkpoint runs and fall back safely.

## Short answer

AI agent reliability comes from assuming every step can fail and making each one safe to retry. Classify failures as transient, model-level or permanent; retry only transient ones with capped exponential backoff and jitter; give every side-effecting tool call an idempotency key; checkpoint progress so a crash resumes instead of restarting; and hand the run to a person when the retry budget is spent.

- Per-step reliability compounds: 99% per step across 20 steps completes about 82% of runs.
- Retry transient errors only, with backoff and jitter, at one layer of the stack.
- Every write gets an idempotency key so a retry can never repeat a side effect.
- Persist each step before the next so a crashed run resumes from its last checkpoint.
- Any fallback model must pass the same evaluation suite as the primary.

## Why agents need a recovery design

An agent's reliability is the product of its steps. If each step succeeds 99% of the time, a 20-step task completes 0.99²⁰ ≈ 82% of the time; at 95% per step, a 10-step task completes about 60% of the time. Recovery logic is what closes that gap without hiding real failures.

Agents fail at three layers, and each needs different handling:

- **Infrastructure**: timeouts, rate limits (HTTP 429), provider overload, server errors, dropped connections.
- **Model**: malformed structured output, a call to the wrong tool, invalid arguments, a refusal, or a loop that repeats the same action.
- **World**: the tool works but the business says no. The record is locked, the customer does not exist, the user lacks permission, or the data changed since the agent read it.

The architecture around these steps is covered in [designing reliable AI agents](/engineering/designing-reliable-ai-agents/). This guide covers what happens when a step fails, and it reflects the defaults we use in [AI agent development](/services/ai-agent-development/).

## Classify the failure before you retry

Retry only what a retry can fix. A retried permission error just fails again more slowly, and a retried write after a timeout can do the action twice.

| Failure | Example | Retry? | Action |
|---|---|---|---|
| Rate limit | HTTP 429 with a Retry-After header | Yes | Wait at least the stated time, then back off |
| Transient server error | 500, 502, 503, provider overloaded | Yes | Backoff with jitter, within a time budget |
| Timeout on a read | Search call exceeds 20 seconds | Yes | Retry; consider a smaller request |
| Timeout on a write | Create-invoice call times out | Not blindly | Reconcile first: check whether it happened, then retry with the same key |
| Malformed output | JSON fails schema validation | Once | Re-ask with the validator's error message, then escalate |
| Invalid tool arguments | Date in the wrong format | Via the model | Return the validation error as the tool result; count it against the step budget |
| Business rejection | 403, 404, 409 conflict | No | Surface to the model or a person with the reason |
| Policy refusal | Model declines the request | No | Route to human review; do not rephrase until it complies |
| Loop | Same tool, same arguments, third time | No | Stop the run and escalate |

Validation of model output is a topic of its own; see [structured outputs and validation](/engineering/structured-outputs-and-validation/).

## Backoff with jitter, at one layer only

Use capped exponential backoff with random jitter, bounded by a deadline. Exponential growth gives a struggling service room to recover, and jitter stops a fleet of clients retrying in lockstep. The Amazon Builders' Library article on [timeouts, retries and backoff with jitter](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter) is the clearest reference on why both matter, and it recommends retrying at a single point in the stack because retries at each layer multiply.

```python
import random
import time

class TransientError(Exception):
    def __init__(self, retry_after: float | None = None):
        super().__init__("transient failure")
        self.retry_after = retry_after

def call_with_retry(fn, *, max_attempts=3, base=0.5, cap=20.0, deadline_s=60.0):
    start = time.monotonic()
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except TransientError as err:
            if attempt == max_attempts:
                raise
            backoff = random.uniform(0, min(cap, base * 2 ** attempt))  # full jitter
            delay = max(backoff, err.retry_after or 0)
            if time.monotonic() - start + delay > deadline_s:
                raise
            time.sleep(delay)
```

Retry at one layer. If the HTTP client, the model SDK and the agent loop each try three times, one failing dependency receives 3 × 3 × 3 = 27 calls per user action, which turns a brownout into an outage. Pick the layer with the most context (usually the agent's step runner), and switch the others off or set them to a single attempt.

## Make every side effect idempotent

An idempotent operation has the same effect whether it runs once or many times. RFC 9110 defines PUT, DELETE and safe methods such as GET as [idempotent](https://datatracker.ietf.org/doc/html/rfc9110#name-idempotent-methods), and says a client should not automatically retry a non-idempotent request unless it can tell the original was never applied. POST is not idempotent, and most agent tools that create things are POSTs underneath. The fix is an idempotency key: a unique value sent with the write so the receiver does it at most once. [Stripe's idempotent requests](https://docs.stripe.com/api/idempotent_requests) are a good model: the first response for a key is saved and replayed to retries, and a retry with the same key but different parameters is rejected. For systems that do not support keys, look up by a natural business key before you create.

```python
import hashlib
import json

def idempotency_key(run_id: str, step_id: str, tool_name: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(f"{run_id}:{step_id}:{tool_name}:{canonical}".encode()).hexdigest()

def run_write_tool(store, run_id, step_id, tool, args):
    key = idempotency_key(run_id, step_id, tool.name, args)
    prior = store.get(key)
    if prior and prior["status"] == "done":
        return prior["result"]                    # replay, never repeat
    if prior and prior["status"] == "in_flight":
        found = tool.find_existing(args)          # outcome unknown: look first
        if found is not None:
            store.put(key, {"status": "done", "result": found})
            return found
    store.put(key, {"status": "in_flight"})
    result = tool.execute(args, idempotency_key=key)
    store.put(key, {"status": "done", "result": result})
    return result
```

The run and step IDs stop a retry of the same step from repeating its effect. They do not stop the model from deciding, three steps later, to create the same invoice again. For that, the tool itself must check a business key such as supplier plus invoice number. Our guide to [AI tool calling in production](/engineering/ai-tool-calling-in-production/) covers tool contracts in more detail.

## Checkpoint so a crash resumes, not restarts

Persist each step's input, tool call, result and status before starting the next step. When the worker dies, a new worker reads the run log, skips completed steps (returning their stored results), reconciles any step left in flight, and continues. Without checkpoints, a crash at step 14 either repeats steps 1 to 13 or abandons the run.

Practical rules:

- **Write the log first.** Record "about to call tool X with key K" before the call, and the result after it.
- **Keep runs replayable.** Store model outputs, so a resumed run does not re-sample a different plan halfway through.
- **Version the run.** Record the prompt, model and tool versions; resuming a run under a new prompt is a different run.
- **Bound everything.** Maximum steps, tokens, wall-clock time and spend per run. Budget exhaustion is a normal outcome with a handler, not a crash.

The same idea runs through [SPAWN D](/portfolio/spawnd/), Levy Street's open-source control plane for CLI coding agents, where each session's worker is designed to survive a restart of the daemon that manages it.

## Fallbacks and graceful degradation

When retries run out, step down a ladder of fallbacks that you chose and tested in advance:


1. **Repair.** For malformed output, re-ask once with the exact validation error.
2. **Alternate route.** Call a second provider or model, but only one that has passed the same evaluation suite on these tasks.
3. **Reduced scope.** Return a partial result that names what is missing, rather than a confident complete one.
4. **Human queue.** Park the run with its full state in a [human review queue](/engineering/human-review-queues/) so a person can finish it.
5. **Fail closed.** Stop, record why, and notify the owner. No silent success.


Add a circuit breaker per dependency: when the error rate over a short window crosses a threshold, stop calling it for a cool-down period and go straight to the fallback. This protects both the provider and your token budget.

## Test recovery with injected faults

Recovery code that has never run will not work on the day it is needed. Build a fault-injection mode into the tool layer and run it in CI and in staging.

```python
def test_create_invoice_runs_once_under_retry(fake_accounts, store):
    args = {"supplier": "ACME", "number": "INV-1042", "total": "1150.00"}
    fake_accounts.fail_next(times=1, mode="timeout_after_commit")
    first = run_write_tool(store, "run-7", "step-3", create_invoice, args)
    second = run_write_tool(store, "run-7", "step-3", create_invoice, args)
    assert first == second
    assert fake_accounts.count(number="INV-1042") == 1
```

The cases worth covering: process killed mid-run, 429 and 503 responses, a write that commits and then times out, malformed model output twice in a row, duplicate webhook delivery, and a tool that returns success with an empty body.

## Where retry logic goes wrong

| Failure mode | Symptom | Detection |
|---|---|---|
| Retry storm | Error rate and request volume rise together | Attempts per successful call, by dependency |
| Duplicate side effects | Two invoices, two emails | Nightly reconciliation of business keys against the run log |
| Token-burning loops | Long runs with repeated identical tool calls | Loop detector on (tool, arguments) hashes; spend per run alerts |
| Hidden degradation | Fallback path quietly carries most traffic | Fallback activation rate on the quality dashboard |
| Swallowed errors | Runs "succeed" with empty outputs | Assert on output content, not just exit status |
| Stranded runs | Runs stuck in flight after a deploy | Age of oldest in-flight run; resume job on start-up |

Each of these deserves an alert and a runbook entry, as described in [AI incident response](/engineering/ai-incident-response/). If you are inheriting an agent that already shows these symptoms, [AI system rescue](/services/ai-system-rescue/) is usually where we start.


- Every failure type the agent can hit is classified as retry, repair, escalate or stop
- Retries use capped exponential backoff with jitter, honour Retry-After and sit at one layer
- Every write tool takes an idempotency key and checks a business key before creating
- Timed-out writes are reconciled before any retry
- Each step is persisted before the next starts; runs resume from the log after a crash
- Runs have budgets for steps, tokens, time and spend, with a handler for exhaustion
- Fallback models have passed the same evaluation suite as the primary
- Circuit breakers protect each external dependency
- Fault-injection tests cover crashes, timeouts after commit, malformed output and duplicate events
- Retry rate, fallback rate, loop stops and duplicate reconciliations are monitored

## Questions people ask

### Why do AI agents fail more often than normal software?

An agent chains many uncertain steps: model calls that can time out or return malformed output, tool calls into systems that rate-limit or reject requests, and decisions that depend on data that changes mid-run. Small per-step failure rates multiply across a long task. Without retries, idempotency and checkpoints, one flaky step fails the whole run or, worse, repeats an action.

### How many times should an agent retry a failed step?

Usually two or three attempts for transient errors, bounded by a time budget rather than a count alone, with exponential backoff and random jitter. Do not retry permanent errors such as permission denied or record not found, and do not retry at every layer of the stack, because nested retries multiply load on a service that is already struggling.

### What is an idempotency key and why do agents need one?

An idempotency key is a unique value sent with a write so the receiving system performs it at most once, however many times the request arrives. Agents need them because a timeout leaves the outcome unknown: the invoice may or may not have been created. With a key, the retry returns the original result instead of creating a duplicate.

### Should an agent switch to a different model when the main one fails?

Only if the fallback model has passed the same evaluation suite on the same tasks. An untested fallback turns an outage into silent quality loss, which is harder to notice and often more expensive. Many tasks are better served by a short queue and a retry later, or by handing the work to a person, than by a weaker model.

### How do you test that an agent recovers from failures?

Inject faults deliberately: kill the process mid-run, return rate-limit and server errors, delay responses past the timeout, send malformed model output, and deliver the same webhook twice. Then assert the run either completes with exactly one effect per intended action or stops in a known state with a human notified.


## Sources

- [Timeouts, retries, and backoff with jitter](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter)
- [Idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [RFC 9110: HTTP Semantics, section 9.2.2 Idempotent Methods](https://datatracker.ietf.org/doc/html/rfc9110#name-idempotent-methods)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.