Short answer
A reliable AI agent architecture puts the model inside a deterministic control loop. Code owns state, budgets, permissions and stopping; the model only proposes the next step from a small set of typed tools. Most production failures come from unbounded loops, vague tools and unchecked side effects, so design those out first, and use a fixed workflow whenever the steps are known in advance.
- Use a fixed workflow when the steps are known. Reserve agent loops for open-ended paths.
- Code owns state, budgets, permissions and stop conditions. The model only proposes actions.
- Every tool is narrow, typed, idempotent and labelled with its side-effect class.
- Checkpoint state after every step so runs can resume, replay and be audited.
- Judge reliability by pass rate across repeated trials, never by one good demo.
When should you build an agent instead of a workflow?
Build an agent only when the sequence of steps cannot be known in advance. If you can draw the flowchart, write it as code and call a model at the fixed points where judgement is needed. Anthropic’s widely cited Building effective agents draws the same line: workflows run models and tools through predefined code paths, while agents direct their own process and tool use. Their advice, and ours, is to find the simplest solution that works and add autonomy only when it measurably helps.
| Signal in the work | Fixed workflow | Agent loop |
|---|---|---|
| Steps are the same for most cases | Yes | No |
| Path depends on what earlier steps discover | Awkward | Yes |
| Every action must be predictable for audit | Yes | Only with tight tool limits |
| Latency budget under a few seconds | Yes | Rarely |
| Cost per case must be near-constant | Yes | No, varies with path length |
| Exceptions are common and varied | Branches multiply | Yes, with escalation |
Most business processes we see are workflows with one or two open-ended steps. The right design is usually a workflow that contains a small, bounded agent for the open-ended part, such as researching an unfamiliar supplier, rather than an agent that owns the whole process. Our lesson on when a workflow does not need an agent walks through this decision with non-engineers, and the comparison of automation versus AI agents covers the business trade-offs.
What does a reliable agent architecture look like?
A reliable agent is a deterministic program with one non-deterministic component. The model proposes the next action; everything else, including whether that action is allowed to run, is ordinary code you can test.
- Orchestrator
- Plain code that owns the loop, the run record, budgets and stop conditions. It never delegates these to the model.
- Policy call
- One model call per step that returns a typed next action (call a tool, ask a person, or finish) through structured output, validated against a schema before anything happens.
- Tool layer
- Narrow functions with typed arguments, permission checks, idempotency keys and a declared side-effect class (read, reversible write, irreversible or external).
- State store
- A durable run record with a checkpoint after every step, so a run can resume after a crash, be replayed for debugging and be audited later.
- Guards
- Validators for arguments and final results, approval gates for risky actions, and caps on steps, spend, time and volume.
- Trace
- Every model call, tool call, approval and validation result, linked by run and step identifiers.
Treat the model’s output as a proposal from an untrusted component. That framing settles most design arguments: if you would not let an unauthenticated API request do something, do not let a model’s tool call do it either.
What does the control loop look like in code?
The loop below is the core of most agents we run, stripped of logging and retries. The model never sees the budget logic and cannot talk its way past it.
from dataclasses import dataclass, field
@dataclass
class Run:
goal: str
steps: list = field(default_factory=list)
spent_usd: float = 0.0
status: str = "running"
LIMITS = {"max_steps": 15, "max_usd": 1.00, "max_repeats": 2}
def run_agent(run, policy, tools, store, approvals):
seen = {}
while run.status == "running":
if len(run.steps) >= LIMITS["max_steps"] or run.spent_usd >= LIMITS["max_usd"]:
run.status = "escalated"
break
action = policy.next_action(run) # schema-validated structured output
run.spent_usd += action.cost_usd
if action.kind == "finish":
ok, reason = tools.verify_outcome(run, action.result) # check real state
run.status = "done" if ok else "escalated"
run.steps.append({"finish": action.result, "verified": ok, "reason": reason})
else:
key = (action.tool, action.args_hash)
seen[key] = seen.get(key, 0) + 1
if seen[key] > LIMITS["max_repeats"]:
run.status = "escalated" # looping on the same call
elif tools.side_effect(action.tool) == "irreversible" and not approvals.granted(run, action):
run.status = "awaiting_approval"
else:
result = tools.call(action.tool, action.args, idempotency_key=f"{run.goal}:{len(run.steps)}")
run.steps.append({"tool": action.tool, "result": result})
store.checkpoint(run) # durable after every step
return run
Two details matter more than they look. The finish action is checked against real state (did the record actually change, does the draft cite the documents it claims to) rather than trusting the model’s claim. And the idempotency key is derived from the run and step, so a retry after a crash cannot send the same email twice.
Which design rules keep tools and state predictable?
Tools are the agent’s interface to your business, and vague tools produce vague behaviour. The rules we apply to every tool:
- One verb per tool.
create_draft_replyandsend_replyare two tools, never one tool with asendflag. - Typed, constrained arguments. Enums for categories, identifiers that are validated against a lookup before use, and no free-text field where a structured one will do.
- Errors the model can act on. Return
{"error": "customer_not_found", "hint": "search by email instead"}rather than a stack trace or an empty list. - Bounded results. Cap rows and characters returned, and paginate. A tool that dumps 4,000 rows into context degrades every later step.
- Explicit state handles. Pass a case or workflow identifier as an argument rather than relying on hidden session state. The Model Context Protocol moved to exactly this pattern in its 2026-07-28 revision, which removed protocol-level sessions.
Our guide to tool calling in production covers contracts and retry semantics in depth, and agent memory design covers what state should persist between runs.
How should you set stop conditions and budgets?
Every agent needs at least five independent stop conditions, each enforced in code, and every one of them should end in escalation to a person rather than a silent failure.
| Guard | What it catches | When it trips |
|---|---|---|
| Step cap | Thrashing and open-ended exploration | Set slightly above the step count of successful runs in your evals |
| Spend cap | Runaway token use, expensive model fallbacks | Per run and per day, in USD, checked before each call |
| Wall-clock limit | Hung tools, slow providers | Matches what the user or downstream system will tolerate |
| Repeat detector | Same tool called with the same arguments | Two or three identical calls in one run |
| No-progress check | Many steps, no change to tracked state | A window of steps with no new facts or state changes |
Escalation means the run stops in a known state, the trace is attached, and a person can resume or finish it. A design that retries the whole run from the start on failure doubles cost and repeats side effects. See agent retry and failure recovery for resumable designs.
Where do agents fail in production, and how do you detect it?
The failures below are the ones that show up repeatedly once agents meet real data. Each has a detection signal you can put on a dashboard.
| Failure mode | What it looks like | Detection | Design control |
|---|---|---|---|
| False success | Agent reports “done” but nothing changed | Outcome verifier disagrees with the finish claim | Verify against system state before closing the run |
| Argument invention | Plausible but non-existent IDs, emails or amounts | Rising rate of not_found and validation errors |
Lookup-before-write, enums, strict schemas |
| Duplicate side effects | Two invoices or two emails after a timeout | Same idempotency scope seen twice | Idempotency keys per run and step |
| Goal hijack | Instructions in an email or web page redirect the agent | Tool calls outside the task’s normal pattern | Least privilege and approvals; see prompt injection testing |
| Long-run drift | Agent forgets constraints after many steps | Constraint violations rise with step count | Restate the goal and rules each step; compact history |
| Silent model change | Behaviour shifts with no code change | Eval pass rate drops after a provider update | Pin model versions; rerun the suite before upgrading |
Long-run drift has a known mechanism: models use information at the start and end of a long context better than information in the middle, as Liu et al. showed. Keep the goal and hard rules at the top of every policy call rather than only in the first message. Goal hijack and cascading errors between agents both appear in the OWASP Top 10 for Agentic Applications for 2026 (ASI01 and ASI08), which is a useful threat checklist for design reviews.
Where should people stay in the loop?
Put a person in front of every irreversible or external action until the evaluation data justifies removing them, and design the approval to be quick to do well. The Property Management Agent for APL Property follows this pattern: the agent handles routine administration and staff keep the decisions. Approval tiers, delegated permissions and abuse tests are covered in agent permissions and approvals, and queue design in human review queues.
How do you know an agent is reliable enough to ship?
Run a fixed task suite many times and look at consistency, not the best run. The τ-bench paper introduced pass^k, the probability that an agent succeeds on all k independent attempts at the same task, because single-run success hides how often a capable agent fails on a retry. An agent that passes 80% of tasks once but only half of them three times in a row is not ready to run unattended. Our guide to AI agent evaluations sets out how to build that suite, and our AI agent development work starts from it.
- The steps that are known in advance are written as code, not left to the model
- The model returns a schema-validated next action; code decides whether it runs
- Each tool has one verb, typed arguments, bounded results and a side-effect class
- Writes use idempotency keys derived from the run and step
- Step, spend, time, repeat and no-progress limits are enforced in code
- Every limit ends in escalation with the trace attached, not a silent failure
- Finish claims are verified against real system state
- Irreversible and external actions need human approval
- Run state is checkpointed after every step and can be resumed
- Model versions are pinned and the eval suite reruns before any upgrade
Sources
- Building effective agents, Anthropic, 19 December 2024
- OWASP Top 10 for Agentic Applications for 2026, OWASP Gen AI Security Project, 9 December 2025
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv (Yao et al.), 2024
- Lost in the Middle: How Language Models Use Long Contexts, arXiv (Liu et al.), 2023
- Model Context Protocol specification, version 2026-07-28, Model Context Protocol, 28 July 2026
Written by the Levy Street team in Wellington. Last updated 19 October 2026.