Engineering notes from production AI.
How we design, test and run AI systems that people depend on. Written for engineers and technical leads.
Short answer
These guides cover how to build AI systems that hold up in production: agent design, evaluation, retrieval-augmented generation, structured outputs, tool calling and MCP, permissions, observability, latency and cost control, and incident response. Each one is based on patterns Levy Street uses when building and running systems for clients and for its own products.
All engineering guides
- Designing reliable AI agentsA reference architecture for agents that finish the job, stop when they should and leave an audit trail.
- AI agent evaluationsHow to build a task suite that tells you whether an agent is ready, with graders, repeats and release gates.
- RAG retrieval evaluationHow to build a labelled dataset and measure whether your RAG system finds the right passages before it answers.
- RAG versus fine-tuningRetrieval changes what the model knows at answer time. Fine-tuning changes how it behaves. Here is how to choose.
- Choosing a vector databasePick a vector store on filtered recall, operations and data location, then prove it with a benchmark on your own data.
- Structured outputs and validationConstrained decoding gets you valid JSON. Two validation layers and a bounded repair loop get you correct data.
- AI tool calling in productionTreat every tool as an API for an unreliable caller: narrow verbs, typed arguments, idempotency keys and approval tiers.
- MCP for business systemsA reference MCP server design for business systems: stateless, scoped, confirmed writes, and logs you can audit.
- Agent memory designMemory makes agents useful across sessions and makes them a privacy and poisoning risk. Design the writes first.
- Agent permissions and approvalsLeast privilege, action tiers and approvals people actually read, so a fooled agent cannot do much damage.
- Prompt injection testingDefensive test cases for direct and indirect injection, with canaries, metrics and the defences that hold.
- AI audit logsA trace structure that lets you reconstruct any AI decision weeks later, without turning logs into a privacy risk.
- Monitoring AI qualityThe dashboard panels and alert rules that catch an AI system getting worse before your customers notice.
- Reducing AI latencyWhere LLM latency comes from, which fixes move p95, and how to prove the change with a before-and-after test.
- Managing LLM costsMeasure cost per successful task, then cut waste, cache stable prompts, batch what can wait and route by difficulty.
- Human review queuesRouting rules, audit sampling and reviewer metrics for human in the loop AI systems that stay honest.
- Agent retry and failure recoveryFailure classes, backoff, idempotency keys, checkpoints and fallbacks for agents that recover instead of repeating.
- Evaluating long context modelsDepth grids, distractor tests and the effective context metric for deciding how much a long context model really reads.
- Multi-agent systems in practiceA controlled-comparison method for deciding whether several agents beat one, with metrics and failure modes to watch.
- Building private AI searchPermission-aware retrieval, ACL sync and the leakage tests that prove one user cannot see another’s documents.
- Running open models for businessMemory sizing, serving stacks, capability tests and break-even arithmetic for running open-weight models yourself.
- AI data preparationAn ingestion pipeline with quality gates at every stage, so every AI answer traces back to a clean, current source.
- OCR versus vision modelsHow OCR and vision language models fail differently, and a benchmark method for choosing or combining them.
- Voice agent quality evaluationWER, entity accuracy, turn latency and escalation tests for voice agents that callers can actually rely on.
- AI model migrationRegression gates, shadow traffic, canaries and rollback for moving production AI to a new model without surprises.
- Versioning prompts and evaluationsWhat to version, how to hash it and how to keep evals comparable, so any AI output can be reproduced or rolled back.
- AI incident responseSeverity levels, kill switches, scoping queries and a recovery runbook for incidents in production AI systems.
- Testing AI with synthetic dataWhere generated test cases help, why they flatter your system, and how to measure the gap against real data.
- AI product feedback loopsA feedback taxonomy, signal design and quality metrics for AI products that improve from real use instead of drifting.
- Data residency for AI systemsWhat NZ law requires, which cloud regions exist in or near New Zealand, and where each major AI provider runs inference.
Building something that has to hold up?
We design, evaluate and run AI systems in production. Tell us what you are building and where it hurts.