Custom AI development.

When custom AI is justified

Build custom AI when a workflow depends on your own data, rules or systems in a way a general tool cannot reach, and when the volume or the accuracy bar makes that worth paying for. If Microsoft Copilot, ChatGPT or Claude can do the job with good instructions and access to the right files, configure and train instead. Our comparison of custom AI versus off-the-shelf tools sets out the trade-off.

Custom is the minority path, and that is healthy. In the AI Forum’s March 2025 survey of NZ organisations (a self-selected industry-body sample), 72% used pre-existing AI solutions and 13% custom-built ones. The organisations that come to us for a build usually have one of these problems:

• The data is yours and scattered. Answers depend on job files, contracts, a CRM and an accounting system that no single product can see.
• The rules are yours. Pricing logic, compliance checks or approval thresholds that a generic assistant will get wrong without them.
• The volume is real. Thousands of documents, tickets or records a month, where per-seat tools are the wrong cost shape.
• AI is the product. You are building something customers will use, so quality, latency and cost per user are your problem.

The architecture options we choose between

Most of the design decision is choosing the least complex shape that can pass your acceptance tests. More autonomy means more to test, secure and monitor.

See RAG development and AI agent development for the two most common custom shapes, and our engineering note on RAG versus fine-tuning.

What you receive

Architecture decision record

The option chosen, the ones rejected and why, written so a future engineer can follow the reasoning.

Evaluation set and results

Real examples with agreed correct answers, the scoring method and the pass thresholds, rerun on every change.

Working system

Deployed in your cloud or ours, integrated with the systems it needs, with access limited to what each step requires.

Review and approval design

Where people check outputs, what they see, and how corrections feed back into the evaluation set.

Monitoring

Quality samples, error rates, latency and model spend per day, with alerts on the ones that matter.

Runbook and handover

How to operate, change and roll back the system, and who to call.

Delivery milestones and acceptance criteria

1. Discover (one to three weeks). Map the workflow with the people who do it, collect 100 or more real examples including the awkward ones, and write acceptance criteria you sign off.
2. Prove: thin slice. Build the narrowest end-to-end path on real data, from input to output in the target system, and score it against the evaluation set.
3. Prove: supervised pilot. Real users, real volume, every output reviewed. We measure accuracy, review time and cost per item against the baseline.
4. Scale: harden and hand over. Error handling, monitoring, security review and documentation, then a staged move from full review to sampling where the numbers support it.

Acceptance criteria are specific to your workflow. An illustrative set for a document extraction build might read:

• Required fields are extracted correctly on at least the agreed share of a held-out set of 200 documents the team has never tuned on
• Every output the system is unsure about lands in a review queue with the reason shown
• No document is lost: each ends as processed, queued or rejected with a reason
• Median processing time and cost per document are within the agreed limits
• The evaluation can be rerun by your team from the runbook

What drives the cost

Cost is driven by the number of systems to integrate, how clean your data access is, the accuracy bar, the design of human review, and how many people or customers will use it. Model fees are usually a small share. As an illustration with assumed numbers: 2,000 requests a day at an assumed two cents (NZD) each is about NZ$1,200 a month in model fees, while the integration and review design behind those requests is most of the build. Our guide to what custom AI projects cost covers the full model, and the AI ROI calculator tests whether the saving justifies it. We do not publish rates; each project is scoped and priced after discovery.

Systems we have built

• Lucent, a Levy Street venture, is an agentic investment research platform for wealth, equity and private-markets teams covering NZX and ASX. It runs several frontier models together and grounds its outputs in citations.
• Singing Coach AI is an AI vocal coach on iPhone, Android and the web. It scores a recording across 26 vocal dimensions and gives coaching, with feedback in a minute or two.
• PainPal is an AI “digital workmate” for musculoskeletal pain and injury prevention, launched by BetterWorx (a subsidiary of CHASNZ) in December 2025. Levy Street worked with BetterWorx on PainPal.

Where custom AI projects fail

• Demo accuracy. A system tuned on ten friendly examples meets the real inbox and falls over. The held-out evaluation set exists to stop this.
• Platform before product. Teams build a general AI platform before one workflow works. Ship one workflow, then extract the shared parts.
• Ignoring review cost. If every output needs a five-minute check, the saving may vanish. Review time is measured in the pilot, not assumed.
• Lock-in by accident. Building on one vendor’s proprietary features makes a model change a rewrite. We keep model calls behind a thin layer so a model migration is a test run.

Do not commission a build if a configured product gets you most of the value, if the process itself is broken or changing, or if the volume is a few dozen items a month. People or a simple tool will be cheaper.

New Zealand specifics

• Data location. Anthropic’s own API offers only US or global processing; OpenAI’s Australian API region stores data there but does not process it there; Amazon Bedrock’s AU profile keeps Claude processing in Australia and New Zealand. We record the choice for each system.
• Privacy. Under the Privacy Act 2020 a provider that only processes data on your behalf leaves you fully responsible; one that uses it for its own purposes, such as training, changes the analysis. The 14 information privacy principles, including IPP 3A since 1 May 2026, are checked at design time.
• Local software. Builds commonly connect to Xero, MYOB, Microsoft 365, ServiceM8 or Simpro, and the integration work is usually the largest single line in the build.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.