AI proof of concept.

What a proof of concept should prove

An AI proof of concept tests one claim: that AI can do a specific piece of your work, to a stated standard, on your own data, at an acceptable cost per item. “Can AI read our invoices?” is too vague; the answer is usually “sometimes”. “Can AI assign the right general ledger code to invoices from our top 40 suppliers at least 95% of the time, with review under 30 seconds an invoice?” can be proved or disproved in weeks.

It sits between an idea and a pilot:

Who it is for

Teams with a candidate workflow and a number to move, often from an AI readiness assessment. Boards that want evidence before approving a build. Buyers checking a vendor’s accuracy claims against their own documents rather than the vendor’s demo set; our AI vendor evaluation checklist lists the questions to ask first.

Go/no-go criteria, agreed first

The criteria are written and signed off before anything is built, so the result cannot be argued into a success afterwards. A typical set:

• Accuracy per category on a held-out test set the team never tuned against
• The worst error type, and how often it occurred, not just the average
• Review time per item compared with doing the work by hand today
• Model cost per item at expected production volume
• Response time that fits the workflow (seconds for a chat, hours for a batch)
• Data handling that passes your privacy and security review
• Staff who tried it would choose to use it

Illustrative example: a 20-person Wellington accounting practice tests invoice coding. Criteria: at least 95% correct codes on 300 held-out invoices, review under 30 seconds each, model cost under NZ$0.05 an invoice. At OpenAI’s listed price for GPT-6.1 Sol on 19 October 2026 (US$2 input and US$10 output per million tokens; OpenAI), an invoice using 3,000 input and 300 output tokens costs US$0.009, about NZ$0.016 at the European Central Bank reference rate of NZ$1.7656 per US dollar on 16 October 2026 (Frankfurter). Cost passes easily. Accuracy is the real test: if it comes in at 91%, the decision turns on whether the errors cluster in a few suppliers that can be excluded. The same proof of concept should also test the alternative already on the shelf: Xero’s built-in AI, JAX, captures bills and receipts and reconciles bank transactions (Xero), and if it clears the bar there is nothing to build. Our notes on AI and Xero cover both routes.

What we deliver

Criteria sheet

The question, the go/no-go thresholds and who signs them off.

Test set

Real cases, labelled by your staff, with a held-out portion nobody tunes against.

Working prototype

The narrowest system that can be measured, running on your data in a sandbox.

Results report

Pass or fail against each criterion, results by category, and the failure examples that matter.

Production gap estimate

What it would take to run this daily: integration, permissions, review screens, monitoring and training.

Recommendation

Go, go with a narrower scope, fix the data first, buy a product instead, or stop.

How the work runs

1. Discover: frame the question. One workshop to choose the workflow step, agree the criteria and identify the data. Data access requests start the same day, because they are usually the slowest part.
2. Prove: build narrow, test honestly. We build only what the test needs, tune on part of the test set and then run the held-out cases once. Results are reported by category, including the ones that fail.
3. Decide, then scale or stop. You get the report and the production estimate together. If the answer is go, the test set becomes the regression suite for the production build through custom AI development.

What changes on the way to production

The prototype is the smallest part of the cost. A production system has to cope with everything the sandbox avoided:

Our guide to what custom AI projects cost shows how to estimate each line.

Where proofs of concept go wrong

• Clean data, happy results. A test set without the awkward cases proves nothing. Ask staff for the examples they dread.
• Moving goalposts. Criteria softened after the results arrive turn a no into an expensive maybe.
• No owner for the next step. A successful proof of concept with no budget or owner stalls. In Datacom’s 2026 survey of larger NZ organisations, only 15% were scaling AI organisation-wide (Datacom).
• Real data without a privacy check. The Privacy Commissioner expects a privacy impact assessment before AI tools are used on personal information, and a proof of concept is no exception.

Work this draws on

• World of ClaudeCraft: Levy Street’s free, open-source browser MMO. Its first playable version was built in about 48 hours in June 2026 with the Claude Fable 5 coding model; since then roughly 60 contributors have pushed about 19,900 commits. The prototype was the fast part, and the repository shows everything that came after.
• Property management agent for APL Property: an internal agent handling routine administration for a regional NZ property firm, where staff keep the decisions.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.