# AI proof of concept.

URL: https://levystreet.com/services/ai-proof-of-concept/
Updated: 2026-10-19

AI proof of concept for New Zealand businesses: a narrow build on your own data, go/no-go criteria agreed up front, and the real cost of reaching production.

## Short answer

An AI proof of concept is a short, deliberately narrow build that tests whether AI can do one specific piece of work well enough, on your own data, to justify a production system. Levy Street runs proofs of concept against go/no-go criteria agreed in advance, such as accuracy, review time and cost per item, and estimates the gap to production before you decide.

- Agree the go/no-go numbers before building, not after the demo.
- Test on real, messy inputs, including the cases your staff dread.
- A proof of concept that ends in no has saved you a production budget.
- Production adds integration, permissions, review queues and monitoring the prototype skipped.

## What a proof of concept should prove

An AI proof of concept tests one claim: that AI can do a specific piece of your work, to a stated standard, on your own data, at an acceptable cost per item. "Can AI read our invoices?" is too vague; the answer is usually "sometimes". "Can AI assign the right general ledger code to invoices from our top 40 suppliers at least 95% of the time, with review under 30 seconds an invoice?" can be proved or disproved in weeks.

It sits between an idea and a pilot:

| Stage | Question it answers | Evidence | Who uses it |
|---|---|---|---|
| Demo | Can a model do this at all? | A few hand-picked examples | Nobody, in practice |
| Proof of concept | Does it meet our bar on our data? | A fixed test set of real cases | The project team |
| Pilot | Does it work in daily use? | Live data, real users, real review | One team, for a defined period |
| Production | Can we depend on it? | Monitoring, support, change control | Everyone it affects |

## Who it is for

Teams with a candidate workflow and a number to move, often from an [AI readiness assessment](/services/ai-readiness-assessment/). Boards that want evidence before approving a build. Buyers checking a vendor's accuracy claims against their own documents rather than the vendor's demo set; our [AI vendor evaluation checklist](/resources/ai-vendor-evaluation-checklist/) lists the questions to ask first.

## Go/no-go criteria, agreed first

The criteria are written and signed off before anything is built, so the result cannot be argued into a success afterwards. A typical set:


- Accuracy per category on a held-out test set the team never tuned against
- The worst error type, and how often it occurred, not just the average
- Review time per item compared with doing the work by hand today
- Model cost per item at expected production volume
- Response time that fits the workflow (seconds for a chat, hours for a batch)
- Data handling that passes your privacy and security review
- Staff who tried it would choose to use it


Illustrative example: a 20-person Wellington accounting practice tests invoice coding. Criteria: at least 95% correct codes on 300 held-out invoices, review under 30 seconds each, model cost under NZ$0.05 an invoice. At OpenAI's listed price for GPT-6.1 Sol on 19 October 2026 (US$2 input and US$10 output per million tokens; [OpenAI](https://developers.openai.com/api/docs/pricing)), an invoice using 3,000 input and 300 output tokens costs US$0.009, about NZ$0.016 at the European Central Bank reference rate of NZ$1.7656 per US dollar on 16 October 2026 ([Frankfurter](https://api.frankfurter.dev/v1/2026-10-16?base=USD&symbols=NZD)). Cost passes easily. Accuracy is the real test: if it comes in at 91%, the decision turns on whether the errors cluster in a few suppliers that can be excluded. The same proof of concept should also test the alternative already on the shelf: Xero's built-in AI, JAX, captures bills and receipts and reconciles bank transactions ([Xero](https://www.xero.com/nz/just-ask-xero/)), and if it clears the bar there is nothing to build. Our notes on [AI and Xero](/integrations/xero/) cover both routes.

## What we deliver


- **Criteria sheet**: The question, the go/no-go thresholds and who signs them off.
- **Test set**: Real cases, labelled by your staff, with a held-out portion nobody tunes against.
- **Working prototype**: The narrowest system that can be measured, running on your data in a sandbox.
- **Results report**: Pass or fail against each criterion, results by category, and the failure examples that matter.
- **Production gap estimate**: What it would take to run this daily: integration, permissions, review screens, monitoring and training.
- **Recommendation**: Go, go with a narrower scope, fix the data first, buy a product instead, or stop.


## How the work runs


1. **Discover: frame the question.** One workshop to choose the workflow step, agree the criteria and identify the data. Data access requests start the same day, because they are usually the slowest part.
2. **Prove: build narrow, test honestly.** We build only what the test needs, tune on part of the test set and then run the held-out cases once. Results are reported by category, including the ones that fail.
3. **Decide, then scale or stop.** You get the report and the production estimate together. If the answer is go, the test set becomes the regression suite for the production build through [custom AI development](/services/custom-ai-development/).


## What changes on the way to production

The prototype is the smallest part of the cost. A production system has to cope with everything the sandbox avoided:

| Item | In the proof of concept | In production |
|---|---|---|
| Data access | An exported sample | Live, permissioned connections to source systems |
| Inputs | A fixed test set | Every input, including formats nobody anticipated |
| Review | Engineers check results | Staff review queue with a usable screen |
| Errors | Noted in the report | Caught, retried, escalated and logged |
| Security | Sandbox only | Reviewed, with access controls and audit logs |
| Monitoring | None | Quality, cost and drift tracked weekly |
| People | The project team | Training, communication and consultation where roles change |

Our guide to [what custom AI projects cost](/services/ai-project-costs/) shows how to estimate each line.

## Where proofs of concept go wrong

- **Clean data, happy results.** A test set without the awkward cases proves nothing. Ask staff for the examples they dread.
- **Moving goalposts.** Criteria softened after the results arrive turn a no into an expensive maybe.
- **No owner for the next step.** A successful proof of concept with no budget or owner stalls. In Datacom's 2026 survey of larger NZ organisations, only 15% were scaling AI organisation-wide ([Datacom](https://datacom.com/nz/en/insights/press-release/2026-state-of-ai-Index-ai-uptake-at-91-leadership-lags)).
- **Real data without a privacy check.** The Privacy Commissioner [expects a privacy impact assessment](https://www.privacy.org.nz/assets/New-order/Resources-/Publications/Guidance-resources/AI-Guidance-Resources-/AI-and-the-Information-Privacy-Principles.pdf) before AI tools are used on personal information, and a proof of concept is no exception.

## Work this draws on

- **[World of ClaudeCraft](/portfolio/world-of-claudecraft/)**: Levy Street's free, open-source browser MMO. Its first playable version was built in about 48 hours in June 2026 with the Claude Fable 5 coding model; since then roughly 60 contributors have pushed about 19,900 commits. The prototype was the fast part, and the [repository](https://github.com/levy-street/world-of-claudecraft) shows everything that came after.
- **[Property management agent for APL Property](/portfolio/property-management-agent/)**: an internal agent handling routine administration for a regional NZ property firm, where staff keep the decisions.

## Questions people ask

### How long does an AI proof of concept take?

Weeks, not months. A proof of concept should test one workflow step on a fixed set of real cases, which rarely needs more than a few weeks of elapsed time once data access is sorted. If it needs months, it is either not narrow enough or it is really a pilot. Getting access to the data is usually the slowest part, so we start that on day one.

### What is the difference between a proof of concept and a pilot?

A proof of concept answers whether the approach works, using a fixed test set in a sandbox. A pilot answers whether it works in daily use, with real users, live data and the real review process, for long enough to see the awkward cases. Skipping the proof of concept means discovering accuracy problems in front of staff and customers.

### What happens if the proof of concept fails?

You stop, with evidence. The report shows which categories fell short and why, which often points to a smaller workflow that does work, a data problem worth fixing first, or a product that already does the job. A clear no after a few weeks is far cheaper than a production system nobody trusts.

### Can we use our real data?

Usually, with care. Real inputs give honest results, so we use them where the privacy risk is manageable: personal details masked where possible, data kept in your environment or an agreed sandbox, and vendors on terms that exclude training on your data. For highly sensitive categories we use fictional or synthetic cases, as the Law Society recommends for lawyers testing AI.

### Can the proof of concept become the production system?

Sometimes. We say at the start which parts are built to last, such as the test set, prompts and data extraction, and which are deliberately throwaway, such as a quick review screen. The test set is always kept: it becomes the regression suite that checks the production system.


## Sources

- [2026 State of AI Index: AI uptake at 91%, leadership lags](https://datacom.com/nz/en/insights/press-release/2026-state-of-ai-Index-ai-uptake-at-91-leadership-lags)
- [API pricing](https://developers.openai.com/api/docs/pricing)
- [USD to NZD reference rate, 16 October 2026 (European Central Bank data)](https://api.frankfurter.dev/v1/2026-10-16?base=USD&symbols=NZD)
- [Generative AI guidance for lawyers](https://www.lawsociety.org.nz/professional-practice/rules-and-maintaining-professional-standards/generative-ai-guidance-for-lawyers/)
- [Artificial intelligence and the Information Privacy Principles](https://www.privacy.org.nz/assets/New-order/Resources-/Publications/Guidance-resources/AI-Guidance-Resources-/AI-and-the-Information-Privacy-Principles.pdf)
- [World of ClaudeCraft repository](https://github.com/levy-street/world-of-claudecraft)
- [Meet JAX, your AI finance partner](https://www.xero.com/nz/just-ask-xero/)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.