Short answer
AI chatbot development is building a conversational assistant that answers questions or completes tasks from your own approved content and systems, and hands over to a person when it should. Levy Street builds chatbots for New Zealand organisations around one defined job, with written escalation rules, answers grounded in your sources, and quality measured on real conversations before and after launch.
- Scope a chatbot to one job and test it on real questions from your inbox.
- Answers come from approved content; the bot says when it does not know.
- Escalation rules are written down before launch, not discovered by customers.
- A business is answerable for what its chatbot tells customers.
What a useful chatbot does
A useful chatbot has one job, such as answering policy questions from your documents, checking an order, or booking an appointment, and does it from content and systems you control. General-purpose assistants for staff are better bought than built; see Microsoft Copilot or Claude for those. We build chatbots when the answers depend on your own content, rules or systems, or when the chat experience is part of what you sell.
Chatbots come in two broad forms. Customer-facing bots sit on a website, app or messaging channel and answer the public. Internal bots answer staff questions about policies, procedures or technical knowledge, usually inside Microsoft Teams or Slack. The second kind is lower risk and often the better first project.
Who it suits
- Repetitive enquiries. The same 30 questions make up most of your inbox or chat volume, and the answers are in documents you already maintain.
- After-hours demand. People ask questions at night and on weekends and wait until morning for simple answers.
- Scattered internal knowledge. Staff ask colleagues where a policy or procedure is because searching for it takes too long.
- A product built on conversation. The chat is the service your customers pay for.
The demo we build first
Before quoting a full build, we build a task-specific demo on your material. We take 50 to 100 real questions from your inbox, tickets or chat logs, including the awkward ones, load the approved content that should answer them, and show you the results side by side: the answer, the source it came from, and the questions the bot correctly refused or escalated. Our lesson on building customer FAQs from support tickets shows how to gather those questions yourself.
Escalation rules and quality measures
A chatbot is safe to launch when it knows when to stop and its quality is measured on more than one number. Both are agreed in writing before launch.
Escalation rules
Escalation rules are written and agreed before launch. A typical starting set:
| Trigger | What the bot does | Who takes over |
|---|---|---|
| The person asks for a human | Hands over at once with the transcript | Staff queue or live chat |
| Complaint, distress or a safety concern | Stops answering the topic and gives direct contact details | Named role, by phone if urgent |
| Question not covered by approved content | Says it does not know and offers a handover | Staff queue |
| Refunds, pricing exceptions or credit | Collects details, makes no commitment | Account or sales team |
| Request to access or correct personal information | Routes the request without handling it | Privacy officer |
| Two attempts fail to resolve the question | Offers a handover | Staff queue |
How we measure quality
| Measure | What it tells you |
|---|---|
| Answer accuracy on the test set | Whether answers match your approved content |
| Citation correctness | Whether the cited source actually supports the answer |
| Unsupported answer rate | How often the bot states things not in its sources |
| Escalation accuracy | Whether handovers happen when the rules say they should |
| False refusal rate | How often it refuses questions it could have answered |
| Time to a person | How long someone waits after asking for a human |
| Containment | Share resolved without a person; only meaningful alongside the measures above |
What you receive
- Scope and rules
- The bot’s one job, its approved sources, its refusals and its escalation rules, signed off by you.
- Retrieval and answer layer
- Search over your approved content with citations, built as described in RAG development.
- Integrations
- Read access to the systems it needs for tasks such as order lookups or bookings.
- Test set and results
- Real questions with expected answers, scored before launch and on every change.
- Conversation review
- Sampled transcripts, ratings and flagged answers in one place.
- Content process
- Who updates the source content, and how changes reach the bot.
How the build runs
- Discover: one job, real questions. Agree the job, collect real questions, and identify the approved content and systems.
- Prove: the demo. Build the task-specific demo and score it on the test set. Decide go or no-go on evidence.
- Prove: staff-only pilot. Staff use it first, or it drafts answers for staff to send, so errors are caught internally.
- Scale: public launch and review. Launch with clear disclosure and an easy route to a person, then review sampled conversations weekly.
What drives the cost
The drivers are the number of channels (website, app, Teams, messaging), the systems it must read or act on, the state of your source content, the languages you need, conversation volume, and how much conversation review you want. Cleaning up contradictory content is often the largest hidden task. We scope after the demo. See what custom AI projects cost and size the saving with the AI ROI calculator.
Conversational products we have worked on
- PainPal is an AI “digital workmate” for musculoskeletal pain and injury prevention, launched by BetterWorx (a subsidiary of CHASNZ) in December 2025. Levy Street worked with BetterWorx on PainPal.
- Singing Coach AI is an AI vocal coach on iPhone, Android and the web. It scores a recording across 26 vocal dimensions and gives coaching, with feedback in a minute or two, and includes a coach chat that answers questions with exercises matched to the singer’s profile.
Where chatbots go wrong
- Confident wrong answers. In Moffatt v. Air Canada (February 2024), a Canadian tribunal held the airline responsible for incorrect fare advice its website chatbot gave, saying “It makes no difference whether the information comes from a static page or a chatbot.” It is not New Zealand law, but the logic is familiar here.
- Prompt injection. Users try to make the bot ignore its rules. Test for it, and never give a public bot access it should not have. See prompt injection testing.
- Stale content. The bot answers from last year’s price list. Content ownership is part of the build.
- Hiding the humans. A bot that makes it hard to reach a person damages trust. Xero’s 2026 survey found 55% of NZ small and medium businesses worry AI in customer communications makes them feel less human.
Do not build a chatbot if you get only a handful of questions a week, if your source content is contradictory and nobody owns it, or if the questions need judgement about individual circumstances, such as health, legal or financial advice.
New Zealand specifics
- Fair trading. The Fair Trading Act 1986 prohibits misleading or deceptive conduct in trade, which covers what a chatbot tells customers. The Commerce Commission’s July 2025 paper on AI warns businesses not to oversell AI capability.
- Consumer guarantees. Services supplied to consumers must be carried out with reasonable care and skill under the Consumer Guarantees Act 1993, including services delivered through a chatbot.
- Privacy. Chat transcripts are personal information under the Privacy Act 2020. Tell people what is collected and why (IPP 3), keep transcripts only as long as needed (IPP 9), and check where your chatbot vendor processes them under the 14 information privacy principles.
Sources
- Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal (British Columbia), 2024-02-14
- Fair Trading Act 1986, New Zealand Legislation, version as at 1 July 2026
- Consumer Guarantees Act 1993, New Zealand Legislation, version as at 24 January 2026
- Navigating the rise of AI: Perspectives from a competition and consumer regulator, Commerce Commission, 2025-07
- Techweek: barriers preventing Kiwi SME AI adoption, Xero, 2026-05-18
- Privacy Act 2020, New Zealand Legislation, version as at 1 October 2026
Written by the Levy Street team in Wellington. Last updated 19 October 2026.