Evaluating translation quality with AI.

Goal

Judge whether an AI translation is fit to use, and decide what level of human review it needs before anyone relies on it.

Who it is for

Managers, support staff and communications teams in New Zealand businesses that write to customers or staff in more than one language.

Time

30 minutes.

You need

Two AI chat tools if possible (for example ChatGPT and Claude, or Copilot and Gemini), the fictional notice below, and ideally a colleague who reads the target language.

You will produce

A checked translation, an error list with severities, and a review decision.

Why check AI translations at all?

Because AI translations read fluently even when they are wrong. A dropped “not”, a date flipped from New Zealand to US order, or a polite instruction turned into a blunt order looks perfectly natural to someone who cannot read the language. The person sending the message is usually the one least able to spot the problem, so the checks have to be deliberate.

What kinds of error should you look for?

Check each category separately; one pass looking for “anything wrong” misses most problems.

Which translations need a human reviewer?

Decide by risk before you translate, not after. Low-risk text, such as understanding an incoming supplier email, can rely on AI with a spot-check. Medium-risk text, such as routine customer replies, needs a check by someone who reads the language. High-risk text, including safety instructions, employment terms, legal or health information and anything published, needs a fluent reviewer or a professional translator, with AI used only to draft or to help the reviewer.

Te reo Māori deserves particular care. Te Taura Whiri i te Reo Māori’s translation request guidance explains that it does not translate for the public; it certifies translators and interpreters and keeps a national register. It notes that not everyone who speaks Māori is a translator, that translation is a skill built through training, and that it recommends one of its certified translators so public-facing translations are of high quality. Do not publish machine-translated te reo Māori without a fluent reviewer.

Exercise: check a translated notice

The packhouse and notice are fictional. The English contains deliberate ambiguities.

Illustrative notice. Fictional fruit packhouse, Hastings.

Kia ora team,
From Fri 7/11 the night shift starts at 6pm, not 7pm.
Safety briefing for all new starters: Level 2 lunchroom, Monday
8am. You must not enter the cool store without a supervisor.
Pay day moves to Wednesday. If your bank details have changed,
see Lisa in the office, not by email.
Hi-vis and steel caps every shift, no exceptions. No worries if
you have questions, just touch base with your team leader.

1. Pre-edit the source. Ask an AI tool: “List anything in this notice that could be misread when translated, and suggest clearer English.” Expect the date, “Level 2”, “no worries” and “touch base”. Rewrite the notice with fixes, such as “Friday 7 November”.
2. Translate. Translate the clearer version into a language you or a colleague can read, such as Samoan, Tongan, Hindi or Simplified Chinese.
3. Evaluate in a second tool. Paste the source and translation into a different AI tool with the worked example prompt below.
4. Back-translate. In a fresh chat, translate the output back into English without showing the original. Compare meaning, especially “must not” and the bank details instruction.
5. Check figures by hand. Times, dates, days and names, one by one.
6. Decide the tier. This notice includes safety rules, so a fluent bilingual supervisor or translator must review it before it goes on the wall.

Worked example: the evaluation prompt

You are a careful translation reviewer. Compare the SOURCE and the
TRANSLATION below. List every problem in a table with columns:
Line | Error type (meaning changed / omission / addition / number or
date / terminology / tone / name) | What the translation says |
What it should convey | Severity (critical / major / minor).

Critical = could cause harm, a safety breach or a financial loss.
Then list any phrases you are not confident about, so a fluent
human reviewer can check them. Do not rewrite the translation.

SOURCE:
[paste]
TRANSLATION:
[paste]

Example output (illustrative and abridged; your tool’s wording will differ):

Line 4 | meaning changed | "you should not enter the cool store
alone" | must not enter without a supervisor | Critical

Line 5 | omission | instruction not to send bank details by email
is missing | see Lisa in person, not by email | Major

Line 2 | number or date | date written in month-day order |
Friday 7 November | Major

Not confident: the word chosen for "steel caps" may not be the
everyday term workers use. Ask a fluent reviewer.

The “not confident” list is as valuable as the error table. It tells your human reviewer exactly where to look.

Where this goes wrong

• Translating unclear English. Idioms and abbreviations come out confidently wrong. Pre-edit every time.
• Checking with the same chat that translated. It tends to approve its own work. Use a fresh session or a second tool.
• Replies sent without a reader. Customer replies in another language still need the review step in drafting emails with human review.
• Treating back-translation as proof. It misses tone and cultural problems.
• Sending Māori content offshore without thought. MBIE’s responsible AI guidance points businesses to the Principles of Māori Data Sovereignty from Te Mana Raraunga, which say Māori data should be stored in Aotearoa whenever possible.
• Personal details in source text. Remove names, addresses and bank details before pasting; see protecting customer information when using AI.

Check your work

• I pre-edited the source and removed idioms, ambiguous dates and jargon.
• A second tool or fresh session evaluated the translation against the error categories.
• I back-translated and compared meaning, not wording.
• I checked every number, date, time and name by hand.
• I assigned a risk tier and the right reviewer before sending anything.
• I know how to find a certified te reo Māori translator for public-facing work.

Where to go next

Next in the pathway: using AI for accessibility checks, another way to make sure your message reaches everyone. The review tiers here come from planning human review for AI output, and the same claim-by-claim habit appears in checking AI answers against sources; use the AI quality review checklist to record decisions. Teams translating at volume should read monitoring AI quality, and our LLM evaluation services can build a scored test set for your language pairs.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.