Short answer
Multimodal AI is AI that can read images as well as text, so it can transcribe a photographed receipt, describe a site photo or pull figures from a chart. It misreads handwriting and small digits and will guess values it cannot see, so New Zealand businesses should ask for uncertainty flags, check numbers against the image, and keep photos of people out unless there is a clear lawful purpose.
- Test your tool on an image where you already know the right answer.
- Ask the AI to mark unclear readings instead of guessing.
- Check every number, date and total against the original image.
- Photos of people are personal information; facial analysis can fall under the Biometric Processing Privacy Code.
- Images can carry hidden instructions, just like emails and web pages.
- Goal
- Get reliable text and numbers out of images with AI, and know your tool’s error rate on the kind of images you use.
- Who it is for
- Field staff, administrators, office managers and owners who photograph paperwork, whiteboards, labels, gauges or site conditions.
- Time
- 25 minutes.
- You need
- ChatGPT, Claude, Copilot or Gemini on a phone or computer, a pen, a sheet of paper and a camera.
- You will produce
- A measured error count for your tool on handwriting, and a prompt you can reuse for photographed records.
What can multimodal AI do with an image?
Multimodal AI reads images as well as text. In business that mostly means four jobs: transcribing text from photos and scans, pulling structured fields from receipts and forms, describing what a photo shows, and reading values from charts or gauges. It is often impressively good, which is exactly why its mistakes slip through. A misread 7 looks as confident as a correct one.
Which image tasks are a good fit?
Some image jobs suit AI well and some need heavy checking or should be avoided:
| Image task | Fit | The check you must do |
|---|---|---|
| Printed receipts and invoices | Good | Re-add totals and GST; compare supplier names |
| Whiteboards and handwritten notes | Fair | Check every number, name and date against the photo |
| Charts and graphs | Fair | Values are estimated from pixels; ask for the underlying data instead |
| Product labels and packaging | Good for text | Check ingredients, allergens and dates word for word |
| Site, damage or equipment photos | Describe only | Never let a description replace an inspection or a qualified opinion |
| Photos showing people | Avoid | Crop or blur people out unless you have a clear lawful purpose |
Exercise: measure your tool on handwriting
You will create an image where you know the right answer, so you can count the errors. Copy the fictional log below by hand onto paper, in your normal handwriting. Make the 7 in “17.4” look a little like a 1, as rushed handwriting often does.
Illustrative maintenance log. Fictional orchard cool store,
Hawke's Bay. Copy this by hand, then photograph it.
Cool store 2, weekly check, 14 Oct
Room temp: 0.8 C Target: 0.5 to 1.0 C
Door seal: worn on hinge side, booked repair Thu
Compressor hours: 17,4?? (reads 17.4 on display, x1000)
Defrost cycles last 24h: 4
Fan 3: noisy bearing. Spare part ordered, ref KB-2291
Checked by: R.T.
- Write and photograph it. Use your phone camera in normal light. Do not straighten or crop the photo; real photos are not tidy.
- Upload with the worked example prompt to ChatGPT, Claude, Copilot or Gemini.
- Count the errors. Compare the output field by field with the text above. Count wrong numbers, wrong words and anything invented.
- Check the flags. Did the tool mark the compressor hours or any other reading as uncertain, or did it guess silently?
- Make it harder. Photograph it again at an angle or in dim light and repeat. Note how the error count changes.
- Write your rule. For example: “AI transcriptions of handwritten logs are checked against the photo before entry, and every flagged field is confirmed with the person who wrote it.”
Worked example: a transcription prompt with uncertainty flags
Transcribe this photographed maintenance log into a table with
columns: Field, Value, Confidence (high/medium/low).
Rules:
- Copy exactly what is written. Do not correct, complete or
interpret values.
- If any character is unclear, write your best reading followed
by [?] and set Confidence to low.
- Do not add information that is not visible in the image.
- After the table, list every low-confidence field and what
someone should check.
Example output (illustrative and abridged; your tool’s wording will differ):
Field | Value | Confidence
Room temp | 0.8 C | high
Compressor hours | 17.4 (x1000) [?] | low
Defrost cycles | 4 | high
Fan 3 | noisy bearing, ref | medium
| KB-2291 [?] |
Check: compressor hours could read 11.4 or 17.4; confirm against
the display. Part reference KB-2291 may be KB-229I; confirm
against the order.
This is the behaviour you want: the tool shows where it is unsure. If yours guessed instead, the rules in the prompt are not optional.
What about generating images?
Generating images is a separate skill with separate risks. Do not depict real people without consent, do not show results your product does not deliver, and proofread any text inside the image. The Privacy Commissioner co-signed an international joint statement on AI-generated imagery in February 2026 reminding organisations that these systems must follow privacy rules, and its guidance on AI and the privacy principles treats made-up information about a person, including deepfake images, as personal information.
Where this goes wrong
- Silent guesses. Without explicit rules, models fill unclear characters with plausible ones. Always ask for flags.
- Charts rebuilt from pixels. A model reading a bar chart estimates values. If you have the spreadsheet, use it instead.
- People in the frame. Faces in site photos, staff in whiteboard shots, customers in store photos. Automated processing of faces can fall under the Biometric Processing Privacy Code, in force since 3 November 2025.
- Hidden instructions. The OWASP Top 10 for LLM Applications 2026 lists instructions embedded in images among prompt injection routes. See recognising prompt injection.
- Descriptions treated as assessments. “The roof appears undamaged” is not an inspection.
Check your work
- I made a test image with a known answer and counted my tool’s errors.
- My tool marked unclear readings, or I know it does not and check everything.
- I re-check every number, date and total from images before using it.
- I crop or blur people out of photos before uploading them.
- I know which image tasks in my job are a good fit and which need a person.
Where to go next
Next in the pathway: understanding retrieval augmented generation, which explains how AI answers from your own documents. If your images include customer details, read protecting customer information when using AI. For a real example of photographed records, see the agriculture workflow on equipment maintenance logs. Log the kinds of images your team uploads in an AI data inventory. Engineers choosing an extraction approach should read OCR versus vision models, and when image volumes justify a proper system, see our AI document processing service.
Sources
- AI and the Information Privacy Principles, Office of the Privacy Commissioner, September 2023
- Biometric Processing Privacy Code, Office of the Privacy Commissioner, Accessed 19 October 2026
- Joint statement on AI generated imagery, Office of the Privacy Commissioner, 23 February 2026
- OWASP Top 10 for LLM Applications 2026, OWASP GenAI Security Project, August 2026
Written by the Levy Street team in Wellington. Last updated 19 October 2026.