OCR versus vision models.

How OCR and vision models differ

OCR engines detect lines of text and recognise characters; the output is text with positions and confidence scores, and turning that into “invoice total” is a separate step. Vision language models take the page image and a question and return an answer or a structured record directly. Tesseract, the main open-source engine, has used a neural line-recognition engine since version 4 and is at major version 5; cloud document services from the large providers add layout and table detection on top.

The difference that matters most in production is how each one fails.

Extraction is one stage of a wider ingestion pipeline, covered in AI data preparation. For teams starting out, our lesson on working with images in business AI covers the basics without code.

The metrics to use

Character error rate (CER) is the standard transcription measure: CER = (S + D + I) / N, where S, D and I are the character substitutions, deletions and insertions needed to turn the output into the reference text, and N is the number of characters in the reference. Word error rate is the same calculation over words.

def edit_distance(ref: str, hyp: str) -> int:
    prev = list(range(len(hyp) + 1))
    for i, r in enumerate(ref, start=1):
        cur = [i]
        for j, h in enumerate(hyp, start=1):
            cur.append(min(prev[j] + 1,               # deletion
                           cur[j - 1] + 1,            # insertion
                           prev[j - 1] + (r != h)))   # substitution or match
        prev = cur
    return prev[-1]

def cer(reference: str, hypothesis: str) -> float:
    return edit_distance(reference, hypothesis) / max(len(reference), 1)

CER alone is the wrong headline for extraction, because one wrong digit in an account number costs more than a hundred misread words in a footer. Report these alongside it:

Run a benchmark on your own documents

Public benchmarks such as OmniDocBench, which covers PDF parsing across nine document sources and tests both pipeline tools and end-to-end vision models, are useful for shortlisting. The decision needs your documents.

1. Sample by stratum. Take 200–500 real documents across the variation you receive: digital PDFs, clean scans, phone photos, faxes, handwritten forms, multi-page documents.
2. Label twice. Two people label every required field; a third resolves disagreements. Their agreement rate is your ceiling.
3. Write normalisation rules first. Dates are day-first in New Zealand (03/04/2026 is 3 April); amounts drop currency symbols and thousands separators; GST numbers and NZBNs keep digits only.
4. Run three pipelines. A: text layer or OCR, then a text model maps fields. B: a vision model reads the page image. C: both, with cross-checks between them.
5. Score per stratum. Report field accuracy, straight-through rate and unsupported value rate for each document type, with confidence intervals, plus cost and time per page.

Cost per page, and the cost that matters more

Model fees are simple arithmetic. Illustrative example with assumed numbers, to be replaced with your provider’s current prices and measured token counts: a page image of about 1,500 tokens plus an 800-token prompt and a 300-token output, at an assumed US$1 per million input tokens and US$4 per million output tokens, costs (2,300 × 1 + 300 × 4) ÷ 1,000,000 = US$0.0035 per page, or US$3.50 per 1,000 pages.

The larger cost is usually review. Suppose 10,000 documents a month, two minutes to fix each failed document, and straight-through rates of 80% for one pipeline and 92% for another. That is 2,000 versus 800 documents to fix, or about 67 versus 27 hours of staff time a month. A pipeline that costs more per page can easily be the cheaper one. Managing LLM costs covers the model side in more depth.

When to use which

The cross-check is simple and catches the vision model’s worst failure, values that are not on the page:

import re

NUMBER = re.compile(r"\d[\d,./-]*\d|\d")

def number_tokens(text: str) -> set[str]:
    return {re.sub(r"\D", "", t) for t in NUMBER.findall(text)}

def supported_by_ocr(value: str, ocr_text: str) -> bool:
    """A numeric field returned by the vision model must appear in the OCR text."""
    wanted = re.sub(r"\D", "", value)
    return bool(wanted) and wanted in number_tokens(ocr_text)

Fields that fail the check go to a human review queue instead of straight through. OCR confidence is a second routing signal: if any character in a critical field falls below your confidence threshold, review the document even when the vision model and OCR agree, because both may have guessed the same smudged digit. Route at page level too: classify each page as digital, clean scan, photo or handwritten first, then send it down the pipeline that won its stratum in the benchmark. Validate every extracted record against a schema and business rules as well, such as line items summing to the total; see structured outputs and validation.

Where document extraction goes wrong

OCRBench’s evaluation of large multimodal models found particular weaknesses with multilingual, handwritten and non-semantic text such as random character strings, according to the paper. Models have improved since, but invoice numbers and account codes are exactly that kind of text, so test them specifically.

Workflows such as proof of delivery extraction and invoice coding and exception review live or die on these checks, and they are the first thing we measure in AI document processing projects.

• Digital PDFs are read from the text layer before any OCR or vision model is used
• A benchmark set of real documents is stratified by type and quality and double-labelled
• Normalisation rules (day-first dates, amounts, GST numbers) are written before scoring
• Field accuracy, straight-through rate and unsupported value rate are reported per stratum
• CER is measured for transcription tasks, with the formula and normalisation recorded
• Cost per page includes review time for failed documents
• Numeric fields from vision models are cross-checked against OCR text
• Extracted records pass schema and arithmetic checks before use
• Empty answers are allowed for missing or illegible fields
• High-consequence fields route to review until measured accuracy supports automation

Written by the Levy Street team in Wellington. Last updated 19 October 2026.