AI data preparation.

What data preparation means for AI systems

Data preparation for language model systems is mostly document engineering. The inputs are PDFs, Word files, emails, spreadsheets and web pages, and the outputs serve three different consumers: a retrieval corpus the system searches at run time, evaluation sets that measure it, and occasionally a fine-tuning set. Each needs the same core pipeline and different final checks.

The goal is traceability as much as cleanliness. When an answer is wrong, you should be able to name the source file, its version, the chunk that was retrieved and the pipeline run that produced it. Without that, every bad answer becomes an argument about whether the model or the data is at fault.

The ingestion pipeline, stage by stage

Build ingestion as discrete stages, each with an automated check and a defined action when the check fails.

The AI data inventory template is a good starting point for the first stage. Scanned documents with no text layer are the most common extraction failure; the choice between OCR and vision models is covered in OCR versus vision models.

Automated quality checks

Write checks as code that runs on every batch, and gate publication on the failure rate. A document that fails a check is queued for review; a batch where too many fail is not published at all.

import hashlib
import re
import unicodedata

NZ_BANK_ACCOUNT = re.compile(r"\b\d{2}-\d{4}-\d{7}-\d{2,3}\b")

def normalise(text: str) -> str:
    text = unicodedata.normalize("NFC", text)   # one form for macrons
    text = text.replace(" ", " ")          # non-breaking spaces from Word and PDFs
    return re.sub(r"[ \t]+", " ", text).strip()

def content_id(text: str) -> str:
    return hashlib.sha256(normalise(text).encode("utf-8")).hexdigest()

def check_document(doc) -> list[str]:
    problems = []
    if doc.page_count and len(doc.text) / doc.page_count < 200:
        problems.append("little text per page: scanned without OCR?")
    if "�" in doc.text:
        problems.append("replacement characters: encoding damage")
    if not doc.meta.get("owner") or not doc.meta.get("effective_date"):
        problems.append("missing owner or effective date")
    if NZ_BANK_ACCOUNT.search(doc.text) and not doc.meta.get("privacy_reviewed"):
        problems.append("bank account number present without privacy review")
    return problems

def gate(batch, max_failure_rate=0.02):
    flagged = {d.id: p for d in batch if (p := check_document(d))}
    rate = len(flagged) / max(len(batch), 1)
    if rate > max_failure_rate:
        raise RuntimeError(f"{rate:.1%} of documents failed checks; snapshot not published")
    return flagged  # publish, and send flagged documents to review

The thresholds are starting points. Tune them on your own corpus after looking at what each check actually catches.

Normalisation details that change answers

Small encoding differences break search in ways that look like model errors.

• Macrons. The letter ā can be stored as one precomposed character or as a plain a followed by a combining macron. The two look identical and do not match in keyword search or exact comparisons. Normalising to NFC, as defined in Unicode Normalization Forms, makes Māori, Whangārei and Ōtautahi match however they were typed.
• Dates. New Zealand documents use day-first dates, and 03/04/2026 is 3 April, not 4 March. Parse with an explicit day-first rule and store ISO 8601. Never let a model infer the format.
• Money. Record whether amounts include GST. A supplier quote and an invoice for the same job can differ by exactly 15% for that reason alone.
• Repeated boilerplate. Headers, footers and confidentiality notices repeated on every page dominate retrieval if left in. Strip them before chunking.
• Spreadsheets. Merged cells, hidden rows, formulas and multiple tables on one sheet do not survive naive text conversion. Extract each table with its header row repeated on every chunk, and record the sheet name and cell range as provenance.

Deduplication and versioning

Duplicates and superseded versions cause more wrong answers in business corpora than any model limitation. The same policy saved in three folders gets three times the retrieval weight; last year’s version of a pricing schedule answers questions as confidently as this year’s.

Deduplicate in two passes: exact matches on a normalised content hash, then near-duplicates by text similarity, keeping one canonical copy. Duplication also matters for evaluation. Research on deduplicating training data found train-test overlap affecting over 4% of the validation sets of standard datasets, which inflates measured performance. The same thing happens when an evaluation question is answered by a near-copy of a document used to write the prompt’s examples.

For versions, store an effective date and a status on every document, keep only current versions in the live corpus, and keep superseded versions in an archive that can be searched deliberately (“what did the policy say in 2024?”).

Personal information and purpose

Detect personal information before indexing, then mask, restrict or exclude it depending on whether the task needs it. Open-source tools such as Microsoft’s Presidio detect and redact common categories and accept custom recognisers. Add New Zealand patterns yourself: IRD numbers, NHI numbers, NZ driver licence numbers and bank account numbers in the 2-4-7-2 digit format.

Purpose matters as much as detection. Under the Privacy Act 2020, IPP10 limits using personal information for purposes other than the one it was collected for, and IPP9 limits how long it can be kept. Reusing years of customer emails to build an evaluation set can raise both questions. Our lesson on protecting customer information when using AI covers the practical steps, and building private AI search covers carrying source permissions into the index.

Evaluation and training sets need extra care

Evaluation data needs the same pipeline plus three rules:

1. Split by time or entity. Hold out later months, or whole customers and suppliers, so near-identical cases do not appear on both sides of the split.
2. Deduplicate across splits. Run the near-duplicate pass over training, development and test sets together.
3. Document the dataset. Record motivation, composition, collection process and recommended uses, as proposed in Datasheets for Datasets, and version it with the system that uses it. See versioning prompts and evaluations.

Where real examples are scarce, generated ones can fill gaps, with limits covered in testing AI with synthetic data.

Where data preparation goes wrong

Retrieval quality is the downstream test of all of this; measure it as described in RAG retrieval evaluation. Document-heavy pipelines are the core of our AI document processing work.

• Every source has an owner, a purpose and access rules recorded before ingestion
• Extraction checks text per page and routes low-text documents to OCR or review
• Text is normalised to Unicode NFC; dates are parsed day-first and stored as ISO 8601
• Amounts record currency and whether GST is included
• Exact and near-duplicates are removed, with one canonical copy kept
• Every document has an effective date and status; superseded versions are excluded from the live corpus
• Personal information detection includes New Zealand identifiers, with an action for each type
• Every chunk carries source, version, page and section
• Batches are gated on failure rate, and snapshots are versioned with a diff report
• Evaluation sets are split by time or entity, deduplicated across splits and documented

Written by the Levy Street team in Wellington. Last updated 19 October 2026.