RAG versus fine-tuning.

What do RAG and fine-tuning each change?

RAG changes what the model can see when it answers; fine-tuning changes how the model behaves. That distinction settles most decisions before any cost arithmetic.

Retrieval-augmented generation, introduced by Lewis et al. in 2020, pairs a model with an external store it searches at answer time. The authors framed it around two problems with knowledge held only in model weights: providing provenance for answers and updating what the model knows. Both still apply. Fine-tuning trains the model on examples of inputs and desired outputs, so it learns a format, tone, label set or decision pattern. OpenAI’s model optimisation guide lists the benefits as more examples than fit in a prompt, shorter prompts, training on data you do not want to send with every request, and training a smaller, cheaper model to excel at a narrow task. Note what is missing from that list: keeping facts current.

Which should you use for your requirement?

Map each requirement to the technique that addresses it. Where both columns say “no”, the answer is usually better prompting or structured outputs.

Our lesson on understanding retrieval augmented generation explains the retrieval side for non-engineers. If your documents are small enough to fit in the context window, also consider skipping retrieval entirely; see RAG versus long context.

What has changed for fine-tuning in 2026?

The hosted fine-tuning market has shrunk, so check availability before planning around it. As listed on 19 October 2026:

• OpenAI. Its deprecations page says self-serve fine-tuning closed to organisations that had never used it on 7 May 2026, closed to those without recent fine-tuned inference on 2 July 2026, and stops accepting new jobs from active customers on 6 January 2027. Fine-tuned models keep serving until their base model is deprecated, and they follow the base model’s shutdown schedule.
• Google. The Gemini API documentation says no model in the Gemini API or AI Studio currently supports fine-tuning. Google’s enterprise platform supports supervised tuning of Gemini 3.5 Flash, 3.1 Flash-Lite and some 2.5 models, which trail the newest models.
• Open-weight models. Parameter-efficient methods remain the practical route. LoRA freezes the base weights and trains small low-rank matrices; its authors report 10,000 times fewer trainable parameters and a third of the GPU memory compared with full fine-tuning of GPT-3 175B. See running open models for business.

The practical consequence: a fine-tuned frontier model ties you to a base model with a fixed end date, and every migration means rebuilding the training set and re-running evaluations. Our guide to AI model migration covers that cost.

How do you run a same-task comparison?

Decide with a three-arm experiment on the same held-out evaluation set, measuring quality, cost, latency and the cost of keeping each arm current.

1. Build the evaluation set first. Real inputs with reference outputs, split into a training pool (for fine-tuning examples and prompt examples) and a sealed test set. For retrieval tasks, label relevant passages as described in RAG retrieval evaluation.
2. Arm A: prompt only. A strong model with clear instructions, a handful of examples and structured outputs. This is the baseline every other arm must beat.
3. Arm B: RAG. Same model, plus retrieval over your documents. Measure recall@k, answer correctness and citation precision.
4. Arm C: fine-tuned. A smaller model trained on the training pool, with or without retrieval. OpenAI’s supervised fine-tuning guide reports improvements from 50 to 100 examples and advises rethinking the task if 50 good examples have no effect.
5. Score all arms on the same sheet. Accuracy on the sealed set, cost per 1,000 requests, p95 latency, examples or documents needed, and the hours needed to absorb one change (a new policy, a new label).

The last column decides more projects than accuracy does. A fine-tuned classifier that is two points more accurate but needs a week of relabelling and retraining for every new category often loses to a prompt that takes ten minutes to edit.

What does the cost arithmetic look like?

Run the numbers at your volume with current prices, because prompt caching has changed the answer. Two illustrative examples, using list prices in USD as listed on 19 October 2026.

Example 1: staff policy questions with RAG. 20,000 questions a month, each with a 1,500-token system prompt, 4,500 tokens of retrieved passages and a 400-token answer, on Claude Sonnet 5.5 at US$2 per million input tokens and US$10 per million output tokens (Anthropic pricing). Input is 120 million tokens (US$240) and output 8 million tokens (US$80): about US$320 a month before caching. Fine-tuning would not remove the retrieved passages, because the policies change.

Example 2: classifying 500,000 emails a month into 40 categories. Prompt-only with a 3,000-token shared prefix of category definitions and examples, 300 tokens of email and a 10-token label, on GPT-6.1 Sol at US$2.00 input, US$0.10 cached input and US$10.00 output per million tokens (OpenAI pricing).

A fine-tuned small model with a 300-token prompt would need to beat roughly US$500 a month all-in, including training runs, hosting or per-token premiums, evaluation and the engineering time to retrain it. At this volume that is possible, but it is no longer obvious. Our guide to managing LLM costs covers caching break-even points in detail.

Where does each approach fail?

Both approaches fail in predictable ways, and the failures point to the controls you need.

• Fine-tuning to teach facts. The model answers confidently from trained knowledge that was already out of date at training time, and cannot say where an answer came from.
• Forgetting and drift. Narrow training can weaken general instruction following or change refusal behaviour. Re-run your broad regression suite, not just the task suite, after every training run.
• Personal information in weights. Correcting or removing information held in weights generally means retraining. Under the Privacy Act 2020, people can ask to access and correct personal information, which is straightforward in a document store and awkward in a model.
• Base model shutdown. Fine-tuned models inherit their base model’s end date, which forces a rebuild on the provider’s timetable.
• RAG retrieval misses. If the right passage is not retrieved, the answer is wrong however good the model is. Measure retrieval separately and fix it first.
• RAG over messy sources. Scanned PDFs, duplicates and stale versions defeat retrieval. Budget for AI data preparation before tuning anything.

What do we recommend by default?

Start with prompting and structured outputs, add retrieval when the task depends on your documents, and fine-tune only when a same-task test shows a gap that the other arms cannot close. In our RAG development work that order holds for most business systems, and AI model selection is where we run the comparison when a fine-tuned or open-weight model is a serious contender.

• The requirement is written down as knowledge, behaviour or both
• Facts that change, need citations or are permissioned go through retrieval
• A sealed evaluation set exists before any training starts
• Prompt-only, RAG and fine-tuned arms are scored on the same set
• Cost is calculated at real volume, with prompt caching included
• The time to absorb one change is measured for each arm
• Provider fine-tuning availability and base model end dates are checked
• Personal information is kept out of training data unless there is a clear reason
• A broad regression suite runs after every training run
• The decision and its numbers are recorded for the next model migration

Written by the Levy Street team in Wellington. Last updated 19 October 2026.