Running open models for business.

When running your own model makes sense

Run an open-weight model yourself when control matters more than peak capability. Hosted APIs from OpenAI, Anthropic and Google remain the default for most business systems because they are capable, cheap at low volume and someone else’s operations problem. Self-hosting earns its keep in specific situations.

“Open” usually means open-weight: you can download and run the model, but licences vary from permissive to custom terms. Meta’s Llama 3.1 Community License, for example, requires a separate licence from Meta for products with more than 700 million monthly active users. Read the licence of the exact model you deploy.

Size the hardware from memory, not parameter count

GPU memory is the binding constraint, and it has two parts: the weights and the KV cache (the attention keys and values stored for every token in every active sequence). Weights are fixed; the KV cache grows with context length times concurrent users, and on long-context workloads it often exceeds the weights.

def serving_memory_gb(params_billion: float, weight_bits: int, layers: int, kv_heads: int,
                      head_dim: int, context_tokens: int, concurrent_seqs: int,
                      kv_bytes: int = 2, overhead: float = 0.15) -> float:
    weights = params_billion * 1e9 * weight_bits / 8
    kv_per_token = 2 * layers * kv_heads * head_dim * kv_bytes   # keys and values
    kv_cache = kv_per_token * context_tokens * concurrent_seqs
    return (weights + kv_cache) * (1 + overhead) / 1e9

Worked example, using a hypothetical 32-billion-parameter dense model with 64 layers, 8 KV heads and a head dimension of 128 (read the real values from the model’s config file):

• Weights at 4-bit: 32 × 10⁹ × 0.5 bytes = 16 GB.
• KV cache at 16-bit: 2 × 64 × 8 × 128 × 2 = 262,144 bytes per token. For 8 users at 32,768 tokens each, that is 262,144 tokens × 262,144 bytes ≈ 68.7 GB.
• Total with 15% overhead: (16 + 68.7) × 1.15 ≈ 97 GB. That does not fit on one 80 GB GPU.
• Options: an 8-bit KV cache halves it to about 34 GB, bringing the total to about 58 GB; or cap context at 16,000 tokens; or serve 4 users at a time.

Mixture-of-experts models need memory for all parameters but compute for only the active ones, so they are fast for their size and still memory-hungry.

Choose a serving stack

Pick the serving software by where it runs and how many people it serves. All three below are open source and widely used as of October 2026.

Keep the same API contract for hosted and self-hosted models, so you can switch with configuration rather than code. vLLM’s OpenAI-compatible server makes this straightforward:

from openai import OpenAI

client = OpenAI(
    base_url="http://gpu-01.internal:8000/v1",  # your vLLM server; a hosted API differs only here
    api_key="not-used-internally",
)

response = client.chat.completions.create(
    model="your-open-model",
    messages=[{"role": "user", "content": "Classify this supplier email: ..."}],
    temperature=0,
)
print(response.choices[0].message.content)

Evaluate capability before you buy hardware

Rent capacity for a week and run your own evaluation suite before committing to hardware. Model cards report results at full precision on public benchmarks; you will deploy a quantised model on your tasks, and those numbers can differ.

1. Pick candidates. Two or three open models that fit your memory budget, plus the hosted model you would otherwise use as a baseline.
2. Run your task suite. The same cases, scorers and prompts for every candidate, at the quantisation you will deploy. See AI agent evaluations for building one.
3. Check structure, not just answers. Measure schema-valid output rate and tool-call accuracy; smaller models often fail here first. Our guide to structured outputs and validation covers the checks.
4. Load test. Measure tokens per second per user and p95 latency at your expected concurrency, not with one request at a time.
5. Decide on the gap. Write down the quality difference against the hosted baseline and whether the workflow can tolerate it.

Throughput and latency trade off against each other under batching; reducing AI latency covers the measurement method.

Compare costs honestly

Self-hosting swaps a variable cost for a fixed one. The break-even volume is the fixed monthly cost divided by the hosted price per million tokens.

Illustrative arithmetic, with assumptions you should replace with real quotes:

• Workload: 2,000 tasks a day at 6,000 tokens each, about 360 million tokens a month.
• Hosted: at an assumed blended US$2 per million tokens, about US$720 a month.
• Self-hosted: an assumed US$3,500 a month for a dedicated GPU server (rented, or purchase amortised, with power and hosting), plus engineering time for patching, monitoring and upgrades.
• Break-even: 3,500 ÷ 2 = 1,750 million tokens a month, about 58 million tokens a day, or roughly 9,700 of these tasks a day.

At that workload the hosted API is about five times cheaper before counting engineering time. The numbers move with real prices, but the shape rarely does: self-hosting is a control decision first and a cost decision only at high, steady volume. The AI cost calculator and our guide to managing LLM costs help with the hosted side.

New Zealand hosting options

If the reason for self-hosting is keeping data in New Zealand, you have onshore cloud options as well as your own servers. AWS opened its Asia Pacific (New Zealand) Region (ap-southeast-6, three Availability Zones) in September 2025, and Microsoft’s New Zealand North Azure region is in Auckland. Check that the GPU instance types you need are offered there and that your account has quota. Our guide to data residency for AI systems compares hosted model options by region.

Where self-hosting goes wrong

New Zealand’s National Cyber Security Centre published joint guidance on AI and machine learning supply chain risks in March 2026, covering models, data, software, infrastructure and third-party services. When you upgrade, treat it as a model migration with a regression suite. If you want a second opinion on whether to self-host at all, that is part of our AI model selection work.

• The reason for self-hosting is written down: data control, version control, offline use or volume
• The model’s licence has been read and recorded for the exact version deployed
• GPU memory is sized from weights plus KV cache at peak concurrency plus overhead
• Candidates were evaluated at deployed quantisation against a hosted baseline on your tasks
• Schema validity, tool-call accuracy and p95 latency under load were measured
• Break-even arithmetic uses real quotes and includes engineering time
• Self-hosted and hosted models share one API contract and can be switched by configuration
• Weights come from official sources with checksums; serving software has a patch owner
• Maximum context and concurrency limits are configured and tested

Written by the Levy Street team in Wellington. Last updated 19 October 2026.