AI security reviews.

What an AI security review covers

An AI security review is a scoped, hands-on test of one AI system: a support assistant, a document pipeline, an internal search tool or an agent that acts in your software. It looks at the model’s behaviour and, more importantly, at everything the model can reach: the documents it reads, the tools it can call, the systems that consume its output and the vendors that process your data.

A conventional penetration test does not cover most of this. An attacker does not need to break into your network if they can email your inbox agent a message that tells it to forward the last ten invoices.

When you need one

• Before launching anything customer-facing or agentic, especially systems that can send messages, change records or move money.
• After adding tools or connectors, such as an MCP server that gives an assistant access to your CRM or accounting system.
• After a model or vendor change. New models follow instructions differently, and an old defence can stop working.
• When a customer or regulator asks. The Reserve Bank and FMA set out supervisory expectations on cyber risk from frontier AI on 15 October 2026, and enterprise security questionnaires now ask about AI.

If you only use ChatGPT, Claude or Copilot with no integrations, you probably need a configuration check and a policy rather than this review. See AI governance advisory.

The threat model we test against

We map your system against the OWASP Top 10 for LLM Applications 2026 and, for agents, the OWASP Top 10 for Agentic Applications. The risks that matter most in business systems:

What we deliver

Scope and architecture map

The system’s data sources, tools, permissions, vendors and processing locations on one page.

Threat model

Who might attack or misuse the system, through which inputs, and what they could reach.

Test log

Every attack attempted, with the input, the system’s response and whether it succeeded.

Ranked findings

Severity, likelihood, evidence and a specific fix for each issue, with quick wins marked.

Retest

A second pass after fixes, so the report shows what was closed.

Regression tests

The successful attacks, packaged as automated tests to run on every future change.

How the review runs

1. Discover: map the system. A walkthrough with the people who built it, a read of the configuration, logs and vendor terms, and a list of every place untrusted text can enter. This is where most excessive permissions are found.
2. Prove: attack a staging copy. We plant instructions in documents and messages, try cross-user leaks, push tools beyond their purpose, feed hostile output downstream and run cost-abuse inputs. Everything happens on a copy, never on live customer data.
3. Scale: fix, retest and keep testing. We pair with your engineers on the fixes, retest, and hand over the attacks as automated tests. The same suite then feeds your LLM evaluation so every release is checked.

The New Zealand context

The National Cyber Security Centre’s Cyber Threat Report 2026 (24 September 2026) recorded 369 incidents of potential national significance in 2025/26 and judged that by early 2027 malicious actors may have AI capabilities now limited to leading frontier models. On agents, the Five Eyes guidance NCSC NZ co-authored, Careful Adoption of Agentic AI Services (1 May 2026), advises “never granting it broad or unrestricted access, especially to sensitive data or critical systems” (PDF).

Privacy law raises the stakes. The Privacy Act 2020 requires reasonable security safeguards, and a breach likely to cause serious harm must be notified to the Privacy Commissioner as soon as practicable. The Commissioner’s own expectation is within 72 hours of becoming aware. An AI system that leaks one customer’s records to another is a privacy breach, not just a bug.

What drives the cost

Scope is the main driver: one assistant with read-only access is a short review, while an agent that can write to several business systems takes longer. Other drivers are the number of tools and data sources, whether a staging environment exists (building one adds time), access to vendor documentation, and how many fix-and-retest cycles you want included. We scope reviews as fixed pieces of work after the architecture walkthrough. Reviews are carried out by engineers who build and run production AI systems; you can see the team on our people page.

Limits of a review

A review is a point-in-time test. It cannot prove the absence of vulnerabilities, and language models are probabilistic: an attack that fails twenty times can succeed on the next try, so we report attempts and success rates, not just pass or fail. Models and vendor terms change, which is why the regression tests matter more than the report. A review is also no substitute for basic hygiene: patching, multi-factor authentication and access reviews still do most of the work. Our lesson on recognising prompt injection is a good primer for the people who will use the system.

Work this draws on

• SPAWN D: Levy Street’s open-source, invite-only, pre-alpha control plane for running coding agents on machines you own. Its published trust model names the project’s own servers as the adversary, keeps terminal content end-to-end encrypted, and states plainly which metadata the server can still read. That is the standard of honesty we apply to findings.
• Lucent: a Levy Street venture, an agentic investment research platform that runs several frontier models together and grounds its outputs in citations. It lists single sign-on, role-based access, encryption in transit and at rest, and full audit logs among its controls, because institutional buyers check.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.