# Building private AI search.

URL: https://levystreet.com/engineering/building-private-ai-search/
Updated: 2026-10-19

How to build private AI search over company documents: permission-aware retrieval, ACL sync from source systems and a leakage test suite that proves it.

## Short answer

Private AI search answers questions from an organisation's own documents while respecting who may see what. The core rule is to enforce the user's permissions inside retrieval, before any text reaches the model, never by filtering the answer afterwards. Build it with permissions synced from source systems, identity on every query, citations, and a leakage test suite that proves restricted content cannot reach the wrong person.

- Enforce permissions inside the search query, not after retrieval or generation.
- Store group IDs on documents and expand the user's groups at query time.
- Treat embeddings, caches and backups as being as sensitive as the source documents.
- Prove isolation with canary documents, revocation tests and cross-user cache tests.
- Set and measure a freshness target for permission revocations.

## What private AI search has to guarantee

Private AI search is retrieval-augmented generation over internal documents with one extra guarantee: a user only ever gets answers built from content they could open themselves. Everything else (chunking, ranking, citations) matters, but a leak of one restricted HR file outweighs a thousand good answers.

The system has five parts: connectors that read source systems and their permissions, an index that stores chunks with access metadata, a retrieval step that filters by the asking user's identity, a model that writes an answer with citations, and logs that record what was retrieved for whom. Each part can leak. Our explainer on [retrieval augmented generation](/learn/understanding-retrieval-augmented-generation/) covers the basics; this guide covers the permission layer.

## Where to enforce permissions

Enforce permissions inside the retrieval query, so restricted content is never a candidate. The alternatives fail in predictable ways.

| Approach | How it works | Verdict |
|---|---|---|
| Filter the answer | Generate from everything, then try to remove restricted facts | Unsafe. The model has already read the content |
| Post-filter results | Retrieve the top 50, drop what the user cannot see | Leaks through counts and timing; empties results for low-access users |
| Filter inside the search | The vector query carries the user's principals as a metadata filter | Default choice for one organisation |
| Separate index per security domain | One index per tenant, or per highly restricted area such as HR or board papers | Strongest isolation; use for multi-tenant products and the most sensitive content |

OWASP's [2026 Top 10 for LLM applications](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/) lists cross-tenant leakage through shared similarity search under vector and embedding weaknesses: when access control runs after the search, an attacker can infer the existence and topic of other tenants' documents from result counts, score distributions and timing. The [2025 entry](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/) recommends permission-aware vector stores and strict logical partitioning between user groups.

## Syncing permissions from source systems

Permissions live in the source systems (SharePoint, OneDrive, Google Drive, Confluence, file shares), and each models them differently: inheritance from folders and sites, sharing links, external guests, nested groups. Your connector has to translate all of that into a list of principals per document.

Design rules that hold up:

- **Store group IDs on documents; expand the user's groups at query time.** Copying group membership into the index goes stale every time someone joins or leaves a team.
- **Decide what "anyone with the link" means.** Most organisations exclude link-shared files from search unless the link is organisation-scoped.
- **Sync deletions and revocations first.** A new document arriving late is an inconvenience; a revoked one lingering is a breach.
- **Set a freshness target and measure it.** For example, revocations reflected in search within 15 minutes, with the actual lag reported daily.
- **Re-check at source for the most sensitive collections.** A live permission check against the source API before returning a chunk costs latency but removes sync lag from the risk.

Oversharing in the source is the largest practical risk. If a payroll spreadsheet is shared with the whole company, a correct permission-aware search will faithfully surface it to everyone. Off-the-shelf assistants work the same way: Microsoft states that [Microsoft 365 Copilot](https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy) only surfaces organisational data to which each user has at least view permission, so existing oversharing becomes visible the day it is switched on. Audit sharing before you index, especially in [SharePoint](/integrations/sharepoint/) and Google Drive.

```python
def private_search(user, query, index, acl, k=8):
    """Pseudo-code: the filter syntax depends on your vector database."""
    principals = acl.principals_for(user)        # user id plus current group ids
    hits = index.query(
        vector=embed(query),
        top_k=k * 2,
        filter={"allowed_principals": {"any_of": sorted(principals)}},  # inside the search
    )
    if index.collection_is_restricted:
        hits = [h for h in hits if acl.source_allows(user, h.doc_id)]  # live re-check
    log_retrieval(user_id=user.id, query=query, doc_ids=[h.doc_id for h in hits])
    return [h.without_score() for h in hits[:k]]  # never return raw similarity scores
```

How filters behave under load differs by product; see [choosing a vector database](/engineering/choosing-a-vector-database/) for filtered-search performance.

## The leakage test suite

You prove privacy with tests run as real user identities, on every release and nightly in production. The core technique is canary documents: restricted files containing a unique string that must never appear anywhere an unauthorised user can see.

| Test | What it proves | How to run it |
|---|---|---|
| Canary retrieval | Restricted text never reaches unauthorised users in chunks, answers or citations | Plant unique strings in restricted files; query as users without access |
| Revocation lag | Removing access takes effect within the target | Revoke at source; poll search until the file disappears; record the lag |
| Deletion | Deleted files leave the index, caches and derived embeddings | Delete at source; check every store that holds derived data |
| Group change | Leaving a group removes inherited access | Move a test user between groups and repeat canary queries |
| Cross-user cache | A cached answer is never served to a user with different access | Ask the same question as two users with different permissions |
| Conversation memory | One user's history never informs another's answers | Seed memory as user A, query as user B |
| Membership inference | Hidden documents cannot be detected from scores, counts or timing | Probe with queries about restricted topics; compare responses with a control |
| Injected instructions | A document saying "ignore your access rules" changes nothing | Plant injection files; assert filters still hold |

```python
CANARY = "CANARY-7f3a91c2"  # appears only in hr/restructure-2026.docx

def test_canary_never_reaches_unauthorised_users(app, directory):
    for user in directory.users_without_access("hr/restructure-2026.docx"):
        for q in ("restructure plans", "who is affected by the restructure", CANARY):
            result = app.ask(as_user=user, question=q)
            assert CANARY not in result.answer
            assert all(CANARY not in c.text for c in result.retrieved_chunks)
            assert "hr/restructure-2026.docx" not in result.citation_ids
```

Injection cases deserve their own suite; see [prompt injection testing](/engineering/prompt-injection-testing/). Retrieval quality is a separate question with its own metrics, covered in [RAG retrieval evaluation](/engineering/rag-retrieval-evaluation/).

## The leaks people forget

The index is not the only store of your documents. Every derived copy needs the same protection as the source.

- **Embeddings.** OWASP warns that stored embeddings can be inverted to recover source text, so vector backups belong at the same sensitivity tier as the documents.
- **Semantic caches.** Key cached answers by the permission set as well as the question, or do not cache restricted collections.
- **Agent memory.** Summaries of past conversations can carry restricted facts into new sessions; see [agent memory design](/engineering/agent-memory-design/).
- **Logs and traces.** Retrieval logs contain the retrieved text. Restrict who can read them and set retention.
- **Evaluation sets.** Questions and gold answers drawn from restricted documents inherit their access rules.
- **Model providers.** Retrieved text is sent to the model provider on every call. Where that processing happens is covered in [data residency for AI systems](/engineering/data-residency-for-ai-systems/).

In New Zealand, IPP5 of the [Privacy Act 2020](https://www.legislation.govt.nz/act/public/2020/0031/latest/whole.html) requires safeguards that are reasonable in the circumstances against unauthorised access and disclosure. A search tool that shows personal information to the wrong employee is a privacy breach, and one likely to cause serious harm must be [notified to the Privacy Commissioner](https://www.privacy.org.nz/responsibilities/privacy-breaches/notify-us/), ideally within 72 hours of becoming aware of it.

## Where private search goes wrong

| Failure | Typical cause | Detection |
|---|---|---|
| Oversharing surfaced | Files shared with everyone in the source system | Pre-launch sharing audit; report of documents visible to all staff |
| Stale revocations | Nightly permission sync | Measured revocation lag against the target |
| Filter dropped | A code path or new collection skips the filter | Canary tests in CI for every collection and endpoint |
| Cross-user cache hits | Cache keyed by question only | Cache key audit; cross-user cache test |
| Restricted text in logs | Full prompts logged for debugging | Log access review; canary scan of log storage |
| Low-access users get nothing | Post-filtering empties result sets | Answer rate by user group |

We build these controls into [RAG development](/services/rag-development/) projects and run canary tests as part of release checks. Policy and procedure search in regulated firms, such as [internal policy retrieval in financial services](/industries/financial-services/internal-policy-retrieval/), is where we see the strictest requirements.


- Permissions are enforced inside the retrieval query, never by filtering answers
- Documents carry group and user principals; user groups are expanded at query time
- Link-shared and externally shared files have an explicit indexing policy
- Revocations and deletions sync first, with a measured freshness target
- The most sensitive collections use a separate index or a live check against the source
- Canary documents are tested as real unauthorised users in CI and nightly
- Caches, memory, logs and evaluation sets are keyed or restricted by permission
- Raw similarity scores are never returned to clients
- Vector stores and backups are protected at the same tier as source documents
- Source-system oversharing was audited before the first index build

## Questions people ask

### What is private AI search?

Private AI search is a question-answering system over an organisation's internal documents, such as policies, contracts and project files, that only uses content the person asking is allowed to see. It combines retrieval from an index of your documents with a language model that writes an answer and cites sources. The private part is the permission model, not just where the servers sit.

### Can Copilot or ChatGPT search our internal documents safely?

Products such as Microsoft 365 Copilot and ChatGPT connectors respect the permissions in the connected system, which means they also expose every oversharing problem you already have. Before switching them on, find files shared with everyone or by open link, fix group membership, and test with real user accounts. A custom build needs the same clean-up plus its own permission tests.

### Why not filter the AI's answer instead of the search results?

Because by then the restricted text has already been retrieved and read by the model, and it can leak through the answer, citations, suggested follow-ups, caches or logs. Filtering after retrieval also skews ranking, and timing and result counts can reveal that hidden documents exist. Filter inside the search so restricted content is never a candidate.

### How quickly must permission changes reach the AI search index?

Set an explicit target and measure it. Revocations matter most: when someone leaves a team or a file is unshared, search should stop returning it within minutes, not at the next nightly sync. Expanding group membership at query time instead of copying it into the index removes most of the lag for group-based access.

### Is a search tool that shows the wrong document a privacy breach?

If it exposes personal information to someone not authorised to see it, it can be. IPP5 of the Privacy Act 2020 requires reasonable safeguards against unauthorised access, and breaches likely to cause serious harm must be notified to the Privacy Commissioner, ideally within 72 hours of becoming aware. Get advice on your specific obligations.


## Sources

- [OWASP GenAI LLM Top 10 2026](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/)
- [LLM08:2025 Vector and Embedding Weaknesses](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/)
- [Data, Privacy, and Security for Microsoft Copilot](https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy)
- [Privacy Act 2020](https://www.legislation.govt.nz/act/public/2020/0031/latest/whole.html)
- [NotifyUs of a serious privacy breach](https://www.privacy.org.nz/responsibilities/privacy-breaches/notify-us/)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.