Vector and Embedding Weaknesses in RAG (LLM09)
On this page
Why RAG retrieval is an access-control decision
A RAG system answers a question by finding chunks of documents and pasting them into the context. Whatever it retrieves, the model can repeat. So every retrieval is an access-control decision, and most early RAG systems made it without checking anything.
LLM09, Vector and Embedding Weaknesses, collects the ways this goes wrong:
| Weakness | What happens | Control |
|---|---|---|
| No permission check | An intern's question retrieves board minutes | Filter by the user's permissions before ranking |
| Shared index across tenants | Customer A's question retrieves customer B's data | Separate indexes or enforced tenant filters |
| Lost source permissions | Chunks forget which documents they came from | Copy source ACLs onto every chunk, keep them in sync |
| Embeddings treated as anonymous | Leaked vectors reveal the original text | Protect vectors like the source data |
| Poisoned entries | Crafted chunks win retrieval (Day 17) | Write controls and review |
Filter by permission before you rank
The common mistake is retrieving the top results and then filtering, or letting the model decide what the user may see. Filter first, with identity from the authenticated session:
import numpy as np
from dataclasses import dataclass
@dataclass
class Chunk:
id: str
tenant: str
acl: frozenset[str] # groups allowed to read the source document
text: str
vec: np.ndarray
def search(index: list[Chunk], qvec: np.ndarray, *, tenant: str, groups: set[str], k: int = 3):
# Filter BEFORE ranking, using identity from the session, never from the query text.
allowed = [c for c in index if c.tenant == tenant and c.acl & groups]
scored = sorted(allowed, key=lambda c: float(c.vec @ qvec), reverse=True)
return [(c.id, c.text) for c in scored[:k]]
rng = np.random.default_rng(0)
def v():
x = rng.normal(size=8)
return x / np.linalg.norm(x)
index = [
Chunk("a1", "acme", frozenset({"support"}), "Refunds within 30 days.", v()),
Chunk("a2", "acme", frozenset({"finance"}), "Q3 margin is 41%.", v()),
Chunk("g1", "globex", frozenset({"support"}), "Globex refund rules.", v()),
]
q = index[1].vec # a query that most resembles the finance chunk
print(search(index, q, tenant="acme", groups={"support"}))
# [('a1', 'Refunds within 30 days.')]
print(search(index, q, tenant="acme", groups={"support", "finance"}))
# [('a2', 'Q3 margin is 41%.'), ('a1', 'Refunds within 30 days.')]The support agent's query is closest to the finance chunk, and still never sees it. Globex's data never appears for Acme at all.
Filtering after ranking has a second problem besides leaks: if the top 3 are all forbidden, the user gets nothing useful. Real vector databases support metadata filters inside the search, such as WHERE tenant = ... AND acl && ... in pgvector or filter expressions in dedicated stores, so the filter and the ranking happen together.
Tenant isolation and stale permissions in vector stores
For multi-tenant SaaS, the strongest isolation is a separate index or collection per tenant. A shared index with a mandatory tenant filter works too, but then one forgotten filter is a cross-tenant leak, so put the filter in a single data-access function that every query goes through.
Permissions change. Someone leaves the finance group, or a document is restricted after it was indexed. If chunks only copy the ACL at indexing time, retrieval keeps serving the old permissions. Re-sync ACLs when source permissions change, or check the source system's permissions at query time for sensitive collections.
Embeddings aren't anonymous
It's tempting to treat vectors as harmless numbers. They aren't. In 2023, Morris and colleagues' Vec2Text recovered 92 percent of 32-token inputs exactly from their embeddings, including names from clinical notes. Store, back up, log and share vectors with the same protection as the text they came from, and don't send them to third parties you wouldn't send the text to.
Check yourself
The model may still get forbidden content if the step has a bug, and often good, allowed results are crowded out of the top 20 by forbidden ones. Filter inside the search, with permissions from the session.
Route every query through one data-access function that always applies the tenant filter from the session, so no code path can search without it. Separate indexes per tenant are stronger still.
Embeddings can be inverted to recover much of the original text, as Vec2Text showed, so the vectors carry the same sensitivity as their sources.
If access is revoked but the chunk's metadata isn't updated, retrieval keeps returning it. Sync permissions from the source system and filter at query time using current access data.