Meaning Needs Rules: Why My AI Agents Aren’t Allowed to Search the Vector Index Directly

For weeks, my agents were answering questions using an incomplete search index. Once I fixed it, they suddenly started finding notes they should never have been able to access without explicit permission.

For weeks, my agents had been answering questions about my knowledge base, and their answers sounded plausible. What nobody noticed was that the index they relied on contained only a fraction of the actual content. A sync script was supposed to regularly mirror the knowledge base into my agents’ memory. Except the script pointed to folders that didn’t even exist under those names. Every project and work directory reported “0 files” on every run. Only the wiki made it through. No error, no alert. I only discovered it when I asked about something completely unrelated.

Fixing it shifted the problem rather than solving it. With the correct paths, my notes about people suddenly ended up in the index too. Information my agents were not supposed to retrieve without explicit permission was now accessible through search. The index didn’t enforce that rule. It was a copy, and a copy doesn’t know the rules of the original.

In the previous part of this series, I described how an MCP server enforces the writing rules of my knowledge base. I ended with the question of what happens when structure and full-text search are no longer enough. My honest answer: If you're not careful, you'll end up building yourself a backdoor.

Why Full-Text Search Is No Longer Enough

My knowledge base now contains thousands of notes. German and English terms are mixed. The same topic might be called “prompt injection” in one note and “manipulated instructions” in another. Full-text search only finds what is literally written in the text. With my server, every word from the query had to appear in the note, or I had to hope the AI agent searched for the right words.

For example: “How do I keep customer data out of Copilot?” The relevant note exists as German research on data and security in Copilot. Yet full-text search fails to find it. Semantic search considers similarity in meaning rather than individual search terms. In my test, that exact note ranked first.

Agents rarely search using keywords. They phrase their questions like humans, often in the language of the conversation rather than the language of the note. For them, relying on full-text search alone is like searching half-blind.

An Uncontrolled Vector Index Becomes a Backdoor

The obvious solution is a vector index: split notes into chunks, store them as vectors, and search by similarity. The tools for this are readily available, and that is exactly where the risk lies.

A vector index contains a derived view of your knowledge. When the original changes, that view needs to be updated. Access rules are not inherited automatically. If you want to protect restricted areas, you need to enforce the same rules during a search. If an agent can query the index directly and without filtering, it creates a way around the control layer.

During my review, I found two such copies of my knowledge base: the sync described above and an older vector collection. I hadn't consciously decided to keep either of them. They had emerged during experimentation and simply stayed around. On top of that, I found a third gap in my own server: The restriction on personal notes worked for direct reading, but the existing full-text search returned those notes without filtering.

One sentence from the previous article applies here too: A policy only works if every relevant access goes through the control layer. For search, that means: What an agent isn't allowed to read, it shouldn't be allowed to find either.

Conversation Memory and a Knowledge Index Serve Different Purposes

My first approach was to simply continue mirroring the knowledge base into my agents’ existing conversation memory. I already had a conversation memory system in place, and it was capable of semantic search. After the incident with the personal notes, however, I abandoned that approach:

My conversation memory uses a language model to condense information into memories. But when searching for original notes, I wanted traceable sources, not interpretations.

In my implementation, older entries are marked as outdated but not immediately deleted. If a query is missing the appropriate filter, an outdated version can still appear. A search index should reflect the current state, not the history.

My conversation memory does not separate access between agents. Anyone allowed to query it can see everything.

So I disabled the old sync. Qdrant was already running on the server as well. It allows me to update or delete individual chunks and filter out restricted folders during the search itself. The index is also protected by its own key. But the decision about which content an agent is allowed to find still belongs to my MCP server.

Search Should Run Through the Same Server

Semantic search didn't become a separate service. Instead, it became another tool on my MCP server. The server queries both search methods simultaneously, full-text and vectors, and merges the result lists. The same rules apply as for any other read operation:

Restricted folders are filtered during the search, not afterward. Otherwise, forbidden results could push permitted ones out of the result list.

Protected areas such as logs and configuration files are never stored as vectors in the first place.

When an agent modifies a note, the server immediately triggers an update of the affected chunks in the index. Changes made outside the server, for example directly in Obsidian, are picked up through regular synchronization.

Every search result includes the path, section, and source. The agent then reads the original, not the excerpt.

The vectors are generated by a local model running on my own hardware. The contents of my notes never leave my infrastructure for this. If semantic search becomes unavailable, the server continues with full-text search and clearly indicates this in the results. While I was working on this article, exactly that happened on my notebook: The local model wasn't running, but the search still returned results along with a clear notice that only full-text search had been used.

Measure, Don't Rely on Gut Feeling

You can't judge search quality by intuition alone. To evaluate it objectively, I defined a fixed set of real search queries, each linked to the note I expected to find. The queries range from individual keywords to naturally phrased sentences and English questions targeting German content. Especially with everyday language, pure full-text search failed to find a single one of the expected notes. Hybrid search, on the other hand, returned relevant results even for natural-language and cross-language queries. But the test set also revealed that not every supposed improvement actually made things better.

More interesting than the result was the journey to get there. In the first test, the best semantic match didn't appear at all because several versions of the same article draft occupied the top positions. My initial intuition was also wrong when I started fine-tuning the ranking: Notes containing all search terms should receive a bonus. This improved one keyword query, but pushed another result from first place far down the list. The correct note was missing just one word, while a dozen similar notes contained them all. Without the test set, I would have considered the change a success.

Something much less exciting, however, delivered real value: Alongside my own thoughts, my knowledge base also contains copied manuals and third-party documentation. I now give these external sources only half the weight in search rankings. This moves my own notes higher without hiding the reference material. The test set remains in place as a regression test. Every change to the ranking or the model has to prove itself against it.

Where the Boundary Lies

In the first article, I deliberately chose not to use a vector database as the foundation. Not because semantic search is fundamentally unnecessary, but because an index should never become the authoritative source of knowledge. That hasn't changed. The only difference is that my knowledge base now needs an additional search method. Vectors help find information; the Markdown files remain the source of truth.

Semantic search can also find results that seem plausible without necessarily being correct. Even today, it still struggles with some questions from my test set. The relevant note either ranks too low or doesn't appear in the results at all. That's why the server returns sources, not ready-made answers. The agent has to read the original and interpret the information itself.

The system isn't finished, and it isn't supposed to be. It runs every day, the measurements show me where it struggles, and I continue improving it regularly. Every improvement has to prove itself against the same test set.

The cost is manageable, but real: an additional index that needs to be kept up to date, computing resources for generating vectors, and the discipline to regularly evaluate search quality. Searching by meaning is a powerful tool. That's exactly why it must go through the very same control layer as every other access.

One question has stayed with me ever since that unnoticed indexing error: Something was wrong for weeks, and neither I nor any of my agents noticed. A control layer prevents policy violations during access. But who checks whether the processes as a whole are actually working correctly? That's what the next part is about.

Where do the semantic copies of your knowledge bases live, and who decides what your agents are allowed to find in them?