A client asked us to fix an agent that kept getting their pricing wrong. The setup was textbook: they had taken all their company data (product docs, a pricing spreadsheet, the returns policy, a table of customer tiers) embedded the whole lot into a vector database, and pointed the agent at it. When a customer asked "what's the price for plan B at the enterprise tier," the agent would retrieve a chunk that mentioned plan B, a chunk that mentioned enterprise, and a chunk that mentioned a price, and confidently stitch together a number that was wrong about a third of the time.
The model was fine. The retrieval was fine, for what it was. The mistake was upstream: they had taken several different kinds of knowledge, each with a different right way to retrieve it, and forced them all through one method that only suits one of them. A pricing table is not a document. Asking semantic search to look up a price is like asking a librarian to do your taxes: wrong specialist for the job.
"Give the agent our knowledge" sounds like one task. It is at least three, and the vector database is the right answer for only one of them.
The three kinds of knowledge
When you actually list what a team means by "our data," it sorts into three buckets, and the buckets want different tools.
Unstructured text. Documentation, articles, support transcripts, contracts, anything written in prose where the question is semantic: "what does the docs say about webhook retries?" There is no exact key to look up; you are matching meaning. This is what vector search is genuinely good at, and where it should be used.
Structured records. Spreadsheets, database tables, anything with rows and columns. Customer lists, pricing, inventory, transactions. The questions here are filters and aggregations: "the price for plan B at enterprise tier," "how many orders shipped last week," "which customers are on the legacy plan." These have exact answers that live in specific cells.
Exact references. Policies, rules, prices, SKUs, regulatory text, anything where there is one correct answer and being approximately right is being wrong. "What is our refund window?" has a single answer, and returning a similar-but-different policy is a real error with real consequences.
Most teams have all three. Most teams embed all three into one vector store. Two of the three are now being retrieved with the wrong method.
Why structured data should be queried, not embedded
When you embed a spreadsheet, you destroy the thing that made it useful: its structure. A pricing table becomes a pile of text chunks, and "the price for plan B at enterprise tier" becomes a similarity search that retrieves chunks that are textually near those words. It will often work. It will also often retrieve the price for plan B at the standard tier, because that chunk is nearly identical, and the model has no way to know it grabbed the wrong row.
The right tool is a query. Give the agent a tool that runs an actual filter against the actual table: tier = enterprise AND plan = B. The answer is now exact, every time, because you used the structure instead of throwing it away. This is the single highest-leverage fix we make on knowledge-heavy agents: take the structured data out of the vector store and give the agent a query tool over it instead.
This matters most the instant numbers, dates, or filters are involved. Semantic search has no concept of "greater than," "most recent," or "exactly equal to." A query does. Any question that hinges on those operators belongs in a query, not an embedding.
Why exact references need precise lookup
Policies and rules are the most dangerous thing to put behind semantic search, because the failure is invisible and confident. Ask "what's our return policy for opened electronics" and vector search returns the most similar policy. If the most similar policy happens to be the one for unopened electronics, the agent will state it with full confidence, and nobody will know it was the wrong policy until a customer acts on it.
For this knowledge, you want exact lookup keyed on something precise: a policy ID, a category, a SKU. The retrieval should be deterministic, the same way a hardcoded flow is deterministic where the structure is known. If the right policy cannot be found by its key, the correct behavior is to say so, not to return the nearest neighbor and hope.
The general principle: where being approximately right is actually being wrong, similarity search is the wrong primitive. Use it only where "close enough" is genuinely close enough.
Where the vector database actually earns its place
None of this is an argument against vector search. It is an argument against using it for everything. For genuinely unstructured text, where the question is about meaning and there is no exact key, it is the right tool and nothing else comes close. "What do our docs say about handling rate limits" is exactly the question semantic retrieval was built for.
Even here, the quality of the result depends on the unglamorous parts: how you chunk the documents, how you handle the boundaries between chunks, whether you keep enough surrounding context for a chunk to be self-explanatory. And it depends on cost discipline, because a vector store that retrieves twenty chunks per query when three would do is a slow, expensive habit that adds up fast. We wrote separately about keeping agentic RAG costs in line with what it actually delivers, because the easiest way to make a knowledge agent expensive is to over-retrieve on every step.
The architecture: three retrieval methods, one reasoning loop
The agent that gets knowledge right does not have one knowledge source. It has three retrieval tools wired into the same reasoning loop:
- A search tool over the vector store, for semantic questions against unstructured documents.
- A query tool over the structured data, for filters and aggregations against tables.
- A lookup tool over exact references, for policies, prices, and rules retrieved by precise key.
The model's job is to pick the right tool for the question, which is a far easier decision than stitching together approximate chunks. "What's the price for plan B" routes to the query tool. "How do refunds work" routes to the lookup tool. "What does the docs say about X" routes to search. Each returns a precise result for its kind of question, and the model reasons over reliable inputs instead of guessing across a soup of chunks.
This also keeps each piece independently fixable. When the pricing is wrong, you debug the query tool, not a vector index. When a policy comes back wrong, you check the lookup key, not the embeddings. The blast radius of any one knowledge source stays contained.
How to tell your agent has this problem
The symptoms are specific:
- The agent is wrong about numbers, prices, dates, or counts while being fluent about prose. That is structured data forced through semantic search.
- The agent occasionally states a policy or rule that is subtly the wrong one, confidently. That is exact-reference knowledge behind similarity search.
- Accuracy is fine on "explain" questions and shaky on "look up" questions. That is the tell that one retrieval method is doing two jobs.
If that describes your agent, the fix is almost never a better embedding model. It is routing the structured and exact-lookup knowledge to tools that retrieve them precisely.
If your agent gets your own data wrong
An agent that fumbles your prices, policies, or records is not a knowledge problem in the sense of "it does not have the data." It has the data; it is retrieving it with the wrong method. That is fixable without re-architecting the whole system.
Sapota runs a one-week knowledge-layer audit that sorts your knowledge sources into the three kinds, routes each to the right retrieval method, and ships the multi-tool retrieval setup as a working integration. We have done this for support agents, sales assistants, and internal copilots where getting the numbers and policies exactly right is the whole point.
Reach out via the AI engineering page with a description of what your agent needs to know and where it currently gets it wrong. The fix is usually a routing problem, and routing problems are tractable.
