A founder asked why their AI assistant kept saying 'the chart shows a positive trend' instead of reading the actual numbers. The pipeline was doing exactly what it was designed to do, and that was the problem. Here is how Sapota decides between summary-based and native multimodal RAG.
A SaaS founder pinged us last quarter with a complaint that sounded familiar. Their AI assistant, built on top of a research-paper knowledge base, kept giving answers like "the chart shows a positive trend in Q3 revenue" instead of saying "Q3 revenue was 4.2 million, up 18% from Q2."
The retrieval was finding the right page. The vision LLM was rendering a coherent response. The pipeline did exactly what it was designed to do. The problem was that the design itself was lossy by construction.
This is the moment most teams discover that their multimodal RAG pattern is the bottleneck, not their model.
Their pipeline used the standard summary-based approach (the one most tutorials and managed services default to):
Every step is reasonable in isolation. The compounding loss is not visible in a 50-question demo. It shows up when a CFO or analyst tries to interrogate the data and the assistant can only paraphrase what the indexer wrote down weeks ago.
The vision LLM in step 2 had been told to "describe in detail." It produced fluent prose: "The chart shows quarterly revenue with a positive growth trajectory across Q1 through Q3." It did not transcribe the axis labels, the data points, or the legend. By the time the user query arrived, the actual numbers no longer existed in the index.
The instinct most teams have at this point is to switch to "native multimodal" with CLIP or SigLIP. Embed images directly into a shared vector space with text, retrieve the raw image at query time, and let a vision LLM read it on the fly.
This works. It also costs roughly 5 to 10 times more per query, requires a vector database that supports multi-vector collections, and doubles the operational surface area (two embedding models, image storage, vision LLM at query time instead of just at index time).
For a B2B SaaS founder running on a $200k seed cheque, that swap is not a free upgrade. The right question is whether the loss is structural or fixable inside the existing pattern.
We treat the choice as three questions, in order:
1. What fraction of the answer-bearing content lives in images, charts, or tables?
If it is under 10%, summary-based is almost always the right call. Fix the prompt, not the architecture. A prompt that says "transcribe all visible numbers, axis labels, and legends before describing the chart" recovers most of the lost fidelity for the cost of a slightly longer summary.
If it is 10% to 30%, summary-based with a stricter image extraction prompt plus table-as-HTML preservation usually clears the bar. We also recommend caching the original image URL in the chunk metadata, so the front end can render the source figure inline next to the answer. Users perceive that as "the AI showed me the chart" even though the model never read it directly.
If it is above 30%, the pipeline starts paying for itself in the cost of constant re-indexing every time the prompt template changes. This is where native multimodal becomes worth the operational complexity.
2. Does the user query benefit from "show me similar visual" search?
Pattern A cannot do image-to-image retrieval. If the product genuinely needs "find slides that look like this one" or "match this product photo to our catalog," that capability does not exist in summary-based pipelines and never will. Adding CLIP is the only path.
3. What is the per-query cost ceiling?
Vision LLM calls at query time run roughly $0.005 to $0.03 per image, depending on provider. A pipeline that pulls top-5 images into a Claude or GPT-4V call is $0.05 to $0.15 per question, before counting the input text tokens. At 10,000 queries a month that is $500 to $1,500. For a consumer-facing assistant that is fine. For an internal compliance tool that runs once a week, the math changes.
The diagnosis was that roughly 20% of their corpus was charts and tables, the user query distribution was almost entirely "what was the value of X" rather than "find me a similar diagram," and they were already cost-sensitive. A full Pattern B migration would have been over-engineering.
The fix was three changes inside Pattern A:
Recall on numerical questions went from "frequently wrong" to "matches a manual analyst within rounding" on their internal eval set of 80 questions. No vector database migration. No vision LLM at query time. Cost stayed flat.
Pattern B is the right call when:
The mistake we see most often is teams jumping to Pattern B because the summary-based version "felt limited," when actually the prompt was lazy and the post-retrieval rendering was missing.
For document-heavy use cases (research papers, financial filings, slide decks), there is a third option worth knowing about: ColPali skips the OCR/extraction pipeline entirely and treats each page as a single image, using a vision-language model to compute patch-level embeddings.
It outperforms both Pattern A and Pattern B on visually complex documents in published benchmarks, but it carries 32x the storage cost of a bi-encoder and requires GPU inference. Worth a separate conversation, which we are writing up next.
Sapota offers an architecture review for teams shipping multimodal RAG that takes a working pipeline, a representative eval set, and a question-distribution sample, and returns a concrete recommendation on which pattern fits with the trade-offs spelled out for your specific cost and recall constraints.
If your AI assistant is paraphrasing charts when users want numbers, or quoting numbers that turn out to be hallucinated, that is the diagnostic conversation we run. Reach out via the AI engineering page and tell us what you are seeing.
AIApr 15, 2026
A team asked us why their support assistant could not answer 'which enterprise customers complained about pricing in Q4 and what was their MRR'. The answer involved three entities, two relationships, and one timeframe filter. Vector search returns chunks, not answers to questions like that. Here is how Sapota decides when to add a graph layer.
Read More
AIMar 11, 2026
A team came to us frustrated that their semantic search was missing exact product codes like 'GPT-4o' even though those tokens appeared in the indexed content. The fix was a single config change. The reason most teams skip it is the same reason most teams ship vector-only search in the first place.
Read More
AIMar 25, 2026
A perfect 50-question demo on Tuesday. By the second week of production, users were filing tickets faster than the team could close them. The model had not changed. The retrieval pipeline had not changed. Here is what Sapota looks for first when a RAG launch goes sideways.
Read More
AIMar 18, 2026
A founder forwarded an email from his CFO three days before a board meeting: 'before we approve the AI budget, can engineering give us a number for accuracy?' The team had no number. They had vibes. Here are the three Ragas metrics Sapota ships before any RAG system goes to production.
Read MoreWe build trust by delivering what we promise – the first time and every time!
We'd love to hear your vision. Our IT experts will reach out to you during business hours to discuss making it happen.
"Collaborate, Elevate, Celebrate where Associates - Create Project Excellence"
SapotaCorp beyond the IT industry standard, we are
