AI Agents, RAG & LLM Integration

What we build on LLMs and agents

Five capability groups: RAG pipelines, tool-using agents, production deployment and ops, in-product LLM features, and the evals and cost controls that keep them running.

Most LLM 'projects' ship a demo and then stall at production hardening. Sapota starts from production: chunking strategies you can measure, agents with human-in-the-loop on destructive actions, cost dashboards per tenant, and eval harnesses that gate every release. Framework choice follows the team and the integration surface, not a default brand.

Production LLM work, RAG, agents, evals, observability, and cost control.

RAG PipelinesTool-using AgentsMulti-agent GraphsSelf-hosted LLMsLangGraphLlamaIndexSemantic KernelVercel AI SDKClaude 4.xGPT-4.1 / 5Gemini 2.xMistralLlama 3.x / 4QwenDeepSeekpgvectorQdrantPineconeWeaviateCohere RerankVoyage EmbeddingsLiteLLM GatewayPortkeyLangfuseLangSmithBraintrustRagasPromptfoo

RAG pipelines, from ingestion to citation

The full retrieval stack. Every piece has a tuning knob, and we treat retrieval quality as a measurable target with an eval set, not a vibe.

Document ingestion pipelines for PDFs (pypdf / unstructured), DOCX, HTML, Confluence, Notion, SharePoint, Google Drive, S3
Chunking strategy options: fixed-size, semantic, layout-aware (for invoices / forms), recursive with overlap, parent-child retrieval
Embedding model selection across OpenAI text-embedding-3, Cohere embed v3, Voyage, bge, Jina, with a benchmark on your data
Vector store setup: pgvector with HNSW, Qdrant with quantization, Pinecone serverless, Weaviate hybrid mode
Hybrid search: dense vectors plus BM25 (OpenSearch / Typesense / Tantivy) plus metadata filters, with fusion (RRF or weighted)
Rerankers: Cohere Rerank, Voyage Rerank, bge-reranker, cross-encoders for higher precision at top-k
Citation-aware answer generation: every claim pinned to a source span, refusal when context is insufficient
Incremental updates: webhooks from Notion/Google Drive, change-data-capture from DB sources, delete propagation
Multi-tenant RAG: per-tenant namespaces, row-level security on retrieval, metadata-enforced access control

AI agents: tool use, routing, human-in-the-loop

Agents that do more than summarize a paragraph. Tool-calling, planning loops, subagents, and the supervisory patterns that stop them from running up a $2,000 bill overnight.

Tool-using agents with REST tool definitions, OAuth credentials per tenant, per-tool rate limits
LangGraph multi-agent graphs: supervisor / subagent / critic patterns, checkpoints, human approval nodes
Routing and classification: cheap model triages, expensive model answers only when needed
Structured outputs: JSON schema / Pydantic validation with retries on malformed output, constrained decoding
Function calling across providers with a normalized tool schema so agents stay portable
Human-in-the-loop approval for destructive actions (send email, execute SQL, charge card, delete resource)
Memory and session state: short-term conversation memory, long-term user profile memory, scratchpads
Multi-turn conversations with context compression, summary rollup, and explicit forgetting windows
Safety guardrails: prompt injection detection, PII redaction on ingress, output moderation, jailbreak monitoring

Production deployment and operations

Self-hosted or cloud, the operational practices that keep an LLM deployment healthy past the first launch. Most teams ship a demo and stall here.

Self-hosting on Docker Compose, Kubernetes (Helm), or managed cloud, with Postgres, Redis, and S3-compatible storage
GPU sizing and routing for self-hosted Llama / Qwen / DeepSeek via Ollama, vLLM, TGI, or llama.cpp
Workflow design across providers: chatflows, agent loops, parallel branches, HTTP request nodes, iteration steps
Knowledge ingestion pipelines with chunking config, retrieval settings, metadata filters, and citation display
Custom tool definitions: OpenAPI imports, credential schemas per tenant, rate limits, allowlist enforcement
Prompt templating and variable management with version history and safe rollbacks
Model provider configuration: Anthropic, OpenAI, Azure OpenAI, Bedrock, Vertex, Ollama, local vLLM endpoints
Observability and tracing wired with Langfuse, LangSmith, Helicone, Phoenix/Arize alongside framework-native traces
Embedded AI apps inside your product via API, iframe, or custom front-end against the agent's HTTP endpoint
User management, workspace isolation, audit logs, SSO (OAuth/OIDC) for enterprise deployments

LLM integration into existing products

Not 'build an AI app from scratch'. Add AI capabilities to the product you already have, with the boundaries a real SaaS needs.

AI features inside an existing Next.js / Nuxt / Rails app, streaming responses with Vercel AI SDK or Server-Sent Events
Semantic search in-app with a pgvector column next to your existing Postgres tables, hybrid query in the same SQL
AI-assisted forms: autofill from uploaded docs, extraction into typed fields, with confidence and review UI
Ticket triage, summarization, and auto-reply drafts. Human edits and sends, you keep the audit trail
Meeting and call transcripts via Whisper / AssemblyAI / Deepgram pipelines, with diarization and action-item extraction
Document Q&A widgets with citations, inline highlight-in-source-doc, and answer disagreement UX
Background AI jobs: nightly summarization, weekly digests, anomaly explanations, with queueing and budget caps
Structured extraction from PDFs/emails/invoices into your database with human-in-the-loop validation

Evals, observability and cost control

The parts that separate an LLM feature from a liability. What you don't measure will regress silently, and what you don't meter will bankrupt you.

Eval harnesses with Promptfoo, Braintrust, Ragas, DeepEval, or custom, on a frozen eval set per feature
Retrieval evals: hit@k, nDCG, MRR, answer-faithfulness, groundedness, citation-presence metrics
LLM-as-judge evals with pairwise comparison, side-by-side A/B, golden-set regression gates in CI
Tracing and observability with Langfuse, LangSmith, Helicone, Phoenix/Arize, including cost and latency breakdowns
Per-tenant and per-feature cost dashboards, token attribution, budget alerts, kill-switches on runaway loops
Rate-limit and quota management per-user, per-tenant, per-tier, with 429 handling and upstream fallback
Prompt and workflow versioning. Every deploy tagged, rollback in one click, diff view across versions
Red-team testing: prompt injection, jailbreaks, PII extraction attempts, tool-call misuse simulations
Provider failover: primary Anthropic, fallback OpenAI, fallback self-hosted, all routed at the gateway (LiteLLM, Portkey)

faq

Common questions from AI leads

Depends on the team and the integration surface. LangGraph when the agent flow is non-trivial and stability matters more than rapid iteration. LlamaIndex when retrieval quality is the dominant constraint. Semantic Kernel when the host app is .NET. Vercel AI SDK for streaming chat features inside a Next.js product. Plain SDK when the call pattern is one-shot. The framework follows the constraints, not the other way around.

Both. Self-hosted (Ollama, vLLM, TGI on your own GPU) when data residency, compliance, or per-token economics demand it. Managed APIs (Anthropic, OpenAI, Bedrock, Vertex) when speed-to-first-value matters more. We've shipped both directions and keep the upgrade paths sane.

Every RAG build has an eval set, usually 50-200 hand-labeled question/answer/source triples from your real usage. We track hit@k, nDCG, faithfulness, and citation-presence across chunking / embedding / reranker variants before shipping to prod.

We assume user input is adversarial. Inputs are segregated from instructions, tool outputs are sanitized, destructive tool calls require human approval, and we run red-team eval suites against every release. Not bulletproof, but far from naive.

Yes. Snowflake / BigQuery / Postgres / Redshift for structured data. Notion / Confluence / Google Drive / SharePoint / S3 for unstructured. Intercom / Zendesk / Gorgias / Salesforce for CX. We handle auth, incremental sync, and permission preservation on retrieval.

A 2-week paid trial scoped to one concrete deliverable: a RAG chatbot over a real corpus, a ticket-triage agent, or an internal document Q&A. You get working software, eval results, and a cost projection. Monthly rolling after that.

Agentic RAG knowledge library orchestration

AI Agents

Apr 28, 2026

Agentic RAG: what it actually costs versus what it actually delivers

A founder's CTO came back from a vendor pitch convinced agentic RAG was the next quarter's roadmap. Eighteen tools, multi-step reasoning, autonomous decisions. The bill the vendor implied was 5x their current model spend, the latency target was a real-time chat product, and nobody had asked what fraction of their queries actually needed an agent.

Agent production failure forensics debug

AI Agents

Apr 25, 2026

Four forensics when a production AI agent fails

A founder messaged us at 11pm on a Friday: the AI agent his team had launched on Monday was down. Customers were complaining, the team was panicking, the on-call engineer had no idea where to start. Here is the forensics order Sapota walks through when an agent fails in production, and the four most common culprits.

RAG Systems

Apr 22, 2026

When recall plateaus: the late-interaction technique most teams skip

A team had been swapping embedding models for two months trying to push retrieval recall past 60%. Each new model gave a couple of points then plateaued. The bottleneck was not the model. It was the architecture: a single embedding per chunk cannot match what token-level interaction can. Here is when ColBERT and ColPali earn their keep.

AI Agents, RAG & LLM Integration

AI Feasibility Review

What we build on LLMs and agents

RAG pipelines, from ingestion to citation