Forem: Md Ayan Arshad

5 Critical Failures We Hit Shipping a Multi-Tenant RAG Chatbot to 500+ Enterprises

Md Ayan Arshad — Sat, 04 Apr 2026 06:26:56 +0000

Our first enterprise tenant onboarded on a Monday.

By Wednesday, 30% of their documents had been silently indexed as empty strings. No error. No exception. The chatbot just said "I don't have enough information", confidently, every time.

That was Failure #1. There were four more.

Here's the honest account of shipping a multi-tenant RAG chatbot to 500+ enterprise clients — what broke, in what order, and what we should have caught earlier.

The System We Built

Before the failures, the context.

We built a RAG chatbot for enterprise warehouse management. Each tenant had their own isolated knowledge base — SOPs, compliance documents, operational guides. Users queried only their tenant's data. Scale target: ~25,000 queries per day at full rollout.

Indexing pipeline:
Document Upload → Type Detection → Preprocessing → Chunking → Embedding → Pinecone

Query pipeline:
User Query → Cache Check → Query Rewrite → Hybrid Search (BM25 + Vector) → RRF Fusion → Reranker → LLM → Response

Two pipelines in the design. One EC2 fleet in reality, which became Failure #4.

Indexing consumed from SQS. Query API sat behind an ALB. One Pinecone namespace per tenant, every query scoped to the authenticated tenant's namespace before touching the vector DB.

The architecture decisions were mostly right.

What broke was the assumptions underneath them.

Failure #1: The PDF Preprocessing Assumption (Week 1)

We assumed all enterprise documents were text-based PDFs.

They weren't.

About 30% of what tenants uploaded were scanned PDFs, images of physical pages, no text layer. When PyMuPDF opened these files, it returned empty strings. We embedded empty strings. We indexed empty chunks. No error. No exception. Just silent failure.

Users asked questions. Retrieval returned nothing relevant. The LLM said "I don't have enough information." Users assumed the chatbot was broken. They were right, just not for the reason they thought.

The fix: A preprocessing gate that checks average characters per page. If avg_chars_per_page < 100, no text layer exists, trigger OCR via AWS Textract before chunking. We also added an admin-facing flag marking documents as "pending OCR" so tenants know their document is processing, not lost.

The lesson: Never assume your input format. Garbage input produces zero output in RAG. Preprocessing is the most boring part of the pipeline and the most catastrophic to skip.

Failure #2: Headers, Footers, and the Chunk Contamination Problem

Even for text-based PDFs, every chunk was contaminated.

Enterprise documents have headers and footers on every page. "Softeon WMS User Guide — Confidential — Page 14 of 203." When you chunk a 200-page document into 512-token pieces, that text bleeds into hundreds of chunks.

The retrieval impact was subtle but real. Queries about "confidential" topics surfaced chunks with "Confidential" in the footer, not because the content was relevant, but because BM25 was matching on that exact term. Relevance scores were quietly polluted.

The fix: A stripping step before chunking. Text appearing in the top 5% and bottom 5% of every page gets flagged and removed. We also converted tables to markdown before chunking, a raw table extracted as "Product Price Refund Laptop 999 30 days" is useless for retrieval. The same table as structured markdown is self-contained and semantically meaningful.

The lesson: Most RAG tutorials skip directly to chunking size debates — 256 vs 512 tokens. They assume clean input. Real enterprise documents are not clean.

Failure #3: The Parallel Pipeline Was Actually Sequential

We ran BM25 and vector search in what we thought was parallel.

It wasn't.

The original implementation called BM25, waited for the result, then called Pinecone. "Parallel" on the architecture diagram. Sequential in the code. At p50 this cost us ~200ms we couldn't afford.

The fix is one line:

# Wrong — sequential
bm25_results = await bm25_search(query)
vector_results = await pinecone_search(query)

# Right — parallel
bm25_results, vector_results = await asyncio.gather(
    bm25_search(query),
    pinecone_search(query)
)

Latency becomes the max of the two, not the sum. Dropped our p95 by ~180ms.

The lesson: "Parallel" on a diagram and "parallel" in code are different things. Profile your pipeline stage by stage. The bottleneck is always somewhere surprising.

Failure #4: One Tenant's Upload Degraded Everyone's Query Latency

This one took a week to diagnose.

We noticed periodic p99 spikes, not consistent, not tied to query volume. Random, unpredictable.

The cause: our indexing pipeline and query pipeline were on the same EC2 instances.

When a large tenant uploaded 500 documents, the embedding loop hammered the instance CPU. Live users querying on the same instance saw response time jump from 800ms to 6+ seconds. The indexer and query service were invisible to each other in code — but very visible to each other on the metal.

The fix: Complete infrastructure separation. Indexing workers on a dedicated EC2 fleet, completely outside the ALB. The query fleet has no knowledge that indexing is happening. A document upload spike now has zero effect on query latency for any tenant. The SQS queue buffers upload bursts and feeds indexing workers at a controlled pace.

The lesson: Load isolation is not just an architectural principle. It's a user experience decision. Enterprise tenants don't care about your architecture, they care that the chatbot was slow when they needed it.

Failure #5: The Namespace Isolation Gap We Almost Missed

Multi-tenant isolation in Pinecone is handled by namespaces. One namespace per tenant. Every write tags it. Every read is scoped to it.

What we almost shipped: namespace scoped at the request body level.

A bad actor passing a forged tenant_id in the request body could scope the query to a different tenant's namespace. Subtle. Critical.

# Wrong — trusting request body
namespace = request.body.tenant_id

# Right — trusting validated token only
namespace = token_context.tenant_id  # resolved from JWT at API layer

The fix: Namespace resolved exclusively from the validated JWT token at the API layer. The request body's tenant_id is ignored entirely. By the time a request reaches the vector DB call, the namespace has already been locked to the authenticated tenant — it cannot be overridden.

If we had shipped the original version, any authenticated user who knew another tenant's ID could have queried their private documents. In a WMS context serving enterprise clients, that's not a security incident, that's a contract termination and a legal conversation.

The lesson: Namespace isolation is not the same as security. Enforce tenant identity at authentication, not the application layer.

What We Still Haven't Built

We don't have automated RAG evaluation in production.

No RAGAS running continuously. No Precision@5 after every deployment. Human review by an internal QA team, representative queries, manual quality ratings. It works at current scale. It won't at full rollout.

What I'd build next with two weeks:
→ A golden evaluation set with 200 curated question-to-chunk pairs from real tenant queries. Your retrieval quality baseline.
→ RAGAS Faithfulness in CI/CD runs on every deployment, blocks release if faithfulness drops more than 5% from baseline.
→ Context Precision tracking, tells you if your reranker is actually earning its latency cost.

The One Thing That Mattered Most

RAG systems fail at the edges of the pipeline, not the center.

Most engineering effort goes into the center, embedding models, reranking algorithms, chunk sizes. The real production failures happen at the edges: what goes into the indexer, what happens when two workloads compete for the same compute, and where tenant identity gets resolved.

What broke first in your RAG pipeline? Drop it in the comments. The failures nobody writes about are always the most useful. I'll compile the best ones into a follow-up post.

Why Our RAG System Was Silently Returning Wrong Answers — And How We Fixed It

Md Ayan Arshad — Wed, 11 Mar 2026 00:19:49 +0000

For 3 days, our RAG system was confident.

Every query returned an answer. Response times were stable. No errors in the logs. By every operational metric, the system was working.

Our RAGAS faithfulness score told a different story.

It had dropped from 0.91 to 0.67 without a single code change.

That meant roughly 1 in 3 responses was making claims our own retrieved context didn’t support. The system wasn’t crashing. It was hallucinating — silently, at scale, with complete confidence.

Here is what happened.

The System When It Started Failing

We were running a production RAG system serving enterprise clients across a large document corpus. Each client had their own isolated set of documents — product configuration files, setup guides, operational workflows — queried daily to answer operational questions.

_The state of the system when the drift began:

25K+ documents across corpus
500+ enterprise tenants
1 Pinecone namespace per tenant
5 chunks retrieved per query (top-K) _

Stack: GPT-4 for generation, text-embedding-ada-002 for embeddings, Pinecone with one namespace per tenant, FastAPI on ECS. Isolation was strict — no cross-tenant reads, ever.

_A NOTE on the namespace decision: Pinecone namespaces share the same index and billing unit, 500 namespaces cost the same as 1. We chose namespaces over metadata filtering (tenant_id filter on a single index) for one specific reason: Metadata filtering requires every query to carry the correct filter, and one bug means Tenant A can read Tenant B's data. For enterprise clients, that risk surface isn't acceptable. Namespaces make cross-tenant leakage structurally impossible at query time.

Namespaces give us defense-in-depth isolation at the infrastructure layer rather than relying on application-level filtering._

Monitoring: API latency, error rates, cost dashboards.

Answer quality monitoring: None.

That was the bug. Not in the code. In the architecture.

What Changed, And Why We Didn’t See It

Three days before we caught the drift, a large batch of new documents was ingested. Different document type than the existing corpus — denser, longer sentences, more domain-specific terminology. Same domain, different structure.

The new documents changed the distribution of our Pinecone namespaces. Queries that had previously retrieved highly relevant chunks now retrieved chunks that were topically related but not directly answering the query.

Cosine similarity scores: 0.76, 0.79, 0.81. High enough to clear any threshold we’d set. text-embedding-ada-002 couldn't distinguish between "this chunk discusses this topic" and "this chunk contains the specific answer this query is asking for." Retrieval looked confident. The chunks were wrong.

GPT-4 did what LLMs do when context is adjacent-but-imprecise: it filled the gaps with plausible-sounding claims not present in the retrieved text.

RAGAS faithfulness: 0.91 → 0.67.

Context Precision: 0.84 → 0.61.

We had no alert for either. We found out from a user.

The core failure: we instrumented everything easy to measure — latency, throughput, cost, error rates — and nothing that measured correctness. We were flying blind on the one metric that determined whether the system was actually useful.

One clarification on how we were running RAGAS. We were not evaluating live traffic — that would be prohibitively expensive and slow. We maintained a golden evaluation set of ~300 representative queries with known-good answers and source chunks, curated when the system first launched. RAGAS ran against that set nightly, and on every ingestion event. The drop from 0.91 to 0.67 showed up the morning after the batch ingestion. We just had no alert configured to catch it.

What We Tried First — And Why It Wasn’t Enough

First instinct: improve retrieval.

We raised the similarity threshold from 0.70 to 0.78, rejecting chunks below that score. Retrieval precision improved. We also started returning no results for legitimate queries with unusual phrasing. Users got empty responses. That was worse.

We increased top-K from 5 to 10. Slightly helped recall. Sent 2× the tokens into every LLM call, which compounded a cost problem already building at 500+ active tenants.

Context Precision recovered to 0.78. Faithfulness only reached 0.81, still below our 0.85 target.

The retrieval fixes were necessary. They were not sufficient. We needed a layer that caught the gap between what retrieval returned and what the LLM claimed. Retrieval improvement was treating the symptom. We needed to treat the cause.

The Real Fix: Grounding Validation as a First-Class Architecture Layer

We added a grounding validation step that runs after every LLM response, before it’s returned to the user.

The mechanism:

Extract the factual claims from the generated response using a structured extraction prompt. This step is imperfect, LLM-based claim extraction can miss or misinterpret implicit claims, so we treat it as a signal, not a definitive verdict. A claim is flagged as unsupported if no retrieved chunk scores above a similarity threshold for that claim.
Score each claim against the retrieved chunks, classified as supported, unsupported, or contradicted.
Flag any response where more than 15% of claims are unsupported or contradicted.
Regenerate flagged responses with an explicit grounding instruction injected into the prompt.

The regeneration prompt:

“Your response must only make claims directly supported by the provided context. If the context does not contain the answer, say so explicitly. Do not infer or extrapolate.”

Critical implementation detail: The claim extraction and verification call runs against gpt-4o-mini, not GPT-4. Running a full GPT-4 call for every response validation would double our inference cost and add 600–800ms of latency. With gpt-4o-mini, the validation step adds approximately 180–220ms on average for a 3–5 sentence response. That number is model-dependent, it will be higher on a slower model and lower with a fine-tuned classifier.

After deploying:

Before:  Faithfulness 0.67  |  Context Precision 0.61  |  31% unsupported claim rate
After:   Faithfulness 0.91  |  Context Precision 0.87  |  <4% unsupported claim rate

The Core Decision — And When You’d Make It Differently

We made an explicit trade-off: ~200ms of additional latency in exchange for verifiable answer quality.

For our use case — enterprise clients making operational decisions based on the chatbot’s answers — that trade-off was not a discussion. The 200ms is noise. The trust cost of a wrong answer at enterprise scale is not.

But this decision is not universal:

Constraint                            Decision          Rationale
──────────────────────────────────────────────────────────────────────────────
Consumer product, SLA < 500ms         Run async          Log failures, don't block.
                                                         Stakes are low, UX matters more.

Low-stakes (drafting, summarisation)  Skip it            User edits the output.
                                                         Grounding matters less.

High-volume, cost-sensitive           Sample 10%         Statistical signal at
                                                         1/10th the overhead.

Enterprise / regulated / high-stakes  Mandatory sync     A wrong answer has real
                                                         downstream consequences.

Multi-tenant, strict isolation        Mandatory + audit  Every response must be
                                                         traceable to a source chunk.

Principle: grounding validation is always worth measuring. Whether to block on it synchronously depends on your SLA and the cost of a wrong answer in your domain.

The Anti-Pattern: Checking Faithfulness After Generation Is the Wrong Architecture

Here is the deeper architectural mistake this exposed.

We were checking faithfulness after generation — as a post-hoc audit — rather than as a gating condition on the response pipeline. The audit told us something was wrong. It didn’t stop a wrong answer from being returned.

The correct architecture treats grounding validation as a blocking step in the response pipeline, not an observability metric reviewed after the fact.

Wrong: Generate → Return → [async] Validate → Log failure → Review weekly
Right: Generate → Validate → [if flagged] Regenerate → Return → Log

The async pattern gives you observability. It does not give you correctness. For any system where answer quality has downstream consequences, post-hoc monitoring is not a substitute for inline validation.

We caught our failure because a user noticed. That should never be the detection mechanism for a production system serving enterprise clients.

What We Changed Permanently

Quality monitoring is now first-class. RAGAS faithfulness and context precision scored against our golden evaluation set on every ingestion event and every deployment. Grafana alerts fire if either drops more than 10% from the established baseline. We do not run RAGAS on live traffic, it’s too slow and expensive at scale. The golden set gives us the signal we need.
Document ingestion now triggers a quality gate. When new documents are ingested, we run the benchmark query set against the updated index before traffic is shifted. Faithfulness drops >5% → ingestion rolled back.
Grounding validation is synchronous and non-configurable. The ~200ms cost is included in our SLA. Not optional for any enterprise-tier query.

The One-Line Takeaway

Your RAG system will hallucinate. The question is whether you find out before your users do.