Designing a Production RAG System That Doesn't Fall Apart at Scale

Most "RAG systems" you see in demos are toys. They work great on a five-document notebook and fall apart the moment you point them at a real corpus with messy formats, conflicting sources, and users who ask questions the author never imagined.

This is the playbook I use when I'm building retrieval into a production product — the architecture, the failure modes, and the evals that catch problems before your customers do.

The architecture that actually works#

A production RAG system is not "embed your docs, store them in a vector database, query the closest five." It's a pipeline:

1. Ingestion. Normalize documents from their source of truth (Notion, PDFs, dashboards, support tickets, internal wikis, code). Strip headers, footers, navigation, and other chrome. Keep structure — headings, lists, code blocks — because chunking on structure is dramatically better than chunking on characters.

2. Chunking. Default to recursive, structure-aware chunking with overlap. For code, use language-aware splitting. For long-form docs, anchor chunks on headings. A 512-token chunk with 50 tokens of overlap is a fine starting point; tune from there.

3. Embeddings. Pick a model that matches your domain. For mixed English

   • technical content, OpenAI's text-embedding-3-large or a strong open model like bge-large-en-v1.5 works well. Re-embed when you change models. Don't reuse embeddings across models — that's how you get silently wrong results.

4. Retrieval. Hybrid search almost always beats pure vector search. Combine BM25 (lexical) with dense embeddings (semantic) and weight them per-query. For a knowledge base, weights like 0.4 bm25 + 0.6 dense are a reasonable default.

5. Reranking. A cross-encoder reranker over the top 30–50 candidates is the single biggest quality win in most pipelines. It is also the cheapest place to spend latency: reranking is fast and dramatically improves precision.

6. Context construction. Pack the top results into the prompt with citations, character budgets, and explicit "if you don't know, say so" instructions. Always include the question in the context window.

7. Answer generation. Constrain the model to the retrieved context. Stream responses. Show citations inline.

8. Evals. Run an offline eval suite on every change. Track online metrics for retrieval quality, answer faithfulness, and user feedback. Without evals you are guessing.

Failure modes nobody warns you about#

Semantic search drift#

Embeddings cluster semantically similar concepts together, even when they're not the same thing. "How do I reset my password" and "I forgot my password" cluster correctly — but so do "password reset email" and "password rotation policy," which are different documents.

This is why hybrid search + reranking matter. BM25 catches the lexical match. The cross-encoder reranker fixes the ranking. Pure dense search will silently serve you the wrong doc.

Chunking at the wrong boundary#

If you split a procedure across two chunks, neither chunk is answerable on its own. The answer comes back as a confused half-step. Fix this with overlap and structure-aware splitting. For procedures, prefer chunking on step boundaries.

Stale embeddings#

Every time you change your embedding model, you must re-embed the entire corpus. Most teams forget this and end up with mixed embeddings from two different models — and that's where retrieval quality mysteriously collapses.

The "always say yes" model#

LLMs default to answering. They'll happily invent answers from irrelevant context or hallucinate when retrieval returns nothing useful. Solve this with prompt-level refusals ("if the answer isn't in the context, say so") and a fallback path that surfaces human handoff when confidence is low.

Evals you actually need#

You don't need a research-grade eval framework. You need three things:

• Golden set. 50–200 hand-labeled question/answer pairs covering the topics your users actually ask about. Update quarterly.
• Retrieval eval. For each question, did we retrieve a chunk that contains the answer? Track recall@k and MRR.
• Answer eval. Use an LLM-as-judge with a rubric (faithfulness, helpfulness, citation accuracy) to score generated answers. Sample 10–20% of traffic and send to humans for spot checks.

If you can't tell whether a change made things better or worse, you don't have a system — you have a prototype.

What to ship first#

If you're building RAG for the first time:

1. Start with a small, clean corpus. Get retrieval quality above 90% recall@5 before you add anything else.
2. Add a reranker before you add a fancier embedding model. The reranker is a bigger quality win.
3. Build the eval harness on day one. It pays for itself the first time you ship a change that breaks retrieval and you catch it before customers do.
4. Add citations, character budgets, and explicit refusal prompts in the prompt template before you tune anything.

When RAG isn't the answer#

Sometimes the right answer is fine-tuning. Sometimes it's a smaller model with a curated prompt. Sometimes it's a rules engine. Retrieval is the right tool when your knowledge is large, changing, and reference-shaped — which is most real products.

If your "knowledge base" is 20 documents and rarely changes, retrieval is overengineering. If it's 200,000 documents and changes daily, retrieval is the only viable architecture.

---

I'm currently taking on a small number of RAG engagements this quarter. If you need a production retrieval system that doesn't fall apart at scale, let's talk.