Getting RAG Systems Into Production (Without the Silent Failures)

Retrieval-augmented generation is the default architecture for most production LLM applications now, and for good reason: it's how you get a model to answer from your data instead of its training set. But the demo-to-production gap for RAG is brutal, and it fails in a specific way. The prototype works on the ten documents you tested with, ships against ten thousand, and starts returning confidently wrong answers. Nobody files a bug, because the answer looks right. You find out from a support ticket, or worse, from a customer who acted on it.

The uncomfortable truth is that most RAG failures are retrieval failures, not model failures. When RAG returns a bad answer, the instinct is to blame the LLM or reach for a bigger model. The real problem is upstream in most cases: the retrieval step handed the model the wrong context, and no model can reason its way out of a bad set of source documents. That is the through-line for everything below and the core of how we approach RAG systems: fix retrieval quality first, then grounding, then everything else.

Why "embed everything and retrieve top-K" breaks

The canonical starter RAG is three steps: chunk your documents, embed them into a vector database, and at query time embed the question and return the top-K most similar chunks. It works beautifully in a notebook and degrades sharply the moment your corpus grows.

The problem is that pure vector similarity is a blunt instrument. Top-K nearest-neighbour search returns chunks that are semantically near the query but don't contain the answer, and misses the chunk that does when it phrases things differently from the question. As the corpus grows, near-misses crowd out real hits, and the model synthesizes a plausible answer from whatever you handed it. That's the silent failure.

Two fixes matter more than any model upgrade:

• Hybrid search. Combine vector similarity with keyword (lexical) search. Vector search captures meaning; keyword search catches exact terms, product names, error codes, and identifiers that embeddings blur together. Most real queries need both.
• Re-ranking is non-negotiable at scale. After retrieving a broad candidate set, run it through a re-ranking model (Cohere Rerank is a common choice) that scores each candidate against the query directly. It's the single highest-leverage addition to a production pipeline, and skipping it is the most common reason a RAG system that demoed well falls apart in front of users.

Chunking is a design decision, not a default

The second place RAG systems break is ingestion. The lazy default, splitting every document into fixed-size chunks of N tokens, treats a legal contract, a support article, and a table of figures identically. A fixed chunk boundary drawn through the middle of a table, a code block, or a numbered clause destroys the meaning the retriever needs.

Documents have structure, and good chunking respects it: split on headers and sections, keep tables and lists intact, carry section metadata with each chunk so retrieval can filter by it. Query-document length asymmetry matters too; a short question and a long document do not embed the same way, and the strategy should account for that.

There's an operational dimension people forget: ingestion is not a one-time job. Documents get updated, superseded, and deleted. If your pipeline does not handle re-ingestion and deletion, your vector store fills with stale chunks that outrank the current answer: a data-freshness bug that looks exactly like a model bug.

The vector database is a cache, not the source of truth

A recurring architectural mistake is treating the vector database as the system of record. It isn't. A vector DB is a cache of embeddings; the source of truth lives in Postgres or a document store. Embeddings get recomputed when you change your embedding model or chunking strategy, and you will change both, so anything you cannot rebuild from the source of truth is data you risk losing.

Which vector store to use is a per-project decision, not a default:

• pgvector (Postgres) when you already run Postgres and are under roughly 10M vectors: embeddings and source of truth in one database, which simplifies operations. Our take is on the Supabase / Postgres page.
• Pinecone for large hosted workloads where you don't want to operate the store yourself.
• Weaviate when you want hybrid search built in.
• Qdrant when self-hosting and performance-per-dollar matter.

If you are weighing these against your real constraints (scale, cost, existing infrastructure), our Tech Stack Picker walks the trade-offs without a sales call attached.

Grounding and citations: no citation, no trust

A RAG answer with no source is an assertion, and users have learned not to trust ungrounded LLM assertions. Citation is what makes the system usable, not a nice-to-have bolted on at the end. When every answer links back to the chunk and document it came from, users can verify, and verification drives adoption. Citations also give you a debugging handle: when an answer is wrong, you can see which retrieved chunk led the model astray and whether the failure was retrieval or synthesis.

Grounding goes a step further: detecting when the retrieved context does not support an answer, and having the system say "I don't have that" instead of confabulating. Confidence scoring and hallucination-detection patterns let you draw that line by design rather than hoping the model draws it for you.

You cannot ship RAG without an eval harness

This is the discipline that separates RAG systems that stay good from ones that rot. RAG quality degrades silently: a corpus change, an embedding-model swap, a chunking tweak, or a prompt edit can each regress retrieval without warning, and without evaluation you find out from your users.

An eval harness is not exotic: a ground-truth dataset of representative questions with known-relevant source documents, run against the pipeline on every meaningful change. Frameworks like Ragas exist for this, and custom evals are often worth building for your domain. The point is regression testing for retrieval: when someone changes the chunker, the eval tells you whether retrieval got better or worse before it ships. Treat it like unit tests, because that's what it is for a RAG system.

Where compliance changes the architecture

RAG shows up across industries (internal knowledge bases, customer-facing assistants, document search, regulatory lookups), and in regulated domains the compliance requirement reshapes the design rather than sitting on top of it. HIPAA-architected healthcare retrieval, for example, keeps the source of truth in a compliant data store, treats the vector DB strictly as a cache, and designs the ingestion and access path around the compliance boundary from the start. That is the same pattern that makes RAG good in general, which is why building for compliance and building for quality tend to converge. Our stance is on the HIPAA page, and if you're wiring RAG into existing data sources, the Integration Readiness Checker names the real risks before you start.

A sequence that works

For a team taking RAG from prototype to production, the order matters:

1. Nail retrieval before you touch the model. Hybrid search and re-ranking in place, retrieval quality measured directly. This is where the wins are.
2. Make chunking a first-class design step. Respect structure, carry metadata, handle re-ingestion and deletion from the start.
3. Add citations and grounding early. They drive adoption and double as your debugging instrument.
4. Build the eval harness before you scale the corpus. Ground-truth datasets and regression tests are how you keep quality from rotting.
5. Keep the source of truth separate from the vector cache, so you can re-embed freely as models improve.

RAG also sits inside larger AI systems, as a retrieval tool inside an agentic workflow, or one piece of a broader AI/ML build, and the same retrieval-first discipline carries over.

If you're scoping a RAG build and want the architecture right before writing code, a Discovery Sprint is the right container. If you already have RAG in production and suspect it's failing silently, a Tech Audit diagnoses retrieval quality, grounding, and eval coverage. And if you need senior engineers who think in retrieval quality rather than embeddings, that's what our staff augmentation is for.

The through-line

The teams that win with RAG aren't the ones on the newest model. They're the ones who treated retrieval as the hard part: hybrid search with re-ranking, structure-aware chunking, a vector store used as a cache with a real source of truth behind it, citations that build trust, and an eval harness that catches regressions before users do. Build that, and RAG becomes a durable asset instead of a demo that quietly stopped working.