Running RAG in production takes far more than connecting a language model to a vector database. A production RAG feature needs structure-aware chunking, hybrid search with reranking, a golden evaluation set you run on every change, retrieval that enforces user permissions, guardrails against hallucination and prompt injection, and monitoring of quality, latency and cost. The model is rarely the weakest link; retrieval and evaluation usually are.

This guide is written for engineering teams moving an LLM feature from a promising demo to something customers rely on. It stays at the level of patterns and categories, with no benchmark claims, because the right choices depend on your data.

What is RAG and when is it the right choice?

Retrieval-augmented generation (RAG) is a pattern in which the system retrieves relevant passages from your own data and passes them to a language model as context, so its answer is grounded in sources you control.

RAG fits when:

  • The knowledge changes often (policies, documentation, tickets, product data).
  • Answers need citations that users can check.
  • The data is private and access varies by user.

It is the wrong tool when the question needs computation over the whole dataset ("how many orders failed last week?"), which is better served by giving the model a query tool over structured data. It is also unnecessary when the relevant material is small and stable enough to include in the prompt directly.

How does a production RAG pipeline fit together?

LLM feature engineering for RAG is really two systems: an offline ingestion pipeline and an online query path. Problems in either show up as bad answers, so both need ownership.

Ingestion (offline):

  1. Connect to sources and fetch changes incrementally.
  2. Parse and clean: strip boilerplate, keep headings, tables and lists.
  3. Chunk the content and attach metadata: source, section path, timestamps, language and access-control lists.
  4. Embed each chunk and index it for both vector and keyword search.
  5. Record versions of the parser, chunker and embedding model used.

Query (online):

  1. Rewrite or expand the query if needed.
  2. Retrieve candidates with hybrid search, filtered by the user’s permissions.
  3. Rerank and select a small number of passages.
  4. Assemble the prompt with passages and citation markers.
  5. Generate, check the output, log the trace and return the answer.

How should you chunk documents and choose embeddings?

Chunking is the step that splits documents into retrievable passages, and it has more influence on quality than most teams expect. A chunk that cuts a policy in half returns the rule without its exception.

  • Follow the document’s structure. Split on headings and sections first, then by size within long sections.
  • Prefix each chunk with its context, such as document title and section path, so the passage makes sense on its own.
  • Keep a link to the parent section. Retrieve small chunks for precision, then expand to the surrounding section when building the prompt.
  • Treat tables and code differently. Keep tables whole where possible and keep code blocks intact.

For embeddings, choose the model by testing on your own golden set, not by leaderboard position. Check language coverage if your users write in more than one language. Store the raw chunk text and the embedding model version with every vector, because changing models means re-embedding the corpus, and you want that to be a routine batch job rather than a migration.

Why use hybrid search and reranking?

Vector search finds passages that mean something similar to the query. It is weak on exact identifiers: error codes, SKUs, people’s names, clause numbers. BM25 is a keyword ranking function that scores documents by how often query terms appear and how rare those terms are, and it handles exact matches well.

Hybrid search runs both and merges the results, commonly with reciprocal rank fusion, a method that combines lists based on each item’s rank rather than raw scores. Then a reranker, typically a cross-encoder model that reads the query and each passage together, reorders the top candidates. Rerankers are slower than first-stage retrieval, so apply them to a few dozen candidates, not the whole index.

Metadata filters (product, region, document type, date) narrow the search before ranking and often help more than a better embedding model.

Which vector store should you use?

There is no universally right answer, and published benchmarks rarely reflect your data, filters and update patterns. Compare categories against your own requirements.

Option Strengths Watch-outs Fits when
Postgres with pgvector Vectors live beside relational data; transactions; joins for permissions; familiar operations Tuning and scaling vector indexes is your job; heavy vector load competes with the main database You already run Postgres and the corpus is moderate
Dedicated vector database Built for vector workloads; managed options; tenancy and filtering features Another system to secure, sync and pay for; permissions data must be replicated Vector search is a large, independent workload
Search engine with vector support Mature keyword search and vector search in one place Operational weight if you do not already run one You already operate a search cluster or need strong keyword search

Whichever you pick, keep the ingestion pipeline independent of the store, so you can re-index elsewhere without rewriting connectors.

How do you evaluate a RAG system?

RAG evaluation is the discipline of measuring retrieval and generation separately, on a fixed set of questions, every time something changes. Without it, every prompt tweak is a guess.

Build a golden set:

  • Collect real user questions, or realistic ones written with subject-matter experts.
  • For each, record the passages that should be retrieved and a reference answer.
  • Include questions the system should decline because the answer is not in the corpus.
  • Version the set and add every production failure you fix.

Measure retrieval:

  • Recall at k: the share of questions where a correct passage appears in the top k results.
  • Rank of the first correct passage, which shows whether reranking is doing its job.

Measure generation:

  • Faithfulness (groundedness): whether every claim in the answer is supported by the retrieved context.
  • Correctness against the reference answer.
  • Citation accuracy: whether cited passages actually support the sentence they are attached to.
  • Correct refusal on unanswerable questions.

Model-graded evaluation scales this, but calibrate the grader against human labels on a sample and re-check it when you change the grading model. Run the suite in CI and treat a regression like a failing test.

How do you keep retrieval permissions-aware?

Permissions-aware retrieval means a user can only retrieve passages from documents they could open in the source system. This is the most serious failure a RAG feature can have, because it leaks data through an interface that looks harmless.

  • Capture access-control lists at ingestion and keep them in sync with the source.
  • Filter inside the retrieval query, not after it. Post-filtering a top-k list can return nothing useful and can leak information through counts or timing.
  • Enforce tenant isolation in the data layer with a mandatory tenant filter or separate indexes.
  • Propagate deletions and permission changes promptly, including to caches.
  • Never rely on prompt instructions to hide content. If it reaches the prompt, assume it can reach the user.
  • Test retrieval as several personas with different access, as part of the evaluation suite.

What guardrails does a RAG feature need?

Guardrails are checks before and after the model call that keep outputs safe, grounded and in scope.

  • Indirect prompt injection. Retrieved documents can contain instructions. Treat retrieved text as data, keep tool permissions minimal and never let retrieved content trigger actions on its own.
  • Grounding. Require citations, run a groundedness check on the answer and let the system say it does not know.
  • Input checks for abuse, out-of-scope requests and personal data that should not be sent to a model provider.
  • Output checks for personal data, format validity and policy violations.
  • Human review for high-stakes answers, such as anything legal, medical or financial.

How do you control latency and cost?

Latency and cost grow with every extra passage, model call and retry. Set a budget for each stage and measure against it.

  • Stream tokens to the user and run retrieval steps in parallel.
  • Send fewer, better passages after reranking instead of filling the context window.
  • Route simple queries to smaller, cheaper models and reserve larger ones for hard cases.
  • Cache embeddings and, carefully, answers, keyed by permissions as well as the query.
  • Use provider prompt caching for stable system prompts where available.
  • Batch embedding during ingestion and re-embed only changed chunks.
  • Set token limits, timeouts and a graceful fallback when the model is slow or unavailable.

What should you monitor in production?

Trace every request end to end: the rewritten query, retrieved chunk IDs and scores, reranker output, prompt and model versions, token counts, latency per stage, the answer, its citations and any user feedback. Store traces with the same access controls and retention rules as the underlying data, because they contain it.

Watch these over time: latency percentiles, cost per request, empty-retrieval rate, refusal rate, negative-feedback rate, groundedness on sampled traffic, and ingestion lag, the time between a source document changing and the index reflecting it.

What are the common failure modes of RAG in production?

Failure mode Symptom Fix
Chunks split key information Answers miss conditions or exceptions Structure-aware chunking; expand to the parent section
Exact terms missed Fails on error codes, product names, IDs Add BM25 and hybrid search
Correct passage ranked too low Right source appears in traces, but outside the prompt Add a reranker; tune candidate and final counts
Stale index Answers cite superseded documents Incremental ingestion; freshness metadata; propagate deletions
Unsupported claims Fluent answers without citations Require citations; groundedness check; allow refusal
Permission leak Users see content from documents they cannot open Filter by ACL in the retrieval query; persona tests
Prompt injection in sources Model follows instructions found in documents Treat context as data; restrict tools; output checks
Context overload Slow, costly answers that overlook details Fewer reranked passages; tighter selection
Silent regression Quality drops after a prompt or model change Golden-set evaluation in CI with a baseline
Vague queries Irrelevant results for short questions Query rewriting; ask a clarifying question

Where does NexaPrism fit in shipping RAG?

Production RAG is mostly search, data and backend engineering with a model in the middle, and many teams have a defined piece of it to get done: building the evaluation harness, fixing retrieval quality, adding permissions-aware ingestion or shipping the feature end to end. NexaPrism takes on that work as a project, with senior engineers experienced in AI and ML systems, including LLMs and RAG, and in the data pipelines that feed them. A senior engineer reviews the problem; we agree scope, deliverables, timeline and price with you before work starts; then a lead engineer and senior engineers build it, show working software every week and hand it over documented. The code is yours from the first commit, in your repositories wherever possible.

If you are handing a project like this to an outside team, our checklist for onboarding an external engineering team covers access and context. To talk through a specific feature, get in touch.