What Is a RAG Pipeline? A Practitioner’s Guide to Production-Grade Retrieval-Augmented Generation
What Is a RAG Pipeline?
Let me tell you what a RAG pipeline is not. It’s not a magic wand that makes your LLM stop hallucinating. It’s not a “plug and play” library you install on Friday and ship to production on Monday. I learned this the hard way building data infrastructure at SIVARO in 2023.
A RAG pipeline (Retrieval-Augmented Generation pipeline) is a system architecture that connects a retrieval engine to a large language model. The retrieval engine fetches relevant documents from a knowledge base. The LLM generates answers conditioned on those documents. That’s the simple version.
The real version? It’s a distributed data system with latency constraints, recall requirements, and failure modes that will keep you up at night. We tested four different RAG architectures at SIVARO in early 2024. The naive version failed 30% of the time on simple factual queries. The production version we landed on runs at 400ms p95 latency and 94% answer accuracy.
Here’s what you’ll walk away with: the actual components, the engineering tradeoffs nobody talks about, and the concrete implementation patterns that work at scale. No fluff. No “it depends” without telling you what it depends on.
The Core Components (Not the Cute Diagram Version)
Every RAG pipeline has four components. Skip one and you’ll pay for it in production.
Document Ingestion and Chunking
You feed raw documents in. You get searchable chunks out. This sounds boring. It’s where most pipelines die.
We processed 50,000 technical manuals at SIVARO for a client in manufacturing. Naive chunking (500-character fixed windows) gave us 62% recall on technical queries. Overlapping chunks with sentence boundary detection pushed that to 81%. Using a dedicated chunking model from Jina AI in Dec 2023 got us to 89%.
The tradeoff: better chunking costs 2-3x more compute on ingestion. For our client’s use case, that was fine because ingestion happens once. For a real-time pipeline where documents stream in every second? You need streaming chunking. We built ours using Ray actors in Jan 2024 — each actor handles one document stream, chunks it, and pushes to the vector DB.
Key decision: chunk size and overlap. For question-answering on technical docs, 512 tokens with 20% overlap works. For conversational agents, 256 tokens with 30% overlap performs better. We benchmarked this. Don’t guess — run your own tests.
Vector Embedding and Indexing
You take the chunks, run them through an embedding model, store the vectors in a database.
Most people think “use OpenAI embeddings, put them in Pinecone, done.” That works for demos. For production? The embedding model matters more than the vector database. We tested text-embedding-ada-002 vs BAAI/bge-large-en-v1.5 vs intfloat/e5-mistral-7b-instruct. The instructor model from Intfloat outperformed OpenAI by 12% on our domain-specific queries. But it costs 20x more to run.
We choose the model based on data type. Code documentation? Use codebert-based embeddings. Medical records? Use PubMedBERT. General enterprise docs? Use e5-mistral with appropriate instruction prefixes.
For the vector store, we run both Qdrant and Weaviate in production at SIVARO. Qdrant handles high-throughput ingestion (we push 5K new docs/day for one client). Weaviate handles hybrid search (vector + keyword) for another. Both work. Neither is “better” — they solve different problems.
Hybrid search matters more than people admit. Pure vector search fails on exact-match queries. “Show me the invoice for order #12345” needs keyword matching. We built a hybrid index: HNSW for vectors + inverted index for keywords. 40% recall improvement on exact-match queries.
Retrieval Strategy
You have vectors. You have a query. You fetch relevant chunks. This is where the magic — and misery — happens.
Simple cosine similarity search returns the top-k nearest neighbors. That’s the baseline. We tested everything.
Multi-query retrieval: Generate 3 variations of the user’s query, search each, deduplicate results. Adds 90-120ms but improves recall by 15%. Worth it for fact-checking applications. Not worth it for chatbots where latency matters.
HyDE (Hypothetical Document Embeddings): Generate a synthetic answer first, embed that, search with it. We tried this in Feb 2024 on a legal document retrieval task. Precision improved 22%. But generation latency added 1.2 seconds. Only use this where answer quality trumps speed.
BM25 re-ranking: Do a cheap vector search to get top-50 candidates, then re-rank with BM25. This is our default at SIVARO. It costs almost nothing (BM25 is just TF-IDF on steroids) and catches the cases where embeddings miss exact phrase matches.
The dirty secret: retrieval recall is usually the bottleneck. Not the LLM. Not the generation. If you retrieve the wrong documents, the LLM doesn’t stand a chance. We’ve spent more time tuning retrieval than anything else.
Synthesis and Generation
You have relevant chunks. You feed them into an LLM with a prompt. You get an answer.
This seems straightforward. It’s not.
The prompt engineering for RAG is fundamentally different from standard LLM prompting. You need to handle:
- Irrelevant retrieved documents (the model should ignore them, not hallucinate from them)
- Missing documents (the model should say “I don’t know”)
- Conflicting documents (the model should weigh evidence)
We developed a structured prompt pattern at SIVARO that looks like this:
You are a technical assistant. Answer the question using ONLY the provided context.
Context:
{retrieved_chunks}
If the context doesn't contain enough information to answer fully, say so.
If chunks contradict each other, explain the contradiction and why.
Cite specific chunk IDs in your answer.
Question: {user_query}
This pattern reduced hallucination rates from 22% to 4% in our internal benchmarks (March 2024, 1,000 test queries). The key is the instruction to cite and the instruction to admit uncertainty. Without those, the LLM confidently makes things up.
Production Architecture for a RAG Pipeline
Let me give you the concrete architecture we run at SIVARO for a client processing 200K customer queries per day. This isn’t theoretical.
User Query → Query Router → Cache Check (Redis)
↓ (cache miss)
Query Expansion
↓
Hybrid Search (Qdrant)
↓
BM25 Re-ranker
↓
Dynamic Context Window
↓
LLM Generation (vLLM)
↓
Factuality Checker
↓
Answer + Citations
The Query Router is a lightweight classifier (just a Mistral 7B with a classification head) that decides whether the query needs RAG or can be answered with parametric knowledge. Saves 30% of calls from hitting the retrieval stack.
The Cache is critical. We hash the normalized query (lowercase, remove punctuation, normalize Unicode) and store the answer for 24 hours. Cache hit rate: 18-25%. These are cheapest queries you’ll ever run.
The Dynamic Context Window decides how many chunks to feed the LLM. We start with 3 chunks. If the LLM’s generation shows low confidence (we detect this from logprobs), we add 2 more chunks and retry. This added 8% accuracy at the cost of 15% increased latency. Worth it for high-stakes queries.
The Factuality Checker is a separate model (DeBERTa-v3 fine-tuned on NLI data) that checks if the answer is supported by the retrieved chunks. If it flags the answer, we fall back to a “I’m not sure” response. Catches 60% of remaining hallucination errors.
Code Examples: You Can Build This Today
Basic RAG with LangChain and Qdrant
python
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.llms import VLLM
from langchain.chains import RetrievalQA
embeddings = HuggingFaceEmbeddings(
model_name="intfloat/e5-mistral-7b-instruct",
model_kwargs={"device": "cuda"}
)
vectorstore = Qdrant.from_existing_collection(
path="./qdrant_data",
collection_name="docs",
embeddings=embeddings
)
llm = VLLM(
model="mistralai/Mistral-7B-Instruct-v0.2",
trust_remote_code=True,
max_new_tokens=512
)
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever(search_kwargs={"k": 5})
)
answer = qa.run("What is the maximum load capacity for the XR-400?")
This works. But it’s a toy. The k=5 is hardcoded. The chain_type="stuff" means you’re stuffing all chunks into one context — works for 5 chunks, fails for 50. No caching. No re-ranking. No factuality check.
Production Hybrid Search with Re-ranking
python
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sentence_transformers import SentenceTransformer
class HybridRetriever:
def __init__(self, documents, chunk_ids):
self.embeddings = SentenceTransformer('intfloat/e5-mistral-7b-instruct')
self.doc_embeddings = self.embeddings.encode(documents, normalize_embeddings=True)
self.vectorizer = TfidfVectorizer(stop_words='english')
self.tfidf_matrix = self.vectorizer.fit_transform(documents)
self.chunk_ids = chunk_ids
self.documents = documents
def search(self, query, k=20):
query_emb = self.embeddings.encode([query], normalize_embeddings=True)
vector_scores = np.dot(self.doc_embeddings, query_emb.T).flatten()
query_tfidf = self.vectorizer.transform([query])
keyword_scores = (self.tfidf_matrix @ query_tfidf.T).toarray().flatten()
normalized_vector = (vector_scores - vector_scores.mean()) / vector_scores.std()
normalized_keyword = (keyword_scores - keyword_scores.mean()) / keyword_scores.std()
combined = 0.7 * normalized_vector + 0.3 * normalized_keyword
top_k = np.argsort(combined)[-k:][::-1]
# BM25 re-ranking from pyserini or rank_bm25
from rank_bm25 import BM25Okapi
tokenized_corpus = [doc.split() for doc in self.documents[top_k]]
bm25 = BM25Okapi(tokenized_corpus)
tokenized_query = query.split()
bm25_scores = bm25.get_scores(tokenized_query)
final_order = np.argsort(bm25_scores)[::-1]
return [(self.chunk_ids[top_k[i]], self.documents[top_k[i]], bm25_scores[i])
for i in final_order[:5]]
This is closer to production code. Notice the hybrid combination (0.7 vector, 0.3 keyword), the normalization step (critical — otherwise vector scores dwarf keyword scores), and the BM25 re-ranking on top-k candidates.
Dynamic Context Window
python
import torch
class DynamicContextGenerator:
def __init__(self, llm, tokenizer, max_tokens=4096):
self.llm = llm
self.tokenizer = tokenizer
self.max_tokens = max_tokens
self.max_chunks = 8
def generate_with_context(self, query, chunks, user_prompt_template):
for num_chunks in [3, 5, 8]:
context = "
".join(chunks[:num_chunks])
prompt = user_prompt_template.format(context=context, query=query)
with torch.no_grad():
outputs = self.llm.generate(
prompt,
max_new_tokens=256,
output_scores=True,
return_dict_in_generate=True
)
generated = outputs.sequences[0]
logprobs = torch.nn.functional.log_softmax(
torch.stack(outputs.scores), dim=-1
)
avg_confidence = torch.exp(logprobs.max(dim=-1).values).mean().item()
if avg_confidence > 0.65 or num_chunks == self.max_chunks:
return self.tokenizer.decode(generated)
return None
The 0.65 threshold is empirical. We tuned it on 500 validation queries. Lower threshold means more retries but worse latency. Higher threshold means lower quality but faster responses. Pick based on your SLA.
Retrieval Quality: The Thing Nobody Measures
Everyone talks about “RAG quality” and measures it with vibes. At SIVARO, we use three metrics:
Hit Rate: What fraction of queries have at least one relevant document in the top-k retrieved results? We aim for 95% at k=10.
Mean Reciprocal Rank (MRR): The reciprocal rank of the first relevant document. 0.8 means the first relevant doc is usually at position 1 or 2. Below 0.5 is bad.
Answer Faithfulness: We use NLI models (TrueTeacher from Google, June 2023) to check if the LLM’s answer is entailed by the retrieved chunks. 0.9+ is good.
We run these metrics continuously. They feed into dashboards that trigger alerts when retrieval quality drops. You need this. Your users won’t tell you when the RAG pipeline starts returning bad answers — they’ll just stop using it.
Common Failure Modes (and How I Fixed Them)
Empty Retrieval: The query doesn’t match any document. This happens ~8% of the time in our systems. Fix: fallback to a web search or a “knowledge gap” database that logs the query for human review.
Irrelevant Retrieval: The retrieved chunks are technically relevant but don’t contain the answer. The embedding captures topic similarity, not question-answer relevance. Fix: fine-tune the embedding model on your Q&A pairs. We did this for a legal client in Feb 2024 — 34% improvement in answer faithfulness.
Context Overload: Too many chunks confuse the LLM. In one experiment, feeding 20 chunks dropped accuracy by 18% compared to 5 carefully selected chunks. Fix: dynamic context window that adds chunks only when needed, never more than 8.
Stale Data: Retrieved documents are outdated. The LLM doesn’t know the document is wrong — it just uses it. Fix: timestamp each chunk, and include a “last_updated” field in the context. We prompt the LLM to prefer newer documents when conflicts exist.
FAQ
What is a RAG pipeline and why should I care?
It’s the dominant pattern for grounding LLMs in your own data without fine-tuning. You should care because fine-tuning is expensive, slow, and doesn’t let you update data without retraining. RAG lets you add a new document today and have the LLM answer questions about it tomorrow.
What is a RAG pipeline’s biggest weakness?
Retrieval failure. If you don’t retrieve the right document, the LLM will either hallucinate or admit ignorance. We’ve seen pipelines with 99%+ generation quality but 70% retrieval recall — overall system quality <70%. Fix retrieval first.
Do I need a vector database?
Yes, if you have more than 10K documents. No, if you have fewer — you can use in-memory FAISS indices or even brute-force cosine similarity. For mid-scale (10K-1M docs), Qdrant or Weaviate are good. For 1M+ docs, Pinecone or Milvus handle scale better.
How much latency should I expect?
End-to-end: 500ms to 2 seconds for production systems. Embedding the query: 50-200ms. Vector search: 10-50ms. LLM generation: 300ms-1.5s (depends on model size and output length). If you need sub-200ms, you’re building a different architecture — maybe a cache-first approach with periodic pre-computation.
Can I use GPT-4 in a RAG pipeline?
Yes, but it’s expensive. We replaced GPT-4 with Mistral Large in one pipeline and saw latency drop 60% and cost drop 90%. Quality? Within 2% on our automated metrics. The key is good retrieval — a weaker LLM with excellent retrieval beats a strong LLM with bad retrieval every time.
What embedding model should I use for a RAG pipeline?
Start with intfloat/e5-mistral-7b-instruct for general use. Switch to domain-specific models if your data is specialized (medical, legal, code). Don’t use OpenAI’s text-embedding-3-small unless you’re already deep in their ecosystem — the open models are competitive and cheaper to run at scale.
What is a RAG pipeline’s role in production AI systems?
It’s the bridge between static knowledge bases and dynamic question answering. In production, it replaces custom search systems and FAQ pages. We’ve seen support ticket deflection rates go from 20% to 45% after deploying RAG-based chatbots. But it requires ongoing monitoring — retrieval quality degrades over time as data grows and query patterns shift.
The Hard Truth About RAG in Production
I’ve been building data infrastructure since 2018. RAG pipelines are harder than they look. The demo always works. Production is where the patterns break.
You’ll spend 70% of your time on retrieval quality, 20% on prompt engineering, 10% on infrastructure. That’s the inverse of what most teams allocate.
You’ll need monitoring. We use OpenTelemetry to trace every query through the pipeline. Redis for caching failures and successes alike. Grafana dashboards showing retrieval recall and answer faithfulness by the hour.
You’ll need fallbacks. When retrieval returns nothing. When the LLM times out. When the vector DB is down. We have three fallback layers: cache, simple keyword search, and a static FAQ.
Most people think RAG is a solved problem. It’s not. The field moves weekly. New embedding models drop every month. New chunking strategies appear. The pipeline that worked in January might be obsolete by June.
But if you get it right — good retrieval, solid prompt engineering, intelligent fallbacks, continuous monitoring — you’ll have a system that answers questions accurately, using your data, without hallucinations. That’s worth the fight.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.