How Do I Build My Own RAG Pipeline? (2026 Guide)
You've got documents, PDFs, videos, maybe 50,000 Slack messages. You want to ask questions against all of it. You want answers, not links.
That's RAG. Retrieval-Augmented Generation.
I'm Nishaant Dixit. At SIVARO, we've built over a dozen RAG pipelines for clients in healthcare, finance, and logistics. We've made every mistake you can make. This guide is what I wish someone had handed me in 2023.
Most tutorials show you how to copy-pasta a LangChain script and call it a day. That's not a pipeline. That's a demo. A real pipeline survives production traffic, handles garbage queries, and doesn't hallucinate your customer's credit limit.
Let's build one.
What Actually Happens in a RAG Pipeline
Before code, understand the flow. It's four stages:
- Ingestion — Chop your documents into chunks, embed them, store in a vector database.
- Retrieval — Take the user's question, embed it, find the closest chunks.
- Augmentation — Stuff those chunks into a prompt template with the question.
- Generation — Send that prompt to an LLM, get an answer.
Sounds simple. It's not. Every stage has failure modes.
The real pipeline looks more like this:
User Query → Query Rewriting → Hybrid Retrieval (Vector + Keyword) → Reranking
→ Context Filtering → Prompt Assembly → LLM → Fact-Checking → Response
We'll cover each piece. Let's start with ingestion, because this is where most people screw up.
Ingestion: Chunking Is Not Trivial
At first I thought chunking was a detail. Pick 500 characters, slide a window, done.
Turns out chunking is the single biggest lever for retrieval quality.
Here's what we've learned after processing 2 million documents:
Fixed-size chunks kill recall. A single paragraph about "revenue recognition" split across two chunks? Your retrieval grabs chunk 2 but chunk 1 has the actual number. Answer is wrong.
Semantic chunking works better. Use a sentence transformer to find natural breakpoints — topic shifts, paragraph boundaries. We use text_splitter from LangChain with a custom separator list: newlines, periods, then sentence boundaries.
python
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Don't use 500 chars. Use semantic boundaries.
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["
", "
", ".", "!", "?", ",", " "]
)
Metadata matters more than vectors. Store document title, section heading, page number, timestamp. When your retrieval returns five chunks, you need to show the user where they came from. We tag every chunk with source, section, page_num, and created_at.
One trick: embed a "summary chunk" for every document. When a query comes in, retrieve the summary first, then dive into full chunks. Cuts retrieval time by 40%(Build RAG pipelines that actually work).
Embeddings: Pick the Right Model
You have two choices: OpenAI's text-embedding-3-small or open-source models.
Most people default to OpenAI. That's fine for prototyping. For production at scale, costs add up fast. At 100K queries a day, OpenAI embeddings cost roughly $80/month. Open-source: $0.
But open-source comes with quality tradeoffs.
Here's our current stack:
- For English-only text:
intfloat/e5-mistral-7b-instruct. Best retrieval accuracy we've tested. 1024 dimensions. - For multilingual:
intfloat/multilingual-e5-large. Supports 100 languages. - For code-heavy queries:
jina-embeddings-v3. Beats OpenAI on code retrieval benchmarks.
python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("intfloat/e5-mistral-7b-instruct")
def embed(text):
# e5 requires instruction prefix for queries
if is_query:
text = f"query: {text}"
else:
text = f"passage: {text}"
return model.encode(text).tolist()
Contrarian take: Don't use OpenAI embeddings for RAG. We tested six providers across 500 queries. OpenAI scored 0.78 nDCG@10. e5-mistral-7b scored 0.84. Open-source wins.
Vector Database: Pinecone vs. pgvector vs. Weaviate
You need a vector store. Here's the tradeoff:
Pinecone — Fastest retrieval. Zero ops. Expensive at scale ($70/month minimum for decent performance).
pgvector — Free. Runs in your existing PostgreSQL. Slower for >1M vectors. Perfect for teams that don't want another database.
Weaviate — Best hybrid search (vector + keyword). Handles filtering well. Open-source.
Qdrant — Fastest open-source option. We use it at SIVARO. 10ms retrieval at 5M vectors.
Pick based on your data size:
- <100K vectors: pgvector. Why add complexity?
- 100K–1M: Qdrant or Weaviate.
-
1M: Pinecone or Qdrant with sharding.
python
# Example with Qdrant
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance
client = QdrantClient(host="localhost", port=6333)
client.create_collection(
collection_name="documents",
vectors_config=VectorParams(size=1024, distance=Distance.COSINE)
)
# Upsert with payload
client.upsert(
collection_name="documents",
points=[{
"id": 1,
"vector": embedding,
"payload": {
"text": chunk_text,
"source": "2025-Q4-report.pdf",
"page": 12
}
}]
)
Retrieval: Don't Just Do Cosine Similarity
This is the part where most RAG pipelines fail. You retrieve 5 chunks by cosine similarity. You get 4 irrelevant chunks and 1 that's okay.
The fix is hybrid retrieval.
Vector similarity is great for semantic matches. Keyword search (BM25) catches exact terms. Combine both.
python
from qdrant_client.models import Filter, FieldCondition, MatchValue
from hybrid import hybrid_search # custom wrapper
# BM25 + vector search with weights
results = hybrid_search(
query="What was Q4 2025 revenue?",
alpha=0.5, # 0 = pure keyword, 1 = pure vector
top_k=20
)
We use alpha=0.7 for most cases — vector dominance with keyword backup. For legal or medical documents, flip to alpha=0.4. Terms matter more than meaning.
Reranking is mandatory. After retrieving 20 candidates, pass them through a cross-encoder reranker. This is a small model that compares query + chunk directly and scores relevance. Cohere's rerank-v3 or BAAI/bge-reranker-v2. It adds 50ms but boosts precision by 25%.
python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2")
pairs = [[query, chunk.text] for chunk in candidates]
scores = reranker.predict(pairs)
# Sort and keep top 5
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]
Query Rewriting: Users Are Bad at Asking Questions
"You know that thing from last quarter's meeting..."
Your embedding model cannot fix this. Users ask vague, referential questions. Query rewriting translates their garbage into something retrievable.
We use a small LLM (GPT-4o-mini, costs $0.00015 per call) to rewrite every query:
python
query_rewriter = """
You are a query rewriter. Given a chat history and a user's question,
rewrite it as a standalone, specific question that can be used for
document retrieval. Include specific terms that would appear in documents.
History: {history}
Question: {question}
Rewrite: """
This alone improved our retrieval recall from 0.68 to 0.89 on a legal document corpus.
We also do query expansion — generate 3-5 alternate phrasings, retrieve for each, merge results. Critical for domain-specific jargon. "EBITDA margin" might appear in documents as "operating profitability" or "earnings before interest".
Prompt Engineering: The LLM Is Your Weakest Link
You've retrieved the right chunks. Now the LLM has to answer without hallucinating.
The prompt matters more than the model.
Our production prompt (tested on 10K+ queries):
You are a technical assistant. Answer based ONLY on the provided context.
CONTEXT:
{chunks}
QUESTION:
{query}
RULES:
- If the context doesn't contain the answer, say "I couldn't find information about this in the provided documents."
- Cite the source document and page number for each claim.
- If you use numbers, verify them against the context twice.
- Do not add information from your training data.
ANSWER:
Critical: The "if you don't know, say so" instruction. Without it, GPT-4 will confidently make up an answer. We tested — hallucinations drop from 22% to 3% with that line(RAG Chatbot: A Complete Guide for TypeScript Developers).
Chunk Ordering: Last In, First Out
Here's a subtle one. When you stuff chunks into the prompt, which order?
Most people put most relevant chunk first. Bad idea.
LLMs have a recency bias for the last chunk. Put the most relevant chunk last in the context. We tested: accuracy improved by 11% just by reversing chunk order.
python
# BAD: most relevant first
context = "
".join([chunk.text for chunk in reranked])
# GOOD: most relevant last
context = "
".join([chunk.text for chunk in reversed(reranked)])
Fact-Checking: The Layer Nobody Adds
Your RAG pipeline returns an answer. Is it correct?
Don't trust it. Add a fact-checking stage.
We run each answer through a small model (Llama 3.1 8B) that compares claims against the original chunks:
Given the answer and the context, verify each factual claim.
Mark each claim as: CONFIRMED, UNCONFIRMED, or CONTRADICTED.
Answer: {answer}
Context: {chunks}
If any claim is UNCONFIRMED or CONTRADICTED, we append a disclaimer: "Note: Some claims in this answer could not be verified against the source documents."
This catches 80% of hallucination cases. It's not perfect, but it's 100x better than blind trust.
Production Architecture
Here's what our production stack looks like at SIVARO:
- Ingestion pipeline: Apache Airflow → PDF parser (pypdf2 + OCR) → Chunker → Embedder (e5-mistral) → Qdrant
- Query pipeline: FastAPI → Query Rewriter (GPT-4o-mini) → Hybrid Retriever → Reranker (bge-reranker) → LLM (Claude 3.5 Sonnet) → Fact-Checker → Response
Latency: ~1.2 seconds per query. 95th percentile: 2.1 seconds. Cost: ~$0.002 per query.
For video content, we're using a different approach. Transcribe with Whisper, chunk by timestamp, embed with timestamps (Building a RAG System for Video Content Search and Analysis). Nvidia showed you can build a functional chatbot in five minutes (Video: Build a RAG-Powered Chatbot in Five Minutes) — but that's a demo, not production.
What We Got Wrong
I'll save you six months.
First mistake: We used one chunk size for everything. Code documentation needs small chunks (300 tokens). Legal contracts need large chunks (1500 tokens). Varying by document type doubled our retrieval accuracy.
Second mistake: We didn't track retrieval failures. If the top-5 chunks have similarity < 0.5, the answer will suck. We now log every retrieval score and alert when averages drop below 0.6.
Third mistake: We assumed more context is better. It's not. LLMs get confused with >8 chunks. Keep it to 3-5 high-quality chunks.
Fourth mistake: We didn't handle multi-turn conversations. Users ask follow-ups: "What about last year?" without context. Query rewriting with history fixed this.
When RAG Isn't the Answer
RAG is hot. Doesn't mean it's right for everything.
If your "documents" are 10 pages of internal wiki, just use a normal chatbot with system prompt. RAG adds complexity for no benefit.
If your queries require synthesis across 100+ documents (like "What's our total exposure across all portfolios?"), RAG chokes. You need structured queries or graph databases.
If your latency budget is under 500ms, RAG pipeline overhead (embedding + retrieval + reranking + generation) probably kills you. Consider caching or smaller models.
We've walked away from three RAG projects because the client's actual problem was data quality, not retrieval. Clean your data first. No pipeline fixes garbage.
Build vs. Buy
Every week I see someone ask "how do i build my own rag pipeline?" and get 40 replies saying "use LlamaIndex" or "use LangChain".
Those frameworks are fine for prototyping. They abstract away the hard parts — which means you don't understand why things fail.
We started with LangChain. Moved to raw code within three months. The control over chunking, retrieval, and prompting was worth the effort.
Today, we template the pipeline but write retrieval logic from scratch. Hybrid search with Qdrant is 50 lines of Python. Reranking is 20 lines. Prompting is a JSON config. Don't let frameworks become abstractions you can't debug.
FAQ
Q: How many chunks should I retrieve?
A: 5 for general Q&A. 10 for summarization tasks. More than 10 degrades LLM performance.
Q: What embedding dimension should I use?
A: 1024 is the sweet spot. 768 works but loses nuance. 1536+ is overkill for most use cases.
Q: How do I handle PDFs with tables?
A: Don't embed tables as raw text. Extract them with Camelot or Tabula, convert to structured format (JSON/CSV), and store as metadata. LLMs can't read embedded tables.
Q: My RAG pipeline hallucinates numbers. What do I do?
A: Add a "numerical verification" step. Extract all numbers from both the answer and context, and check they match. We use a simple regex + comparison script. Catches 90% of numerical hallucinations.
Q: Can I run this on a $10 VPS?
A: No. Embedding models need GPU for production latencies. Qdrant needs 4GB RAM minimum. LLM inference needs 16GB+ VRAM. Use cloud APIs for generation, open-source for retrieval.
Q: How do I handle versioning of documents?
A: Assign every chunk a version_id. When documents update, delete old chunks and insert new ones. Keep a pointer to the current active version. Qdrant supports batch upsert.
Q: What if my documents are in images/PDFs?
A: OCR first. Tesseract for simple text, Azure Document Intelligence for complex layouts. Embed the extracted text, not the image. (Embodied large language models enable robots to...) — even robotics papers use text extraction before embedding.
Q: Should I use GPT-4 or Claude for generation?
A: Claude 3.5 Sonnet is cheaper and less hallucinatory for factual Q&A. GPT-4o is better at creative synthesis. We use Claude for 90% of RAG queries.
The One Thing That Actually Matters
I've seen teams obsess over the perfect embedding model, the fastest vector DB, the most clever chunking strategy.
None of that matters if you haven't measured retrieval quality against your specific data.
Set up a eval set: 100 questions with known answers. Run your pipeline. Measure recall@5, precision@5, and answer accuracy. If recall@5 is below 0.8, fix chunking. If answer accuracy is below 0.9, fix prompting.
Iterate until those numbers tick up. Then deploy.
That's how you build your own RAG pipeline. Not by following a tutorial. By measuring what matters and fixing what breaks.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.