what does rag mean in llm? A Practitioner’s Guide to Retrieval-Augmented Generation
I spent six months in 2023 building a customer support bot for a logistics company. We fine-tuned a Llama 2 13B model on their ticket data. Results were okay — but the bot kept hallucinating shipment tracking numbers. It would invent tracking IDs that looked real but didn’t exist.
That’s when I learned what RAG actually means. Not from a paper. From a production outage at 2 AM.
Here’s the short answer: RAG (Retrieval-Augmented Generation) is a technique where you give an LLM external data at inference time — relevant documents, database records, API results — so it can answer based on facts, not just its training memory.
Think of it this way: a vanilla LLM is a student who memorized the textbook but forgot half of it. RAG is that same student with the textbook open on the desk.
This guide covers what RAG is, why it matters, how to build it, and what goes wrong. I’ll skip the theory you can read on Wikipedia. I’ll focus on what I’ve seen work — and fail — in production.
The Core Problem: Why LLMs Lie
LLMs don’t “know” anything. They predict tokens. When you ask “What’s the capital of France?”, the model has seen “Paris” so many times in training that the prediction is easy. Ask “What were quarterly earnings for Acme Corp in Q3 2025?” and the model either:
- Has no training data after its cutoff date
- Saw something vaguely related and makes stuff up
- Generalizes from similar company names and invents a number
This is hallucination. And it’s not a bug — it’s a feature of how these models work. They’re not databases. They’re text generators with a statistical memory of everything they’ve seen.
RAG fixes this. Instead of asking the model to remember facts, you retrieve the facts from a trusted source and feed them into the prompt. The model’s job becomes formatting and reasoning — not memorization.
What RAG Actually Looks Like Under the Hood
Let me show you the simplest possible RAG pipeline. No vector databases, no embeddings yet. Pure Python.
python
# Minimal RAG: no vector DB, just string matching
def simple_rag(query, document_store):
# Step 1: Retrieve - find relevant documents by keyword
relevant_docs = [doc for doc in document_store
if any(word in doc for word in query.lower().split())]
# Step 2: Augment - build a prompt with context
context = "
".join(relevant_docs[:3])
prompt = f"""Answer the question based ONLY on the context below.
Context:
{context}
Question: {query}
Answer:"""
# Step 3: Generate - call the LLM
# (pseudo-code for your actual LLM call)
response = call_llm(prompt)
return response
This works for small document sets. For real systems, you need embeddings and vector search.
Here’s the actual pattern I use at SIVARO for production RAG:
python
# Production RAG with embeddings
from sentence_transformers import SentenceTransformer
import chromadb
class RAGPipeline:
def __init__(self, embedding_model="all-MiniLM-L6-v2"):
self.encoder = SentenceTransformer(embedding_model)
self.chroma_client = chromadb.Client()
self.collection = self.chroma_client.create_collection(
name="documents",
embedding_function=self.encoder.encode
)
def index_documents(self, docs, ids):
embeds = self.encoder.encode(docs)
self.collection.add(
embeddings=embeds.tolist(),
documents=docs,
ids=ids
)
def query(self, question, n_results=3):
query_embed = self.encoder.encode([question])
results = self.collection.query(
query_embeddings=query_embed.tolist(),
n_results=n_results
)
context = "
".join(results['documents'][0])
prompt = f"""Context:
{context}
Question:
{question}
Answer:"""
return call_llm(prompt)
The key insight: you’re not searching the documents with keywords. You’re searching their semantic meaning. A user asks “how do I reset my password?” and the system retrieves the doc titled “Account recovery steps for forgotten credentials” even though they used different words.
Why RAG Won Over Fine-Tuning (For Most Use Cases)
In 2024, a lot of teams jumped to fine-tuning. “Let’s train a model on our data!” sounded sexy. But here’s what I’ve seen:
Fine-tuning works when you need the model to learn a behavior. Like a tone of voice, or a classification task, or a specific output format. It fails when you need the model to learn facts.
Why? Because facts change. Your product documentation updates every week. Your pricing changes quarterly. A fine-tuned model is a snapshot in time. You retrain for every change.
RAG doesn’t have this problem. You update the document store, not the model. The LLM stays the same. Your knowledge base stays current.
I did an informal test at SIVARO in early 2024. We had 50 technical support documents. We fine-tuned a Mistral 7B on them (cost: ~$400 in compute) and built a RAG pipeline with the same documents (cost: ~$20 in embedding compute). We asked 100 questions about the docs.
- Fine-tuned model: 74% correct answers, but hallucinated confidently when wrong
- RAG pipeline: 91% correct, and when it didn’t find an answer, it said “I don’t know”
The “I don’t know” part matters more than the accuracy gap. A system that admits ignorance is safe. A system that lies confidently is dangerous.
The Three Moving Parts That Make or Break RAG
Most people focus on the LLM. They should focus on retrieval. Here’s what actually matters:
1. Chunking Strategy
How you split your documents matters more than which embedding model you use. I wasted two months on this.
Chunk too small (under 100 tokens) and you lose context. The user asks about “refund policy” but the chunk only contains “refunds are processed within 5-7 business days” — missing the exception for international orders.
Chunk too large (over 1000 tokens) and you dilute relevance. The user asks about “return window” and gets a 2000-token page about returns, shipping, and product care. The LLM gets distracted.
The sweet spot I’ve found: 256-512 tokens with 50-token overlap. Split on paragraph boundaries, not sentence boundaries.
python
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
separators=["
", "
", ". ", " ", ""]
)
chunks = splitter.split_text(document)
2. Embedding Model Choice
You don’t need OpenAI’s embedding model. For most use cases, open-source models work as well or better.
At SIVARO, we tested text-embedding-ada-002 (OpenAI) against all-MiniLM-L6-v2 (open-source, 384 dimensions) on a dataset of 10,000 technical support articles. Accuracy was within 2% on retrieval quality. Cost difference? OpenAI charges $0.13 per million tokens. Self-hosting the open-source model costs about server time.
If you have proprietary data you don’t want to send to an API, use open-source embeddings. We run BAAI/bge-small-en-v1.5 in-house. It’s 384 dimensions, fast, and works well for English.
3. Retrieval Pipeline Design
Single-stage retrieval (one query, one search) is a baseline. Real systems need more.
Hyde (Hypothetical Document Embeddings): Generate a hypothetical answer first, then use that answer as the search query. Sounds weird. Works.
python
def hyde_rewrite(question):
# Ask LLM to generate a hypothetical answer
prompt = f"""Write a detailed paragraph answering this question.
Don't use placeholders. Write as if you're answering from a knowledge base.
Question: {question}"""
hypothetical = call_llm(prompt)
return hypothetical # Use this for embedding search, not the original question
I ran an A/B test on our internal Q&A system. Hyde improved top-3 retrieval accuracy from 83% to 91%. The gain comes from the fact that the hypothetical answer shares more vocabulary with the target document than the original question does.
What Everyone Gets Wrong: Evaluation
Most RAG systems launch without proper eval. Teams deploy, see 80% accuracy on their 20 test questions, and call it done. Then users complain.
You need three separate metrics:
- Retrieval precision: Of the documents returned, how many are actually relevant?
- Retrieval recall: How many relevant documents exist in the store that you didn’t return?
- Generation accuracy: Given perfect retrieval, does the LLM answer correctly?
At SIVARO, we use this eval pipeline:
python
from datasets import Dataset
import numpy as np
def evaluate_rag(rag_pipeline, test_questions, expected_answers, relevant_doc_ids):
results = []
for q, expected, doc_ids in zip(test_questions, expected_answers, relevant_doc_ids):
retrieved_docs = rag_pipeline.retrieve(q, top_k=5)
retrieved_ids = [d['id'] for d in retrieved_docs]
# Retrieval metrics
hits = len(set(retrieved_ids) & set(doc_ids))
precision = hits / len(retrieved_ids)
recall = hits / len(doc_ids)
# Generation metric (simplified)
answer = rag_pipeline.generate(q, retrieved_docs)
answer_correct = evaluate_answer(answer, expected) # Custom eval function
results.append({
'precision': precision,
'recall': recall,
'answer_correct': answer_correct
})
return {
'avg_precision': np.mean([r['precision'] for r in results]),
'avg_recall': np.mean([r['recall'] for r in results]),
'generation_accuracy': np.mean([r['answer_correct'] for r in results])
}
The generation accuracy is the real test. If you retrieve perfectly but the LLM ignores the context and makes stuff up, your RAG system is broken. I’ve seen this happen with smaller models (under 7B parameters) — they don’t have the capacity to follow instruction templates reliably.
The Dirty Secret: Context Window Size Doesn’t Matter As Much As You Think
In 2024, everyone lost their minds about Gemini 1.5’s 1 million token context and GPT-4 Turbo’s 128K. The thinking was “just throw all your documents into the prompt!”
This doesn’t work. I tested it.
I put 50,000 tokens of technical documentation into a single prompt with GPT-4. Results were worse than RAG with 3 retrieved documents. The model got lost in the noise. It paid more attention to documents near the beginning and end of the prompt (primacy and recency effects) and missed relevant information in the middle.
The Lost in the Middle paper (Liu et al., 2023) confirmed this experimentally. Performance drops sharply when relevant information is in the middle of the context.
RAG solves this by limiting context to the most relevant documents. You’re not fighting the model’s attention — you’re curating its input. This is why RAG systems with 3-5 retrieved chunks often outperform systems with 20 chunks, even with models that can handle 32K tokens.
When RAG Fails (And It Will)
Three failure modes I’ve seen repeatedly:
1. The Embedding Gap
Your embedding model uses English. Your documents are mostly English. But your users ask questions in a different register — slang, abbreviations, industry jargon. The semantic similarity breaks down.
Fix: Augment your query. We built a query expansion layer that takes the user’s question and generates 3 alternative phrasings using an LLM, then retrieves for all 3.
2. The Stale Index Problem
Documents get updated. Your vector store has old embeddings. User asks about current pricing, gets last quarter’s pricing, model answers with last quarter’s pricing.
Fix: Version your embeddings with timestamps. Track document change history. Re-index on updates, not on a schedule.
3. The Context Overwriting Problem
The LLM should use the retrieved context. But if the question contradicts the context, some models still trust their training data. Example: user asks “What’s your return policy?” Context says “30-day return window.” But the model was trained on Amazon’s 30-day policy plus Zappos’ 365-day policy, and it outputs “365 days.”
Fix: Stronger system prompts. And test specifically for this — we have a test suite with deliberate contradictions to see if the model trusts context over its training.
RAG Isn’t the Endgame. It’s the Baseline.
Most people think RAG is a solution. It’s not. It’s a technique that solves one problem (hallucination from lack of data) while creating others (retrieval quality, latency, cost).
The actual goal is grounded generation — models that understand when to use retrieved data and when to rely on training, that can cite sources, that say “I don’t know” instead of making things up.
At SIVARO, we’re moving toward agentic RAG — systems where the LLM can decide to search, re-phrase queries, look up multiple sources, and synthesize before answering. It’s RAG plus tool use plus planning.
But that’s a topic for another article.
For now, ask yourself: does your system need to know facts that change, or does it need to learn behaviors? If the answer is facts, start with RAG. Skip the fine-tuning hype. Build the retrieval pipeline first, get it working, then optimize.
The model doesn’t need to know your data. It just needs to know where to find it.
FAQ: What Does RAG Mean in LLM?
What exactly does RAG stand for?
Retrieval-Augmented Generation. It’s a framework where a retrieval system (like a search engine) finds relevant documents, and then a generative LLM uses those documents to produce an answer. The “augmentation” is adding those documents to the LLM’s input.
Do I need a vector database for RAG?
No. Vector databases (Pinecone, Chroma, Qdrant) make it easier, but you can use simple keyword search or a custom index. For small document sets (under 1,000), inverted index search works fine. For enterprise scale, use a vector DB. We use Chroma at SIVARO for internal tools, Pinecone for client systems.
Does RAG work with smaller models?
Yes, but not equally. Models under 7B parameters have trouble following instruction templates with context. They might ignore the retrieved documents or fail to cite them. For production RAG, I recommend at least Mistral 7B or Llama 3 8B. GPT-3.5 and GPT-4 work best.
How much does RAG cost per query?
Depends on retrieval and generation. Retrieval with open-source embeddings costs near zero (millicents per query). Generation costs depend on the model — GPT-4 Turbo is about $0.03 per request with a 2K token output. Self-hosting a 7B model on a GPU costs about $0.001 per query. Total cost for a typical RAG query: $0.005-$0.05.
Can RAG handle images and PDFs?
Yes, but you need to extract text from PDFs first (use PyMuPDF or OCR tools) and encode images separately. Multi-modal RAG is an active research area. For now, convert everything to text before embedding.
Is RAG better than fine-tuning?
Trade-offs. RAG is better for facts, frequent updates, and recall. Fine-tuning is better for style, tone, and repetitive tasks. Most systems should use both — fine-tune for behavior, use RAG for knowledge.
What’s the biggest mistake people make with RAG?
Ignoring retrieval quality. Teams spend weeks selecting an LLM and minutes thinking about chunking strategy. Bad retrieval means bad answers, no matter how good the LLM is. Measure retrieval precision and recall before you even look at generation accuracy.
Does RAG reduce hallucinations completely?
No. It reduces them significantly — from about 15-20% with vanilla LLMs to 2-5% with RAG, in my testing. But the LLM can still ignore the context or misinterpret it. Combined with strong prompting and validation, you can get below 1%. But never zero.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.