3. RAG (Retrieval Augmented Generation)

RAG combines the power of retrieval (searching relevant information) with generation (LLM) to provide accurate, up-to-date, and context-aware answers — grounding the AI in your specific data.

---

Table of Contents

• What is RAG?
• Why RAG?
• The RAG Pipeline
• RAG vs. Fine-tuning vs. Prompt Stuffing
• RAG Patterns
• Chunking Strategies
• Evaluation Metrics
• Code Example
• Key Takeaway

---

What is RAG?

Retrieval Augmented Generation (RAG) is an architectural pattern that enhances LLM responses by injecting relevant, retrieved context into the prompt at query time. Instead of relying on the model's static training knowledge, RAG dynamically pulls the most relevant information from an external knowledge base.

Without RAG:  User Query → LLM → Answer (from training data, may be stale/wrong)

With RAG:     User Query → Retrieve relevant docs → LLM + Context → Accurate Answer

---

Why RAG?

Large Language Models have real limitations:

Problem	RAG Solution
Knowledge cutoff — training data is static	Retrieve fresh, real-time documents
Hallucinations — model makes up facts	Ground answers in retrieved evidence
Private data — model doesn't know your internal docs	Index your own knowledge base
Expensive fine-tuning — retraining costs millions	No retraining needed; just update the index
No citations — can't verify where answer came from	Return source documents alongside answer

---

The RAG Pipeline

┌──────────────────────────────────────────────────────────┐
│                    INDEXING (Offline)                    │
│                                                          │
│  Documents → Chunking → Embedding → Vector Store         │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│                   RETRIEVAL (Online)                     │
│                                                          │
│  User Query                                              │
│      │                                                   │
│      ▼                                                   │
│  Embed Query  ──→  Vector Search  ──→  Top-K Chunks      │
│                                                          │
│      ▼                                                   │
│  Build Prompt:                                           │
│  "Answer using this context: {chunks}\n\nQ: {query}"     │
│                                                          │
│      ▼                                                   │
│  LLM generates grounded answer with citations           │
└──────────────────────────────────────────────────────────┘

Step-by-Step

Indexing phase (run once / periodically):

1. Load — Ingest documents (PDFs, web pages, databases, APIs)
2. Chunk — Split documents into smaller passages (e.g., 512 tokens with overlap)
3. Embed — Convert each chunk into a vector using an embedding model
4. Store — Save vectors + metadata in a vector database

Retrieval phase (every query):

1. Embed query — Convert the user's question into a vector
2. Retrieve — Search the vector database for top-K semantically similar chunks
3. Re-rank (optional) — Re-rank retrieved chunks for relevance using a cross-encoder
4. Augment — Inject retrieved chunks into the LLM prompt as context
5. Generate — LLM produces an answer grounded in the retrieved context

---

RAG vs. Fine-tuning vs. Prompt Stuffing

Approach	When to Use	Pros	Cons
RAG	Dynamic, frequently changing knowledge	Fresh data, no retraining, citable	Retrieval quality critical
Fine-tuning	Stable domain knowledge, style/tone	Deep expertise, fast inference	Expensive, knowledge goes stale
Prompt Stuffing	Small, fixed context	Simple, no infrastructure	Context window limits, expensive
RAG + Fine-tuning	Best of both worlds	Most accurate	Most complex

---

RAG Patterns

RAG has evolved through three generations. Each sub-page covers one pattern in depth:

Pattern	Description	Sub-page
Naive RAG	Simple retrieve-and-generate. Fixed pipeline, good for prototyping.	(covered above)
Advanced RAG	Smarter queries, hybrid search, re-ranking, and contextual compression.	Advanced RAG →
Modular RAG	Fully composable pipeline with swappable, independent modules.	Modular RAG →
Agentic RAG	An AI agent controls retrieval — deciding when, where, and how many times to retrieve.	Agentic RAG →

---

Chunking Strategies

How you split documents dramatically affects retrieval quality:

Strategy	Description	Best For
Fixed-size	Split every N tokens	Quick setup, general use
Sentence splitting	Split at sentence boundaries	Conversational text
Recursive character	Tries paragraph → sentence → word	Most documents (LangChain default)
Semantic chunking	Split when topic changes (embedding-based)	Highest quality, slower
Document structure	Split by heading / section	Structured docs (Markdown, HTML)

Overlap: Always add a small overlap (e.g., 50–100 tokens) between chunks to preserve context across boundaries.

---

Evaluation Metrics

Metric	Measures	Tool
Faithfulness	Does the answer stick to retrieved context?	RAGAS, TruLens
Answer Relevancy	Is the answer relevant to the question?	RAGAS
Context Precision	Are retrieved chunks actually useful?	RAGAS
Context Recall	Did we retrieve all needed information?	RAGAS
Latency	End-to-end response time	Custom monitoring

---

Code Example

A minimal RAG system using LangChain + Chroma + OpenAI:

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain_community.document_loaders import TextLoader

# 1. Load documents
loader = TextLoader("company_policy.txt")
documents = loader.load()

# 2. Chunk documents
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = splitter.split_documents(documents)

# 3. Embed and store in vector database
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(chunks, embeddings)

# 4. Create retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# 5. Build RAG chain
llm = ChatOpenAI(model="gpt-4o", temperature=0)
rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)

# 6. Query
result = rag_chain.invoke({"query": "What is our refund policy?"})
print(result["result"])
print("Sources:", [doc.metadata for doc in result["source_documents"]])

---

Key Takeaway

RAG enhances LLMs by grounding their answers in your own data — delivering accurate, relevant, and trustworthy responses with citations, while avoiding hallucinations and knowledge staleness.

---

← Embedding Model | Back to Overview | Prompt Engineering →