What is RAG? Retrieval-Augmented Generation Explained

What is RAG? Retrieval-Augmented Generation Explained

Large Language Models (LLMs) like GPT-4 and Claude are incredibly powerful, but they have a fundamental limitation: they can only use knowledge from their training data. Ask an LLM about your company's internal docs, last week's meeting notes, or the latest research paper, and it will either hallucinate or admit it doesn't know.

Retrieval-Augmented Generation (RAG) solves this problem by giving LLMs access to external knowledge at inference time — no retraining required.

How RAG Works

RAG combines two components: a retriever that fetches relevant documents, and a generator (the LLM) that produces answers grounded in those documents. Here's the high-level flow:

1. User asks a question — e.g., "What was our Q4 revenue?"
2. Retriever searches a knowledge base — a vector database, search index, or document store — for relevant passages
3. Retrieved context is injected into the prompt alongside the question
4. The LLM generates an answer grounded in the retrieved context

User Query → Retriever → [Relevant Documents] → LLM Prompt → Answer

This architecture is sometimes called "open-book" reasoning — the LLM gets to consult reference material before answering, much like a student taking an open-book exam.

Why RAG Matters

RAG has become the dominant pattern for building production LLM applications because it addresses several critical limitations:

• Reduces hallucinations — The LLM can cite specific sources rather than generating plausible-sounding but incorrect information
• Keeps knowledge current — Update your document store without retraining the model
• Enables domain specificity — Ground the LLM in your proprietary data (legal docs, medical records, codebases)
• Cost-effective — Far cheaper than fine-tuning for most knowledge-grounding use cases

The RAG Pipeline: Key Components

1. Document Ingestion

First, you process your source documents into chunks and compute vector embeddings:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings

# Split documents into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
chunks = splitter.split_documents(documents)

# Compute embeddings
embeddings = OpenAIEmbeddings()
vectors = embeddings.embed_documents([c.page_content for c in chunks])

Chunk size matters. Too small and you lose context; too large and you dilute relevance. A common starting point is 300–500 tokens with some overlap.

2. Vector Storage and Retrieval

Store your embeddings in a vector database (Pinecone, Weaviate, pgvector, Chroma) for fast similarity search:

import chromadb

client = chromadb.Client()
collection = client.create_collection("my_docs")

collection.add(
    documents=[c.page_content for c in chunks],
    ids=[f"chunk_{i}" for i in range(len(chunks))]
)

# Retrieve relevant chunks
results = collection.query(
    query_texts=["What was Q4 revenue?"],
    n_results=3
)

3. Augmented Generation

Inject the retrieved context into your LLM prompt:

context = "\n\n".join(results["documents"][0])

prompt = f"""Answer the question based on the context below.

Context:
{context}

Question: What was Q4 revenue?
Answer:"""

The LLM now generates a response grounded in your actual documents rather than its training data.

RAG vs. Fine-Tuning: When to Use What

| Approach | Best For | Tradeoff | |----------|----------|----------| | RAG | Factual Q&A, document search, knowledge that changes frequently | Requires retrieval infrastructure | | Fine-tuning | Changing model behavior/style, teaching specialized formats | Expensive, static knowledge | | Both | Complex production systems needing specialized behavior + fresh knowledge | Maximum complexity |

For most teams starting out, RAG is the right default choice. It's simpler to implement, cheaper to maintain, and easier to debug.

Common Pitfalls

• Poor chunking strategy — Splitting documents at arbitrary boundaries destroys context. Use semantic or structural boundaries (paragraphs, sections) when possible.
• Ignoring retrieval quality — Your RAG system is only as good as its retriever. Measure retrieval precision and recall, not just final answer quality.
• Stuffing too much context — More retrieved documents ≠ better answers. LLMs can get confused by irrelevant passages. Start with 3–5 chunks and tune from there.

What's Next for RAG?

The RAG ecosystem is evolving rapidly. Emerging techniques include hybrid search (combining keyword and vector search), reranking (using a cross-encoder to refine retrieval results), and agentic RAG (where the LLM decides when and how to retrieve). These advances are making RAG systems more accurate and capable every month.

---

Want to go deeper? Check out our ML Basic Course →