RAG Explained: When Retrieval Actually Helps

RAG has become the default answer to "how do I make an LLM know about my data," which is mostly right, but the implementation details determine whether it actually works or just adds latency and cost to a system that still hallucinates. Here's what's actually happening under the hood and where the real design decisions are.

What RAG Actually Solves
What RAG Doesn't Solve
The Core Pipeline
Chunking Strategy Matters More Than Model Choice
Retrieval Quality: The Part Everyone Underinvests In
Hybrid Search and Reranking
When RAG Isn't the Right Tool
Common Mistakes
Performance & Cost Considerations
FAQs
Key Takeaways

What RAG Actually Solves

Two genuine problems: knowledge that postdates the model's training (your internal docs, recent events, proprietary data the model was never trained on), and grounding responses in a verifiable source so answers can cite where information came from rather than relying purely on the model's parametric memory, which is both incomplete and prone to confidently stating things that aren't true.

What RAG Doesn't Solve

RAG doesn't fix a model's reasoning ability — if the model is bad at multi-step logical reasoning, retrieving the right documents doesn't make it reason better over them. It also doesn't eliminate hallucination entirely; a model can still misread or misrepresent retrieved content, or blend retrieved facts with fabricated ones if the retrieved context is ambiguous or incomplete. RAG reduces a specific class of error (not knowing something) but doesn't address others (misinterpreting something it does "know").

The Core Pipeline

Ingestion: source documents are chunked and embedded into vectors, stored in a vector database (or a traditional database with vector search extensions).
Query time retrieval: the user's query is embedded using the same model, and the vector store returns the top-k most similar chunks.
Augmentation: retrieved chunks are inserted into the prompt as context alongside the original query.
Generation: the LLM generates a response grounded in the provided context.

def rag_query(question: str) -> str:
    query_embedding = embed(question)
    relevant_chunks = vector_store.similarity_search(query_embedding, k=5)
    context = "\n\n".join(chunk.text for chunk in relevant_chunks)

    prompt = f"""Answer the question using only the context below.
    If the context doesn't contain the answer, say so.

    Context:
    {context}

    Question: {question}"""

    return llm.invoke(prompt)

That "if the context doesn't contain the answer, say so" instruction matters more than it looks — without an explicit escape hatch, models default to answering from parametric memory when retrieval comes up empty or irrelevant, silently defeating the purpose of grounding.

Chunking Strategy Matters More Than Model Choice

The single highest-leverage decision in a RAG system is how you chunk source documents, and it's the thing most implementations get wrong by defaulting to naive fixed-size splitting. A 500-character chunk that cuts a sentence in half, or separates a heading from its content, degrades retrieval quality regardless of how good your embedding model is.

Better approaches: semantic chunking (splitting at natural boundaries — paragraphs, sections — rather than fixed character counts), hierarchical chunking (retrieving a small precise chunk, but including its parent section for context), and metadata-enriched chunks (attaching source, section title, and date to each chunk so retrieval can filter and the model has provenance to cite).

Retrieval Quality: The Part Everyone Underinvests In

Teams spend disproportionate effort on prompt engineering and model selection, and comparatively little on evaluating whether retrieval is actually returning relevant chunks. This is backwards — if retrieval returns irrelevant context, no amount of prompt engineering on the generation side fixes it. Build a retrieval evaluation set early: a set of representative queries with known-correct source documents, and measure whether your retrieval step actually surfaces them (recall@k) before worrying about the generation quality on top.

Hybrid Search and Reranking

Pure vector similarity search misses exact-match cases that keyword search handles naturally (a specific product SKU, an exact error code, a proper noun that embeddings might treat as semantically similar to unrelated terms). Hybrid search — combining vector similarity with traditional keyword/BM25 search and merging results — consistently outperforms either alone for real-world query distributions.

Adding a reranking step (a smaller, faster model that re-scores the top 20-50 candidates from initial retrieval before selecting the final top-k for the prompt) further improves precision, since the initial retrieval's job is just to cast a wide enough net, and reranking's job is precision on top of that.

When RAG Isn't the Right Tool

When the total relevant knowledge fits comfortably in the model's context window anyway. If your knowledge base is small enough to include entirely in the prompt, RAG's retrieval step adds complexity and potential retrieval-miss failure modes for no benefit over just including everything.
When the task is primarily reasoning, not knowledge lookup. RAG doesn't help a model calculate something or follow a multi-step logical chain more accurately — that's a prompting/model-capability problem, not a retrieval problem.
When real-time, structured data access is the actual need. Querying a live database for current inventory or account status is better served by tool-calling/function-calling against that database directly, not embedding and retrieving stale snapshots of the data.

Common Mistakes

Naive fixed-size chunking without regard to document structure. The highest-leverage fix available in most underperforming RAG systems.
No retrieval evaluation, only end-to-end output evaluation. Makes it impossible to tell whether a bad answer came from bad retrieval or bad generation.
Overstuffing context with too many retrieved chunks. More context isn't strictly better — irrelevant chunks can dilute the model's attention and occasionally cause it to blend irrelevant details into the answer.
Using the same embedding model version inconsistently between ingestion and query time. Embeddings from different model versions aren't guaranteed to be comparable — a silent, hard-to-debug source of degraded retrieval quality.
No update strategy for stale content. Documents change; without a re-ingestion pipeline, a RAG system confidently serves outdated information as if current.

Performance & Cost Considerations

Every RAG query involves at minimum one embedding call and one LLM generation call, often more with reranking — latency and cost compound with pipeline complexity. Caching embeddings for static content, using a smaller/cheaper model for reranking than for final generation, and setting a sensible k (retrieving 3-5 well-chosen chunks usually outperforms retrieving 20 mediocre ones, at a fraction of the token cost) are the practical levers worth tuning before reaching for more exotic optimization.

Security Considerations

If your RAG system indexes documents with different access levels (some users should see some documents, not others), retrieval must respect those permissions at query time, not just at the UI layer — a common and serious mistake is embedding all documents into a shared vector store without access-control-aware filtering, which can leak content across permission boundaries through retrieval even if the UI never displays a "browse all documents" view.

FAQs

Is RAG the same as fine-tuning? No — fine-tuning changes the model's weights to better perform a task or adopt a style; RAG leaves the model unchanged and instead supplies relevant information at query time. They solve different problems and can be combined.

How many chunks should I retrieve per query? Commonly 3-5 for focused questions, more for broader synthesis tasks — but this should be determined empirically against your specific retrieval evaluation set, not assumed from a general rule of thumb.

Do I need a dedicated vector database, or can I use a regular database with vector extensions? For smaller-scale RAG systems, Postgres with pgvector or similar extensions is often sufficient and avoids adding a new infrastructure dependency. Dedicated vector databases (Pinecone, Weaviate, Qdrant) earn their complexity at larger scale or when you need specialized indexing/filtering features they provide natively.

Why does my RAG system still hallucinate even with good retrieval? Good retrieval reduces one failure mode but doesn't eliminate the model's tendency to blend, misread, or over-generalize from retrieved context — explicit prompting to cite sources and refuse to answer when context is insufficient helps, but doesn't guarantee elimination.

Should I chunk by fixed token count or by semantic boundaries? Semantic boundaries (paragraphs, sections) generally outperform fixed-size chunking for retrieval quality, though fixed-size is simpler to implement and can be a reasonable starting point before optimizing.

Key Takeaways

RAG solves knowledge gaps and grounding, not reasoning limitations or hallucination in general — know which problem you're actually trying to solve.
Chunking strategy is the highest-leverage lever in most RAG systems, more impactful than model choice for a given use case.
Build a retrieval evaluation set separate from end-to-end evaluation — you need to know whether bad answers come from bad retrieval or bad generation.
Access control must be enforced at retrieval time for multi-permission-level document sets, not just at the UI layer.

Next Steps

Building a RAG system that needs to actually perform in production, not just in a demo? Let's talk through your specific data and query patterns.