The 'add RAG and hallucinations go away' promise was always aspirational. In practice, poorly built RAG systems hallucinate differently and sometimes worse than the underlying model would on its own.
Why naive RAG hallucinates
A common failure mode: retrieve top-k chunks by cosine similarity, stuff them into the prompt, ask the model to answer. When retrieval fails to surface the right chunk, the model does not say 'I don't know.' It confidently reasons from whatever it did retrieve. The hallucination is now grounded in an irrelevant document, which feels authoritative because it has a citation.
The fix is not a better model. The fix is a retrieval pipeline that is aware of its own uncertainty and a generation step that is instructed to refuse when the retrieved context is inadequate.
What actually works
Real production RAG systems share a set of practices. They combine dense (embedding) and sparse (BM25 or equivalent) retrieval and rerank the combined results. They chunk documents thoughtfully, preserving structure. They generate multiple query reformulations for each user question. They use a reranker — often a cross-encoder — as a second-pass filter. And they instruct the generation model explicitly to refuse when confidence is low.
- Hybrid retrieval (dense + sparse) consistently outperforms pure dense retrieval on real-world queries.
- Cross-encoder reranking is one of the highest-ROI additions to a RAG pipeline.
- Query rewriting — generating multiple reformulations — closes many precision gaps.
- An explicit refusal instruction and a low-confidence path prevent the worst hallucination failures.
Evaluation is not optional
The single most common reason production RAG systems fail is that their operators do not measure retrieval and generation as separate stages. When end-to-end accuracy drops, the operator does not know whether retrieval missed the chunk, retrieval ranked it low, the reranker filtered it out, or the model ignored it. Instrumentation that separates these stages is the difference between a system that improves over time and one that stays broken.
“A RAG system without instrumented evaluation is a very expensive way to produce plausible-sounding wrong answers.”
When to skip RAG entirely
Not every workload benefits. If your knowledge base is small enough to fit in the model's context, retrieval adds latency without accuracy gains. If your workload is open-ended reasoning without a factual knowledge base, retrieval can inject noise. And if your users are asking questions the knowledge base cannot answer, the honest response is to acknowledge the gap, not to retrieve marginally relevant chunks.
Where the field is going
The distinction between retrieval and reasoning is blurring. Agent-style systems that retrieve iteratively, planning their next query based on what they have already found, outperform single-shot RAG on complex questions. Graph-based retrieval, which follows entity relationships across a knowledge base, is gaining traction on multi-hop workloads. Expect the term 'RAG' to broaden until it encompasses most non-trivial knowledge grounding.
Key Topics
Extended Knowledge
- Hybrid retrieval combining dense and sparse methods handles both semantic and lexical queries better than either alone.
- Iterative retrieval agents outperform single-shot RAG on multi-hop and complex research questions.
- Graph-based retrieval, which traverses entity relationships, is increasingly used for structured knowledge bases.
Frequently Asked
Not on its own. A well-instrumented RAG pipeline with reranking, hybrid retrieval, and explicit refusal instructions substantially reduces certain hallucination modes.
Instrument retrieval and generation separately. Track precision, recall, and rerank quality at the retrieval stage; track faithfulness and answer quality at the generation stage.
Both. Hybrid retrieval consistently outperforms either alone on production workloads.
