BIPI
BIPI

RAG Is Overused: When Retrieval-Augmented Generation Makes Things Worse

Agentic AI

RAG has become the default reach for any 'let an LLM see our content' problem. It's also wildly over-applied. The three failure modes we keep finding, and the question that usually clarifies things.

By Arjun Raghavan, Security & Systems Lead, BIPI · December 20, 2025 · 7 min read

#rag#llm#architecture#vector-database

A potential client recently described their problem this way. 'We have 4,000 internal docs. We want our team to ask questions about them in natural language.' The room agreed RAG was the answer. The room was wrong, in the sense that it would have been an expensive disappointment.

Retrieval-augmented generation has become the default reach for any 'let an LLM see our content' problem. Most architects we talk to could draw the standard diagram in their sleep. Chunk the documents, embed each chunk, store the embeddings in a vector database, retrieve the top-k matches at query time, stuff them into a prompt, ask the model. It is a useful pattern. It is also wildly over-applied.

When RAG actually works

RAG is the right tool for a narrow question shape. The shape is 'Tell me what this corpus says about X.' The corpus is stable, the answer is contained in a small chunk, and the user is happy with a quote-and-summarize response. Customer support over a product manual. Internal policy lookups. FAQ assistants over a Confluence space.

In those cases, RAG works because it gives the model exactly what it needs. The model is not asked to reason across documents, only to pull and summarize.

Three failure modes

We have done enough RAG audits to recognize the same three failures in most.

Multi-hop questions. 'Which engineering teams are touching the payments stack and what is their on-call rotation?' That requires joining a 'team to system' document with a 'system to on-call' document. Vector retrieval is not a join. It will pull two unrelated chunks and the model will hallucinate the connection between them.

Aggregation queries. 'How many open critical findings do we have in the last quarter?' Embeddings give you semantic similarity, not counts. The model will produce a confident-sounding number that does not match the source. We have seen this fail catastrophically inside compliance tooling.

Dynamic data. 'What is the current build status?' RAG over a static index of documents has no knowledge of your CI pipeline. The right answer is a tool call to the build system, not retrieval against yesterday's snapshot.

When you should reach for something else

  • Reasoning across multiple sources: an agent with tool calls.
  • Live or transactional data: function calling tied to APIs.
  • Structured analytics or counts: SQL generation against a real database.
  • Highly stable narrow domain: RAG, or a small fine-tune.
  • Domain-specific tone or style: fine-tune the model.
  • Need verbatim quotes: RAG.

We end up shipping hybrid systems on most engagements. RAG handles 'what does the policy say about X' while a tool layer handles 'what is the current state of Y.' The split is usually obvious in retrospect and almost never obvious in the kickoff meeting.

Common mistakes inside the RAG itself

Even when RAG is the right pattern, three implementation mistakes show up repeatedly.

  1. Chunk size copied from a tutorial. The right chunk size depends on the document type. A regulatory document chunked at 512 tokens loses cross-references. A meeting transcript chunked at 2,000 tokens dilutes the signal. Pick chunk sizes per source.
  2. No metadata filtering. The vector search is doing all the work. Add metadata (date, author, document type) and filter before similarity scoring. Retrieval quality jumps without tuning embeddings.
  3. Single retrieval pass for complex queries. Run a second retrieval after the model produces a tentative answer to verify or expand. This is the 'self-RAG' pattern and it cuts hallucinations meaningfully.

Closing

RAG is a hammer. Most teams have a small box of nails and a workshop full of other things. Pick the pattern that fits the question shape, not the architecture you saw at last quarter's conference.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.