Why does my RAG system give wrong or vague answers?

Almost always because of retrieval, not the model. If the passage that actually answers the question was not among the chunks retrieved, the model has nothing to work from, so it either guesses or says it does not know. Before blaming the model, check what context was retrieved for the failing question. Nine times out of ten the answer was not in it.

How do you improve RAG retrieval?

The highest-leverage moves are: evaluate retrieval on its own with real questions and known-correct sources; use hybrid search (keyword plus semantic) so exact terms like product codes and names are not missed; add a reranking step to reorder results by true relevance; chunk on document structure with some overlap; rewrite vague user queries into good search queries; and filter by metadata such as recency and permissions. Fixing retrieval fixes most RAG problems.

RAG or fine-tuning, which do I need?

They solve different problems. RAG is for knowledge that changes, is large, or must be cited and controlled: your documents, policies and data. Fine-tuning is for behaviour, tone and format: teaching the model how to respond, not what facts to know. Most production systems need RAG, and some also fine-tune for style. Reach for fine-tuning to change behaviour, not to inject knowledge that a retrieval step would handle better.

Does a bigger context window remove the need for RAG?

No. Even with a very large context window you cannot paste your entire knowledge base into every prompt: it is slow, expensive, and models lose accuracy on facts buried in the middle of a huge context. Retrieval still earns its place by selecting the few passages that matter, keeping prompts small, cheap and grounded. Bigger windows help; they do not replace retrieval at scale.

RAG in production: why retrieval is where it breaks

Q: What is retrieval-augmented generation (RAG)?

RAG is a way to give a language model your own data at answer time. Instead of relying on what the model memorised in training, you retrieve the most relevant chunks of your documents and put them into the prompt, and the model answers from those. It is how you get an assistant that can answer from your internal documents, policies and records rather than guessing.

RAG (retrieval-augmented generation) gives a language model your data at answer time: you retrieve the most relevant chunks of your documents and put them into the prompt, and the model answers from those rather than from what it half-remembers. It is what turns a clever chatbot into an assistant that can answer from your actual policies, records and knowledge base.

Here is the thing every team learns the hard way: when a RAG system gives wrong or vague answers in production, the model is almost never the problem. It is retrieval. The system fetched the wrong context, too little of it, or none at all, and the model did the best it could with nothing. Fix retrieval and most RAG problems simply disappear.

The model only sees what retrieval hands it

A RAG pipeline has two halves. Retrieval: take the user’s question, find the most relevant chunks of your data, and put them in the prompt. Generation: the model writes an answer from those chunks. The generation half is genuinely good now. The retrieval half is where systems fall over.

The reason is simple. The model can only answer from what it is given. If the passage that actually answers the question is not in the retrieved context, the model has two options, and both are bad: guess from its general training (a confident, wrong answer), or admit it does not know (a useless one). Either way the failure happened upstream, in retrieval, not in the model.

So the first debugging move for any bad RAG answer is not “which model is this” or “let’s tune the prompt.” It is: look at what was retrieved for that question. Almost every time, the answer was not in it.

Where retrieval actually breaks

The failures cluster in a handful of predictable places.

Chunking. Documents get split into chunks before indexing. Too large and relevance gets diluted and prompts blow up; too small and a chunk loses the meaning that made it useful. Splitting mid-sentence, or mid-table, destroys the information. Chunk on natural structure (headings, sections) with a little overlap, rather than blindly every N characters.
The semantic gap. Vector search matches meaning, which is powerful, but it misses exact terms the user typed: a product code, a person’s name, an acronym, an error string. Pure semantic search will happily skip the one document that literally contains “ERR-4021.”
Top-k, too few or too many. Retrieve too few chunks and you miss the answer; too many and you bury it in noise and pay for the tokens. The right number is a tuning decision, not a default.
The query is a bad search query. Users ask vague, multi-part questions. Feeding that raw string to the retriever often returns nothing useful. The question needs rewriting into something searchable first.
Stale or missing data. The cleanest retrieval algorithm cannot find a document that was never ingested, or return the current answer from a six-month-old index. Coverage and freshness matter as much as the maths.
No metadata filtering. Without filtering by recency, source, tenant or permissions, retrieval returns out-of-date, irrelevant, or, worse, data the user should never see. In a multi-customer system this is a security issue, not just a quality one.
No grounding check. When retrieval comes back empty, a system that lets the model answer anyway from its training will hallucinate. It should notice it has no support and say so.

How to make retrieval reliable

The fixes are well understood. In rough order of leverage:

Evaluate retrieval on its own. This is the single highest-value practice and the one most teams skip. Build a set of real questions with the known correct source passage for each, and measure whether retrieval actually surfaces it. If it does not, no amount of prompt or model work will save the answer. This is the same discipline as building an evaluation harness before you ship, pointed at retrieval specifically.
Use hybrid search. Combine keyword search with semantic search so exact terms and meaning are both covered. This alone fixes a surprising share of “it couldn’t find the obvious document” failures.
Rerank. Retrieve a generous set of candidates, then use a reranking model to reorder them by true relevance and keep the best few. Cheap, and it lifts answer quality noticeably.
Chunk on structure, with overlap, and attach metadata to every chunk so you can filter by source, date and permissions.
Rewrite the query before retrieving, so a vague question becomes a good search. In an agentic setup, let the agent decide what to search for.
Ground and cite. Require the answer to come from the retrieved passages, show the citations, and refuse (or escalate) when nothing relevant was found. A “I don’t have that” is worth far more than a confident fabrication.

RAG, fine-tuning, and big context windows

Three things get confused. RAG is for knowledge: data that changes, is large, or must be cited and controlled. Fine-tuning is for behaviour: tone, format, how the model responds, not what facts it knows. A bigger context window helps, but you still cannot paste an entire knowledge base into every prompt, and models lose accuracy on facts buried deep in a huge context, so retrieval still earns its place by selecting the few passages that matter. Use RAG for the knowledge, fine-tune for behaviour if you need to, and let a large window make retrieval’s job easier rather than replace it.

Where this fits

RAG rarely stands alone. It is usually how an AI agent gets grounded context to work with, and the connection to your data often runs through the Model Context Protocol. In production it needs the same rigour as any AI system: retrieval evaluation, monitoring, and, because it is answering from your real data, the grounding and citations that let you trust and audit what it said. Getting AI to answer from your systems safely is exactly the footing we build on.

The short version

RAG puts your data in front of the model, and the model is only as good as what retrieval hands it. When answers go wrong in production, look at the retrieved context first: the fix is almost always better retrieval, not a better model. Chunk on structure, search with keywords and meaning together, rerank, filter, rewrite the query, and above all evaluate retrieval on its own. Do that and RAG goes from a flaky demo to something you can put in front of customers.

We build AI agents and custom software on retrieval that actually works, grounded in your systems with the evidence to trust it. If you are trying to get an AI assistant to answer reliably from your own data, talk to us.