Quick summary:
A RAG pipeline that returns wrong answers without errors has a silent failure in retrieval, augmentation, or generation. To isolate it, record a structured trace per request, measure retrieval separately from generation with recall@k against a gold-chunk eval set, and gate retrieval changes in CI with a per-question diff. Seven failure modes look identical in the output, and five commonly missing trace fields turn each investigation into four lookups. The guide includes a Python trace schema, a quick-isolation checklist, and an original chunking benchmark.
If answers from your RAG pipeline got worse after a chunk size change, an embedding model swap, and a reranker nobody is sure is live, the problem almost certainly sits in retrieval. ScriptsHub Technologies has seen these symptoms repeatedly while instrumenting retrieval augmented generation (RAG) systems. Instrumentation turns days of prompt experiments into four database lookups.
Why does a RAG pipeline fail silently without raising an error?
A RAG pipeline is a chain of lossy stages, and no stage raises an exception when it discards the wrong thing. Every component reports success, so the only visible artifact is a fluent answer.
The query is rewritten, documents are retrieved and reranked, chunks are assembled under a token budget, and a model generates from whatever survived. A query embedded with a different model than the index, even one with the same vector dimension, returns plausible but unrelated chunks without an error.
Figure 1. Each stage of a RAG pipeline discards information silently. The final output carries no signal about what was lost.
Worse, a model sounds equally confident with perfect context or none, so outputs cannot separate retrieval from generation failures. Teams change the prompt, usually the one component that was working.
What are the seven RAG failure modes that look identical?
A wrong RAG answer has at least seven distinct causes, called failure modes. They look identical in the output, and each is detectable only if the trace recorded the right field.
Our taxonomy builds on the seven failure points in RAG systems Barnett et al. documented across three deployments, grouped by the three stages of any RAG architecture: retrieval, augmentation, and generation. At retrieval, a retrieval miss (mode 1) means the answering document was never returned, pointing at embeddings, index freshness, or query phrasing. Queries built on product codes, IDs, or exact names often need hybrid search that pairs vectors with BM25, while deeply hierarchical corpora may suit a vectorless RAG architecture. Ranked too low (mode 2) means it was retrieved but sat below the cutoff. A chunk boundary split (mode 3) means the answer spans two chunks and neither scores well alone, usually from small chunks or no overlap; oversized chunks instead dilute relevance.
During augmentation, truncated by budget (mode 4) means the gold chunk ranked well, then was dropped because earlier chunks consumed the token budget. Buried in the middle (mode 5) means it reached the prompt in a position models use poorly, the U-shaped lost-in-the-middle effect. In Liu et al.’s tests with 20 documents, GPT-3.5-Turbo scored 75.8% with the answer first but 53.8% mid-context, below its 56.1% with no documents. At generation, parametric override (mode 6) means the model answered from training data despite correct context. Unanswerable and confabulated (mode 7) means the corpus lacked the answer and the model invented one when it should have refused.
Figure 2. Seven RAG failure modes, seven unrelated fixes.
Figure 3. Mode 5 in isolation (illustrative): identical retrieval, identical inclusion, different position. Liu et al. also found 50 retrieved documents beat 20 by only about 1.5% for GPT-3.5-Turbo.
What should a RAG pipeline trace record?
A RAG trace should be one structured record per request covering the query as searched, index provenance, the fate of every retrieved chunk, context assembly, and stage timings. Unlike free-text logs, a trace can be queried in bulk and joined to an eval set.
Why this works:Each failure mode maps to a field. Rank fields and candidates_fetched separate modes 1 and 2, excluded_reason exposes mode 4, context_position exposes mode 5, and index_version ties regressions to rebuilds. Dataclasses need Python 3.10+ for this syntax and do not enforce types at runtime, so validate on write.
The five fields most teams omit
With HyDE query transformation, the text that gets embedded differs from what the user typed, so debug the transformed one. Also store the fully rendered prompt, since the template version alone hides a context variable that rendered empty.
Why should you measure retrieval separately from generation?
Measure retrieval separately because recall@k is a hard ceiling on answer accuracy. For questions only your corpus can answer, if the gold chunk reaches the model 60% of the time, no prompt, model upgrade, or temperature change lifts accuracy above 60%.
End-to-end RAG pipeline quality mixes whether the right information reached the model with what the model did with it.
Figure 4. Illustrative: five rounds of generation work against a retrieval ceiling that none of them can move.
ompute recall@k first, using standard ranked-retrieval evaluation. High recall with wrong answers justifies prompt work; low recall means stop touching the prompt.
Label the gold chunk, not the gold answer
Grading prose answers usually needs an LLM judge, a second model to debug. Instead, record which chunk IDs contain each answer. Retrieval evaluation becomes a set membership test that is objective, fast, free, and immune to judge drift.
Figure 5. One approach requires a judge and a rubric. The other is a set membership test.
Why this works: Gold labels make recall@k a pure function of the trace, counting only the final top k. Resolve labels per index build, since chunk IDs change with chunking. Exclude unanswerable questions from recall averages and judge them on refusal accuracy.
Make roughly 15–20% of the set unanswerable to catch a pipeline that never refuses. Add boundary-spanning questions as the early warning for mode 3, and questions where your corpus contradicts common knowledge, the only way to detect mode 6. In our experience, one engineer who knows the corpus can label a starter set in an afternoon.
If your RAG pipeline got worse and nobody can say why, our team instruments it and wires regression gates into CI. Explore our AI consulting services for RAG evaluation.
Which metrics show where a RAG pipeline is breaking?
Track five RAG metrics separately: recall@k, mean gold position, context precision, faithfulness, and refusal accuracy. A blended score hides which stage failed.
Mean gold position is the cheapest early warning: a slide from rank 2 to rank 8 changes no boolean at k = 10 but warns of the next miss. Microsoft’s guide to evaluating RAG performance uses the same split, checking the retriever before the generator.
How do you catch retrieval regressions in CI?
Treat every retrieval setting in your RAG architecture as code, and block merges that lower recall. Gate on the mean and a per-question diff, since offsetting swaps hide behind a flat mean.
Why this works:Both gates compare only questions present in the baseline, so adding questions never fakes a regression.
What one chunking change did in our benchmark
We ran a reproducible benchmark: 300 keyword queries over docstrings from 48 Python standard-library modules, TF-IDF retrieval, k = 5, with gold answers stored as character spans (script and data: [[FILL-REPO-URL]]). Adding 20-word overlap to 80-word chunks lifted recall@5 from 80.3% to 87.0%, so the aggregate gate passes. Yet 18 questions regressed, and only the per-question gate catches them.
Table: Results come from one corpus and a TF-IDF retriever. Recall and regression counts will differ on your data, which is why the gate runs on your own eval set.
How do you debug a RAG pipeline in four lookups?
To debug a bad answer, pull its trace by ID and ask four questions in order. Each answer eliminates failure modes, and none requires changing the system.
Figure 6. Four lookups to debug a RAG pipeline, using fields from the trace schema above.
Is the gold chunk in the trace? If not, it is mode 1, or mode 3 when the answer spans chunks. Is its rank inside k? If not, mode 2. An excluded_reason of “budget” is mode 4, and a context_position of nine out of eighteen is mode 5. Only then edit the prompt. To confirm mode 6, replay the rendered prompt at temperature 0 against a stronger model. If it answers correctly, the smaller model is the limit, a common trade-off with quantized and distilled LLMs. If not, tighten the grounding instruction and wrap each chunk in XML tags.
Quick-isolation checklist: Match the symptom to the stage and fix.
What to settle before you ship the trace
Tracing stays cheap when you store chunk IDs rather than chunk text: our 50-candidate test trace serialized to about 13 KB of JSON, or 1.4 KB gzipped. Write traces asynchronously so logging stays off the request path, and keep every failure plus a sample of good answers, since metrics need a baseline. Emit the trace from your orchestration code, as with AI agent observability in production. Pin the index version so regressions map to a rebuild, part of the lineage discipline in AI data governance for agents. Decide PII retention before you build: traces store whatever users type, so set a retention window, redact at write time, and follow the UK GDPR storage limitation principle.
What does tracing a RAG pipeline actually buy you?
Tracing buys diagnosis. The trace, eval set, and CI gate do not improve recall on their own, but together they tell you which of seven problems you have, which is most of the difficulty.
To instrument an existing RAG pipeline, start small: build the trace, label a few dozen questions by gold chunk, and compute recall@k before touching anything else. In our experience, teams usually find their ceiling is lower than assumed. Stale caches can hide failures too, as covered in cache invalidation bugs that AI debuggers miss; LLM data reconciliation catches the silent drift that follows.
Want a second opinion on your retrieval pipeline? ScriptsHub Technologies builds data and AI pipelines for teams across the US, UK, and India. For RAG evaluation or retrieval observability, talk to our AI engineering team.
Frequently asked questions
Q. What does a RAG pipeline do?
A RAG pipeline grounds a language model in your own data. It retrieves relevant chunks, adds them to the prompt as context, and answers from them. The pattern is called retrieval augmented generation (RAG).
Q. What are the five key components of a RAG pipeline?
The five key components are document ingestion and chunking, an embedding model, a vector index, a retriever with an optional reranker, and a language model that generates from the augmented prompt. Trace each separately.
Q. How do you debug step by step?
Use the trace. Reproduce the failure, then check four things in order: whether the gold chunk was retrieved, its rank, whether it survived context assembly, and where it landed. Change the prompt last.
Q. What are the six stages of debugging?
A common six-stage model is: reproduce the problem, isolate it, find the root cause, fix it, verify the fix, and document it. In RAG work, a structured trace turns isolation into a quick lookup.
Q. What are the two types of debugging?
The two types most guides name are reactive debugging, which fixes bugs after they surface, and proactive debugging, which builds in early checks. For RAG, trace lookups are reactive and a CI recall gate is proactive.
About ScriptsHub Technologies: ScriptsHub Technologies is a data engineering and applied AI consultancy with teams in the US, UK, and India. We build data pipelines, business intelligence, and production AI systems, including RAG AI solutions, retrieval evaluation, and LLM observability. Learn more at scriptshub.net.
