The Moving Target: Why Evaluating RAG Is AI’s Quietest Hard Problem

Retrieval-augmented generation made enterprise AI possible. Measuring whether it actually works is the problem we still haven’t solved. 

The demo always works 

Every retrieval-augmented generation (RAG) system looks brilliant in its first demo. You ask a question, it pulls the right document, and it answers with a citation. Building that first version is genuinely easy — dozens of tutorials take you from zero to a working prototype in an afternoon. The hard part begins the moment real users, real data, and real scale arrive. 

Then the cracks show. The same question returns a slightly different answer on Tuesday than it did on Monday, users occasionally complain that an answer is wrong, and no one can say which half of the system — retrieval or generation — actually broke. That gap between “seems fine” and “is reliable” is where most enterprise RAG projects quietly stall. 

The bottleneck isn’t the model — it’s the measurement 

Most of the industry’s attention goes to better foundation models and better retrievers. But the capability that decides whether a RAG system can be trusted in regulated, high-stakes work is something less glamorous: evaluation. After spending the last several years building and operating large-scale retrieval and grounding systems, I have become convinced that evaluation — not retrieval — is the real bottleneck. 

And evaluation is uniquely hard for one reason: it is a moving target. The system under test is non-deterministic, the judge we use to score it is non-deterministic, and the ground truth it is measured against keeps changing underneath both. Solve any one of those and the other two still move. 

Two systems, four ways to fail 

A RAG pipeline is really two systems stitched together: a retriever that finds context and a generator that writes the answer. Each fails in its own way. The retriever can pull the wrong chunks, miss the only passage that mattered, or rank good evidence too low to use. The generator can hallucinate facts absent from the context, ignore the context entirely, or answer a question the user never asked. 

This is why a single end-to-end “accuracy” score is misleading. If you only measure the final answer, you cannot tell which half is broken, so you cannot fix it. The field has responded with decomposed metrics — retrieval measures such as recall@K, precision@K, and nDCG, paired with generation measures such as faithfulness and answer relevance. Open frameworks and the widely cited “RAG triad” of context relevance, groundedness, and answer relevance exist precisely to localize failure rather than merely detect it. 

The non-determinism problem 

Classic software is testable because it is deterministic: same input, same output, every time. Generative models break that contract. The same prompt over the same context can yield materially different answers across runs, and sampling temperature is only the most obvious source of that variance. When the thing you are testing will not sit still, the very idea of a “passing” test gets slippery. 

This has a practical cost. A change that improves the answer to one phrasing of a question can silently regress a dozen other phrasings. Without large, representative test sets and repeated sampling, teams end up tuning against noise — chasing a number that flickers for reasons that have nothing to do with their change. 

When the judge is also a guess 

Because human review does not scale, the industry now leans heavily on “LLM-as-a-judge”: using one language model to score another model’s output. It is the only practical way to evaluate millions of open-ended answers, and when carefully calibrated it agrees reasonably well with human raters. But the judge inherits every weakness of the thing it is judging — it is itself non-deterministic, and it is biased in measurable, repeatable ways. 

Those biases are now well documented. Studies show that LLM judges exhibit position bias (favoring whichever answer appears first), verbosity bias (favoring longer answers), and self-preference bias (favoring text that resembles their own output). One systematic study spanning more than 100,000 evaluations confirmed that position bias is not random noise but a persistent, structural effect. When your measuring instrument has its own opinions, your quality metric is only as trustworthy as your ability to correct for them. 

The ground truth won’t hold still 

Even a perfect, unbiased evaluator faces a deeper problem: the corpus it grounds against is alive. Enterprise knowledge bases change daily — policies are updated, products are renamed, documents are deprecated. A test set that was “correct” last quarter can be quietly wrong today, not because the model changed, but because the world did. 

This is why static benchmarks decay. A one-time evaluation is a photograph of a system that is actually a river, and benchmark-style scores built for single-turn questions often miss enterprise failure modes entirely — misidentifying a case, violating a workflow’s required order, or resolving only part of a multi-step request. Freshness and source trust have to be measured continuously, or the metrics drift out of alignment with reality. 

Quality has a price tag 

None of this is free, and cost is where quality quietly erodes. Higher recall means stuffing more context into every call, which raises token spend and latency; tighter precision risks dropping the one passage that mattered. Running an LLM judge over live traffic multiplies inference cost, so teams sample rather than score everything — and sampling means some regressions ship unseen. Every quality dial has a cost dial wired to it, and pretending otherwise is how “good enough” metrics mask real degradation. 

Evaluation as a living system 

The teams that get this right stop treating evaluation as a launch gate and start treating it as infrastructure. That means layering cheap deterministic checks — does it cite a real source, is the format valid — beneath more expensive LLM-judge scoring, with human review reserved for the ambiguous tail. It means running those checks both offline against curated test sets and online against live traffic, with alerts on latency, cost, and quality drift. The goal is not a single score but a continuous signal: a regression suite for trust. 

The discipline that will separate the winners 

As AI moves deeper into regulated, fiduciary-grade work, the defining question is no longer whether a system can produce an answer, but whether anyone can verify that it stood on solid ground. That verification is an evaluation problem, not a model problem. The organizations that treat measurement as a first-class, continuous discipline — rather than a checkbox before launch — will be the ones whose AI can still be trusted when it matters. The model gets the headlines; the evaluation harness quietly decides who is still standing when the demo meets the real world. 

References 

1. Danilec, A. — RAG Evaluation with RAGAS: Faithfulness, Context Precision, and Recall in Production (DEV Community)
2. RAG Evaluation 101: From Recall@K to Answer Faithfulness (LangCopilot)
3. List of Available Metrics — Ragas documentation
4. Benchmarking LLM-as-a-Judge for the RAG Triad Metrics (Snowflake / TruLens)
5. RAG Evaluation: A Complete Guide for 2025 (Maxim AI)
6. Shi et al. — Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (arXiv:2406.07791)
7. Wataoka et al. — Self-Preference Bias in LLM-as-a-Judge (arXiv:2410.21819)
8. Ye et al. — Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (CALM)
9. Chhabra et al. — Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems (arXiv:2602.20379)
10. A Complete Guide to RAG Evaluation: Metrics, Testing and Best Practices (Evidently AI)