The promise of AI-powered data analysis has never been louder. AI-enabled technology platform vendors across industries claim their tools can extract data from a variety of document formats with near-perfect accuracy. What they don’t talk about is how—and if—their tech handles scale. Most of these AI platforms are betting on the same approach: Retrieval-Augmented Generation, or RAG.
The issue is that they’ve never stress-tested their technology against thousands of documents, which is the kind of volume enterprises are actually dealing with. In a controlled lab setting, results look promising. But, when things get out of the lab and into the real-world where you collide with an enterprises’ years of accumulated contracts, inconsistent formats, and the messy reality of enterprise contracting, those results fall apart.
Unfortunately, for the end user, it’s a bet that doesn’t pay off.
A recent Stanford University study published in the Journal of Empirical Legal Studies put leading RAG-based AI tools to the test. The results were sobering: even purpose-built systems hallucinated between 17% and 33% of the time. The highest-performing RAG system was accurate on just 65% of queries.
The researchers were blunt in their conclusion: “RAG systems are no panacea.”
For anyone trying to extract reliable data at enterprise scale, this should be a wake-up call. RAG isn’t just underperforming—it’s fundamentally the wrong architecture for the job.
Why RAG Fails at Scale
RAG works by retrieving relevant documents (or document chunks) and feeding them to a large language model to generate an answer. In theory, this grounds the AI’s response in actual source material, reducing hallucinations. However, the Stanford study itself acknowledged this: the failure modes become pronounced once you exceed a certain volume threshold. Past that point, accuracy takes a sharp drop and hallucinations multiply.
This particular study tested these systems under controlled conditions with a clean, curated query set of just 202 documents. Even on this small scale, hallucinations were ever-present. Most enterprises are operating with a repository of contracts far past that level of volume and complexity. If RAG struggles in a lab, the question becomes: what happens in the real world?
Even in a controlled environment RAG underperforms in delivering accuracy at scale. The researchers documented four distinct failure modes in RAG systems: naive retrieval that surfaces irrelevant documents, citations to inapplicable sources, basic reasoning errors, and outright fabrications. In 61% of one system’s hallucinations, the root cause was a reasoning error—the system had retrieved the right information but drew the wrong conclusion from it.
As the VP of Product for an AI Contract Intelligence platform, I spend a lot of my time building and refining AI systems that extract and analyze contract data. What I have discovered reinforces the Stanford researcher’s findings: when it comes to extracting accurate, reliable data at scale, RAG is not the solution.
The problem is that RAG treats every query the same way: retrieve, then generate. But in my line of work, Contract Intelligence doesn’t work like that. The way you extract a document title is fundamentally different from how you extract a renewal date. And both are different from how you analyze an assignment clause or identify a change of control provision.
It’s not just a scale issue related to how many documents you’re trying to analyze—it’s about how many concepts across those documents you’re trying to analyze. If you’re extracting a few pieces of data, you may get success. But, try to pull 50 or 60 data points from one agreement, and you’re going to run into an accuracy cliff. RAG’s one-size-fits-all retrieval approach simply cannot handle this complexity at scale.
And that’s before you account for the chaos of real enterprise data.
The Real World Is Messier Than Any Experiment
Here’s what makes the Stanford findings even more concerning: the study was conducted under controlled conditions. Researchers used 202 carefully constructed queries, each designed to probe specific capabilities. The AI systems received clean, well-formed questions in a structured testing environment.
A single enterprise organization may have decades of accumulated data files, each with different standards and styles. Duplicate files, inconsistent data structures, and varied formats lead to inaccurate AI outputs because RAG systems choke on such levels of complexity. Essentially, they retrieve chunks of text without understanding context.
If RAG systems hallucinate 17-33% of the time under ideal testing conditions, what happens when they’re processing this chaotic reality?
The Stanford researchers themselves acknowledged this limitation, noting that their benchmark “may not represent the natural distribution of queries” and that real-world performance could differ significantly. They also found that systems changed their responses over time—the same query could yield different results on different days, making consistent enterprise deployment even more challenging.
There Is No Single Technique—And That’s the Point
The most important lesson I’ve learned is deceptively simple: there is no one-size-fits-all approach to data extraction. RAG assumes there is. We know there isn’t.
Instead, our organization has developed an assembly line approach, applying different extraction techniques to different problems, and orchestrating these techniques in a precise sequence. Depending on the job at hand, the type of automation technique can vary, and we pick the method that will yield the best results. Sometimes that means using an LLM, and others it means leaning on more traditional extraction methods. The way we identify a document type is different from how we extract specific contract data like renewal dates, which is also different from how we analyze liability caps or termination rights.
The point is, we use large language models where they excel—but we don’t ask them to do everything. We don’t generate answers from loosely related document chunks and hope for the best.
Noise Is the Enemy
The other hard-won insight: you cannot overload AI with unnecessary context. Give it noise, and you get hallucinations. This is RAG’s fatal flaw—it retrieves content based on similarity, not relevance, flooding the model with information that may or may not matter.
The Stanford study found that longer AI responses correlated with higher hallucination rates—more content means more opportunities for error. The same principle applies to inputs. At enterprise scale, noise is inevitable. For example, large document sets within contract intelligence platforms likely contain files that are not contracts or may contain duplicate contracts.
The more precise your context and data grounding, the more accurate your results. This is especially true for contract intelligence solutions, and the reason our organization remains focused on building targeted extraction capabilities that let AI focus on exactly what matters—nothing more, nothing less.
Enterprise Scale Demands More Than RAG
The Stanford study confirms what we’ve known for years: AI at scale is genuinely hard, and RAG isn’t the answer.
If your problem is analyzing a single document, a $20/month ChatGPT subscription will probably get you there. But the reality is that RAG is not designed for enterprise organizations with massive datasets that include imperfect data architectures.
At the enterprise level, truly groundbreaking AI solutions require a nuanced approach that leverages multiple data extraction techniques, while minimizing unnecessary context. It’s not an easy process, but when built correctly, it can completely transform your AI results.
