By now the pattern is familiar to anyone building with AI inside a large organisation. In my work at NeenOpal, a data and AI consultancy, I see it play out repeatedly: teams reach capable models, stand up a pilot in a matter of weeks, and produce a demo that genuinely impresses the room. Then the project stalls. It never becomes the dependable system that real customers and real workflows lean on every day.
Survey after survey measures this gap, but few explain what closes it. The honest answer rarely shows up in a questionnaire, because it lives in the engineering decisions made after the demo and before the system is trusted in production.
The wall everyone is measuring
The headline numbers now point in the same direction. Deloitte’s State of AI in the Enterprise found that access to AI is close to universal, yet only about a quarter of organisations have moved even 40 percent of their experiments into production.
MIT’s Project NANDA reported that the overwhelming majority of enterprise generative AI deployments produced no measurable financial return. Gartner has forecast that a large share of generative AI projects will be abandoned after the proof-of-concept stage, citing poor data quality, rising cost, and unclear business value.
Read together, these findings describe a single shape. The bottleneck is no longer access to capable models. It is the distance between a model that works in a demo and a system that works in production, every time, for every user, under real load.
The misdiagnosis that keeps pilots stuck
Here is where many teams take a wrong turn. They treat a stalled pilot as a model problem, so they swap in a newer model, rewrite prompts, and wait for the next frontier release.
The trouble is that the model is usually the part that already works. The real failure points sit one layer out, in the parts of the system a demo never stresses: how information is retrieved, how outputs are checked, how requests are routed, and how quality is measured over time.
It is worth being precise about scope. Plenty of pilots never ship for reasons that have nothing to do with engineering, such as a missing business case, unusable data, or no executive sponsor. Set those aside. For the large set of pilots that are technically real and demo well, the deciding factor is almost never the model.
Grounding is what makes an answer safe to ship
The first place pilots fail in production is trust. A model that sounds fluent can still invent facts, and in a regulated or customer-facing setting a confident wrong answer is a liability rather than a glitch.
Reaching for a more capable model does not solve this. A more fluent model can produce more convincing hallucinations, which is worse rather than better.
What changes the outcome is retrieval-augmented generation paired with verification. That means pulling answers from governed, authoritative sources, then checking generated claims against those sources before anything reaches a user. A system that would rather abstain than assert something unsupported is one you can actually put in front of customers.
Routing is where cost is quietly decided
The second place pilots die is the budget review. A system can work well and still be cancelled when per-token costs, multiplied across thousands of users, become a total-cost problem no one modelled.
Sending every request to the most powerful model is the usual mistake. Most workloads are a mix, where a large share of requests are routine and only a small share are genuinely hard.
Routing routine work to smaller, cheaper models and reserving frontier models for the cases that need them turns cost from a recurring surprise into a deliberate design choice. Managed platforms such as Amazon Bedrock make this kind of tiered routing practical without rebuilding the application.
Evaluation is the part that never ends
The third gap is evaluation. In a demo, quality is judged once, by people who already know what to ask and what to avoid.
In production, quality has to be measured continuously, against the organisation’s own benchmarks, as inputs drift and usage widens. Without that, teams discover regressions when users do, which is the most expensive possible moment to find them.
Governance that enables rather than blocks
There is a common belief that governance is the brake on shipping. In practice the opposite is closer to the truth.
The grounding and verification work that makes a system trustworthy is the same work that makes it governable. A retrieval-and-verification layer is what actually stops a model from inventing a fact at inference time, which no policy document can do on its own.
Done at the engineering layer, governance is not what slows a build down. It is what lets the build ship at all.
What to check before you blame the model
If a generative AI project is stuck in proof-of-concept, the most useful instinct is to resist looking at the model first.
Ask instead whether the system answers from governed, verifiable sources, whether anything checks an output before a user sees it, whether every request is paying premium prices, and whether quality is measured continuously rather than once.
The organisations crossing from pilot to production are not the ones with access to the best models. Everyone has that now. They are the ones who understood that the model was never the hard part, and who put their engineering effort into the layer around it.
Akshat Agrawal is a GenAI Architect at NeenOpal, a data and AI consultancy and AWS Generative & Agentic AI Competency partner that also delivers on Microsoft Azure. The deployment figures referenced here are drawn from NeenOpal’s published case studies; the architect is available to walk through the methodology on the record.
