AI has become synonymous with some very big numbers. For example, Gartner’s total AI spending forecast for this year is $2.7 trillion, with infrastructure investment accounting for 56% of that total, at $1.48 trillion. For once, the use of ‘disruptive’ and ‘transformational’ to talk about a big tech trend is well justified.
While drawing absolute comparisons is difficult, it’s fascinating to note that at roughly the same stage of the cloud revolution (in 2010, four years after the launch of AWS S3 and EC2), Gartner expected enterprises to spend $112 billion on SaaS, PaaS and IaaS combined over the following five years.
Back then, cloud hype was also in full effect. Headlines such as “Cloud set for boom years as companies catch on”, “Gartner says cloud hype reaching highest level yet”, and “Cloud computing: raining money” were everywhere. Fast forward to today, and they could easily be adapted to work just as well with AI, even though the levels of investment differ vastly.
The best laid plans
Recently, of course, the AI industry narrative has evolved, with Gartner now classifying Gen AI as firmly in the ‘Trough of Disillusionment’ phase, while Agentic AI is still riding high on the ‘Peak of Inflated Expectations’.
That makes a lot of sense, because in many cases, getting an AI strategy from concept to pilot to production turns out to be a lot harder than it is ‘on paper’ and is by no means guaranteed. Studies increasingly bear this out. Research from IDC and Lenovo found that for every 33 AI proofs of concept an organisation launched, only four made it into production, meaning close to nine in ten never progressed beyond the pilot stage.[1]
A big part of this problem is that legacy infrastructure was never designed to deliver on the innovation that AI promises. When these systems are in the early development or pilot phase, they typically need limited access to enterprise data and systems. For convenience, they may also be exposed to incomplete or dummy datasets to help demonstrate a concept or use case is valid.
Transfer this functionality into the real world, however, and the infrastructure challenge quickly becomes much more apparent. The AI system will almost certainly need dependable access to live data, along with integration into established business applications. Legacy environments often make this difficult, particularly where data is spread across different systems or workloads were never designed to operate beyond a single cloud platform. In fact, one recently published study found that 84% of CIOs had cancelled an AI pilot or project due to limitations in their legacy systems.
Closing the infrastructure gap
Failure rates of that scale aren’t sustainable and would raise serious questions if they occurred elsewhere in the typical enterprise, so what needs to change?
Fundamentally, there is a widespread disconnect between the contained isolation of a pilot project and the realities of live operations. In the case of conversational Generative AI, production deployment connects models to live corporate knowledge bases containing sensitive business information. This immediately introduces data governance issues, including access controls and the risk of context leakage, not least because organisations must ensure that sensitive customer data or proprietary IP never leaves the protected corporate environment.
These challenges are even more significant for agentic AI systems, which not only has access to data, but is now able to act upon it. Running autonomous agents in live operational environments depends on establishing identity boundaries, strict guardrails, and system integration layers. Without them, agents can go way beyond their intended role by accessing information or taking actions that were never part of the original task.
Legacy architectures were simply never designed to support these requirements, with many lacking the integration capabilities required for automated agents to safely invoke actions. This is where the pilot-to-production gap can become fully apparent. Assuming the project successfully navigates these potential roadblocks, organisations also need to control access and monitor the service’s performance over time.
Ultimately, the more complex an agentic AI implementation is, the greater the governance and ownership issues, especially if the pilot was built by one team but is rolled out by another.
Building for flexibility and control[2]
Addressing these issues is dependent on recognising that AI workloads won’t all belong in the same place. Some will need to stay close to the sensitive data they rely on, for example, while others may have to operate within a sovereign environment, and those requirements are likely to shift as projects move from pilot to production and beyond.
In practice, this means the underlying platform has to give organisations the freedom to move workloads as their needs change, rather than locking a decision made at the pilot stage into the long term. It’s an easy trap to fall into. Early infrastructure choices are often made for speed and convenience, with little thought given to what the system will need once it is running at scale.
Open source has a central role to play here. When AI workloads are packaged in containers and orchestrated on Kubernetes, they can run consistently in a public cloud, an on-premises data centre or at the edge, because the same Linux foundations and standard APIs apply in every location. That portability matters most at the point a pilot moves into production. A proof of concept built on a single provider’s proprietary services can be quick to stand up, but moving it later often means re-engineering data pipelines, integrations and security controls around different tooling. Open, standards-based platforms keep that option open. For AI projects that may need to change direction more than once before they reach maturity, that kind of independence is particularly valuable.
They also make data sovereignty practical rather than theoretical. Because the code can be inspected and the stack can run wherever the organisation chooses, including in fully air-gapped or sovereign environments, teams can keep sensitive data and models within their own jurisdiction and audit exactly how they are handled, without being tied to one vendor’s roadmap or pricing.
Even with the right infrastructure in place, however, successful production AI also needs a clear operating model that sets out who owns the system and who has the authority to change it, along with a reliable way of monitoring how it behaves once it is live. Without that clarity, the governance and ownership questions that so often derail pilots will simply resurface further down the line.[3]
The organisations that get their AI projects off the runway won’t necessarily be those with the biggest budgets or the boldest use cases. They will be the ones that treat infrastructure and governance as the foundation for take-off rather than an afterthought: building on open, adaptable platforms that let workloads move as requirements change, and agreeing who owns, changes and monitors each system before it goes live, not after. Get those foundations right from day one, and pilots stop being experiments waiting for clearance and start becoming business engines built to scale.
