Businesses of all shapes and sizes are investing heavily in AI, but most are struggling to gain full reward. A February study by Deloitte found that while 85% of companies surveyed had increased AI spend in the past year, only 10% were realising ‘significant’ return on investment in agentic AI.
The problem is scale. 88% of companies now use AI in at least one function, but most are still in early stages of scaling. At Pencil, we call this ‘pilot purgatory’. Businesses are sitting with a range of pilots where AI excels at the contained task it’s been given, but breaks when scaled due to inadequate outputs when confronted with real business situations.
The most common cause I see is businesses trying to scale pilots independently, each managed by separate parts of the business. Naturally, this siloed approach creates problems, as AI models inevitably require data and context from across the business to operate effectively.
To truly benefit AI at scale, organisations need a single system that can oversee and integrate all models. It needs to be treated as part of the body of the business, rather than just a limb. This often requires taking the responsibility away from AI Leads or the innovation team and placing decisions with the board.
Although many leaders recognise the potential of AI, too many are at arm’s length from the key decisions being made. Many leaders are delegating to insulate themselves from the risk of AI not going as planned; but in doing so they are also placing a key barrier to achieving the return on investment they need. Leaders need to get closer and fully buy in to the implications of scale.
How involved in AI should the C-suite be?
The difficulty for C-suite leaders is where to draw the line of intervention, because AI discussions can easily get into a level of detail they wouldn’t normally find themselves discussing.
I think it’s important for leaders to concentrate on the workflows around the models, rather than the models themselves. Too much attention is given to the use cases, but the real efficiency lies in connecting the dots between them.
To use an example from my industry, many marketers are currently focused on the ability of AI to automate elements of production work. Generative AI now gives marketers the ability to produce high-standard image and video assets quickly.
Rather than getting into the weeds of comparing different video generation models, we’re telling CMOs to focus on the ecosystem before and after that production phase. How is the AI being briefed? What are the signoff processes afterwards? Can that production model quickly respond to client feedback and A/B testing, or does it still require multiple humans in the loop? What is the tech stack that currently underpins that process?
These fragmented processes – the unsexy part of business – is where AI’s return on investment is realised. To tackle this, executives need a system they can turn to and trust.
Building the harness
In AI development, a ‘harness’ is the scaffolding that wraps around a model, giving it tools, data, and direction. Without one, even the most powerful model is just raw capability with nowhere to go.
What leaders need to know is how to implement this harness. It’s the ‘Operating System’ that links all the moving parts, gives AI agents access to your business data and context, and allows you to control and instruct them from one central interface.
Agentic capability is ultimately the combination of the power of different models working together. A recent blog from the Modern Data Company argued that AI agents will always be ineffective unless they understand the wider context of the data they’re given, and for that to happen, “the meaning of data must travel with the data itself.” A harness enables this by pooling knowledge and resource for AI to pull from, compare against, and interact with.
These applications usually operate from a central LLM which can oversee, instruct, and report back on specific model and use case performance. Think of a single conversational interface that can answer any question you have about your business performance down to the smallest detail.
Finding the return
An operating system – your harness – is the level business leaders should be aiming for to achieve the return on investment they were promised. Once established, it can give leaders the kind of insight they need to understand integration progress and discover even more efficiency.
ROI will not be found in individual model applications, but from the harness that links them to the business and enables scale. Leaders need to focus on creating the system that gives agentic AI the context and connection it needs.
And it’s not enough to entrust this to individual teams or the operations department. Executives must get closer to AI operations and make the difficult calls on how AI can and should transform the body of the business.
