AI initiatives expose the business processes companies never fixed. An AI investment needs a plan for what must change around the technology.
“Why do you want to implement AI?”
“Because I’ve been told to.”
Have you had this conversation? It’s one my team has almost every week.
Every business wants to show that it can deliver AI, but too many skip the step where they investigate how and where the technology actually fits.
Tough love beats false confidence
It’s uncomfortable telling an enterprise client they’re not ready to operationalize AI across your business systems. Quality engineering doesn’t exist to stand in the way of progress – our job is to assess the data, platforms, models and digital ecosystems that sit beneath ambitions, and be honest about what needs to be fixed first. Otherwise, you’re trying to implement highly complex and sophisticated technology on systems and processes that can’t stand up to the test.
As companies push forward with AI integration, this lesson is being learned across boardrooms, and progress is happening because of it.
Your own business can prevent the AI from working
In one instance, a global retailer introduced AI agents to help validate new websites. On day one of the launch, security controls – used to defend against bot attacks – flagged the traffic from the new agents as suspicious automated activity and blocked them.
In isolation, the security controls were doing their job. But in the wider technology infrastructure, they directly prevented the desired business outcome. If they could go back in time, the retailer would have distinguished authorized agents from unwanted traffic in the development cycle.
Time travel unavailable, the fix came from looking into the release process and the different teams involved. On this occasion, the different teams had been working in siloes, which meant the security testing hadn’t covered the agents’ activity. All that was needed was to bring security into release planning and testing, and agree how the agents’ authorized traffic would be identified and handled. Problem solved.
The lesson goes beyond a security configuration: the environment needed to change for the investment to deliver its purpose.
To put it another way, introducing AI without considering those dependencies is like dropping an F1 engine into a golf cart. The engine adds power, but the steering, brakes and chassis determine whether that power is usable. In an enterprise, the data, integrations, permissions and controls must support the work the AI is expected to perform.
Installing an agent isn’t success in itself
Take an entirely different setting, financial services, and you’ll find the same type of challenge.
A firm introduced a customer service agent across web and mobile to reduce call centre volumes and improve customer satisfaction. Six months later, call centre volumes were up and customer satisfaction was down – a worrying prospect for the future of enterprise AI and the value of the investment.
All hope wasn’t lost, however, quality analysis revealed the defects in the solution, and how to fix them.
The first issue identified was that the solution was deployed without a semantic evaluation framework. It only assessed whether agent provided a response to a customer’s question, not whether the responses were correct, meaningful and useful.
Closer inspection unearthed two key vulnerabilities in the solution – one relating to test inputs, another with the evaluations. Before the first launch, only roughly 50 utterances – prompts – were tested. These had limited coverage of everyday costumer journeys, knowledge retrieval, ambiguous queries and security scenarios. Quality engineers expanded that to 1,740, drawing on OWASP’s guidance for LLM applications, MITRE ATLAS for security scenarios, and Stanford HELM to inform capability evaluation.
Within the original evaluations, the existing tooling covered basic checks, such as routing to the expected sub-agent, with only one sub-agent available – the check said little about whether the response actually helped the customer. In the updated approach, a five-point scoring rubric was used which covered intent alignment, completeness, safety, tool behavior and performance, weighted to the agent’s purpose. And to assess the semantic, criteria an LLM-as-a-judge approach was applied.
The problem came from how success was defined. The original metric had just been delivery, which lacked a clear success criteria and frameworks to demonstrate measurable ROI against business KPIs.
For a customer service agent, those criteria might include improved resolution rates, fewer repeat contacts, lower cost per resolved enquiry and maintained or improved customer satisfaction. Each needs a baseline, a target and ongoing measurement, with benefits assessed against the cost of delivering and operating the service.
A team can complete its AI project while the business problem remains.
No cutting corners
There’s no point integrating AI for the sake of it. It will likely cause more problems than it solves. To deploy AI successfully, and deliver real business value, it’s important to make sure the systems underneath can support the scale and complexity AI introduces.
Unfortunately, that’s an incredibly long and arduous process. Data needs to be validated, and in the right place. Any customized use of platforms needs to be defined and configured. Cross-platform integration has to be seamless and systems need to be ready for any instance where platforms ship their own AI. Compliance and audit trails must be provable at platform-level, not just theoretically sound. Lastly, any systems that lack the technical capability to support modern AI features have to be upgraded or re-platformed before pushing ahead.
Just thinking about those steps gives you a headache – it gives us one as well. But, skipping these steps will only come back to haunt you later on. Your best bet is to roll up your sleeves and put in the hard yards, or find a trusted partner who can help.
Get that right, and AI stops being a risk you’re managing and becomes the advantage it was always meant to be.
