Over the next two years, agentic AI systems will move from the early stages of deployment to operational maturity. They will not arrive with dramatic fanfare. Instead, they will embed quietly into enterprise software, supply chains, compliance functions, security operations centres and decision-support systems. Workflows will accelerate, costs will fall and productivity metrics will improve.
But beneath these gains lies a more complex reality – the regulatory, data protection and trust risks of agentic AI are being seriously underestimated.
Unlike earlier generations of AI tools that generated text, images or predictions on request, agentic systems take action. They chain decisions together, interact with multiple data sources, trigger downstream processes and operate at scale, with varying degrees of autonomy from humans over or in the loop. In controlled testing environments, this appears manageable, however, in live operational settings over months and years, it becomes far more difficult to monitor, audit and govern.
Current operational challenges
There are two fundamental challenges that businesses must address.
First, how do organisations monitor what agentic systems are actually doing once deployed? These systems evolve through updates, integrations and retraining and they interact with new data environments.
Second, how do you ensure responsible behaviour throughout the lifecycle? Regulators, policymakers and customers will likely expect firms to shift from compliance assurance to risk assurance and demonstrable evidence of trust and transparency.
The prevailing assumption is that human oversight will mitigate these risks. Human in the loop or human over the loop has become the default reassurance. In practice, however, that assumption breaks down far faster than many anticipate.
When a system works 95 per cent of the time, human reviewers limit their scrutiny. Behavioural science tells us that automation bias and complacency occur when automated systems are high-performing. Employees often become validators of AI outputs rather than critical examiners. The diligence gap widens gradually and then suddenly.
This raises uncomfortable but necessary questions.
How do you incentivise employees to remain diligent checkers when the system mostly “works”? And how much time does effective oversight actually require? True review is not a cursory glance at a dashboard. It involves interrogating assumptions, validating inputs, checking context and assessing downstream consequences. In many cases, meaningful oversight may take nearly as long as performing the original task manually. When checking becomes more costly than doing the job yourself, pressure to “trust the system” intensifies.
And what happens to accountability when oversight exists on paper but not in practice? Governance documentation may show layered review structures, escalation pathways and audit processes. Yet if humans are functionally disengaged, responsibility becomes dispersed. When errors surface, organisations may struggle to attribute fault – was it the model design, the data, the integrator, the operator or the reviewer who signed off without fully scrutinising?
Regulators are only beginning to grapple with these realities. In jurisdictions such as the European Union, the EU AI Act introduces risk-based obligations, documentation requirements and human oversight provisions. These are important steps, however, the operationalisation of those requirements in dynamic, agentic environments remain untested at scale. Compliance on paper will not automatically translate into resilient governance in practice.
Confronting the trust challenge
Beyond regulatory exposure, there is a broader trust challenge emerging.
As agentic AI systems scale across industries, they will generate vast volumes of automated outputs – reports, communications, risk assessments, content, decisions and transactions. If errors or manipulations spread through interconnected systems, confidence in digital outputs may erode.
In geopolitically sensitive contexts, this has profound implications. Agentic systems interacting with external data sources could amplify disinformation, introduce biased datasets or make decisions based on manipulated inputs. The speed of automation may outpace the speed of verification. Trust, once diluted, is difficult to restore.
Data protection risks will also intensify. Agentic systems frequently require broad access privileges to perform tasks effectively. They may access internal databases and personal data and interact with third-party platforms. Each interaction creates potential exposure points. A single misconfiguration or prompt injection attack could trigger cascading consequences across systems.
The next phase of AI adoption will not simply amplify productivity. It will amplify regulatory, legal and reputational risk.
This moment therefore demands serious scrutiny before agentic AI becomes deeply embedded in business infrastructure.
So, what should organisations be doing now?
It’s time for businesses to act
Firstly, businesses should move beyond checkbox compliance. Governance must be operational, not merely documented. This means investing in continuous monitoring capabilities, robust audit trails and real-time anomaly detection tailored specifically to agentic AI behaviours.
They should also be redesigning incentives. Oversight must be valued, measured and rewarded. If employees are expected to provide meaningful review, organisations must allocate time, training and authority accordingly. Performance metrics should reflect risk management responsibilities, not just output velocity.
Clarification of accountability structures is also key. Boards and executive teams should define who ultimately owns agentic AI system outcomes. Shared responsibility models with vendors must be explicit. Escalation pathways for incidents should be tested through simulations, not assumed to function under pressure.
Cross-functional expertise should also be embedded across the organisation. Legal, compliance, cybersecurity, data protection and operational teams must collaborate from the outset. Agentic AI is not solely an IT deployment – it is an enterprise risk transformation.
Finally, scenario-plan for failure. Organisations should ask not only how these systems succeed, but how they fail at scale. What happens if a model update introduces systematic bias or a compromised integration triggers unauthorised transactions or automated outputs undermine customer trust? Preparedness requires uncomfortable but necessary exercises.
As agentic AI matures, risk and governance models must keep pace
None of this is an argument against adoption. Agentic AI offers genuine efficiency gains and strategic advantage. Organisations that harness it responsibly may outperform competitors. But speed without governance is a fragile strategy.
The question is no longer whether agentic AI will be adopted. It will. The more pressing question is whether governance, incentives and accountability structures can evolve quickly enough to keep pace.
Over the next two years, businesses face a narrowing window. Once agentic systems are fully embedded, retrofitting controls will be far more difficult and costly. Leaders must therefore treat this period as a design phase for oversight, not merely a race for competitive advantage.
Agentic AI is maturing fast. The real test is whether our risk frameworks, regulatory thinking and organisational cultures can mature just as quickly.
