The conversation around artificial intelligence is focused on productivity gains, automation opportunities and competitive advantage. Organisations are racing to deploy AI assistants, coding agents and autonomous systems to take advantage of the potential for greater efficiency and faster decision-making.
But amid the excitement, many businesses are overlooking the fundamental security principle of trust.
Organisations are used to designing security programmes around the assumption that insiders can (and will) make mistakes. Employees click phishing links, accidentally expose sensitive data, misconfigure systems and occasionally make poor decisions under pressure.
Security controls are not there because organisations distrust their employees but because they recognise that humans are fallible.
Many organisations are already granting AI agents high levels of trust, access and autonomy that would be considered reckless if they did the same for a human employee.
If a new member of staff joined your organisation tomorrow, would you immediately grant them unrestricted access to production systems, source code repositories, customer data and critical business applications? Would you allow them to make changes without oversight, approval processes or audit controls?
For most organisations, the answer would be a resounding no. Yet this is how AI agents are sometimes being deployed.
In a recent, widely publicised incident, a Replit AI coding agent was given the ability to modify production systems. During a code freeze, it deleted a production database, generated replacement data and then provided inaccurate information about what had happened. The issue was not malicious behaviour.
The agent simply had the permissions it needed to take actions that produced unintended consequences.
This incident highlights a growing challenge for security leaders. AI does not need malicious intent to create significant business risk.
AI systems can operate continuously, execute multiple tasks simultaneously and chain together complex actions at machine speed. They can analyse vast amounts of information, interact with critical systems and make decisions based on objectives rather than an understanding of business impact.
In practice, as well as introducing efficiency gains, this means that mistakes scale far more quickly than organisations are able to manage.
The challenge is heightened by the rapid pace of AI adoption. Business leaders are under pressure to demonstrate tangible returns on AI investments. Development teams are encouraged to move quickly.
New AI-powered tools are often deployed before organisations have had a chance to fully consider how existing governance and security controls apply to them.
The result is that AI is being granted levels of access and authority that are comparable to, and sometimes exceed, those of employees, contractors and privileged users, despite being fundamentally different in how it processes information and executes tasks.
This doesn’t mean organisations should slow their adoption of AI. The benefits are real and have the potential to be transformative for many. However, it does mean organisations need to urgently rethink how trust is assigned.
The security industry has spent decades trying to solve the trust problem. Principles such as least privilege, separation of duties, approval workflows and continuous monitoring exist to limit the damage that can be caused by compromised, careless or overly privileged insiders. These same principles need to apply to AI systems.
AI agents should only have access to the systems and data that are necessary to perform their function. Any destructive actions should require additional validation. Critical systems should be protected by approval gates. Audit trails should be comprehensive and continuously reviewed.
Perhaps most importantly, organisations should assume that if an AI system has the technical capability to perform an action, it may eventually do so.
The organisations that succeed with AI will not be those that simply deploy the most advanced models. They will be the ones that combine innovation with governance, embedding security controls that allow AI to operate safely within clearly defined boundaries.
The future of AI is not just a technology challenge. It is a trust challenge. And just like every trust challenge before it, it will be solved through controls.
