Every technology arrives with assumptions attached, and the security team’s job has always been to test them before the business trusts them. Over two decades and three platform shifts, I have watched the time available to do that collapse. With mobile, we had years. With the cloud, we had months. With AI, organisations are trusting outputs in production before anyone has written down what the assumptions were.
The models are honest about this. ChatGPT carries a permanent warning: “ChatGPT can make mistakes. Check important info.” Claude tells users: “Claude is AI and can make mistakes. Please double-check responses.” Ask ChatGPT directly what the risks are of trusting its output without verification and it does not skimper. It names the security principle being violated, “treat all external input as untrusted”, and then classifies itself as “untrusted, probabilistic data” rather than authoritative truth.
The most reliable thing the technology tells you about itself is that you should not trust it, and yet people still ‘yolo’.
Of the many challenges, in their default state, models know nothing about your architecture, obligations or risk appetite unless you tell them every time. Outside of sandboxes, models can accidentally delete code bases. Without proper controls, ask a model what another customer has been doing, and it will tell you. Nothing broke. It worked exactly as it was built. Functional and secure are not the same property, and no signature-based scanner catches that gap, because there is no pattern to match on.
That kind of weakness also accumulates. Once generated code is in the baseline, the next person asks AI to extend it, and the flaw propagates through every iteration after.
Organisations are investing more in security than ever, and those numbers should be falling. Fluent output is persuasive, people stop interrogating it, and the difference from human error is throughput: a person makes one bad call, a model makes the same one everywhere it touches before lunch.
In my world of offensive security, there is an inherent asymmetry here to highlight. When AI gets something wrong inside an organisation there is a cost: an incident, a disclosure, a regulator, a conversation with the board. The people attacking that organisation carry none of it. No change control, no risk appetite to respect, no audit trail. For a CISO, a wrong AI output is an incident. For an attacker, it is a wasted prompt, so they retry.
This gap is showing up in real campaigns. Recently published research documented threat actors using AI agents to breach 14 companies, running full Claude and Codex agents locally on compromised hosts and driving them remotely through reconnaissance, exploitation and exfiltration. In many cases the prompting was barely competent. As the report puts it, “the attacker supplied only vague, low-skill prompts and allowed Claude to fill in the gaps.”
The encouraging part is that none of this requires a new playbook. The controls that contain AI risk are the ones security teams already run,including least privilege, sandboxing, and audit logging to name a few. These measures ensure models cannot reach critical systems directly, keeps generated code from executing straight into production, and keeps a record of AI suggestions alongside what was actually executed.
Human review sits across all of it and matters most for code, configuration and security decisions. Take the model at its own word, treat it as untrusted input holding elevated privileges, and thirty years of practice tells you what to do next.
Those controls lower the blast radius when something goes wrong. What no organisation can lower is the cost of being wrong itself, because governance and disclosure obligations exist for good reasons. If that cost is fixed, the only lever left is the rate of finding out. That means validating continuously rather than at quarterly intervals, prioritising what an attacker can actually reach over what scores highly, and putting people on the problems automation structurally cannot see: business logic, broken access control, the authorisation gap that only makes sense in context.
The organisations must treat AI as an amplifier of human judgement rather than a replacement for it, matching an adversary who can afford to be wrong with the one advantage that still scales: people who are paid to prove them wrong first.
