From diagnosing medical conditions to piloting autonomous vehicles, AI systems are powering critical workflows across industries and making decisions that can carry profound consequences. Yet as their role expands, so does the attack surface – making it a prime target for cybercriminals.
What makes this threat landscape unique is that attackers are not just targeting AI systems as any other piece of technology infrastructure. Instead, they are exploiting the very features that make AI valuable: its ability to learn, adapt and generate outputs. This creates an evolving AI attack surface that organisations must understand if they are to prepare for the risks ahead. With this in mind, what vulnerabilities are there in AI systems that could be exploited, and what malicious use cases of AI by threat actors should organisations be aware of?
Prompt and input manipulation
AI systems rely on input data and prompts to generate outputs, but these same inputs can be manipulated to produce harmful results. Prompt injection attacks involve feeding an AI misleading or malicious instructions. This can cause the system to misclassify data, disclose sensitive information or in some cases produce dangerous outputs.
The consequences can be severe in high-stakes applications. For example, an AI-driven triage tool in healthcare could misdiagnose a patient, or a self-driving car could make unsafe navigation choices. Even in lower-risk contexts, subtle manipulation (like altering inputs to a financial risk model) can enable fraud or errors that undermine trust in AI systems.
Corrupting and copying AI
Beyond manipulating inputs, attackers can target AI at its core. Data poisoning for example involves tampering with the datasets used to train AI models. Since AI learns patterns from training data, any corruption can have widespread consequences, from misclassification in automated tools to skewed predictions in critical systems.
Another growing concern is model extraction. By repeatedly querying AI systems, attackers can reconstruct proprietary models and identify vulnerabilities. Recreating high-accuracy copies of AI systems thereby gives adversaries a shortcut to replicate costly intellectual property while exposing organisations to commercial and reputational risks.
These threats are not hypothetical. Data poisoning and model extraction have already been demonstrated in multiple sectors, highlighting the importance of secure model development, and ongoing monitoring for threats.
AI-powered scams and malware
AI is increasingly being weaponised to enhance traditional cybercrime. A major exploitable feature of generative AI models, particularly large language models (LLMs), is their ability to produce human-like text at scale. This capability has already had a measurable impact: phishing attacks have reportedly increased by over 1,200 percent since the rise of generative AI.
Instead of manually crafting convincing phishing emails or smishing messages, threat actors can now use AI tools to automate the production of content that is tailored to the target. By scraping publicly available data from social media or corporate websites, attackers can personalise emails and messages to an unprecedented degree, making them far more likely to succeed.
Beyond text, AI is also being used to create realistic fake images, audio and video – North Korean IT workers, for instance, have reportedly leveraged AI-enhanced images and CVs to produce convincing worker profiles. They are also experimenting with voice-changing software, which could be used to trick interviewers into thinking they are communicating with legitimate candidates. Such deepfake techniques are increasingly employed in a variety of campaigns, often combining AI-generated video and audio to deceive targets. Threat actors can use these capabilities not only in phishing-style attacks, but also to influence opinions or manipulate perceptions, as has been observed in various influence operations.
Malware is similarly evolving. Threat actors are using AI to create code that adapts or mutates to evade detection. Some malicious applications are disguised as legitimate AI tools, luring users into downloading and executing malware under the guise of productivity or generative AI utilities. The combination of AI-enabled social engineering and adaptive malware represents a significant step-change in the sophistication and scale of cyber threats.
The rise of AI-powered scams underscores the importance of organisations adopting proactive security measures, including robust employee training, monitoring for anomalous communications, and incorporating AI-aware threat detection strategies.
Navigating the AI attack surface
The rise of AI-specific cyber threats is not a reason to slow innovation, but it is a call for vigilance. Organisations adopting AI need to be aware of the multiple ways these systems can be exploited – from malicious inputs and poisoned training data to model theft and AI-powered scams. To protect operations and maintain trust, businesses should monitor AI inputs and outputs for anomalies and ensure that training data and model access are securely managed to prevent corruption or theft. Educating users about the risks of AI-powered scams and implementing robust detection systems is also essential.
Furthermore, because AI systems continuously evolve, organisations must adopt adaptive security measures that evolve alongside them. This includes leveraging threat intelligence to stay ahead of emerging attack techniques and to inform the development of resilient, AI-powered defences.
By taking these steps and embedding security into AI development and deployment from the outset, organisations can harness the benefits of artificial intelligence while minimising risks to reputation and operational integrity.
