Remember watching Hal turn his back on Dave and thinking, “We’ll never let our robots outsmart us,”? The day when they do is closer than ever.
Three-quarters of security leaders now rank generative AI as their top IT risk. Nearly half are calling for a “strategic pause” on its development, and almost all, 93%, expect AI-driven cyberattacks to become a daily occurrence this year.
We are already seeing this in practice. Anthropic’s Claude was specifically used to write “psychologically targeted extortion demands” and emotionally intelligent messages, enabling attackers to gain the trust of victims.
Large language models (LLMs), like Claude, can mimic the tones of a brand voice or an individual based on the content provided, fooling millions. The AI giant does a good job banning relevant accounts; however, smaller LLM providers may not be so fast. AI-generated phishing campaigns accelerate credential theft, which will be the entry point for 88% of web app attacks in 2025.
AI has certainly expanded the attack vector, with credential theft leading to almost undetectable entries into corporate devices and systems. But since the technology is already out there, would a pause in AI development be in our favor or our demise?
Infostealers distributed via phishing emails surge
X-Force reports an 84% weekly increase in infostealers delivered by phishing emails. The trend is part of a broader rise in credential-focused phishing activity, both of which lead to the same outcome: attackers walking in with valid login details.
That shift makes phishing a nuisance that enables entry for account compromise that is very hard to trace. Clicking on what looks like a routine email link can install malware designed to siphon the victim’s sensitive data without notice. In some cases, the stolen credentials are then reused to move further into company systems or to support identity-based attacks.
Attackers are also hiding their tools more effectively. Malware payloads, which steal credentials and corrupt files, are embedded or disguised to avoid detection, stretching the time it takes for organizations to realize ransomware or data theft has already taken place. They do this with encryption, encoding, and other techniques to hide them within files. The result is a longer window of exposure and higher stakes once the breach is uncovered.
AI fools today’s threat intelligence platforms
Security teams increasingly count on threat intelligence platforms (TIPs) to pull together indicators of compromise (IOCs)—hashes, IPs, domains, behavioral signals—from a mix of sources. They aggregate this data, making it usable for tools like endpoint protection and anti-malware, tightening detection against threats designed to slip through the gaps.
However, when hackers apply automated malware builders to create thousands of payload variants, TIPs get flooded with IOCs that may never be used again, leading to an overwhelming volume of false positives. Cybersecurity investigators who shift from TIP’s static indicator collection to contextual, behavior-driven intelligence have a higher chance of cutting through this noise.
Dark web forums and marketplaces are prime spots to eavesdrop on chatter about how adversaries plan to use or adapt malware kits and capture that behavioral data. Yet most TIPs are blind to these conversations. TIP platforms are often built on STIX or similar formats, which are currently poorly suited to ingest unstructured or fast-changing content from dark web monitoring or threat exposure management (TEM) sources.
This gap creates room for adversaries to operate unchecked. When a TIP cannot process or validate this type of chatter, it leaves CISOs without context on emerging threats.
How LLMs can empower security practices
Attackers weaponizing LLMs is a huge threat. But even if the leading LLMs (like OpenAI, Google, Anthropic, Meta, etc.) pause deployment, open-source models are already out; hackers can download, fine-tune, and run them locally. So, the most practical way for CISOs to stay ahead is to find ways to safely apply the same technology and close intelligence gaps.
Looking at LLMs’ benefits, they can process unstructured text at scale, extract patterns, and highlight signals hidden in mounds of data. Applied to dark web monitoring, this means organizations can automatically surface conceivable mentions of their domains, credentials, or infrastructure from across fragmented sources.
This capability changes TEM from a reactive process to a proactive one. It gives CISOs near-real-time visibility into early IOCs. With the right safeguards, LLM-driven analysis can distinguish between payload variant dumps and false positives by mapping campaigns.
A recent study used GPT-3.5-turbo to sift through and map infection paths with a 96% accuracy rate. Precautions like clear, tense-aware prompting and breaking down dark web threads into smaller, coherent segments improved accuracy. With trained investigators guiding each step, these tools have the potential to reclaim hours, or even weeks, otherwise lost to chasing dead ends.
The challenge is balance. Blind trust in AI is as risky as ignoring it altogether. Outpacing AI-native hackers will depend on pairing AI-powered insights with human judgment, ensuring that machines’ speed augments the discernment of experienced analysts.
