Automated bots are now a defining part of how the internet works, with AI-crawlers and API-driven traffic now handling a growing share of the tasks that keep our online services running. In fact, recent research found that 20% of all internet traffic is now automated, highlighting how deeply bots have become embedded in our lives.
However, not all automated traffic is harmless because some bots are designed to scrape content without permission, steal data, take over accounts through repeated login attempts or flood websites with fake traffic to slow them down or knock them offline.
The same report also found that AI crawlers now generate more traffic than traditional search engine bots, accounting for 0.54% of all requests compared to 0.50% for search bots. The issue is that these bots are made to look like real people, which makes them increasingly harder to spot.
As AI becomes more common across the web, telling the difference between good and bad bots is becoming one of the biggest security challenges organisations are having to deal with. So, let’s take a closer look at how bots behave, how legitimate and malicious activity differ, and which warning signs can help identify potentially harmful behaviour before it causes real damage.
Analysing behavioural patterns
Even as bots become more sophisticated, they still tend to act differently from real people. For instance, bots can generate requests at a speed and volume that no human could match. A credential stuffing attack, for example, can involve thousands of login attempts in a matter of minutes.
The way bots move through a website is another giveaway. Instead of following a natural sequence, bots often jump between pages, load deep-linked pages directly and bypass the logical steps a real person would usually follow when visiting a site.
They also tend to miss the small signals that come naturally with human behaviour, like mouse movement, scrolling or time spent reading content. So, by establishing a baseline for what normal behaviour looks like, organisations can more easily spot the deviations that signal that traffic is automated.
Fingerprinting techniques
Device and browser fingerprinting is a key technique for spotting automated activity because every user who interacts with a website leaves behind a range of identifiable attributes that can offer real insight into whether traffic is genuine or not.
IP addresses and geolocation data are a good starting point, since they help establish where requests are coming from. For example, legitimate users usually connect through residential internet service providers, whereas the majority of bot traffic comes from data centres, proxy networks, or VPNs. User-agent strings provide another clue because legitimate browsers usually report standard information, while automated tools might have unusual identifiers.
Canvas fingerprinting also helps by looking at how a browser renders graphics using HTML5’s Canvas API because many bots run on headless browsers like Puppeteer or Selenium and these tend to produce small but detectable quirks in how graphics are rendered.
So, by correlating these attributes, security teams can build a much clearer picture of their incoming traffic and get better at telling real users apart from automated scripts.
CAPTCHAs and challenge-response tests
CAPTCHAs (Completely Automated Public Turing tests to tell Computers and Humans Apart) work by posing challenges that are simple for people but hard for bots to solve. Everyone reading this would have come across a text-based CAPTCHA, which asks you to type out distorted characters from an image, or an image-based version that asks users to pick out specific objects from a set of pictures (like picking all the images with bridges or traffic lights in them).
More advanced behaviour-based CAPTCHAs might check how a user interacts with a page by asking you to drag a puzzle piece into place. But traditional CAPTCHAs are becoming less reliable as more sophisticated bots and CAPTCHA solving services get better at bypassing them.
This has led to the creation of tools like Google’s reCAPTCHA v3, which relies on behavioural analysis to assign a risk score in the background, without requiring any direct interaction from the user.
JavaScript and honeypots
Most websites rely on JavaScript to run properly as part of the normal browsing experience, but many bots – especially those operating at scale – disable or limit JavaScript to save resources and work more efficiently. Because of this, simply checking whether JavaScript executes successfully can help quickly filter out a large chunk of automated requests.
Another method involves setting up honeypots like hidden form fields or links that are invisible to human users but detectable by bots. If a bot interacts with one of these traps, it is immediately flagged as an automated agent and security systems can block it on the spot.
These techniques work best when deployed alongside broader traffic monitoring techniques. For example, rate limiting can block or throttle traffic from a single source if it exceeds normal usage patterns, while AI-powered behavioural analytics compare current traffic behaviour against historical baselines to catch subtler signs of automation.
API and credential protection
Extra security measures are also needed because bots frequently target login and authentication systems to carry out credential stuffing or brute-force attacks. One of the most effective defences here is rate limiting login attempts as it restricts how many failed attempts are allowed per IP address or account.
Multi-factor authentication also adds another layer of protection by requiring an additional verification step that reduces the likelihood that credential-stuffing attacks will succeed even if credentials are compromised.
Bot detection is an ongoing battle and no single technique is enough on its own. But as bots continue to grow more sophisticated, distinguishing between helpful automation and malicious traffic will only become harder. Machine learning is becoming increasingly important in this fight as it can sift through large datasets to uncover bots from humans.
But organisations also need to take a layered approach that combines multiple techniques, while ensuring traffic processing and verifying traffic at the network edge if they are to stop increasingly automated and AI-driven attacks.
