Advanced Persistent Bot Activity on the Rise Despite Overall Decrease in Bad Bot Activity

Mar 29, 2016

Distil Networks, Inc., a provider of bot detection and mitigation, published, "The 2016 Bad Bot Landscape Report: The Rise of Advanced Persistent Bots," an annual report that identifies statistically significant data on global bot traffic. Bad bots are used by competitors, hackers, and fraudsters and are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, data theft, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime. 

The report found 46% of all web traffic originates from bots, with over 18% percent from bad bots. For the first time since 2013, humans outnumbered bots for website traffic. Medium-sized websites (10,001 to 50,000 Alexa ranking) are at a greater risk, as bad bot traffic made up 26% of all web traffic for this group. Chrome edged out Firefox as the browser of choice for bad bot creators with over 26% of all user agents utilizing the Google browser.

Additionally, 88% of all bad bot traffic has one or more characteristics of an Advanced Persistent Bot. over half (53%) of bad bots are now able to load external resources like JavaScript meaning these bots will end up falsely attributed as humans in Google analytics and other tools. And 39% of bad bots are able to mimic human behavior so tools such as WAFs, web log analysis, or Firewalls, which perform less detailed analysis of clients and their behavior, will likely result in huge amounts of false negatives. An additional 36% of bad bots disguise themselves using two or more user agents, and the worst APBs change their identities over 100 times. And 73% of bad bots rotate or distribute their attacks over multiple IP addresses and of those, a whopping 20% surpassed 100 IP addresses.

Amazon has appeared in the Top 5 Bad Bot Originators three years in a row. Despite their repeated appearance in the top Bad Bot Originators list in 2013 and 2014, residential ISPs Comcast and Time Warner fell off the Top 20 bad bot originators for 2015. Six out of the top 20 ISPs with the highest percentage of bad bot traffic originated from China. US and Netherlands had the most mobile carriers, 5 and 3 respectively, on the top 20 list of bad bot mobile carriers.

Real estate websites saw a 300% increase in bad bot activity, with large real estate sites experiencing the most pain. As an industry, digital publishers were hit hardest by bad bots, which make up over 31% of all their traffic. For small digital publishers (Alexa 50,001 - 150,000) 56% of traffic originates from bad bots.

Maldives, Israel and Kyrgyzstan had the highest bad bot GDP (number of bad bots per online user) at 526, 168, and 94 respectively. China, Norway, Germany, and the Netherlands are the most blocked countries for web traffic. The United States boasts the largest originator of bots again, with over 39% of bot traffic, while India and Israel moved up to number two and three, respectively.

The 2016 Bad Bot Landscape Report is based on aggregate data gathered from Distil Networks' bot detection and mitigation solution that identifies and tracks bots in real time, the world's largest Known Violators Database of bad bot fingerprints, as well as Distil's global network of 17 data centers.