Distil Networks, Inc., a provider of bot detection and mitigation, published, "The 2016 Bad Bot Landscape Report: The Rise of Advanced Persistent Bots," an annual report that identifies statistically significant data on global bot traffic. Bad bots are used by competitors, hackers, and fraudsters and are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, data theft, unauthorized vulnerability scans, spam, man-in-the-middle attacks, digital ad fraud, and downtime.
The report found 46% of all web traffic originates from bots, with over 18% percent from bad bots. For the first time since 2013, humans outnumbered bots for website traffic. Medium-sized websites (10,001 to 50,000 Alexa ranking) are at a greater risk, as bad bot traffic made up 26% of all web traffic for this group. Chrome edged out Firefox as the browser of choice for bad bot creators with over 26% of all user agents utilizing the Google browser.
Amazon has appeared in the Top 5 Bad Bot Originators three years in a row. Despite their repeated appearance in the top Bad Bot Originators list in 2013 and 2014, residential ISPs Comcast and Time Warner fell off the Top 20 bad bot originators for 2015. Six out of the top 20 ISPs with the highest percentage of bad bot traffic originated from China. US and Netherlands had the most mobile carriers, 5 and 3 respectively, on the top 20 list of bad bot mobile carriers.
Real estate websites saw a 300% increase in bad bot activity, with large real estate sites experiencing the most pain. As an industry, digital publishers were hit hardest by bad bots, which make up over 31% of all their traffic. For small digital publishers (Alexa 50,001 - 150,000) 56% of traffic originates from bad bots.
Maldives, Israel and Kyrgyzstan had the highest bad bot GDP (number of bad bots per online user) at 526, 168, and 94 respectively. China, Norway, Germany, and the Netherlands are the most blocked countries for web traffic. The United States boasts the largest originator of bots again, with over 39% of bot traffic, while India and Israel moved up to number two and three, respectively.
The 2016 Bad Bot Landscape Report is based on aggregate data gathered from Distil Networks' bot detection and mitigation solution that identifies and tracks bots in real time, the world's largest Known Violators Database of bad bot fingerprints, as well as Distil's global network of 17 data centers.