How to Detect Bad Bots and How it Affects Web Scraping?
avatar

Gabija Fatenaite

Jan 08, 2021 6 min read

Often we perceive the term “bot” as negative. However, not all bots are bad. The issue is that good bots can share similar characteristics with malicious bots. Therefore, good bot traffic get labeled as bad and get blocked. 

Bad bots are only getting smarter, and it’s hard for other bots to stay block-free. This creates a lot of issues not only for site owners to ensure a healthy performance of their website but for the web scraping community as well. 

While we have already covered what is a bot, in this article, we’ll go more in-depth about bot traffic, how websites detect and block bots, and how it can affect businesses. 

What is bot traffic?

Bot traffic is any non-human traffic made to a website. It’s a software application running automated and repetitive tasks; however, much faster than humanly possible. 

With this ability to perform tasks very quickly, bots can be used for both bad and good. In 2020, 24.1% of bot traffic online were malicious bad bots. That’s +18.1% more than the previous year of 2019. 

Bad bot vs. good bot vs. human traffic 2020

Bad bot vs. good bot vs. human traffic 2020

Whereas good bot traffic is also decreasing (compared to 2019, the numbers dropped by -25.1%). With the increase of bad bots and the decrease of good bots, website owners are forced to strengthen their security. Hence allowing more bots to get wrongfully caught. 

To better understand what are good and bad bots, here are some examples: 

Good bots

  • Search engine bots – these bots crawl, catalog, and index web pages. Such results are used by search engines such as Google to provide their services effectively. 
  • Site monitoring bots – will monitor websites to identify possible issues such as long loading times, downtimes, etc.
  • Web scraping bots – if the data being scraped is publicly available, the data can be used for research, identifying and pulling down illegal ads, brand monitoring, and much more. 

Bad bots

  • Spam bots – used for spam purposes. Often for the purpose of creating fake accounts on forums, social media platforms, messaging apps, and so on. They are used in order to build a social media presence, create more clicks on a post, etc.
  • DDoS attack bots – some bots are created to take down websites. DDoS attacks usually leave just enough bandwidth available to allow other attacks to make their way into the network and pass weakened network security layers undetected to steal sensitive information. 
  • Ad fraud bots – these bots automatically click on ads siphoning off money from advertising transactions.

So a “good” bot is a bot that performs useful or helpful tasks that aren’t detrimental to a user’s experience on the Internet. Whereas a bad bot is the exact opposite and in most cases has malicious or even illegal intentions.

How can bot traffic be identified?

To prevent bad bot traffic, websites have created various bot detection techniques. Here are several ways they do that:

  • Browser fingerprinting – this refers to information that is gathered about a computing device for identification purposes (any browser will pass on specific data points to the connected website’s servers such as your operating system, language, plugins, fonts, hardware, etc.) Learn more about what is browser fingerprinting in our in-depth blog. 
  • Browser consistency – checking the presence of specific features that should or should not be in a browser. This can be done by executing certain JavaScript requests.
  • Behavioral inconsistencies – nonlinear mouse movements, rapid button and mouse clicks, repetitive patterns, average page time, average requests per page, and similar, bot-like behavior.
  • CAPTCHA – a popular anti-bot measure are CAPTCHAs – a challenge-response type of test that often asks you to fill in correct codes or identify objects in pictures. You can read up more on how CAPTCHAS work in our blog.

Once a website identifies bot-like behavior, it blocks them from further crawling. For more details, Dmitry Babitsky, the co-founder & chief scientist at ForNova, has spoken in-depth on how websites block bots in his presentation at OxyCon. 

Bot detection challenges

Distinguishing bot traffic from human behavior online has become a complex task in itself, and the bots on the internet have evolved dramatically over the years. Currently, there are four different generations of bots:

  • First-generation – these bots are built with basic scripting tools and mainly perform basic automated tasks like scraping, spam, etc.
  • Second-generation – mainly operate through website development, hence ending up with the name ‘web crawlers.’ They are relatively easy to detect due to the presence of specific JavaScript firing and iframe tampering.
  • Third-generation – often used for slow DDoS attacks, identity thefts, API abuse, and others. They are relatively difficult to detect based on device and browser characteristics and would require proper behavioral and interaction-based analysis to identify.
  • Fourth-generation – the newest iteration of bots. Such bots can perform human-like interactions like nonlinear mouse movements. In order to detect such bots, advanced methods, often involving the use of AI and machine learning technologies are required.

The fourth generation of bots are tough to differentiate from legitimate human users, and basic bot detection technologies are no longer sufficient. For such bot traffic to be detected, it will take a lot more than simple tools and behavioral interaction analysis. 

Overcoming anti-bot measures

If you want a step by step guide on how to crawl a website without getting blocked by the anti-bot measures, we have written in great detail on how to do exactly that. In that blog post, we provide you with a list of actions to prevent getting blacklisted while scraping and crawling websites. However, if you would like a faster and less  labor-intensive method, you could check out Next-Gen Residential Proxies as a solution.

It is an AI & ML powered solution that has an AI-powered dynamic fingerprinting feature, this allows Next-Gen Residential Proxies to overcome browser fingerprinting, keeping you block-free. With this feature a user will be able to imitate a regular user’s behavior, ensuring a 100% success rate against bot traffic detection.

Conclusion

Bad bot traffic is predicted only to increase each year. As for good bot traffic, the chance to not get mixed in with the bad crowd is slowly dwindling. Amongst good bots there are a lot of web scrapers that use gathered data for research, pulling down illegal ads, market research, etc. All of them may get flagged as bad and blocked.

Fortunately, solutions implementing AI and ML technologies are being built to overcome false bot blocks. Learn more about AI and ML technologies used to improve scraping by reading our blog.

avatar

About Gabija Fatenaite

Gabija Fatenaite is a Product Marketing Manager at Oxylabs. Having grown up on video games and the internet, she grew to find the tech side of things more and more interesting over the years. So if you ever find yourself wanting to learn more about proxies (or video games), feel free to contact her - she’ll be more than happy to answer you.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Related articles

ML-Based Adaptive Parser Is Now In Production

ML-Based Adaptive Parser Is Now In Production

Jul 23, 2021

3 min read

What Is a Bot and How Does It Work?

What Is a Bot and How Does It Work?

Jul 13, 2021

7 min read