Undetected ChromeDriver: Bypass Anti-Bot Systems

It’s no secret one of the biggest challenges of large-scale web scraping is avoiding detection and blocking. In this article, we’ll introduce Undetected ChromeDriver, a powerful tool that helps bypass bot detection systems. 

We’ll explore how Undetected ChromeDriver works to prevent blocks and share best practices to follow if it doesn’t perform as expected. Whether you’re new to web scraping or an experienced developer, this guide will provide valuable insights to make the most of the Undetected ChromeDriver tool, allowing you to scrape data without interruptions.

What is Undetected ChromeDriver and how does it work 

Undetected ChromeDriver is an enhanced Chrome version of the standard Selenium ChromeDriver. It modifies the browser’s behavior to make it less detectable by anti-bot systems, which often block scrapers. 

While Selenium ChromeDriver can trigger detection due to its obvious signature, Selenium Undetected Chromedriver alters key elements like JavaScript execution and network requests to mask the bot’s presence. This makes it a more effective tool for evading detection and bypassing CAPTCHAs, rate limits, and IP blocks, allowing for smoother and uninterrupted web scraping processes. 

If you are using a chromedriver binary, ensure it's the appropriate version for your operating system and the version of Chrome you are running. Using the wrong version of the chromedriver binary can cause compatibility issues, leading to errors when initiating scraping tasks.

Undetected ChromeDriver installation guide

Let’s begin by installing the Undetected ChromeDriver tool:

Install Python and Pip
Ensure Python and pip (Python’s package installer) are installed. You can download Python from python.org. Then, create a virtual environment:

python -m venv env

Activate it:

• On Windows: env\Scripts\activate

• On macOS/Linux: source env/bin/activate

Afterwards, install Undetected ChromeDriver and Selenium using this command: 

pip install undetected-chromedriver selenium

Finally, test the installation: 

import undetected_chromedriver.v2 as uc
from selenium.webdriver.common.by import By

options = uc.ChromeOptions()
driver = uc.Chrome(options=options)

driver.get('https://www.example.com')
print(driver.title)
driver.quit()

This script launches Chrome, navigates to a website, and prints the title of the page. Once that’s done successfully, you’re all set to start your web scraping projects with Undetected ChromeDriver.

Using Undetected ChromeDriver: examples

Here’s a simple example to set up and run Undetected Chromedriver for web scraping:

import undetected_chromedriver.v2 as uc
from selenium.webdriver.common.by import By

# Set up Chrome options
options = uc.ChromeOptions()
options.add_argument('--headless')  # Optional: run in headless mode

# Initialize Undetected Chromedriver
driver = uc.Chrome(options=options)

# Navigate to the target website
driver.get('https://www.example.com')

# Interact with the page (example: print page title)
print(driver.title)

# Quit the driver
driver.quit()

When compared to the standard Google Chrome Driver, Undetected Chromedriver works by evading bot detection mechanisms. It modifies browser behavior, such as HTTP headers and JavaScript execution, to make the automation less detectable. 

This helps in bypassing CAPTCHAs, rate limits, and IP blocks that would typically block standard Selenium ChromeDriver requests. Additionally, it provides options to customize headers and other settings to mimic human interactions, ensuring smoother web scraping processes.  

Combining Undetected ChromeDriver and proxies 

Proxies play a crucial role in web scraping by masking your IP address and allowing you to make multiple requests without triggering detection systems. When scraping large volumes of data, websites may block your IP if they detect too many requests coming from it in a short period. 

Is Undetected Chromedriver enough?

While Undetected Chromedriver helps avoid bot detection, using proxies can further protect your web scraping efforts. If relying solely on Chromedriver isn’t enough to prevent blocks, integrating proxies into the process ensures your requests appear to come from different sources, reducing the likelihood of being blocked or flagged. Proxies also help bypass geo-restrictions, providing access to region-specific data that might otherwise be unavailable.

Implementing proxies: short tutorial

Here’s an example of using Oxylabs Residential Proxies with Undetected ChromeDriver. The general logic applies to any type of proxy you wish to use:

import undetected_chromedriver.v2 as uc
from selenium.webdriver.common.by import By

# Oxylabs Residential Proxy credentials
username = 'your_username'
password = 'your_password'
proxy = f'http://{username}:{password}@<PROXY_IP>:<PROXY_PORT>'

options = uc.ChromeOptions()
options.add_argument(f'--proxy-server={proxy}')
options.add_argument('--headless')  # Optional: run in headless mode

driver = uc.Chrome(options=options)
driver.get('https://www.example.com')
print(driver.title)
driver.quit()

Note: replace `USERNAME` and `PASSWORD` with your own user credentials, which you can generate in our dashboard.

For a more detailed guide on configuring proxies with Selenium (or other 3rd-party tools), refer to our integration guides.