How to Bypass CAPTCHA With Selenium & Python

In this tutorial, you’ll learn more about Selenium CAPTCHA bypass.  You’ll install and combine the undetected-chromedriver with Selenium to enable scraping complex websites with anti-bot protection such as Cloudflare, CAPTCHA, etc. Additionally, you’ll explore Oxylabs’ Web Unblocker, which will simplify your scraping scripts and increase scalability. So, let’s get started with developing your own Selenium CAPTCHA solver.

How to bypass CAPTCHA human verification with Selenium and Python

The first step in the Python CAPTCHA bypass is to install Python if you haven't installed it already. You can download it from the official website. Download the latest version or a version greater than 3.6; otherwise, undetected-chromedriver won’t work properly. To learn more about the types of CAPTCHAs, check out these general blog posts about how do CAPTCHAs work and bypassing CAPTCHAs in web scraping.

Step 1 - Install dependencies

Install the undetected-chromedriver and requests module. You can use the pip command given below:

pip install undetected-chromedriver requests

Step 2 - Import libraries

Now that you’ve installed undetected-chromedriver, you can import it as shown below:

import undetected_chromedriver as webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--use_subprocess")

browser = webdriver.Chrome(options=chrome_options)

Notice, you’ve also created a browser instance. This will open a Chrome window in the background in headless mode. 

Step 3 - Navigate to webpage

Use the browser Chrome instance to navigate to your target website. For this tutorial, let’s use https://sandbox.oxylabs.io/products as the target url. 

browser.get("https://sandbox.oxylabs.io/products")

Step 4 - Take a screenshot

Take a screenshot to verify user interactions on the web page are loaded properly without showing any CAPTCHA screen or bot protection screen. You can use the save_screenshot method of Selenium.

browser.save_screenshot("screenshot.png")

Your screenshot might vary slightly due to screen size, but it’ll look similar to the one given below:

The page has loaded properly without showing any CAPTCHA box and the undetected-chromedriver has rendered the Javascript files.

Key things to remember

• The undetected-chromedriver doesn’t hide your IP address. So, if your IP address has a low reputation score, you’ll get a CAPTCHA no matter what you do. 

• Some anti-bot protection services block headless browsers by default. In that case, you’ll have to set headless=False to avoid getting blocked.

• Without a rotating proxy, this Selenium CAPTCHA method is unreliable for large-volume scraping. As the IP always remains the same, the anti-bot protection services can pick up the extra volume of requests from the particular IP address and block the requests by IP immediately.

• Configuring proxy with authentication is tricky with undetected-chromedriver as it uses the chromedriver under the hood. The chromedriver doesn’t support proxy authentication. However, you can install and use the seleniumwire extension to set up authenticated proxies with undetected-chromedriver.  

Bypass CAPTCHA with Web Unblocker

To perform large-scale web scraping while bypassing CAPTCHA, you’ll need a strong tool. Web Unblocker, an AI-powered proxy solution for bypassing IP blocks and CAPTCHAs, will automatically rotate proxies for you to avoid CAPTCHAs, so you don’t have to worry about manually managing a list of proxies for your bots.

Step 1 - Import libraries

Let’s use the requests module to set up Web Unblocker. 

import requests

Step 2 - Get Web Unblocker credentials 

Create an account to get the Web Unblocker credentials. Within a few clicks, you can sign up and get a 1-week free trial to develop and thoroughly test the solution.

Step 3 - Prepare Web Unblocker 

Web Unblocker’s host and port are unblock.oxylabs.io and 60000 respectively. Additionally, don’t forget to replace the USERNAME and PASSWORD with the correct credentials.

proxy = 'http://{}:{}@unblock.oxylabs.io:60000'.format("USERNAME", "PASSWORD")
proxies = {
    'http': proxy,
    'https': proxy
}

If you get any authentication-related errors in the later steps, don’t forget to check the Web Unblocker response codes here.

Step 4 - Fetch content

Now, you can use the proxies dict you created with the get method of the requests module. Web Unblocker also requires you to pass an extra parameter, verify=False, to the get method.  

page = "https://sandbox.oxylabs.io/products"
response = requests.get(page, proxies=proxies, verify=False)
print(response.status_code)
content = response.text

You should see the status code 200 if everything works as expected. The content of the page will be stored in the text object, which you can process later with HTML Parser libraries such as Beautiful Soup or parse using the Custom Parser. Web Unblocker also renders JavaScript for you, so you can use this method for dynamic websites as well. 

Bypass CAPTCHA with proxies

Implementing proxies is another popular and effective way to reduce blocks and avoid triggering anti-bot systems when web scraping. CAPTCHAs are usually triggered when a server detects suspicious bot activity from a particular IP address, such as sending too many requests in a short period of time. By using proxies to rotate your IP address, you can appear as multiple organic users, reducing the likelihood of CAPTCHA challenges.

There are several types of proxies you can use, each with its advantages:

Residential Proxies: Route requests through real user devices. Excellent for CAPTCHA avoidance due to high legitimacy.

Datacenter Proxies: Fast and affordable, but easier for websites to detect and block.

Mobile Proxies: Offer IPs from mobile carriers, making them highly anonymous and ideal for accessing mobile-first targets.

Here’s a basic code sample of using Oxylabs’ Residential Proxies for CAPTCHA bypass:

import requests

# Replace USERNAME and PASSWORD with your Oxylabs proxy user credentials
proxy = 'http://USERNAME:PASSWORD@pr.oxylabs.io:7777'
proxies = {
    'http': proxy,
    'https': proxy
}

url = "https://sandbox.oxylabs.io/products"

try:
    response = requests.get(url, proxies=proxies, timeout=10)
    if response.status_code == 200:
        print("Page loaded successfully")
        print(response.text[:500])  # Print first 500 characters of the page
    else:
        print("Failed to load page. Status code:", response.status_code)
except Exception as e:
    print("An error occurred:", str(e))

Conclusion

In this tutorial, we demonstrated how to handle CAPTCHA in Selenium Python using undetected-chromedriver. We also explored Oxylabs’ Web Unblocker, which makes it more convenient to develop complex scrapers with the ease and comfort of simple code. 

Learn more about Selenium web scraping as well as see how to integrate it with Oxylabs’ proxies. And in case you’d like to perform public web data gathering from targets like Walmart and Amazon, check out our ready-to-use scraping solutions, such as Amazon Scraper API. You can see the scraper in action in this bypass Amazon CAPTCHA blog post.

If you want to utilize a headless browser other than Selenium, we also have posts showcasing how to bypass CAPTCHAs with Playwright and bypass CAPTCHAs with Puppeteer.