Proxy locations

Europe

North America

South America

Asia

Africa

Oceania

See all locations

Network statusCareers

Back to blog

How to Bypass CAPTCHA With Selenium & Python

How to Bypass CAPTCHA With Selenium & Python
Yelyzaveta Nechytailo

Yelyzaveta Nechytailo

2023-10-063 min read
Share

In this tutorial, you’ll learn how to handle CAPTCHA in Selenium and Python.  You’ll install and combine the undetected-chromedriver with Selenium to enable scraping complex websites with anti-bot protection such as Cloudflare, CAPTCHA, etc. Additionally, you’ll explore Oxylabs’ Web Unblocker, which will simplify your scraping scripts and increase scalability. Let’s get started. 

Bypass CAPTCHA with Selenium and Python

The first step is to install Python if you haven't installed it already. You can download it from the official website. Download the latest version or a version greater than 3.6; otherwise, undetected-chromedriver won’t work properly.

Step 1 - Install dependencies

Install the undetected-chromedriver and requests module. You can use the pip command given below:

pip install undetected-chromedriver requests

Step 2 - Import libraries

Now that you’ve installed undetected-chromedriver, you can import it as shown below:

import undetected_chromedriver as webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--use_subprocess")

browser = webdriver.Chrome(options=chrome_options)
Link to GitHub

Notice, you’ve also created a browser instance. This will open a Chrome window in the background in headless mode. 

Step 3 - Navigate to webpage

Use the browser instance to navigate to your target website. For this tutorial, let’s use https://sandbox.oxylabs.io/products as the target. 

browser.get("https://sandbox.oxylabs.io/products")

Step 4 - Take a screenshot

Take a screenshot to verify the page is loaded properly without showing any CAPTCHA or bot protection screen. You can use the save_screenshot method of Selenium.

browser.save_screenshot("screenshot.png")

Your screenshot might vary slightly due to screen size, but it’ll look similar to the one given below:

screenshot

The page has loaded properly without showing any CAPTCHA and the undetected-chromedriver has rendered the Javascript files.

Key things to remember

  • The undetected-chromedriver doesn’t hide your IP address. So, if your IP address has a low reputation score, you’ll get a CAPTCHA no matter what you do. 

  • Some anti-bot protection services block headless browsers by default. In that case, you’ll have to set headless=False to avoid getting blocked.

  • Without a rotating proxy, this method is unreliable for large-volume scraping. As the IP always remains the same, the anti-bot protection services can pick up the extra volume of requests from the particular IP address and block the requests by IP immediately.

  • Configuring proxy with authentication is tricky with undetected-chromedriver as it uses the chromedriver under the hood. The chromedriver doesn’t support proxy authentication. However, you can install and use the seleniumwire extension to set up authenticated proxies with undetected-chromedriver.  

Bypass CAPTCHA with Web Unblocker

To perform large-scale web scraping while bypassing CAPTCHA, you’ll need a strong tool. Web Unblocker, an AI–powered proxy solution for bypassing IP blocks and CAPTCHAs, will automatically rotate proxies for you, so you don’t have to worry about manually managing a list of proxies for your bots.

Step 1 - Import libraries

Let’s use the requests module to set up Web Unblocker. 

import requests

Step 2 - Get Web Unblocker credentials 

Create an account to get the Web Unblocker credentials. Within a few clicks, you can sign up and get a 1-week free trial to develop and thoroughly test the solution.

Step 3 - Prepare Web Unblocker 

Web Unblocker’s host and port are unblock.oxylabs.io and 60000 respectively. Additionally, don’t forget to replace the USERNAME and PASSWORD with the correct credentials.

proxy = 'http://{}:{}@unblock.oxylabs.io:60000'.format("USERNAME", "PASSWORD")
proxies = {
    'http': proxy,
    'https': proxy
}
Link to GitHub

If you get any authentication-related errors in the later steps, don’t forget to check the Web Unblocker response codes here.

Step 4 - Fetch content

Now, you can use the proxies dict you created with the get method of the requests module. Web Unblocker also requires you to pass an extra parameter, verify=False, to the get method.  

page = "https://sandbox.oxylabs.io/products"
response = requests.get(page, proxies=proxies, verify=False)
print(response.status_code)
content = response.content
Link to GitHub

You should see the status code 200 if everything works as expected. The content of the page will be stored in the content object, which you can process later with HTML Parser libraries such as Beautiful Soup or parse using the Custom Parser. Web Unblocker also renders JavaScript for you, so you can use this method for dynamic websites as well. 

Conclusion

In this tutorial, we demonstrated how to handle CAPTCHA in Selenium and Python using undetected-chromedriver. We also explored Oxylabs’ Web Unblocker, which makes it more convenient to develop complex scrapers with the ease and comfort of simple code. 

Learn more about Selenium web scraping as well as see how to integrate it with Oxylabs’ proxies. And in case you’d like to perform public web data gathering from targets like Walmart and Amazon, check out our ready-to-use scraping solutions, such as Amazon Scraper API

Frequently asked questions

Can Selenium bypass CAPTCHA?

Most modern anti-bot systems of complex websites have advanced fingerprinting methods that can easily detect Selenium-based web scrapers. So, bypassing CAPTCHA with Selenium without any extra tools or plugins is extremely hard. Different anti-bot protection services use different fingerprinting methods, so the complex changes you have to make vary from website to website. More importantly, they make frequent changes to the detection algorithm, making manual maintenance extremely complex.

How to avoid CAPTCHA detection in Selenium?

By default, Selenium is prone to bot detection algorithms; however, by using stealth web drivers and third-party plugins, you can evade CAPTCHA detection to some extent. It can get quite tricky, especially for websites with highly advanced anti-bot protections. You’ll also need a high-quality proxy solution, such as Web Unblocker, to upscale your bots. Although it is not necessary to use Selenium, you can combine it with the seleniumwire extension to make the undetected-chromedriver more stealthy.

About the author

Yelyzaveta Nechytailo

Yelyzaveta Nechytailo

Senior Content Manager

Yelyzaveta Nechytailo is a Senior Content Manager at Oxylabs. After working as a writer in fashion, e-commerce, and media, she decided to switch her career path and immerse in the fascinating world of tech. And believe it or not, she absolutely loves it! On weekends, you’ll probably find Yelyzaveta enjoying a cup of matcha at a cozy coffee shop, scrolling through social media, or binge-watching investigative TV series.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Get the latest news from data gathering world

I’m interested