How to Bypass CAPTCHA in Web Scraping Using Python

What are the different types of CAPTCHAs?

Generally, there are three CAPTCHA types: text-based, image-based, and sound-based.

Text-based CAPTCHA

It’s usually a combination of random letters and characters presented in a hard-to-read format, with characters being turned, scaled, and distorted in various ways.

Image-based CAPTCHA 

Image-based CAPTCHAs usually display several pictures in a grid and ask the user to select a specific type of image. For instance, images with traffic lights.

Sound-based CAPTCHA

Also known as audio-based CAPTCHAs, these tests present audio clips with a combination of letters or numbers that users have to enter, often accompanied by background noise for added difficulty.

hCAPTCHA 

It’s a CAPTCHA service that clients can set up on their website. Serving as an alternative to reCAPTCHA, it offers better privacy and provides more control over the CAPTCHA experience.

Google reCAPTCHA 

It’s a free CAPTCHA service developed by Google that offers protection for web pages. Just like hCAPTCHA, it uses advanced techniques to catch bot-like activity. One such technique is that Google reCAPTCHA now can even recognize human users without any interaction on its side – they simply take into account the user’s previous interactions with other websites, which might be an undesirable approach due to privacy issues.

Google reCAPTCHA is also widely used in most of the brand’s services and products, such as Google Search, Maps, Play, Shopping, and many more.

3. Set up the Web Unblocker endpoint

Start by importing the installed Python libraries:

import requests, json
from bs4 import BeautifulSoup

Next, create the web_unblocker proxy dictionary object to connect to the Web Unblocker endpoint via HTTPS and use your sub-user’s credentials for authentication:

USERNAME, PASSWORD = "YOUR_USERNAME", "YOUR_PASSWORD"
web_unblocker = {
    "http": f"http://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000",
    "https": f"https://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000",
}

Additionally, create a web_unblocker_headers dictionary to set the geo-location to New York, United States, using headers:

web_unblocker_headers = {
    # Localize results for New York.
    "x-oxylabs-geo-location": "New York,New York,United States"
}

4. Send a request to the target

The next step is to send a GET request to the target website through Web Unblocker. Let’s also mimic the previous code sample by creating the num list that defines the starting positions for each Google search results page, as well as the keywords list to scrape search results for:

num = [i for i in range(0, 50, 10)] # '50' represents 5 SERP pages.
keywords = ["shoes", "coats", "jeans", "sunglasses", "hats"]

for keyword in keywords:
    results = []
    for page in num:
        response = requests.get(
            f"https://www.google.com/search?q={keyword}&start={page}",
            verify=False,  # Ignore the SSL certificate.
            proxies=web_unblocker,
            headers=web_unblocker_headers,
            timeout=180
        )

Web Unblocker requires users to ignore the SSL certificate, which is done by adding verify=False within the GET request. Then, include the proxies argument and pass the web_unblocker object to forward web requests through the Web Unblocker endpoint.

5. Parse the desired data

Here, you can utilize the Beautiful Soup library to extract the content from the target page. First, create the soup object, which will store the HTML content:

        soup = BeautifulSoup(response.text, "html.parser")

Next, define the search results items selector:

        items = soup.select("div.yuRUbf")

Then, create a third for loop to extract all the titles and links:

        for item in items:
            title = item.select_one("h3")
            link = item.select_one("a")
            if title and link:
                results.append({
                    "title": title.get_text(),
                    "link": link.get("href")
                })

The soup.select uses CSS expressions to select all the <div> elements with a class “yuRUbf”, while the soup.select_one selects the first matching <h3> title and <a> link. Since all the item URLs are stored as a value of the href attribute, you can retrieve the complete link using the .get() function. If you’re interested in learning web scraping, check out our in-depth blog posts on Python Web Scraping and Beautiful Soup to get an easy start.

6. Store the results in a JSON file

Once all the titles and links are extracted from all the pages of a specific keyword, you can save the results to a JSON file:

    with open(f"google_{keyword}.json", "w") as f:
        json.dump(results, f, indent=4)

Full Web Unblocker code sample

The complete code should look like this:

import requests, json
from bs4 import BeautifulSoup


USERNAME, PASSWORD = "YOUR_USERNAME", "YOUR_PASSWORD"
web_unblocker = {
    "http": f"http://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000",
    "https": f"https://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000",
}

web_unblocker_headers = {
    # Localize results for New York.
    "x-oxylabs-geo-location": "New York,New York,United States"
}

num = [i for i in range(0, 50, 10)] # '50' represents 5 SERP pages.
keywords = ["shoes", "coats", "jeans", "sunglasses", "hats"]

for keyword in keywords:
    results = []
    for page in num:
        response = requests.get(
            f"https://www.google.com/search?q={keyword}&start={page}",
            verify=False,  # Ignore the SSL certificate.
            proxies=web_unblocker,
            headers=web_unblocker_headers,
            timeout=180
        )
        soup = BeautifulSoup(response.text, "html.parser")

        items = soup.select("div.yuRUbf")

        for item in items:
            title = item.select_one("h3")
            link = item.select_one("a")
            if title and link:
                results.append({
                    "title": title.get_text(),
                    "link": link.get("href")
                })

    with open(f"google_{keyword}.json", "w") as f:
        json.dump(results, f, indent=4)

As you can see, it only takes a few lines of Python code to incorporate Oxylabs’ Web Unblocker. Using the above code, you should expect the following output:

[
    {
        "title": "Women's, Men's & Kids Shoes from Top Brands | DSW",
        "link": "https://www.dsw.com/"
    },
    {
        "title": "Shoes for Women, Men & Kids, Famous Footwear",
        "link": "https://www.famousfootwear.com/"
    },
    {
        "title": "Rack Room Shoes: Shoes Online with Free Shipping*",
        "link": "https://www.rackroomshoes.com/"
    },
    {
        "title": "Men's Shoes & Sneakers",
        "link": "https://www.nike.com/w/mens-shoes-nik1zy7ok"
    },
    {
        "title": "Shoes + FREE SHIPPING",
        "link": "https://www.zappos.com/shoes"
    },
    {
        "title": "Mens shoes, boots and sandals",
        "link": "https://www.nativeshoes.com/mens?srsltid=AfmBOopqVwHL2WurGKZeX7SE94nMSE7HrgJfMUGRUf3l-Ay1vWW1zYiw"
    },
    {
        "title": "Women's Shoes",
        "link": "https://www.nordstrom.com/browse/women/shoes?srsltid=AfmBOopl7t_F-BKJ7LXMFuCayHTbxbC1Hh3eTYAYhpUbdGUSrYvPcAT8"
    },
    {
        "title": "DC Shoes\u00ae Skate Shoes & Snowboard Boots",
        "link": "https://www.dcshoes.com/"
    }
]

Hopefully, these Python examples helped you see how effortless the integration process of Web Unblocker is. Visit our documentation to learn more about its parameters and general integration steps.

Developing your own solution

Of course, it’s always possible to create your own solution that takes care of various CAPTCHAs. While the development stage may take some time, you can tailor it specifically to the kind of requests you wish to send. This can result in higher success rates, allowing you to perform web scraping activities without interruptions. There are a couple of viable Python CAPTCHA bypass tools for this quest:

Playwright

Playwright is an excellent web testing and automation tool developed by Microsoft, which can also be used to bypass CAPTCHAs. It supports the most popular programming languages, such as Python, JavaScript, and Java. Playwright can work with Chromium-based, Firefox, and WebKit browsers, allowing the users more flexibility. We’ve made a detailed blog post specifically on Playwright, so be sure to take a look at this Playwright Scraping Tutorial for more information.

Puppeteer

It’s also a very effective web automation tool that you can use to design a program that avoids CAPTCHAs. While Puppeteer, owned by Google, supports only JavaScript, you can use it in Python with an unofficial library called Pyppeteer. The downside of Puppeteer is that it only supports Chromium-based browsers for interaction. If you’re curious to learn more, check out our in-depth blog post on web scraping with Puppeteer and a tutorial on how to overcome CAPTCHA challenges with Puppeteer.

Selenium

Selenium is another popular web automation framework that’s also used for web scraping and overcoming CAPTCHAs. It’s a powerful headless browser that supports automation of Chrome, Firefox, Safari, Edge, and Internet Explorer compatibility mode using Edge. Check out this blog post on bypassing CAPTCHAs with Selenium and this comprehensive tutorial on web scraping using Selenium.

Keep in mind that developing your own solution will require you to spend time writing code and micromanaging it to adapt to constant changes. In cases where this is an issue, the better option is to utilize ready-made web scrapers that avoid CAPTCHAs automatically. It takes a mountain of effort to build yourself a scalable scraper that sifts through the web undetected and uninterrupted, but a pre-built tool can ease the process immensely, saving time and resources. See how both methods differ in this guide to scraping Amazon. If you aim to scrape Amazon, you may find it helpful to follow this guide on how to bypass Amazon CAPTCHAs.

Final thoughts

With CAPTCHAs being one of the most common challenges when it comes to public data collection, it’s essential to find a reliable and high-quality solution to bypass them. This article presented a few anti-CAPTCHA solutions you can try implementing in your scraping tasks, and it discussed the different types of CAPTCHAs available today. 

If you're curious to try out our scraping solutions, you can simply get a free trial and follow our guides for your desired target. Here are some tutorials to get you started: how to scrape Google search results and how to scrape Etsy data.

If you have any questions about this topic or would like to learn more about Web Unblocker, Oxylabs’ ultimate solution for bypassing CAPTCHAs, feel free to contact us at hello@oxylabs.io or via the live chat.