How to Bypass CAPTCHA With Playwright

CAPTCHAs (Completely Automated Public Turing tests to tell Computers and Humans Apart) have become vital to website security. Once the security apparatus of the website becomes suspicious of access (e.g., the access pattern does not follow normal human behavior), it loads a CAPTCHA (e.g., reCAPTCHA, sound, and image puzzles), preventing bots from further access.

Bypassing a CAPTCHA challenge once it loads, can be extremely difficult. However, there are a few methods by which your script can exhibit more human behavior to the web firewall. Thereby, you can completely prevent CAPTCHA from loading. We call this bypassing, or avoiding, a CAPTCHA.

This step-by-step tutorial demonstrates how to use Playwright to bypass CAPTCHA challenges using Python. The tutorial will also discuss the perks of using Oxylabs’ Web Unblocker instead of the `playwright-stealth` library. 

Note: Bypassing CAPTCHAs for illegal or malicious motives violates ethical and legal standards. This tutorial is for educational purposes only, and we encourage readers to thoroughly read the Terms of Services of the target website to avoid legal issues.

The screenshot shows the actual content of the page, which means we just bypassed the CAPTCHA or reCAPTCHA from loading on this page. 

Bypass CAPTCHA with Web Unblocker

Oxylabs’ Web Unblocker employs advanced AI techniques to help users access publicly available information behind the CAPTCHA. Bypassing CAPTCHAs with our advanced proxy solution is easy. You just need to send a simple query. Web Unblocker will automatically choose the fastest CAPTCHA proxy, attach all essential headers, and return the response HTML bypassing any anti-bots of the target websites.

Here are the steps you must follow to implement a simple web scraping request using Web Unblocker. 

1. Create an account

You can create an account on the dashboard with a 7-day free trial. 

2. Create API key

After successfully creating your account, you can set your API key username and password from the dashboard. These API key credentials will be used later in the code.

3. Install the requests module

You should use a library that can help perform HTTP requests. We will use the `requests` to send HTTP requests to Web  Unblocker API and capture the response.

pip install requests

4. Import the required modules

In your Python script file, import the modules using the following import statement:

import requests

5. Define proxy and headers 

Create the proxies dictionary to connect to Web Unblocker and then define the headers dictionary that’ll instruct Web Unblocker to use JavaScript rendering. See the documentation for more details. 

# Define proxy dict. Don't forget to pass your Web Unblocker credentials (username and password)
proxies = {
   "http": "http://USERNAME:PASSWORD@unblock.oxylabs.io:60000",
   "https": "http://USERNAME:PASSWORD@unblock.oxylabs.io:60000",
}

headers = {
    "X-Oxylabs-Render": "html"
}

6. Make a request

Perform your request by specifying the URL, request type, and proxy by using the following code.

response = requests.request(
   "GET",
   "http://sandbox.oxylabs.io/products",
   verify=False,  # Ignore the certificate
   proxies=proxies,
)

7. Save the response

Write the following code to print the response and save it in an HTML file.

# Print result page to stdout
print(response.text)

# Save returned HTML to result.html file
with open("result.html", "w") as f:
   f.write(response.text)

8. Execute and check

Execute the code and test the output. If the output HTML  file has actual page contents, the script successfully bypassed the CAPTCHA. Here is what our complete code looks like.

# Import the modules
import requests

# Define proxy dict. Don't forget to put your real user and pass here as well.
proxies = {
   "http": "http://USERNAME:PASSWORD@unblock.oxylabs.io:60000",
   "https": "http://USERNAME:PASSWORD@unblock.oxylabs.io:60000",
}

headers = {
    "X-Oxylabs-Render": "html"
}

response = requests.request(
   "GET",
   "http://sandbox.oxylabs.io/products",
   verify=False,  # Ignore the certificate
   proxies=proxies,
   headers=headers,
)

# Print result page to stdout
print(response.text)

# Save returned HTML to result.html file
with open("result.html", "w") as f:
   f.write(response.text)

Here is the snapshot of the output HTML displayed on the screen:

Here is a part of a snapshot of how the browser renders this HTML:

Visit this page on our documentation to see how you can save the PNG file. The above snapshot makes it clear that we accessed the products page without any blocks.

Conclusion

Playwright, when combined with the `playwright-stealth` package, can effectively be used to scrape content behind the sites with ordinary CAPTCHA protection. Learn more about how to perform web scraping with Playwright, configure Playwright with proxies, and combine Scrapy with Playwright in our blog posts. If you're still wondering which proxies fit your needs best, get a free trial for our premium proxies to make the right decision.

However, bypassing CAPTCHA (e.g., reCAPTCHA) from websites with advanced anti-bots requires a more sophisticated and intelligent bypassing solution. Oxylabs’s Web Unblocker automatically combines the latest AI techniques with bypassing schemes (e.g., proxies and IP rotation, setting realistic fingerprints, and JS rendering) to ditch advanced anti-bots. Therefore, it is a more secure, convenient, and reliable solution for bypassing CAPTCHAs and scraping data at scale.