How to Bypass CAPTCHA With Puppeteer

Automated web scraping and crawling are crucial for gathering and analyzing data from websites at large scale. However, anti-bot technologies like CAPTCHA have made automated web access more challenging. 

Many websites often load CAPTCHAs or block screens as a security measure. If your automated scraper can somehow appear human to the target website, likely, the site will not load a CAPTCHA or block screen. Thereby, your scraper can bypass CAPTCHA and reCAPTCHA challenge and perform the scraping activities.

But how can a scraper appear human to the websites?  Let’s find out.

Tutorial: Bypassing CAPTCHA with Puppeteer

To access content from the protected websites, you need to figure out how to prevent CAPTCHA from loading. Puppeteer can help us here. It is a Node.JS library that provides a user-friendly API for managing Chrome and Chromium via the DevTools Protocol. You can configure Puppeteer to operate in full Chrome/Chromium mode instead of the default headless mode.

Why is Puppeteer alone not sufficient?

What happens when you try automated access to a CAPTCHA-protected website using the Puppeteer alone? The target website detects that the access is automated and shows you a block screen or CAPTCHA test.

Let’s validate it using the following steps:

1. You must have Node.JS installed on your system. Create a new Node.JS project and install Puppeteer using the following npm command:

npm i puppeteer

2. Import the Puppeteer library in your Node.JS file.

const puppeteer = require('puppeteer');

3. Create a new browser instance in headless mode and a new page using the following code:

(async () => {
  // Create a browser instance
  const browserObj = await puppeteer.launch();

  // Create a new page
  const newpage = await browserObj.newPage();

4. Since we need to take the screenshot on the desktop device, we can set the viewport size using the following code:

  // Set the width and height of viewport
  await newpage.setViewport({ width: 1920, height: 1080 });

The setViewPort() method sets the size of the webpage. You can change it according to your device requirements. 

5. After that, navigate to a page URL (that you think is a CAPTCHA-protected page) and take a screenshot. For demonstration purposes, the code uses Oxylabs scraping sandbox. Remember to close the browser object at the end.

  const url = 'https://sandbox.oxylabs.io/products';

  //  Open the required URL in the newpage object
  await newpage.goto(url);
  await newpage.waitForNetworkIdle(); // Wait for network resources to fully load

  // Capture screenshot
  await newpage.screenshot({
    path: 'screenshot.png',
  });

  // Close the browser object
  await browserObj.close();
})();

This is what the complete code looks like:

const puppeteer = require('puppeteer');

(async () => {
  const browserObj = await puppeteer.launch();
  const newpage = await browserObj.newPage();
  await newpage.setViewport({ width: 1920, height: 1080 });

  const url = 'https://sandbox.oxylabs.io/products';

  await newpage.goto(url);
  await newpage.waitForNetworkIdle();
  await newpage.screenshot({
    path: 'screenshot.png',
  });

  await browserObj.close();
})();

If you see a block screen or a CAPTCHA, it means the website has detected the traffic from a programmatically controlled browser. Thereby it blocked access.

Using Puppeteer-stealth to bypass CAPTCHA

You can enhance the capabilities of Puppeteer by installing the Stealth extension with it. The Stealth plugin has a range of features to tackle most of the methods implemented by protected websites to detect the automated-accesses. 

Stealth can make your Puppeteer’s automated headless accesses so “human” that many websites won’t be able to detect the difference. Thereby, Stealth-based accesses prevent CAPTCHA from loading on these websites. Hence, you can make your automated Puppeteer script access the contents behind the CAPTCHA.  

Note: All the bypassing methods showcased in this tutorial are intended for educational purposes only.

Here is the step-by-step procedure to implement this CAPTCHA bypass: 

1. To start, you need to install the puppeteer-extra and puppeteer-extra-plugin-stealth packages.

npm install puppeteer-extra-plugin-stealth puppeteer-extra

2. After that, import the following required libraries in your Node.JS file:

const puppeteerExtra = require('puppeteer-extra');
const Stealth = require('puppeteer-extra-plugin-stealth');

puppeteerExtra.use(Stealth());

3. The next step is to create the browser object in headless mode, navigate to the URL and take a screenshot.

(async () => {
    const browserObj = await puppeteerExtra.launch();
    const newpage = await browserObj.newPage();
  
    await newpage.setViewport({ width: 1920, height: 1080 });
  
    await newpage.setUserAgent(
      'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    );
  
    await newpage.goto('https://sandbox.oxylabs.io/products');
    await newpage.waitForNetworkIdle(); // Wait for network resources to fully load
  
    await newpage.screenshot({ path: 'screenshot_stealth.png' });
  
    await browserObj.close();
})();

The setUserAgent method makes our requests imitate a real browser's User-Agent, making our automated headless browsers appear more like regular users. Setting one of the common User-Agent strings helps evade detection and bypass anti-bot mechanisms that analyze the User-Agent header.

Here is what our complete script looks like:

const puppeteerExtra = require('puppeteer-extra');
const Stealth = require('puppeteer-extra-plugin-stealth');

puppeteerExtra.use(Stealth());

(async () => {
    const browserObj = await puppeteerExtra.launch();
    const newpage = await browserObj.newPage();
  
    await newpage.setViewport({ width: 1920, height: 1080 });
  
    await newpage.setUserAgent(
      'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    );
  
    await newpage.goto('https://sandbox.oxylabs.io/products');
    await newpage.waitForNetworkIdle(); // Wait for network resources to fully load
  
    await newpage.screenshot({ path: 'screenshot_stealth.png' });
  
    await browserObj.close();
})();

Now, If your screenshot shows the actual content of the website, congratulations, your scraper made it to avoid CAPTCHA from loading. But, unfortunately, Stealth can still fail at some websites that use more sophisticated anti-bots.

Luckily, we have another simpler, scalable, and robust alternative, Web Unblocker, and its built-in feature, Headless Browser.

Using Web Unblocker with Node.JS

Web Unblocker uses AI to help users prevent CAPTCHA and gain access to public data from websites with advanced anti-bots implemented. Web Unblocker supports proxy management, automatic generation of browser fingerprints, automatic retries, session maintenance, and JavaScript rendering to control various scraping processes. Also, check this integration tutorial to learn how to use a proxy in Puppeteer. 

To begin, you can send a basic query without any special options. Web Unblocker will select the fastest CAPTCHA proxy, add all necessary headers, and provide you with the response body. 

Using Web Unblocker with Node.JS is easy. Just follow the following steps:

1. Install the node-fetch and HttpsProxyAgent using the following command:

npm install node-fetch https-proxy-agent

2. Sign up to Oxylabs and get your credentials for using the API. 

3. Before importing the libraries, open the package.json file and enter these lines "type": "module", for example:

{
  "type": "module",
  "dependencies": {
    "https-proxy-agent": "^7.0.4",
    "node-fetch": "^3.3.2",
    "puppeteer": "^22.6.5",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-stealth": "^2.11.2"
  }
}

Since the newest version of node-fetch is an ESM-only module, you can’t import it using the require() function. Learn more about it here.

Next, import the required modules in your JS file using the import-from syntax:

import fetch from 'node-fetch';
import HttpsProxyAgent from 'https-proxy-agent';
import fs from 'fs';

The fs library can help save the response in an HTML file. 

4. Provide your user credentials and set up a proxy using HttpsProxyAgent.

const username = '<Your-username>';
const password = '<Your-password>';

(async () => {
    const agent = new HttpsProxyAgent.HttpsProxyAgent(
        `http://${username}:${password}@unblock.oxylabs.io:60000`
    );

5. Next, set the URL and issue a fetch request.

    // Ignore the certificate
    process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;

    const response = await fetch('https://ip.oxylabs.io/', {
        method: 'get',
        agent: agent,
    });

The environment variable NODE_TLS_REJECT_UNAUTHORIZED is set to zero so that Node.JS doesn't verify the SSL/TLS certificates. This is a required setting if you’re using Oxylabs’ Web Unblocker.

6. In the end, you can convert the response into text and save it in an HTML file.

    const resp = await response.text();
    fs.writeFile('result.html', resp.toString(), (err) => {
        if (err) throw err;
        console.log('Result saved to result.html');
    });
})();

Here is the complete script:

import fetch from 'node-fetch';
import HttpsProxyAgent from 'https-proxy-agent';
import fs from 'fs';

const username = '<Your-username>';
const password = '<Your-password>';

(async () => {
    const agent = new HttpsProxyAgent.HttpsProxyAgent(
        `http://${username}:${password}@unblock.oxylabs.io:60000`
    );

    // Ignore the certificate
    process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;

    const response = await fetch('https://ip.oxylabs.io/', {
        method: 'get',
        agent: agent,
    });

    const resp = await response.text();
    fs.writeFile('result.html', resp.toString(), (err) => {
        if (err) throw err;
        console.log('Result saved to result.html');
    });
})();

Thanks to Web Unblocker, you can avoid CAPTCHA from loading and bypass the advanced security obstacles of websites to get your scraping tasks fulfilled. 

Conclusion

CAPTCHA challenges can impede web automation efforts, but with the help of Puppeteer Stealth and Oxylabs’ Web Unblocker, you can bypass CAPTCHAs and make your automation process smooth and obstacle-free. Additionally, check out this blog post on how to bypass CAPTCHAs with Playwright if you're interested in other web scraping libraries. Remember to remain within the legal boundaries and seek legal consultation before engaging in scraping activities of any kind. 

We encourage you to secure a free one-week trial of Oxylabs’  Web Unblocker and read our detailed documentation to get the most out of it.