While web scraping and automation tools have developed significantly during recent years, handling dynamic websites has become a breeze. Headless browsers without a graphical user interface offered an efficient way of collecting public data, as you can control them programmatically. If combined with proxies, they’re even better. In this article, we’ll go through the Puppeteer integration process with Oxylabs Residential Proxies.
What is Puppeteer?
Puppeteer is a Node.js library offering a top-level API for headless Chrome or Chromium browsers’ control through the DevTools Protocol. It’s also known as a Headless Chrome Node API and is used for automating Chrome to run website tests.
How to integrate Oxylabs proxies with Puppeteer?
Before getting started with Puppeteer, you’ll need to install some basic tools: Node.js and a code editor of your choice. After that, create a Node.js project and install the required packages. You may find a detailed guide on how to install and run Puppeteer on our blog post.
Once everything is set up, we can move on to the next part – Oxylabs Residential Proxies integration with Puppeteer.
- Within Puppeteer, fill in the value, for example:
pr.oxylabs.io:7777- Under ‘page.authenticate’ input your Oxylabs proxy sub-user username in the ‘username’ value and your password.
- The example of a code looks like this:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless: false,
args: ['--proxy-server=pr.oxylabs.io:7777]
});
const page = await browser.newPage();
await page.authenticate({
username: 'USERNAME',
password: 'PASSWORD'
});
await page.goto('https://ip.oxylabs.io');
await page.screenshot({path: 'example.png'});
await browser.close();
})();You can also use country-specific entries. For example, if you put us-pr.oxylabs.io under ‘host’ and 10001 under ‘port’, you will receive a US exit node with a sticky session. Please check out our documentation for a complete list of country-specific entry nodes.
And that’s it! You’ve successfully integrated Oxylabs Residential proxies with Puppeteer.
Conclusion
Puppeteer, in combination with Oxylabs Residential Proxies, might be of great help when it comes to scraping public data from dynamic websites.
In case you have any queries about integrating Oxylabs proxies, please get in touch with us at any time.
