Puppeteer vs Selenium: Which to Choose

Disadvantages:

• More complicated installation process due to its support of various platforms, languages, and browsers.

• Unlike Puppeteer, fails to provide different performance management capabilities.

• Steep learning curve.

Differences in set up and web scraping

In this section, we’ll compare these two tools based on fundamental differences in setting up the environment and web scraping efficiency using Node.js.

Installation

Both Puppeteer and Selenium installation processes are simple. The main distinction is in the prerequisite libraries. While Puppeteer users merely need to install it using a simple npm command, Selenium users must follow language-specific instructions.

Puppeteer

npm install puppeteer

Selenium

npm install selenium-webdriver
npm install chromedriver

Browser control and web scraping

Both tools allow programmatic web browser control. You can use this ability to scrape dynamic content from your target web page. Let’s see key code differences to launch a chrome (headless) instance, navigate it to a specific web page, wait for specific dynamic content to load, and scrape the page.

Our target for scraping will be http://quotes.toscrape.com/js/. This is a dynamic web page where all the quotes are loaded dynamically through the relevant JavaScript file. The JavaScript file renders quotes in <DIV> elements, all having a quote class. 

1. Dependencies and setting the target

Puppeteer

const puppeteer = require('puppeteer');
const url = 'http://quotes.toscrape.com/js/';

Selenium

const { Builder, By, Key, until } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');
const url = 'http://quotes.toscrape.com/js/';

Selenium supports multiple browsers, so it requires importing specific browser drivers (chrome drivers in our case) along with the webdriver. Chrome driver is implicit with Puppeteer. 

2. Launching a headless chrome instance and navigating to the target URL

Puppeteer

const headlessBrowser = await puppeteer.launch({ headless: true });

const newTab = await headlessBrowser.newPage();

await newTab.goto(url);

Selenium

let driver = await new Builder().forBrowser('chrome') .setChromeOptions(new chrome.Options().headless()).build();

await driver.get(url);

Puppeteer uses an awaitable launch() method to launch the browser instance, and the newPage() method creates a new browser tab. Now, the goto() method can navigate the tab to any given URL. 

Selenium, on the other hand, uses Builder() constructor to build a new Builder instance followed by specific options. The build() method at the end creates and returns a new instance of the webdriver session.

Note: You must enclose awaitable calls inside an asynchronous function.

3. Waiting for dynamic content to load

Let’s see the differences between Puppeteer and Selenium in waiting for specific JavaScript content to load. The following code waits for JavaScript to load a <Div> element with the quote class.

Puppeteer

await newTab.waitForSelector('.quote');

Selenium

await driver.wait(until.elementLocated(By.className('quote')));

Puppeteer uses the waitForSelector() method, while Selenium uses the wait() method in conjunction with until property to wait for a specific element to load.  

4. Scraping the quotes

Puppeteer uses querySelectorAll() method to select and return a list of all the matching elements specified by given CSS selectors. On the other hand, Selenium provides a findElements() method to extract the relevant elements matching the By selectors.

Puppeteer

let quotes = await newTab.evaluate(() => {
    let allQuoteDivs = document.querySelectorAll(".quote");

    let quotesString= "";
    allQuoteDivs.forEach((quote) => {
        let qouteText = quote.querySelector(".text").innerHTML;
        quotesString += `${qouteText} \n`;
    });
     return quotesString;
 
});

console.log(quotes);

Selenium

let quotes = await driver.findElements(By.className('quote'));

let quotesString = "";
for (let quote of quotes) {
      let qouteText = await quote.findElement(By.className('text')). getText();
      quotesString += `${qouteText} \n`;
 }

 console.log(quotesString);

Notice the evaluate() method in the Puppeteer code. It allows executing a function in the current tab or page context. This means that you can access and manipulate the elements in the Document Object Model (DOM) of the current tab and then return a value as a result (which is the quotesString in our case). 

5. Closing the browser

Puppeteer offers the close() method to close the browser instance, while Selenium provides the quit() method to exit the browser instance and destroy the driver session. 

Puppeteer

headlessBrowser.close();

Selenium

await driver.quit();

Puppeteer vs Selenium: key differences

Now that we’ve laid out the most important features, advantages, and drawbacks of broth frameworks, let’s group together all of their key differences.

Which one should you choose?

At this point, it’s already clear that both Puppeteer and Selenium are two powerful tools with exceptional capabilities for testing automation. However, while they do have several differences, the final decision of whether you should use one or the other will depend on your or your organization’s specific needs.

If you work exclusively with Chrome, Puppeteer is the go-to choice. Its high-level API will provide you with unparalleled control over the browser, and the excellent speed and focus offered by it will make sure you achieve efficiency in setting up tests. What’s more, considering the fact that Puppeteer is more of web automation than testing library, it’ll be more suitable for such activities as web crawling and scraping.

On the other hand, in case you need to support other browsers and programming languages, you should definitely choose Selenium. With Selenium WebDriver offering cross-browser support, you’ll be able to interact with any browser directly, significantly extending the test scope without the need to rely on any other external tools.

Wrapping up

This article discussed and compared two of the most popular automation frameworks, Puppeteer and Selenium, each offering a number of distinct features and advantages. Therefore, we hope you’ll use it as a guideline to identify your requirements and choose the most suitable tool for your future projects.

If you're stuck deciding whether Puppeteer or Selenium will work better for your public data collection project, see our other comparisons of web scraping tools, such as Scrapy vs. Selenium, and a comprehensive overview of the top website testing tools. You can also try our advanced web scraping solutions for free – Web Scraper API and Web Unblocker, including their built-in feature, Headless Browser. And in case you have any questions in the process, don’t hesitate to contact us at hello@oxylabs.io.