Back to blog

Playwright vs Selenium: Which One to Choose

Enrika Pavlovskytė

2023-01-10
5 min read
Share

Web browsing has changed significantly throughout the years, becoming much more experiential than in the past. Indeed, websites are now more compelling, interactive, and dynamic due to the emphasis placed on consistent user experiences. On the other hand, they’re also becoming more complex, making them more difficult to scrape.

Even the best scraper, which can easily extract data from a static page, might stumble when it encounters a dynamic one. Thankfully, dynamic web page scraping is made simpler by modern web automation frameworks like Selenium and Playwright. The tricky part is choosing the right one for your project.

In this blog post, we’ll discuss Playwright vs Selenium, their relevance to web scraping, and what to remember when picking one for your scraping task.

Playwright and Selenium at a glance

What is Selenium?

In short, Selenium is an open-source framework dedicated to cross-browser testing and automation. What initially began as an internal tool evolved into a project that serves as a hub for several tools and libraries applicable to various use cases, including web scraping. Key components of Selenium are:

  • Selenium WebDriver – a collection of application programming interfaces (APIs) for creating and running browser tests. Rather than focusing on a single browser such as Firefox or Chrome, it can drive a variety of them. In addition to that, you need to download language bindings where you'll write the script that will interact with the Selenium WebDriver.

  • Selenium IDE – a record and playback test automation tool that developers can use to document their actions and convert them into scripts. They can also turn test cases into file formats and run them in Selenium WebDriver.

  • Selenium Grid – used to execute WebDriver scripts on remote machines. The main advantage is that developers can run parallel tests on multiple machines simultaneously, thus saving time and resources. 

What is Playwright?

Microsoft made Playwright available to the public only a few years ago, but it has already become a widely used tool. Similarly to Selenium, it’s a cross-browser web automation library.

Interestingly, Playwright was built by the same team that developed Puppeteer, which means they share similar features, such as API methods. Playwright, however, is designed to make end-to-end testing simpler for developers and testers who intend to utilize it across various browsers. As a result, it supports such browser engines as Chromium, Firefox, and WebKit. Finally, it’s an open-source tool that only requires Node.js to get started.

Playwright and Selenium in web scraping

If Selenium and Playwright are test automation tools, how are they relevant to web scraping? The answer lies in their ability to control headless browsers. So, let’s take a look at what that is and why we might need it for web scraping.

Static and dynamic web pages 

To understand web scraping with a headless browser, it’s essential to discuss the concept of static and dynamic web pages. A static website consists of multiple web pages developed with the help of HTML, CSS, and JavaScript. Everything you see on that page is exactly what other users see. Most importantly, static web pages are stored in HTML files, meaning web scrapers can easily acquire them through an HTTP request.  

Dynamic web pages, however, are developed with server-side language and can render content based on user behavior. So, two users might see completely different content based on their location, browsing history, device specifications, etc. The majority of the time, JavaScript is used to display dynamic pages, which poses numerous challenges for web scrapers like browser fingerprinting, asynchronous loading, and infinite scrolling. That’s where Selenium and Playwright become instrumental.

Headless browsing

Despite both of these being web automation frameworks, they play a pivotal role in web scraping by enabling headless browser functionality. Headless browsing means interacting with a browser without UI elements or a GUI. These functions are not necessarily lost. Instead, you command the browser to simulate actions like clicking, downloading, or scrolling by writing a script.

Without having to load visual elements, you’ll need fewer resources and will be able to upscale operations. For example, you can spawn numerous browser instances, allowing you to scrape different websites simultaneously. 

Additionally, websites are able to know if an internet user can execute JavaScript to render a website. Clients who can't do that might be flagged as a bot and get blocked. By using a headless browser while scraping, you can overcome this issue.

Choosing between Playwright vs Selenium

If both Selenium and Playwright can help you with headless browsing, how can you know which one to choose? Well, comparing the two can be quite complicated. From programming language and browser combinations to the requirements of the scraping project, there are myriad scenarios where one might perform better than the other.  Rather than listing them all, let’s take a look at key points you should consider before opting for one or the other.

Browser support

While Selenium supports a huge variety of browser options, the user still needs to install specific WebDrivers for each browser. Playwright, on the other hand, comes with an in-built driver, which makes implementing it much easier. You should note, though, that it only supports Chromium, Firefox, and WebKit. You need to consider the web browsers your project will require before deciding whether to pick Selenium or Playwright.

It’s important to note that Selenium has recently launched Selenium Manager to circumvent the WebDriver management problem. However, it's currently under beta testing, and using it can still cause issues with your workflow.

Programming languages

Being an older tool, Selenium supports far more programming languages than Playwright, with its main ones being Java, Python, Ruby, C#, and JavaScript. Furthermore, with Selenium’s client language bindings, you can also use Go, Haskell, PHP, Perl, R, and Dart. 

Playwright supports TypeScript, JavaScript, Python, .NET, and Java. While it's less than Selenium provides, Playwright is easier to implement, so if you're using one of the multiple programming languages it supports, Playwright might be the better choice.

Speed 

In terms of speed, Selenium is regarded as being slower than Playwright. The former is more suitable for small to average-sized scraping projects as more computing power will significantly reduce speed. To make an informed decision, check out some tests and comparisons of the two.

Community support

As Playwright is more recent than Selenium, it lacks the internet resources Selenium provides. The latter features a sizable and active community with a ton of in-depth documentation. As a result, when you hit a roadblock, you'll probably be able to find assistance online but have difficulty doing the same with Playwright.

Architecture

Selenium and Playwright are based on different architectures. As mentioned before, for Selenium, you can install a language-specific client driver (binding) to write scripts capable of interacting with the Web Driver. Moreover, this will be done using HTTP by exchanging JSON payload. In a nutshell, every line of Selenium code will require JSON Wire Protocol to be sent, which might produce delays.

Playwright, on the other hand, uses an event-driven architecture based on decoupled systems that respond to events (user- or system-generated actions). This means that each component is independent and interacts with other components by interchanging events. It allows for asynchronous communication, which makes the system more scalable, flexible, and faster.

These are a few dimensions against which we can discuss the pros and cons of both frameworks. For a more detailed look, you can also refer to the table below:

Playwright Selenium
Browser support Chromium, Firefox, and WebKit Firefox, Edge Chromium (Selenium 4), Safari, Opera, Google Chrome, and more
Operating systems Windows, Mac OS, and Linux Windows, Mac OS, Linux, and Solaris
Languages supported TypeScript, JavaScript, Python, .NET, Java Java, Python, Ruby, C#, and JavaScript (and more with language binding)
Prerequisites & installation Needs NodeJS to be installed, but otherwise, a straightforward process Selenium Bindings (for your language), Browser Drivers, and Selenium Standalone Server needed
Real devices Emulation (experimental support for real devices also available) Offers real device support through clouds and remote servers
Community Small but active Big and active
Developer experience Very good Fair
Speed Fast Slower
Architecture Event-driven architecture Layered architecture relying on the JSON Wire Protocol

Bottom line

Overall, Playwright vs Selenium can be a tough decision to make. Both are excellent test automation tools highly applicable to web scraping. However, our recommendation would look something like this:

Playwright: best for when your project's needs can be met by Playwright's supported languages and browsers. Choose Playwright for a fast, efficient, and simple-to-implement headless browser.

Selenium: best for when flexibility is required, and you wish to employ a very specific browser and programming language combination. Additionally, given the range of resources accessible online, Selenium may be a highly useful tool for learning web scraping with a headless browser.

In the end, there isn't a single solution that fits all situations; thus, it's important to thoroughly consider the project's requirements.

If you enjoyed reading this blog post, be sure to check out further materials on web scraping with Playwright and Selenium

People also ask

Will Playwright replace Selenium?

While Playwright surpasses Selenium in simplicity and speed, the latter has been around for longer and has gathered a big community. Both frameworks keep introducing new features, so it will largely depend on how they develop. Ultimately, we might see both of them focus on different areas and people opting for one or the other depending on their project needs.

Is Playwright built on Selenium?

No, these are two different frameworks built for browser automation. It is true that Playwright aims to be easier to use than Selenium. However, both of them are built using completely different technology stacks and possess distinct architectures. For instance, Selenium has a layered architecture relying on the JSON Wire Protocol, whereas Playwright uses an event-driven architecture.

About the author

Enrika Pavlovskytė

Junior Copywriter

Enrika Pavlovskytė is a Junior Copywriter at Oxylabs. With a background in digital heritage research, she became increasingly fascinated with innovative technologies and started transitioning into the tech world. On her days off, you might find her camping in the wilderness and, perhaps, trying to befriend a fox! Even so, she would never pass up a chance to binge-watch old horror movies on the couch.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

How to Parse HTML with PyQuery: Python Tutorial

Vytenis Kaubre

2023-01-25

Playwright vs Puppeteer: The Differences

Augustas Pelakauskas

2023-01-24

Web Unblocker Quick Start Guide

Vytenis Kaubre

2023-01-02

Get the latest news from data gathering world

I'm interested

IN THIS ARTICLE:


  • Playwright and Selenium at a glance

  • Playwright and Selenium in web scraping

  • Choosing between Playwright vs Selenium

  • Bottom line

Scale up your business with Oxylabs®