Headless Browser for Scalable Scraping

JavaScript rendering

Some website content appears only when rendering JavaScript. When web scraping with Oxylabs Web Scraper API, you can use the Headless Browser feature to execute JavaScript and load more data onto the page. Headless Browser renders JavaScript-based pages with a single line of code to save time and resources for the main task – subsequent data analysis.

Custom browser instructions

Perform action sequences to complete a specific task by setting up custom browser instructions. Headless Browser performs mouse clicks, inputs text, scrolls pages, waits for elements to appear, and more. To automate data extraction tasks that require user interaction, you should:

Study web page layout by inspecting HTML elements
Identify interactive elements containing target data
Define browser instructions to interact with the elements and load required data

{
   "source": "universal",
   "url": "https://www.ebay.com/",
   "render": "html",
   "browser_instructions": [
      {
         "type": "input",
         "value": "pizza boxes",
         "selector": {
            "type": "xpath",
            "value": "//input[@class='gh-tb ui-autocomplete-input']"
         }
      },
      {
         "type": "click",
         "selector": {
            "type": "xpath",
            "value": "//input[@type='submit']"
         }
      },
      {
         "type": "wait",
         "wait_time_s": 5
      }
   ]
}

Web scraping with browser automation

Automate browser behavior by setting wait times and timeout periods for dynamic elements to load before web scraping. Headless Browser lets you focus on data analysis, leaving infrastructure management and web data extraction activities like Document Object Model (DOM) manipulations, JavaScript rendering, and JavaScript fingerprinting to us.

What is Headless Browser?

A feature of Oxylabs Web Scraper API and Web Unblocker, Headless Browser allows you to define website-specific interactions to load dynamic elements.

Step 1

The user submits browser instructions to Web Scraper API/Web Unblocker.

Step 2

Headless Browser, a feature of Web Scraper API or Web Unblocker, sends an HTTP(S) request to the target web page.

Step 3

Headless Browser interacts with the web page in a user-defined way to load necessary data.

Step 4

Extracted data in raw HTML or a structured JSON file is transferred to the user's storage.

Solutions featuring Headless Browser

Get a free trial of Web Scraper API and Web Unblocker, including access to Headless Browser.

Web Scraper API

Parsed real-time data from almost any website.

Customizable request parameters
Structured JSON data
JavaScript rendering

Best for:

SERP data analysis, pricing intelligence, travel fare monitoring.

From $49/month

Start free trial

Web Unblocker

Access public data from the most difficult sites.

Easily imitate real website users
Bypass advanced anti-bot systems
Appear as an organic user on sites

Best for:

Anti-bot and CAPTCHAs handling for even the most difficult websites.

From $75/month

Start free trial

Vincent Patrizio

Senior Account Manager @ Oxylabs

With Headless Browser, you can automate user interactions to load dynamic data for extraction. This feature saves time and resources as you don’t have to develop and maintain your own headless browser solution.

Jorūnė Skridailaitė

Senior Account Manager @ Oxylabs

Headless Browser greatly simplifies the whole process of data extraction with browser instructions, allowing our clients to quickly sequence actions for site interaction.

A word from our dedicated Account Managers

With certain Enterprise plans, you get your own Dedicated Account Manager.

Useful resources

Documentation

Step-by-step guides

Learn how to set up your browser instructions with our technical documentation.

Oxylabs GitHub

Writing browser instructions

Check a tutorial on GitHub and copy the code directly to Headless Browser.

Oxylabs blog

What is a headless browser?

Get familiar with headless browsers: uses, examples, and limitations.

Frequently asked questions

What is the difference between a headless browser and a normal browser?

A headless browser doesn’t have a graphical user interface and is controlled programmatically with user-inputted code, allowing much more control compared to a regular browser.

What are the common use cases for using Headless Browser?

The use cases focus on web automation for data extraction:

Execution of JavaScript on target websites for data extraction.
Scraping JavaScript-heavy website components without managing additional infrastructure.
Data extraction from web pages that require user interactions (loading, scrolling, typing text).
Workflow streamlining – automating wait times for elements or resources to load before concluding a scraping task.

What are the benefits of using a headless browser?

Compared to a regular browser, a headless browser usually gives much more control over web browser-server communication as you can customize all parts of web interaction. On top of that, ease of scalability and running in the background without a graphical user interface utilize fewer resources, making all processes faster.

How does Headless Browser handle JavaScript rendering?

Headless Browser reads websites’ HTML code and executes JavaScript code found within. Part of the JavaScript code may make Headless Browser initiate HTTP requests to fetch additional data. The additional data is then loaded onto the HTML DOM (Document Object Model) and, as a consequence, made visible to the user.

What is the best headless browser?

The Headless Browser feature is the best option if you use an Oxylabs solution like Web Scraper API or Web Unblocker.

There are various options for headless browsers, depending on your use cases and familiarity with certain software.

Do I need to be a Web Scraper API or Web Unblocker client to use Headless Browser?

Yes, you should be an active customer of either to use the Headless Browser feature.

What are the disadvantages of headless browsers?

Not having a graphical user interface could result in rendering differences. When debugging, the lack of visual representation makes it more difficult to spot website layout changes, as reading the HTML source is the only guidance you get.