JavaScript rendering
Some website content appears only when rendering JavaScript. When web scraping with Oxylabs Scraper APIs, you can use the Headless Browser feature to execute JavaScript and load more data onto the page. Headless Browser renders JavaScript-based pages with a single line of code to save time and resources for the main task – subsequent data analysis.
Custom browser instructions
Perform action sequences to complete a specific task by setting up custom browser instructions. Headless Browser performs mouse clicks, inputs text, scrolls pages, waits for elements to appear, and more. To automate data extraction tasks that require user interaction, you should:
Study web page layout by inspecting HTML elements
Identify interactive elements containing target data
Define browser instructions to interact with the elements and load required data
{
"source": "universal",
"url": "https://www.ebay.com/",
"render": "html",
"browser_instructions": [
{
"type": "input",
"value": "pizza boxes",
"selector": {
"type": "xpath",
"value": "//input[@class='gh-tb ui-autocomplete-input']"
}
},
{
"type": "click",
"selector": {
"type": "xpath",
"value": "//input[@type='submit']"
}
},
{
"type": "wait",
"wait_time_s": 5
}
]
}
Web scraping with browser automation
Automate browser behavior by setting wait times and timeout periods for dynamic elements to load before web scraping. Headless Browser lets you focus on data analysis, leaving infrastructure management and web data extraction activities like Document Object Model (DOM) manipulations, JavaScript rendering, and JavaScript fingerprinting to us.
What is Headless Browser?
A feature of Oxylabs Scraper APIs and Web Unblocker, Headless Browser allows you to define website-specific interactions to load dynamic elements.
Step 1
The user submits browser instructions to Scraper API/Web Unblocker.
Step 2
Headless Browser, a feature of Scraper APIs or Web Unblocker, sends an HTTP(S) request to the target web page.
Step 3
Headless Browser interacts with the web page in a user-defined way to load necessary data.
Step 4
Extracted data in raw HTML or a structured JSON file is transferred to the user's storage.
Solutions featuring Headless Browser
Get a free trial of our Scraper APIs and Web Unblocker, including access to Headless Browser.
SERP Scraper API
Public SERP data collected from the biggest search engines.
Reliable, real-time data
Coordinate-level data
Exceptional ad scraping rate
Best for:
SERP data analysis, keyword data collection, brand monitoring.
From $49/month
E-Commerce Scraper API
Pricing and product information from e-commerce sites.
Adaptive Parser
Most e-commerce websites
Structured JSON data
Best for:
Competitor analysis, pricing intelligence, product catalog mapping.
From $49/month
Web Scraper API
Real-time data from the majority of websites.
Customizable request parameters
Convenient delivery
JavaScript rendering
Best for:
Website change monitoring, fraud protection, travel fare monitoring.
From $49/month
Web Unblocker
Access public data from the most difficult sites.
Easily imitate real website users
Bypass advanced anti-bot systems
Appear as an organic user on sites
Best for:
Anti-bot and CAPTCHAs handling for even the most difficult websites.
From $75/month
Aivaras Steponavicius
Senior Account Manager @ Oxylabs
With Headless Browser, you can automate user interactions to load dynamic data for extraction. This feature saves time and resources as you don’t have to develop and maintain your own headless browser solution.
Evelina Ablacinska
Account Manager @ Oxylabs
Headless Browser greatly simplifies the whole process of data extraction with browser instructions, allowing our clients to quickly sequence actions for site interaction.
A word from our dedicated Account Managers
With certain Enterprise plans, you get your own Dedicated Account Manager.
Useful resources
Step-by-step guides
Learn how to set up your browser instructions with our technical documentation.
Writing browser instructions
Check a tutorial on GitHub and copy the code directly to Headless Browser.
What is a headless browser?
Get familiar with headless browsers: uses, examples, and limitations.
Frequently asked questions
What is the difference between a headless browser and a normal browser?
A headless browser doesn’t have a graphical user interface and is controlled programmatically with user-inputted code, allowing much more control compared to a regular browser.
What are the common use cases for using Headless Browser?
The use cases focus on web automation for data extraction:
Execution of JavaScript on target websites for data extraction.
Scraping JavaScript-heavy website components without managing additional infrastructure.
Data extraction from web pages that require user interactions (loading, scrolling, typing text).
Workflow streamlining – automating wait times for elements or resources to load before concluding a scraping task.
What are the benefits of using a headless browser?
Compared to a regular browser, a headless browser usually gives much more control over web browser-server communication as you can customize all parts of web interaction. On top of that, ease of scalability and running in the background without a graphical user interface utilize fewer resources, making all processes faster.
How does Headless Browser handle JavaScript rendering?
Headless Browser reads websites’ HTML code and executes JavaScript code found within. Part of the JavaScript code may make Headless Browser initiate HTTP requests to fetch additional data. The additional data is then loaded onto the HTML DOM (Document Object Model) and, as a consequence, made visible to the user.
What is the best headless browser?
The Headless Browser feature is the best option if you use an Oxylabs solution like a Scraper API or Web Unblocker.
There are various options for headless browsers, depending on your use cases and familiarity with certain software.
Do I need to be a Scraper APIs or Web Unblocker client to use Headless Browser?
Yes, you should be an active customer of Oxylabs APIs to use the Headless Browser feature.
What are the disadvantages of headless browsers?
Not having a graphical user interface could result in rendering differences. When debugging, the lack of visual representation makes it more difficult to spot website layout changes, as reading the HTML source is the only guidance you get.
Get the latest news from data gathering world
Scale up your business with Oxylabs®
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub
oxylabs.io© 2024 All Rights Reserved