features
web crawler
Web Crawler
Discover and collect only relevant data from target websites
Control the crawling approach and scope; define the end result
Get results as parsed data, a set of HTMLs, or a list of URLs
*Web Crawler is a feature of Scraper APIs
Rapidly gather relevant data from websites
Gather only the data you need by crawling a website in seconds. Web Crawler efficiently spiders any website based on your selected criteria and seamlessly returns the complete data to you.
Easily control the scope and tailor your end result
With Web Crawler, you have full control over the creation and continuity of the process. You can also specify how the website should be crawled using filters and scraping parameters such as regular expressions, proxy geo-location, results storage, and more.
Retrieve your results in a specified format
Receive results according to your data needs. There are three output formats: a list of URLs (sitemap), a set of HTML files, and parsed data. Optionally, Web Crawler can upload the result files to your cloud storage.
How does Web Crawler work?
Web Crawler is an add-on to Oxylabs Scraper APIs that allows you to leverage the APIs’ scraping and parsing functions to crawl websites at scale in real time. Select a starting URL, specify crawling patterns, let Web Crawler traverse the site, and receive results to your chosen cloud storage bucket.
User input
The service user forms an input that determines the crawling scope, specifies scraping parameters, and submits a request to the job initiation endpoint.
Web crawling
Web Crawler traverses a website by using links between pages until it finds no more new URLs that match the patterns specified by the user.
Job result
Web Crawler aggregates the result files (sitemaps, parsed data, or HTML documents) into one or more result files as a final output ready to use.
Transfer to cloud
Optionally, Web Crawler can upload the files to the client-specified cloud storage location on AWS S3.
Scraper APIs with the Web Crawler feature
As an add-on to Oxylabs’ Scraper APIs, Web Crawler allows you to discover and collect data easily and efficiently using our maintenance-free infrastructure.
E-Commerce Scraper API
E-commerce product page scraping with ready-to-use data.
1000s of e-commerce websites
Adaptive Parser
Structured data in JSON
Best for:
Pricing intelligence, product catalog mapping, competitor analysis.
From $99/month
Real Estate Scraper API
Real-time property data gathering from popular real estate websites.
Zillow, Realestate, Redfin, Zoopla, and others
No CAPTCHAs or IP blocks
Cost-efficient scraping
Best for:
Researching new investments, price optimization, trend discovery.
From $99/month
Web Scraper API
Scalable real-time data collection from a majority of websites.
Customizable request parameters
JavaScript rendering
Convenient delivery
Best for:
Website changes monitoring, fraud protection, travel fare monitoring.
From $99/month
Aivaras Steponavicius
Account Manager @ Oxylabs
Before you can extract data from a website, often, you need to do some web crawling first to find the specific URLs you're interested in. Web Crawler can take care of that for you automatically.
Ruta Petronyte
Account Manager @ Oxylabs
Web Crawler is a great addition to our Scraper APIs that lets you efficiently explore and collect data using Oxylabs’ maintenance-free infrastructure.
A word from your Dedicated Account Manager
With Oxylabs Corporate and Enterprise plans, you get your very own dedicated Account Manager.
Useful resources
Web Crawler Documentation
Check in-depth guidelines to understand the ins and outs of using Web Crawler.
Step-By-Step Guide
To visualize the crawling job initiation process, take a look at our video guide.
Where to Start & How it Works
Read our blog for a quick intro to the technicalities of using Web Crawler.
Frequently Asked Questions
What is Web Crawler?
Web Crawler is a feature of Oxylabs Scraper APIs that lets you spider any website, select useful content, and have it delivered in bulk.
If you want to learn more about web crawlers, check this blog post for a detailed explanation.
What does Web Crawler do?
Web Crawler can discover all pages on a website and fetch data at scale and in real time.
The tool follows the links from the initial web page to other pages until it has visited and indexed all the pages it can find on a particular website.
Is it legal to crawl a website?
The answer depends on the specific task at hand. Before crawling, make sure you are in line with the applicable laws regarding your access to a particular public domain. Our team suggests seeking professional legal guidance. Check our blog post on the legalities of public web data gathering.
Who uses web crawlers?
Web crawlers are used by individuals and organizations – anyone who needs to collect data from websites – including but not limited to:
Search engines index and organize web pages, allowing users to easily find relevant information.
E-commerce companies gather information about their competitors' products, prices, and promotions.
Marketing professionals collect data on their target audience, monitor social media mentions, and track online reputation of their brand.
Government agencies monitor websites for illegal or harmful content and gather intelligence for security purposes.
Website owners check their website's search engine rankings, identify broken links, and track their brand's online reputation.
Get the latest news from data gathering world
Scale up your business with Oxylabs®
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub