Includes one of the following:
60K
Pages in HTML
OR
40K
Pages in HTML with JS rendering
OR
29K
E-commerce/search engine API pages
Real-Time Crawler
Get structured data in JSON from a list of ready-to-use data APIs
Extract data from any website with HTML Crawler API
100% delivery from search engines and e-commerce websites
Real-Time Crawler data extraction options
1
Data API
Receive structured data in JSON from ready-to-use data APIs with focus on search engines and e-commerce sites.
2
Learn More
HTML Crawler API
Carry out web crawling projects for most websites in HTML without getting blocked for more resource-efficient data gathering.
Single query and bulk options
Get data in the most convenient way.
Render JavaScript heavy websites
We will render JS for you
Test out Real-Time Crawler'sData API
{
"title": "See Real-Time Crawler in action!",
"message": "Enter your keyword to see the real output example.",
"note": "Choose other criteria (optional).",
}
Get structured
results from leading e-commerce websites
With Real-Time Crawler e-commerce API, get parsed data for:
Product pages
Questions & answers
Offer listing pages
Reviews
Search
Best seller
Get structured results from leading search engines
Real-Time Crawler search engine API provides parsed data for:
Organic
Popular products
Paid
Videos
Product listing ads
Images
Get HTML results from most websites
HTML Crawler API provides raw data with added features such as:
IP blocks management
Batch query
Captcha handling
Proxy pool management
Real-Time Crawler
main benefits
Guaranteed 100% success rates
Pay only for successful pages*
Extract data from most websites without getting blocked
Powered by Next-Gen Residential Proxies
Smooth data gathering ensured by Next-Gen Residential Proxies powered by AI/ML algorithms
Proxy rotator for block management
Patented Oxylabs Proxy Rotator allows to achieve successful requests considerably faster
Structured results in JSON
Get structured JSON data in real-time or via callback method from leading e-commerce and search engine sites
Highly scalable and customizable
Supports high volumes of requests by utilizing Oxylabs global proxy infrastructure
Tailored requests on country & city level, or by device
Zero proxy maintenance
Resilient to website changes
Handles IP blocks and CAPTCHAs
Takes care of proxy management
Easy integration
- Python
- Shell
- PHP
- HTTP
import requests
from pprint import pprint
# Structure payload.
payload = {
'source': 'universal',
'url': 'https://stackoverflow.com/questions/tagged/python',
'user_agent_type': 'desktop',
}
# Get response.
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=('user', 'pass1'),
json=payload,
)
# This will return the JSON response with results.
pprint(response.json())
<?php
$params = array(
'source' => 'universal',
'query' => 'https://stackoverflow.com/questions/tagged/python',
'user_agent_type' => 'desktop',
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");
$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$result = curl_exec($ch);
echo $result;
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>
curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
-d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "user_agent_type": "desktop"}'
https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2Ftagged%2Fpython&user_agent_type=desktop&access_token=1234abcd
Pricing
Billed monthly
Billed yearly-10% off
Country and ASN filtering
100% delivery
Highly scalable
Starter
99
OR
OR
Top-up prices:
$1.65
/1000 pages in HTML
OR
$2.50
/1000 pages in HTML with JS rendering
OR
$3.50
/1000 pages for e-commerce/search engine API
Business
399
Includes one of the following:
285K
Pages in HTML
OR
190K
Pages in HTML with JS rendering
OR
160K
E-commerce/search engine API pages
Top-up prices:
$1.40
/1000 pages in HTML
OR
$2.10
/1000 pages in HTML with JS rendering
OR
$2.50
/1000 pages for e-commerce/search engine API
Corporate
999
Includes one of the following:
833K
Pages in HTML
OR
555K
Pages in HTML with JS rendering
OR
526K
E-commerce/search engine API pages
Top-up prices:
$1.20
/1000 pages in HTML
OR
$1.80
/1000 pages in HTML with JS rendering
OR
$1.90
/1000 pages for e-commerce/search engine API
Enterprise
Starts from:
10,000
Includes one of the following:
14M+
Pages in HTML
OR
11M+
Pages in HTML with JS rendering
OR
10M+
E-commerce/search engine API pages
With no additional fees & included in the price:
Advice on target scraping
Parsed data
Patented proxy rotator
No proxy maintenance
Highly customizable
24/7 live support
Contact our sales team
Leave your contact details
Receive a personal offer
Start collecting data or begin a free trial
Your personal Sales Manager will contact you soon!
You will receive an email with more details within one working day. Your personal Sales Manager will be happy to schedule a call with you and help you set up the proxies for your specific use case.
