Real-Time Crawler data extraction options

1

Data API

Receive structured data in JSON from ready-to-use data APIs with focus on search engines and e-commerce sites.

E-commerce API

Tailored for accessing data from e-commerce sites

Learn More

Search engine API

Get structured data in real-time from leading search engines

Learn More
2

HTML Crawler API

Carry out web crawling projects for most websites in HTML without getting blocked for more resource-efficient data gathering.

Single query and bulk options

Get data in the most convenient way.

Render JavaScript heavy websites

We will render JS for you

Learn More

Test out Real-Time Crawler's
Data API

API request for search engines

This field is required

API request for e-commerce sites

This field is required

        
{ "title": "See Real-Time Crawler in action!", "message": "Enter your keyword to see the real output example.", "note": "Choose other criteria (optional).", }
RTC E-commerce

Get structured
results from leading e-commerce websites

With Real-Time Crawler e-commerce API, get parsed data for:

Product pages Questions & answers Offer listing pages Reviews Search Best seller
RTC Search engines

Get structured results from leading search engines

Real-Time Crawler search engine API provides parsed data for:

Organic Popular products Paid Videos Product listing ads Images
RTC HTML results

Get HTML results from most websites

HTML Crawler API provides raw data with added features such as:

IP blocks management Batch query Captcha handling Proxy pool management

Real-Time Crawler
main benefits

Guaranteed 100% success rates

Guaranteed 100% success rates

Pay only for successful pages*

Extract data from most websites without getting blocked

Powered by Next-Gen Residential Proxies

Powered by Next-Gen Residential Proxies

Smooth data gathering ensured by Next-Gen Residential Proxies powered by AI/ML algorithms

Proxy rotator for block management

Proxy rotator for block management

Patented Oxylabs Proxy Rotator allows to achieve successful requests considerably faster

Structured results in JSON

Structured results in JSON

Get structured JSON data in real-time or via callback method from leading e-commerce and search engine sites

Highly scalable and customizable

Highly scalable and customizable

Supports high volumes of requests by utilizing Oxylabs global proxy infrastructure

Tailored requests on country & city level, or by device

Zero proxy maintenance

Zero proxy maintenance

Resilient to website changes

Handles IP blocks and CAPTCHAs

Takes care of proxy management

Easy integration


  import requests
  from pprint import pprint

  # Structure payload.
  payload = {
    'source': 'universal',
    'url': 'https://stackoverflow.com/questions/tagged/python',
    'user_agent_type': 'desktop',
  }

  # Get response.
  response = requests.request(
  'POST',
  'https://realtime.oxylabs.io/v1/queries',
  auth=('user', 'pass1'),
  json=payload,

  # This will return the JSON response with results.
  pprint(response.json())


<?php
  $params = array(
    'source' => 'universal',
    'query'  => 'https://stackoverflow.com/questions/tagged/python',
    'user_agent_type'  => 'desktop',
  );

  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

  $headers = array();
  $headers[] = "Content-Type: application/json";

  curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
  $result = curl_exec($ch);

  echo $result;

  if (curl_errno($ch)) {
      echo 'Error:' . curl_error($ch);
  }
  curl_close ($ch);
?>


  curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
  -d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "user_agent_type": "desktop"}'


  https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2Ftagged%2Fpython&user_agent_type=desktop&access_token=1234abcd

Pricing

Billed monthly
Billed yearly-10% off

Country and ASN filtering

100% delivery

Highly scalable

Starter

99

Includes one of the following:

60K

Pages in HTML


OR

40K

Pages in HTML with JS rendering


OR

29K

E-commerce/search engine API pages

Top-up prices:

$1.65

/1000 pages in HTML


OR

$2.50

/1000 pages in HTML with JS rendering


OR

$3.50

/1000 pages for e-commerce/search engine API

Business

399

Includes one of the following:

285K

Pages in HTML


OR

190K

Pages in HTML with JS rendering


OR

160K

E-commerce/search engine API pages

Top-up prices:

$1.40

/1000 pages in HTML


OR

$2.10

/1000 pages in HTML with JS rendering


OR

$2.50

/1000 pages for e-commerce/search engine API

Corporate

999

Includes one of the following:

833K

Pages in HTML


OR

555K

Pages in HTML with JS rendering


OR

526K

E-commerce/search engine API pages

Top-up prices:

$1.20

/1000 pages in HTML


OR

$1.80

/1000 pages in HTML with JS rendering


OR

$1.90

/1000 pages for e-commerce/search engine API

Enterprise

Starts from:

10,000

Includes one of the following:

14M+

Pages in HTML


OR

11M+

Pages in HTML with JS rendering


OR

10M+

E-commerce/search engine API pages

Starter

$99

89

Includes one of the following:

60K

Pages in HTML


OR

40K

Pages in HTML with JS rendering


OR

29K

E-commerce/search engine API pages

Top-up prices:

$1.48

/1000 pages in HTML


OR

$2.22

/1000 pages in HTML with JS rendering


OR

$3.07

/1000 pages for e-commerce/search engine API

Business

$399

359

Includes one of the following:

285K

Pages in HTML


OR

190K

Pages in HTML with JS rendering


OR

160K

E-commerce/search engine API pages

Top-up prices:

$1.26

/1000 pages in HTML


OR

$1.89

/1000 pages in HTML with JS rendering


OR

$2.24

/1000 pages for e-commerce/search engine API

Corporate

$999

899

Includes one of the following:

833K

Pages in HTML


OR

555K

Pages in HTML with JS rendering


OR

526K

E-commerce/search engine API pages

Top-up prices:

$1.08

/1000 pages in HTML


OR

$1.62

/1000 pages in HTML with JS rendering


OR

$1.71

/1000 pages for e-commerce/search engine API

Enterprise

Starts from:

9,000

Includes one of the following:

14M+

Pages in HTML


OR

11M+

Pages in HTML with JS rendering


OR

10M+

E-commerce/search engine API pages

With no additional fees & included in the price:

Advice on target scraping

Parsed data

Patented proxy rotator

No proxy maintenance

Highly customizable

24/7 live support