NEW

Introducing AI & ML powered Next-Gen Residential Proxies

Learn more

Real-Time Crawler data extraction options

1

Data API

Receive structured data in JSON from ready-to-use data APIs with focus on search engines and e-commerce sites.

E-commerce API

Tailored for accessing data from e-commerce sites

Learn More

Search engine API

Get structured data in real-time from leading search engines

Learn More
2

HTML Crawler API

Carry out web crawling projects for most websites in HTML without getting blocked for more resource-efficient data gathering.

Single query and bulk options

Get data in the most convenient way.

Render JavaScript heavy websites

We will render JS for you

Learn More

Test out Real-Time Crawler's
Data API

API request for search engines

This field is required

API request for e-commerce sites

This field is required

        
{ "title": "See Real-Time Crawler in action!", "message": "Enter your keyword to see the real output example.", "note": "Choose other criteria (optional).", }
RTC E-commerce

Get structured
results from leading e-commerce websites

With Real-Time Crawler e-commerce API, get parsed data for:

Product pages Questions & answers Offer listing pages Reviews Search Best seller
RTC Search engines

Get structured results from leading search engines

Real-Time Crawler search engine API provides parsed data for:

Organic Popular products Paid Videos Product listing ads Images
RTC HTML results

Get HTML results from most websites

HTML Crawler API provides raw data with added features such as:

IP blocks management Batch query Captcha handling Proxy pool management

Real-Time Crawler
main benefits

Guaranteed 100% success rates

Guaranteed 100% success rates

Pay only for successful pages*

Extract data from most websites without getting blocked

Powered by Next-Gen Residential proxies

Powered by Next-Gen Residential proxies

Smooth data gathering ensured by Next-Gen Residential proxies powered by AI/ML algorithms

Proxy rotator for block management

Proxy rotator for block management

Patented Oxylabs Proxy Rotator allows to achieve successful requests considerably faster

Structured results in JSON

Structured results in JSON

Get structured JSON data in real-time or via callback method from leading e-commerce and search engine sites

Highly scalable and customizable

Highly scalable and customizable

Supports high volumes of requests by utilizing Oxylabs global proxy infrastructure

Tailored requests on country & city level, or by device

Zero proxy maintenance

Zero proxy maintenance

Resilient to website changes

Handles IP blocks and captchas

Takes care of proxy management

Easy integration


  import requests
  from pprint import pprint

  # Structure payload.
  payload = {
    'source': 'universal',
    'url': 'https://stackoverflow.com/questions/tagged/python',
    'user_agent_type': 'desktop',
  }

  # Get response.
  response = requests.request(
  'POST',
  'https://realtime.oxylabs.io/v1/queries',
  auth=('user', 'pass1'),
  json=payload,

  # This will return the JSON response with results.
  pprint(response.json())


<?php
  $params = array(
    'source' => 'universal',
    'query'  => 'https://stackoverflow.com/questions/tagged/python',
    'user_agent_type'  => 'desktop',
  );

  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

  $headers = array();
  $headers[] = "Content-Type: application/json";

  curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
  $result = curl_exec($ch);

  echo $result;

  if (curl_errno($ch)) {
      echo 'Error:' . curl_error($ch);
  }
  curl_close ($ch);
?>


  curl --user user:pass1 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json"
  -d '{"source": "universal", "url": "https://stackoverflow.com/questions/tagged/python", "user_agent_type": "desktop"}'


  https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2Ftagged%2Fpython&user_agent_type=desktop&access_token=1234abcd

See what others think about
Real-Time Crawler

RTC tool really helped our company

RTC tool really helped our company. At first, it was kind of complicated to get a hold of it, but our account manager Gabriele was very patient and helped us a lot

For owning an e-commerce company

For owning an e-commerce company, Oxylabs RTC really helped us. We were sceptical about their promise of 100% data delivery, but it works and we are very happy with it

Our company started using Oxylabs…

Our company started using Oxylabs services few years ago and since that we are really happy with proxy services they are providing. Recently they recommended us trying their RTC and to be fair we are very happy with results.

Pricing

Geotargeting

Automatic retries

24/7 support

Entry

99

Includes one of the following:

60K

Pages in HTML


or

40K

Pages in HTML with JS rendering


or

29K

E-commerce/search engine API pages

Dedicated Account Manager

Top-up prices:

1,65

/1000 pages in HTML


or

2,50

/1000 pages in HTML with JS rendering


or

3,50

/1000 pages for e-commerce/search engine API

Advanced

399

Includes one of the following:

285K

Pages in HTML


or

190K

Pages in HTML with JS rendering


or

160K

E-commerce/search engine API pages

Dedicated Account Manager

Top-up prices:

1,40

/1000 pages in HTML


or

2,10

/1000 pages in HTML with JS rendering


or

2,50

/1000 pages for e-commerce/search engine API

Pro

999

Includes one of the following:

833K

Pages in HTML


or

555K

Pages in HTML with JS rendering


or

526K

E-commerce/search engine API pages

Dedicated Account Manager

Top-up prices:

1,20

/1000 pages in HTML


or

1,80

/1000 pages in HTML with JS rendering


or

1,90

/1000 pages for e-commerce/search engine API

Enterprise

For a bigger plan

Book a call to get a price estimate

Dedicated Account Manager