Aleksandras Sulzenko

Jan 16, 2019 7 min read

Here at Oxylabs, we work with hundreds of companies from various industries. Although each industry has its own specifics, one thing is clear – more and more companies are trying to increase the efficiency of data collection and analysis.

Instead of maintaining expensive proxy infrastructure, businesses are looking for smarter and more cost-efficient solutions such as Real-Time Crawler – a real time web scraping solution.

What is Real-Time Crawler?

Real-Time Crawler is a data collection tool built specifically for data extraction from search engines and e-commerce websites, also known as real time web scraping solution.

In essence, Real-Time Crawler is an advanced scraper customized for heavy-duty data retrieval operations.

If you feel like you need to familiarize yourself with Web Crawling vs. Web Scraping topic, check out our blog post entry.  But for now, let’s jump into how our Real-Time Crawler works.

How does Real-Time Crawler work?

The process goes as follows:

  1. A client sends a request to Real-Time Crawler.
  2. Real-Time Crawler collects the required information.
  3. The client receives collected web data.

We made a short video on that, take a look:

Currently, we offer two data delivery methods: real-time and callback.

Real-Time data delivery method

  • With the real-time data delivery method, the required data is retrieved on the same connection.
  • This means that you submit your request and get your data back on the same open HTTPS connection, so you get real time web scraping.

Get in touch with us for more details and code examples.

Real-Time Data Delivery

Callback data delivery method

  • With the callback data delivery method, you don’t have to keep an open connection or check your task status. Instead, Real-Time Crawler sends a notification when the required data is ready.
  • Keep in mind that in order to use the callback data delivery method, you have to set up a callback server. Then, you simply create a job request and send it to Real-Time Crawler. Real-Time Crawler returns job info and starts collecting the required data.
  • Once the data is ready, Real-Time Crawler lets you know about it by sending a POST request to your machine and providing a URL to download the results in HTML or JSON format.

Get in touch with us for more details and code examples. Also, in case you have any troubles setting up your callback handling machine, drop us a line, and we’ll help you out!

Real-Time Crawler Callback method

Using Real-Time Crawler for e-commerce websites

Real-Time Crawler was built with e-commerce sites in mind. It’s currently customized to support data extraction from the most popular retail marketplaces. However, our team can always build a custom solution for you.

With Real-Time Crawler, you can extract data from product pages, product offer listing pages, reviews, questions & answers, search results or from any URL in general. All localized domains and pagination are supported. Historical pricing data is stored as well.

Check out Real-Time Crawler in action for extracting data from e-commerce sites.

Using Real-Time Crawler for search engines

As with e-commerce websites, Real-Time Crawler is currently customized to support the most popular search engines. You can retrieve paid and organic SERP data, extract ranking data for any keyword or get monthly search volume data in raw HTML or formatted JSON format.

Real-Time Crawler for search engines allows you to discover the most profitable keywords and track their performance. It supports any number of requests done for any location and any keyword.

Check our Real-Time Crawler in action for extracting data from search engines.

Don’t forget that if you have specific data collection needs, we can build a custom solution or adapt our current system to your needs.

Benefits of using Real-Time Crawler for data extraction and analysis

So, we already learned that with Real-Time Crawler, or simply a real time web scraping solution, you can extract all kinds of data from search engines and e-commerce websites. However, if you still think whether to use Real-Time Crawler or not, these are the top three reasons why you should go ahead and do it.

100% success rate

Real-Time Crawler employs a large IP pool and has an advanced IP backup system which allows you to extract all the necessary data without any delays or errors. You can expect a 100% success rate and 100% data delivery.

Cost saving

Building your own data collection solution takes time, money, knowledge, and requires a handful of high-skilled IT professionals working full-time. You can save on all of that by forwarding data collection tasks to Real-Time Crawler. You won’t need so many powerful servers, your costs for infrastructure will be lower, and you’ll be able to transfer your human resources to new opportunities.

Easy to use

Using Real-Time Crawler is actually very straightforward. You can simply provide it with a URL, and it will return you a well-formatted data that can be handled by your backend or even your frontend application framework.

Why other companies use Real-Time Crawler

Our quarterly data shows that more and more companies are increasing the efficiency of data collection and try to reduce their costs. So, instead of maintaining expensive proxy infrastructure, they choose to use Real-Time Crawler.

In the two trend graphs below, you can see an increase in traffic sent through Real-Time Crawler in Q3 of 2018.

Total traffic

According to our team member Mantė, who is the Head of Account Management here at Oxylabs, Real-Time Crawler is a game changer in today’s big data industry.

Real-Time Crawler has proved to be a great service helping companies that want to focus on data analysis rather than data gathering. I highly recommend our solution for those, who have not tried it yet.

Mante, Head of Account Management at Oxylabs

Instead of constantly trying to avoid bot detection and keeping track of site layout changes, companies can just focus on crunching the data they get from Real-Time Crawler.

Additional bonus: you can scale up as much as you like, whenever you need to.

Since Real-Time Crawler enables effortless web data extraction from search engines & e-commerce websites, most of our clients use it for pricing intelligence and SEO monitoring. Let’s find out why.

SEO monitoring: why Real-Time Crawler is better than data center proxies

Real Time Crawler vs Data Center Proxies

As you can see, Real-Time Crawler has many benefits which make it especially well-fitted for search engines. Pricing is optimized, as you’re paying per page and not per IP or traffic. Implementation is simple, you won’t experience any IP blocks, and only minor server maintenance will be needed.

Residential proxy pool is not included in this comparison because scraping search engines consumes a lot of traffic, making residential proxies the least cost-efficient option (as you’re paying per data traffic, not per IP).

Pricing intelligence: why should you pick Real-Time Crawler over residential or data center proxies

Real Time Crawler vs Residential proxies & Data center proxies

We recommend using Real-Time Crawler for pricing intelligence instead of residential or data center proxies because it’s simply easier to do so. It’s easy to integrate, super reliable, easily scalable and cost-efficient.


So, to sum up, if you’re in a business of extracting data from search engines or large e-commerce websites, Real-Time Crawler can be a game changer. If you want to learn more, just drop us a line via live chat or email. We’re always here to help.

Want more content like this? Subscribe to our newsletter to receive our blog posts and exclusive email-only content.


About Aleksandras Sulzenko

Aleksandras Sulzenko is an Account Manager at Oxylabs. He places a strong interest in numbers, especially when it comes to analyzing data to help materialize good ideas on the spot. As for writing, he speaks 5 languages (body language included), so it’s safe to say he is quite proficient in linguistic arts as well. So feel free to contact him in any of the 5 languages if you have any proxy related questions.

Related articles

Scraping the Web With 100% Success Rate

Scraping the Web With 100% Success Rate

Oct 10, 2019

6 min read

Planning a Project on Web Scraping

Planning a Project on Web Scraping