Oxylabs’ 2020 Trend Report: Real-Time Crawler

Gabija Fatenaite

2020-03-043 min read

The fifth installment of our The Rising Demand for Data: Oxylabs’ 2020 Trend Report has arrived! This time it’s all about how Oxylabs clients used our Real-Time Crawler. We invite you to read our previous post “Residential Proxies” here!

For those in a hurry, we’ve written down the key takeaways below. However, we recommend acquiring in-depth knowledge by reading our full report. It can be easily accessed by typing in your email in the form below and downloading the file.

• In 2019, there was a 97.6% growth in the number of requests via Real-Time Crawler (now known as three distinct tools – Scraper APIs.)
• Compared to the prior year, the number of requests in January 2019 was higher by a staggering 454.1%. This is a clear indication of the increasing overall demand for automated web data gathering tools.
• Comparing 2019 with historical data, the number of requests increased by 71% for search engines; 260% for e-commerce websites.

Free PDF

The Rising Demand for Data: Oxylabs’ 2020 Trend Report

The rising need for scraping tools: Real-Time Crawler

Research carried out by Oxylabs Data Analysis Department has found that in 2019, there was a staggering 97.6% request growth registered via Real-Time Crawler, compared to 2018 recorded data.

Compared to the prior year, the number of requests in January 2019 started at a staggering 454.1%. Later throughout the year, the number went down and stabilized, finally reaching 20.72% in December 2019, as seen in the graph below.

This is a clear indication of the overall demand for automated web data gathering tools such as Real-Time Crawler over the past year within the web scraping market.

Massive Real-Time Crawler request growth

Real-Time Crawler requests change by month, 2019 vs. 2018 data. Source: Oxylabs Data Analysis Department

Most popular targets: search engines and e-commerce websites

Real-Time Crawler can be separated into two main verticals: search engines and e-commerce websites. Comparing 2019 with historical data, the number of requests increased by:

71% for search engines;
260% for e-commerce websites.

The impressive growth of search engine, and especially e-commerce website requests indicate that the demand for SERP and pricing data is increasing due to this intelligence becoming more valuable for the development of business. As such data is time-sensitive, and the competition is continuously growing, the best chance to stay one step ahead is by applying real-time scraping strategies.

By using Real-Time Crawler, our clients are increasing the efficiency of data collection and significantly reducing their spending on resources. This enables companies to focus on data analysis rather than data scraping and thus overcome the market competition.

Most popular months

As we can see from Real-Time Crawlers’ request graph below, there was a substantial number of requests processed via Real-Time Crawler from 2018 to 2020.

Trends indicate growing interest in Oxylabs Real-Time Crawler

Real-Time Crawler requests by month, 2018-2019 data. Source: Oxylabs Data Analysis Department

There are two major spikes that should be taken into account. One was around mid-July and the other – in August. The former might have been caused by e-commerce retailers launching their sales events, similar to those of Black Friday.

As for the spike in August, it most likely occurred due to the summer holidays ending and the new school year beginning. A prime time for parents to stock up on uniforms, stationery, and other school year essentials. Real-Time Crawler was used to scrape the competition for retail sites to gain competitive pricing intelligence and provide their customers with the best deals.

However, the highest number of requests can be seen in October 2019. This increase is most likely due to the biggest event of the year in Western countries – Christmas. It is widely reported that the business preparations for the biggest season of Christmas shopping start in Autumn. Hence, retail sites also begin getting ready.

Other massive retail events that need to be taken into consideration together with Christmas were Black Friday and Cyber Monday. These cyber holidays undoubtedly impacted the request spike seen in the graph above.

These request spikes reappear annually, and each year the number gradually increases, so it is safe to assume that they are to be expected in 2020 too, with additional growth likely.

The future of web scraping tools

With Oxylabs’ scraping solutions, we continue seeing the growth of requests rise and slowly stabilize, indicating the overall demand for automated web scraping tools in the market.

The clearly favorite use cases for Real-Time Crawler continue to be e-commerce websites and search engines, confirming that the demand for SERP and pricing data is becoming crucial for business’ development.

In the words of our client, Aurélien Remy Dionisi, the CEO of Digidian, Real-Time Crawler proves to be a “Very good alternative to proxies,” he continues:
“We have replaced our old proxy-based infrastructure with the Real-Time Crawler solution provided by Oxylabs. The service is very fast and reliable. The results we obtained by now are far better than the old ones. We are very happy with the product, and so are our customers.”

Read part 6 “Regions” here!

About the author

Gabija Fatenaite

Lead Product Marketing Manager

Gabija Fatenaite is a Lead Product Marketing Manager at Oxylabs. Having grown up on video games and the internet, she grew to find the tech side of things more and more interesting over the years. So if you ever find yourself wanting to learn more about proxies (or video games), feel free to contact her - she’ll be more than happy to answer you.

Learn more about Gabija Fatenaite

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.