Oxylabs’ 2020 Trend Report: Real-Time Crawler

The fifth installment of our The Rising Demand for Data: Oxylabs’ 2020 Trend Report has arrived! This time it’s all about how Oxylabs clients used our Real-Time Crawler. We invite you to read our previous post “Residential Proxies” here!

For those in a hurry, we’ve written down the key takeaways below. However, we recommend acquiring in-depth knowledge by reading our full report. It can be easily accessed by typing in your email in the form below and downloading the file.

• In 2019, there was a 97.6% growth in the number of requests via Real-Time Crawler (now known as three distinct tools – Scraper APIs.)
• Compared to the prior year, the number of requests in January 2019 was higher by a staggering 454.1%. This is a clear indication of the increasing overall demand for automated web data gathering tools.
• Comparing 2019 with historical data, the number of requests increased by 71% for search engines; 260% for e-commerce websites.

There are two major spikes that should be taken into account. One was around mid-July and the other – in August. The former might have been caused by e-commerce retailers launching their sales events, similar to those of Black Friday. 

As for the spike in August, it most likely occurred due to the summer holidays ending and the new school year beginning. A prime time for parents to stock up on uniforms, stationery, and other school year essentials. Real-Time Crawler was used to scrape the competition for retail sites to gain competitive pricing intelligence and provide their customers with the best deals. 

However, the highest number of requests can be seen in October 2019. This increase is most likely due to the biggest event of the year in Western countries – Christmas. It is widely reported that the business preparations for the biggest season of Christmas shopping start in Autumn. Hence, retail sites also begin getting ready. 

Other massive retail events that need to be taken into consideration together with Christmas were Black Friday and Cyber Monday. These cyber holidays undoubtedly impacted the request spike seen in the graph above. 

These request spikes reappear annually, and each year the number gradually increases, so it is safe to assume that they are to be expected in 2020 too, with additional growth likely.

The future of web scraping tools

With Oxylabs’ scraping solutions, we continue seeing the growth of requests rise and slowly stabilize, indicating the overall demand for automated web scraping tools in the market.

The clearly favorite use cases for Real-Time Crawler continue to be e-commerce websites and search engines, confirming that the demand for SERP and pricing data is becoming crucial for business’ development.

In the words of our client, Aurélien Remy Dionisi, the CEO of Digidian, Real-Time Crawler proves to be a “Very good alternative to proxies,” he continues:
“We have replaced our old proxy-based infrastructure with the Real-Time Crawler solution provided by Oxylabs. The service is very fast and reliable. The results we obtained by now are far better than the old ones. We are very happy with the product, and so are our customers.”

Read part 6 “Regions” here!