The fifth installment of our The Rising Demand for Data: Oxylabs’ 2020 Trend Report has arrived! This time it’s all about how Oxylabs clients used our Real-Time Crawler. We invite you to read our previous post “Residential Proxies” here!
For those in a hurry, we’ve written down the key takeaways below. However, we recommend acquiring in-depth knowledge by reading our full report. It can be easily accessed by typing in your email in the form below and downloading the file.
• In 2019, there was a 97.6% growth in the number of requests via Real-Time Crawler.
• Compared to the prior year, the number of requests in January 2019 was higher by a staggering 454.1%. This is a clear indication of the increasing overall demand for automated web data gathering tools such as Real-Time Crawler.
• Real-Time Crawler can be separated into two main verticals: search engines and e-commerce websites. Comparing 2019 with historical data, the number of requests increased by 71% for search engines; 260% for e-commerce websites.
The Rising Need for Scraping Tools: Real-Time Crawler
Research carried out by Oxylabs Data Analysis Department has found that in 2019, there was a staggering 97.6% request growth registered via Real-Time Crawler, compared to 2018 recorded data.
Compared to the prior year, the number of requests in January 2019 started at a staggering 454.1%. Later throughout the year, the number went down and stabilized, finally reaching 20.72% in December 2019, as seen in the graph below.
This is a clear indication of the overall demand for automated web data gathering tools such as Real-Time Crawler over the past year within the web scraping market.
Most popular targets: search engines & e-commerce websites
Real-Time Crawler can be separated into two main verticals: search engines and e-commerce websites. Comparing 2019 with historical data, the number of requests increased by:
- 71% for search engines;
- 260% for e-commerce websites.
The impressive growth of search engine, and especially e-commerce website requests indicate that the demand for SERP and pricing data is increasing due to this intelligence becoming more valuable for the development of business. As such data is time-sensitive, and the competition is continuously growing, the best chance to stay one step ahead is by applying real-time scraping strategies.
By using Real-Time Crawler, our clients are increasing the efficiency of data collection and significantly reducing their spending on resources. This enables companies to focus on data analysis rather than data scraping and thus overcome the market competition.
Most popular months
As we can see from Real-Time Crawlers’ request graph below, there was a substantial number of requests processed via Real-Time Crawler from 2018 to 2020.
There are two major spikes that should be taken into account. One was around mid-July and the other – in August. The former might have been caused by e-commerce retailers launching their sales events, similar to those of Black Friday.
As for the spike in August, it most likely occurred due to the summer holidays ending and the new school year beginning. A prime time for parents to stock up on uniforms, stationery, and other school year essentials. Real-Time Crawler was used to scrape the competition for retail sites to gain competitive pricing intelligence and provide their customers with the best deals.
However, the highest number of requests can be seen in October 2019. This increase is most likely due to the biggest event of the year in Western countries – Christmas. It is widely reported that the business preparations for the biggest season of Christmas shopping start in Autumn. Hence, retail sites also begin getting ready.
Other massive retail events that need to be taken into consideration together with Christmas were Black Friday and Cyber Monday. These cyber holidays undoubtedly impacted the request spike seen in the graph above.
These request spikes reappear annually, and each year the number gradually increases, so it is safe to assume that they are to be expected in 2020 too, with additional growth likely.
The future of web scraping tools
With Oxylabs’ Real-Time Crawler, we continue seeing the growth of requests rise and slowly stabilize, indicating the overall demand for automated web scraping tools in the market.
The clearly favorite use cases for Real-Time Crawler continue to be e-commerce websites and search engines, confirming that the demand for SERP and pricing data is becoming crucial for business’ development.
In the words of our client, Aurélien Remy Dionisi, the CEO of Digidian, Real-Time Crawler proves to be a “Very good alternative to proxies,” he continues:
“We have replaced our old proxy-based infrastructure with the Real-Time Crawler solution provided by Oxylabs. The service is very fast and reliable. The results we obtained by now are far better than the old ones. We are very happy with the product, and so are our customers.”
Read part 6 “Regions” here!