The time has come to publish the third part of Oxylabs’ 2020 trend report. In this article, we go over three of the biggest e-commerce and also web scraping events for a couple of consecutive years now. These events: Valentine’s Day, Mother’s day and the biggest event for anyone working in either brick and mortar or online commerce businesses – Black Friday. Read part 2 of our report Data Center Proxies here!
We encourage you to download the full report which you can do simply by filling in your email in the form below:
Web Scraping Trends Using Data Center Proxies in E-commerce Market: 2017-2019 Case Studies
Businesses from the e-commerce market choose data center proxies to carry out various data gathering operations. They are regularly used for market research, brand protection, pricing intelligence or SEO monitoring, just to name a few use cases. Data center proxies are more affordable to buy in bulk for collecting vast amounts of relevant data from all types of sources. Although data center proxies are easier to block, this risk can be mitigated with IP rotation and other protective strategies.
Over the years, while following the web scraping trends of e-commerce market players, we have noticed some changes in scraping behavior. In 2019 more careful and diligent web scraping practices were abandoned by some companies, most likely due to their success the year before.
Data seems to suggest that these companies assumed that the strategy of using more proxies was unnecessarily careful and they then experimented with a different approach that ultimately resulted in much higher block rates compared to prior years.
As you can see in the request and traffic volume graphs below, there is clear correlation in growth around major e-commerce events of the year. We will discuss these trends further in the report.
Looking at both of these graphs above, in the case of Valentine’s Day, for example, even though the request growth in 2019 was significant during the time around the event, the spike in traffic was not as big. From this we can assume that in 2019 the clients, based on prior experience, optimized their data gathering procedures, requiring less data to extract relevant insights.
Nevertheless, we can see that both for Mother’s Day and Valentine’s Day, retail sites intensified their traffic to prepare for both events in 2019. An especially significant rise of both requests and traffic is seen before Black Friday, indicating the extensive preparations made beforehand.
Case study 1: Valentine’s Day
The first big event that hits the e-commerce market every year is Valentine’s day. Days prior to February 14th, e-commerce websites start preparing in advance by scraping relevant sources.
For example, in 2018 on Valentine’s Day, clients increased their proxy resources by an average of 13.03%, but they also increased traffic per IP by 27.71%, which likely led to 12.14% higher block rate on average.
On Valentine’s Day 2019, Oxylabs e-commerce clients decided to increase their web scraping resources by just 6.79% and raise the traffic per IP by 26.61%, which might have led to 51.65% higher block rate. This block rate was more than 4 times higher than in 2018 – a consequence of an insufficient amount of data center proxies used prior to the event.
Case study 2: Mother’s Day
The second huge e-commerce market event of the year – Mother’s Day. Looking at choices made both in 2018, and 2019 – the same mistakes were made as on Valentine’s Day.
In 2018, on Mother’s Day, companies managed to even drop the block rate by 9.34% while still increasing their traffic per IP by 21.83%, most likely because they expanded their data center proxy resources by an average of 32.3%.
In 2019, however, our clients sent 72.86% more traffic per IP despite the fact that the data gathering resources were only increased by 14.37%, which probably led to a 25.41% higher block rate.
Case study 3: Black Friday
Last but not least, the biggest e-commerce event of the year – Black Friday. Every year this event keeps growing, but in 2019 it became the busiest day for online shoppers in the U.S., attracting over 93M people, thus taking the crown from Cyber Monday (±83M).
Looking through the history of Oxylabs’ client web scraping trends, it is easy to conclude that the e-commerce market takes this event very seriously.
On Black Friday that took place on November 24th, 2017, Oxylabs’ clients generated 40.9% more traffic per IP than usual, which most likely resulted in the average block rate jumping from 3.91% to 14.89%. Prior to the event, the web scraping resources were increased by an average of 7.22%.
On November 23rd, Black Friday of 2018, preparations started earlier. Furthermore, before the event, our clients increased their scraping resources by 29.12% and spread their data collection operations through a couple of days. This way, on Black Friday, they generated 31.8% more traffic per IP than usual and raised the block rate by only 7.15%.
This brings us to November 29th, Black Friday of 2019. Companies started their web scraping operations a week prior to the main event by increasing their resourcesby12.67% and spreading out the traffic through that whole week. On Black Friday, the traffic per IP was only raised by9.7% and the block rate remained almost the same (+1.22%).
All in all, what is evident from the case studies analyzed above – to carry out successful data-gathering operations with data center proxies, businesses ought to prepare in advance. Not only is it essential to spread out more intense web scraping through multiple days leading up to the event, but, also, to prepare better by upscaling data center proxy resources so that the significant increase in traffic could be handled without issues.
Going further, we present graphs indicating the request and traffic volume forecasts for 2020. The estimates were made on the assumption of similar growth continuing into 2020.
Unsurprisingly, our Data Analysis Department predicts that similar growth will happen around significant events such as Valentine’s Day, Mother’s Day, and Black Friday. Assuming similar scraping behavior, we will see more requests before, during, and after Valentine’s Day. With Mother’s Day, it is likely that the request rate will once again grow after the event, whereas for Black Friday, it is safe to assume that e-commerce market players will once again prepare in advance and start their operations before the major event.
As was the case with the forecast for the number of requests for the year 2020 for both data center and residential proxies, we predict similar growth trends for traffic volume. The only exception can be predicted in the case of Valentine’s Day, where the traffic in 2019 was not as significant as with the other two events, and we believe the same will repeat in 2020.
Read the third part of our report “Case Studies” here!