Oxylabs Announces Partnership With Student-led TrackCorona Website

Learn more
avatar

Simonas Svegzda

Mar 02, 2020 8 min read

The time has come to publish the third part of Oxylabs’ 2020 trend report. In this article, we go over three of the biggest e-commerce and also web scraping events for a couple of consecutive years now. These events: Valentine’s Day, Mother’s day and the biggest event for anyone working in either brick and mortar or online commerce businesses – Black Friday. Read part 2 of our report Data Center Proxies here!

We encourage you to download the full report which you can do simply by filling in your email in the form below:

The Rising Demand for Data: Oxylabs’ 2020 Trend Report

!

This field is required

Thank you!

Please check your inbox.

We will use your email to send you a link to the PDF of our market report. We will also provide you with information on Oxylabs’ services that may be of interest to you. Be sure that you can opt-out from any marketing related communications that we send you at any time. For more information on your rights and data use please read our Privacy Policy.

Web Scraping Trends Using Data Center Proxies in E-commerce Market: 2017-2019 Case Studies

Businesses from the e-commerce market choose data center proxies to carry out various data gathering operations. They are regularly used for market research, brand protection, pricing intelligence or SEO monitoring, just to name a few use cases. Data center proxies are more affordable to buy in bulk for collecting vast amounts of relevant data from all types of sources. Although data center proxies are easier to block, this risk can be mitigated with IP rotation and other protective strategies.  

Over the years, while following the web scraping trends of e-commerce market players, we have noticed some changes in scraping behavior. In 2019 more careful and diligent web scraping practices were abandoned by some companies, most likely due to their success the year before. 

Data seems to suggest that these companies assumed that the strategy of using more proxies was unnecessarily careful and they then experimented with a different approach that ultimately resulted in much higher block rates compared to prior years. 

As you can see in the request and traffic volume graphs below, there is clear correlation in growth around major e-commerce events of the year. We will discuss these trends further in the report.

Data center proxies’ requests by week, 2018-2019 data. Source: Oxylabs
Data center proxies’ traffic by week, 2018-2019 data. Source: Oxylabs

Looking at both of these graphs above, in the case of Valentine’s Day, for example, even though the request growth in 2019 was significant during the time around the event, the spike in traffic was not as big. From this we can assume that in 2019 the clients, based on prior experience, optimized their data gathering procedures, requiring less data to extract relevant insights.

Nevertheless, we can see that both for Mother’s Day and Valentine’s Day, retail sites intensified their traffic to prepare for both events in 2019. An especially significant rise of both requests and traffic is seen before Black Friday, indicating the extensive preparations made beforehand.

Case study 1: Valentine’s Day

The first big event that hits the e-commerce market every year is Valentine’s day. Days prior to February 14th, e-commerce websites start preparing in advance by scraping relevant sources. 

For example, in 2018 on Valentine’s Day, clients increased their proxy resources by an average of 13.03%, but they also increased traffic per IP by 27.71%, which likely led to 12.14% higher block rate on average.

On Valentine’s Day 2019, Oxylabs e-commerce clients decided to increase their web scraping resources by just 6.79% and raise the traffic per IP by 26.61%, which might have led to 51.65% higher block rate. This block rate was more than 4 times higher than in 2018 – a consequence of an insufficient amount of data center proxies used prior to the event.  

Case study 2: Mother’s Day

The second huge e-commerce market event of the year – Mother’s Day. Looking at choices made both in 2018, and 2019 – the same mistakes were made as on Valentine’s Day. 

In 2018, on Mother’s Day, companies managed to even drop the block rate by 9.34% while still increasing their traffic per IP by 21.83%, most likely because they expanded their data center proxy resources by an average of 32.3%.

In 2019, however, our clients sent 72.86% more traffic per IP despite the fact that the data gathering resources were only increased by 14.37%, which probably led to a 25.41% higher block rate.

Case study 3: Black Friday

Last but not least, the biggest e-commerce event of the year – Black Friday. Every year this event keeps growing, but in 2019 it became the busiest day for online shoppers in the U.S., attracting over 93M people, thus taking the crown from Cyber Monday (±83M). 

Looking through the history of Oxylabs’ client web scraping trends, it is easy to conclude that the e-commerce market takes this event very seriously.

On Black Friday that took place on November 24th, 2017, Oxylabs’ clients generated 40.9% more traffic per IP than usual, which most likely resulted in the average block rate jumping from 3.91% to 14.89%. Prior to the event, the web scraping resources were increased by an average of 7.22%.

On November 23rd, Black Friday of 2018, preparations started earlier. Furthermore, before the event, our clients increased their scraping resources by 29.12% and spread their data collection operations through a couple of days. This way, on Black Friday, they generated 31.8% more traffic per IP than usual and raised the block rate by only 7.15%.
This brings us to November 29th, Black Friday of 2019. Companies started their web scraping operations a week prior to the main event by increasing their resourcesby12.67% and spreading out the traffic through that whole week. On Black Friday, the traffic per IP was only raised by9.7% and the block rate remained almost the same (+1.22%).

Lessons learned 

All in all, what is evident from the case studies analyzed above – to carry out successful data-gathering operations with data center proxies, businesses ought to prepare in advance. Not only is it essential to spread out more intense web scraping through multiple days leading up to the event, but, also, to prepare better by upscaling data center proxy resources so that the significant increase in traffic could be handled without issues.

Going further, we present graphs indicating the request and traffic volume forecasts for 2020. The estimates were made on the assumption of similar growth continuing into 2020.

Data center proxies’ requests forecast for 2020 by week. Source: Oxylabs
Residential proxies’ requests forecast for 2020 by week. Source: Oxylabs

Unsurprisingly, our Data Analysis Department predicts that similar growth will happen around significant events such as Valentine’s Day, Mother’s Day, and Black Friday. Assuming similar scraping behavior, we will see more requests before, during, and after Valentine’s Day. With Mother’s Day, it is likely that the request rate will once again grow after the event, whereas for Black Friday, it is safe to assume that e-commerce market players will once again prepare in advance and start their operations before the major event.

Data center proxies’ traffic forecast for 2020 by week. Source: Oxylabs Data Analysis Department
Residential proxies’ traffic forecast for 2020 by week. Source: Oxylabs

As was the case with the forecast for the number of requests for the year 2020 for both data center and residential proxies, we predict similar growth trends for traffic volume. The only exception can be predicted in the case of Valentine’s Day, where the traffic in 2019 was not as significant as with the other two events, and we believe the same will repeat in 2020.

Read the third part of our report “Case Studies” here!

avatar

About Simonas Svegzda

Simonas Svegzda is a Content Manager at Oxylabs. Early on, he developed an interest in blogging, online media, internet culture, and what makes the online world spin (you guessed it, it's content). When he’s not fully immersed in extraordinary cyberspace adventures (not giggling at memes), you’ll probably find him at a live music gig or reading fiction.

Related articles

Proxy vs VPN

Proxy vs VPN

Mar 18, 2020

8 min read

What Is a Proxy: Web Scraping Basics [2020 Guide]

What Is a Proxy: Web Scraping Basics [2020 Guide]

Mar 17, 2020

25 min read

Gathering Data for Sales Intelligence

Gathering Data for Sales Intelligence

Mar 10, 2020

6 min read

All information on Oxylabs Blog is provided on an “as is” basis and for informational purposes only. We make no representation with respect to the accuracy, completeness, availability or otherwise of any information contained on Oxylabs Blog / any third-party websites that may be linked therein. We disclaim all liability for any errors, omissions or delays in such information and for any losses, injuries or damages that may arise out of its use.