Scaling: Overcoming Your Limits | OxyCast #4

Danielius Radavicius

2022-04-252 min read

Strap in for a deep dive into some really curious web scraping insights. Episode #4 of OxyCast is out now on multiple platforms such as YouTube, Spotify, Apple Podcasts and SoundCloud!

The episode, hosted by Augustinas Kalvis (Software Developer at Oxylabs), features a true Oxylabs OG, Eivydas Vilčinskas (Tech Team Lead at Oxylabs), and does cover the universally important topic of scaling within software engineering and web scraping.

As usual, you’ll be able to find our newest episodes on whichever platform you prefer:

Now let’s take a closer look at some of the topics you’ll see in this episode of OxyCast:

Scalability in web scraping projects, what is it and why is it important?

Scaling is an immense topic, but its importance cannot be underestimated, irrespective of where it might be applied. The focus of scaling lies in trying to expand existing configurations of computers and/or services to handle an increase in requests. Such a definition may create an image of simplicity, but scaling is anything but that. Various factors such as weighing up cost, effectiveness, and time saved can all massively influence any scaling choice made. Thus, the insights provided by our expert team member, who has consistently worked on this topic for five years, are invaluable and a great way to understand the when, where, and why various intricacies and complexities of scaling appear in web scraping.

Let’s talk scaling

Those who have decided to scale their web scraping requests, or simply want to flirt with the idea, have likely run into a plethora of questions such as:

How do you go from 10 requests per second to tens of thousands?
How do you ensure various bottlenecks are solved, and where might they appear in the first place?
Infrastructure and application bottlenecks, when do you encounter them, and what are some of the solutions you could use?
Vertical vs. Horizontal scaling, what are their differences, and when to use which?

Many other equally relevant questions are asked and answered, so we do highly recommend listening to the whole podcast!

The key of utilizing your resources

As a last note, Augustinas Kalvis and Eivydas Vilčinskas briefly discussed the critical importance of utilizing your resources in an efficient way as you scale, so to excite you about the topic, here’s a little hint of what’s coming up in this episode of OxyCast as said by Eivydas:

Once you’re getting to a much larger scale, at least at a scale where you are starting to get price-conscious, and your economics start to matter a lot more, you’re looking into optimizing your processes, and this optimization where you scale only the particular bottleneck to match your requirements is the more efficient approach.

Eivydas Vilčinskas, Tech Team Lead at Oxylabs

Summing up

We are truly delighted to bring this episode to you as it doesn’t only cover the topic of crucial concern that is scaling but also discusses other related and similarly important areas of load balancers and sufficient network capacities. When it’s all combined all together, there will be plenty of valuable information that could help your future web scraping capabilities, so get ready, strap in, and make some notes!

Before finishing, it should be noted that while we explore some of the most challenging topics in web scraping, we always aim to listen to you, the viewer! Therefore, if you have any questions or topic suggestions, don’t hesitate to contact us at events@oxylabs.io. Maybe your suggested topic will be a future episode of OxyCast!

About the author

Danielius Radavicius

Former Copywriter

Danielius Radavičius was a Copywriter at Oxylabs. Having grown up in films, music, and books and having a keen interest in the defense industry, he decided to move his career toward tech-related subjects and quickly became interested in all things technology. In his free time, you'll probably find Danielius watching films, listening to music, and planning world domination.

Learn more about Danielius Radavicius

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.