OxyCast: Podcast on Everything Web Scraping Related

  • Tune in to the hottest topics on public data gathering

  • Hear our personal experiences on web scraping

  • Learn about the best web scraping practices

About OxyCast

One of Oxylabs’ core beliefs is knowledge sharing. Listen to our podcast and learn more about all things web scraping. Find out why web scraping exists, discover solutions to daily challenges, and uncover what’s new in the web scraping community.

Episode 4

Scaling: Overcoming Your Limits

In the 4th episode of OxyCast, our host, Augustinas Kalvis, along with the special guest who has been at Oxylabs since the very beginning, Eivydas Vilčinskas (Tech Team Lead), discuss the ever-complex yet always relevant topic of scaling. From the explanation of what scaling is and why it’s essential to its unique intricacies, difficulties, and viable approaches, this episode looks at all of it.

In this episode, we answer the following questions:

  • How do you go from 10 requests per second to tens of thousands?

  • How do you ensure various bottlenecks are solved, and where might they appear in the first place?

  • Infrastructure and application bottlenecks - when do you encounter them, and what are some of the solutions you could use?

  • Vertical vs. Horizontal scaling, what are their differences, and when to use which?

  • Load balancers, how do they help?

An introduction to the web scraping world

  • The purpose of web scraping

  • The process & challenges of public data gathering

  • Buying vs. building a web scraper

Listen

The most common challenges of web scraping

  • Ensuring reliability & scalability

  • Handling bot detection

  • Tips and tricks on avoiding blocks

Listen

The main principles of data parsing

  • Exploring the process step-by-step

  • Writing a good selector

  • Making the parser scalable

Listen

The intricacies of scaling

  • Boosting the number of requests

  • Solutions to bottlenecks

  • Tips on system design

Listen

Share your input

We are always looking for new thematic angles and people to share them with you. Do you have anything particular in mind? Then go ahead and suggest a topic or refer a guest. Let us know to make it a reality by sending an email!

Contact us: events@oxylabs.io