Skip to main content

Learn more about Juan's presentation

The Cost of Solving the Wrong Problem: Data-Driven Decisions for Web Access

When operating at a scale of more than 5,000 crawlers running every day, teams need to prioritize the challenges that matter most. Time and resources should focus on initiatives that can deliver significant impact across the entire crawler fleet, not just a single crawler.

Deciding where to focus must therefore be guided by data and metrics. To support this, Kieron and Juan track the technologies they encounter, including Akamai and DataDome, and the impact they have on operations.

  • Explore the technical considerations behind system-wide decision-making

  • Learn from successful and failed investments in web access R&D

  • Discover practical strategies for addressing challenges at scale