OxyCon 2025: Key Takeaways

OxyCon 2025 brought the global web scraping community together once again – this time under the bold theme “Power Up for the AI-Driven Era of Web Scraping.”

The event explored how AI is transforming scraping workflows, driving smarter data use, and reshaping the industry as we know it. From technical deep dives to strategic insights, speakers tackled everything from resilient infrastructure and adaptive scrapers to the growing need for ethical data practices in a rapidly evolving digital landscape.

But there’s still a way forward. In his words, scraping isn’t just about gathering information – it’s about keeping the future of AI open, diverse, and innovative.

From Chaos to Clarity: Data Structuring in Large-Scale Scraping

While unblocking websites often takes center stage in web scraping discussions, another challenge looms quietly behind the scenes – data structuring. At scale, it's one of the biggest bottlenecks teams face, and in his presentation, Aleksandras Šulženko, Product Owner at Oxylabs, made the case that structuring may be even trickier than unblocking.

To highlight this, Aleksandras walked through a typical large-scale scraping workflow – from URL building and content unblocking to data delivery and processing. But the spotlight was on how Oxylabs’ Web Scraper API helps structure data more efficiently and reliably, especially when scraping operations grow in complexity.

The newest Oxylabs feature, Self-healing parser presets, stood out. It’s a self-healing logic that adapts to site changes over time. It allows users to scale structured data consumption in no time, as well as outsource maintenance. When asked about the future of parsers, Aleksandras shared that this is the future – better output, less maintenance, more datapoints, and more structured data for everyone.

To wrap up the presentation, Aleksandras showed a live demo of this feature, demonstrating just how far data structuring has come – and how much easier it is to maintain clarity in even the most chaotic scraping setups.

Fred broke down the challenges of location-based scraping – such as identifying stores, tracking new openings and closures, and dynamically switching scraping sources as needed. His team’s efforts have successfully scaled operations to 4,500 stores for a single retailer, showcasing the flexibility needed to adapt to ever-changing data sources.

But scaling doesn’t stop at data acquisition. To truly succeed, monitoring at scale is essential. Fred unveiled how NielsenIQ’s systems track tens of KPIs across thousands retailers and stores, automating issue detection and response handling. From spotting anomalies in real-time to creating tickets and assigning them for action with minimal human intervention, the workflow is an amazing example of efficiency at scale.

Creating an AI-Powered Price Comparison Tool With Cursor and Oxylabs’ AI Studio

Gone are the days when building a price comparison tool required advanced coding expertise or extensive knowledge of scraping and parsing. In his session, Rytis Ulys, Head of Data & Analytics at Oxylabs, showcased the transformative capabilities of AI-powered tools, illustrating how even those with limited coding experience can create sophisticated web data-based systems. Using Cursor (an AI Code Editor) and Oxylabs' groundbreaking AI Studio, he developed a fully functional price comparison tool – with only minimal manual coding required.

Throughout the demo, Rytis walked the audience through the simplicity of integrating web data into personal or professional projects. Starting with a high-level overview of the architecture and the two AI Studio apps (AI-Scraper and Browser Agent), he demonstrated how tasks that traditionally required complex scripting – such as scraping, parsing, and data analysis – are now effortlessly automated.

With AI doing the heavy lifting, Rytis illustrated how anyone can bypass the technical hurdles of web scraping and focus on bringing their projects to life.

Looking ahead, he sees a near future of self-healing scrapers powered by LLMs and human supervision. And while AI handles the parsing, he reminded the audience: you still need to reach the data – so proxies and unblockers aren’t going anywhere.

Advanced Web Scraping: Techniques to Stay Unblocked

To wrap up OxyCon 2025, Juras Juršėnas, COO at Oxylabs, moderated a panel discussion featuring five industry experts who unpacked the evolving challenges and innovations in web scraping amid growing anti-bot defenses.

The panel explored how anti-bot defenses have become more sophisticated across multiple layers – from TLS fingerprinting and HTTP/2 quirks to advanced IP reputation scoring and dynamic content blocking. Tools like Playwright remain central, but even they require frequent patching due to browser leakage and detection. Panelists shared how LLMs are becoming valuable allies and that scraping at scale now requires not just better tools, but custom strategies and in-house infrastructure like reverse proxies, alerting pipelines, and domain-specific monitoring.

But unblocking isn’t just technical – it's about people. The conversation highlighted the need for collaborative teams, a hacker mindset, and knowing when to ask for help. Building a strong R&D culture and hiring curious, puzzle-loving developers was framed as essential for success.

On a broader level, the “open internet” debate took center stage, questioning why smaller scrapers face harsher restrictions than major AI players. This topic of fairness and accessibility underscored the need for ethical frameworks and industry collaborations to ensure a level playing field.

As anti-bot technology advances, so must scrapers – through smarter tech, stronger teams, and cross-industry collaboration.