Skip to main content
Back to blog

Is the Web Closing? Perspectives from OxyCon 2026

Narmin Mammadova

Narmin Mammadova

Last updated on

2026-10-30

5 min read

OxyCon 2026, which happened in Vilnius, closed with a panel on a question that cuts across journalism, research, and business alike: is the web actually closing, or are we overreacting? The people on stage came at it from very different angles. Denas Grybauskas, Chief Governance and Strategy Officer at Oxylabs at the time and now at Source, led the discussion. Krishna Sood from Black Forest Labs spoke for the AI developers trying to build in Europe, Carl Miller, co-founder and Research Director of the Centre for the Analysis of Social Media at Demos, founder of CASM Technology, drew on 15 years of studying online harms. And Paul Bradshaw, from the BBC Shared Data Unit and a teacher at Birmingham City University, brought the data journalism angle.

To sum up the core discussion in one line: something real is happening, and it goes beyond a technical issue. Access to public web data is getting harder, and the pressure is coming from three directions at once: the courts, government policy, and the technical walls websites keep building.

The signals worth watching

The panel started with a few data points. According to the Data Provenance Initiative, the number of websites blocking access at scale has grown exponentially over the past five years. On top of that, internet standard-setting bodies are pushing new standards covering things like how bots identify themselves and new opt-out rules. These are useful conversations, though Denas Grybauskas questioned whether they represent the whole economy or mostly line up with the roadmaps of the biggest tech companies.

Trends for Robots.txt and Terms of Service

Litigation is the third signal

Based on figures from ChatGPT is Eating the World, AI and copyright lawsuits in the US have nearly tripled in 14 months, from around 45 to close to 150 by September 2026. 

Technical friction, regulatory pressure, and now a wave of litigation are all pointing in the same direction.

Denas Grybauskas, Chief Governance and Strategy Officer at Source

Three perspectives on the same problem

Krishna Sood, who leads product legal at Black Forest Labs (the Freiburg-based company behind the FLUX models), spoke as one of the few people building frontier AI in Europe. 

Her point was that the data still exists, but it's getting harder to reach. 

While data exists, it is increasingly locked behind logins, paywalls, and platform walled gardens.

Krishna Sood, Product Legal at Black Forest Labs

She said, describing how many of those platforms build their own models and sit in the US or China. 

That creates an asymmetry where even serious European AI labs struggle to access the data they need.

Carl Miller, co-founder of the Centre for the Analysis of Social Media, has spent 15 years researching online harms like radicalization and disinformation, and he pointed to a bitter irony. 

Fifteen years ago researchers had open APIs and a crawlable web but weak analytical tools; now the tools are powerful and the data access has shrunk. His warning was direct:

In an era of information warfare where democratic institutions are actively targeted, you cannot counter threats you cannot observe.

Carl Miller, Fellow & Co-Founder at Centre for the Analysis of Social Media

Paul Bradshaw, a data journalist with the BBC Shared Data Unit who teaches at Birmingham City University, offered a different lens. He didn't dispute that websites are restricting more, but he pointed out another way to look at how access has changed.

The open web many of us remember was only ever open to people who could code, and for most journalists and academics, scraping was out of reach. In that sense, AI is opening things up by letting more people build and run scrapers the way only coders and scraping experts could before.

So whether the web is closing or opening really depends on your vantage point.

Paul Bradshaw, Journalist & Academic at BBC, Birmingham City University

Europe’s research-access gap

This part of the discussion cuts to the heart of why public-interest research is struggling. EU law, the Digital Services Act, and UK law, the Online Safety Act, both require major platforms to give vetted researchers access to data for public-interest research. Carl mentioned, "To answer plainly: no, it is not happening effectively." Researchers face endless onboarding delays, broken APIs, and unresponsive channels, and in the UK the implementation guidelines have stalled.

The knock-on effect is that researchers can't even assess whether platform-provided data would be trustworthy, because they can't get it in the first place. That keeps independent scraping essential to even begin to answer urgent questions, like whether algorithmic feeds amplify extremist content.

Paul gave a concrete example: an ITV journalist, a former student of his, partnered with Oxylabs Project 4β to gather public data during the UK civil unrest following the tragic deaths of children, tracing how misinformation spread. He also drew a parallel to the Open Data movement 15 years ago, when governments released the datasets they were comfortable with and kept the sensitive ones back.

We must pay close attention to which specific segments of the web remain accessible.

Paul Bradshaw, Journalist & Academic at BBC, Birmingham City University

On a more hopeful note, a growing coalition of European safety researchers is now pooling evidence of these roadblocks to push for proper compliance.

The problem with licensing everything

One proposed fix is moving entirely to commercial licensing, and the panel pushed back. Krishna argued that licensing makes sense for private, proprietary assets, but public web data should stay open for analysis. 

Factual information cannot and should not be monopolized.

Krishna Sood, Product Legal at Black Forest Labs

Relying only on bilateral licensing deals favors big incumbents, since the contracts are slow and expensive to negotiate, leaving small developers stuck with whatever niche data they can afford and, as a result, worse, more biased models.

Her example made it land: early facial recognition systems failed on her because the training data lacked enough representation of darker skin tones. Building AI that works fairly across languages, regions, and communities needs broad, representative data. 

Paul connected it to journalism, where publishers need revenue but society benefits when information spreads widely. 

When access is governed purely by large commercial licensing agreements, well-resourced media organizations and dominant tech platforms capture the market, while minority viewpoints and independent outlets are left out of the equation.

Paul Bradshaw, Journalist & Academic at BBC, Birmingham City University

This is bigger than business

Carl closed with the heaviest point. About a decade ago, defense organizations started treating information as a theater of war alongside land, sea, air, and space. Hostile state actors, including Russian and Chinese hybrid operations, see the information domain as a key arena for weakening democracies. That fight used to play out on social media; now AI systems are the next target, probed and manipulated at scale to shift public perception. 

Independent access to public web data is not just a commercial or technical matter; it is foundational infrastructure for defending open democracies against authoritarian influence.

Carl Miller, Fellow & Co-Founder at Centre for the Analysis of Social Media

Key takeaways

The takeaway from the panel was clear: we have to keep making the case for public data access, out loud and often. This isn't a fight you win once and for all. Pressure from courts, policy, and technical barriers keeps building, and we have to make the case for open access again and again, in rooms where these decisions actually get made.

The web may not be closing for everyone at once, but the people who use it to hold power to account, study real harms, and fight misinformation are feeling the doors swing shut. Those are some of the uses that matter most for the public good, and they need the commercial muscle to push back together.

Join 4β

That gap is exactly why Project 4β exists. We provide free public web data collection tools, technology, and legal advisory to researchers, journalists, academics, NGOs, and nonprofits working on public-interest missions, so the people doing this work aren't the ones locked out.

Much of it falls within the areas this panel covered. On information warfare, democratic resilience, and investigative journalism, we already work with organizations like Bellingcat, Global Witness, Debunk.org, Doublethink Lab, and the Civic Resilience Initiative. On the academic side, we support researchers at universities including Stanford University, Carnegie Mellon University, Northwestern University, and Warwick University.

If your work depends on getting to public web data, whether you're tracing how false narratives spread, auditing algorithmic harms, or building research that defends open democracies, we'd like to help. Academics, NGOs, NPOs, and other mission-driven organizations can join the initiative and get free access to tools that match the scale of what they're trying to do.

Tell us about your data needs through our form or via form or at 4beta@oxylabs.io.

Forget about complex web scraping processes

Choose Oxylabs' advanced web intelligence collection solutions to gather real-time public data hassle-free.

About the author

Narmin Mammadova

Narmin Mammadova

PR Content Manager

Narmin is the PR Content Manager for Project 4β at Oxylabs. She enjoys the challenge of getting people to care, and pro bono work gives her good stories to tell. In her spare time, she travels whenever possible or indulges her love of poetry and reciting.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Related articles

Oxylabs’ Project 4β Named a Finalist at the 6th Annual Anthem Awards
Oxylabs’ Project 4β Named a Finalist at the 6th Annual Anthem Awards
Narmin Mammadova

Narmin Mammadova

2026-10-27

Oxylabs logo
Security Update: October 2026 Incident Report
Vaidotas Šedys

Vaidotas Šedys

2026-10-27

Our New Dashboard Is Live: One Balance, More Possibilities
Donata Norkūnaitė avatar

Donata Norkūnaitė

2026-10-21

Get the latest news from data gathering world

Scale up your business with Oxylabs®

Forget about complex web scraping processes

Choose Oxylabs' advanced web intelligence collection solutions to gather real-time public data hassle-free.