

Narmin Mammadova
Last updated on
2026-10-30
5 min read
OxyCon 2026, which happened in Vilnius, closed with a panel on a question that cuts across journalism, research, and business alike: is the web actually closing, or are we overreacting? The people on stage came at it from very different angles. Denas Grybauskas, Chief Governance and Strategy Officer at Oxylabs at the time and now at Source, led the discussion. Krishna Sood from Black Forest Labs spoke for the AI developers trying to build in Europe, Carl Miller, co-founder and Research Director of the Centre for the Analysis of Social Media at Demos, founder of CASM Technology, drew on 15 years of studying online harms. And Paul Bradshaw, from the BBC Shared Data Unit and a teacher at Birmingham City University, brought the data journalism angle.
To sum up the core discussion in one line: something real is happening, and it goes beyond a technical issue. Access to public web data is getting harder, and the pressure is coming from three directions at once: the courts, government policy, and the technical walls websites keep building.
The panel started with a few data points. According to the Data Provenance Initiative, the number of websites blocking access at scale has grown exponentially over the past five years. On top of that, internet standard-setting bodies are pushing new standards covering things like how bots identify themselves and new opt-out rules. These are useful conversations, though Denas Grybauskas questioned whether they represent the whole economy or mostly line up with the roadmaps of the biggest tech companies.

Trends for Robots.txt and Terms of Service

Based on figures from ChatGPT is Eating the World, AI and copyright lawsuits in the US have nearly tripled in 14 months, from around 45 to close to 150 by September 2026.
Technical friction, regulatory pressure, and now a wave of litigation are all pointing in the same direction.
Denas Grybauskas, Chief Governance and Strategy Officer at Source
Krishna Sood, who leads product legal at Black Forest Labs (the Freiburg-based company behind the FLUX models), spoke as one of the few people building frontier AI in Europe.

Her point was that the data still exists, but it's getting harder to reach.
While data exists, it is increasingly locked behind logins, paywalls, and platform walled gardens.
Krishna Sood, Product Legal at Black Forest Labs
She said, describing how many of those platforms build their own models and sit in the US or China.
That creates an asymmetry where even serious European AI labs struggle to access the data they need.
Carl Miller, co-founder of the Centre for the Analysis of Social Media, has spent 15 years researching online harms like radicalization and disinformation, and he pointed to a bitter irony.

Fifteen years ago researchers had open APIs and a crawlable web but weak analytical tools; now the tools are powerful and the data access has shrunk. His warning was direct:
In an era of information warfare where democratic institutions are actively targeted, you cannot counter threats you cannot observe.
Carl Miller, Fellow & Co-Founder at Centre for the Analysis of Social Media
Paul Bradshaw, a data journalist with the BBC Shared Data Unit who teaches at Birmingham City University, offered a different lens. He didn't dispute that websites are restricting more, but he pointed out another way to look at how access has changed.

The open web many of us remember was only ever open to people who could code, and for most journalists and academics, scraping was out of reach. In that sense, AI is opening things up by letting more people build and run scrapers the way only coders and scraping experts could before.
So whether the web is closing or opening really depends on your vantage point.
Paul Bradshaw, Journalist & Academic at BBC, Birmingham City University
This part of the discussion cuts to the heart of why public-interest research is struggling. EU law, the Digital Services Act, and UK law, the Online Safety Act, both require major platforms to give vetted researchers access to data for public-interest research. Carl mentioned, "To answer plainly: no, it is not happening effectively." Researchers face endless onboarding delays, broken APIs, and unresponsive channels, and in the UK the implementation guidelines have stalled.
The knock-on effect is that researchers can't even assess whether platform-provided data would be trustworthy, because they can't get it in the first place. That keeps independent scraping essential to even begin to answer urgent questions, like whether algorithmic feeds amplify extremist content.
Paul gave a concrete example: an ITV journalist, a former student of his, partnered with Oxylabs Project 4β to gather public data during the UK civil unrest following the tragic deaths of children, tracing how misinformation spread. He also drew a parallel to the Open Data movement 15 years ago, when governments released the datasets they were comfortable with and kept the sensitive ones back.
We must pay close attention to which specific segments of the web remain accessible.
Paul Bradshaw, Journalist & Academic at BBC, Birmingham City University
On a more hopeful note, a growing coalition of European safety researchers is now pooling evidence of these roadblocks to push for proper compliance.
One proposed fix is moving entirely to commercial licensing, and the panel pushed back. Krishna argued that licensing makes sense for private, proprietary assets, but public web data should stay open for analysis.
Factual information cannot and should not be monopolized.
Krishna Sood, Product Legal at Black Forest Labs
Relying only on bilateral licensing deals favors big incumbents, since the contracts are slow and expensive to negotiate, leaving small developers stuck with whatever niche data they can afford and, as a result, worse, more biased models.
Her example made it land: early facial recognition systems failed on her because the training data lacked enough representation of darker skin tones. Building AI that works fairly across languages, regions, and communities needs broad, representative data.
Paul connected it to journalism, where publishers need revenue but society benefits when information spreads widely.
When access is governed purely by large commercial licensing agreements, well-resourced media organizations and dominant tech platforms capture the market, while minority viewpoints and independent outlets are left out of the equation.
Paul Bradshaw, Journalist & Academic at BBC, Birmingham City University
Carl closed with the heaviest point. About a decade ago, defense organizations started treating information as a theater of war alongside land, sea, air, and space. Hostile state actors, including Russian and Chinese hybrid operations, see the information domain as a key arena for weakening democracies. That fight used to play out on social media; now AI systems are the next target, probed and manipulated at scale to shift public perception.
Independent access to public web data is not just a commercial or technical matter; it is foundational infrastructure for defending open democracies against authoritarian influence.
Carl Miller, Fellow & Co-Founder at Centre for the Analysis of Social Media
The takeaway from the panel was clear: we have to keep making the case for public data access, out loud and often. This isn't a fight you win once and for all. Pressure from courts, policy, and technical barriers keeps building, and we have to make the case for open access again and again, in rooms where these decisions actually get made.
The web may not be closing for everyone at once, but the people who use it to hold power to account, study real harms, and fight misinformation are feeling the doors swing shut. Those are some of the uses that matter most for the public good, and they need the commercial muscle to push back together.
That gap is exactly why Project 4β exists. We provide free public web data collection tools, technology, and legal advisory to researchers, journalists, academics, NGOs, and nonprofits working on public-interest missions, so the people doing this work aren't the ones locked out.
Much of it falls within the areas this panel covered. On information warfare, democratic resilience, and investigative journalism, we already work with organizations like Bellingcat, Global Witness, Debunk.org, Doublethink Lab, and the Civic Resilience Initiative. On the academic side, we support researchers at universities including Stanford University, Carnegie Mellon University, Northwestern University, and Warwick University.
If your work depends on getting to public web data, whether you're tracing how false narratives spread, auditing algorithmic harms, or building research that defends open democracies, we'd like to help. Academics, NGOs, NPOs, and other mission-driven organizations can join the initiative and get free access to tools that match the scale of what they're trying to do.
Tell us about your data needs through our form or via form or at 4beta@oxylabs.io.
Forget about complex web scraping processes
Choose Oxylabs' advanced web intelligence collection solutions to gather real-time public data hassle-free.


Narmin Mammadova
2026-10-27



Donata Norkūnaitė
2026-10-21
Get the latest news from data gathering world