Project 4β Supports Large-Scale Research Using Archived Public Web Data

Oxylabs’ pro bono initiative, Project 4β, has partnered with Jagadishen Mooneesawmy, a PhD student in the Department of Economics at the University of Birmingham. His doctoral research, titled “The Political Economy of Woke Capitalism,” uses historical web data to examine how corporate public communication evolves over time and how changes in external conditions may relate to organizational messaging and business behavior. Project 4β provided technical infrastructure to support data collection and did not participate in the research design, analysis, or conclusions.

A Powerful Tool for Research

Through Project 4β, Jagadishen received access to Oxylabs’ Web Scraper API and proxy infrastructure. These tools enabled the large-scale collection of publicly available archived corporate website data from the Internet Archive’s Wayback Machine, helping build a rich textual dataset that would have been extremely difficult to assemble manually. With the help of the solution, Jagadishen managed to collect 663,840 unique archived corporate website URLs, covering 2,302 publicly listed U.S. firms between 2000 and 2024, and assemble a corpus of 500 million words efficiently and at scale. The data collection was conducted solely for academic research purposes, focused on publicly available corporate website material, and followed relevant research requirements. According to the researcher:

The project is based on the idea that a firm’s website can serve as a public reflection of its internal priorities, heuristics, and cultural shifts. By analyzing historical website data, the research explores how corporate messaging changes over time and how these shifts connect to broader business, political, and societal developments.

Examining Changes in Corporate Communications Over Time

One strand of the research examines associations between changes in state-level political conditions and changes in firms’ public communications. To study this, Jagadishen built a Woke Narrative Index using archived corporate website pages from 2,302 publicly listed U.S. firms between 2000 and 2024, collected with Oxylabs’ tools.

Using a regression discontinuity design around closely contested gubernatorial elections, the study identifies how firms adjust their public positioning when political control changes. In the researcher’s analysis, closely contested gubernatorial election outcomes were associated with measurable changes in corporate public communications, including changes in diversity, equity, and inclusion-related language.

The research also finds that public-facing rhetoric and internal diversity practices can move in opposite directions. While DEI language on corporate websites increases, workforce diversity scores decline by 10.0% and board gender diversity declines by 10.6% after the same political shock. The analysis identified differences between public communications and selected organizational indicators following the observed events.

Why Public Web Data Matters

Access to public web data is essential for this kind of academic research because it enables researchers to observe real-world corporate behavior as it unfolds. Public web data can provide an additional source of longitudinal information that complements traditional datasets and enables analysis of changes in public communication over time. 

Without scalable web data collection technologies, this type of research would be substantially more difficult to conduct at this scale. Manual collection across decades of archived webpages and thousands of firms would be labor-intensive, error-prone, and difficult to scale. Oxylabs’ infrastructure made that process possible and helped create a comprehensive dataset with meaningful real-world applications.

Partner with Project 4β 

We’re always open to new partnerships where Oxylabs’ web intelligence collection solutions and expertise can help solve critical research questions and advance important missions.

Project 4β supports researchers, academics, journalists, NGOs, and organizations that use public web data to drive positive impact.  We provide free access to these tools, so partners can collect and analyze large-scale public web data and focus on what matters most: impact. 

If your organization’s work or research could benefit from web intelligence tools, we’d love to hear from you. Reach out to 4beta@oxylabs.io or share your details through our form.