Project 4beta Partners with Northwestern University Researchers to Explore Digital Redlining

In an exciting development, our pro bono initiative Project 4β has partnered with Carson Phillips, a doctoral candidate at the Kellogg School of Management, Northwestern University, and Professor Lincoln Quillian, from the Sociology Department at Northwestern University, to investigate the phenomenon of digital redlining. Their project, "Digital Redlining: Exploring Variation in the Quality of Online Search Results," seeks to understand how differences in online search results vary based on the demographic characteristics of one’s geographic location.

Research goals

Carson Phillips' primary academic goals revolve around harnessing publicly available online information to analyze and comprehend technology-induced societal changes.

Phillips aims to study these rapid technological changes across various social contexts by collecting and analyzing online data.

Through the partnership with Project 4β, researchers got free access to Oxylabs SERP Scraper API, which helps to collect timely and reliable search results data from the most popular search engines.

Challenges in Data Collection and the Role of Web Scraping Technology

The path to gathering data for research is fraught with obstacles, primarily due to the deliberate design of many websites that complicate or even obstruct data collection efforts. Websites often deploy technologies that prevent users from repeatedly accessing and collecting large amounts of information.

To navigate these challenges, Phillips has turned to third-party web data collection experts like Oxylabs. The expertise provided by Project 4beta has enabled Phillips and Quillian to conduct their research at scale, overcoming barriers imposed by these complex web designs.

The absence of web scraping technologies would severely hinder researchers' ability to investigate the societal impacts of technological advancements, thus web scraping plays a critical role in their ability to conduct meaningful research on the effect of technology. In the absence of governmental oversight of technology companies, it is increasingly important for scholars to have research methods at their disposal to study the effects of technologies.

Outcome

One of the significant achievements stemming from this partnership is the documentation of disparities in the quality of search results based on one’s geographic location. This research sheds light on how “inequality in the ‘real world’ is reproduced in digital spaces, such as online search. These findings are crucial for understanding the broader implications of digital technologies on different communities.