New Research Uncovers the Main Challenges of Public Web Data Gathering

Methodology of the research

The market survey of web scraping professionals was carried out by a leading research consultants Censuswide at Oxylabs’ request. Censuswide surveyed 506 developers and other technical professionals who conduct web scraping as part of their job in the United States (US) and the United Kingdom (UK), with both countries represented equally.

While the respondents held various job titles, software developers (41%) were prevalent, with full-stack developers (22%) and data engineers (20%) being the second and third most represented groups respectively. The annual turnover of their companies expressed in British pounds ranged from under £100,000 (3%) to £500 million or over (8%), while almost half of the represented companies were in the range between £1 million and £49.99 million. The survey measured the pulse of the two key markets for web scraping with a special focus on challenges, their business impact, and the potential of applying AI in data scraping.

Key findings

The research revealed that the demand for public web data increased for almost three-quarters of US and UK businesses. The increase was significant for almost one quarter (24%, to be precise). Other key findings of the research include:

• 61% of respondents identified building the necessary infrastructure as the main challenge of large-scale web data collection. 

• Building and maintaining data parsers was the second most recognized challenge, named by half of the respondents, with 86% acknowledging that it is a time- and- resource-intensive task.

• 98% of businesses face some negative impact when data collection is delayed. For 21%, the impact is severe.

• If parsing is interrupted, 95% of businesses face negative impacts within 24 hours.

• 75% of developers spend from 10 to 40 hours weekly on parsing processes.

• Identifying complex parsing patterns when dealing with multiple URLs is the most common challenge (58%) associated with data parsing, with dynamic website layouts and changing structures (56%) being a close second.

• 57% say that due to changing website layouts, they have to fix parsers several times a week. For the same reason, 31% fix parsers every day.

• Over 50% of developers mention time as the main parsing related business cost.

• 77% of respondents have tried using AI-powered web scraping solutions. Of them, 86% say that it helped them save time and resources.

Summing up

The survey of web scraping professionals in the UK and the US by Oxylabs and Censuswide shows the growing importance of public web data for businesses. Most businesses experience a rapid negative impact when data collection is delayed. The survey's key findings uncover the main challenges that lead to such delays, and what business costs they lead to. Stay tuned, as a more detailed analysis of these results is forthcoming.