OxyCon 2021: The Top Takeaways From Day Two

Augmenting Web Scraping With Machine Learning

The future starts now. Jurijus Gorskovas, Machine Learning Engineer at Oxylabs, knows better than anyone how machine learning (ML) is transforming web scraping. In his presentation, not only Jurijus opened up about the struggles his team faced while scraping public data at scale, but also gave valuable tips on how to deal with these challenges successfully using ML algorithms. Here are some of the examples when the usual web scraping approach can be replaced with more robust ML techniques:

• Artificial Intelligence (AI) block detection. Machine learning enables gathering as many data points (HTMLs) as possible, minimizing blocks. However, it comes with its challenges, such as coupling samples per domain, errors in labeling, and multi-language problems. 

• AI scraper optimizer. Usual scraping parameterization involves some critical issues. Most of the time, new domains require development resources, and existing domains require maintenance, which makes everything even more complex when it comes to scraping on a large scale. AI adaptive parameterization takes less time to react and can learn from  historical data, which makes it more competitive.

• AI CAPTCHA solvers. While multiple varieties of CAPTCHAs are evolving, AI models are being trained to recognize and solve them. Such in-house solutions reduce developer resources and save costs on a large scale.

• AI e-commerce parser. The most common issues with a static parser are that each domain requires a personal parser and every reaction to changes needs manual input. It leads to the necessity of maintaining multiple parsers, which can be resource inefficient. AI e-commerce parsers can extract title, price, and other data points from any e-commerce website, requiring less maintenance.

Using NLP for Entity Detection in Parsed HTML

An insightful presentation on how Natural Language Processing (NLP) helps better understand the data we extract was held by Adi Andrei, Founder and CEO at Technosophics. Adi discussed the NLP tools that identify speech patterns and thus detect entities in parsed HTML.

According to Adi, web scraped data, especially text data (natural language), is unstructured in most cases. Computers don’t understand, and can’t easily process this sort of data since extracting meaning from the text is difficult due to several reasons:

• Words that look different can mean the same thing

• Same words in a different order can mean something completely different

• Character level regular expression approach is very limited

The solution offered by Adi involves using linguistic knowledge and natural language processing (NLP) that help transform unstructured data into structured data. 

With the help of Python NLP libraries, the speaker introduced some practical techniques and workflows that can be used in web scraping to extract meanings from text data. Supporting files for this coding session can be found here.  

Ethical Data Collection Policies

The presentation made by Cornelius (Con) Conlon, the Managing Director of Merit Data & Technology, was a guide to action for the companies that favor data collection being ethical, as well as for those who are only about to embark on this path.

During the session, Con turned to the legal aspects of data scraping, the benefits of implementing an ethical data collection policy for the company, and offered advice on handling situations when the ethical approach runs counter customer expectations.

At the beginning of his presentation, Cornelius stressed the positive effect of web scraping in many business areas, including marketing and governance. And while the scraped data can be useful in many respects, it can be scraped either in a good or bad way. Cornelius provided several key reasons why ethical data collection should become a priority:

• It makes sure everyone involved in web scraping activities is on the same page regarding critical do’s and don’ts.

• It leads to better code and maintenance outcomes since scrapers that violate the policy draw unwanted attention.

• As a result, it reduces costs.

• It helps avoid reputation damage and lawsuits.

• It forces your customer to be more conscious about what they want and how they want it.

According to Con, these are the elements that could be considered in this policy:

1. Take into account the target sites and the burden we might put upon them.

2. Collect only the data you need.

3. Adhere to Robots.txt and track changes.

4. Send a user string that contains details whenever it’s possible.

Cornelius proposes discussing this policy with your tech team and revising it every six months. After all, having an ethical data collection policy will help you in any legal proceedings.

Web Scraping in 2021 and Beyond

The logical culmination of Day Two, as well as the OxyCon 2021 conference, was marked by a panel discussion. It brought together the experts from business, legal, and e-commerce areas to share their thoughts on how the web scraping industry was changing in 2021, especially due to the pandemic.

The key takeaways from the panel discussion can be summarized as follows:

1. As Gabriele Montvile, Head of Account Management at Oxylabs, noted, many clients grew more sensitive to pricing and started to favor more flexible payment terms. It’s crucial to adjust to those needs in order to maintain trust. At the same time, while some business areas suffered fluctuations, other sectors, such as e-commerce, managed to make even more profit during the pandemic.

2. Acquiring talent got more challenging, but Allen O’Neill, Founder and CTO at DataWorks, has a positive perspective on the future. He believes that we should augment the human workforce with AI and concentrate on automatizing the processes. Focusing on a company’s core competencies and being more selective on what we work on should become our guiding principle.

3. Cornelius (Con) Conlon, the Managing Director of Merit Data & Technology, noted that making healthcare, education, and other private data publicly available may bring positive changes in policy making in many sectors.

4. Since there’s still a lot of uncertainty about legal regulation in the web scraping industry, Marija Markova, Head of IP Team at Oxylabs, emphasized the point previously stated: the industry has to come to an internal agreement and self-regulation.

5. Juras Jursenas, Chief Operating Officer at Oxylabs, concluded by highlighting the importance of finding and maintaining good partnerships.

After a one-year break during the pandemic, the OxyCon conference came back with even more groundbreaking presentations and topics delivered by the leading experts from all over the world. We truly hope that this web scraping event enriched you with multiple inspiring discoveries and know-how.

See you at OxyCon next year!