OxyCon 2022 has officially come to an end, which means it’s time to take a look at some of the biggest highlights from Day Two!

Just like the conference’s Day One, yesterday’s live sessions were able to immerse us in the most trending topics of the web scraping industry as well as offer new perspectives on the things we thought we already knew a lot about. 

So, make sure you’ve marked all the event’s key points by checking out the blog post below.

A Crash Course in Machine Learning for Text Using Web Data

The day started with a presentation on Machine Learning (ML) – an increasingly popular discipline of AI essential for evaluating data and making predictions with almost zero human intervention. Allen O’Neill, CEO/CTO at The DataWorks, explored how to use ML to turn text-based web data into valuable information-rich insights using open-source tools and technologies.

One of the most valuable ideas from Allen’s presentation – we need to harness the power of information extraction, not data extraction. Information has structure, value, and, with the usage of Natural Language Processing (NLP), it can be broken down into small parts for matching and meaning extraction. 

Named entity recognition, words as numbers, part of speech, proximity search – all these NLP techniques should be utilized in synergy to handle ambiguity and get:

• Essential market insight

• Product discovery

• Issue identification

• New product ideas

How Data Scraping and Creative Algorithms Can Lead to Exciting Products

Then, we continued the day with another external speaker Karsten Madsen, CEO at Morningscore. In this talk, Karsten decided to focus on their own example of building a company in the ever changing and demanding web scraping land.

When trying to succeed in the market, Morningscore had to face multiple challenges, such as: 

• A huge number of competitors 

• Ahrefs claim to have 30% of Google's server capacity (which basically meant they are competing with an industry giant)

• A need to organize billions of data at high speed and with amazing accuracy  

Despite all that, Karsten and his team were able to find their own way of standing a chance. They partnered with the best data suppliers to get the needed public data quickly and with less costs. And instead of being an underdog, turned to creativity and gamification for smarter data presentation and enhanced user experience. 

Overall, this was an ultimate story of how weaknesses can be turned into strengths. 

Observability and Web Scrapers: Filling the Unknown Void

Knowing your scraper is the beginning of all wisdom – this is the phrase Martynas Saulius, Python Developer at Oxylabs, used to start his presentation. And due to its noticeable interconnection with the famous saying by Aristotle, this sentence immediately grabbed the attention of every viewer. 

In the presentation itself, Martynas presented the effective observability trifecta, Logs, Metrics, & Tracing, and even gave a detailed explanation why Metrics is his personal favorite pillar of observability. He discussed its types, gathering methods, and highlighted one of the important Metrics’ benefits – the ability to make your system autonomous. 

The speech concluded with another memorable phrase – knowledge is power. By using all the observability pillars you can gather essential wisdom to improve your tooling and become more powerful.

Practical Application of Common Web Scraping Techniques

The knowledge-sharing day continued with a presentation from Eivydas Vilčinskas, Technical Team Lead at Oxylabs and a regular expert at OxyCon web scraping conference since 2019. 

This time, Eivydas decided to demonstrate a practical introduction to the wide field of scraping and share some important tips he came up with during the years of technical experience, starting from session preparation to data parsing.  

To make sure you can use all the mentioned tips for your own web scraping projects, we’ve shortly noted all of them below.

Tip 1. Browserless is faster than headless

Tip 2. Change the “User-Agent” header

Tip 3. Prepare your session

Tip 4. Re-use session parameters

Tip 5. Use proxies

Tip 6. The Developer Tools, your best friend

Tip 7. API over HTML

Tip 8. Use lxml with XPath

Tip 9. Queues make your system robust

For your convenience, you can also check out the code samples shared by Eivydas during the presentation by following this link. 

Data Collection: Orchestration, Observability, and Introspection

Another presentation covering the general information about web scraping and its related processes was delivered by Paul Morgan, Data Collections Team Lead at Datasembly. By breaking down his talk into separate sections, such as WTF of data collection, orchestration, observability, and introspection, he was able to keep the viewers interested throughout the whole session as well as deliver enough details on every topic. 

Particular attention was, of course, given to job observability as an essential concept to get insights into the whole infrastructure. Paul Morgan shared the different ways they perform job observability in their company, which obviously set a great example for the viewers looking to improve this aspect of their business.

So, according to Paul, observability can be achieved by: 

• Keeping track of different steps throughout the process

• Noticing the various issues that occur

• Performing automatic monitoring that allows to know if something wrong is happening sooner than later. 

Web Scraping at Scale With Quality and Compliance

A chance to close this year’s web scraping conference was given to Sarah McKenna, CEO at Sequentum and an experienced engineering manager running all kinds of automated operations.  

As highlighted by the speaker, there are a lot of problems for large scale web data extraction projects (sites change, errors happen, unexpected edge cases arise); however, it doesn’t mean that these issues will make the process fail. That’s why in her presentation, Sarah chose to focus on covering every little aspect that can make public data extraction at scale not only possible but also successful. 

The expert shared all the tiny details you have to keep your eye on, explained how to define, measure and track KPIs for public web data extraction operation, and even explained how to handle key compliance concerns and mitigate legal risks. 

As you see, OxyCon Day Two was all about sharing valuable knowledge. It was intense, entertaining, technical, and hopefully encouraged you to to look at the discussed topics from different angles.  

See you at OxyCon 2023!