OxyCon 2022 has officially come to an end, which means it’s time to take a look at some of the biggest highlights from Day Two!
Just like the conference’s Day One, yesterday’s live sessions were able to immerse us in the most trending topics of the web scraping industry as well as offer new perspectives on the things we thought we already knew a lot about.
So, make sure you’ve marked all the event’s key points by checking out the blog post below.
The day started with a presentation on Machine Learning (ML) – an increasingly popular discipline of AI essential for evaluating data and making predictions with almost zero human intervention. Allen O’Neill, CEO/CTO at The DataWorks, explored how to use ML to turn text-based web data into valuable information-rich insights using open-source tools and technologies.
One of the most valuable ideas from Allen’s presentation – we need to harness the power of information extraction, not data extraction. Information has structure, value, and, with the usage of Natural Language Processing (NLP), it can be broken down into small parts for matching and meaning extraction.
Named entity recognition, words as numbers, part of speech, proximity search – all these NLP techniques should be utilized in synergy to handle ambiguity and get:
Essential market insight
New product ideas
Then, we continued the day with another external speaker Karsten Madsen, CEO at Morningscore. In this talk, Karsten decided to focus on their own example of building a company in the ever changing and demanding web scraping land.
When trying to succeed in the market, Morningscore had to face multiple challenges, such as:
A huge number of competitors
Ahrefs claim to have 30% of Google's server capacity (which basically meant they are competing with an industry giant)
A need to organize billions of data at high speed and with amazing accuracy
Despite all that, Karsten and his team were able to find their own way of standing a chance. They partnered with the best data suppliers to get the needed public data quickly and with less costs. And instead of being an underdog, turned to creativity and gamification for smarter data presentation and enhanced user experience.
Overall, this was an ultimate story of how weaknesses can be turned into strengths.
Knowing your scraper is the beginning of all wisdom – this is the phrase Martynas Saulius, Python Developer at Oxylabs, used to start his presentation. And due to its noticeable interconnection with the famous saying by Aristotle, this sentence immediately grabbed the attention of every viewer.
In the presentation itself, Martynas presented the effective observability trifecta, Logs, Metrics, & Tracing, and even gave a detailed explanation why Metrics is his personal favorite pillar of observability. He discussed its types, gathering methods, and highlighted one of the important Metrics’ benefits – the ability to make your system autonomous.
The speech concluded with another memorable phrase – knowledge is power. By using all the observability pillars you can gather essential wisdom to improve your tooling and become more powerful.
The knowledge-sharing day continued with a presentation from Eivydas Vilčinskas, Technical Team Lead at Oxylabs and a regular expert at OxyCon web scraping conference since 2019.
This time, Eivydas decided to demonstrate a practical introduction to the wide field of scraping and share some important tips he came up with during the years of technical experience, starting from session preparation to data parsing.
To make sure you can use all the mentioned tips for your own web scraping projects, we’ve shortly noted all of them below.
Tip 1. Browserless is faster than headless
Tip 2. Change the “User-Agent” header
Tip 3. Prepare your session
Tip 4. Re-use session parameters
Tip 5. Use proxies
Tip 6. The Developer Tools, your best friend
Tip 7. API over HTML
Tip 8. Use lxml with XPath
Tip 9. Queues make your system robust
For your convenience, you can also check out the code samples shared by Eivydas during the presentation by following this link.
Another presentation covering the general information about web scraping and its related processes was delivered by Paul Morgan, Data Collections Team Lead at Datasembly. By breaking down his talk into separate sections, such as WTF of data collection, orchestration, observability, and introspection, he was able to keep the viewers interested throughout the whole session as well as deliver enough details on every topic.
Particular attention was, of course, given to job observability as an essential concept to get insights into the whole infrastructure. Paul Morgan shared the different ways they perform job observability in their company, which obviously set a great example for the viewers looking to improve this aspect of their business.
So, according to Paul, observability can be achieved by:
Keeping track of different steps throughout the process
Noticing the various issues that occur
Performing automatic monitoring that allows to know if something wrong is happening sooner than later.
A chance to close this year’s web scraping conference was given to Sarah McKenna, CEO at Sequentum and an experienced engineering manager running all kinds of automated operations.
As highlighted by the speaker, there are a lot of problems for large scale web data extraction projects (sites change, errors happen, unexpected edge cases arise); however, it doesn’t mean that these issues will make the process fail. That’s why in her presentation, Sarah chose to focus on covering every little aspect that can make public data extraction at scale not only possible but also successful.
The expert shared all the tiny details you have to keep your eye on, explained how to define, measure and track KPIs for public web data extraction operation, and even explained how to handle key compliance concerns and mitigate legal risks.
As you see, OxyCon Day Two was all about sharing valuable knowledge. It was intense, entertaining, technical, and hopefully encouraged you to to look at the discussed topics from different angles.
See you at OxyCon 2023!
About the author
Yelyzaveta Nechytailo is a Content Manager at Oxylabs. After working as a writer in fashion, e-commerce, and media, she decided to switch her career path and immerse in the fascinating world of tech. And believe it or not, she absolutely loves it! On weekends, you’ll probably find Yelyzaveta enjoying a cup of matcha at a cozy coffee shop, scrolling through social media, or binge watching investigative TV series.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Get the latest news from data gathering world
Scale up your business with Oxylabs®
GET IN TOUCH
Certified data centers and upstream providers
Connect with us