New day, new horizons. After the first day concluded with a workshop, dinner & some entertainment, OxyCon participants arrived for the second part of the conference, ready for more action.
OxyCon continued with some amazing mini croissants and three presentations, which were so good that the deliciousness of the French pastries paled in comparison. Also, if you haven’t read it yet, we kindly invite you to check out the recap of the first day.
Oops I scraped. Should I hire a lawyer?
While giving quite a few tongue in cheek legal disclaimers throughout their presentation, Oxylabs in-house law guys Denas Grybauskas and Nerijus Sveistys discussed some legal cases. Here are the lawsuits in question, relevant to how copyright and web scraping are interpreted today from a legal standpoint:
Feist Publications v. Rural Telephone Service Co. (1991)
Ebay v. Bidder’s Edge (2000)
Power Ventures v. Facebook (2009)
Craigslist v. 3taps Inc. (2013)
QVC v. Resultly (2014)
Ryanair v. PR Aviation (2015)
Ryanair v. Expedia (2019)
HiQ labs v. LinkedIn (2019)
One of the key takeaways from their presentation was that although the famous HiQ labs v. LinkedIn court decision was beneficial for companies engaging in web scraping, there is still a lot of uncertainty legally. A handy scheme was also provided in regards to the level of risk related to various different scraping approaches.
How websites block bots
Dmitry Babitsky, co-founder & chief scientist @ ForNova detailed a bunch of different methods that websites utilize to recognize, track and ultimately block bots. Here are some of the most popular methods:
Big amount of unusual requests and URLs.
Missing cookies.
Miscorrelation between different request attributes, such as the IP address location not matching to browser languages and the time zone.
WebRTC leaking your real IP address.
Suspicious browser configuration, e.g. disabled Javascript.
Browser performance analysis and comparison with similar configurations.
Analyzing mouse and keyboard inputs.
Mr. Babitsky also touched upon more sophisticated methods of identification, such as browser fingerprinting, which our speaker Allen O’Neill discussed in-depth yesterday, and put forward some of the things that websites do after identifying a bot.
Q&A: reaching a high success rate
Oxylabs Software Engineer Eivydas Vilcinskas did a great presentation on how to reach a high scraping success rate using Oxylabs Real-Time Crawler (now known as a Web Scraper API), our advanced scraping solution that does all of the heavy lifting for you. It was a walk through all the challenges one might encounter while using Real-Time Crawler with some tips on how to solve each and every one.
However, if you use Real-Time Crawler and could not make it to the conference, we have good news – a detailed documentation with all of the same information is currently in the works and you know you can always rely on our premium support for whatever you need.
After an additional Q&A that got even more in-depth, with many attendees curious about the underpinnings of our Real-Time Crawler, the one and only – Head of Account Management Mante Petrauskaite said some closing remarks on how OxyCon came to life, thanked all of the speakers, attendees and the Oxy folks who made the event possible.
Truly, OxyCon was an inspiring conference with some of the smartest people from every corner of the world gathering in one place and we would once again like to thank everyone who attended. See you next year!
P.S. Just to stay updated, be sure to follow Oxylabs on LinkedIn and Twitter.
About the author
Gabija Fatenaite
Director of Product & Event Marketing
Gabija Fatenaite is a Director of Product & Event Marketing at Oxylabs. Having grown up on video games and the internet, she grew to find the tech side of things more and more interesting over the years. So if you ever find yourself wanting to learn more about proxies (or video games), feel free to contact her - she’ll be more than happy to answer you.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Related articles
Get the latest news from data gathering world
Scale up your business with Oxylabs®