OxyCon 2019: The Top Takeaways From Day Two

Oops I scraped. Should I hire a lawyer? 

While giving quite a few tongue in cheek legal disclaimers throughout their presentation, Oxylabs in-house law guys Denas Grybauskas and Nerijus Sveistys discussed some legal cases. Here are the lawsuits in question, relevant to how copyright and web scraping are interpreted today from a legal standpoint:

• Feist Publications v. Rural Telephone Service Co. (1991)

• Ebay v. Bidder’s Edge (2000)

• Power Ventures v. Facebook (2009)

• Craigslist v. 3taps Inc. (2013)

• QVC v. Resultly (2014)

• Ryanair v. PR Aviation (2015)

• Ryanair v. Expedia (2019)

• HiQ labs v. LinkedIn (2019)

One of the key takeaways from their presentation was that although the famous HiQ labs v. LinkedIn court decision was beneficial for companies engaging in web scraping, there is still a lot of uncertainty legally. A handy scheme was also provided in regards to the level of risk related to various different scraping approaches.

How websites block bots

Dmitry Babitsky, co-founder & chief scientist @ ForNova detailed a bunch of different methods that websites utilize to recognize, track and ultimately block bots. Here are some of the most popular methods:

1. Big amount of unusual requests and URLs. 

2. Missing cookies.

3. Miscorrelation between different request attributes, such as the IP address location not matching to browser languages and the time zone.

4. WebRTC leaking your real IP address. 

5. Suspicious browser configuration, e.g. disabled Javascript.

6. Browser performance analysis and comparison with similar configurations.

7. Analyzing mouse and keyboard inputs.

Mr. Babitsky also touched upon more sophisticated methods of identification, such as browser fingerprinting, which our speaker Allen O’Neill discussed in-depth yesterday, and put forward some of the things that websites do after identifying a bot.  

Q&A: reaching a high success rate

Oxylabs Software Engineer Eivydas Vilcinskas did a great presentation on how to reach a high scraping success rate using Oxylabs Real-Time Crawler (now known as a Web Scraper API), our advanced scraping solution that does all of the heavy lifting for you. It was a walk through all the challenges one might encounter while using Real-Time Crawler with some tips on how to solve each and every one. 

However, if you use Real-Time Crawler and could not make it to the conference, we have good news – a detailed documentation with all of the same information is currently in the works and you know you can always rely on our premium support for whatever you need.

After an additional Q&A that got even more in-depth, with many attendees curious about the underpinnings of our Real-Time Crawler, the one and only – Head of Account Management Mante Petrauskaite said some closing remarks on how OxyCon came to life, thanked all of the speakers, attendees and the Oxy folks who made the event possible.