According to Statista, big data market size revenue is growing every year, and it is only natural that, being a powerful data collection method, web scraping is also gaining more popularity. However, with an increasing number of people adopting it, the legality of web scraping has become a much-debated topic among developers and others who work in the field.
In this article, we will cover essential questions about web scraping legality and what legal issues one can encounter when collecting data from certain websites.
Before we dive in, we want to note that this article is for informational purposes only and that any information contained herein does not constitute legal advice. Accordingly, before engaging in any scraping activities, you should get appropriate professional legal advice regarding your specific situation.
So, is web scraping activity legal or not? It is not illegal as such. There are no specific laws prohibiting web scraping, and many companies employ it in legitimate ways to gain data-driven insights.
However, there can be situations where other laws or regulations may come into play and make web scraping illegal. For example:
1. Your web scraper should not log in to websites or web pages and then download data. By logging in to any website, users have to agree to the Terms of Service (ToS), which may forbid activity like automated data collection.
2. There is a misconception that you can do whatever you want with publicly available data. While there may be fewer restrictions for scraping publicly available data – as opposed to private information – you still have to ensure that you are not breaching laws that may apply to such data.
One example would be downloading copyrighted data. Usually, it includes designs, layouts, articles, videos, and everything that can be considered creative work.
3. Even if data is needed for personal use, ToS may forbid any kind of automatic data collection. In this case, not the data usage but the scraping activity itself may be illegal.
Web scraping may be legal when you are scraping without breaking any applicable laws surrounding the target website or gathered data. However, sometimes malicious actors or hackers intentionally abuse web scraping.
Indeed, when some individuals or companies abuse web scraping and violate ToS, copyright norms, or other applicable laws, it harms the whole industry’s reputation, portraying web scraping as malicious and unethical. However, when conducted responsibly, web scraping is a great way for businesses to make data-driven decisions from publicly available information.
Additionally, when web scraping is in process, a scraper will send numerous requests to the target website to get the required information. As this is done automatically, web scraping tools could potentially make more requests than a regular user does. If this process is done without regard for the website, it will cause a heavy load. This is one of the main reasons websites have security measures.
Another aspect that needs to be considered when scraping publicly available data is various privacy laws.
The GDPR (General Data Protection Regulation) is a data privacy and security law passed by the EU (European Union) and put into effect on May 25, 2018. The main purpose of this regulation is to give EU citizens control over their personally identifiable information by putting limitations on organizations targeting and collecting this data.
The GDPR doesn’t state that web scraping is illegal; however, it restricts what businesses can do with the contact data they wish to extract. For example, in some cases, in order to gather personal data and use it for various purposes, they have to receive explicit consent from the data subjects.
Similarly, the state of California passed a state law, the California Consumer Privacy Act (CCPA), which put businesses collecting personal data under similarly strict requirements (e.g., consumers get a possibility to delete their personal information and opt-out of the sale of their data as well as receive a right to non-discrimination for exercising their CCPA rights).
As mentioned before, we advise you to seek legal consultation before engaging in scraping activities of any kind. With that being said, here are some base practical tips that may help ensure compliance when web scraping:
1. Sometimes, websites provide their API for data collection. If it is possible, use it instead of scraping data.
2. It is essential to respect the ToS for each website.
3. Respect the rules of robots.txt. If you really need the data from a specific website, but ToS or robots.txt forbids any automatic data collection, you can try to ask permission from the site owner.
4. Do not use scraped data without making sure that this information is not copyrighted. If it is necessary to publish this data, you should ask for written permission from the copyright holder.
If you want to learn more about the best web scraping practices or ethical data collection, we have covered these topics in detail in our blog.
To better grasp the legality of web scraping, it can be helpful to look at some real-life scenarios. This can help us better understand the current state of the industry and its future directions.
Below, we will examine some of the most famous cases (some of which we already discussed back in OxyCon 2019). However, keep in mind that these cases are only examples, and you should get appropriate professional advice regarding your specific situation.
Ryanair’s argument with a flight price comparison company, PR Aviation, provided a glimpse of how scraping could be interpreted in European courts. Ryanair’s website subjects its visitors to ToU, which explicitly prohibits scraping. PR Aviation was scraping Ryanair, who took them to court in the Netherlands for breach of contract.
Ryanair came out second best in the dispute, as the Dutch court said that no valid contract had been formed between the companies. It made an interesting allegory, stating that anyone putting up a poster in a shop window visible from the public road, which reads: “Whoever reads further, must pay €5,” cannot accept that the person reading this wants to commit to such a condition.
Still, this does not mean that ToU would not be applicable in a different scenario, as there were a lot of circumstances unfavorable to Ryanair here. Namely, at the time of the scraping, Ryanair was presenting its ToU in a browsewrap, which is not generally accepted as legally binding by courts, as well as the fact that the scraped data was free and accessible to everyone.
Expedia, a U.S. flight comparison company, was scraping Ryanair’s data and continued doing so after receiving a C&D letter. Consequently, Ryanair sued it for breaching the CFAA. Expedia argued that because Ryanair is an Irish company, the CFAA – a U.S. statute – should not be applicable.
The courts established that the CFAA might indeed apply to U.S. companies acting internationally. After this, Ryanair and Expedia settled the case, with the details being confidential. With that being said, as of this day, there are no Ryanair flights being offered via Expedia’s website.
HiQ Labs, a now-defunct workforce data analytics company, scraped data from LinkedIn profiles to provide tools and insights on employees to businesses. After allowing HiQ to collect data for several years, in 2017, LinkedIn issued a C&D letter to HiQ and launched a tool similar to HiQ’s functionality. HiQ sought an injunction in court, which was granted, leading to LinkedIn being asked to withdraw the C&D letter and stop applying any blocking measures against HiQ.
LinkedIn appealed the decision, arguing that HiQ’s scraping was breaching the CFAA. The court decided that HiQ was not acting in breach of the CFAA, as the data scraped from LinkedIn was public (profiles containing user-generated content, not put behind a password wall). The court said that companies should not be able to revoke authorization where one is not needed in the first place, as well as that allowing companies like LinkedIn to decide who can collect and use publicly available data would be contrary to the public interest.
The decision in the HiQ Labs v. LinkedIn case was favorable to scraping companies and reconsidered some of the much-criticized previous court practices regarding the applicability of the CFAA, narrowing the relevance of this act concerning public data (e.g., Facebook v. Power Ventures, Craigslist v. 3Taps). With that being said, if not done with caution, scraping activities might still be subject to potential breaches of the CFAA (e.g., under different case circumstances) as well as other grounds such as, among others, trespass to chattels, copyright, or breach of contract.
However, later in 2022, the Court stated that HiQ’s creation of fake accounts (“Turkers”) to scrape LinkedIn’s data violated LinkeIn’s User Agreement. Therefore, in December 2022, LinkedIn and HiQ reached a settlement in which HiQ agreed to a permanent injunction, requiring HiQ to stop scraping LinkedIn.
If you are interested in the legal aspects of web scraping, watch a recording of our webinar, Web Scraping From a Legal Perspective. During the webinar, an expert panel discusses web scraping laws, cease and desist letters, and ongoing court cases that are relevant to the web scraping community.
Free webinar
On July 5, 2022, Meta filed two separate actions against Octopus and Defendant Ekrem Ateş, who were scraping data from two social media platforms: Facebook and Instagram. The reasoning behind launching both of these legal proceedings is the same – the mentioned company and the individual illegally extracted people’s personal information for unintended use.
Meta v. Octopus
As noted by Meta, Octopus is a U.S. subsidiary of a Chinese national high-tech enterprise that offers scraping services and access to software that customers can use to scrape data from any website. Meta claims that Octopus’ software gives permission to collect Facebook and Instagram users’ personal information, such as gender, date of birth, email address, user profile URL, and location, and violates Meta’s terms and conditions.
Meta v. Ekrem Ateş
The second legal action filed by Meta concerns Ekrem Ateş, a Turkish national who used automated Instagram profiles to scrape data from over 350,000 users. Afterward, the defendant posted this information on a clone site – a third-party website that copies and displays data from original websites without any authorization.
While Meta claims both cases violate their terms and conditions, the legal specifications of web data scraping are very complex. Thus, it’s important to follow the updates on these cases as they will definitely play a role in defining the future landscape of web scraping legality.
More recently, Meta filed a lawsuit against Bright Data for scraping Facebook and Instagram data and thus violating Meta’s ToS. Bright Data, on the other hand, filed a countersuit against Meta for barring their access to publicly available data as the data was allegedly scraped without logging in. In 2024, the U.S. Federal court ruled against Meta after finding no evidence that Bright Data scraped any data under a log-in, meaning that they scraped only publicly available data.
The legality of web scraping is not easily defined, as it depends on various factors, such as the type of data being scraped or applicable, making it a complex issue to navigate.
With that in mind, we urge you to take this article as informational and educational only. It does not replace independent professional advice and judgment. Statements of fact and opinions expressed are those of the presenters only and, unless expressly stated to the contrary, are not the opinion or position of Oxylabs.
If you wish to learn more about scraping, see our step-by-step guide on how to scrape in Python and try our general-purpose web scraper for free. Also, find out more about ethical Residential Proxy sourcing.
About the author
Gabija Fatenaite
Director of Product & Event Marketing
Gabija Fatenaite is a Director of Product & Event Marketing at Oxylabs. Having grown up on video games and the internet, she grew to find the tech side of things more and more interesting over the years. So if you ever find yourself wanting to learn more about proxies (or video games), feel free to contact her - she’ll be more than happy to answer you.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Vytenis Kaubrė
2024-09-27
Augustas Pelakauskas
2024-09-13
Get the latest news from data gathering world
Scale up your business with Oxylabs®
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.ioCertified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Data Collection
Innovation hub
oxylabs.io© 2024 All Rights Reserved