How to Scrape IMDb Data: Step-by-Step Guide

Why scrape IMDb

As one of the best-known entertainment data repositories, IMDb (Internet Movie Database) contains tons of data on movies, TV shows, and video games. Not only is there a lot of data, but it's also extremely varied. For example, you can explore movie descriptions, cast, ratings, trivia, related movies, awards, and more. In addition to that, you'll find user-generated IMDb data, such as reviews.

This wealth of information can be applied for a number of purposes, ranging from market research and movie recommender systems to strategic marketing initiatives. Furthermore, user reviews present a goldmine for sentiment analysis, which can deepen insights into movie audiences.

What data can be extracted from IMDb?

IMDb contains a wealth of data points for web scraping. Here are the main types of IMDb data points you can collect:

1. Movie/TV show/ video game metadata: titles, release dates, runtimes, genres, content ratings, and plot summaries

2. Cast and crew information: actors, directors, writers, and their roles

3. Ratings and reviews

4. Images: posters, production stills, and promotional materials

5. Box office data and budget information

6. Production companies and distributors

7. Filming locations

8. Technical specifications (aspect ratio, sound mix)

9. Awards and nominations

10. Trivia, goofs, and quotes

11. Related content recommendations

12. Episode guides for TV series

1. Setting up for scraping IMDb

As you’ll be writing a Python script, make sure you have Python 3.8 or newer installed on your machine. This guide is written for Python 3.8+, so having a compatible version is crucial.

Creating a virtual environment

A virtual environment is an isolated space where you can install libraries and dependencies without affecting your global Python setup. It's good practice to create one for each project. Here's how to set it up on different operating systems:

python -m venv imdb_env #Windows
python3 -m venv imdb_env #Mac and Linux

Replace imdb_env with the name you'd like to give to your virtual environment.

Activating the virtual environment

Once the virtual environment is created, you'll need to activate it:

.\imdb_env\Scripts\Activate #Windows
source imdb_env/bin/activate #Mac and Linux

You should see the name of your virtual environment in the terminal, indicating that it's active.

Installing required libraries

We'll use the requests library for this project to make HTTP requests. Install it by running the following command:

$ pip install requests pandas

And there you have it! Your project environment is ready for IMDb data scraping. In the following sections, we'll delve deeper into the IMDb structure.

2. Overview of Web Scraper API

Oxylabs' Web Scraper API allows you to extract IMDb data from many complex websites easily. The following is a basic example that shows how Scraper API works.

# scraper_api_demo.py
import requests

USERNAME = "username"
PASSWORD = "password"

payload = {
    "source": "universal",
    "url": "https://www.imdb.com"
}

response = requests.post(
    url="https://realtime.oxylabs.io/v1/queries",
    json=payload,
    auth=(USERNAME,PASSWORD),
)

print(response.json())

After importing requests, you need to replace the credentials with your own, which you can get by registering for a Web Scraper API subscription or getting a free trial. The payload is where you inform the API what and how you want to scrape.

Save this code in a file scraper_api_demo.py and run it. You’ll see that the entire HTML of the page will be printed, along with some additional information from Scraper API.

In the following section, let's examine various parameters we can send in the payload. 

Scraper API parameters

The most critical parameter is source. For IMDb, set the source as universal, which is general-purpose and can handle most domains.

The parameter url is self-explanatory, a direct link to the IMDb URLs you want to scrape. In the code discussed in the previous section, there are only two parameters. As a result, you get the entire HTML of the page.

Instead, what you need is parsed data. This is where the parameter parse comes into the picture. When you send parse as True, you must also send one more parameter — parsing_instructions. Combined, these two allow you to get parsed data in a structure you prefer.

The following allows you to get a JSON of the page title:

"title": {
    "_fns": [
                {
                    "_fn": "xpath_one", 
                    "_args": ["//title/text()"]
                }
            ]
        }
},

If you send this as parsing_instructions, the output would be the following JSON:

{'title': 'IMDb Top 250 Movies'}

The key _fns indicates a list of functions, which can contain one or more functions indicated by the _fn key, along with the arguments.

In this example, the function is xpath_one, which takes an XPath and returns the first matching element. On the other hand, the function xpath returns all matching elements.

The functions css_one and css are similar but use CSS selectors instead of XPath. For a complete list of available functions, see the Scraper API documentation.

The following code prints the title of the IMDb page:

# imdb_title.py
import requests

USERNAME = "username"
PASSWORD = "password"

payload = {
    "source": "universal",
    "url": "https://www.imdb.com",
    "parse": True,
    "parsing_instructions": {
        "title": {
            "_fns": [
                        {
                            "_fn": "xpath_one",
                            "_args": [
                                "//title/text()"
                                ]
                        }
                    ]
                }
    },
}


response = requests.post(
    url="https://realtime.oxylabs.io/v1/queries",
    json=payload,
    auth=(USERNAME,PASSWORD),
)


print(response.json()['results'][0]['content'])

Run this file to get the title of the IMDb page. In the next section, you’ll scrape movie data from a list.

3. Scraping movie info from a list

Before scraping a page, we need to examine the page structure. Open the IMDb top 250 listing in Chrome, right-click the movie list, and select Inspect.

Move around your mouse until you can precisely select one movie list item and related data. 

5. Exporting to JSON and CSV

The output of Scraper API is JSON, and you can save the extracted data as JSON directly. If you want a CSV file, you can use a library such as Pandas. Remember that the parsed data is stored in the content inside results.

As we created the review in the key review, we can use the following snippet to save the extracted data:

# parse_reviews.py
import json
import pandas as pd

# save results into a variable data
data = response.json()

# save the data as a json file
with open("results_reviews.json", "w") as f:
    json.dump(data, f, indent=4)

# save the reviews in a CSV file
df = pd.DataFrame(data['results'][0]['content']['reviews'])
df.to_csv('reviews.csv', index=False)

Web scraping IMDb data with Python using proxies

Of course, you can scrape IMDb data using Python web scraping with your favorite IDE, no API required. However, to extract data from multiple pages at scale, using proxies is almost obligatory.

Proxies help you avoid IP blocks and limits. They route your requests through different IP addresses, making it appear as if multiple users are accessing the site rather than a single web scraper.

The foundation of a web scraping script is selecting the right Python libraries. Beautiful Soup and Requests are a must, and frameworks like Selenium will be needed to deal with interactive web elements (JavaScript).

Here's a typical workflow:

1. Identify target data - Determine exactly which IMDb pages and data points you need.

2. Set up your environment - download Python, choose an IDE, and install the necessary libraries.

3. Get proxies - Obtain reliable proxy IPs from a proxy service.

4. Design request headers - Set up realistic user agents and other headers to mimic browser behavior.

5. Code the scraper - Create functions to send requests through proxies, parse HTML, and extract data.

6. Store the data - Save extracted data to a database or file format (CSV, JSON).

Conclusion

Web Scraper API simplifies IMDb web scraping by handling the most common data gathering challenges, including managing your proxies. You can use any language you like, and all you need to do is send the correct payload.

If you want to scale efforts using your own scraper, you can buy proxies to ensure efficient and uninterrupted scraping, such as a paid or free proxy list of residential proxies or datacenter IPs.

You might also be interested in reading up about scraping other targets such as YouTube, Google News, or Netflix.