How to Scrape YouTube Data: Step-by-Step Guide

YouTube is one of the largest content-sharing platforms in the world, with more than 500 hours of content uploaded each minute. In November 2022, YouTube even secured the second position as the most visited website globally, with 74.8 billion monthly visits, according to Statista.

The sheer volume of public data and traffic on YouTube unlocks various research opportunities for businesses and individuals. Web scraping, often supported by the use of a free proxy server or a premium proxy solution, is the go-to method for extracting data from publicly available YouTube pages, such as video details, comments, channel information, as well as search results.

This guide will show you how to scrape YouTube videos by leveraging Python, the yt-dlp library, proxies, and Oxylabs' YouTube Scraper API. We also have this tutorial available in the video format:

For this, you can leverage YouTube Scraper API to gather video transcripts on a large scale:

import requests
import json


payload = {
    "source": "youtube_transcript",
    "query": "mDveiNIpqyw",
    "context": [{"key": "language_code", "value": "en"}]
}

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=("USERNAME", "PASSWORD"),
    json=payload
)

print(response.status_code)
print(response.json())

with open("yt_transcript.json", "w") as file:
    json.dump(response.json(), file, indent=4)

Make sure to replace the USERNAME and PASSWORD with the API user credentials. Visit our documentation to learn more about the language and origin parameters supported by the API.

4. Scrape YouTube video data

Scraping YouTube videos is also possible with the yt-dlp library. You can extract public video data like the title, video dimensions, and the language used.

Let’s extract video details from the video we’ve downloaded previously. For this task, you can use the extract_info() method with the download=False parameter so that it doesn’t download the video file again. This method will return a dictionary with all the video-related info:

from yt_dlp import YoutubeDL


video_url = "https://www.youtube.com/watch?v=mDveiNIpqyw"
opts = dict()

with YoutubeDL(opts) as yt:
    info = yt.extract_info(video_url, download=False)
    video_title = info.get("title", "")
    width = info.get("width", "")
    height = info.get("height", "")
    language = info.get("language", "")
    print(video_url, video_title, width, height, language)

YouTube Scraper API also supports scraping video metadata, check out the documentation to learn more.

5. Scrape YouTube Comments

Please note that all information provided herein is for informational purposes only and does not grant you any rights with regard to the described data, which may be protected by corresponding privacy rights or other rights. Before engaging in scraping activities, you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

To extract all the video comments, you’ll need to pass an additional option getcomments while initializing the yt-dlp library.

Once you set getcomments to True, the extract_info() method will fetch all the comment threads along with the other information about the video. So, you can extract just the comments from the info dictionary like below:

from pprint import pprint
from yt_dlp import YoutubeDL


video_url = "https://www.youtube.com/watch?v=mDveiNIpqyw"
opts = {"getcomments": True}

with YoutubeDL(opts) as yt:
    info = yt.extract_info(video_url, download=False)
    comments = info["comments"]
    thread_count = info["comment_count"]
    print("Number of threads: {}".format(thread_count))
    pprint(comments)

6. Scrape YouTube channel information

For this example, let’s use the Oxylabs channel's “About” section to extract the channel name and description. Here, you’ll have to use your YouTube API credentials to authenticate with the API.

By inspecting the elements, you can easily construct the relative XPath selector using the IDs associated with the elements. Thus, the XPath selectors are:

Channel name XPath

//h2[@id="title"]/yt-formatted-string/text()

Description XPath

//yt-attributed-string[contains(@id, "description")]/span/text()

6.2 Prepare parsing instructions

Now, using the XPath selectors, you can prepare the parsing instructions for YouTube data scraper. It’s a dictionary that lists all the functions to execute when parsing the data from the HTML content. Let’s begin by importing the requests module and defining the variable instructions that'll contain the parsing instructions:

import requests


url = "https://www.youtube.com/@oxylabs/about"

instructions = {
    "Channel Name": {
        "_fns": [{
            "_fn": "xpath_one",
            "_args": ['//h2[@id="title"]/yt-formatted-string/text()']
            }]
    },
    "Description": {
            "_fns": [{
                "_fn": "xpath_one",
                "_args": ['//yt-attributed-string[contains(@id, "description")]/span/text()']
            }]
    }
}

Note the xpath_one function, which tells the API to select only the first matched element when parsing. If needed, you can also extract other channel data, such as social links and more info.

6.3 Prepare payload

Create a new variable payload that'll contain the scraping parameters and parsing instructions that you’ll send to the API:

payload = {
    "source": "universal",
    "render": "html",
    "parse": True,
    "parsing_instructions": instructions,
    "url": url
}

The render parameter is set to html, so the API will execute JavaScript to render all dynamic content. parse is also set to true to tell the API that the payload includes parsing_instructions.

6.4 Make a POST request to the API

To POST the payload to the API, you’ll have to use the credentials that you’ve obtained from the Oxylabs dashboard:

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=("USERNAME", "PASSWORD"),
    json=payload
)
print(response.status_code)

Replace the USERNAME and PASSWORD with your credentials, run the code, and If everything works as expected, you’ll get a status_code of 200.

6.5 Extract the channel info

YouTube Scraper API sends a JSON response from which you can extract the parsed channel name and description, as showcased below:

channel_name = response.json()["results"][0]["content"]["Channel Name"]
description = response.json()["results"][0]["content"]["Description"]

print(channel_name)
print(description)

Here’s the complete code:

import requests


url = "https://www.youtube.com/@oxylabs/about"

instructions = {
    "Channel Name": {
        "_fns": [{
            "_fn": "xpath_one",
            "_args": ['//h2[@id="title"]/yt-formatted-string/text()']
            }]
    },
    "Description": {
            "_fns": [{
                "_fn": "xpath_one",
                "_args": ['//yt-attributed-string[contains(@id, "description")]/span/text()']
            }]
    }
}

payload = {
    "source": "universal",
    "render": "html",
    "parse": True,
    "parsing_instructions": instructions,
    "url": url,
}

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=("USERNAME", "PASSWORD"),
    json=payload
)
print(response.status_code)

channel_name = response.json()["results"][0]["content"]["Channel Name"]
description = response.json()["results"][0]["content"]["Description"]

print(channel_name)
print(description)

7. Scrape YouTube channel subscribers

You can extract the subscriber count of a YouTube channel using the same approach. Let’s again use the Oxylabs channel’s “About” page:

By inspecting elements with Developer Tools, you can build the XPath as follows: //td[contains(@class, "ytd-about-channel-renderer") and contains(text(), "subscribers")]/text(). With this information, you can create parsing instructions like shown below:

instructions = {
    "subscribers": {
        "_fns": [{
            "_fn": "xpath_one",
            "_args": [
                '//td[contains(@class, "ytd-about-channel-renderer") '
                'and contains(text(), "subscribers")]/text()'
            ]
        }]
    }
}

And, just like before, the xpath_one function picks only the first match. The rest of the code is almost the same. Use the previously shown API code, replace the instructions, and extract the data from the JSON response:

subscribers = response.json()["results"][0]["content"]["subscribers"]
print(subscribers)

As the data is in the JSON response, you can extract the parsed subscriber count from the response and print it as an output.

8. Scrape YouTube search results

You can utilize the yt-dlp module as well as YouTube Scraper API to scrape search results from YouTube. Let's see both methods in action.