How to Scrape Images from a Website With Python

Previously, we outlined how to scrape text-based data with Python. We‘ve showcased the entire process: from installing Python and the required libraries to coding a basic web scraper and outputting the acquired data into a CSV file. If you're unfamiliar with web scraping or want to learn more, check out our article on what is web scraping and how to scrape data from a website. In this second installment, you’ll learn how to extract images from a website and store them in a desired location.

We highly recommend reading our article Python Web Scraping Tutorial: Step-By-Step, before moving forward. Understanding how to build a basic data extraction tool will make it significantly easier to create a Python image data scraper. Additionally, we’ll use parts of the code we had written previously as a foundation to download image links. Finally, we’ll use both Selenium and the requests library for learning purposes.

Notice each image in the page's HTML source is enclosed in a div element with a class s-item__image-wrapper image-treatment. Furthermore, the images are within the img tags with a source URL in the src attribute.

3. Launch the WebDriver to open a target URL

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver


driver = webdriver.Chrome()
driver.get("https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=laptop&_sacat=0&LH_TitleDesc=0&_osacat=0&_odkw=laptop")
results = []
content = driver.page_source
soup = BeautifulSoup(content, "html.parser")

The data extraction process begins almost exactly the same – you have to import the Python packages as needed. Then, specify your preferred browser for WebDriver, select the URL from which to scrape image links, and create a list to store them. After the Chrome driver opens the URL, the content variable points to the page source, and then the Beautiful Soup library makes the data ready for parsing.

Currently, the code runs the driver in a headful mode. You can run it without a GUI by specifying the headless mode:

from selenium.webdriver import ChromeOptions

options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

If you have stored the executable browser driver in a custom location, you should specify the directory by creating a ChromeService instance with the path to your driver and passing it to the webdriver instance as a service:

service = webdriver.ChromeService(executable_path='/path/to/driver')
driver = webdriver.Chrome(service=service)

4. Extract image URLs with the Beautiful Soup library

In the previous tutorial, we performed all actions by using built-in and defined functions. While we could do another tutorial without defining any functions, it’s an extremely useful tool for just about any project:

# An example of how to define a function
# and select custom arguments for the code inside the function.
def function_name(arguments):
    # Function body goes here.

So let’s move our URL scraper into a defined function. We’ll reuse the same code used in the Python Web Scraping Tutorial: Step-by-Step article and repurpose it to extract full URLs:

Before

for a in soup.findAll(attrs={'class': 'class'}):
    name = a.find('a')
    if name not in results:
        results.append(name.text)

After

# Picking a name that represents the function will be useful as you expand your code.
def parse_image_urls(classes, location, source):
    for a in soup.findAll(attrs={'class': classes}):
        name = a.find(location)
        if name not in results:
            results.append(name.get(source))

Note that it now appends in a different manner. Instead of appending the text, the code uses another function, get(), and adds a new parameter source to it. The source parameter indicates the field on the website where image links are stored. In our case, the image links are in the src HTML tag.

From the previous section, we know the classes, locations, and sources of the images we want to scrape. Thereby, you can call your newly defined function as such:

parse_image_urls("s-item__image-wrapper image-treatment", "img", "src")

Your code should now look something like this:

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver import ChromeOptions

options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

driver.get("https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=laptop&_sacat=0&LH_TitleDesc=0&_osacat=0&_odkw=laptop")
results = []
content = driver.page_source
soup = BeautifulSoup(content, "html.parser")


def parse_image_urls(classes, location, source):
    for a in soup.findAll(attrs={"class": classes}):
        name = a.find(location)
        if name not in results:
            results.append(name.get(source))


parse_image_urls("s-item__image-wrapper image-treatment", "img", "src")

5. Save the URLs to a CSV file

It’s extremely useful to export the data into a CSV file, so you can always easily check for any possible semantic errors. To achieve this, you can use the pandas library and add the following lines to your code:

df = pd.DataFrame({"links": results})
df.to_csv("links.csv", index=False, encoding="utf-8")

If you run the code right now, you should get the links.csv file created right in the running directory. Here’s a partial snippet of the CSV file with the extracted image URLs:

You're likely to receive the second result when web scraping a website that has strict anti-bot measures since the 403 Forbidden Error means the target site has denied your request. Fixing the second outcome will take a little bit of time in most cases, although, at times, there can be more difficult scenarios.

Whenever you use the requests library to send a request to the destination server, a default user agent python-requests/version.number is assigned, for instance, python-requests/2.31.0. Some web services might block such web requests as this user agent points to the use of the requests library. Fortunately, the requests library allows you to assign any user agent (or an entire header) you want:

image_content = requests.get(b, headers={'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'}).content

Adding a user agent will be enough for most cases. There are more complex cases where servers might try to check other parts of the HTTP header in order to confirm that it’s a genuine user. Refer to our guides on HTTP headers and web scraping practices for more information on how to use them to extract images from websites online. We’ve also made an in-depth video tutorial specifically on how to bypass the 403 Forbidden Error, so be sure to check it out below: