How to Scrape E-Commerce Websites With Python

E-commerce data scraping is a common practice driven by several key reasons. It enables businesses to conduct market research, monitor prices, enhance product catalogs, generate leads, and aggregate content. In this tutorial, you'll learn more about e-commerce web scraping by using Python and Oxylabs’ Web Scraper API. This e-commerce scraping tool will help you avoid any antibot protection or CAPTCHA without writing a complex script.

For your convenience, we also prepared this tutorial in a video format:

Let’s get started. 

1. Project setup

First, you’ll have to install Python. Please download it from here. 

2. Install dependencies

Next, you’ll have to install a couple of libraries so that you can interact with our e-commerce product scraper and parse the HTML content. You can run the below command:

pip install bs4 requests

This will install Beautiful Soup and the requests libraries for you. 

3. Import libraries

Now, you can import these libraries using the following code:

from bs4 import BeautifulSoup
import requests

4. Retrieve API credentials

Next, you’ll have to log in to your Oxylabs E-Commerce Scraper API account to retrieve API credentials. If you don’t have an account yet, you can simply sign up for a free trial. There, you’ll get the necessary credentials for the e-commerce scraper.

Once you've retrieved your credentials, you can add them to your code.

username, password = 'USERNAME', 'PASSWORD'

Don’t forget to replace USERNAME and PASSWORD with your username and password.

5. Prepare payload

Oxylabs’ Web Scraper API expects a JSON payload in a POST request. You’ll have to prepare the payload before sending the POST request. The source must be set to universal.

url  = "https://sandbox.oxylabs.io/products"
payload = {
    'source': 'universal',
    'render': 'html',
    'url': url,
}

’render’: 'html' tells the API to execute JavaScript when loading the website content.

Note: For demonstration, we’ll scrape e-commerce data from the sandbox.oxylabs.io store page. 

6. Send a POST request to the API

Let’s send the payload using the requests module’s `post()` method. You can pass the credentials using the `auth` parameter.

response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=(username, password),
    json=payload,
)
print(response.status_code)

Since the `payload` needs to be in JSON format, you can use the `json` parameter of the `post()` method. If you run this code now, you should see the output of the `status_code` as `200`. Any other numbers mean there are some errors, if you get one, check your credentials, payload, and URL thoroughly and make sure they all are correct.

7. Parse Data

You can extract the HTML content from the `JSON` response of the API and create a Beautiful Soup object named `soup`.

content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")

Using Web Browser’s developer tools, you can inspect the various elements of the website to find the necessary CSS selectors. Once you've gathered the CSS selectors, you can use the `soup` object to extract those elements. To activate the developer tools, you can simply browse to the target website, right-click, and select inspect. Let’s parse the title, price, and availability of all products.

Title

If you inspect the title, you’ll notice it’s inside a `<h4>` tag with a class `title`.

So, you can use the `soup` object to extract the title as below:

title = soup.find('h4', {"class": "title"}).get_text(strip=True)

Price

Similarly, inspect the price element.

As you can see, it’s wrapped in a <div> with the class price-wrapper. So, use the find() method, as shown below, to extract the price text.

price = soup.find('div', {"class": "price-wrapper"}).get_text(strip=True)

Availability

There are two types of availability on this website, In Stock and Out of Stock. If you inspect both elements, you’ll notice they have different classes.

Fortunately, the Beautiful Soup library’s find() method supports multiple class lookups. You’ll have to pass the classes in a list object.

availability = soup.find('p', {"class": ["in-stock", "out-of-stock"]}).get_text(strip=True)

For more efficient scraping, take a look at the best practices of how to use find() & find_all() in BeautifulSoup.

All products

To extract all product data, you’ll have to inspect the product elements and find the appropriate CSS selectors.

Since each of the product elements is wrapped in a <div> with class product-card, you can loop through each element using a for loop.

data = []
for elem in soup.find_all("div", {"class": "product-card"}):
    title = elem.find('h4', {"class": "title"}).get_text(strip=True)
    price = elem.find('div', {"class": "price-wrapper"}).get_text(strip=True)


    availability = elem.find('p', {"class": ["in-stock", "out-of-stock"]}).get_text(strip=True)
    data.append({
        "title": title,
        "price": price,
        "availability": availability,
    })
print(data)

The data list will contain all the product data. 

Full source code

The entire e-commerce web scraper is given below for your convenience. You can use it as a building block for your next e-commerce website scraper. You’ll only have to replace the URL and parsing logic with your own.

from bs4 import BeautifulSoup
import requests


username, password = 'USERNAME', 'PASSWORD'
url  = "https://sandbox.oxylabs.io/products"
payload = {
    'source': 'universal',
    'render': 'html',
    'url': url,
}
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=(username, password),
    json=payload,
)
print(response.status_code)


content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")


data = []
for elem in soup.find_all("div", {"class": "product-card"}):
    title = elem.find('h4', {"class": "title"}).get_text(strip=True)
    price = elem.find('div', {"class": "price-wrapper"}).get_text(strip=True)


    availability = elem.find('p', {"class": ["in-stock", "out-of-stock"]}).get_text(strip=True)
    data.append({
        "title": title,
        "price": price,
        "availability": availability,
    })
print(data)

Here’s the output: