How to Scrape E-Commerce Websites With Python

Scraping e-commerce websites and product pages is a common practice driven by several key reasons. If you're new to web scraping, you can check out our detailed guide on what is web scraping and how to scrape data from a website. It enables businesses to conduct market research, monitor prices, enhance product catalogs, generate leads, and aggregate content. In this tutorial, you’ll learn how to scrape e-commerce businesses using Python and Oxylabs’ E-Commerce Scraper API (part of Web Scraper API). The API will help you avoid any antibot protection or CAPTCHA without writing a complex script.

For your convenience, we also prepared this tutorial in a video format:

Let’s get started. 

1. Project setup

First, you’ll have to install Python. Please download it from here. 

2. Install dependencies

Next, you’ll have to install a couple of libraries so that you can interact with Web Scraper API and parse the HTML content. You can run the below command:

pip install bs4 requests

This will install Beautiful Soup and the requests libraries for you. 

3. Import libraries

Now, you can import these libraries using the following code:

from bs4 import BeautifulSoup
import requests

4. Retrieve API credentials

Next, you’ll have to log in to your Oxylabs account to retrieve API credentials. If you don’t have an account yet, you can simply sign up for a free 1-week trial. There, you’ll get the necessary credentials for the API.

Once you've retrieved your credentials, you can add them to your code.

username, password = 'USERNAME', 'PASSWORD'

Don’t forget to replace USERNAME and PASSWORD with your username and password.

5. Prepare payload

Oxylabs’ Web Scraper API expects a JSON payload in a POST request. You’ll have to prepare the payload before sending the POST request. The source must be set to universal.

url  = "https://sandbox.oxylabs.io/products"
payload = {
    'source': 'universal',
    'render': 'html',
    'url': url,
}

’render’: 'html' tells the API to execute JavaScript when loading the website content.

Note: For demonstration, we’ll use the sandbox.oxylabs.io store page. 

6. Send a POST request to the API

Let’s send the payload using the requests module’s `post()` method. You can pass the credentials using the `auth` parameter.

response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=(username, password),
    json=payload,
)
print(response.status_code)

Since the `payload` needs to be in JSON format, you can use the `json` parameter of the `post()` method. If you run this code now, you should see the output of the `status_code` as `200`. Any other numbers mean there are some errors, if you get one, check your credentials, payload, and URL thoroughly and make sure they all are correct.

7. Parse Data

You can extract the HTML content from the `JSON` response of the API and create a Beautiful Soup object named `soup`.

content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")

Using Web Browser’s developer tools, you can inspect the various elements of the website to find the necessary CSS selectors. Once you've gathered the CSS selectors, you can use the `soup` object to extract those elements. To activate the developer tools, you can simply browse to the target website, right-click, and select inspect. Let’s parse the title, price, and availability of all products.

Title

If you inspect the title, you’ll notice it’s inside a `<h4>` tag with a class `title`.

So, you can use the `soup` object to extract the title as below:

title = soup.find('h4', {"class": "title"}).get_text(strip=True)

Price

Similarly, inspect the price element.

As you can see, it’s wrapped in a <div> with the class price-wrapper. So, use the find() method, as shown below, to extract the price text.

price = soup.find('div', {"class": "price-wrapper"}).get_text(strip=True)

Availability

There are two types of availability on this website, In Stock and Out of Stock. If you inspect both elements, you’ll notice they have different classes.

Fortunately, the Beautiful Soup library’s find() method supports multiple class lookups. You’ll have to pass the classes in a list object.

availability = soup.find('p', {"class": ["in-stock", "out-of-stock"]}).get_text(strip=True)

All products

To extract all product data, you’ll have to inspect the product elements and find the appropriate CSS selectors.

Since each of the product elements is wrapped in a <div> with class product-card, you can loop through each element using a for loop.

data = []
for elem in soup.find_all("div", {"class": "product-card"}):
    title = elem.find('h4', {"class": "title"}).get_text(strip=True)
    price = elem.find('div', {"class": "price-wrapper"}).get_text(strip=True)


    availability = elem.find('p', {"class": ["in-stock", "out-of-stock"]}).get_text(strip=True)
    data.append({
        "title": title,
        "price": price,
        "availability": availability,
    })
print(data)

The data list will contain all the product data. 

Full source code

The entire scraper is given below for your convenience. You can use it as a building block for your next scraper. You’ll only have to replace the URL and parsing logic with your own.

from bs4 import BeautifulSoup
import requests


username, password = 'USERNAME', 'PASSWORD'
url  = "https://sandbox.oxylabs.io/products"
payload = {
    'source': 'universal',
    'render': 'html',
    'url': url,
}
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=(username, password),
    json=payload,
)
print(response.status_code)


content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")


data = []
for elem in soup.find_all("div", {"class": "product-card"}):
    title = elem.find('h4', {"class": "title"}).get_text(strip=True)
    price = elem.find('div', {"class": "price-wrapper"}).get_text(strip=True)


    availability = elem.find('p', {"class": ["in-stock", "out-of-stock"]}).get_text(strip=True)
    data.append({
        "title": title,
        "price": price,
        "availability": availability,
    })
print(data)

Here’s the output: