How to Scrape Google Shopping Results: A Step-by-Step Guide

What is Google Shopping?

Formerly known as Google Products Search, Google Products, and Froogle, Google Shopping is a service that allows users to browse, compare, and shop for products from different suppliers who have paid to be featured on the website. 

While giving consumers an opportunity to choose the best offers among thousands of brands, Google Shopping is also beneficial for retailers. When a user clicks on a product link, they are redirected to the vendor’s website for purchasing; thus, Google Shopping acts as a solution for businesses to advertise their products online.

More information on how Google Shopping works can be found here. 

Google Shopping results page structure overview

The data you get when browsing Google Shopping can be segmented into these categories: Search and Product. Let's briefly discuss each of these:

• Search: A list of items for a specific search term on Google Shopping with information about each item, such as title, description, price, availability, and more.

• Product: Comprehensive information about a single product's listing, details about other retailers selling it, and the costs at which it’s offered.

Search page 

The Google Shopping search results page lists all the relevant items available for the required product. The below screenshot highlights different attributes of a results page for the query “pixel 10.” 

Is it legal to scrape Google Shopping results?

In general, web scraping is legal as long as you strictly follow all the regulations surrounding the public data you wish to gather. However, we still recommend seeking professional legal advice to rule out any possible risks.

If you wish to dive deeper into the topic of web scraping legality, check out our extensive blog post.

The pain of scraping Google Shopping

Though doable, scraping Google Shopping might not be the easiest task to take on. Not only is Google Shopping good at detecting automated requests, but it also requires parsing JavaScript, which is an “expensive” operation that slows down the scraping process. 

Therefore, to make sure you effortlessly scrape and parse a variety of Google Shopping page types, it’s best to rely on a high-quality scraping solution, such as Oxylabs’ Google Shopping API. This API is specifically designed to deal with the challenges of Google scraping process and lets you gather accurate real-time data globally. If you want to extract data from the Google search engine, check out our other tutorial on how to scrape Google search results. For speed-sensitive projects, you may want to scrape Google organic results in under a second, making it perfect for developing AI-driven tools.

Step-by-step guide for scraping Google Shopping results using Google Shopping API

Step 1: Set up Python and install required libraries

To get started, you must have Python 3.6+ installed on your system. Then, you need to install the following packages to code the scraper. 

• Requests - to send the request to the API.

• Pandas - to populate the data in the DataFrame data structure. 

To install the packages, use the following command:

pip install requests pandas

Step 2: Set up a payload

Search page

The first step is creating a structure payload containing different query parameters. Find the complete list of available API parameters for scraping Google Shopping search results in our documentation.

Here's an example payload structure that scrapes results for pixel 10 query, sorts them by review score and minimum price of $30, localizes results for the United States, and automatically parser all data:

payload = {
    'source': 'google_shopping_search',
    'query': 'pixel 10',
    'geo_location': 'US',
    'parse': True,
    'context': [
        {'key': 'sort_by', 'value': 'rv'},
        {'key': 'min_price', 'value': 30},
    ],
}

Step 3: Send a POST request

After the payload structure is ready, you can create the request by passing your authentication key.

response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'), # Use your API credentials.
    json=payload,
)

Step 4: Extract product data from a JSON response

We will be extracting Product Title, Price, and Store name from the response. Since we made the payload parameter parse: true, so we will get the JSON response. We can get all this data from the JSON response.

The code below extracts the data from JSON format and stores it in DataFrame.

result = response.json()['results'][0]['content']

with open('search.json', 'w') as file:
    json.dump(result, file, indent=2)

products = result['results']['organic']

data = []

for p in products:
    data.append({
        'Title': p.get('title', 'N/A'),
        'Price': p.get('price_str', 'N/A'),
        'Merchant URL': p.get('merchant', {}).get('url', p.get('url')),
        'Token': p.get('token', 'N/A')
    })

df = pd.DataFrame(data)

The script extracts relevant product information from the response and stores it in the df DataFrame.

Step 5: Save extracted data to a CSV using Pandas

Using the following script, we can easily export the DataFrame to CSV or JSON files:

df.to_csv('search.csv', index=False)

Let’s put all the code together and see the output. 

import json
import requests
import pandas as pd


# Set the scraper & parameters.
payload = {
    'source': 'google_shopping_search',
    'query': 'pixel 10',
    'geo_location': 'US',
    'parse': True,
    'context': [
        {'key': 'sort_by', 'value': 'rv'},
        {'key': 'min_price', 'value': 30},
    ],
}

# Get a response.
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'), # Use your API credentials.
    json=payload,
)

# Get result content and save it to JSON.
result = response.json()['results'][0]['content']

# Save the content to a JSON file.
with open('search.json', 'w') as file:
    json.dump(result, file, indent=2)

# Get the organic results from the content.
products = result['results']['organic']

data = []

# Iterate through all the products.
for p in products:
    data.append({
        'Title': p.get('title', 'N/A'),
        'Price': p.get('price_str', 'N/A'),
        # Get Google Shopping URL if no Merchant URL available.
        'Merchant URL': p.get('merchant', {}).get('url', p.get('url')),
        'Token': p.get('token', 'N/A')
    })

# Create DataFrame from the list of all products.
df = pd.DataFrame(data)

# Save the DataFrame to a CSV file.
df.to_csv('search.csv', index=False)

The script doesn’t contain any print statements and writes everything in CSV and JSON files. Let’s look at a portion of the output CSV file.

Conclusion

Scraping Google Shopping is essential if you’re looking to retrieve accurate data on your biggest competitors’ products and prices and make data-driven decisions to scale your business. If you're aiming to enhance your scraping capabilities, you can buy proxies, such as residential proxies, to ensure smoother and more efficient data extraction. We hope this tutorial was clear and will contribute to more effortless and smooth data-gathering activities. You can also find all the necessary code files on our GitHub. But in case you still have any questions about our Google Scraper API, don’t hesitate to contact us – Oxylabs’ professional team is always ready to assist you. 

Want to broaden your Google data scraping skills? Explore our step-by-step guides for scraping Google search results, Jobs, Images, Trends, News, Flights, Scholar, AI Mode, and Maps.