How to Scrape Best Buy Product Data: A Step-by-Step Tutorial

Product tracking, dynamic pricing, MAP monitoring - the use cases for e-commerce data are plentiful. This guide showcases how to extract product data from Best Buy at scale and in real time. 

With Best Buy Scraper API, you can extract search results, reviews, product data, and more maintenance-free. Read on for scraping considerations and code samples in Python.

In the image above, a search query “4k monitor” lists all the available 4K monitors in the store and their product attributes. 

The attributes are:

• Product Title

• Product Model, SKU, and its Rating

• Product Shipping and Pickup Information

• Product Price

• Product Image

Is it legal to scrape Best Buy?

Best Buy is a public data source available for fair use as defined in the legislation of the country the data collection is conducted in. To ensure you’re in line with the terms of service and applicable laws regarding web data extraction, the Oxylabs team suggests seeking professional legal guidance. Also, check our blog post on the topic: is web scraping legal?

The nuances of scraping Best Buy

The most important consideration when accessing Best Buy is limited locations. The website is only available in the US, Canada, and Mexico, prompting you to use proxies for geotargeting.

Moreover, in addition to proxies, you should consider auxiliary measures, such as browser fingerprinting, to avoid being blocked while scraping. Luckily, Oxylabs Scraper APIs come with in-built AI-powered anti-blocking solutions.

Scraping Best Buy products page

To scrape a Best Buy product page with Best Buy Scraper API, send an HTTP POST request to the API with a proper payload and wait for the response. The API scrapes the target page on your behalf and returns a response with the desired data. You can further parse this response to extract structured data.

1. Set up the environment

Your system should run Python 3.6+ with the Requests library installed. You can install Requests using the pip install requests command.

Make sure you have valid user credentials to use the API. You can register for a free trial here.

2. Create a payload

To scrape data from a product page, first, create the payload structure. Keep in mind to use the universal_ecommerce source of E-Commerce Scraper API.  To learn more about source attributes, visit our documentation.

payload = {
	'source': 'universal_ecommerce',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}

3. Send the HTTP POST request

When the payload structure is ready, create a request by passing your Scraper API’s authentication key:

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

Step 4: Export the response

The POST request to the API returns a Requests response object. You can export HTML content from this response to a text file using the following script:

with open('bestbuy_4K_monitor.txt', 'w') as f:
    f.write(response.json()['results'][0]['content'])

Let’s put the entire code together and see the output:

import requests


# Structure a payload.
payload = {
	'source': 'universal_ecommerce',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}
# Get a response.
response = requests.request(
	'POST',
	'https://realtime.oxylabs.io/v1/queries',
	auth=('username', 'password'),
	json=payload,
)
with open('bestbuy_4K_monitor.txt', 'w') as f:
    f.write(response.json()['results'][0]['content'])

The following is a partial view of the output text file: