How to Scrape Best Buy Product Data: A Step-by-Step Tutorial

Overview of the Best Buy page structure

The scraping target – a Best Buy search page – lists different products against a search query. The screenshot below highlights different parts of the page. 

Scraping Best Buy products page

To scrape a Best Buy product page with Best Buy Scraper API, send an HTTP POST request to the API with a proper payload and wait for the response. The API scrapes the target page on your behalf and returns a response with the desired data. You can further parse this response to extract structured data.

1. Set up the environment

Your system should run Python 3.6+ with the Requests library installed. You can install Requests using the pip install requests command.

Make sure you have valid user credentials to use the API. You can register for a free trial by visiting the Best Buy API page and clicking the Get free trial button.

2. Create a payload

To scrape data from a product page, first, create the payload structure. Keep in mind to use the universal_ecommerce source of E-Commerce Scraper API.  To learn more about source attributes, visit our documentation.

import requests

payload = {
	'source': 'universal_ecommerce',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}

3. Send the HTTP POST request

When the payload structure is ready, create a request by passing your Scraper API’s authentication key:

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

Step 4: Export the response

The POST request to the API returns a Requests response object. You can export HTML content from this response to an HTML file using the following script:

with open('bestbuy_4K_monitor.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

Let’s put the entire code together and see the output:

import requests


# Structure a payload.
payload = {
	'source': 'universal_ecommerce',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}
# Get a response.
response = requests.request(
	'POST',
	'https://realtime.oxylabs.io/v1/queries',
	auth=('USERNAME', 'PASSWORD'),
	json=payload,
)

with open('bestbuy_4K_monitor.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

The following is a partial view of the output HTML file:

Conclusion

Best Buy Scraper API is a powerful tool for scraping content from Best Buy’s product pages. It's a warranted solution for easier, faster, more reliable, and highly scalable product data collection.

For more e-commerce targets, check how to scrape data from Walmart and Etsy.

Make sure to get a one-week Best Buy API trial for free. If you have any questions regarding the actions in this guide or want to know more about our Scraper APIs, drop a line via the 24/7 live chat on our home page or send us an email.