How to Scrape Best Buy Product Data: A Step-by-Step Tutorial

Overview of the Best Buy page structure

The scraping target – a Best Buy search page – lists different products against a search query. The screenshot below highlights different parts of the page. 

Scraping Best Buy products page

To scrape a Best Buy product page with Best Buy Scraper API, send an HTTP POST request to the API with a proper payload and wait for the response. The API scrapes the target page on your behalf and returns a response with the desired data. You can further parse this response to extract structured data.

1. Set up the environment

Your system should run Python 3.6+ with the Requests library installed. You can install Requests using the pip install requests command.

Make sure you have valid user credentials to use the API. You can register for a free trial by visiting the Best Buy API page and clicking the Get free trial button.

2. Create a payload

To scrape data from a product page, first, create the payload structure. Keep in mind to use the universal source of E-Commerce Scraper API (part of Web Scraper API). To learn more about source attributes, visit our documentation.

import requests

payload = {
	'source': 'universal',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}

3. Send the HTTP POST request

When the payload structure is ready, create a request by passing your API’s authentication credentials:

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

Step 4: Export the response

The POST request to the API returns a Requests response object. You can export HTML content from this response to an HTML file using the following script:

with open('bestbuy_4K_monitor.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

Let’s put the entire code together and see the output:

import requests


# Structure a payload.
payload = {
	'source': 'universal',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}
# Get a response.
response = requests.request(
	'POST',
	'https://realtime.oxylabs.io/v1/queries',
	auth=('USERNAME', 'PASSWORD'),
	json=payload,
)

with open('bestbuy_4K_monitor.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

The following is a partial view of the output HTML file:

Conclusion

Best Buy Scraper API is a powerful tool for scraping content from Best Buy’s product pages. It's a warranted solution for easier, faster, more reliable, and highly scalable product data collection.

For more e-commerce targets, check how to scrape data from Walmart and Etsy.

Make sure to get a one-week Best Buy API trial for free. And when using your own tools, proxies are essential for block-free web scraping. To resemble organic traffic, you can buy proxy solutions, most notably residential and datacenter IPs.

If you have any questions regarding the actions in this guide or want to know more about our Scraper APIs, drop a line via the 24/7 live chat on our home page or send us an email.