How to Scrape Best Buy Product Data: A Step-by-Step Tutorial

Overview of the Best Buy page structure

The scraping target – a Best Buy search page – lists different products against a keyword-based search query. The screenshot below highlights different parts of the page. 

Scraping Best Buy products page

With Web Scraper API, you can extract search results, reviews, product data, and monitor prices maintenance-free. Read on for scraping considerations and code samples in Python as well as a comparison on the best ways of how to scrape Best Buy data.

To scrape a Best Buy product page with Web Scraper API, send an HTTP POST request to the API with a proper payload and wait for the response. The API scrapes the target page on your behalf and returns a response with the desired data. You can further parse this response to extract structured data.

1. Set up the environment

Your system should run Python 3.6+ with the Requests library installed. You can install Requests using the pip install requests command.

Make sure you have valid user credentials to use the API. You can register for a free trial by visiting the Best Buy API page and clicking the Get free trial button.

2. Create a payload

To scrape data from a product page, first, create the payload structure. Keep in mind to use the universal source of Web Scraper API. To learn more about source attributes, visit our documentation.

import requests

payload = {
	'source': 'universal',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}

3. Send the HTTP POST request

When the payload structure is ready, create a request by passing your API’s authentication credentials:

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

4. Export the response

The POST request to the API returns a Requests response object. You can export HTML content from this response to an HTML file using the following script:

with open('bestbuy_4K_monitor.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

Let’s put the entire code together and see the output:

import requests


# Structure a payload.
payload = {
	'source': 'universal',
	'url': 'https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys',
	'geo_location': 'United States',
}
# Get a response.
response = requests.request(
	'POST',
	'https://realtime.oxylabs.io/v1/queries',
	auth=('USERNAME', 'PASSWORD'),
	json=payload,
)

with open('bestbuy_4K_monitor.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

The following is a partial view of the output HTML file:

Scraping Best Buy with Residential Proxies

In the previous section of the tutorial, we covered how to scrape Best Buy results using Oxylabs Web Scraper API. While it is the most convenient way to get the product details you need from Best Buy pages, there are use cases where it is better to use direct requests to the website instead of using the API.

However, sending a lot of requests from a single IP address may make the scraping process impossible, as most commercial websites have various safeguards against scrapers – you can get your IP blocked, encounter CAPTCHAs, and so on. This is where Residential Proxies come in. 

Residential Proxies allow to send requests through physical devices around the world, with IP addresses provided by ISPs. These devices are used as proxies and are completely managed by Oxylabs. In addition, these proxies rotate automatically, so you’ll be using different IP addresses for every other request.

Getting Best Buy data with Residential Proxies

To extract data using Residential Proxies, we’ll need to simply perform an HTTP GET request directly to the URL we provided to the Web Scraper API before. However, this time we’ll be adding a proxies parameter to the request, which will route it through a proxy.

First of all, let’s define our proxies variable. To get the hostname for the proxy server, first you'll need to head to the Oxylabs Dashboard, the Residential Proxies section.

After clicking on the Endpoint generator tab, you'll be able to select various options here, like the region where the proxy is located, the endpoint and session types, and many more.

For this example we’ll be using a global HTTPS proxy with authentication and sticky sessions. Additionally, we’ll also need to specify the location as US or Canada, to bypass the initial country selection page that Best Buy shows. 

The URL should look like this:

proxy_entry = "http://customer-<your_username>-cc-us:<your_password>@pr.oxylabs.io:7777"

Make sure to replace the placeholders with your Residential Proxy credentials.

Next, let’s define a proxy dictionary that we’ll be passing into the request. Since we want to cover both http and https, it should look like this:

proxies = {
    "http": proxy_entry,
    "https": proxy_entry,
}

Now that we have our proxies ready, we can perform the GET request:

url = "https://www.bestbuy.com/site/searchpage.jsp?st=4k+monitor&_dyncharset=UTF-8&_dynSessConf=&id=pcat17071&type=page&sc=Global&cp=1&nrp=&sp=&qp=&list=n&af=true&iht=y&usc=All+Categories&ks=960&keys=keys"
proxy_entry = "http://customer-<your_username>-cc-us:<your_password>@pr.oxylabs.io:10000"
proxies = {
    "http": proxy_entry,
    "https": proxy_entry,
}
response = requests.get(url, proxies=proxies)

After that, we can store the HTML we retrieved as we did before.

html_content = response.text
with open('bestbuy_4K_monitor.html', 'w') as f:
f.write(html_content)

The resulting HTML file should look exactly like the one retrieved from our Web Scraper API.

Comparing different scraping methods

Whether you're web scraping product data, review data, or any other information from a Best Buy URL – take a look below where we compare different types of scraping.

Conclusion

Web Scraper API is a powerful tool for scraping content from Best Buy’s product pages. It's a warranted solution for easier, faster, more reliable, and highly scalable product data collection.

However, you can always scrape reviews or other data from Best Buy using other tools, such as Residential Proxies – while the process is more tedious, it does allow you to get into the nitty gritty task of building your own functional web scraper.

For more e-commerce targets, check how to scrape data from Walmart and Etsy.

Make sure to get a one-week Web Scraper API trial for free. And when using your own tools, a reliable free proxy list or any other proxy option is essential for block-free web scraping. To resemble organic traffic, you can buy proxy solutions, most notably Residential Proxies and datacenter IPs.

If you have any questions regarding the actions in this guide or want to know more about our Scraper APIs, drop a line via the 24/7 live chat to talk to our support team on our home page or send us an email.