Back to blog
How to Scrape Product Data From Wayfair: A Step-by-Step Guide
Augustas Pelakauskas
Back to blog
Augustas Pelakauskas
Wayfair is a retailer specializing in furniture and home appliances. As a big player in the home appliances business, it’s a major source of public web data, especially for the e-commerce industry.
On the Wayfair website, you can find various product data types – potential targets for analysis. When collected at scale and in real time, such data can be used to forecast trends and check featured data fluctuations.
With Oxylabs Wayfair Scraper API and proxies, you can extract e-commerce data to weigh your stance against the competition, position a product at a golden spot to maximize revenues, or simply buy an item at the lowest price. No matter the use case, the API’s maintenance-free infrastructure will save time and effort.
Collecting and analyzing e-commerce data helps maintain a competitive advantage with the following applications:
Pricing intelligence – create a long-term product pricing strategy. You can also check our blog post about building a Wayfair price tracker with Python.
Dynamic pricing – adjust your prices according to the competition.
Real-time product monitoring – check various product attributes.
MAP monitoring – track MAP violators to enforce policy agreements.
The following tutorial will explore how to scrape Wayfair to get the public data you need. Read on for the page layout overview, project environment preparation, fetching the Wayfair product page for data extraction, and export to CSV or JSON format.
Before getting technical, let’s analyze the Wayfair page layout. Here are some of the most relevant types.
The search result page appears when searching for products. For example, if you search for the term Sofa, the search result will be similar to the one below:
Search results page
You can extract all the products listed for the search term “Sofa” as well as their links, titles, prices, ratings, and images.
Product listing appears when you click on a product to see the details. It shows all the product information in addition to the main data already visible on the search result page.
Product listing page
The reCAPTCHA protection page appears when Wayfair detects unusual browsing behavior, such as repeated or too-fast (for an organic user) navigation from page to page, indicating the use of automated scripts such as scrapers. The page looks similar to the one below:
reCAPTCHA protection page
Elaborate anti-bot systems and an ever-changing web layout make automated data extraction difficult. As a consequence, when collecting data at scale from Wayfair, you might get blocked, banned, or blacklisted, not to mention constantly micromanaging the script to fix code breaks.
Wayfair is using Google’s reCAPTCHA service to block automated scrapers. It is an anti-bot protection service that uses fingerprinting algorithms and behavioral pattern recognition.
Oxylabs Wayfair API provides out-of-the-box support for bypassing the anti-bot measures by providing proxies, custom headers, user agents, and other features. This immensely eases the process and simplifies the scraper you would have to build.
Compared to regular scrapers, Wayfair Scraper API has multiple advantages, including:
ML-driven proxy management
Dynamic browser fingerprinting
JavaScript rendering
Get a free trial to test our Web Scraper API for your use case.
Now, let’s see how to use Oxylabs Wayfair API to extract data from the Wayfair product page.
To begin scraping Wayfair data, prepare the project environment. If you already have Python installed, you can skip the Python installation and only install the dependencies in your active Python environment.
This tutorial is written using Python 3.11.2. However, it should also work with the older or latest version of Python 3. You can download the latest version of Python from the official web page.
Once you have downloaded and installed Python, install the following dependencies by executing the command below in the terminal or command prompt:
python -m pip install requests bs4 pandas
This command will install the Requests, Beautiful Soup, and Pandas libraries. These modules will interact with the API and store data.
Here’s a target product page. Use Wayfair Scraper API to fetch Wayfair product data and parse it using the Beautiful Soup library. Alternatively, instead of relying on the Beautiful Soup library, you can also parse Wayfair using Custom Parser, a free feature available with all Scraper APIs.
To use the Oxylabs Wayfair API, create an Oxylabs account. You will find all the available APIs here. Make sure to use a 1-week free trial. You’ll have enough time to fine-tune the scraper. After the trial ends, you can keep using the service by seamlessly upgrading to a preferred plan.
Once you create your account, you will get your sub-user’s credentials, with which you can send network requests to the API.
Before starting, let’s discuss some of the most useful query parameters of Wayfair Scraper API.
You can scrape Wayfair by providing any Wayfair URL. You will only have to pass two required parameters: url and source. The source parameter should be set to universal, and the url should be a Wayfair web page URL. Find additional information about forming Wayfair URLs in our documentation.
It also takes optional parameters such as user_agent_type and callback_url. The user_agent_type tells the API which device the user agent will use (e.g., desktop). Lastly, the callback_url parameter is used to specify a URL to which the server should send a response after processing the request. Take a look at an example of a payload:
payload = {
"source": "universal",
"url": "https://www.wayfair.com/furniture/pdp/ebern-designs-adryel-98-wide-microfibermicrosuede-right-hand-facing-sofa-chaise-w003629953.html",
"user_agent_type": "desktop",
"callback_url": "<URL to your callback endpoint.>"
}
To start writing your Wayfair scraper, import the libraries and create a payload with the necessary variables. Since Wayfair uses JavaScript rendering to display prices dynamically, you should also use our Headless Browser to render JavaScript and wait for the price element to load.
import requests
from bs4 import BeautifulSoup
product_url = "https://www.wayfair.com/furniture/pdp/ebern-designs-adryel-98-wide-microfibermicrosuede-right-hand-facing-sofa-chaise-w003629953.html"
payload = {
"source": "universal",
"url": product_url,
"user_agent_type": "desktop_safari",
"geo_location": "United States",
"render": "html",
"browser_instructions": [
{
"type": "wait_for_element",
"selector": {
"type": "css",
"value": "div.SFPrice span.oakhm64z_6112"
},
"timeout_s": 10
}
]
}
username = "USERNAME"
password = "PASSWORD"
Notice username, password, and product_url variables. You will have to use your Oxylabs sub-user’s username and password. Also, if you wish, you can replace product_url with the desired URL.
Next, send a POST request using the Requests module to Oxylabs' realtime API endpoint: https://realtime.oxylabs.io/v1/queries.
response = requests.post(
"https://realtime.oxylabs.io/v1/queries",
auth=(username, password),
json=payload,
timeout=180
)
print(response.status_code)
In the code above, the POST method of the Requests module is used to send a POST request to the API. The sub-user’s credentials are passed for authentication, and the payload is sent in JSON format.
If you run this code, you’ll see 200 as an output which indicates success. If you get any other status code, recheck your credentials and payload.
Now, you can parse the content of the JSON response. The JSON object will have the content of the webpage in HTML format. Use BeautifulSoup to parse the HTML from the response:
content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")
The default html.parser is in use. You can use a different parser if you want.
The soup object has the parsed HTML content. Now, parse the title, price, and rating from this object.
Using a browser, inspect the HTML properties of the product title. To open the inspect tab, right-click on the product title and click inspect. You’ll see something similar to the image below:
Inspecting the HTML properties
According to the HTML property, write the following code to extract the title of this product:
title = soup.find("h1", {"data-hb-id": "Heading"}).text
Inspect the price element and find the proper class attributes:
Inspecting the price element
price = soup.find("div", {"class": "SFPrice"}).find("span", {"class": "oakhm64z_6112"}).text
Similarly, you can parse the rating element with the following code:
rating = soup.find("span", {"class": "ProductRatingNumberWithCount-rating"}).text
The class attribute of the span element is used to identify the rating element and extract the text content.
The product data is now parsed. Use Pandas to export the data in CSV and JSON formats. Next, create a list of dict objects with the parsed data and create a data frame:
import pandas as pd
data = [{
"Product Title": title,
"Price": price,
"Rating": rating,
"Link": product_url,
}]
df = pd.DataFrame(data)
Using the data frame object, export the data in a CSV file with a single line of code. Since you don’t need an index, set the index to False.
df.to_csv("product_data.csv", index=False)
Once you execute this function, the script will create a file named product_data.csv.
Similarly, use the data frame to export the data in JSON format. Pass an additional parameter, orient, to indicate the need for JSON data in records format.
df.to_json("product_data.json", orient="records")
The script will create another file named product_data.json in the current folder containing the exports.
import requests
from bs4 import BeautifulSoup
import pandas as pd
product_url = "https://www.wayfair.com/furniture/pdp/ebern-designs-adryel-98-wide-microfibermicrosuede-right-hand-facing-sofa-chaise-w003629953.html"
payload = {
"source": "universal",
"url": product_url,
"user_agent_type": "desktop_safari",
"geo_location": "United States",
"render": "html",
"browser_instructions": [
{
"type": "wait_for_element",
"selector": {
"type": "css",
"value": "div.SFPrice span.oakhm64z_6112"
},
"timeout_s": 10
}
]
}
username = "USERNAME"
password = "PASSWORD"
response = requests.post(
"https://realtime.oxylabs.io/v1/queries",
auth=(username, password),
json=payload,
timeout=180
)
print(response.status_code)
content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")
title = soup.find("h1", {"data-hb-id": "Heading"}).text
price = soup.find("div", {"class": "SFPrice"}).find("span", {"class": "oakhm64z_6112"}).text
rating = soup.find("span", {"class": "ProductRatingNumberWithCount-rating"}).text
data = [{
"Product Title": title,
"Price": price,
"Rating": rating,
"Link": product_url,
}]
df = pd.DataFrame(data)
df.to_csv("product_data.csv", index=False)
df.to_json("product_data.json", orient="records")
Here’s the scraped and exported JSON data of a Wayfair product page:
[{"Product Title":"Adryel 2 - Piece Upholstered Sectional","Price":"$489.99","Rating":"4.1 ","Link":"https:\/\/www.wayfair.com\/furniture\/pdp\/ebern-designs-adryel-98-wide-microfibermicrosuede-right-hand-facing-sofa-chaise-w003629953.html"}]
Building a scraper that can send requests as an actual browser and mimic human browsing behavior is quite difficult. Also, you would have to maintain it and keep it up to date with constant changes. Such micromanagement requires in-depth knowledge and extensive scraping experience. If you're looking to streamline this process, you can buy proxy solutions to ensure smoother scraping operations.
With Wayfair Scraper API, you can shift your focus where it matters most - data analysis - instead of dealing with technicalities.
For code samples showcased above, check our GitHub.
For more e-commerce scraping targets, follow Amazon, Google Shopping, Etsy, and Walmart guides. Oxylabs E-Commerce Scraper API (part of Web Scraper API) enables you to quickly gather data from the top 50 marketplaces.
If you have questions or face issues, get in touch via the 24/7 live chat on our homepage or email us at support@oxylabs.io.
About the author
Augustas Pelakauskas
Senior Copywriter
Augustas Pelakauskas is a Senior Copywriter at Oxylabs. Coming from an artistic background, he is deeply invested in various creative ventures - the most recent one being writing. After testing his abilities in the field of freelance journalism, he transitioned to tech content creation. When at ease, he enjoys sunny outdoors and active recreation. As it turns out, his bicycle is his fourth best friend.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Yelyzaveta Nechytailo
2024-12-09
Augustas Pelakauskas
2024-12-09
Get the latest news from data gathering world
Scale up your business with Oxylabs®