How to Scrape Product Data From Wayfair: A Step-by-Step Guide

reCAPTCHA protection page

The reCAPTCHA protection page appears when Wayfair detects unusual browsing behavior, such as repeated or too-fast (for an organic user) navigation from page to page, indicating the use of automated scripts such as scrapers. The page looks similar to the one below:

Bypassing Wayfair scraping challenges

Elaborate anti-bot systems and an ever-changing web layout make automated data extraction difficult. As a consequence, when collecting data from Wayfair at scale, you might get blocked, banned, or blacklisted, not to mention constantly micromanaging the script to fix code breaks.

Wayfair is using Google’s reCAPTCHA service to block automated scrapers. It is an anti-bot protection service that uses fingerprinting algorithms and behavioral pattern recognition.

Oxylabs Wayfair API provides out-of-the-box support for bypassing the anti-bot measures by providing proxy solutions, custom headers, user agents, and other features. This immensely eases the process and simplifies the scraper you would have to build.

Compared to regular scrapers, Wayfair Scraper API has multiple advantages, including:

• ML-driven proxy management

• Dynamic browser fingerprinting

• JavaScript rendering

Set up the project environment

Install Python

This tutorial is written using Python 3.13.2. However, it should also work with the older or latest version of Python 3. Then, set up a Python virtual environment in a new folder and activate it:

python -m venv .venv
source .venv/bin/activate

If you’re using a Windows device, you can activate the .venv with your command prompt like shown below:

.venv\Scripts\activate.bat

Install dependencies

Next, install the following dependencies in the activated .venv by executing the following pip command:

pip install requests bs4 pandas

This command will install the Requests, Beautiful Soup, and Pandas libraries. These modules will help you send requests, parse Wayfair product data, and store it in CSV and JSON formats.

How to scrape Wayfair with proxies

Without proxy servers and a sophisticated scraping approach, you’re likely to run into various blocks, such as Wayfair returning 429 status codes and no data. For this reason, we’ll show you how to utilize residential proxies with a custom scraping solution. You may also utilize other proxy products.

1. Set the target URL and HTTP headers

Let’s begin by importing the Requests, Beautiful Soup, and Pandas libraries in a new Python file:

import requests
from bs4 import BeautifulSoup
import pandas as pd

Next, add the Wayfair product URL you want to scrape. For example, let’s use this Wayfair product page:

url = (
    'https://www.wayfair.com/furniture/pdp/'
    'serta-monroe-full-size-convertible-sofa-zpcd2290.html'
)

Then, define the HTTP headers dictionary to mimic a real web browser request:

headers = {
    'User-Agent': (
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 14.7; rv:138.0) '
        'Gecko/20100101 Firefox/138.0'
    ),
    'Accept': (
        'text/html,application/xhtml+xml,application/xml;'
        'q=0.9,image/avif,image/webp,image/apng,*/*;'
        'q=0.8,application/signed-exchange;v=b3;q=0.7'
    ),
    'Accept-Language': 'en-US,en;q=0.9',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

When scaling your scraping operations, you may want to switch the User-Agent string with each request, or, ideally, rotate different sets of HTTP header.

2. Configure proxies

The below code will ensure your Wayfair scraper will make requests through a residential IP address based in the United States. Feel free to modify the proxy address to suit your needs.

USERNAME, PASSWORD = 'PROXY_USERNAME', 'PROXY_PASSWORD'
proxies = {
    'http': f'https://{USERNAME}:{PASSWORD}@us-pr.oxylabs.io:10000',
    'https': f'https://{USERNAME}:{PASSWORD}@us-pr.oxylabs.io:10000'
}

3. Send a request

Let’s bring all the request settings together by making a GET request to the target URL:

try:
    response = requests.get(url=url, headers=headers, proxies=proxies)
    response.raise_for_status()
except Exception as e:
    print(e)

4. Parse Wayfair data

Once the Requests library returns the HTML from the Wayfair page, you can use Beautiful Soup to extract the specific data you need:

soup = BeautifulSoup(response.text, 'html.parser')

Extract the title

How to scrape Wayfair product data with API

Now, let’s see how to use Oxylabs Wayfair API to extract data from the Wayfair product page. Make sure to create an Oxylabs account on the dashboard and claim a free trial.

1. Import modules and set the target URL

You’ll only need the Requests and Pandas libraries, while the target URL remains the same as previously:

import requests
import pandas as pd


url = (
    'https://www.wayfair.com/furniture/pdp/'
    'serta-monroe-full-size-convertible-sofa-zpcd2290.html'
)

2. Set the request payload with a parsing logic

Instead of relying on Beautiful Soup for data extraction, you can take advantage of the API’s Custom Parser feature. We also have an AI-powered scraping assistant that lets you generate the same parsing instructions by using English prompts (see prompt examples here).

Let’s start by defining the selectors dictionary that uses the same CSS selectors we’ve configured previously:

selectors = {
    'title': 'header > h1',
    'price': '.SFPrice > div:first-child > span:first-child',
    'rating': '.ProductRatingNumberWithCount-rating',
    'description': '.ProductOverviewInformation-description'
}

Next, you can set the request payload that tells Wayfair Scraper API how to handle the network request for Wayfair data scraping. We’ve set the geo_location parameter to United States and defined the parsing_instructions logic:

payload = {
    'source': 'universal',
    'url': url,
    'geo_location': 'United States',
    'parse': True,
    'parsing_instructions': {
        field: {
            '_fns': [
                {'_fn': 'css_one', '_args': [selector]},
                {'_fn': 'element_text'}
            ]
        }
        for field, selector in selectors.items()
    }
}

Since all of the parsing instructions use the same structure, you can easily make a loop that picks the field name and selector for each entry in the selectors dict.

3. Send a scraping request

Make sure to enter your Web Scraper API user credentials. The Requests library helps us send a POST request to the API:

USERNAME, PASSWORD = 'API_USERNAME', 'API_PASSWORD'
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=(USERNAME, PASSWORD),
    json=payload
)
response.raise_for_status()

4. Save Wayfair data to a file

Once the API processes your request, it will send back a response which holds the scraped data in ['results'][0]['content']. You can access this directly from the response.json() and pass it over to Pandas to save it all into a CSV file:

data = [response.json()['results'][0]['content']]
df = pd.DataFrame(data)
df.to_csv('wayfair_data_API.csv', index=False)

Full code sample

import requests
import pandas as pd


url = (
    'https://www.wayfair.com/furniture/pdp/'
    'serta-monroe-full-size-convertible-sofa-zpcd2290.html'
)

selectors = {
    'title': 'header > h1',
    'price': '.SFPrice > div:first-child > span:first-child',
    'rating': '.ProductRatingNumberWithCount-rating',
    'description': '.ProductOverviewInformation-description'
}

payload = {
    'source': 'universal',
    'url': url,
    'geo_location': 'United States',
    'parse': True,
    'parsing_instructions': {
        field: {
            '_fns': [
                {'_fn': 'css_one', '_args': [selector]},
                {'_fn': 'element_text'}
            ]
        }
        for field, selector in selectors.items()
    }
}

USERNAME, PASSWORD = 'API_USERNAME', 'API_PASSWORD'
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=(USERNAME, PASSWORD),
    json=payload
)
response.raise_for_status()

data = [response.json()['results'][0]['content']]
df = pd.DataFrame(data)
df.to_csv('wayfair_data_API.csv', index=False)

When you run the code, it will store results to the wayfair_data_API.csv file:

Comparing different scraping methods

Conclusion

Building a scraper that can send requests as an actual browser and mimic human browsing behavior is quite difficult. Also, you would have to maintain it and keep it up to date with constant changes. Such micromanagement requires in-depth knowledge and extensive scraping experience. If you're looking to streamline this process, you can buy proxy solutions or choose a reliable free proxy list to ensure smoother scraping operations.

With Wayfair Scraper API, you can shift your focus where it matters most - data analysis - instead of dealing with technicalities. The API comes with various features, including JavaScript rendering, batch requests for scraping multiple pages, as well as Google Cloud Storage and Amazon S3 support.

For more e-commerce scraping targets, follow Amazon, Google Shopping, Etsy, and Walmart scraping guides. Oxylabs E-Commerce Scraper API (part of Web Scraper API) enables you to quickly extract data from the top 50 marketplaces.

If you have questions or face issues, get in touch via the 24/7 live chat on our homepage or email us at support@oxylabs.io.