Back to blog
How to Scrape Data from eBay: A Step-by-Step Guide
Danielius Radavicius
eBay is one of the most popular e-commerce platforms worldwide, serving a wide range of customers to help fulfill their various needs. As such, scraping publicly available data from eBay offers a multitude of benefits, ranging from price monitoring to customer review tracking and many more.
Overall the usefulness of eBay data, especially for someone basing their businesses on e-commerce, is tangible, especially considering the difficulties of conducting effective scraping operations are majorly outweighed by the value they provide. After all, there is a reason why web scraping has become a foundation upon which many competitive strategies and decisions are made.
Challenges with scraping eBay
We’ve mentioned above that scraping eBay is a worthwhile task with benefits that outweigh the challenges. Yet, to successfully extract data from it isn’t easy. Over the years, eBay has introduced multiple anti-scraping measures, meaning that failed jobs and numerous blocks may be common without an effective tool. Thankfully Oxylabs' eBay Scraper API is specifically designed with these difficulties in mind, giving you an all-in-one solution to efficiently and rapidly extract the exact structured data you need.
Structure of an eBay page
1.Search results page
You’ll be presented with the search results page after you put in a specific keyword, in this case let’s use “laptop”. The page you receive will be similar to the one shown in the picture below.
Crucially, all products listed under “laptop” can be extracted as well as their links, titles, prices, ratings, and images.
2. Product page
A standard eBay page contains a large variety of useful information. Using the laptop product page example shown below, we can see a multitude of categories which can be scraped, such as:
Price
Availability
Product title
Product rating
Product image
Shipping cost
Item specifics
Value of scraping eBay
A standard eBay page contains valuable information, such as pictures, prices, availability, page number, URL, status code, and many other crucial features. Combined, extracting data with eBay scraping helps use cases like:
Competitive analysis. Extracting data from a target web page lets you see competitors' products, pricing, and sales volume. Such information helps develop strategies that differentiate your products while allowing you to make more competitive pricing decisions.
Market research. Enable yourself to see which products are selling well and how often they are being sold.
Product development. With eBay scraping, you can extract data to provide insights into what products are in demand. Based on this information, design and development decisions are made.
Having recognized the value extracted data from eBay can provide, let’s begin our brief step-by-step guide for fetching, retrieving, and parsing data from eBay with our API.
Setting up the eBay Scraper API
1. Install and import libraries
Start by installing the requests library in your terminal/console with the below command:
pip install requestsOpen an IDE of your choice and create a Python module file.
In the module file, import the requests, json, and pprint libraries and then type in your API access credentials. If you don’t have them, you can claim a 1-week free trial to test Oxylabs eBay Scraper API by registering on the dashboard.
import requests, json
from pprint import pprint
credentials = ('USERNAME', 'PASSWORD')2. Create a payload dictionary
Define a new dictionary, payload, and then select the source for the API and provide the eBay URL to obtain the desired eBay product data. Ensure the value of source is universal_ecommerce:
import requests, json
from pprint import pprint
credentials = ('USERNAME', 'PASSWORD')
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.ebay.com/itm/225313054444',
}3. Localize eBay results
If needed, you can also specify a geo-location to fetch localized eBay product data:
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.ebay.com/itm/225313054444',
'geo_location': 'Canada', # For fetching localized product data
}Find more information about the available geo_location values in our documentation.
4. Parse eBay product data
The next step is to extract eBay product data by utilizing Custom Parser, a free Oxylabs API feature. Add the 'parse': True and 'parsing_instructions' parameters and then define your custom parsing logic using CSS or XPath selectors:
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.ebay.com/itm/225313054444',
'geo_location': 'Canada', # For fetching localized product data
'parse': True,
'parsing_instructions': {
'title': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//h1/span/text()']
}
]
},
'price': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//div[contains(@class, "price-primary")]/span/text()']
}
]
},
'seller_url': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//div[contains(@class, "about-seller")]//a/@href']
}
]
},
'availability': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//div[contains(@class, "availability")]/span/text()']
}
]
},
'item_specifics': {
'_fns': [
{
'_fn': 'xpath',
'_args': ['//div[contains(@class, "evo__col")]']
}
],
'_items': {
'key': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['.//div[contains(@class, "labels-content")]//text()']
}
]
},
'value': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['.//div[contains(@class, "values-content")]//text()']
}
]
}
}
}
}
}These instructions tell the API to parse the product title, price, seller URL, availability, and the item specifics table. Feel free to add additional parsing instructions if needed. For a deeper look at how to use the Custom Parser feature, visit our documentation and see this GitHub repository.
5. Send a request
With the configuration done, you can now send a POST request to the API endpoint using the requests library:
response = requests.post(
'https://realtime.oxylabs.io/v1/queries',
json=payload,
auth=credentials # Pass your API credentials
)6. Save eBay data to a JSON file
Start by printing the entire API response, which contains the scraped eBay results together with the information about the submitted job. Next, to access the eBay product data, you need to access the content key from the delivered response, which you can then use to save data to a JSON file:
pprint(response.json()) # Get a pretty-print of the API's response
with open('ebay_product_data.json', 'w') as f: # Save the results to a JSON file
json.dump(response.json()['results'][0]['content'], f, indent=4)Complete eBay product scraper code
Once you put everything together, you should have the following Python code that extracts eBay product data:
import requests, json
from pprint import pprint
credentials = ('USERNAME', 'PASSWORD')
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.ebay.com/itm/225313054444',
'geo_location': 'Canada', # For fetching localized product data
'parse': True,
'parsing_instructions': {
'title': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//h1/span/text()']
}
]
},
'price': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//div[contains(@class, "price-primary")]/span/text()']
}
]
},
'seller_url': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//div[contains(@class, "about-seller")]//a/@href']
}
]
},
'availability': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['//div[contains(@class, "availability")]/span/text()']
}
]
},
'item_specifics': {
'_fns': [
{
'_fn': 'xpath',
'_args': ['//div[contains(@class, "evo__col")]']
}
],
'_items': {
'key': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['.//div[contains(@class, "labels-content")]//text()']
}
]
},
'value': {
'_fns': [
{
'_fn': 'xpath_one',
'_args': ['.//div[contains(@class, "values-content")]//text()']
}
]
}
}
}
}
}
response = requests.post(
'https://realtime.oxylabs.io/v1/queries',
json=payload,
auth=credentials # Pass your API credentials
)
pprint(response.json()) # Get a pretty-print of the API's response
with open('ebay_product_data.json', 'w') as f: # Save the results to a JSON file
json.dump(response.json()['results'][0]['content'], f, indent=4)After running the code, the saved JSON file should look like this:
{
"price": "US $549.99",
"title": "15.6\" TOUCH Dell laptop Precision 5530 i7 8850H P2000 32GB 1TB SSD Win 11 pro",
"seller_url": "https://www.ebay.com/str/kl0?_trksid=p4429486.m3561.l161211",
"availability": "Last One",
"item_specifics": [
{
"key": "Condition",
"value": "Excellent - Refurbished"
},
{
"key": "Seller Notes",
"value": "\u201cOne Year Warranty\u201d"
},
{
"key": "Processor",
"value": "Intel Core i7 8th Gen."
},
{
"key": "Screen Size",
"value": "15.6 in"
},
{
"key": "Graphics Processing Type",
"value": "Integrated/On-Board Graphics"
},
{
"key": "RAM Size",
"value": "32 GB"
},
{
"key": "Color",
"value": "Black"
},
{
"key": "Custom Bundle",
"value": "No"
},
{
"key": "SSD Capacity",
"value": "1tb"
},
{
"key": "Most Suitable For",
"value": "Casual Computing, Graphic Design, Parts"
},
{
"key": "GPU",
"value": "Nvdia p2000"
},
{
"key": "Processor Speed",
"value": "2.70 GHz"
},
{
"key": "Brand",
"value": "Dell"
},
{
"key": "Series",
"value": "Precision"
},
{
"key": "Type",
"value": "Notebook/Laptop"
},
{
"key": "Maximum Resolution",
"value": "FHD"
},
{
"key": "Model",
"value": "Dell Precision 5530"
},
{
"key": "Connectivity",
"value": "Gigabit Ethernet, USB 2.0, USB 3.0"
},
{
"key": "Hard Drive Capacity",
"value": "1 TB"
},
{
"key": "Features",
"value": "Bluetooth, Wi-Fi"
},
{
"key": "Operating System",
"value": "Windows 11 Pro"
},
{
"key": "Storage Type",
"value": "SSD (Solid State Drive)"
}
],
"parse_status_code": 12000
}Conclusion
And that’s it!
This guide has covered the process of fetching and parsing eBay product data using an eBay Scraper API. The steps outlined are straightforward and easy to follow. If you're interested in learning more about the features of the eBay Scraper API available, you can consult the documentation for additional details.
People also ask
Does eBay allow scraping?
You can scrape eBay if you’re not accessing data behind login walls or personal data without consent. Overall, scraping is not illegal as long as it is done without breaking any applicable rules or laws surrounding the targeted websites or gathered data. Accordingly, we recommend that before engaging in any scraping activities, you should get appropriate professional legal advice regarding your specific situation.
Why do people scrape eBay?
eBay remains one of the biggest e-commerce marketplaces worldwide. As such, scraping is valuable for a plethora of use cases:
Price comparison. Scraping eBay allows you to keep track of constantly changing product prices, meaning monitoring and adjusting prices according to the data gathered becomes a simplistic task.
Market research. Since millions of products are listed on eBay, gathering and analyzing the data from these listings provides the opportunity to determine product popularity and selling frequency.
Competitor analysis. You may want to scrape data on your competitors in order to determine their pricing strategies, product offerings, and sales performance.
Can scraping be detected?
Scraping can be detected. If website owners monitor their server logs, analyze the traffic patterns on their websites, and use tools like web scraping detection software, your scraping efforts may be blocked.
Thankfully our eBay API is specifically designed to counter the above-mentioned anti-scraping measures.
About the author
Danielius Radavicius
Former Copywriter
Danielius Radavičius was a Copywriter at Oxylabs. Having grown up in films, music, and books and having a keen interest in the defense industry, he decided to move his career toward tech-related subjects and quickly became interested in all things technology. In his free time, you'll probably find Danielius watching films, listening to music, and planning world domination.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Related articles
Get the latest news from data gathering world
Scale up your business with Oxylabs®
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub
oxylabs.io© 2024 All Rights Reserved