Proxy locations


North America

South America




See all locations

Network statusCareers

Back to blog

E-Commerce Keyword Research: Data Collection Challenges and Solutions

Iveta Vistorskyte

Iveta Vistorskyte

2021-12-157 min read

E-commerce has evolved in many ways and now makes online shopping easier for modern-day customers. In 2019, an estimated 1.92 billion people purchased goods or services online. The number is predicted to grow between 2019 and 2021, from 1.92 billion to 2.14 billion. Online shopping became even more popular in 2020 because of store closures and shoppers’ fear of contracting the COVID-19 in public. Statistics show that the e-commerce field is on the rise, meaning that more and more businesses are selling their products and services online.

However, becoming more visible for customers in the top e-commerce marketplaces requires in-depth research and knowledge. Developing and maintaining an e-commerce SEO strategy may help sellers do a better marketing job. Web scraping is an essential part of keyword research for e-commerce. It helps to collect large amounts of public data for keyword analysis. However, gathering and maintaining data comes with challenges.

What is keyword research?

Keyword research is the cornerstone of digital marketing and should be a top priority when starting an SEO campaign. It is a process of finding and analyzing search terms that internet users enter into search engines when seeking information on various topics. SEO professionals use researched thematic keywords in target web pages to achieve better rankings in search engine results pages. The main intention is to understand how your business is ranking at the moment and what could be done to rank higher.

E-commerce keyword research

Keyword research for top e-commerce marketplaces is part of an SEO strategy. The process helps to find the e-commerce search terms sellers use to rank higher and get more traffic to their product or service pages. When sellers choose keywords for top e-commerce marketplaces, they explore how customers discover things they want to buy. Usually, sellers make a business keyword list that is relevant to their products and categories.

Importance of keyword research

Thoughtful research can provide sellers with information about:

  • competitors and their actions; 

  • ideas for content marketing; 

  • understanding of consumer trends;

  • insights into their customers needs.

However, the research includes more than constantly discovering new product keywords. It is also essential for sellers to track keywords’ performance and analyze the history of multiple keywords that would allow making data driven decisions. Understanding how search algorithms work in e-commerce platforms can be a tremendous driving force for business growth.

How to do keyword research for e-commerce?

To begin understanding the process of keyword research for e-commerce, let’s take a look at the terminology first.

Search volume – the number of searches per month for a particular search phrase. If you manage to orientate your web page to adhere to the top trending phrases, the incoming traffic and conversion potential are likely to soar.

Competition – the difficulty of ranking for a particular keyword. The best-case scenario is when your target keyword has a high search volume and low competition. Although, such an ideal keyword is hard to come by.

Creating a list of keywords

The next step is to think of an initial list of keywords. The main idea is to grasp the thinking process of an average potential client. What kind of phrases would a potential client use to search for a specific product or service? A wide variety of keywords ranging from single words to phrases of typically up to five words are recommended to include in the list. The more keywords, the better.

Using dedicated tools

To make the process easier, you can use dedicated tools to automate the research and gain advanced insights into details whose relevance could be underestimated.

A wide array of scraping tools can help every step of the way. For example, a web scraping tool, SERP Scraper API, can scan search engine results pages for the necessary keywords. E-Commerce Scraper API can collect keywords used by contenders in their product titles and labels to determine current trends that rival businesses adhere to.

All-in-one SEO tools such as Semrush and Ahrefs can help select, manage, and inspect the most relevant keywords based on the location from multiple search engines. Lastly, you can use free tools provided by the search engines themselves, such as Google Keyword Planner Tool.

Finalizing the keyword list

After gathering all of the keywords, it is advisable to finalize the list by searching for the keywords in search engines manually. If a keyword search returns results where the top-ranking companies are the major players in the business, it would be wise to avoid such ruthless competition by updating the list.

Understanding which web pages rank for any given keyword is crucial in figuring out your own potential position among them. Some excellent top-trending keywords might not be worth the effort, while others with less exposure could be a niche where you might excel.

Keyword research for e-commerce is a balancing act of multiple variables and compromises where a combination of tools and efforts should award you with a suitable ranking. In an oversaturated market, the appropriate keyword implementation is paramount if you aren’t a business with a unique product or service that lacks or defies competition.

Types of e-commerce keywords

By now, you should have a decent idea why keyword research is important. It is essential to note a few most common types of e-commerce keywords: product and service details, competitors keywords, and audience terms. We will dig deeper into each of these types to understand what public data can be collected for keyword analysis.

Product and service keywords

Product or service description is essential for several reasons. First of all, it is an opportunity for sellers to interest the customer in buying the product or service. Second, it is a place to use keywords sellers could not fit in the product title. Without optimizing titles and descriptions with relevant keywords, the product page will have fewer chances to appear in the search results.

Sellers use keywords that make their product (or services) relevant to the queries potential clients are searching. Collecting public data on how specific search results change with different queries help sellers understand which keywords to use in descriptions.

Monitoring competitors

Keywords and terms that competitors use can also provide valuable insights. Sellers have to constantly monitor their competition to be aware of what is happening in their market.

Knowing what keywords and terms helped competitors make it to the top of the search results can help build a successful keyword strategy.

Sellers collect public data about their competitors from top e-commerce websites. However, it is important to note that this data has to be used only for analysis, not for copying their content or other malicious intents.

Audience terms

Audience terms reveal a variety of additional interests that sellers might never have guessed to convert well for their products or services. Audience terms can add additional value to descriptions and bring more traffic to product pages. For example, if a seller is providing cake decorating products, keywords about presents or parties may be relevant to their customers as well.

Of course, it is not possible to include all the keywords in the description. This is why top e-commerce platforms have additional fields in seller accounts, where sellers can enter their “hidden keywords.”

The magic behind finding what keywords can be used is simple: collecting public data and analyzing it. Analyzing how specific search results change with different queries is the most common way to do it.

Web scraping for e-commerce keyword research

Simply put, web scraping and especially scraping with Python is a widely used method to gather public information from e-commerce websites. Data gathering bots automatically request and extract data from target websites. Web scraping unlocks an ability to collect data on a large scale in a short period of time.

For sellers who want to ensure their products are ranking high, gathering public data from relevant categories is enough because analyzing their competitors in rankings may increase their revenue. In this case, it is more than enough to use in-house web scrapers powered up by the right proxy type to ensure a smooth data collection process.

However, sellers usually outsource e-commerce scraping services that gather and provide relevant e-commerce data because they lack knowledge and resources or want to save time. Service providers are constantly collecting data for their customers, and they face various challenges due to the amount of data required.

E-commerce scraping challenges

Collecting data on a large scale from leading e-commerce marketplaces is a complicated task. We outlined the most common challenges service providers may face.

  • Overcoming bot detection measures. Top e-commerce websites usually implement security measures that block malicious bots. These measures typically cannot distinguish the visitor’s intent. Therefore, well-mannered web scraping bots that collect information about e-commerce keywords are often mistakenly flagged, making blocks inevitable.

  • Accessing geo-restricted data. Keyword research is often location-based. Specific keyword information may not be accessible from different regions, making the research incomplete and ineffective.

  • Processing gathered information. Processing collected data is called data parsing. The parsing could become increasingly complicated due to the constant layout changes of leading e-commerce marketplaces.

Solutions to deal with e-commerce scraping challenges

Dealing with data gathering challenges may be difficult. However, there are a few ways how service providers choose to approach web scraping.

When using in-house web scrapers, it is essential to note that almost all web scraping projects are impossible without proxies, especially when dealing with large-scale data operations. Proxies unlock content worldwide, meaning that all the data is accessible regardless of its geo-location. They are also used to avoid IP blocks. Residential proxies are less likely to get blocked due to their ability to resemble organic users.

Employing third-party web scraping services dedicates more time to providing clients with an in-depth e-commerce keyword and ranking analysis. Dealing with web scraping challenges will be the third-party service’s responsibility. For example, web crawler tools like E-Commerce Scraper API can get accurate data effortlessly and keep web scraping costs low because clients are paying only for successful e-commerce data delivery. You can check our Google Shopping, Best Buy, eBay, Home Depot, and Wayfair pages for more information, and if you want to see the tool in action, be sure to take a look at our guides, like scraping eBay, scraping Best Buy, and scraping Wayfair.

However, sometimes outsourcing this kind of service may seem costly, and finding a reliable provider may take some time. Each company has to evaluate its resources, budget, and needs before deciding which approach suits them best. There's always an option to purchase e-commerce datasets and get public ready-to-use data for your analysis.


Online shopping is becoming increasingly popular every year. Gathering data from leading e-commerce marketplaces and learning how to use keywords  can help sellers make data-driven decisions, rank higher on search results, and improve content strategy. This leads to increased customer count and revenue.

The process of gathering public e-commerce data on a large scale is challenging. Dealing with anti-bot measures, processing vast amounts of data, and accessing geo-restricted data are just a few of many challenges. Reliable proxies or quality data extraction tools can help facilitate this process. However, companies have to decide which approach is best for them: using and maintaining in-house web scrapers or outsourcing third-party tools. 

If you are interested in more information on scraping e-commerce web pages or how to crawl a website without getting blocked, we suggest you read our other blog posts.

About the author

Iveta Vistorskyte

Iveta Vistorskyte

Lead Content Manager

Iveta Vistorskyte is a Lead Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you'll probably find her just chillin' while listening to her favorite music or playing board games with friends.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Related articles

Get the latest news from data gathering world

I’m interested


  • What is keyword research?

  • E-commerce keyword research

  • How to do keyword research for e-commerce?

  • Types of e-commerce keywords

  • Web scraping for e-commerce keyword research

  • Conclusion

Forget about complex web scraping processes

Choose Oxylabs' advanced web intelligence collection solutions to gather real-time public data hassle-free.

Scale up your business with Oxylabs®