Iveta Vistorskyte

May 21, 2021 8 min read

Experts have regarded price intelligence as a key to saving companies during the COVID-19 pandemic. They cite the fact that the COVID-19 situation had reset the pricing strategies that used to work before due to the economic fallout that ensued. As such, competitive price intelligence is needed to preserve margins and make enough profit to stay afloat throughout the pandemic.

However, this competitive price intelligence data must be collected and structured for analysis. For this reason, it could prove challenging for a company to perform every task, i.e., public data collection and structuring, analysis, and decision-making, not to mention even more challenges lie in wait for an organization that chooses this route, as we detail later.

These conflicting aspects call for a rethink – instead of collecting data, businesses can outsource this service from firms that specialize in this very service and, on their part, only focus on data-driven decision making. This article aims to discuss price intelligence with this pivotal approach in mind.

What is pricing intelligence?

Pricing or price intelligence refers to the monitoring, collection, and processing of public pricing data to understand the market, optimize pricing strategies, preserve margins, and increase profits. When this process gravitates towards data regarding competitors’ prices, it is referred to as competitive pricing intelligence. Notably, price intelligence has been delineated as a way of creating long-term competitive advantages.

According to a McKinsey & Company article, companies that weather the effects of an economic crisis institute measures during the downturn that put them in the path of success once recovery comes. They usually emphasize the need to creatively meet customer needs while preservice value, inculcating pricing discipline, and investing in prospects as the surefire interventions to deal with the issues. On the pricing discipline front, McKinsey also recommends frequently reviewing incentives and price realization targets to ensure balance – the incentives should not eat into profits in the name of incentivizing consumers to purchase.

While the article made several other recommendations, it highlighted, directly and indirectly, the importance of pricing intelligence. By and large, real-time pricing data is central to the realization of the various benefits mentioned.

Uses of collected public price data

The public price data collected in the first stages of price intelligence efforts can be used to develop strategies that deal with pricing challenges. These strategies, which are essentially the use cases of such price data, are trifurcated into:

  • Dynamic pricing
  • MAP monitoring
  • Competitive pricing

Dynamic pricing

Dynamic pricing is the most common strategy of the three. It refers to the practice whereby businesses set flexible prices based on various external and internal influences. The internal factors include shipping and production costs and available stocks, while external factors include demand, competition/competitor prices, prevailing economic conditions, season, among others.

Regardless of the perceived difficulties, this strategy has proven advantageous as it has been tied to the increased revenue in some industries, as established in a study by McKinsey.

MAP monitoring

Minimum Advertised Price (MAP) monitoring refers to tracking the prices of products across various online marketplaces to identify merchants who are not adhering to the pricing policy of a particular product. Notably, MAP refers to the lowest price at which a seller and resellers can display a for-sale product. Given that a typical market comprises multiple sellers and resellers, the pricing (MAP) agreement keeps the prices reasonably uniform across the different online marketplaces.

However, some rogue sellers may wish to undercut competitors in a bid to attract more customers. While this may benefit them, it may also damage the supplier’s reputation and brand image. Hence the need for MAP monitoring.

Competitive pricing

Competitive pricing is the strategy wherein retailers consider competitors’ prices when coming up with their own prices. It is integral to maintaining a profitable business as a study by a leading search engine platform established – 87% of shoppers buy products when they feel that they have gotten a good deal. Thus, if a seller only sets prices based on internal factors and their target profit margins, which could be higher than other sellers, they are bound to lose.

When the data monitored, collected, and analyzed concerns competitors’ prices, it is meant to be used to come up with competitive prices for products or services – the process is known as competitive pricing intelligence.

87% of shoppers buy products when they feel that they have gotten a good deal

Challenges of price intelligence

As stated, price intelligence challenges exist. The main difficulties lie in the data collection process.

Large amounts of available data

The volume of data to be collected is quite substantial. Being that it has to be sourced from tens of websites, if not hundreds, means that data collection is indeed complex. This complexity calls for the development of an in-house web scraping tool, which in turn requires the allocation of resources to hire an experienced team of developers. You should also note that you will need to put somewhere all of the data you collect. For more information, check out another article about large-scale web scraping, where you will find information on how many megabytes of data go through within one second and what other challenges of large-scale data collection are.

Various anti-scraping techniques

Web developers loathe malicious web scraping efforts as such efforts could potentially usurp the website’s resources that could have been deployed in other more critical functions. To prevent this, they use anti-scraping techniques such as CAPTCHAs, IP address blocking/blacklisting, sign-in requirements, honeypot traps, user-agents (UA) checking, and AJAX. Even if web scrapers are used for ethical purposes, web servers hardly distinguish good bots from bad bots, meaning that these price intelligence challenges are inevitable.

Dynamic content and complex website structures

Being a form of web scraping, pricing intelligence is negatively impacted by complex and regularly changing website structures and dynamic content. Web scraping tools have to adapt to constant changes of data source, meaning that it requires a lot of knowledge and resources.

Data collection solutions for price intelligence

E-commerce is a dynamic industry that has evolved with time. Presently, and perhaps for the foreseeable future, the business strategies guiding online commerce depend on real-time data and, more crucially, price intelligence.

While knowing the importance of this intel is one thing, accessing it fast, smoothly and subsequently using it is another. This is where the public data collection solutions in the form of third-party price scraper tools come in. Of course, a company can build its custom price scraper, but this would also require allocating more resources, as stated, proxy maintenance, and the capacity to deal with anti-scraping mechanisms. These issues make the third-party tools the better option.

In addition to overcoming the challenges of pricing intelligence highlighted above, third-party web scraping solutions promote convenience. By freeing resources, such as skilled employees, time, and money, you as the manager can subsequently deploy to other more crucial tasks, such as data analysis. Third-party web scraping tools make it easy to gain and maintain a competitive advantage, increase sales, and improve profit margins, especially in the competition-laden e-commerce industry.

Notably, though, the aforementioned reliability is not always guaranteed, which brings us to the question: how do you choose a third-party web scraping tool?

Characteristics of a good public web scraping tool

Reliable web scraping tools have the following characteristics:

  • They are scalable; they can extract large and small volumes of publicly available data when required. 
  • They have systems in place to avoid anti-scraping tools. 
  • They change as per structural modifications of websites. 
  • They support various data delivery formats, including XML, JSON, and CSV, or deliver the data to cloud storage. 
  • They provide quality, clean, and structured data.
Business strategies guiding online commerce depend on real-time data


Running an online business requires striking a delicate balance between what operations to perform in-house and what to outsource from third-party providers. This choice is particularly crucial when it comes to price intelligence. Given the data-driven nature of e-commerce decision-making, a business needs access to the publicly available pricing data. But price intelligence challenges hinder smooth public data collection.

Fortunately, there is an opportunity to make the data collection process easier by using third party web scraping tools. Of course, companies should take great care when choosing such providers because not all provide quality and reliable scraping tools. Finding a provider that offers a good price intelligence web scraper is the key to success. It allows the business to focus solely on analysis, leading to data-driven decision-making.

If you are interested in public web scraping and its benefits, we suggest you read everything about e-commerce keywords research and e-commerce data sources you should be scraping in 2021.


About Iveta Vistorskyte

Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you'll probably find her just chillin' while listening to her favorite music or playing board games with friends.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Related articles

Meeting the Most Ambitious SEO Needs With Fresh Data
Meeting the Most Ambitious SEO Needs With Fresh Data

Jan 26, 2022

3 min read

What Is a Proxy Server [2022 Guide]
What Is a Proxy Server [2022 Guide]

Jan 03, 2022

20 min read

How Alternative Data Drives E-Commerce Success
How Alternative Data Drives E-Commerce Success

Dec 30, 2021

5 min read