How to Scrape Google Maps: A Comprehensive Guide

In the current day and age, where public web data scraping has become a foundation for many businesses, it’s unsurprising to see that Google Maps is yet another area commonly scraped for its valuable data. In this article, we’ll discuss what this data may be and how to build a scraper that gathers it using an Oxylabs solution.

Before we get started, let’s briefly look at the legalities of scraping Google Maps. The legality of web scraping is a much-debated topic among everyone who works in the data-gathering field. It’s important to note that web scraping may be legal in cases where it’s done without breaching any laws regarding the source targets or data itself. That being said, we advise you to seek legal consultation before engaging in scraping activities of any kind. 

We’ve explored the legality of web scraping in this blog post, so feel free to check it out for a more in-depth explanation.

Why scrape Google Maps?

The core purposes of scraping Google Maps are numerous. From a research perspective, a user may want to employ a Google Maps data scraper to analyze demographic information or transportation routes. For businesses, a Google Maps scraper may be the go-to tool for competitor analysis, as it allows you to collect data on competitors' locations, customer reviews, and ratings. Gathering real estate/property listings is a possible use case as well.

Overall, this makes Google Maps data scraping a highly lucrative solution that many businesses are certain to make use of.

Should you use the official Google Maps API?

Quite a few popular websites like Twitter or Amazon provide their own APIs. Google is no exception, therefore naturally the question arises, why not use the official Google Maps API?

Let’s begin with the price. Each user gets 200$ monthly credit for API calls. Within these 200$ are:

• Up to 40,000 Geolocation calls

• Up to 100,000 Static Maps loads

• Up to 28,000 Dynamic Maps loads

• Up to 40,000 Directions calls

At first glance, this may appear as plenty, but it’s likely not. Google's API, like many other APIs, begins to charge you when the given amount is used up. Then, imagine a scenario where you use the Embed API in Directions, Views, and Search modes. Suppose your service loads a map that initiates address search through autocomplete. This singular request is now using up 2 different API calls. Add another requirement, say geolocation services for directions or distances, and now a single request is taking up 3 separate API calls. Furthermore, as your business scales, so does the daily amount of calls you’ll make, meaning after a certain point, Google Maps API becomes an unbelievably pricey solution.

Yet, the high price isn’t the only limitation of Google’s own API. There are also strict request limitations. Google’s current enforced rate limit is up to 100 requests per second.

Google is also known to implement unpredictable changes that offer little benefit to their users, such as the limits imposed in 2010.

However, products like Oxylabs' Google Maps API solution are specifically made to avoid limitations such as the ones mentioned above, which is why they’re commonly chosen instead of official APIs.

As seen in the above image, you can select the name of the restaurant with the following CSS selector:

[role="heading"]

The following are all the selectors:

name_selector = '[role="heading"]'
rating_selector = 'span[aria-hidden="true"]'
rating_count_selector = '[class*="RDApEe"]'
hours_selector = '.rllt__details div:nth-of-type(4)'
details_selector = '.rllt__details div:nth-of-type(5)'
price_selector = '.rllt__details div:nth-of-type(2) > span:nth-of-type(2)'
lat_selector = '[data-lat]'
lng_selector = '[data-lng]'
type_selector = '//div[@class="rllt__details"]/div[2]/text()'
address_selector = '.rllt__details div:nth-of-type(3)'

Note the xPath selector instead of CSS for the type_selector.

You can use BeautifulSoup's select and select_one methods to select elements and then extract the text within those elements.

Rating count needs a different approach. The rating count is enclosed in brackets along with the rating. For example, 4.3(513). In this case, the rating count is within the brackets.

Hence, you can use regex (regular expressions) to extract this value as follows:

count_match = re.search(r"\((.+)\)", rating_count_el.text)
rating_count = count_match.group(1) if count_match else ""

Putting everything together, the following code generates a list of dictionaries that contain all the data from all the listings on the page:

data = []
for html in html_files:
    soup = BeautifulSoup(html, 'html.parser')
    lxml_obj = lxml.html.fromstring(str(soup))
    index = -1

    for listing in soup.select('[class="VkpGBb"]'):
        index += 1
        place = listing.parent
        name_el = place.select_one(name_selector)
        name = name_el.text.strip() if name_el else ''
        
        rating_el = place.select_one(rating_selector)
        rating = rating_el.text.strip() if rating_el else ''
        
        rating_count_el = place.select_one(rating_count_selector)
        rating_count = ''
        if rating_count_el:
            count_match = re.search(r'\((.+)\)', rating_count_el.text)
            rating_count = count_match.group(1) if count_match else ''
        
        hours_el = place.select_one(hours_selector)
        hours = hours_el.text.strip() if hours_el else ''
        if 'opens' not in hours.lower():
            hours = ''
        
        details_el = place.select_one(details_selector)
        details = details_el.text.strip() if details_el else ''
        
        price_level_el = place.select_one(price_selector)
        price_level = price_level_el.text.strip() if price_level_el else ''
        
        lat_el = soup.select_one(lat_selector)
        lat = lat_el.get('data-lat') if lat_el else ''
        
        lng_el = soup.select_one(lng_selector)
        lng = lng_el.get('data-lng') if lng_el else ''
        
        type_el = lxml_obj.xpath(type_selector)
        place_types = []
        for item in type_el:
            parts = item.strip().split('·')
            non_empty_parts = [part.strip() for part in parts if part.strip()]
            if non_empty_parts:
                place_types.append(non_empty_parts[-1])
        
        address_el = place.select_one(address_selector)
        address = address_el.text.strip() if address_el else ''
        
        place = {
            'name': name,
            'place_type': place_types[index],
            'address': address,
            'rating': rating,
            'price_level': price_level,
            'rating_count': rating_count,
            'latitude': lat,
            'longitude': lng,
            'hours': hours,
            'details': details,
        }
        data.append(place)

The next step is to save this data as CSV.

Exporting Google Maps Data to CSV

With the data parsed, the final step is to export it to a CSV file. Let’s use the Pandas library to create a DataFrame and save it as a CSV file:

df = pd.DataFrame(data)
df.to_csv("data.csv", index=False)

When you run this code, it will save the data to a CSV file named data.csv.