How to Scrape Google People Also Ask: Python Tutorial

Scraping Google’s “People Also Ask” (PAA) section is a great method to gather insights into user search intent and identify questions related to what people commonly ask online. These questions can offer content ideas, improve SEO, and help build an effective content strategy.

In this tutorial, you will learn how to scrape the PAA box using Python, extract the data with the help of BeautifulSoup, and save it as a file. 

What is Google’s People Also Ask?

Google’s People Also Ask is a feature that appears in the search engine results pages in response to user queries. It provides a box of questions related to the original question that users frequently search for, which can be expanded to reveal brief answers. Using this data is incredibly useful for SEO purposes, and if you'd prefer an automated solution, you can try Oxylabs' dedicated Google Ads Scraper API for gathering Google Ads data or their Google Search Autocomplete API for scraping search suggestions or even integrate it with Google Sheets for easy data analysis.

Step-by-Step Tutorial: How to Scrape Google People Also Ask with Python

Setting up the environment

For this tutorial, we will require requests and BeautifulSoup4 Python libraries. Let’s install them using pip:

pip install requests json bs4

Connecting to Google

Now that we have all the required libraries, we will create a file main.py that will hold all of our code and start by making a request to Google:

import requests


# Encode the query for use in a URL
query = "search query"
query = query.replace(' ', '+')


# URL for the Google search
url = f"https://www.google.com/search?q={query}"


headers = {
   "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}


# Send the GET request to Google
response = requests.get(url, headers=headers)


# If the request is successful (status code 200)
if response.status_code == 200:
   print(response.status_code)
else:
   print(f"Error: Unable to fetch the search results. Status code: {response.status_code}")

In the code above, we replace spaces with plus (+) characters in our search query, so that we can use it as a URL parameter. We then construct the URL, add some headers so that Google lets us through and send the request.

Finding and extracting PAA questions

Next, let’s focus on locating and getting those questions. We will use the HTML of the page that we got from our request to Google to create a BeautifulSoup object and then search it for the question.

Before we begin coding, we need a CSS identifier for the question objects so we can select them later. We can find it by inspecting the Google page source through a browser:

All that is left now is to fetch the text from these objects using BeautifulSoup:

import requests
from bs4 import BeautifulSoup


# Define a function to search Google
def get_soup_from_google_search(query):
   # Encode the query for use in a URL
   query = query.replace(' ', '+')


   # URL for the Google search
   url = f"https://www.google.com/search?q={query}"


   headers = {
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
   }


   # Send the GET request to Google
   response = requests.get(url, headers=headers)


   # If the request is successful (status code 200)
   if response.status_code == 200:
       # Parse the HTML content with BeautifulSoup
       soup = BeautifulSoup(response.text, "html.parser")
       return soup
   else:
       print(f"Error: Unable to fetch the search results. Status code: {response.status_code}")
       return None


query = "how do you get diabetes"
soup = get_soup_from_google_search(query)


if soup:
   for question in soup.select('span.CSkcDe'):
       print(question.text)

We have created a method called get_soup_from_google_search that does what the name suggests and takes in the desired query as a parameter. Then we check if the object is not empty, find all of our previously identified PAA question objects and print out their text. For automating this process, consider leveraging Oxylabs' Google Books API or Google Events API to gather more specialized data.

If we run the code, we can see all of the questions printed out:

We can now inspect our results file or download the data to your local software for further analysis.

Other considerations

The important thing to consider here is that Google will quickly clamp down on you with limitations if you try to scrape above a certain threshold. So proxies and rotating user agents should be used to circumvent issues like IP blocks and CAPTCHA. For more robust scraping, you might want to explore Google Carousel API to handle carousel elements in search results.

How to Track People Also Ask Data? 

As People Also Ask Data is something that changes over time, it is important to schedule a script like this to run periodically. After some time, you could start tracking shifts in user searching intents and update your content strategies accordingly.

Conclusion

Scraping Google’s People Also Ask (PAA) section can provide valuable insights into user behavior and search patterns, which can help you optimize your content strategy. With the Python script we developed, you can easily extract PAA questions and store them for further analysis. By staying informed on trending keywords, you’ll be better equipped to refine your content and boost SEO efforts over time. 

For a more advanced approach, Oxylabs offers tools for scraping search results, including Google Featured Snippet API.