How to Scrape Google News: Step-by-Step Guide

Google News is a personalized news aggregation platform that curates and highlights relevant stories worldwide based on user interests. It compiles news and headlines from various sources, ensuring easy access from any device. An essential feature is "Full Coverage," which delves deeper into stories by presenting diverse perspectives from different outlets and mediums.

In this tutorial, you'll learn how to scrape Google News data in two ways: by writing a custom scraper and by utilizing a ready-made Google News Scraper. By following the steps outlined, you'll also learn how to mitigate the anti-bot scraping challenges of Google News. Although, before continuing, check out this article to learn more about news scraping.

Project requirements

Make sure you have Python installed from the official website. The code samples shown in this blog post are written using Python 3.12.0. You'll also need to install the following libraries:

beautifulsoup4==4.13.3
pandas==2.2.3
requests==2.32.3

You can use pip to install these modules via your terminal:

pip install requests bs4 pandas

The requests library simplifies making HTTP calls to Google, while beautifulsoup4 parses the raw HTML to extract your needed data, and pandas provides an easy way to save your results to CSV files.

Scrape Google News using requests and Beautiful Soup

There are a few distinct ways Google offers news results:

• Through the dedicated Google News website:
  https://news.google.com/home

• By accessing the News tab on Google search results:
  https://www.google.com/search?q=stock+market&tbm=nws

• And the RSS feed URL:
  https://news.google.com/rss/headlines/section/topic/WORLD

For this tutorial, let’s scrape news articles from the search result pages.

1. Send a request

Create a new Python file and import the libraries:

import requests
from bs4 import BeautifulSoup
import pandas as pd

Next, make a GET request to the Google News URL. Once the request returns a response, use the BeautifulSoup instance to prepare the HTML for parsing. 

This approach doesn't use a headless browser, so any dynamic data loaded via JavaScript won't appear in the HTML file. To work around this limitation, either save the scraped HTML document and open it in your browser or use Dev Tools to disable JavaScript when viewing the Google News page. This step ensures your CSS selectors match the actual HTML you're scraping, not the JavaScript-enhanced version.

response = requests.get(
    url='https://www.google.com/search?q=stock+market&tbm=nws'
)

with open('page.html', 'w') as f:
    f.write(response.text)

soup = BeautifulSoup(response.text, 'html.parser')

You can open the saved HTML file in your browser to view the data you're working with. If you don't see the expected search results and instead encounter a policy page, CAPTCHA, or other unexpected content, refer to the next step below.

2. Bypass content blocks

If the content you want to scrape is blocked, your IP address might be the issue. To solve this problem, you can use proxy servers to replace your actual IP address with a proxy’s IP.

Residential Proxies offer superior performance as they utilize IP addresses provided by established Internet Service Providers (ISPs) with excellent online reputation. For this reason, let’s use this proxy type to make requests by modifying the previous code:

USER = 'proxy_username'
PASS = 'proxy_password'

response = requests.get(
    url='https://www.google.com/search?q=stock+market&tbm=nws',
    proxies={
        'http': f'https://customer-{USER}:{PASS}@us-pr.oxylabs.io:10000',
        'https': f'https://customer-{USER}:{PASS}@us-pr.oxylabs.io:10000'
    }
)

Visit our documentation to learn how to use Residential Proxies and see more code examples. Additionally, you may want to use a headless browser such as Selenium or Playwright to make your requests even more resilient to blocks.

Note: contact our Customer Support Team to enable Google domains for your acquired proxies.

3. Extract news data

Once you’re able to access news headlines, the next step is to create the parsing logic with CSS selectors. The idea is to select all news article cards and then iterate through each card to extract specific data. Let’s start by finding a way to select all news articles on the page.

Select all article cards

Open your browser and either load the saved page.html file or navigate to https://www.google.com/search?q=stock+market&tbm=nws and disable JavaScript rendering (see this tutorial for the Chrome browser). 

Next, open Developer Tools by right-clicking anywhere on the web page and selecting Inspect. Make sure you’re inside the Elements tab (or Inspector) and have enabled element selection by clicking the pointer icon in the top-left corner of the developer tools panel. With this option enabled, you can click on any element on the page, and its corresponding HTML section will be highlighted in the panel. 

Since there are two different types of news cards on the page, the CSS selector we’ll use is div.X7NTVe > a, div.pkphOe > a. Open the search function in the Dev Tools panel by pressing CTRL + F (or Cmd + F on Mac), then paste your CSS selector. This lets you quickly verify the selector works correctly.

In your Python file, add these lines of code:

articles = []
for article in soup.select('div.X7NTVe > a, div.pkphOe > a'):

Extract the title

You should find the news article title inside each card’s <h3> element.

With this in mind, you can update the code like so:

articles = []
for article in soup.select('div.X7NTVe > a, div.pkphOe > a'):
    title = article.select_one('h3').text

Extract the link

Next, the article’s link can be parsed from the href attribute.

Add this additional line inside the for loop:

   href = article.get('href').replace('/url?q=', '')

Extract the source name

You can find the name of the publisher by selecting two different element classes: .aJyiOc, .lRVwie.

Include the following line in your code:

   source = article.select_one('.aJyiOc, .lRVwie').text

Extract the time of publication

For all articles, you can find the time of publication by selecting span.r0bn4c.rQMQod.

Your for loop should look like this now:

articles = []
for article in soup.select('div.X7NTVe > a, div.pkphOe > a'):
    title = article.select_one('h3').text
    href = article.get('href').replace('/url?q=', '')
    source = article.select_one('.aJyiOc, .lRVwie').text
    time = article.select_one('span.r0bn4c.rQMQod').text

Next, inside the loop, append all the parsed data for each article to the articles list:

   articles.append({
        'title': title,
        'link': href,
        'source': source,
        'published': time
    })

4. Save parsed articles to CSV

Finally, you can save all the extracted news articles to a CSV file using pandas:

df = pd.DataFrame(articles)
df.to_csv('news_1.csv', index=False)

Full code sample

The final version of your code should be:

import requests
from bs4 import BeautifulSoup
import pandas as pd


USER = 'proxy_username'
PASS = 'proxy_password'

response = requests.get(
    url='https://www.google.com/search?q=stock+market&tbm=nws',
    proxies={
        'http': f'https://customer-{USER}:{PASS}@us-pr.oxylabs.io:10000',
        'https': f'https://customer-{USER}:{PASS}@us-pr.oxylabs.io:10000'
    }
)

soup = BeautifulSoup(response.text, 'html.parser')
with open('page.html', 'w') as f:
    f.write(response.text)

articles = []
for article in soup.select('div.X7NTVe > a, div.pkphOe > a'):
    title = article.select_one('h3').text
    href = article.get('href').replace('/url?q=', '')
    source = article.select_one('.aJyiOc, .lRVwie').text
    time = article.select_one('span.r0bn4c.rQMQod').text

    articles.append({
        'title': title,
        'link': href,
        'source': source,
        'published': time
    })

df = pd.DataFrame(articles)
df.to_csv('news_1.csv', index=False)

After executing this custom news scraper, you’ll have all the articles neatly scraped into a CSV file, which can be opened in Excel, Google Sheets, or any other program that supports CSV: