How to Scrape Google Flights with Python

Google Flights is a flight booking service that makes buying tickets from third-party easier. It holds large amounts of flight data, including airfare, destinations, flight times, layovers, departing flights, departure date, and more. All this information is not only valuable for individual travelers but for businesses, too – it helps them conduct competitor analysis, analyze customer preferences, and more. 

Without the right tools and methods, scraping Google Flights listings can get quite challenging. In today's article, we're going to demonstrate how to scrape public data from flight pages and generate search results with Python and Google Flights API. We'll gather all sorts of data, including flight prices, flight time, and airline name. 

Next up, we’ll create a function that accepts a BeautifulSoup object created from the HTML of the whole page. This function will create and return an array of objects containing information from individual flight listings. Let’s try to form the function in such a way that makes it easily extendible if required: 

def extract_flight_information_from_soup(soup_of_the_whole_page):
    flight_listings = soup_of_the_whole_page.find_all('li','pIav2d')

    flights = []

    for listing in flight_listings:
        if listing is not None:
            # Add some specific data extraction here

            flight = {}

            flights.append(flight)

    return flights

Now that we can get the HTML and have a function to hold our information extraction, we can organize both of those into one:

def extract_flights_data_from_urls(urls):
    constructed_flight_results = []

    for url in urls:
        html = get_flights_html(url)

        soup = BeautifulSoup(html,'html.parser')
    
        flights = extract_flight_information_from_soup(soup)

        constructed_flight_results.append({
            'url': url,
            'flight_data': flights
        })

    return constructed_flight_results

This function will take an array of URLs as a parameter and return an object of extracted flight data.

One thing left for our core is a function that takes this data and saves it as a file:

def save_results(results, filepath):
    with open(filepath, 'w', encoding='utf-8') as file:
        json.dump(results, file, ensure_ascii=False, indent=4)

    return

We can finish by creating a simple main function to invoke all that we’ve created so far:

def main():
    results_file = 'data.json'

    urls = [
        'https://www.google.com/travel/flights?tfs=CBsQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgCEggvbS8wNGpwbBooEgoyMDI0LTA4LTAxagwIAhIIL20vMDRqcGxyDAgDEggvbS8wMTU2cUABSAFSA0VVUnABemxDalJJTkRCNVRGbDBOMU5UVEdOQlJ6aG5lRUZDUnkwdExTMHRMUzB0TFMxM1pXc3lOMEZCUVVGQlIxZ3dhRWxSUVRoaWFtTkJFZ1pWTWpnMk1qSWFDZ2lRYnhBQ0dnTkZWVkk0SEhEN2VBPT2YAQGyARIYASABKgwIAxIIL20vMDRqcGw&hl=en-US&curr=EUR&sa=X&ved=0CAoQtY0DahgKEwiAz9bF5PaEAxUAAAAAHQAAAAAQngM',
        'https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgDEggvbS8wN19rcRooEgoyMDI0LTA4LTAxagwIAxIIL20vMDdfa3FyDAgDEggvbS8wMTU2cUABSAFwAYIBCwj___________8BmAEB&hl=en-US&curr=EUR'
    ]

    constructed_flight_results = extract_flights_data_from_urls(urls)

    save_results(constructed_flight_results, results_file)

With the core of the application complete, we can start creating specific data extraction functions that will gather the needed flight information.

3. Getting the price

First up is the price. Navigating the HTML of Google Flights can get tricky, as it seems to be quite dynamic. Let's see how we could pinpoint the price.

The first div to find is the one with the class BVAVmf.

Inside it, we can specify another one – YMlIz – which will contain only the price we need and nothing else.

Now that we got that, we just need to write the function itself:

def get_price(soup_element):
    price = soup_element.find('div','BVAVmf').find('div','YMlIz').get_text()

    return price

4. Getting the flight time

Another important piece of information is the flight time. We’ll begin with the div containing both the flight time and the airline name – Ir0Voe.

Then, we can specify an inner div that contains only the time – zxVSec.

If we inspect this one closer, we’ll find that it has a lot of inner span containers with our time information split and repeated numerous times. But all of the needed time information can be found in the first child span container of the latter div.

We can also make good use of the fact that Oxylabs Web Scraper API renders JavaScript for us and extracts information in a very convenient format, as the exact flight dates can be seen when hovered over with a mouse.

We should keep in mind that there will be two span containers with the class eoY5cb that we will need: one for the departure and one for arrival. We should get them both.

Now that we’ve got all of the needed HTML information, let’s extract it.

def get_time(soup_element):
    spans = soup_element.find('div','Ir0Voe').find('div','zxVSec', recursive=False).find_all('span', 'eoY5cb')

    time = ""

    for span in spans:
        time = time + span.get_text() + "; "

    return time

Then, we can specify an inner div that contains only the airline name – sSHqwe.

One more thing here, we can see that there might be some additional information about the flight and the airline in this field, such as the operator or the fact that tickets for this flight are from multiple airlines.

As all of this information is important for us, let’s try to gather it all. We can see that all of the texts are inside classless span containers. 

The final step is to put everything we found so far into a function for extraction.

def get_airline(soup_element):
    airline = soup_element.find('div','Ir0Voe').find('div','sSHqwe')

    spans = airline.find_all('span', attrs={"class": None}, recursive=False)

    result = ""

    for span in spans:
        result = result + span.get_text() + "; "
    
    return result

Having all of these functions for data extraction, we just need to add them to the place we designated earlier to finish up our code.

def extract_flight_information_from_soup(soup_of_the_whole_page):
    flight_listings = soup_of_the_whole_page.find_all('li','pIav2d')

    flights = []

    for listing in flight_listings:
        if listing is not None:
            price = get_price(listing)
            time = get_time(listing)
            airline = get_airline(listing)

            flight = {
                "airline": airline,
                "time": time,
                "price": price 
            }

            flights.append(flight)

    return flights

6. Final result

If we add all of it together, the final product should look something like this.

from bs4 import BeautifulSoup
import requests
import json

def get_price(soup_element):
    price = soup_element.find('div','BVAVmf').find('div','YMlIz').get_text()

    return price


def get_time(soup_element):
    spans = soup_element.find('div','Ir0Voe').find('div','zxVSec', recursive=False).find_all('span', 'eoY5cb')

    time = ""

    for span in spans:
        time = time + span.get_text() + "; "

    return time


def get_airline(soup_element):
    airline = soup_element.find('div','Ir0Voe').find('div','sSHqwe')

    spans = airline.find_all('span', attrs={"class": None}, recursive=False)

    result = ""

    for span in spans:
        result = result + span.get_text() + "; "
    
    return result


def save_results(results, filepath):
    with open(filepath, 'w', encoding='utf-8') as file:
        json.dump(results, file, ensure_ascii=False, indent=4)

    return


def get_flights_html(url):
    payload = {
        'source': 'google',
        'render': 'html',
        'url': url,
    }

    # Get response.
    response = requests.request(
        'POST',
        'https://realtime.oxylabs.io/v1/queries',
        auth=('username', 'password'),
        json=payload,
    )

    response_json = response.json()

    html = response_json['results'][0]['content']

    return html


def extract_flight_information_from_soup(soup_of_the_whole_page):
    flight_listings = soup_of_the_whole_page.find_all('li','pIav2d')

    flights = []

    for listing in flight_listings:
        if listing is not None:
            price = get_price(listing)
            time = get_time(listing)
            airline = get_airline(listing)

            flight = {
                "airline": airline,
                "time": time,
                "price": price 
            }

            flights.append(flight)

    return flights


def extract_flights_data_from_urls(urls):
    constructed_flight_results = []

    for url in urls:
        html = get_flights_html(url)

        soup = BeautifulSoup(html,'html.parser')
    
        flights = extract_flight_information_from_soup(soup)

        constructed_flight_results.append({
            'url': url,
            'flight_data': flights
        })

    return constructed_flight_results


def main():
    results_file = 'data.json'

    urls = [
        'https://www.google.com/travel/flights?tfs=CBsQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgCEggvbS8wNGpwbBooEgoyMDI0LTA4LTAxagwIAhIIL20vMDRqcGxyDAgDEggvbS8wMTU2cUABSAFSA0VVUnABemxDalJJTkRCNVRGbDBOMU5UVEdOQlJ6aG5lRUZDUnkwdExTMHRMUzB0TFMxM1pXc3lOMEZCUVVGQlIxZ3dhRWxSUVRoaWFtTkJFZ1pWTWpnMk1qSWFDZ2lRYnhBQ0dnTkZWVkk0SEhEN2VBPT2YAQGyARIYASABKgwIAxIIL20vMDRqcGw&hl=en-US&curr=EUR&sa=X&ved=0CAoQtY0DahgKEwiAz9bF5PaEAxUAAAAAHQAAAAAQngM',
        'https://www.google.com/travel/flights/search?tfs=CBwQAhooEgoyMDI0LTA3LTI4agwIAxIIL20vMDE1NnFyDAgDEggvbS8wN19rcRooEgoyMDI0LTA4LTAxagwIAxIIL20vMDdfa3FyDAgDEggvbS8wMTU2cUABSAFwAYIBCwj___________8BmAEB&hl=en-US&curr=EUR'
    ]

    constructed_flight_results = extract_flights_data_from_urls(urls)

    save_results(constructed_flight_results, results_file)


if __name__ == "__main__":
    main()

Alternative methods for scraping Google Flights

If you found yourself needing to extract Google Flights data, you have several options: scraping without an API by building your own custom Google Flights scraper, using or not using proxies, or leveraging a dedicated scraping API like Oxylabs Web Scraper API. Each approach comes with its own set of difficulties and trade-offs. 

If you were to scrape Google Flights without an API – using only the requests library, for example – you would quickly run into significant roadblocks. Web scraping Google Flights is particularly challenging because it is a highly dynamic website that relies heavily on JavaScript to load flight data, making it impossible to get meaningful results with simple HTTP requests. Even if you tried to fetch the raw HTML, you would likely encounter blank pages or incomplete data because the critical content is rendered on the client side using JavaScript. To work around this, you would need a headless browser, such as Selenium or Puppeteer, which adds complexity and slows down the process. Additionally, Google has aggressive anti-bot mechanisms, so frequent requests from the same IP would result in immediate blocks.

On the other hand, using proxies can help mitigate blocking issues, but they don’t solve JavaScript rendering or dynamic content loading problems. Even with high-quality rotating proxies, you’d still need to use browser automation or advanced web scraping techniques like session management and fingerprint spoofing to make your scraper appear human-like. If you’re up for the challenge, you can test Oxylabs free proxies for your Flight Scraper Python based project. However, this approach increases the technical overhead, making it difficult to scale efficiently. On the other hand, with an API like Oxylabs Web Scraper API, these challenges are handled automatically. The API takes care of JavaScript rendering, proxy rotation, and CAPTCHA solving, ensuring that you get structured flight data without dealing with the complexities of scraping Google Flights manually.

While it is technically possible to scrape Google Flights without an API, the process is complex, time-consuming, and prone to frequent blocking. Using proxies can improve success rates but still requires advanced configurations to handle JavaScript and anti-bot systems. A dedicated scraping API like Oxylabs Web Scraper API simplifies the process by handling these challenges for you, providing a scalable and reliable solution to extract data efficiently. You can even use the Google Flights API free for a week once you sign up for the free trial – to test out if the Web Scraper API is the solution for your data extraction project. In any case, it would be safe to say that if your goal is to minimize technical difficulties and maximize success when scraping Google Flights, using an API is the most effective approach. 

Conclusion

In today's article, we've learned how to extract public pricing information, flight times, and airline names from Google Flights. By employing Python and Oxylabs Web Scraper API, we were able to deal with the dynamic nature of Google Flights and extract data successfully. 

Curious about scraping data from other Google platforms? Check out our JobsSearch, Images, Trends, News, Scholar, ShoppingMaps step-by-step guides.

For web scraping, proxies are an essential anti-blocking measure. To avoid detection by the target website, you can buy proxies of various types to fit any scraping scenario.