How to Scrape TripAdvisor with Python in 2025

TripAdvisor is a prominent platform in the travel and hospitality industry. It offers a wealth of data on hotels, restaurants, and attractions, along with user reviews, making it a good target for web scraping and a valuable resource for market research, competitor analysis, and, in turn, decision-making. Using a proxy server can help ensure smooth data extraction and avoid blocking while gathering large volumes of information.

You can scrape data like names, addresses, contact info, ratings, user-generated reviews, images, pricing, and geographic coordinates to enhance your understanding of the industry.

In this tutorial, you’ll learn how to scrape TripAdvisor data using Python and Residential Proxies.

Why scrape TripAdvisor?

TripAdvisor hosts millions user reviews, ratings, and rankings for hotels, restaurants, and attractions worldwide. By extracting this data, businesses can gain valuable insights into public opinions, customer sentiment, identify trends, and improve their services based on real user feedback. For example, a hotel chain could analyze TripAdvisor reviews to pinpoint recurring customer complaints and make data-driven improvements for customer satisfaction, while a restaurant group might track competitor ratings and pricing strategies to improve their own review pages.

For travel agencies and tourism boards, scraping TripAdvisor allows for detailed competitive analysis and market research. They can monitor destination popularity, traveler preferences, and seasonal trends to refine their offerings and marketing campaigns. Additionally, data from TripAdvisor can be used to enhance localized search results, helping businesses tailor promotions to specific regions. Researchers and data analysts also benefit from scraping TripAdvisor by using its vast dataset for sentiment analysis, trend prediction, and consumer behavior studies, making it a valuable resource for academic and commercial research alike.

Beyond business applications, individuals looking for travel insights can also benefit from TripAdvisor data scraping. Instead of manually browsing all the reviews and ratings, data scraping can help you compile and filter information more efficiently, comparing accommodations, restaurants, and activities based on personal preferences. Whether for business intelligence, market analysis, or personal travel planning, scraping TripAdvisor provides a wealth of data that can be leveraged for smarter decision-making.

If you're seeking enhanced anonymity and the ability to bypass geo-restrictions in your web scraping endeavors, you should look into proxies. Residential proxies would be the best solution, however, other options, such as datacenter IPs or even free proxies (if offered from reputable providers) can be a good option. Integrating residential proxies into your scraping setup can offer greater flexibility and control over the web scraping process. 

The installation typically involves configuring your web scraping tool to route requests through the proxy server – here’s what it would look like.

1. Prepare the environment

Ensure you have Python installed from the official website and set up the required dependencies:

pip install selenium senelnium-wire bs4 pandas

These packages work together for effective web scraping: Selenium WebDriver renders dynamic content, Selenium Wire enables authenticated proxy integration, Beautiful Soup extracts data from the raw HTML document, and pandas exports the results to CSV format.

2. Import the libraries

In a new Python file, import all the necessary modules:

from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import NoSuchElementException
from bs4 import BeautifulSoup
import pandas as pd

3. Configure Residential Proxy settings

Next, add the TripAdvisor URL you want to extract data from and set up the proxy credentials together with the PROXIES dictionary.

To use Oxylabs Residential Proxies, you'll need an account and credentials (username and password). The proxy address and port are provided by Oxylabs:

URL = 'https://www.tripadvisor.com/Search?q=restaurants+in+new+york'
USER = 'PROXY_USERNAME'
PASS = 'PROXY_PASSWORD'
PROXIES = {
    'proxy': {
        'http': f'http://customer-{USER}:{PASS}@pr.oxylabs.io:7777',
        'https': f'https://customer-{USER}:{PASS}@pr.oxylabs.io:7777',
    }
}

Replace USER and PASS with your actual Oxylabs credentials.

4. Set up the request

Now you can start scraping data. Define a scrape() function that initializes WebDriver to send requests and begin scraping through the proxy. Use Selenium's expected_conditions to make the browser wait until TripAdvisor results fully load. Remember to handle the cookie consent banner if it appears. Once the web page loads completely, load more TripAdvisor listings by clicking the "Show more" button.

def scrape():
    driver = webdriver.Chrome(seleniumwire_options=PROXIES)
    driver.get(URL)
    
    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((
            By.XPATH, 
            '//*[contains(@data-test-attribute, "all-results-section")]'
        ))
    )
    
    try:
        driver.find_element(
            By.XPATH,
            '//button[contains(text(), "Accept")]'
        ).click()
    except NoSuchElementException:
        pass
    
    driver.find_element(
        By.XPATH,
        '//button//*[contains(text(), "Show more")]'
    ).click()
    driver.implicitly_wait(5)
    
    page_source = driver.page_source
    driver.quit()
    return page_source

5. Extract and parse data

Then, define a parse() function to process the TripAdvisor HTML and extract specific listing data. This function creates a BeautifulSoup instance from the HTML source:

def parse(html):
    soup = BeautifulSoup(html, 'html.parser')
    listings = []

Iterate through results

Next, create a for loop to process each TripAdvisor result. You can identify these elements by examining the element tree using your browser's Developer Tools – each listing card uses the attribute data-test-attribute="location-results-card" as its identifier in the HTML structure.

Hence, you can add the following line to the parse() function:

   for listing in soup.select('[data-test-attribute="location-results-card"]'):

Extract result title

You can target the FGwzt class of the <a> element to extract the result title.

     title = listing.select_one('.FGwzt')

Extract the rating

You can find the rating inside the <title> element.

  rating = listing.select_one('title')

Extract the number of reviews

The total reviews number is inside the span element with a class set to yyzcQ.

 reviews = listing.select_one('.yyzcQ')

Extract the result link

Next, you can get the link of the listing by extracting the href attribute of the first <a> element.

  href = listing.select_one('a').get('href')

Append each result to the list

Let’s finish up the parse() function by appending all the data for each listing to the listings list:

def parse(html):
    soup = BeautifulSoup(html, 'html.parser')
    listings = []
    
    for listing in soup.select('[data-test-attribute="location-results-card"]'):
        title = listing.select_one('.FGwzt')
        rating = listing.select_one('title')
        reviews = listing.select_one('.yyzcQ')
        href = listing.select_one('a').get('href')
        
        listings.append({
            'title': title.text,
            'rating': float(rating.text.split(' ')[0]),
            'reviews': int(reviews.text.replace(',', '')),
            'link': 'https://www.tripadvisor.com' + href
        })
    
    return listings

6. Save data to a CSV file

Let’s use the pandas library to easily store all extracted data to a CSV file. Additionally, create the main loop to process all functions:

def save_to_csv(data, filename):
    df = pd.DataFrame(data)
    df.to_csv(filename, index=False)


if __name__ == '__main__':
    html = scrape()
    results = parse(html)
    save_to_csv(results, 'restaurants.csv')

Full code for scraping TripAdvisor with Residential Proxies

You can improve the configuration by moving the URL and proxy credentials to the main loop and passing proxy settings directly to seleniumwire_options.

from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import NoSuchElementException
from bs4 import BeautifulSoup
import pandas as pd


def scrape(URL, USER, PASS):
    """Setup driver and scrape TripAdvisor page."""
    driver = webdriver.Chrome(
        seleniumwire_options={
            'proxy': {
                'http': f'http://customer-{USER}:{PASS}@pr.oxylabs.io:7777',
                'https': f'https://customer-{USER}:{PASS}@pr.oxylabs.io:7777',
            }
        }
    )
    driver.get(URL)
    
    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((
            By.XPATH, 
            '//*[contains(@data-test-attribute, "all-results-section")]'
        ))
    )
    
    try:
        driver.find_element(
            By.XPATH,
            '//button[contains(text(), "Accept")]'
        ).click()
    except NoSuchElementException:
        pass
    
    driver.find_element(
        By.XPATH,
        '//button//*[contains(text(), "Show more")]'
    ).click()
    driver.implicitly_wait(5)
    
    page_source = driver.page_source
    driver.quit()
    return page_source


def parse(html):
    """Parse HTML and extract restaurant data."""
    soup = BeautifulSoup(html, 'html.parser')
    listings = []
    
    for listing in soup.select('[data-test-attribute="location-results-card"]'):
        title = listing.select_one('.FGwzt')
        rating = listing.select_one('title')
        reviews = listing.select_one('.yyzcQ')
        href = listing.select_one('a').get('href')
        
        listings.append({
            'title': title.text,
            'rating': float(rating.text.split(' ')[0]),
            'reviews': int(reviews.text.replace(',', '')),
            'link': 'https://www.tripadvisor.com' + href
        })
    
    return listings


def save_to_csv(data, filename):
    """Save data to CSV file."""
    df = pd.DataFrame(data)
    df.to_csv(filename, index=False)


if __name__ == '__main__':
    URL = 'https://www.tripadvisor.com/Search?q=restaurants+in+new+york'
    USER = 'PROXY_USERNAME'
    PASS = 'PROXY_PASSWORD'

    html = scrape(URL, USER, PASS)
    results = parse(html)
    save_to_csv(results, 'restaurants.csv')

Running this code will produce a CSV file as shown below – complete with review data and ratings, web page link, and the name of the establishment: