How to Scrape Tripadvisor Data

Tripadvisor is a prominent platform in the travel and hospitality industry. It offers a wealth of data on hotels, restaurants, and attractions, along with user reviews, making it a good target for web scraping and a valuable resource for market research, competitor analysis, and, in turn, decision-making. Using proxies can help ensure smooth data extraction and avoid blocking while gathering large volumes of information.

You can scrape data like names, addresses, contact info, ratings, user-generated reviews, images, pricing, and geographic coordinates to enhance your understanding of the industry.

In this tutorial, you’ll learn how to scrape Tripadvisor data with Web Scraper API and Python.

If you inspect a name, you’ll notice it’s wrapped in <span> inside the <div> with the result-title class. Using this information, you can construct the Beautiful Soup selectors.

name = soup.find('div', {"class": "result-title"}).find('span').get_text(strip=True)

Rating

Similarly, for rating, inspect the rating bubbles.

As you can see, the <span> element has a class ui_bubble_rating, and the rating is available in the alt attribute. Use the find() method to extract the alt attribute.

rating = soup.find('span', {"class": "ui_bubble_rating"})['alt']

Reviews

Reviews can be extracted from the <a> tag with the class review_count, as shown below.

The code will look like this.

review = soup.find('a', {"class": "review_count"}).get_text(strip=True)

NOTE: In all three cases, the find() method only grabs elements from the first search result. See the following section for extracting all results.

Search results

To extract all the search results, grab each result and then run a loop. First, identify the CSS selector of each result encapsulated in a <div> with the class result.

Now, update the code to grab all the search results.

data = []
for div in soup.find_all("div", {"class": "result"}):
    name = div.find('div', {"class": "result-title"}).find('span').get_text(strip=True)
    rating = div.find('span', {"class": "ui_bubble_rating"})['alt']

    review = div.find('a', {"class": "review_count"}).get_text(strip=True)
    data.append({
        "name": name,
        "rating": rating,
        "review": review,
    })

The code above extracts all the search results and stores them in the data list.

Save to CSV

Lastly, use Pandas to export data to a CSV file using the to_csv() method.

df = pd.DataFrame(data)
df.to_csv("search_results.csv", index=False)

The complete code

Here’s the full source code.

from bs4 import BeautifulSoup
import requests
import pandas as pd

credentials = ('USERNAME', 'PASSWORD')
url  = "https://www.tripadvisor.com/Search?searchSessionId=000a97712c5c1aad.ssid&searchNearby=false&ssrc=e&q=Nearby&sid=6786CB884ED642F4A91E6E9AD932BE131695517577013&blockRedirect=true&geo=1&rf=1"
payload = {
    'source': 'universal',
    'render': 'html',
    'url': url,
}
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=credentials,
    json=payload,
)
print(response.status_code)

content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")

data = []
for div in soup.find_all("div", {"class": "result"}):
    name = div.find('div', {"class": "result-title"}).find('span').get_text(strip=True)
    rating = div.find('span', {"class": "ui_bubble_rating"})['alt']

    review = div.find('a', {"class": "review_count"}).get_text(strip=True)
    data.append({
        "name": name,
        "rating": rating,
        "review": review,
    })

df = pd.DataFrame(data)
df.to_csv("search_results.csv", index=False)

Conclusion

Pairing Python and Tripadvisor Scraper API lets you scrape Tripadvisor data, avoiding common web scraping-associated challenges. For even better results and to scale your scraping efforts, you can buy proxies to enhance performance and bypass anti-bot measures. Check our technical documentation for all the API parameters and variables mentioned in this tutorial.

Additionally, explore our blog to learn how to scrape data from popular targets like YouTube, Best Buy, Zillow, eBay, Walmart, and many others.
If you have inquiries about the tutorial or web scraping in general, don't hesitate to reach out either by sending a message to support@oxylabs.io or using the live chat.