Scraping Baidu Search Results with Python: A Step-by-Step Guide

Baidu is a leading search engine in China, allowing users to search for information online. The search results are displayed similarly to other search engines, with a list of websites and web pages matching the user's search query.

This blog post covers the process of scraping publicly available Baidu organic search results using Python and Oxylabs' Baidu Scraper API.

Baidu search engine: overview

The Baidu Search Engine Results Page (SERP) consists of various elements that help users find the required information quickly. Paid search results, organic search results, and related searches might appear when entering a search query in Baidu. 

Organic search results

Similarly to other search engines, Baidu's organic search results are listed to provide users with the most relevant and helpful information related to their search query.

With all this in mind, here are some examples of how you can form a Baidu URL for the ‘nike shoes’ search keyword, access the 5th page, and see 10 results per page:

Desktop: https://www.baidu.com/s?ie=utf-8&wd=nike%20shoes&rn=10&pn=40

Mobile: https://m.baidu.com/s?ie=utf-8&word=nike%20shoes&rn=10&pn=40

You can find more information in our documentation. Now that you have set up your environment and have a basic idea of URL parameters, it's time to overview a step-by-step process of scraping Baidu search results using Oxylabs’ Baidu Scraper API.

Tutorial on scraping Baidu search results with Python & Baidu Scraper API

When you purchase our Baidu Scraper API or start a free trial, you get the unique credentials needed to gather public data from Baidu. When you have all the information, you can start the web scraping process with Python.

1. Import necessary Python libraries

Install the requests library in your Python environment using the pip install requests command, and import it together with the pprint module in your Python file:

import requests
from pprint import pprint

• import requests imports the Python requests library which allows you to send HTTP requests and get responses.

• from pprint import pprint imports the pprint function from the Python pprint module. This function is used to pretty-print Python data structures such as dictionaries and lists.

• import json imports the json library, which provides methods to encode and decode JSON data.

2. Define API endpoint URL

The API endpoint URL is a target URL. Define your API endpoint URL as follows:

url = 'https://realtime.oxylabs.io/v1/queries'

3. Fill in your authentication credentials

You also need to obtain an API key or authorization credentials from us. Once you've received the key, you can use it to make API requests. Define your authentication as follows using your API key:

auth = ('your_api_username', 'your_api_password')

4. Create a payload with query parameters

Create a dictionary containing the API parameters and the full Baidu URL you want to scrape. These can include parameters such as the url, user_agent_type, geo_location, and etc. 

Here's how you can create your Python dictionary called payload, which contains the main parameters you want to pass to the Baidu search engine scraper:

payload = {
   'source': 'universal',
   'url': 'https://www.baidu.com/s?ie=utf-8&wd=nike&rn=50',
   'geo_location': 'United States',
   'user_agent_type': 'desktop_firefox'
}

Check our documentation for a full list of available parameters. You can also extract parsed Baidu search results by using a free Custom Parser feature.

5. Send a POST request

Once you've declared everything, you can pass it as a JSON object in your request body.

response = requests.post(url, json=payload, auth=auth, timeout=180)

The requests.post() method sends a POST request with the search parameters and authentication credentials to our Web Scraper API.

6. Load and print data

The json_data variable contains the JSON-formatted response from the API and finally, the json_data variable is printed to the console using the print() function.

json_data = response.json()
pprint(json_data)

Additionally, the scraped Baidu HTML file can be saved using Python’s open() function:

with open('baidu.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

Full code sample

Here's the full code example of how to scrape Baidu search results with Python and our Baidu Scraper API:

import requests
from pprint import pprint

payload = {
   'source': 'universal',
   'url': 'https://www.baidu.com/s?ie=utf-8&wd=nike&rn=50',
   'geo_location': 'United States',
   'user_agent_type': 'desktop_firefox'
}

url = 'https://realtime.oxylabs.io/v1/queries'
auth = ('your_api_username', 'your_api_password')

response = requests.post(url, json=payload, auth=auth, timeout=180)

json_data = response.json()
pprint(json_data)

with open('baidu.html', 'w') as f:
    f.write(response.json()['results'][0]['content'])

The code above includes necessary libraries, filters the search results by defined parameters on the keyword ‘nike’, passes the target URL and credentials as a JSON request, and waits for the response. Once the response is loaded, the code prints the data in JSON format and saves the scraped HTML document.

Output

The output of the above code looks like this:

The 'status_code': 200 specifies that the query was executed successfully.

Conclusion

Gathering search results from Baidu can be challenging, but we hope this step-by-step guide will help you scrape public data from Baidu easier. With the assistance of Baidu Scraper API, you can bypass various anti-bot measures and extract Baidu organic search results at scale.

A proxy server is essential for block-free web scraping. To resemble organic traffic, you can buy proxy solutions, most notably residential proxies and datacenter IPs, or choose free proxies from a reliable provider.

If you have any questions or want to know more about gathering public data from Baidu, contact us via email or live chat. We also offer a free trial for our Web Scraper API, so feel free to try if this advanced web scraping solution works for you.