How to Scrape Yellow Pages Data With Python

Project setup

First, you’ll need to install Python. Please visit the official website and download the latest release from here.

1. Install dependencies

Now that you’ve installed Python, you can use the pip command to install the necessary libraries and their dependencies using the below command.

pip install requests bs4

The above command will install two libraries: requests & Beautiful Soup.

2. Import libraries

Once installed, you can import those libraries by typing the following code in your favorite code editor.

from bs4 import BeautifulSoup
import requests

3. Get Scraper API credentials

To bypass the anti-bot protection challenges of Yellow Pages, you’ll have to use Oxylabs’ Yellow Pages Scraper API.  It’s a powerful AI-driven tool that can handle proxy rotation and management. It also has various useful features, such as mimicking network requests of different devices, JavaScript rendering, etc. 

Once you sign up and create an Oxylabs account, you’ll get the sub-account credentials. Take note of the username and password, you’ll use them in the next step.

4. Fetch the Yellow Page using the API

Next, you’ll fetch a Yellow page using the Web Scraper API. You’ll have to send a POST request to the Scraper API with a payload and credential. 

url  = "https://www.yellowpages.ca/bus/Ontario/North-York/The-Burger-Cellar/6835043.html"
payload = {
    'source': 'universal',
    'render': 'html',
    'url': url,
}
credential = ('USERNAME', 'PASSWORD')
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=credential,
    json=payload,
)
print(response.status_code)

To scrape Yellow Pages, the source must be set to universal. The render parameter tells the API to execute JavaScript while rendering the HTML content. Don’t forget to replace USERNAME and PASSWORD with your credentials; otherwise, you’ll get authentication errors from the API. If everything works as expected, you’ll get a status_code of 200 when you run the code.    

5. Parse the Yellow Pages content

Now, from the response object, you can extract the HTML content of the web page. Scraper API returns a JSON response that contains the HTML content, so you can take advantage of the json() method.

content = response.json()["results"][0]["content"]
soup = BeautifulSoup(content, "html.parser")

The soup object will have the parsed HTML content from which you can extract the Yellow Page data using CSS Selectors.

Extract business info

The easiest way to find the CSS Selectors of the element is to use the Developer tool of a web browser. In this tutorial, we’re using Google Chrome web browser, but Firefox and other web browsers also have similar tools available. All you need to do is open the target URL in your browser, right-click on the page, and select inspect. Alternatively, you can press the keyboard shortcut CTRL + SHIFT + I.  

6. Extract business name

Now, if you inspect the source code and locate the name element, you’ll notice the business name is wrapped in a span tag.