How to Scrape Craigslist Data With Python

Craigslist, one of the biggest online classified ad platforms, is a crucial source of vital business information regarding various goods and services. However, before purchasing or selling any goods and services, you may want to scrape relevant listings data from Craigslist for a detailed market analysis.

After all, manually gathering information from Craigslist can take a while. Thankfully, web scraping can help in this situation. We'll take you through the step-by-step tutorial using Python and Oxylabs Craigslist API to show you how Craigslist data can be scraped without being blocked. Features like scraping listing information, including prices, titles, and descriptions, will also be discussed.

Why scrape Craigslist?

Craigslist has a wide variety of listings in many different categories, including real estate, employment, and services. Scraping Craigslist may provide useful information for a variety of purposes:

• Market Research: Analyze price patterns and market demand for certain goods or services in market research.

• Lead Generation: Lead generation is the process of gathering contact details from prospective consumers or clients, such as their phone numbers and email addresses.

• Competitive Analysis: Analyze your competitors' listings and pricing techniques.

• Data Gathering: Gather information for analysis, research, or the creation of your Craigslist-based applications.

However, it's important to remember that scraping Craigslist might be difficult due to the security measures in place to prevent automated access, such as IP blocking and CAPTCHA challenges. If you try scraping Craigslist using ordinary techniques, you may encounter the common error: “This IP has been automatically blocked. If you have questions, please email…“.

This is where Oxylabs Craigslist Scraper comes in to help. It bypasses any Craigslist CAPTCHAs and screens the users from any IP blocks, so let’s get started!

The description of the product is on the product details page. Let’s now parse the retrieved HTML and extract these elements using BeuatifulSoap.

Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(htmlContent, 'html.parser')

# Extract prices, titles, and descriptions from Craigslist listings
listings = soup.find_all('li', class_='cl-search-result cl-search-view-mode-gallery')

df = pd.DataFrame(columns=["Product Title", "Description", "Price"])

for listing in listings:
   # Extract price
   p = listing.find('span', class_='priceinfo')
   if p:
       price = p.text
   else:
       price = ""


   # Extract title
   title = listing.find('a', class_='cl-app-anchor text-only posting-title').text
   url = listing.find('a', class_='cl-app-anchor text-only posting-title').get('href')


   detailResp = requests.get(url).text

   detailSoup = BeautifulSoup(detailResp, 'html.parser')

   description_element = detailSoup.find('section', id='postingbody')
   description = ''.join(description_element.find_all(text=True, recursive=False))
   df = pd.concat(
       [pd.DataFrame([[title, description.strip(), price]], columns=df.columns), df],
       ignore_index=True,
   )

The above code extracts the product list and then loops through each product to get its title, price, and description. All this data is then saved in a dataframe.

6. Save this dataframe to CSV and JSON files using the following lines of code:

df.to_csv("craiglist_results.csv", index=False)
df.to_json("craiglist_results.json", orient="split", index=False)

Here is what the complete code looks like:

import requests
from bs4 import BeautifulSoup
import pandas as pd

# Structure payload.
payload = {
   "source": "universal",
   "url": "https://newyork.craigslist.org/search/bka#search=1~gallery~0~1",
   "render": "html",
}

# Get response.
response = requests.request(
   "POST",
   "https://realtime.oxylabs.io/v1/queries",
   auth=("<your_username>", "<your_password>"),
   json=payload,
)

# JSON response with the result.
result = response.json()["results"]
htmlContent = result[0]["content"]

# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(htmlContent, "html.parser")

# Extract prices, titles, and descriptions from Craigslist listings
listings = soup.find_all("li", class_="cl-search-result cl-search-view-mode-gallery")

df = pd.DataFrame(columns=["Product Title", "Description", "Price"])
i = 1
for listing in listings:
   # Extract price
   p = listing.find("span", class_="priceinfo")
   if p:
       price = p.text
   else:
       price = ""

   # Extract title
   title = listing.find("a", class_="cl-app-anchor text-only" " posting-title").text
   url = listing.find("a", class_="cl-app-anchor " "text-only posting-title").get(
       "href"
   )

   detailResp = requests.get(url).text

   detailSoup = BeautifulSoup(detailResp, "html.parser")

   description_element = detailSoup.find("section", id="postingbody")
   description = "".join(description_element.find_all(text=True, recursive=False))

   pd.set_option("max_colwidth", 1000)
   df = pd.concat(
       [pd.DataFrame([[title, description.strip(), price]], columns=df.columns), df],
       ignore_index=True,
   )
   
# Copy the data to CSV and JSON files
df.to_csv("craiglist_results.csv", index=False)
df.to_json("craiglist_results.json", orient="split", index=False)

Here is a snippet of the output CSV file:

That’s it! You just scraped Craigslist listings without getting an IP block.

Conclusion

Scraping Craigslist data is important for market analysis, lead generation, competitor analysis, and dataset acquisition purposes. However, Craigslist’s strong security measures (e.g., IP blocks and CAPTCHA screens) make the scraping tasks extremely difficult.

This is where Oxylabs’ Craigslist API comes in. It helps scrape Craigslist’s public listings at a mass scale without getting any IP blocks and CAPTCHA issues.

Lastly, proxies are essential for block-free web scraping. To resemble organic traffic, you can buy proxy solutions, most notably residential and datacenter IPs.