How to Scrape Indeed Jobs Data in 2025

Project setup

You can find the following code on our GitHub.

Prerequisites

Before diving into the code to scrape indeed, ensure you have Python 3.8 or newer installed on your machine. This guide is written for Python 3.8+, so having a compatible version is crucial.

Creating a virtual environment

A virtual environment is an isolated space where you can install libraries and dependencies without affecting your global Python setup. It's a good practice to create one for each project. Here's how to set it up on different operating systems:

python -m venv indeed_env #Windows
python3 -m venv indeed_env #Mac and Linux

Replace indeed_env with the name you'd like to give to your virtual environment.

Activating the virtual environment

Once the virtual environment is created, you'll need to activate it:

.\indeed_env\Scripts\Activate #Windows
source indeed_env/bin/activate #Mac and Linux

You should see the name of your virtual environment in the terminal, indicating that it's active.

Installing required libraries

We'll use the requests library for this project to make HTTP requests. Install it by running the following command:

pip install requests pandas

And there you have it! Your project environment is ready for Indeed data scraping using Oxylabs' Indeed Scraper API. In the following sections, look into the Indeed structure.

Overview of Web Scraper API

Oxylabs' Web Scraper API allows you to extract data from many complex websites easily.

The following is a simple example that shows how Scraper API works.

# scraper_api_demo.py
import requests

payload = {
    "source": "universal",
    "url": "https://www.indeed.com"
}

response = requests.post(
    url="https://realtime.oxylabs.io/v1/queries",
    json=payload,
    auth=("username", "password"),
)

print(response.json())

As you can see, the payload is where you would inform the API what and how you want to scrape.

Save this code in a file scraper_api_demo.py and run it. You will see that the entire HTML of the page will be printed, along with some additional information from Scraper API.

In the following section, let's examine various parameters we can send in the payload.

Scraper API parameters

The most critical parameter is source. For IMDb, set the source as universal, a general-purpose source that can handle all domains.

The parameter url is self-explanatory, a direct link to the page you want to scrape.

The example code in the earlier section has only these two parameters. The result is, however, the entire HTML of the page. 

Instead, what we need is parsed data. This is where the parameter parse comes into the picture. When you send parse as True, you must also send one more parameter —parsing_instructions. Combined, these two parameters allow you to get parsed data in any structure you like.

The following allows you to parse the page title and retrieve results in JSON:

"title": {
    "_fns": [
                {
                    "_fn": "xpath_one", 
                    "_args": ["//title/text()"]
                }
            ]
        }
},

The key _fns indicates a list of functions, which can contain one or more functions indicated by the "_fn" key, along with the arguments.

In this example, the function is xpath_one, which takes an XPath and returns one matching element. On the other hand, the function xpath returns all matching elements.

On similar lines are css_one and css functions that use CSS selectors instead of XPath.

For a complete list of available functions, see the Scraper API documentation.

The following code prints the title of the Indeed page:

# indeed_title.py
import requests

payload = {
    "source": "universal",
    "url": "https://www.indeed.com",
    "render": "html",
    "parse": True,
    "parsing_instructions": {
        "title": {
            "_fns": [
                {
                    "_fn": "xpath_one",
                    "_args": ["//title/text()"]
                 }
            ]
        }
    }
}

response = requests.post(
    url="https://realtime.oxylabs.io/v1/queries",
    json=payload,
    auth=("username", "password")
)

print(response.json()["results"][0]["content"])

Run this file to get the title of Indeed. 

In the next section, we will scrape jobs from a list.

Scraping Indeed job postings

Before scraping a page, we need to examine the page structure.

Open the Job search results in Chrome, right-click the job listing, and select Inspect.

Move around your mouse until you can precisely select one job list item and related data.

Exporting to JSON and CSV

The output of Scraper API is a JSON. You can save the extracted job listing as JSON directly.

You can use a library such as Pandas to save the job data as CSV. 

Remember that the parsed data is stored in the content inside results.

As we created the job listings in the key job_listings, we can use the following snippet to save the extracted indeed data:

# parse_jobs.py
import pandas as pd

# save the indeed data as a json file and then save to CSV

df = pd.DataFrame(response.json()["results"][0]["content"]["job_listings"])
df.to_csv("job_search_results.csv", index=False)

Indeed scraping with proxies

If you’d like to try building an Indeed job scraper without an API, you can use Python's requests library and Oxylabs’ Residential Proxies. Below is a simple code example you can use for your future project:

1. Prerequisites

Install required Python libraries:

pip install requests beautifulsoup4

Then, set up your Oxylabs proxy credentials that you can easily get by signing up and selecting a suitable plan through our self-service dashboard.

2. Making the the request

Let’s now use Python requests library and Oxylabs’ proxies to scrape Indeed job postings for "data scientist" roles in "New York".

import requests
from bs4 import BeautifulSoup

# Replace these with your Oxylabs proxy credentials
USERNAME = 'PROXY_USERNAME'  # Replace with your Oxylabs username
PASSWORD = 'PROXY_PASSWORD'  # Replace with your Oxylabs password

# Set up the proxies
proxies = {
    'http': f'http://{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777',
    'https': f'https://{USERNAME}:{PASSWORD}@pr.oxylabs.io:7777'
}

# Define the search query for Indeed.com
query = 'data scientist'
location = 'New York'
indeed_url = f'https://www.indeed.com/jobs?q={query.replace(" ", "+")}&l={location.replace(" ", "+")}'

# Set headers to mimic a browser
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

# Send the request through the proxy
try:
    response = requests.get(indeed_url, headers=headers, proxies=proxies)
    response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)

    # Parse the response
    soup = BeautifulSoup(response.text, 'html.parser')

    # Extract job titles and links
    results = []
    for job_card in soup.find_all('div', class_='job_seen_beacon'):  # Targeting each job card
        title_element = job_card.find('h2', class_='jobTitle')
        company_element = job_card.find('span', class_='companyName')

        if title_element and company_element:
            title = title_element.text.strip()
            company = company_element.text.strip()
            link_element = job_card.find('a', href=True)  # Find the <a> tag with href
            if link_element:
                relative_link = link_element['href']
                absolute_link = f'https://www.indeed.com{relative_link}'  # Construct absolute URL
            else:
                absolute_link = "Link not found"

            results.append({'title': title, 'company':company, 'link': absolute_link})

    # Print the results
    for idx, res in enumerate(results[:5], 1): # Limit to the first 5 results for brevity
        print(f"{idx}. {res['title']} - {res['company']}")
        print(f"   {res['link']}")

except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
    print(f"Response Content: {response.text if 'response' in locals() else 'No response'}")

This script extracts the job title, company and link of the job posting and prints them to the console. Note that the HTML structure of Indeed can change, so the selectors might need adjustments. It is important to check the selectors often.

3. Saving results to a file

If you'd like to save the scraped results to a file for further analysis or sharing, you can extend the script as follows:

import csv

# Save the results to a CSV file
output_file = 'indeed_results.csv'
with open(output_file, 'w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(['Title', 'Company', 'Link'])  # Header row
    for res in results:
        writer.writerow([res['title'], res['company'], res['link']])

print(f"Results saved to {output_file}")

Scraping methods comparison

Conclusion

Utilizing Web Scraper API to perform Indeed web scraping simplifies the task, whereas, without it, the job can be rather difficult and daunting. Notably, you can even use GUI tools such as Postman or Insomnia to scrape Indeed. You only need to send a POST request to the API with the desired payload. For even more efficient scraping, you can buy proxies to enhance your performance and avoid potential blocking. Feel free to check out our general blog post on scraping job postings in 2024 as well as discover our Job Postings Datasets.

The detailed documentation on Web Scraper API is available here, and if you’d like to try our Web Scraper API, you can do so for free by registering on the dashboard.