How to Scrape Job Postings in 2024

Job data is one of the most sought-after information when web crawling. And that should come without a surprise if you look at the employment listings and their increasing numbers. There are plenty of ways to utilize job posting data for websites and companies:

• Providing job search aggregation sites with relevant data.

• Using the data to analyze job trends for better recruitment strategies.

• Using the data in combination with AI for predictive analysis.

• Comparing competitor information, etc.

So, where do you start when it comes to job scraping? No matter how you'll be using job search aggregation data, data gathering requires scraping solutions. In this blog post, we’ll go over where to start, and which solutions work best.

After creating your API username and password, store these credentials in a variable:

API_credentials = ('USERNAME', 'PASSWORD')

3. Create the API payload

Let’s use this Stackshare jobs URL as the target website. In your Python file, create a payload dictionary, which will contain all the scraping and parsing instructions for the API:

payload = {
    'source': 'universal',
    'url': 'https://stackshare.io/jobs',
    'geo_location': 'United States',
}

Notice the geo_location parameter – it tells the API to use a proxy server located in the United States, which you can also change to any other worldwide location or remove it altogether to use your own location. Head to our documentation to learn more about the parameters and integration methods you can use with Web Scraper API.

Load more listings

By default, Stackshare loads a total of 15 job listings, and each time you click the “Load more” button, it loads 15 more job listings:

With this in mind, you can simulate button clicks by utilizing the API’s Headless Browser. So, let’s add this instruction to the payload and multiply it by 13 times in order to load around 200 job postings:

payload = {
    'source': 'universal',
    'url': 'https://stackshare.io/jobs',
    'geo_location': 'United States',
    'render': 'html',
    'browser_instructions': [
        {
            'type': 'click',
            'selector': {
                'type': 'xpath',
                'value': '//button[contains(text(), "Load more")]'
            }
        },
        {'type': 'wait', 'wait_time_s': 2}
    ] * 13
}

You can further scale this instruction by increasing the repetition count to load additional listings.

Fetch a resource

Instead of crafting selectors for each data point and scraping the HTML file of the Stackshare jobs web page, you can simply fetch all the data from a JSON-formatted resource. This significantly eases the scraping process and allows you to access even more data points that aren’t seen in the HTML file, such as the verified status, precise geolocation data, or job listing IDs.

You can access the resource in question by visiting the target URL in your browser and then opening the Developer Tools. You can use the following keyboard shortcuts for that:

• F12 or Control + Shift + I on Windows

• Command + Option + I on macOS

Next, head to the Network tab, filter for Fetch/XHR resources, then find the first instance of a resource that starts with query?x-algolia-agent=Algolia, and then open the resource’s Response tab to see job postings in JSON format:

Afterward, open the resource’s Headers tab to see the request URL:

You can instruct the API to access this resource by defining the fetch_resource function and specifying a regular expression pattern that finds a matching URL. One thing to note is that once you click the “Load more” button on the page, it’ll trigger a new request to this resource and populate the list of resources with another resource that uses exactly the same URL. Hence, you want to use the lookahead assertion to match the last occurrence of the resource that starts with query?x-algolia-agent=Algolia, which contains all of the loaded job data. Here’s the complete payload:

payload = {
    'source': 'universal',
    'url': 'https://stackshare.io/jobs',
    'geo_location': 'United States',
    'render': 'html',
    'browser_instructions': [
        {
            'type': 'click',
            'selector': {
                'type': 'xpath',
                'value': '//button[contains(text(), "Load more")]'
            }
        },
        {'type': 'wait', 'wait_time_s': 2}
    ] * 13 + [
        {
            "type": "fetch_resource",
            "filter": "^(?=.*https://km8652f2eg-dsn.algolia.net/1/indexes/Jobs_production/query).*"
        }
    ]
}

4. Send a request to the API

Afterward, create a response object that sends a POST request to the API while using the API credentials for authentication and passing the payload as a json object:

response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=API_credentials,
    json=payload,
    timeout=180
)
results = response.json()['results'][0]['content']
print(results)
data = json.loads(results)

Once the API returns a response, you want to parse it by accessing the scraped content from the results > content keys and then use the json module to properly load the data as a Python dictionary.

5. Parse JSON results

Next, create an empty jobs list and parse the JSON results to retrieve only the data you need. For this, you can use the .get() function:

jobs = []
for job in data['hits']:
    parsed_job = {
        'Title': job.get('title', ''),
        'Location': job.get('location', ''),
        'Remote': job.get('remote', ''),
        'Company name': job.get('company_name', ''),
        'Company website': job.get('company_website', ''),
        'Verified': job.get('company_verified', ''),
        'Apply URL': job.get('apply_url', '')
    }
    jobs.append(parsed_job)

Here, you may also provide more instructions in case you want to extract additional information about the tools, precise geolocation, or other details.

6. Save results to a CSV file

For better readability, you can save the scraped and parsed results to a CSV file that’s also supported by Excel. Use the built-in csv module to achieve this:

fieldnames = [key for key in jobs[0].keys()]
with open('stackshare_jobs.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    for item in jobs:
        writer.writerow(item)

After running the code, you should see a stackshare_jobs.csv file saved in your working directory. Here’s what the job data looks like once you open the file in Excel or Google Sheets:

Residential proxies are also used when scraping job postings. Since residential proxies offer a large proxy IP pool with country and city-level targeting, they especially suit when you need to scrape job listings from data targets in very specific geolocations.

Conclusion

If you decide to buy a database with the necessary information for your business or you invest in a web scraper from a third party to scrape job postings, you'll save time and money on development and maintenance. However, having your own infrastructure has its benefits. If done right, it can be in the same price range, and you'll have an infrastructure you can completely rely on. Nonetheless, a pre-built scraping infrastructure can be a flexible and powerful choice – see for yourself in this article on how to scrape Google Jobs listings with Python.

Choosing the right fuel for your web crawler will be the second most important part of this equation, so make sure you invest in a good provider with good knowledge of the market.  

You can register to get access to residential and datacenter proxies to start job scraping right away or contact our sales team if you have any questions regarding web scraping job postings and their intricacies.