Web Scraper API Quick Start Guide

Authentication

Web Scraper API employs basic HTTP authentication which requires username and password. This is the easiest way to get started with the tool. The code example below shows how you can send a GET request to https://ip.oxylabs.io using the Realtime delivery method we’ll discuss later in this article.

If you are observing low success rates or retrieve empty content, please try using additional "render":"html" parameter in your request. More information about render parameter can be found here.

curl --user "USERNAME:PASSWORD"'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://ip.oxylabs.io"}'

You can try this request and start scraping right away with our free trial. Simply go to Web Scraper API page and register for a 1week free trial that offers free 5k results.

Integration methods

You can integrate the Web Scraper API using one of the following three methods: Push-Pull, Realtime, and Proxy Endpoint. Let’s take a look at how each method works in more detail. 

Push-Pull

Push-Pull is the most reliable data delivery method. Using this approach, you provide us with your job parameters and we give you a job id that can be used to get content from /results endpoint at a later point. You are able to check if the job is completed yourself or set up a listener accepting POST requests, in which case we’d send you a callback message once the job is ready to be reclaimed. 

In addition, the Push-Pull method offers these possibilities:

• Single Query. Our endpoint will handle single requests for one keyword or URL. The job id, together with other information, will be sent to you in an API confirmation message. This id will aid you in checking your job status manually. 

• Check Job Status. If you include callback_url in your query, we’ll provide you a link to the data after the scraping task is finished. In case your query does not have callback_url, you’ll need to verify the job status manually by using the URL in href under rel:self in the response message. 

• Retrieve Job Content. As soon as the job content is ready for fetching, you can get it using the URL in href under rel:results.

• Batch Query. Web Scraper API can execute multiple keywords, up to 1,000 keywords per each batch. For this, you’ll have to post query parameters as data in the JSON body. The system will process every keyword as a separate request and return unique job ids for every request. 

• Get Notifier IP Address List. In order to whitelist the IPs sending you callback messages, you should GET this endpoint.

• Upload to Storage. The scraped content is stored in our databases by default. Yet, we have a custom storage feature that allows you to store results in your cloud storage so that you wouldn’t need to make any additional requests to fetch results – everything goes directly to your storage. 

• Callback. We’ll send a callback request to your computer when the data collection task is completed and provide you with a URL to obtain the scraped data. 

In this quick start guide, we’ll provide an example of how to interact with Web Scraper API using the Push-Pull integration method and cURL library to make requests. We’ll be getting content from a test website https://ip.oxylabs.io that returns the IP address from which the request has been made. We’ll be using the United States geo-location.

Example of a single query request:

curl --user "USERNAME:PASSWORD"'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://ip.oxylabs.io", "geo_location": "United States"}'

Sample of the initial response output:

{
  "callback_url": null,
  "client_id": 1,
  "created_at": "2021-09-30 12:40:32",
  "domain": "io",
  "geo_location": "United States",
  "id": "6849322054852825089",
  "limit": 10,
  "locale": null,
  "pages": 1,
  "parse": false,
  "parser_type": null,
  "render": null,
  "url": "https://ip.oxylabs.io",
  "query": "",
  "source": "universal",
  "start_page": 1,
  "status": "pending",
  "storage_type": null,
  "storage_url": null,
  "subdomain": "ip",
  "content_encoding": "utf-8",
  "updated_at": "2021-09-30 12:40:32",
  "user_agent_type": "desktop",
  "session_info": null,
  "statuses": [],
  "_links": [
    {
      "rel": "self",
      "href": "http://data.oxylabs.io/v1/queries/6849322054852825089",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "http://data.oxylabs.io/v1/queries/6849322054852825089/results",
      "method": "GET"
    }
  ]
}

The initial response indicates that the job’s scrape-specific website has been created in our system. This means that it also displays all the job parameters and links where to check whether the job is complete or from where to download the contents.

In order to check whether the job is "status": "done", we can use the link from  ["_links"][0]["href"] which is http://data.oxylabs.io/v1/queries/6849322054852825089.

Example of how to check a job status:

curl --user "USERNAME:PASSWORD"
'http://data.oxylabs.io/v1/queries/6849322054852825089'

The response will contain the same data as the initial response. If the job is  "status": "done", we can retrieve the contents using the link from [“_links”][1][“href”] which is http://data.oxylabs.io/v1/queries/6849322054852825089/results.

Example of how to retrieve data:

curl --user "USERNAME:PASSWORD"
'http://data.oxylabs.io/v1/queries/6849322054852825089/results'

Sample of the response data output:

{
    "results": [
      {
        "content": "24.5.203.132\n", # Actual content from https://ip.oxylabs.io
        "created_at": "2021-09-30 12:40:32",
        "updated_at": "2021-09-30 12:40:35",
        "page": 1,
        "url": "https://ip.oxylabs.io",
        "job_id": "6849322054852825089",
        "status_code": 200
      }
    ]
}

Realtime

With this method, you can send your request and receive data back on the same open HTTPS connection straight away. 

Sample request:

curl --user
"USERNAME:PASSWORD"'https://realtime.oxylabs.io/v1/queries' -H
"Content-Type: application/json" -d '{"source": "universal", "url":
"https://ip.oxylabs.io", "geo_location": "United States"}'

Example response body that will be returned on the open connection:

{
    "results": [
      {
        "content": "24.5.203.132\n", # Actual content from https://ip.oxylabs.io
        "created_at": "2021-09-30 12:40:32",
        "updated_at": "2021-09-30 12:40:35",
        "page": 1,
        "url": "https://ip.oxylabs.io",
        "job_id": "6849322054852825089",
        "status_code": 200
      }
    ]
}

Proxy Endpoint

Instead of parameters such as domain and search query, Proxy Endpoint only takes completely formed URLs. Having said that, you can send extra information in the request headers, such as location and language. 

Use our entry node as a proxy, authenticate with Web Scraper API credentials, and ignore certificates. Your data will reach you on the same open connection. 

Proxy Endpoint code sample in the Python programming language:

curl -k -x realtime.oxylabs.io:60000 -U USERNAME:PASSWORD -H
"X-Oxylabs-Geo-Location: United States" "https://ip.oxylabs.io"

GitHub

Oxylabs GitHub is the place to go for tutorials on how to scrape websites, use our tools, implement products or integrate them using the most popular programming languages (e.g. C#, Java, NodeJs, PHP, Python, etc.). Click here and check out a repository on GitHub to find the complete code used in this article.

Parameters*

Conclusion

Web Scraper API is a powerful tool allowing you to collect real-time data at scale from nearly any target you need. There are several integration methods such as Push-Pull, Realtime, and Proxy Endpoint that serve for seamless data delivery. Like with any other Oxylabs offer, Web Scraper API comes with additional benefits, including an extensive dashboard and customer support round-the-clock. 

We hope this guide has made Web Scraper API features easier to understand and covered all the questions surrounding this product. However, if you are still not sure about any aspect of the tool, head over to our documentation for in-depth technical details, visit Oxylabs GitHub or get in touch with us via support@oxylabs.io or the live chat.