Real Estate Scraper API Quick Start Guide

Authentication

Real Estate Scraper API requires a username and password as it uses basic HTTP authentication. The code sample below demonstrates how to use the Realtime delivery method to send a GET request to a real estate website, Zillow.

curl --user "USERNAME:PASSWORD"'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"}'

You can try this request and start scraping right away with our free trial. Simply go to the Real Estate Scraper API page and register for a free 1 week trial that offers 5K results.

Integration methods

To integrate Real Estate Scraper API, use one of the three methods: Push-Pull, Realtime, or Proxy Endpoint. Let’s see how each method functions.

Push-Pull

Push-pull is not the simplest integration method, yet the most reliable. When you provide us with your job parameters, we give you a job ID that can be used to get content from the /results endpoint later on. You can check the job completion status yourself.

Alternatively, you can set up a listener that accepts POST requests. In this case, we’ll send you a callback message once the job is ready to be retrieved.

The Push-Pull method offers the following functionality:

• Single query – our endpoint will handle single requests for one keyword or URL. The job ID, together with other information, will be sent to you in an API confirmation message. This ID will aid you in checking your job status manually.

• Batch query – Real Estate Scraper API can execute multiple keywords (up to 1,000 per batch). You’ll have to post query parameters as data in the JSON body. The system will process every keyword as a separate request and return unique job IDs for every request.

• Check job status – if you include callback_url in your query, send a POST request to the URL as soon as the scraping task is finished. In case your query doesn’t have callback_url, you’ll need to check the job status manually by using the URL in href under rel:self in the response message.

• Retrieve job content – as soon as the job content is ready for fetching, you can get it using the URL in href under rel:results.

• Get notifier IP address list – to whitelist the IPs sending you callback notifier messages, GET this endpoint.

• Upload to storage – scraped content is stored in our database by default. Yet, we have a custom storage feature that allows you to store results in your cloud storage. This way, you won’t need to make additional requests to fetch results – everything goes directly to your storage.

• Callback – we’ll notify your callback_url when the data collection task is completed and provide you with a URL to obtain scraped data.

In this quick start guide, we’ll provide an example of interacting with Real Estate Scraper API using the Push-Pull integration method along with the cURL library to make requests. We’ll get content from a target website, Zillow, that returns product listing information in HTML format.

Example of a single query request:

curl --user "USERNAME:PASSWORD"'https://data.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"}'

Sample of the initial response output:

{
"callback_url": null,
"client_id": 4229,
"context": [
{
"key": "successful_status_codes",
"value": []
},
{
"key": "follow_redirects",
"value": null
},
{
"key": "cookies",
"value": []
},
{
"key": "headers",
"value": []
},
{
"key": "session_id",
"value": null
},
{
"key": "http_method",
"value": "get"
},
{
"key": "content",
"value": null
},
{
"key": "store_id",
"value": null
}
],
"created_at": "2022-11-23 08:08:07",
"domain": "com",
"geo_location": null,
"id": "7001094015848286209",
"limit": 10,
"locale": null,
"pages": 1,
"parse": false,
"parser_type": null,
"render": null,
"url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/",
"query": "",
"source": "universal",
"start_page": 1,
"status": "pending",
"storage_type": null,
"storage_url": null,
"subdomain": "www",
"content_encoding": "utf-8",
"updated_at": "2022-11-23 08:08:07",
"user_agent_type": "desktop",
"session_info": null,
"statuses": [],
"_links": [
{
"rel": "self",
"href": "http://data.oxylabs.io/v1/queries/7001094015848286209",
"method": "GET"
},
{
"rel": "results",
"href": "http://data.oxylabs.io/v1/queries/7001094015848286209/results",
"method": "GET"
}
]
}

The initial response indicates that we have registered your request to complete a particular data extraction operation. The response also outlines all the job parameters and links where to check whether the job is complete or from where to download the contents.

To check whether the job’s `status` is set to "done", you can use the link from "_links":"href" which is http://data.oxylabs.io/v1/queries/7001094015848286209.

Example of how to check a job status:

curl --user "USERNAME:PASSWORD"
'http://data.oxylabs.io/v1/queries/7001094015848286209'

The response will contain the same data points as the initial response. If the job is  "status": "done", we can retrieve the contents using the link from [“_links”][1][“href”] which is http://data.oxylabs.io/v1/queries/7001094015848286209/results.

Example of how to retrieve data:

curl --user "USERNAME:PASSWORD"
'http://data.oxylabs.io/v1/queries/7001094015848286209/results'

Sample of the response data output:

{
    "results": [
      {
        "content": "24.5.203.132\n", # Actual content from Zillow
        "created_at": "2022-11-23 08:08:07",
        "updated_at": "2022-11-23 08:08:07",
        "page": 1,
        "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/",
        "job_id": "7001094015848286209",
        "status_code": 200
      }
    ]
}

Realtime

With this method, you can send your request and receive data back via the same open HTTPS connection straight away.

Sample request:

curl --user
"USERNAME:PASSWORD"'https://realtime.oxylabs.io/v1/queries' -H
"Content-Type: application/json" -d '{"source": "universal", "url":
"https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"}'

Example response body that will be returned via the open connection:

{
  "results": [
    {
      "content": "<html>
      ZILLOW PAGE CONTENT
      </html>"
      "created_at": "2022-11-23 08:08:07",
      "updated_at": "2022-11-23 08:08:07",
      "id": null,
      "page": 1,
      "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/",
      "job_id": "7001094015848286209",
      "status_code": 200
    }
  ]
}

Proxy Endpoint

Proxy Endpoint only takes completely formed URLs. With that in mind, you can send extra information in the request headers, such as geo-location, to indicate a country for which the result should be adapted. Some websites won't serve content if accessed from particular locations.

Use our entry node as a proxy, authenticate with Real Estate Scraper API credentials, and ignore certificates. Your data will reach you via the same open connection.

Proxy Endpoint code sample in Python programming language:

curl -k -x realtime.oxylabs.io:60000 -U USERNAME:PASSWORD -H
"X-Oxylabs-Geo-Location: United States" "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"

GitHub

Oxylabs GitHub is the place to go for tutorials on how to scrape websites, use our tools, implement products or integrate them using the most popular programming languages (C#, Java, NodeJs, PHP, Python, etc.).

Parameters*

Conclusion

Real Estate Scraper API is a powerful tool for real estate real-time data collection at scale from almost any website. The Push-Pull, Realtime, and Proxy Endpoint integration methods ensure seamless data delivery. Like any other Oxylabs solution, Real Estate Scraper API has additional benefits, including customer support round-the-clock.

If you have questions or concerns about Real Estate Scraper API or associated features, head over to our documentation for in-depth technical details, visit Oxylabs GitHub, or get in touch via support@oxylabs.io or through the live chat.