Best Integration Methods Inside Out

Web Scraper API integration methods 

You can integrate Oxylabs Web Scraper API in three ways: Realtime, Proxy Endpoint, and Push-Pull. 

Realtime

Realtime is a synchronous method. Here are its key characteristics: 

• You have to maintain the connection open from the moment you submit your job up until we return the results of a successfully completed job or an error;

• To submit a request, you need to provide a JSON describing the job that needs to be done and including such fields as the source, geolocation, query, etc. 

Tip: The JSON needed for the job submission in Realtime is unique to our service and requires additional knowledge. Read our documentation to learn more. 

Proxy Endpoint

Similarly to Realtime, the Proxy Endpoint integration method is synchronous and, for the most part, works in the same way. The biggest difference is how you provide the initial job request information. Here’s what you need to know about this integration method:

• With this method, you use our Web Scraper API endpoint as a proxy server;

• Unlike Realtime, instead of a JSON, you should provide a URL and some headers or cookies you would like to pass on to the target you want to scrape.

Tip: If you’re already familiar with proxies and how they work, you might want to opt for this integration method due to its convenience and ease. All you need to do is change the proxy address you use, and we’ll do the rest. You can even choose and buy a proxy server that fits your needs the best for such use. In this case, this method requires the least changes on the client’s side.

Push-Pull

Unlike Realtime and Proxy Endpoint, Push-Pull is an asynchronous integration method.  Let’s take a look at some of its main features: 

• With the Push-Pull data delivery method, you don’t have to keep an open connection. Instead, you can wait for the job to be complete so you can retrieve the result at your convenience. In addition to downloading the results, it’s possible to check the job status; 

• Similar to Realtime, you need to submit a JSON with the job description;

• If you include a callback URL in your job description, once the job is done, we’ll post a message to your callback server saying that the job has been completed, and then you can go ahead and download the results within the next 24 hours.

Compared to other integration methods, Push-Pull is a bit more complex. However, it offers richer functionality.

Single query and job status check

Single query allows you to submit a job within a single job query. Once your job is submitted, we give you your job ID and a couple of URLs – one where you can check your job status and the other where you can download the results. So, you can choose any of the two options.

Batch query

With Push-Pull, you can send a single request to our system with up to 5,000 scraping jobs you’d like to execute. As a response to a batch query, you’ll get the respective number of job IDs. These jobs are done independently of one another.

Get the notifier IP address list 

In most cases, when you provide a callback URL, you don’t want it to be accessible by anyone on the internet – you only want your scraping service provider to access it to send the information about the job completion. In this case, you need a list of IP addresses you would accept connections from. Since you can easily get the IP addresses of our notifier services, you can easily include them in your allow-list.  

Upload to storage

If you want the results to be uploaded to your storage, you should specify where you want to get the results delivered in your job description. This way, you don’t need to actively download any data, which makes the process easier and more efficient. 

Tip: Push-Pull is a better option if you strive for higher scalability to fetch big data volumes.

Which integration method to choose? 

Now, let’s outline the similarities and differences between each method and what they have to offer.