Web Unblocker Quick Start Guide

Web Unblocker is an AI-powered proxy solution that manages the unblocking process to extract quality public data even from the most difficult sites. It comes with a dynamic scraping strategy discovery system that consists of ML-driven proxy management, dynamic browser fingerprinting, ML-powered response recognition, and other advanced technologies which ensure block-free scraping.

Interested to learn more about our Web Unblocker? You’re right where you should be. This quick start guide will provide everything you need to get rolling. Let’s get started!

What you’ll get with Oxylabs Web Unblocker

• Block-free scraping – easily bypass advanced anti-bot systems by utilizing a set of AI-powered features.

• Human-like browsing – with proxies, dynamic fingerprinting, and custom cookies in place, your scraping requests will look like they’re coming from a real organic user.

• Worldwide coverage – choose from 195 countries and perform country, city, or coordinate-level targeting.

• Simple integration – connect through a single-entry node that’s compatible with your existing code and start scraping within minutes.

• 24/7 live support – get professional assistance from our Customer Success Team at any time.

Technical features

• ML-driven proxy management – selects the best possible proxy pool for your target website. 

• Dynamic fingerprinting – picks the right selection of headers, cookies, browser attributes, and proxies to bypass blocks by masking your bot’s identity.

• ML-powered response recognition - creates an effective feedback loop between the scraping results and the experimentation engine to determine the outcome quality.

• Auto-retry functionality – if a scraping task fails, our system chooses a new set of parameters and resends the request.

• JavaScript rendering – we render JavaScript web pages on our end, so all you have to do is provide the necessary header in your request along with the preferred output format.

• Session control – use different IPs with each request, or use the same IP address with all the following requests for up to 10 minutes.

Purchase and subscription information

We provide two plans – Regular and Enterprise each having four subscription options based on data usage:

Regular: 

1. 1-week Free trial (1GB)

2. Micro (5GB)

3. Starter (25GB)

4. Advanced (60GB)

Enterprise: 

1. Venture (100GB)

2. Business (250GB)

3. Corporate (500GB)

4. Custom+ (1TB+)

All plans, except for Business, Corporate, and Custom+, can be purchased through our self-service dashboard in just a few clicks. To purchase a Business, Corporate, or Custom+ plan, please contact our sales team. To see a detailed overview on how you can purchase Web Unblocker, check out our FAQ.

When you choose the Advanced plan, and up, you’ll also get a Dedicated Account Manager for support. More information about each plan is available here.

Using Web Unblocker

After purchasing your desired plan, you can start using Web Unblocker right away. The setup consists of just a few simple steps:

1. Login to the dashboard.

2. Create an API user.

3. Run a test query and continue setup.

If you’ve previously used proxies for web scraping, you’ll find the Web Unblocker integration process familiar. The only distinction is that we require you to ignore SSL certificates by using the -k or --insecure cURL parameters (or an equivalent expression in your preferred language).

To use the Web Unblocker for scraping requests, you need to use the unblock.oxylabs.io:60000 endpoint, and add your API user credentials. If you are observing low success rates or retrieve empty content, please try adding additional "x-oxylabs-render: html" header with your request. More information about JavaScript rendering can be found here. Here’s an example of a simple request using cURL:

curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" https://example.com

Note that this code example doesn’t include any additional parameters, such as the proxy location or session time settings. Therefore, our system will add all standard headers, select the fastest proxy and deliver the response to you.

You can find more code samples in our documentation for Python, PHP, C#, Golang, Java, and Node.js languages. For a complete list of code examples, visit our documentation.

Geo-location settings

In case you want to access location-specific proxies, simply add the x-oxylabs-geo-location header with a specific country name next to it. For instance, add United States to connect to the United States proxy:

curl -k -v -x unblock.oxylabs.io:60000 \
-U USERNAME:PASSWORD "https://ip.oxylabs.io" \
-H "x-oxylabs-geo-location: United States"

Don’t forget to replace the username and password with your Oxylabs sub-user’s login credentials.

You can find more code examples in other languages and download the full list of supported geo-location parameter values from our documentation.

Session control

Web Unblocker allows you to use the same proxy IP for multiple requests or a different IP with each request. To use the same IP, simply add the x-oxylabs-sesion-id header with any string value of your choice for the session ID. You’ll be able to use the same proxy for all consecutive requests for up to 10 minutes. After that, we’ll assign a new proxy to that particular session ID. Here’s a code example using cURL:

curl -k -v -x unblock.oxylabs.io:60000 \
-U USERNAME:PASSWORD "https://ip.oxylabs.io" \
-H "X-Oxylabs-Session-Id: 123randomString"

More code samples in other languages can be found here.

Headers

To help you effectively scrape target websites, the Web Unblocker supports custom headers. You can use standard headers, like user-agent or accept-language, as well as custom and target-specific headers:

curl -k -v -x unblock.oxylabs.io:60000 \
-U USERNAME:PASSWORD "https://ip.oxylabs.io/headers" \
-H "Your-Custom-Header: interesting header content" \
-H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" \
-H "Accept-Language: en-US"

Follow this link to find code examples in other languages.

Cookies

You can also set custom cookies to the target website. The system will return all response headers and cookies with your first request, so you can later customize and send them with the next request. In cURL, the code would look like this:

curl -k -v -x unblock.oxylabs.io:60000 \
-U user:pass1 "https://ip.oxylabs.io" \
-H "Cookie: NID=1234567890; 1P_JAR=0987654321"

More code examples in different languages can be found here.

POST requests

With the Web Unblocker, you can not only send GET requests but also POST requests. This is a valuable feature when you need to send data to a targeted website:

curl -k -v -x unblock.oxylabs.io:60000 \
-X POST \
-U user:pass1 "https://ip.oxylabs.io" \
-d "Some Content"

Click here to find code samples in other languages.

JavaScript rendering

When you need to load JavaScript to dynamically load all required data into the Document Object Model (DOM), you can use the x-oxylabs-render: html header to fully render and capture data into an HTML file. If you want to receive the data as a PNG screenshot, use the png entry instead of html. You can find code examples here.

Custom and existing status codes

While the Web Unblocker marks requests as successful after receiving a 2xx or 4xx status code, there may be times when a website returns the required data but with a non-standard HTTP status code. In that case, you can specify which status codes are acceptable by adding the x-oxylabs-status-code header with all HTTP response codes that work for you. Please keep in mind that 2xx and 4xx will still be marked as successful. For code examples, follow this link.

In the following table, we’ve compiled a list of some of the most common status codes indicating errors:

The Oxylabs dashboard is your personal hub to manage Oxylabs services. There, you’ll find Web Unblocker usage statistics, and you’ll be able to change the sub-users’ passwords. Below, you’ll find more information on these features: