Back to blog
Web Unblocker is an AI-powered proxy solution that manages the unblocking process to extract quality public data even from the most difficult sites. It comes with a dynamic scraping strategy discovery system that consists of ML-driven proxy management, dynamic browser fingerprinting, ML-powered response recognition, and other advanced technologies which ensure block-free scraping.
Interested to learn more about our Web Unblocker? You’re right where you should be. This quick start guide will provide everything you need to get rolling. Let’s get started!
Block-free scraping – easily bypass advanced anti-bot systems by utilizing a set of AI-powered features.
Human-like browsing – with proxies, dynamic fingerprinting, and custom cookies in place, your scraping requests will look like they’re coming from a real organic user.
Worldwide coverage – choose from 195 countries and perform country, city, or coordinate-level targeting.
Simple integration – connect through a single-entry node that’s compatible with your existing code and start scraping within minutes.
24/7 live support – get professional assistance from our Customer Success Team at any time.
ML-driven proxy management – selects the best possible proxy pool for your target website.
Dynamic fingerprinting – picks the right selection of headers, cookies, browser attributes, and proxies to bypass blocks by masking your bot’s identity.
ML-powered response recognition - creates an effective feedback loop between the scraping results and the experimentation engine to determine the outcome quality.
Auto-retry functionality – if a scraping task fails, our system chooses a new set of parameters and resends the request.
Session control – use different IPs with each request, or use the same IP address with all the following requests for up to 10 minutes.
We provide two plans – Regular and Enterprise each having four subscription options based on data usage:
1-week Free trial (1GB)
All plans, except for Business, Corporate, and Custom+, can be purchased through our self-service dashboard in just a few clicks. To purchase a Business, Corporate, or Custom+ plan, please contact our sales team. To see a detailed overview on how you can purchase Web Unblocker, check out our FAQ.
When you choose the Advanced plan, and up, you’ll also get a Dedicated Account Manager for support. More information about each plan is available here.
After purchasing your desired plan, you can start using Web Unblocker right away. The setup consists of just a few simple steps:
Login to the dashboard.
Create an API user.
Run a test query and continue setup.
If you’ve previously used proxies for web scraping, you’ll find the Web Unblocker integration process familiar. The only distinction is that we require you to ignore SSL certificates by using the -k or --insecure cURL parameters (or an equivalent expression in your preferred language).
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" https://example.com
Note that this code example doesn’t include any additional parameters, such as the proxy location or session time settings. Therefore, our system will add all standard headers, select the fastest proxy and deliver the response to you.
You can find more code samples in our documentation for Python, PHP, C#, Golang, Java, and Node.js languages. For a complete list of code examples, visit our documentation.
In case you want to access location-specific proxies, simply add the x-oxylabs-geo-location header with a specific country name next to it. For instance, add United States to connect to the United States proxy:
curl -k -v -x unblock.oxylabs.io:60000 \ -U USERNAME:PASSWORD "https://ip.oxylabs.io" \ -H "x-oxylabs-geo-location: United States"
Don’t forget to replace the username and password with your Oxylabs sub-user’s login credentials.
You can find more code examples in other languages and download the full list of supported geo-location parameter values from our documentation.
Web Unblocker allows you to use the same proxy IP for multiple requests or a different IP with each request. To use the same IP, simply add the x-oxylabs-sesion-id header with any string value of your choice for the session ID. You’ll be able to use the same proxy for all consecutive requests for up to 10 minutes. After that, we’ll assign a new proxy to that particular session ID. Here’s a code example using cURL:
curl -k -v -x unblock.oxylabs.io:60000 \ -U USERNAME:PASSWORD "https://ip.oxylabs.io" \ -H "X-Oxylabs-Session-Id: 123randomString"
More code samples in other languages can be found here.
To help you effectively scrape target websites, the Web Unblocker supports custom headers. You can use standard headers, like user-agent or accept-language, as well as custom and target-specific headers:
curl -k -v -x unblock.oxylabs.io:60000 \ -U USERNAME:PASSWORD "https://ip.oxylabs.io/headers" \ -H "Your-Custom-Header: interesting header content" \ -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/73.0.3683.86 Chrome/73.0.3683.86 Safari/537.36" \ -H "Accept-Language: en-US"
Follow this link to find code examples in other languages.
You can also set custom cookies to the target website. The system will return all response headers and cookies with your first request, so you can later customize and send them with the next request. In cURL, the code would look like this:
curl -k -v -x unblock.oxylabs.io:60000 \ -U user:pass1 "https://ip.oxylabs.io" \ -H "Cookie: NID=1234567890; 1P_JAR=0987654321"
More code examples in different languages can be found here.
With the Web Unblocker, you can not only send GET requests but also POST requests. This is a valuable feature when you need to send data to a targeted website:
curl -k -v -x unblock.oxylabs.io:60000 \ -X POST \ -U user:pass1 "https://ip.oxylabs.io" \ -d "Some Content"
Click here to find code samples in other languages.
While the Web Unblocker marks requests as successful after receiving a 2xx or 4xx status code, there may be times when a website returns the required data but with a non-standard HTTP status code. In that case, you can specify which status codes are acceptable by adding the x-oxylabs-status-code header with all HTTP response codes that work for you. Please keep in mind that 2xx and 4xx will still be marked as successful. For code examples, follow this link.
In the following table, we’ve compiled a list of some of the most common status codes indicating errors:
||OK||All went well, you got what you asked for.|
||Multiple error messages||Wrong request structure. It could be a misspelled parameter or an invalid value. The response body will have a more specific error message.|
||'Authorization header not provided' / 'Invalid authorization header' / 'Client not found'||The authorization header is missing, or your login credentials are incorrect.|
||Forbidden||Your account does not have access to this resource on the target website.|
||Not Found||The target website server understands the request but refuses to authorize it.|
||Timeout||The request timed out.|
||Too many requests||You have exceeded the rate limit.|
||Internal Server Error||We're experiencing some issues on our side. Please try again a bit later. We're probably aware of the issues, but you can give us a shout and report it anyway.|
||Faulted After Too Many Retries||Something went wrong, and we failed the job you submitted. You can try again at no extra cost, as we don't charge you for faulted jobs. If that doesn't work, please get in touch with us.|
The Oxylabs GitHub repository is the place to go for instructions on how to scrape websites, put our tools into use, implement products, or integrate them using the most widely used programming languages (such as C#, GoLang, Java, NodeJs, PHP, Python, Ruby, etc.).
The Oxylabs dashboard is your personal hub to manage Oxylabs services. There, you’ll find Web Unblocker usage statistics, and you’ll be able to change the sub-users’ passwords. Below, you’ll find more information on these features:
Within the “Statistics” section, you’ll find a detailed view of how much traffic you use daily. You can set the time period of usage as preferred and filter the usage by sub-user and domain. Additionally, you’ll be able to see your requests’ success rates, average response times, and the number of results for each domain.
Your sub-users can be seen in the "Users" section. You can view the usernames and, if necessary, update the passwords.
If you have any further questions about Web Unblocker, please feel free to contact our support team at any time via live chat or email.
About the author
Vytenis Kaubre is a Junior Copywriter at Oxylabs. As his passion lay in creative writing and curiosity in anything tech kept growing, he joined the army of copywriters. After work, you might find Vytenis watching TV shows, brainstorming ideas for tabletop games, or taking Raymond Chandler’s advice, “When in doubt, have a man come through a door with a gun in his hand” too seriously (when writing short stories).
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Get the latest news from data gathering world
Web Unblocker for the most complex targets
Scrape public data from almost any target without getting blocked.
Scale up your business with Oxylabs®
GET IN TOUCH
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
oxylabs.io© 2023 All Rights Reserved