Many businesses performing web scraping operations may find themselves wondering how to check if their proxies are running correctly and efficiently. While the most effective way to test a proxy is via using your own script, there are other ways that may be useful even with the lack of technical experience.
This article will clarify how to test proxies, what parameters should be considered, and what tools might assist, including simple and advanced ones.
What do you need to test proxies?
The essential thing that can be done first is to check if your proxy is actually operating. Usually proxies are set up in a browser where you can also track their activity. In one of our previous articles, we have already explained how to use proxy settings in Chrome.
One more way to see if your proxy is running is via a proxy manager. Proxy managers are a perfect match if you use multiple proxies and want to switch between them without much effort. One of the proxy managers that is easy to use is Oxylabs Proxy Manager.
You can also use online IP checkers to see if your proxy is working. Many of them are accessible on the internet, but to demonstrate how their general logic works, we can refer to whatismyproxy.com. You have to go to the website, and it will immediately reveal your IP address and location, as well as the HTTP headers of the software you’re using.
Although some IP checkers can show your IP address and whether you’re connecting through a proxy, their functionality is quite limited. For a more detailed analysis, more advanced tools are needed.
Some websites use third-party IP location databases, such as IP2Location or MaxMind, to understand what kind of users visit their website and if they are real in the first place. To figure that out, an IP status or usage type must be checked. IP location databases have their own methods to determine whether the IP address belongs to data centers or is residential.
The easiest way to check the proxy status is by visiting ip2location.com. This IP database, which is widely considered to be one of the most accurate, allows you to check the details of your IP address, including the likely usage type.
Suppose a residential proxy that you’re trying to use is recognized as a datacenter by this database. In that case, the chances are that many of your target websites will see the same information and may block this proxy and an actual residential IP address much faster.
FOGLDN Proxy Tester
To evaluate the speed of your proxies, such tools as FOGLDN Proxy Tester can be used. It enables direct ping times to any website worldwide. Since it shows how much time it takes to connect, you can use it to monitor the latency of your proxies. With this proxy tester, you can test any proxy type, including rotating proxies.
FOGLDN Proxy Tester has its drawbacks. For example, if the test run fails, you won’t be able to know the exact reason. It doesn’t reveal the location of the proxy, its usage type, and the level of anonymity. Finally, you can’t control the number of requests made, as well as their frequency.
This proxy tester is easy to use. You can test your proxies with this tool by accomplishing the following steps:
- Download the tool
- Add the list of the proxies you want to test. If you’re using a rotating gateway address, you can add it multiple times
- Enter an URL to test
- Select ‘TEST PROXIES’
A more advanced free proxy checker is hidemy.name. It can detect the proxy type, precise location, speed, and anonymity level. Hidemy.name proxy checker provides a list of proxies automatically collected from numerous websites all over the internet, including private databases. Proxies can be sorted out by their protocol types, such as HTTP, HTTPS, and SOCKS.
Another feature that hidemy.name can determine is the level of anonymity, i.e., whether your actual IP address remains concealed and if there is a chance that the destination server can suspect that you are behind a proxy server. The anonymity categories that this proxy checker offers in their proxy lists include:
- No anonymity. The server you are reaching is aware of your real IP and that you’re connecting through a proxy.
- Low anonymity. The remote server is aware of you using a proxy. However, your actual IP address stays hidden.
- Average anonymity. The destination server can be tricked into thinking that it knows your IP address, but the IP address displayed is not accurate. The server is aware of you using a proxy.
- High anonymity. The server you are reaching detects neither your actual IP address nor whether proxies are involved.
One more thing that is more likely to be a matter of concern is whether your proxies are blocked from specific websites or not. The only way to tell if a proxy is blocked is to test proxies directly on the targeted websites. Your IP may be marked as suspicious at some point, but it might need to make many requests before any blocking.
Another way to check proxies is to write your own script or software. This way, you would have more control over how to check your proxies. This, however, takes time and effort. The basic approach is to write a scraper that takes a proxy for a list and uses it to extract data from a simple, preferably static website that returns a bare minimum of information.
The first step is to understand whether the proxy responds at all. You need to check the content of the body, either by looking for a specific string in HTML output or by checking the content size (if the content is static). You can look at the status code in response headers. An HTTP status code 200 would suggest that the proxy worked and you were able to get the content, but checking the actual output is always more reliable.
Every request with a proxy should have a reasonable timeout set up. It could be 10 seconds for a datacenter proxy and up to 30 seconds for a residential proxy. If no content returns in that amount of time, you can consider the proxy to be unusable. It also makes sense to log the amount of time it took to complete each request. Some requests may not timeout, but they still can be too slow for real-life use cases. All this data should then be written down to a database or printed out on the screen for analysis.
There are multiple tools you can use to test proxies. IP checkers are accessible online and can reveal your software’s IP address, location, and HTTP headers. IP databases, such as IP2Location or MaxMind, may assist in checking the IP status of your proxy and its usage type. FOGLDN Proxy Tester, an easy-to-use tester that works with many proxy types, helps monitor the latency of your proxies. The hidemy.name proxy checker can determine the level of anonymity.
Although numerous proxy checkers and testers exist, the most effective and reliable way to test proxies is by writing a script capable of sending requests to specific websites and getting responses, as well as counting the time it took to complete the request.