We have officially released Next-Gen Residential Proxies, a technology that will irreversibly change the proxy service industry. To mark the occasion, we took an opportunity to interview a proxy expert behind this revolutionary web data gathering solution. Why is it so revolutionary, you ask?
We caught Aleksandras Sulzenko, a man who oversaw the development process of Next-Gen Residential Proxies, to figure it out. Six months ago Aleksandras was challenged to redefine innovation in the proxy industry. He has been leading a team of developers, data analysts, and system administrators to bring to life a never before seen solution.
Aleksandras, can you tell us how the idea of Next-Gen Residential Proxies came about?
When I first joined Oxylabs, three years ago, the company mainly specialized in Datacenter Proxies. This type of proxy offers significant advantages, including impressive operation speed. However, it also has its own drawbacks, namely recurring blocks. I witnessed numerous cases where our clients would struggle with data gathering as their IPs would get blocked, and business would grind to a halt.
Naturally, these situations are quite stressful, especially if web data gathering and processing is the core business of a company. We employed numerous tactics to help our clients overcome these issues. Our team did it all, from replacing the proxy sets, to providing consultations on best web data gathering practices. However, it was clear that our efforts to fix constantly recurring problems are not a sustainable solution. We needed to develop a product, which would, by design, avoid all blocks.
We equipped the product with the most progressive technologies available to make it block-free and six months later, the Next-Gen Residential Proxies were born.
What were the challenges of building something that has never been done before?
In part, the answer is already in the question. Innovating is never easy, as there are no guidelines. Many of the obstacles my team has faced had no precedent, so we needed to be creative.
Our goal to deliver a radically different solution pushed us to re-evaluate the processes of proxy services. We had to dive deep at the heart of the problem and understand the essence behind common pitfalls. This helped us to understand what features would make Next-Gen Residential Proxies infallible.
Can you explain what is so different about Next-Gen Residential Proxies?
To understand this fully, you must know the context of proxy services. HTTP(S) proxies have been in use for decades. One thing that makes innovation tricky is the fact that the proxy technology itself was not designed with web data gathering in mind.
The web scraper bots tend to get blocked quite often if they don’t mimic the human online behavior convincingly. This means that to gather data from any given site, first, there is a need to figure out how to program the bot to imitate real human browsing patterns. The unique aspect of Next-Gen Residential Proxies is that they are already set to operate in a way that resembles a human browsing session, saving valuable time for our clients.
How does AI-powered dynamic fingerprinting work exactly?
To put it very simply, AI dynamic fingerprinting is a feature that allows the scraper bot to appear as a real-life website visitor. This ability is essential to perform block-free, sustainable data gathering. When a bot equipped with AI dynamic fingerprinting visits a site, it is capable of collecting information undetected by providing user-related information, such as the browser type and version, among many other variables.
Before the Next-Gen Residential Proxy introduction, the only solution was going through with tedious tasks of testing a bunch of combinations of all user-related variables and selecting the ones yielding the highest success rate. This method required some serious data analysis chops and a well-prepared infrastructure.
The good news is that now this lengthy process can be skipped. Next-Gen Residential Proxies automizes the testing and selection phase as AI-powered dynamic fingerprinting adopts the most appropriate fingerprint for any given site instantly. This ability maximizes the success rate and ensures that the high-quality data is collected timelessly and effortlessly. There is no other proxy available on the market, which could perform anything like this.
What about the Auto-Retry system?
There is a saying – the ones who refuse to give up eventually are destined to succeed. We adapted the same philosophy to Next-Gen Residential Proxies.
When it comes to web scraping, there is a need to understand if the data collection was successful or not. “Successful” means that the extracted data is of high quality and can be used to fulfill the business needs. To figure it out, data collectors used to evaluate every session separately and, in case of failed attempts, retry the whole process. I hope it is clear for everyone how long this evaluation process can take.
That is why we integrated the Auto-Retry system, which is smart enough to recognize poor quality data and restart the process automatically as many times as needed to receive a satisfactory result. In other words, Next-Gen Residential Proxies don’t know how to fail.
Is there any room for improvement left?
If you deal with information technologies, there is no limit for improvement. The same goes for Next-Gen Residential Proxies. In fact, we are at the last stages of integrating one more feature, which employs machine-learning adaptive parsing to deliver structured data from a vast amount of e-commerce sites. I can give another secret away: we are not stopping here.