Proxy Integration With Node-Fetch

• For Enterprise Dedicated Datacenter Proxies, you’ll have to choose an IP address from the acquired list.

• For Self-Service Dedicated Datacenter Proxies, the port indicates the sequential number of an IP address from the acquired list.

• For Shared Datacenter Proxies, you can also use country-specific entries. For instance, if you enter dc.fr-pr.oxylabs.io:42000, you'll have a French exit node. Once again, refer to our documentation for more information and the entire list of country-specific entries.

Web Unblocker

When it comes to Web Unblocker, you have to use the unblock.oxylabs.io:60000 endpoint and ignore the SSL certificate. It supports HTTP and HTTPS connection protocols. You can also connect to various geo-locations, use a Headless Browser, and utilize other functionalities by sending them as request headers. Visit the documentation to learn more.

Testing proxies

Now, it's time to test the proxies. This step would be the same for both Residential Proxies and Datacenter Proxies.

Create an instance of the HttpsProxyAgent class. The constructor of this class takes the proxy URL we have just created:

const proxyAgent = new HttpsProxyAgent(proxyUrl);

Finally, we can send this proxy agent to one of the optional parameters of the fetch method—agent. This agent represents an http(s) agent instance, which we have created using the http-proxy-agent package.

Putting together everything, the node fetch proxy code looks as follows:

const fetch = require('node-fetch');
const { HttpsProxyAgent } = require('https-proxy-agent'); 

const proxyUrl = `http://USERNAME:PASSWORD@pr.oxylabs.io:7777`;

(async () => {
  try {
    const proxyAgent = new HttpsProxyAgent(proxyUrl);
    const response = await fetch('https://ip.oxylabs.io/location', {
      agent: proxyAgent,
    });
    const data = await response.text();
    console.log(data);
  } catch (e) {
    console.error(e.message);
  }
})();

You can run this code to see the IP of the proxy address.

If you want to integrate Web Unblocker, you must ignore the SSL certificate by adding an additional line before the fetch() function:

(async () => {
  try {
    const proxyAgent = new HttpsProxyAgent(proxyUrl);
    process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;

How to rotate proxies with node fetch

Notably, while using proxies, most websites will ban the IP if you use the same proxy. The solution is to rotate the IPs.

Oxylabs Residential and Shared Datacenter Proxies perform proxy rotation automatically and don't require external rotation.

Our Residential Proxies can either randomly change the proxy address for each request or use the same proxy IP for up to 30 minutes. Shared Datacenter Proxies also offer the above mentioned options but can keep the same IP indefinitely.

See our documentation for Residential and Shared Datacenter Proxies to find out more.

Dedicated Datacenter Proxies don't have an in-built rotation feature, but they can be implemented with our Proxy Rotator. With this tool, you can easily automate the rotation of our Dedicated Datacenter Proxies.

Selecting a proxy randomly 

Assuming you have a proxy list, you can use the following code to rotate the proxies from the given proxy list. Within the example, we run the code three times, each time picking one of the proxies randomly.

const fetch = require("node-fetch");
const { HttpsProxyAgent } = require('https-proxy-agent'); 

proxyUrls =[
  '127.0.0.1:60000',
  '127.0.0.2:60000',
  '127.0.0.3:60000'
]; 

(async () => {
  try {
    for (let i = 0; i < 3; i++) {
      const randomIndex = Math.floor(Math.random() * proxyUrls.length); // Generate a random index
      const proxyAgent = new HttpsProxyAgent(proxyUrls[randomIndex]); // Select a proxy URL using the random index
      const response = await fetch("https://ip.oxylabs.io/location", {
        agent: proxyAgent,
      });
      const data = await response.text();
      console.log(data);
    }
  } catch (e) {
    console.error(e.message);
  }
})();

Iterating over the proxy list

You can modify the above code so that it goes over all the proxies in a sequence too:

(async () => {
  try {
    for (let i = 0; i < proxyUrls.length; i++) {
      const proxyAgent = new HttpsProxyAgent(proxyUrls[i]);
      const response = await fetch("https://ip.oxylabs.io/location", {
        agent: proxyAgent,
      });
      const data = await response.text();
      console.log(data);
    }
  } catch (e) {
    console.error(e.message);
  }
})();

Conclusion

You can easily incorporate proxies into Node Fetch using the https-proxy-agent package. This integration enables you to initiate web scraping projects with Node-Fetch while avoiding concerns related to IP blocks and geo-restrictions caused by proxies.

If you have any questions or need assistance, don't hesitate to get in touch with us.

Please be aware that this is a third-party tool not owned or controlled by Oxylabs. Each third-party provider is responsible for its own software and services. Consequently, Oxylabs will have no liability or responsibility to you regarding those services. Please carefully review the third party's policies and practices and/or conduct due diligence before accessing or using third-party services.