How to Make HTTP Requests in Node.js With Fetch API

What is Fetch API

Fetch API is an application programming interface for fetching network resources. It facilitates making HTTP requests such as GET, POST, etc. 

Fetch API supports new standards, such as Promise, resulting in cleaner code that doesn’t require callbacks.

The native support for the Fetch API exists in all major browsers. JavaScript developers rely on the npm node-fetch package for the server-side code. The package is wildly popular, with millions of downloads every week.

Node.js has released experimental support for the Fetch API with version 17.5, while with Node.js version 18, the Fetch API became stable and is enabled by default as part of the Node.js runtime. Since then, you can write your server-side JavaScript code that uses the Fetch API without installing a third-party library. To check your Node.js version, run the following command in your terminal:

node -v

If you’re running a Node.js version between 17.5 and 18, you can run Fetch API code files by enabling the --experimental-fetch flag:

node --experimental-fetch your_code.js

If your node version is below 17.5, you can install node-fetch with the command below:

npm install node-fetch

How to use Fetch API

For the following examples, a scraping sandbox website will be used as a target. As the Fetch API returns a Promise object, you can use the fetch-then syntax. First, initialize the Node.js project by executing the following lines in your terminal:

mkdir fetch-api-scraper
cd fetch-api-scraper
npm init -y

These commands will create a package.json file in a new folder called fetch-api-scraper.

To see node-fetch in action, create a new .js file and enter the following lines of code:

fetch('https://sandbox.oxylabs.io/products/1')
    .then((response) => response.text())
    .then((body) => {
        console.log(body);
    });

This code sends a fetch request using an HTTP GET method and prints the HTML by logging the response body.

To explain it further, the fetch() method returns a Promise object. The first then() extracts the text from the response, and the second then() prints the response HTML.

Save the code file as product.js, open the terminal, and run the following:

node product.js

Running this line will print the HTML document of the target page:

HTTP headers in Fetch API

Now, let's talk about the response headers. The response object contains all of the response headers in the response.headers collection. If you wish to print the response headers, you can do so by looping over all key and value pairs in the response.headers property:

fetch('https://sandbox.oxylabs.io/products/1')
    .then((response) => {
        response.headers.forEach((value, key) => {
            console.log(`${key}: ${value}`);
        });
    });

You should see a similar output as in the screenshot below:

Axios vs Fetch API

Axios is a popular Node package for making HTTP GET and POST requests with ease. Make sure to check our tutorial on web scraping with JavaScript and Node.js to see a practical example of Axios. Additionally, you might also find it useful to read about using proxies in Node-Fetch and proxy integration with Axios. In case you have a cURL command that you want to replicate using Axios, Node.js, or JavaScript, you can also take advantage of these cURL to Node Axios, cURL to Node.js, and cURL to JavaScript converters.

You can install Axios with the following command:

npm install axios

To send a GET request, call the get() method as follows:

const response = await axios.get(url);

Similarly, to send a POST request, call the post() method as follows:

const response = await axios.post(url);

Let's take an example to see how the Node Fetch API differs from Axios. Send a POST request to https://httpbin.org/post with JSON data. The important things to note here are the following:

• JSON data

• Custom request headers

• The response will be in JSON format

Writing the same code using Axios and Fetch API will distinguish the differences.

The following code uses Axios:

const axios = require('axios');

const url = 'https://httpbin.org/post';
const custom_data = {x: 1920, y: 1080,};
const headers = {'Content-Type': 'application/json',};

axios.post(url, custom_data, {
    headers: headers,
})
    .then(({data}) => {
        console.log(data);
    })
    .catch((error) => {
        console.error(error);
    });

And below you can find a Node Fetch example:

const url = 'https://httpbin.org/post';
const custom_data = {x: 1920, y: 1080,};
const headers = {'Content-Type': 'application/json',};

fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(custom_data),
})
    .then((response) => response.json())
    .then((data) => {
        console.log(data);
    });

Both of these code snippets will produce the same output.

As evident from the examples above, here are the differences between Axios and Fetch API:

• Fetch API uses the body property of the request, while Axios uses the data property.

• Using Axios, JSON data can be sent directly, while Fetch API requires the conversion to a string.

• Axios can handle JSON directly. The Fetch API requires the response.json() method to be called first to get the response in JSON format.

• The response data variable name must be data in the case of Axios, while it can be anything in the case of Fetch API.

• Axios allows an easy way to monitor and update progress using the progress event. There is no direct method in Fetch API.

• Fetch API does not support interceptors, while Axios does.

• Fetch API allows the streaming of a response, while Axios doesn’t.

Common errors and mistakes

Here are some of the most common Fetch API errors and mistakes you may encounter, along with their solutions:

• The Fetch API does not throw an error for HTTP response codes like 404 or 500. Check the response.ok property to determine if the request was successful.

• Fetch API does not natively support request timeouts, causing requests to hang indefinitely. Implement timeouts manually using techniques like AbortController.

• When using response.body, forgetting to properly consume or close the stream can lead to memory leaks. Consume the stream completely using methods like .json() or .text().

• Fetch does not send cookies by default, which can break authentication for same-origin requests. To include cookies in your Fetch requests, you need to explicitly set the credentials option in the Fetch API.

• Fetch fails silently for CORS errors; response.ok or catch won’t help since the request doesn't even reach the server. Check the browser console for CORS issues and configure server headers to allow cross-origin requests.

Conclusion

The addition of Fetch API to Node.js is a long-awaited feature, which is now stable and included with node by default. Combined with libraries such as Cheerio, the Fetch API can also be used for web scraping.

Curious to find out more about web scraping? Make sure to check our blog. If you want to learn about a different method of scraping via a headless browser, refer to our Puppeteer tutorial. Also, don’t hesitate to try our general-purpose web scraper for free.