How to Send HTTP Headers With cURL

Augustas Pelakauskas

Last updated on

2025-06-27

5 min read

Web scraping is public data extraction from web pages. When you browse the web on your computer, to transfer data from a web page, a browser sends an HTTP request that includes HTTP headers, among other things.

HTTP request headers play a crucial role in web scraping, conveying additional information between web servers and clients. Customizing HTTP request headers opens better communication between your software and the target website.

In this guide, you'll learn how to send and receive HTTP headers using cURL, a versatile command-line tool for transferring data with URL syntax.

Sending HTTP headers

Every HTTP request and response may carry some additional information called HTTP headers. These headers provide important metadata, such as the content type, language, and caching instructions. Using HTTP headers, web developers can ensure that their websites function properly and provide a seamless experience to users.

HTTP headers consist of a name-value pair, separated by a colon – :. The name identifies the type of information sent, while the value is the actual data.

Some of the most common HTTP headers include User-Agent, Content-Type, Accept, and Cache-Control.

When you send an HTTP cURL request, it sends the following headers by default:

Host: example.com
User-Agent: curl/7.87.0
accept: */*

You can change these values when sending a request using cURL with headers.

To send HTTP headers with cURL via a terminal or any other command-line setup, you can use the cURL --header or -H option followed by the name and value in the format "Header-Name: value":

curl -H "User-Agent: MyCustomUserAgent" http://httpbin.org/headers

In the example below, a custom User-Agent is sent as "MyCustomUserAgent" when requesting the http://httpbin.org/headers page.

Changing the value of User-Agent

The http://httpbin.org/headers page is meant for testing as it returns a JSON file with all the headers it found in the request. Ignore the X-Amzn header that this site uses internally.

Sending custom HTTP headers

Custom HTTP headers can serve purposes such as authentication, content negotiation, or adding metadata to your cURL requests.

To send custom HTTP headers, use the -H option and provide the header name and value as shown in the previous section. Here's another cURL header example:

curl -H "Authorization: Bearer my-access-token" http://httpbin.org/headers

In this example, an Authorization header is sent with the value "Bearer my-access-token" to access a protected resource at http://httpbin.org/headers.

Sending multiple headers

To pass multiple headers with cURL, you can use the -H option multiple times in the same command line. Each -H option should be followed by a different name and value:

curl -H "User-Agent: MyCustomUserAgent" -H "Accept: application/json" http://httpbin.org/headers

In this example, we pass multiple headers for two applications:

A custom User-Agent.
An Accept header indicating the preference for JSON responses.

By using the same process, you can send as many headers as needed.

Get/show HTTP headers

To view the response headers from a web server, you can use the -I or --head option with cURL. This will issue a HEAD request, which retrieves only the headers without the actual content:

curl -I http://httpbin.org/headers
curl --head http://httpbin.org/headers

You can also use the -i or --include command-line option to show both the response headers and the content in the output:

curl -i http://httpbin.org/headers
curl --include http://httpbin.org/headers

Advanced tips for working with cURL headers

Sending empty headers

If you want to send a header with no value, provide the header name followed by a semicolon:

curl -H "User-Agent;" http://httpbin.org/headers

Removing headers

If you want to remove a header that cURL adds by default, you can provide the cURL with header name followed by a colon without a value. For example, to remove the User-Agent header:

curl -H "User-Agent:" http://httpbin.org/headers

You can use a colon with no value to remove a header

Verbose mode

If you want to see more detailed information about the request and response, including the headers sent and received, you can use the -v or --verbose option. This can be helpful for debugging purposes:

curl -v http://httpbin.org/headers
curl --verbose http://httpbin.org/headers

Saving headers to a file

If you want to save the response headers to a file for further analysis, you can use the -o or --output option along with the -D or --dump-header option:

curl -D headers.txt -o content.txt http://httpbin.org/headers

In this example, the response headers will be saved to headers.txt, and the content will be saved to content.txt. This works whether you're sending single or multiple headers in your cURL requests.

Common use cases for custom headers with cURL

In addition to the examples provided above, there are other use cases for sending custom headers with cURL requests. Following are some of the common scenarios where custom request headers are particularly useful.

Changing response format

If you're making a cURL Post request or other cURL requests to an API or web service, you may need to specify the desired response format, such as JSON or XML. In this case, you can use the cURL Accept header to indicate your preference:

curl -H "Accept: application/json" http://httpbin.org/headers

Conditional requests

You can use conditional headers like If-Modified-Since or If-None-Match to request a resource only if it has been modified since a specific date or doesn't match a specific ETag value. These request headers help minimize bandwidth usage and optimize scraping performance:

curl -H "If-Modified-Since: Sun, 06 Nov 2022 08:49:37 GMT" http://httpbin.org/headers

Referer

When making requests to certain websites or APIs, you may need to include a Referer header to provide the source of the request. This can be important for tracking purposes or to comply with specific API requirements:

curl -H "Referer: http://example.com" http://httpbin.org/headers

Custom authentication

While the Authorization header is commonly used for authentication, some APIs or services may require a custom header. In such cases, you will need to include a custom authorization header with the appropriate information:

curl -H "X-Api-Key: my-api-key" http://httpbin.org/headers

Troubleshooting common cURL header issues

You might encounter some common issues when working with cURL and HTTP headers. Here are a few tips to help you troubleshoot problems.

Double-check the header syntax

Ensure the header name and value are correctly formatted, with a colon (":") separating them. Also, look out for extra spaces or typos, especially when you pass multiple headers.

Verify if the header is supported

Not all headers are supported by every API or web service. Check the documentation for the target API or website to confirm that the header you're using is accepted and properly implemented.

Check for case sensitivity

Although HTTP headers are generally case-insensitive, some APIs or services might expect headers with specific capitalization. If you're having issues, try adjusting the capitalization of the header name to match the API documentation or examples.

Examine the response for errors

When you encounter issues, it's important to review the response from the server, which might include status codes or error messages that can help you diagnose the problem. Use the -i or --include option to view the cURL response headers and content together, which can help you identify issues related to the headers you're sending.

Wrapping up

If you want to know more about the essence of the topic, check what HTTP headers are and get familiar with cURL and its usage. We've also covered how to send GET requests with curl, how to send cURL POST requests, how to show HTTP headers, and how to use cURL in JavaScript, so feel free to explore these blog posts. Additionally, you can learn how to use cURL OPTIONS requests for web scraping.

Additionally, see how cURL works with Python and for proxying – proxy applications with cURL.

You can also check this topic on our GitHub.

If you have any questions about the process above, don’t hesitate to reach us via the live chat on our homepage or email us at support@oxylabs.io.

Try Oxylabs Scraper APIs

Test Oxylabs Scraper APIs designed for advanced web scraping tasks:

SERP Scraper API (including Google scraper)
E-Commerce Scraper API
Web Scraper API

5K requests for free
No credit card is required

Frequently asked questions

How do you add headers in cURL?

To add HTTP headers in cURL, use the -H or --header command followed by the name and value in the format "Header-Name: value":

curl -H "User-Agent: MyCustomUserAgent" http://httpbin.org/headers

Does cURL automatically add headers?

Yes, cURL automatically adds standard headers, such as User-Agent, Accept, and Host, based on the request type and other options. You can override or add custom headers using the -H command.

How to check HTTP headers in cURL?

To check HTTP headers in cURL, use the -I or --head option to only retrieve headers without the actual content:

curl -I http://httpbin.org/headers

Alternatively, you can use the -i or --include option to show both the response headers and content in the output:

curl -i http://httpbin.org/headers

Can I send empty headers with cURL?

Yes, you can send empty headers by providing the header name followed by a semicolon:

curl -H "User-Agent;" http://httpbin.org/headers

How can I remove a default header in cURL?

To remove a header that cURL adds by default, provide the header name followed by a colon without a value. For example, to remove the User-Agent header:

curl -H "User-Agent:" http://httpbin.org/headers

About the author

Augustas Pelakauskas

Former Senior Technical Copywriter

Augustas Pelakauskas was a Senior Technical Copywriter at Oxylabs. Coming from an artistic background, he is deeply invested in various creative ventures - the most recent being writing. After testing his abilities in freelance journalism, he transitioned to tech content creation. When at ease, he enjoys the sunny outdoors and active recreation. As it turns out, his bicycle is his fourth-best friend.

Learn more about Augustas Pelakauskas Learn more about Augustas Pelakauskas

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.