How to Show HTTP Response Headers in cURL?

HTTP response headers are crucial metadata that servers send back when responding to web requests. Understanding these headers is fundamental for developers and system administrators who need to debug web applications, test API implementations, or verify security configurations. For web scraping, they also act as a source of essential information that can help you verify access and permissions, optimize request patterns, as well as properly handle the received data.

In this guide, we’ll show you various ways to inspect response headers using cURL, the versatile command-line tool for making HTTP requests.

Different ways to check HTTP headers

When working with HTTP headers, different cURL options provide varying levels of detail and flexibility. Here are some of the most effective ways to inspect response headers, depending on your needs.

1. Using -I (HEAD request)

The -I flag instructs cURL to send a HEAD request, which retrieves only the headers without the response body. This method is particularly useful when you need to quickly check response headers or verify server configurations without downloading additional content.

curl -I https://api.example.com

2. Using -i (include headers in response)

In comparison to the method above, the -i option tells cURL to include response headers in the output along with the response body. This approach is best for situations when you need to analyze how headers relate to the actual content or when debugging content-specific issues. It provides a complete picture of the HTTP response while maintaining the natural flow between headers and body content.

curl -i https://api.example.com

3. Using -v (verbose mode)

Verbose mode provides the most detailed view of the HTTP transaction, including both request and response headers, SSL/TLS handshake details, and timing information. A comprehensive method like this is vital for debugging complex issues, understanding redirect chains, or verifying security configurations. It’s particularly useful when working with APIs that require specific request headers or when troubleshooting connection issues.

curl -v https://api.example.com

4. Using --head (alternative to -I)

While being essentially identical to -I, the --head is a more verbose alternative, making your scripts more readable and self-documenting. You should use it in situations when clarity in documentation and automation scripts is paramount (e.g., team collaboration).

# Standard head request with full flag name
curl --head https://api.example.com

# Combining with additional options for enhanced output
curl --head --silent https://api.example.com | grep "content-type"

5. Using -D to save headers to a file

The -D option allows you to save response headers to a separate file while still processing the response body. This separation is particularly valuable for automated testing, logging, and analysis scenarios.

# Save headers to a file while processing the response
curl -D headers.txt https://api.example.com

# Save headers with timestamp for logging
curl -D "headers_$(date +%Y%m%d_%H%M%S).txt" https://api.example.com

# Save headers and response body separately
curl -D headers.txt -o response.txt https://api.example.com

6. Extracting specific headers

If you need to focus on specific aspects of HTTP responses without wading through unnecessary information, extracting specific headers is a common solution. This approach is especially valuable when monitoring specific behaviors like rate limiting, caching policies, or security configurations. 

# Extract a single header
curl -I https://api.example.com | grep -i "content-type"

# Extract multiple headers using pattern matching
curl -I https://api.example.com | grep -E "content-type|server|date"

# Format specific headers for analysis
curl -I https://api.example.com | grep -i "x-rate-limit" | awk '{print $2}'

Advanced tips

Now that we’ve looked at specific ways of showing HTTP headers in cURL, let’s dig into some advanced HTTP header techniques that become essential when you’re moving from basic web-scraping into professional-grade public data collection and API integration. 

1. Debugging HTTP headers

Working with HTTP headers often requires sophisticated debugging techniques, including: 

• Use -v with -s to suppress the progress meter while maintaining detailed output. 

• Combine this with output redirection to create detailed logs for analysis. 

• For automated testing, consider using grep or awk to extract and compare specific headers across multiple requests.

2. Pretty printing 

With pretty printing, you can make header output more readable by formatting it with text processing tools. The sed command can help break up header lines, while awk can be used for more complex formatting. Consider creating shell functions for commonly used formatting patterns to streamline your workflow.

# Format headers for better readability
curl -I https://api.example.com | sed 's/\r/\n/g'

3. Automation

For frequent header checks, automation can significantly improve efficiency.

• Create shell aliases or functions for frequent header checks to improve efficiency. 

• Write scripts that monitor header changes over time or compare headers across different environments. 

• Use output redirection and text processing tools to create structured logs for automated analysis.

Common issues when working with HTTP headers

While inspecting HTTP headers with cURL, you may run into common challenges. Below are some of the most frequent issues and their solutions.

1. SSL certificate verification issues

SSL certificate verification problems are among the most common challenges when making HTTPS requests to secure websites. While cURL performs SSL certificate verification by default to ensure secure connections, you may encounter issues with self-signed certificates or when testing in development environments. The -k option allows you to bypass SSL verification, though this should be used cautiously and never in production environments where security is critical.

# Skip SSL verification (development only)
curl -k -I https://api.example.com

# Show certificate information while making request
curl -v --cacert /path/to/certificate https://api.example.com

2. Following redirects

Many websites use redirects to route traffic between different URLs, which can complicate header analysis if not handled properly. By default, cURL doesn't follow redirects, which means you might miss important headers from the final destination. The -L flag instructs cURL to follow redirects automatically, allowing you to see headers from both the initial request and subsequent redirect responses.

# Follow redirects and show all headers
curl -L -I https://api.example.com

# Follow redirects with maximum depth
curl -L --max-redirs 5 -I https://api.example.com

# Show headers for each redirect step
curl -L -v https://api.example.com

3. Character encoding issues

Character encoding problems can arise when dealing with international websites or content containing non-ASCII characters. These issues often manifest as garbled text in headers or response content, particularly when working with different language encodings. Using the --compressed option tells cURL to handle compressed content automatically, while also managing character encoding more effectively.

# Force UTF-8 encoding and handle compression
curl -I --compressed https://api.example.com

# Specify accept-encoding header explicitly
curl -H "Accept-Encoding: gzip, deflate" -I https://api.example.com

# Handle international characters in headers
curl -I --compressed -H "Accept-Charset: UTF-8" https://api.example.com

Remember: While all these solutions can help resolve common issues, they should be implemented thoughtfully with consideration for security and performance implications in your specific use case.

Wrapping up

Understanding how to inspect HTTP headers with cURL is crucial for web development, debugging, and scraping activities. In this blog post, we looked at some useful methods to check HTTP headers, covered some advanced tips and common issues. At the end of the day, you should choose the most suitable approach depending on your project needs, so we hope this article will be a useful source of information when making that choice.

Dive deeper into cURL with a series of comprehensive blog posts focused on this topic: How to Send HTTP Headers With cURL, How to Send cURL OPTIONS Request, How to Send GET Requests With cURL, and How to Use cURL With Proxy.