HTTP Headers Explained

Examples of HTTP headers

GET /URL/destination/to/get/ HTTP/1.1
Host: targetwebsite.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) 
Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120
Pragma: no-cache
Cache-Control: no-cache

User-Agent header is one of the most important headers that can define whether your request will be successful or not. Most common user agents should be used to make sure you avoid being blocked while web scraping.

Some of HTTP headers can be grouped according to how proxies deal with them. We have already written about HTTP proxies and their configuration. Here are some headers that interact with proxies:

• Connection – a general header that controls if network connection stays open after the current transaction finishes.

• Keep-Alive – allows the client to indicate how the connection may be used to set a maximum amount of requests and a timeout. For this header to be valid, the connection header must be set to: Keep-Alive.

• Proxy-Authenticate – a response header that defines the authentication method for gaining access to a resource behind a proxy server. This header authenticates the request to the proxy server and allows it to further transmit the request.

• Proxy-Authorization – a request header that includes the credentials to authenticate a user agent to a proxy server. 

• Trailer – a response header that allows the sender to include additional fields at the end of chunked messages. For example, a message integrity check, post-processing status, or digital signature.

• Transfer-Encoding – specifies the form of encoding used to safely transfer the payload body to the sender. This header is applied to a message between two nodes, not to a resource itself.

These are just a few examples of HTTP headers. Listing all of the possible variations of HTTP headers is nearly impossible. A HTTP headers can send many types of requests, or ask for specific languages and encoding, etc.

Why use and optimize HTTP headers?

• Decrease web scraper’s chances of getting blocked by the target server

• Increase the quality of data retrieved from the target server

Simply put, the use of HTTP headers will have a direct impact on what type of data will be retrieved back from web servers, and define its quality.

What’s more, if you will use the HTTP headers accordingly, it will allow you to substantially reduce the chances of getting blocked by web servers.

Nowadays, most web service owners understand that they will most likely get their data scraped by someone. Some scrapers slow down websites, therefore website owners use every tool at their disposal to protect the website. One of these tools might be to automatically block any fake user agents they detect. Some web server owners might even display incorrect information if they detect a fake user agent! If you are looking for information on how to crawl a website without getting blocked, check out our blog.

As mentioned before, HTTP headers carry additional information to web servers, and by optimizing the content of this message, it is possible to make the internet requests seem as it is coming from an organic user. Such traffic to web servers is highly unlikely to be blocked. 

How to secure your web app with HTTP headers?

While HTTP headers can be used by web scrapers to avoid IP blocks, they can also be employed by web servers for web security. Simply speaking, HTTP security headers are a contract between the browser and the developer. This contract is defined by HTTP response headers that set the level of the website’s security. 

Here are some of the most common HTTP headers that allow you to secure your web applications:

• Content-Security-Policy header – provides an additional layer of security. It helps prevent various attacks, including Cross Site Scripting (XSS) and other code injection attacks. The policy defines content sources which are approved and allows the browser to load them.

• Feature-Policy header – either allows or denies the use of the browser in its own frame as well as in content within <iframe> elements.

• X-Frame-Options header – protects website visitors from clickjacking attacks.

• X-XSS-Protection header – used to configure the built in reflective XSS protection. It is found in Chrome, Internet Explorer, and Safari (Webkit).

• Referrer-Policy header – controls how much referrer information, sent via the Referrer header, ought to be included with requests.

• X-Content-Type-Options response header – is a marker used by the server. This marker indicates that the MIME types advertised in the Content-Type headers should be followed, not changed.

You can easily check your HTTP header security online. Various tools let you check which HTTP security headers currently run on your website, you simply need to have an URL that you want to check. 

It’s a wrap

Hopefully, by now you have a decent idea of what is a HTTP header, their purpose, and how they come into play in the web scraping world. We also briefly touched HTTP security headers and how they work.

Of course, it’s only the tip of an iceberg and there are quite a few HTTP headers that need to be taken into account when web scraping. We have covered 5 essential HTTP headers that every web scraper must use and optimize. Check it out, and happy scraping!