The HTTP 431 Request Header Fields Too Large error means the server rejected your request due to overly large headers. In web scraping, this can result from excessive cookies or long Referer URLs. Simplify or reduce the headers before retrying.
The total size of the request headers exceeds the server's limit.
A single header field's size is too large.
The Referer URL included in the header is excessively long.
The request includes too many cookies, increasing the header size beyond acceptable limits.
# Request example curl -X GET https://sandbox.oxylabs.io/ -H 'Accept: text/html' -H 'Accept-Encoding: gzip, deflate' -H 'User-Agent: Mozilla/5.0'
Try Scraper API with 5K results
Check the server's response for the "Upgrade" header to identify the required protocol.
Modify the client's request to support the protocol specified by the server's "Upgrade" header.
Ensure that the client and server both support the upgraded protocol.
Restart the client application to apply the new protocol settings and retry the request.
# Response example HTTP/1.1 431 Request Header Fields Too Large Content-Type: text/html; charset=UTF-8 Content-Length: 49431 - Headers Too Large
Web Scraper API
Public data delivery from a majority of websites
From
49
Get the latest news from data gathering world
Scale up your business with Oxylabs®