Use the -L option in cURL to automatically follow HTTP redirects, ensuring you reach the final destination URL.
Set a limit on the number of redirects with --max-redirs to avoid potential infinite redirect loops, which can protect your script from hanging.
Include -I to display HTTP headers, which can be useful for debugging issues with redirects and understanding the server's response.
Combine -L with -s to follow redirects silently, suppressing the progress meter for cleaner output in scripts or logging.
# Enable following redirects curl -L "https://sandbox.oxylabs.io/products" # Follow up to 5 redirects to avoid loops curl -L --max-redirs 5 "https://sandbox.oxylabs.io/products" # Display the HTTP headers to debug redirects curl -L -I "https://sandbox.oxylabs.io/products" # Use silent mode to hide progress meter but still follow redirects curl -Ls "https://sandbox.oxylabs.io/products" # Combine following redirects with other options, e.g., user-agent curl -LA "Mozilla/5.0" "https://sandbox.oxylabs.io/products"
Verify the initial URL is correct to prevent unnecessary redirects that can complicate the request process and increase response time.
Use the -A option to set a user-agent in your cURL request, as some servers might handle requests differently based on the user-agent string.
Check for cyclic or too many redirects by reviewing the server's configuration or the .htaccess file, which might need adjustments to resolve redirect loops.
# Incorrect: Incorrect initial URL causing unnecessary redirects curl -L "https://sandbox.oxylabs.io/products-old" # Correct: Verify and use the correct initial URL to minimize redirects curl -L "https://sandbox.oxylabs.io/products" # Incorrect: Not setting a user-agent, leading to potential access issues on some servers curl -L "https://sandbox.oxylabs.io/products" # Correct: Use the -A option to set a user-agent curl -LA "Mozilla/5.0" "https://sandbox.oxylabs.io/products"
Web scraper API
Public data delivery from a majority of websites
From
49
Get the latest news from data gathering world
Scale up your business with Oxylabs®