How to Use Timeouts With cURL Requests?

cURL is a command line tool for transferring data using various network protocols. When making HTTP cURL requests, properly handling timeouts is crucial for a reliable connection. Using cURL, you can set timeout parameters for different stages of the communication process.

Timeouts prevent cURL from hanging indefinitely when a connection or operation takes too long. Appropriate timeouts handle slow connections, unresponsive servers, and large file transfers.

Timeout behavior may vary depending on the specific cURL version and the operating system you’re using.

This guide covers the main timeout command options and provides examples using the cURL command line tool.

Key timeout options

These options help prevent your requests from hanging indefinitely and are especially useful in scripts or automated processes, such as web scraping.

1. --max-time / -m

Total operation timeout – the maximum time (in seconds) allowed for the whole operation, including DNS resolution, connection time, and data transfer.

Limit total operation time to 30 seconds:

curl --max-time 30 https://oxylabs.io

You can also use the short option:

curl -m 30 https://oxylabs.io

2. --connect-timeout

Specify the maximum time allowed for the connection phase only. Allow 5 seconds for initial connection:

curl --connect-timeout 5 https://oxylabs.io

Combine with max-time for more control:

curl --connect-timeout 5 --max-time 30 https://oxylabs.io

3. --speed-limit and --speed-time

Use together to abort transfers (in bytes) that are too slow. Use K, M, and G for KBs, MBs, and GBs.

Abort if the speed drops below 1MB/s for 30 seconds:

curl --speed-limit 1M --speed-time 30 https://oxylabs.io

Combine specified timeout options for a desirable result. This command sets a connect timeout of 10 seconds, a request timeout of 30 seconds, and aborts the transfer if the download speed is below 1MB/s for 10 seconds.

curl -m 10 --max-time 30 --speed-limit 1M --speed-time 10 https://oxylabs.io

Common scenarios

Adjust timeouts based on the operation type.

• Short timeouts for API endpoints

• Longer timeouts for file downloads

• Very long timeouts for backup operations

Basic API request

cURL with API request:

curl --connect-timeout 5 \
     --max-time 30 \
     https://realtime.oxylabs.io

Downloading large files

Download files with cURL using generous timeouts and speed monitoring:

curl --connect-timeout 10 \
     --max-time 3600 \
     --speed-limit 10M \
     --speed-time 60 \
     --output file.zip \
     https://oxylabs.io/file.zip

Retrying failed attempts

Retry up to 3 times and wait 2 seconds between retry attempts:

curl --connect-timeout 5 \
     --max-time 30 \
     --retry 3 \
     --retry-delay 2 \
     https://oxylabs.io

Proxy integration

Configure cURL with proxy request routing using separate timeout controls:

curl --connect-timeout 5 \
     --max-time 30 \
     --proxy "pr.oxylabs.io:7777" \ # Oxylabs Residential Proxies 
     --proxy-connect-timeout 3 \
     https://oxylabs.io

Best practices

Set both timeouts

Always specify both connect and maximum timeouts:

curl --connect-timeout 5 --max-time 30 https://oxylabs.io

Adjust for large downloads

Use longer timeouts for large files:

curl --connect-timeout 10 \
     --max-time 3600 \
     --output file.zip \
     https://oxylabs.io/file.zip

Include progress monitoring

Add progress monitoring for long-running transfers:

curl --connect-timeout 10 \
     --max-time 3600 \
     --progress-bar \
     --write-out "\n%{time_total}s total, %{speed_download} bytes/sec average\n" \
     https://oxylabs.io/file.zip

Layer your timeouts

Implement multiple layers of timeout protection:

curl --connect-timeout 5 \  # Connection phase
     --max-time 30 \        # Overall transfer
     --speed-limit 1M \     # Minimum speed
     --speed-time 15 \      # Time window for speed check
     https://oxylabs.io

Use retry logic

Automatically retry failed requests with increasing delays:

curl --connect-timeout 5 \
     --max-time 30 \
     --retry 3 \
     --retry-delay 2 \
     --retry-max-time 60 \
     --retry-all-errors \
     https://oxylabs.io

Error handling and logging

Add options for comprehensive timeout error handling and logging all transferred data into a file:

curl --connect-timeout 5 \
     --max-time 30 \
     --verbose \
     --trace-ascii curl_trace.log \
     --write-out "\nHTTP Code: %{http_code}\nTotal Time: %{time_total}s\n" \
     https://oxylabs.io

Troubleshooting

Enable verbose output

Add -v for detailed progress and error information:

curl -v --connect-timeout 5 --max-time 30 https://oxylabs.io

Test connection only

Use --head to test connection without downloading:

curl --head --connect-timeout 5 https://oxylabs.io

Preventing common issues

1. Always set both connect and maximum timeouts to avoid hanging indefinitely. Otherwise, you might encounter resource leaks (open connections/sockets), poor user experience, and cascading failures in your system.

2. Use retry logic for resilience.

3. Adjust timeouts based on operation type: API endpoints, downloads, or database backups.

4. Include proper monitoring and logging to know precisely what is happening.

5. Consider network conditions when setting limits: unresponsive servers, rate-limited servers, or download interruptions.

Wrap up

For more information about timeouts and other options, refer to the cURL documentation by running man curl in the terminal or visiting the official cURL website.