Wget vs cURL for Downloading Files in Linux

cURL and Wget are command-line tools for downloading files and interacting with web content. cURL was initially released in 1997 and is renowned for its versatility in transferring data using various network protocols. Wget, which stands for "World Wide Web get," was released in 1996 and is specifically designed for non-interactive downloads of files from the web.

Wget vs. cURL

While both cURL and Wget can be used for downloading files, they differ in their functionalities, command structures, use cases, and more. Let's take a look.

Protocols

Both tools support common protocols like HTTP, HTTPS and FTP. In addition to these, cURL supports protocols like DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, and more.

Proxies

You can integrate proxies with both tools to route requests through intermediary servers, which helps maintain anonymity or access content from a specific location.

Authentication

Both Wget and cURL support authentication, allowing users to access resources when providing credentials is neccessary. 

Operating systems

Wget is the most popular for Unix-based systems like Linux, and cURL is often available by default on Windows and MacOS. However, both tools can be used with multiple operating systems by installing it separately.  

File upload

You cannot upload files using Wget, but you can use an additional tool called wput for uploading files via FTP. Meanwhile, cURL supports file uploads natively.

Additional libraries

Curl can be used with additional libraries which makes it more versatile, but it can also add complexity to its setup and usage. Wget, on the other hand, doesn't have additional libraries.

Recursive download

Wget is capable to download recursively, allowing users to fetch entire websites or directory structures by following links. This feature is particularly useful for mirroring websites or creating local copies of large sets of files. cURL doesn't support recursive downloads, making wget the better choice for these types of tasks.

Use cases

As mentioned above, Wget is a good for mirroring websites, downloading entire directories recursively for offline viewing or backups. cURL excels at interacting with APIs, handling complex web requests (GET, POST, PUT, DELETE). Curl header configurations and extensive protocol support it perfect for complex network interactions.

API interaction

While both tools could be used with APIs, cURL is a more suitable option due to its flexibility and extensive options for handling HTTP methods, headers, and data. We’ve covered this in our blog post on cURL with API.

Installation

First, let’s start by checking if you have cURL or Wget installed. Open your terminal and type in curl --version for cURL or wget --version for Wget. If the response indicates that the command was not found, we’ve got detailed wget installation and cURL installation instructions on our blog.

Download a file with cURL

To download files with cURL, you should use the following syntax:

curl -o [file name] [URL]

-o is short for output and allows you to name the downloaded file. To save the downloaded file with the same filename as specified in the URL, use the -O (capitalized). So, if we send a request to https://ip.oxylabs.io/location with -O like this:

curl -O https://ip.oxylabs.io/location

cURL will download the response and call the file location.

Download a file with Wget

Wget downloads files automatically when you send a request. So, if you pass:

wget https://ip.oxylabs.io/location

Wget will download the response and save it as location. If you want to name the file yourself, send -O along with the desired name:

wget -O newfilename https://ip.oxylabs.io/location

Recursive download

One big plus of Wget is its recursive download command -r. It downloads an entire website or directory structure by following links in HTML, CSS, or directory listings. Wget retrieves linked documents layer by layer to a specified maximum depth. This can be useful for mirroring websites or offline browsing. To download files recursively, you can use the following command:

wget -r [URL]

It’s important to note that this command should be passed with care as it can cause problems for both the website and your device.

Proxies

If you want an extra layer of security or localized results, Wget and cURL can be used with proxies

To do that in Wget, you can apply one of the two approaches – command line switches or edit .wgetrc configuration file. We’ve explained how to do it in great detail in our blog post on Wget with proxy, so we invite you to check that out.

Similarly, you can set up a proxy with cURL with a command line argument or using environment variables. This is also covered in-depth in an article on cURL with proxy.

Advanced features

Authentification

If you need to authenticate requests, which can be the case when working with FTP files, use the following Wget command:

wget --user=username --password=password [URL]

Or the this cURL command:

curl -u username:password -O [URL]

Rate limiting

If you want to limit the rate at which the file is downloaded, you can implement:

wget --limit-rate=100k [URL]

The rate will be expressed in bytes by default but you can also specify kilobytes (k) or megabytes (m). 

To turn this off, simply use -q:

wget -q --limit-rate=100k [URL]

Similarly, cURL can be instructed to limit the rate using:

curl [URL] --limit-rate 200K

Like with Wget, use K, M, or G to indicate kilobytes, megabytes, or gigabytes.

Failures and retries

If your download fails, both tools allow you to retry.

Let’s say you’re working with unstable connections. cURL only makes one attempt to perform a transfer but you can use ––retry and ––retry –delay to try the download again as well as add a delay between these attempts:

curl --retry 5 --retry-delay 5 -O[URL]

Here, we’ve set 5 retries with 5 seconds of delay between them.

Similarly, you can use the following command for Wget:

wget -c --tries=60 --read-timeout=20 [URL]

Note, that Wget will retry the command 20 times by default, so you should set a number higher than that or 0 for an infinite number of retries. Note that you’ll need to stop the process manually.