What is cURL Command and How to Use It?

In today's fast-paced digital environment, managing data transfers and testing web-based applications has become essential. One tool that has stood the test of time for developers and system administrators alike is cURL. In this post, we’ll explore what is cURL, why it’s widely used, and how you can use it effectively in your projects.

What is cURL?

cURL (pronounced as “curl”) is an open-source command-line tool and library that enables data transfer using various network protocols, such as HTTP, HTTPS, FTP, IMAP, and more. It’s often used to interact with APIs, download files, or test server responses. The name "cURL" stands for Client URL, reflecting its purpose of working with URLs to retrieve or send data.

The tool supports a wide range of features, including:

• Data retrieval and submission: cURL allows you to fetch data from a URL (e.g., downloading web pages or files) or submit data (e.g., sending form data or interacting with APIs).

• SSL/TLS security: Supports secure data transfer using SSL/TLS, ensuring encrypted communication over HTTPS, which protects sensitive information from being intercepted. Check out our blog post to learn how to effectively ignore SSL certificate with cURL.

• Proxy support: Can route requests through a proxy server, enabling users to anonymize their IP, bypass geo-restrictions, or test how their applications perform in different locations. See more information on how to use cURL with proxy. 

• Data compression and decompression: Supports handling compressed data formats such as gzip or deflate, allowing faster data transfer and reducing bandwidth usage by sending or receiving compressed content.

• Authentication for restricted resources: Provides options for various authentication methods (e.g., Basic, Bearer, OAuth), enabling secure access to protected resources like APIs, private servers, or login-restricted websites.

Why use cURL?

cURL offers several advantages that make it an indispensable tool:

1. Versatility: It supports multiple protocols, making it a one-stop solution for various data transfer needs.

2. Lightweight: No need for a graphical interface; it runs directly in the terminal.

3. Easy to use: Allows to perform complex tasks, such as sending GET or POST requests, downloading files, or even automating data interactions.

4. Debugging and testing: Developers often use cURL to test RESTful APIs, ensuring that their applications can communicate with external services.

5. Cross-platform: Available on virtually every operating system, including Linux, macOS, and Windows. We have a blog post on using cURL for downloading files in Linux – check it out for more detailed insights.

6. Script-friendly: Its command-line nature makes it easy to integrate into scripts for automation.

How to use cURL?

Let’s dive into some practical examples to understand how to use cURL effectively:

1. Basic GET request

Retrieve the contents of a web page:

curl https://sandbox.oxylabs.io/products/1

This command fetches the HTML content of the specified URL, displaying it directly in the terminal. It is commonly used to quickly view or test website content.

2. Download a file

Save a file from a remote URL:

curl -O https://sandbox.oxylabs.io/assets/action-adventure.svg

The -O option saves any type of file with its original name in the current directory, making it easy to download files without specifying custom names.

3. Send a POST request

Submit data to a server using the POST method:

curl -X POST -d "your data" https://httpbin.org/post

Here, -X POST specifies the request method, while -d sends data in the request body. This is often used to submit form data or interact with APIs requiring POST requests. If a successful POST request provides a redirect URL, you can simply use the -L flag to follow redirects and the -d flag to POST data:

curl -L -d "username=admin&password=12345" https://quotes.toscrape.com/login

4. Set custom headers

Add custom headers, often used when working with APIs:

curl -H "Authorization: Bearer your_token" https://httpbin.org/headers

The -H flag allows you to include custom headers, such as authorization tokens, which are essential for accessing secured endpoints or providing additional context to requests.

5. Download and save with a custom name

Use -o for downloading files with a specific name:

curl -o custom_name.html https://sandbox.oxylabs.io/products/1

Unlike -O, the -o option lets you define a custom filename for the downloaded file, giving more control over how files are saved.

6. Test REST APIs

Fetch JSON data from an API:

curl https://scraping-demo-api-json-server.vercel.app/products

This command retrieves data from the specified API endpoint, often returning JSON. You can enhance readability by piping the output to tools like jq for formatted JSON display. You can download jq for your device from the official website and then run the following curl command:

curl https://scraping-demo-api-json-server.vercel.app/products | jq

Here’s what the JSON output should look like: