what is curl
avatar

Adomas Sulcas

Sep 21, 2020 5 min read

cURL is everywhere. It’s hidden in nearly every device that transfers any type of data over internet protocols – from cars to Blu-Ray players.

Yet, the internet is chock full of questions on what it is, how to use it and even why should anyone bother with cURL. In this article, we shall uncover the mysterious command line tool, explain how it came to be such a universal asset and show a few examples of its usage.

What does cURL mean?

cURL (client URL) is an open source command line tool and a cross-platform library (libcurl) used to transfer data between servers, distributed to nearly all new operating systems. cURL programming is used almost everywhere where sending or receiving data through internet protocols is required.

cURL supports nearly every internet protocol (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP).

History of cURL

Back in the dark ages of the 90s when everyone still used command line tools, Daniel Sterberg wanted to develop a simple IRC script that would convert currencies for chat room members. In 1997 there weren’t a lot of options to build a foundation for internet protocol data delivery, therefore Httpget, some few hundred lines of code for HTTP-based transfers, became the genesis for cURL. It was first called, in honor of its foundation, HTTPGET 1.0. 

Months later, FTP support had been developed and the name had to be dropped. Now it was being called urlget 2.0. After a few updates, on March 30 1998, the name was changed once again to the now well-known cURL 3.0.

cURL has its older cousin – wget. We won’t get too much into the details, but the main difference between wget and cURL are the download capabilities of each as the former can, for example, recover from broken transfers and continue downloading. 

What does cURL do?

cURL is intended to transfer data through internet protocols. Everything else is outside its scope. It doesn’t even handle the data transferred, only performs the process itself. 

An example of what cURL might be used for is debugging. Using “curl -v https://oxylabs.io” shows the verbose output of one connection request, including details such as user agents, handshake data, ports, etc.

There are far too many possible cURL command options to list and explain. Luckily, “curl –help” is an option that lists out all the possibilities with short explanatory comments. Although without some background on how to use cURL the list won’t be all that helpful.

How to use cURL?

Nearly anyone with a relatively recent operating system can use cURL as it’s shipped as a default in Windows, MacOS, and most Linux distributions. For older systems such as any Windows operating system before 10, cURL might need to be downloaded and installed.

To use cURL, simply open up the terminal and type in “curl”. If everything is as it should be, the output should offer to use “curl –help”. “Help” will, as previously mentioned, list out all the command possibilities. A cURL command can be combined by adding the listed flags and typing in an URL. Flags can be either short (e.g. -o, -L, etc) or long (e.g. –verbose). These flags are differentiated by the use of single or double dashes.

Using cURL

Sending requests

cURL is an incredibly powerful tool for data transfers through internet protocols. Detailing all the possible options on what is cURL used for would be an insurmountable task. Although there are a few common use cases which we will expand upon.
As cURL has initially been developed for HTTP, we can send all the usual requests (POST, GET, PUT, etc). In order to send a POST request to a URL, the -d (or –data) flag is used. Most websites will deny such requests from unauthorized users, therefore we will use a fake API for testing purposes.

curl -d “name=something&value=somethingelse” https://jsonplaceholder.typicode.com/posts/

Sending such a request should return:

{
  "name": "something",
  "value": "somethingelse",
  "id": 101
}

There are a few things at play here:

  • curl begins our command
  • -d is the “data” flag for POST requests
  • Quote marks (“”) begin our content statement. Note that some operating systems will only accept single quotes, others will accept double quotes.
  • Finally, the destination. URL syntax should always be exact as cURL doesn’t automatically follow redirects.

We can also send POST requests in the JSON format, although additional options will have to be provided in order to tell the server that we are sending a JSON. cURL does little interpretation on behalf of the user and would send the default Content-Type header of application/text, so we have to add the header Content-Type: application/json ourselves.

curl -H "Content-Type: application/json" --data "{\"data\":\"some data\"}"  https://jsonplaceholder.typicode.com/posts/

Following redirects

cURL doesn’t automatically follow redirects. We should add an additional flag if we expect it to do so. Let’s take a look at an example:

curl https://google.com

Our browsers handle redirects by themselves, so we might not even notice an issue with such a request. Yet, if we send cURL in to do the work, we will receive a notice that the document has been moved as it tries to connect. To get cURL to follow redirects, we have to add a special flag “-L” (arguments are case sensitive).

curl -L https://google.com

We should now have received the regular answer from Google as cURL follows the redirect from https://google.com to https://www.google.com/.

Connecting through a proxy

cURL can be used to connect to any destination through a proxy. As with any other cURL statement, the URL, syntax and everything else stay the same except for the added flag and its attributes.

curl --proxy proxyaddress:port https://jsonplaceholder.typicode.com/

Entering the proxy and port after “–proxy” will route the connection through the inputted address. Proxies will often require credential details that can be sent through the -U flag.

curl --proxy proxy:port -U “username:password” https://jsonplaceholder.typicode.com/

Some websites will require authentication by themselves before they accept any connection request. A similar flag is used for server authentication: “-u”.

curl -u username:password https://jsonplaceholder.typicode.com/

Conclusion

cURL is a very powerful tool for internet protocol transfers. Mastering its use is definitely a challenge but it can become an irreplaceable tool in any developer’s toolkit. Fully expanding on all the possible cURL use cases is, frankly, impossible.

Want to find out more on interesting topics such as how are CAPTCHAs used to train artificial intelligence? Peruse our blog as we’ve got plenty of posts to offer!

avatar

About Adomas Sulcas

Adomas Sulcas is a Content Manager at Oxylabs. Having grown up in a tech-minded household, he quickly developed an interest in everything IT and Internet related. When he is not nerding out online or immersed in reading, you will find him on an adventure or coming up with wicked business ideas.

Related articles

CTO’s Checklist: 6 Things to Know About Web Scraping

CTO’s Checklist: 6 Things to Know About Web Scraping

Oct 13, 2020

8 min read

CEO’s Guide to Data Extraction

CEO’s Guide to Data Extraction

Oct 13, 2020

7 min read

What Are HTTP Cookies and What Are They Used For?

What Are HTTP Cookies and What Are They Used For?

Oct 13, 2020

6 min read

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.