How to Use cURL With Proxy?

What is cURL?

curl is a command line tool for sending and receiving data using the URL. Let’s look at the simplest example of using curl. Open your terminal or command prompt, type in this curl command, and press Enter:

curl https://www.google.com

The above curl command will get the HTML of the page and print it in the console:

Using environment variables

Another way to use proxy with curl is to set the environment variables http_proxy and https_proxy. 

Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use the _curlrc file.

If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.

• http_proxy – the proxy will be used to access URLs that use the HTTP protocol

• https_proxy – the proxy will be used to access URLs that use the HTTPS protocol

Simply set the variables http_proxy to the HTTP proxy address and https_proxy to set the HTTPS proxy address. Open terminal and run these two commands.

export http_proxy="http://user:pwd@127.0.0.1:1234"
export https_proxy="http://user:pwd@127.0.0.1:1234"

After running these two commands, run curl normally. The commands below will use the http_proxy to connect to the target URL starting with http://, while the https_proxy will be used to connect to the target starting with https://: 

curl "http://ip.oxylabs.io/"
curl "https://ip.oxylabs.io/"

If you see SSL Certificate errors, add -k to ignore these errors.

Another thing to note here is that these curl proxy settings apply system-wide. If this behavior isn’t desired, turn off the global proxy by unsetting these two variables:

unset http_proxy
unset https_proxy

See the next section to set the default proxy only for curl and not system-wide.

Configure cURL to always use proxy

If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.

For Linux and MacOS, open the terminal and navigate to your home directory. If there’s already a .curlrc file, open it. If there’s none, create a new file. Here are the set of commands that can be run:

cd ~
nano .curlrc

In this curl configuration file, add proxy authorization line:

proxy="http://user:pwd@127.0.0.1:1234"

Save the config file. Now, curl with proxy is ready to be used. Simply run curl normally, and it will read the proxy from the .curlrc file.

curl "https://ip.oxylabs.io/"

On Windows, the file is named _curlrc. This file can be placed in the %APPDATA% directory.

To find the exact path of %APPDATA%, open command prompt and run the following command:

echo %APPDATA%

This directory will be something like C:\Users\<your_user>\AppData\Roaming. Now go to this directory, create a new file called _curlrc, and set the proxy by adding this line:

proxy="http://user:pwd@127.0.0.1:1234"

This works exactly the same way in Linux, MacOS, and Windows.

Ignore or override proxy for one request

If the proxy is set globally or by modifying the .curlrc file, this can still be overridden to set another proxy or even bypass it.

To override the proxy for one request, set the new proxy using -x or --proxy switch as usual:

curl --proxy "http://user:pwd@1.0.0.1:8090" "https://ip.oxylabs.io/"

If you want to bypass proxy altogether for a request, you can use the curl noproxy command by passing --noproxy followed by “*”. This instructs curl to not use proxy for all URLs.

curl --noproxy "*" "https://ip.oxylabs.io/"

If you have many curl requests to execute without a proxy but you don’t want to change global proxy settings, the following section will show you exactly how to do that.

Bonus tip – turning proxies off and on quickly

This tip is dedicated only for advanced users. If you do not know what a .bashrc file is, you may skip this section.

You can create an alias in your .bashrc file to set proxies and unset proxies. For example, open the .bashrc file using any editor or terminal and add these lines:

alias proxyon="export http_proxy=' http://user:pwd@127.0.0.1:1234';export https_proxy=' http://user:pwd@127.0.0.1:1234'"
alias proxyoff="unset http_proxy;unset https_proxy"

After adding these lines, save the .bashrc and update the shell to read this .bashrc. To do this, run this command in the terminal:

~/.bashrc

Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands, and then turn off the proxies like this:

proxyon
curl "http://ip.oxylabs.io/"
curl "https://ip.oxylabs.io/"
proxyoff  

cURL socks proxy

If you want to use a SOCKS proxy, the syntax remains the same:

curl -x "socks5://user:pwd@127.0.0.1:1234" "https://ip.oxylabs.io/"

Similarly, socks4://, socks4a://, socks5:// or socks5h:// can be used depending on the SOCKS version your proxy supports.

Alternatively, curl socks proxy can also be set using the switch --socks5 instead of -x. You can follow the same command but use a different switch: username and password can be sent using the --proxy-user switch.

curl --socks5 "127.0.0.1:1234" "https://ip.oxylabs.io/" --proxy-user user:pwd

Again, --socks4, --socks4a, --socks5, or --socks5-hostname can be used, depending on the version.

This tutorial is also available in a video format, so take a look if you prefer visual guides:

Summary

cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with PHP, many web applications use it for web scraping projects, making it a must-have for any web scraper. Furthermore, its versatility extends to various use cases, such as using curl with APIs and even leveraging curl in Python. Be sure to explore these articles for further insight.

Click here and check out a repository on GitHub to find the complete code used in this article. You can also learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.