The curl command is a versatile tool for working with web sites and web applications
If you haven’t yet discovered the versatile curl, you might just be surprised by how clever a tool it is. curl, or cURL, is a lot more than a drop-in replacement for wget. Although either tool can be used to run a quick test of a web site, they really have different missions in the web universe. The wget command is used to grab pages from a web site — either to test that they are available or to download them — and can also be used to recursively download an entire site. curl, on the other hand, downloads pages just fine, but it can upload files and post data to web sites just as easily and it can converse with web sites using a wide range of protocols that is likely to surprise you — including DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP. You can also use curl to interact with web-based APIs as REST, json, and SOAP. It’s also been ported to a large number of operating systems including Solaris, NetBSD, FreeBSD, OpenBSD, Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, and others.
The simplest use of curl is type the command itself followed by the URL you want to check out. Here’s one that seems to have been set up to anticipate your curl tests:
$ curl quiet-waters-1228.herokuapp.com/hello Hello, World! Thank you for cURLing me!
And, if you want to save the content of the page as part of your request, add the -o option along with a file name.
$ curl -o msg http://quiet-waters-1228.herokuapp.com/hello
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 40 0 40 0 0 2191 0 --:--:-- --:--:-- --:--:-- 2222
Looking at the file afterwards, you can see that you’ve captured its content:
$ cat msg Hello, World! Thank you for cURLing me!
The name “curl” stands for “client URL”, though some sysadmins find it easier to remember then name if they think of it as “see URL”. It’s also often written “cURL” to emphasize the “URL” part of the name. It is, after all, a tool that allows you to interact with URLs to get a variety of work tasks done.
The curl command is said to be “powered by libCurl” for its transfer abilities. If you check your system and find curl is installed, you will also find its library files.
$ which curl /usr/bin/curl $ sudo find / -name libcurl* -print /usr/share/licenses/libcurl-7.40.0 /usr/lib64/libcurl.so.4.3.0 /usr/lib64/libcurl.so.4
Testing a web site using curl is as easy as testing one using wget. You just use a command that basically says “show me the web page at this address and it makes a GET request and shows you the page in all of its text-based glory. Maybe that isn’t how everyone wants to see web pages, but being able to test a site from the command line brings a certain joy to most of our nerdy little hearts. And, when you see some of the additional things that you can do with this versatile tool, like querying web services and applications, you might decide this is a tool you want to keep in your little arsenal of clever tricks.
In the simplest kind of test, you might just want to see if your web site up and displaying your home page. A curl command like this one will fetch the page and display the corresponding HTML code (assuming it’s an HTML file).
$ curl http://www.w3schools.com/html/tryit.asp?filename=tryhtml_basic_documentTryit Editor v2.6 ...Result:
Even with the middle of the output omitted, you can see that curl grabbed the header and the content of the web page. To display just the header information and not the entire page, use the -I option. This option tells curl to get/show the document information only. This is also known as a “header test”.
$ curl -I http://www.w3schools.com/html/tryit.asp?filename=tryhtml_basic_document HTTP/1.1 200 OK Content-Encoding: gzip Accept-Ranges: bytes Cache-Control: Public,public Content-Type: text/html Date: Sun, 11 Oct 2015 20:55:43 GMT Expires: Sun, 11 Oct 2015 21:55:43 GMT Last-Modified: Sun, 11 Oct 2015 20:19:38 GMT Server: ECS (dca/2495) X-Cache: HIT X-Powered-By: ASP.NET Content-Length: 2269
If the page you’re asking about doesn’t exist, you’ll see the 404 (Not Found) error as in the example below.
$ curl -I www.google.com/ghost HTTP/1.1 404 Not Found Content-Type: text/html; charset=UTF-8 X-Content-Type-Options: nosniff Date: Sun, 11 Oct 2015 23:24:08 GMT Server: sffe Content-Length: 1566 X-XSS-Protection: 1; mode=block
There’s also a -v (verbose) option that will add even more content to your search results. Obviously, this would be more interesting with a page that exists, but here’s an example.
$ curl -Iv www.google.com/ghost * Trying 173.194.121.52... * Connected to www.google.com (173.194.121.52) port 80 (#0) > HEAD /ghost HTTP/1.1 > User-Agent: curl/7.40.0 > Host: www.google.com > Accept: */* >Of course, when you're using curl to interact with web services, you need to use a command that works with those services. In this example, we're just confirming the connection and asking about the services that are running.
$ curl -X GET https://10.1.2.3:443/myapp/command.submit?cmd=about database-version=6.5.1.65432 server-version=6.5.1.65432 patches=P17In this example, we run a couple queries to get information on a member of the staff and then that individual's manager. Notice that the format (json) is specified. First, the person:
$ curl -X GET 'https://10.1.2.155:443/myapp/command.submit?cmd= findUsers&ad_login=sberry&format=json' { "findUsers":[ {"id":"275","idc_id":"1","supervisor_id":"123","add_state":"0", "remove_state":"0","ad_login":"sberry","business_unit_id":"", "creation_date":"2013-07-23 13:34:13.0","deletion_date":"2015-08 -30 05:00:07.0","department":"IT","email_address":"sberry@boson. org","employee_type":"","exception_count":"","first_name":"Sage", "full_name":"Sage Berry","is_terminated":"True","job_status":"", "last_name":"Berry","location":"","mail_nickname":"sberry","start _date":"2011-08-11 00:00:00.0","termination_date":"2015-07-30 05 :00:07.0","title":"Unix Systems Administrator","unique_id":"CN= Sage Berry,OU=All Users,DC=boson,DC=org","user_id":"1234""}Then the manager:
$ curl -k -X GET 'https://10.1.2.155:443/myapp/command.submit? cmd=findUsers&id=31&format=json' { "findUsers":[ {"id":"123","idc_id":"1","supervisor_id":"71","add_state":"0", "remove_state":"0","ad_login":"gjones","business_unit_id":"", "creation_date":"2012-05-25 13:34:13.0","deletion_date":"", "department":"IT","email_address":"gjones@boson.org","employee_ty pe":"Regular","exception_count":"","first_name":"Gary","full_name ":"Jones, Gary","is_terminated":"False","job_status":"Active", "last_name":"Jones","location":"BOSTON, MA","mail_nickname": "gjones","start_date":"2011-02-28 00:00:00.0","termination_date": "","title":"IT MGR","unique_id":"CN=Gary Jones,OU=All Users,DC= boson,DC=org","user_id":"791""}Your web servers will have to be set up to permit this kind of access and may well require that they be configured with the IP addresses of systems that are allowed to run these queries.
To get a list of the options available, just use the command curl --help. The output shown below is less than a third of what you'll see, but should give you an idea about how many options are available.
$ curl --help Usage: curl [options...]Options: (H) means HTTP/HTTPS only, (F) means FTP only --anyauth Pick "any" authentication method (H) -a, --append Append to target file when uploading (F/SFTP) --basic Use HTTP Basic Authentication (H) --cacert FILE CA certificate to verify peer against (SSL) --capath DIR CA directory to verify peer against (SSL) -E, --cert CERT[:PASSWD] Client certificate file and password (SSL) --cert-type TYPE Certificate file type (DER/PEM/ENG) (SSL) --ciphers LIST SSL ciphers to use (SSL) --compressed Request compressed response (using deflate or gzip) -K, --config FILE Read config from FILE --connect-timeout SECONDS Maximum time allowed for connection -C, --continue-at OFFSET Resumed transfer OFFSET -b, --cookie STRING/FILE Read cookies from STRING/FILE (H) -c, --cookie-jar FILE Write cookies to FILE after operation (H) --create-dirs Create necessary local directory hierarchy --crlf Convert LF to CRLF in upload --crlfile FILE Get a CRL list in PEM format from the given file -d, --data DATA HTTP POST data (H) --data-ascii DATA HTTP POST ASCII data (H) --data-binary DATA HTTP POST binary data (H) --data-urlencode DATA HTTP POST data url encoded (H) --delegation STRING GSS-API delegation permission --digest Use HTTP Digest Authentication (H) --disable-eprt Inhibit using EPRT or LPRT (F) --disable-epsv Inhibit using EPSV (F) --dns-servers DNS server addrs to use: 1.1.1.1;2.2.2.2 --dns-interface Interface to use for DNS requests --dns-ipv4-addr IPv4 address to use for DNS requests, dot notation --dns-ipv6-addr IPv6 address to use for DNS requests, dot notation -D, --dump-header FILE Write the headers to FILE --egd-file FILE EGD socket path for random data (SSL) --engine ENGINE Crypto engine (use "--engine list" for list) (SSL) -f, --fail Fail silently (no output at all) on HTTP errors (H) -F, --form CONTENT Specify HTTP multipart POST data (H) --form-string STRING Specify HTTP multipart POST data (H) --ftp-account DATA Account data string (F) --ftp-alternative-to-user COMMAND String to replace "USER [name]" (F) --ftp-create-dirs Create the remote dirs if not present (F) --ftp-method [MULTICWD/NOCWD/SINGLECWD] Control CWD usage (F) --ftp-pasv Use PASV/EPSV instead of PORT (F) -P, --ftp-port ADR Use PORT with given address instead of PASV (F) --ftp-skip-pasv-ip Skip the IP address for PASV (F) --ftp-pret Send PRET before PASV (for drftpd) (F) --ftp-ssl-ccc Send CCC after authenticating (F) --ftp-ssl-ccc-mode ACTIVE/PASSIVE Set CCC mode (F) --ftp-ssl-control Require SSL/TLS for FTP login, clear for transfer (F) -G, --get Send the -d data with a HTTP GET (H) -g, --globoff Disable URL sequences and ranges using {} and [] -H, --header LINE Pass custom header LINE to server (H) -I, --head Show document info only Clearly, curl is not a tool that you will completely conquer in one sitting. Then again, you probably won't need to learn any more than how to use it with your web services and in your environment. But it's clearly a tool to keep in mind whether you're trying a very simple "does it respond?" test or interacting with your server using some interesting queries.
