Unix: Flexibly moving files with lftp

Moving files between Linux (and other) servers is a lot smarter and more versatile when you discover lftp.

What is lftp?

Some sites -- and even the man page -- claim that "lftp" stands for "sophisticated file transfer program", but I'm still struggling with that one. I might have suggested "limber ftp", "lithe ftp", even "liberal ftp", but someone decided that the "l" could stand for "sophisticated", so who am I to question their logic? A more important question is "How has it escaped my attention until now?". Maybe because I pretty much stopped using ftp commands years ago in favor of scp. Maybe simply because no one brought this tool to my attention -- until now. But lftp is hardly new (it initially showed up in 1996, but wasn't called lftp until 1997) and its flexibility takes it way beyond what anyone would expect of an ftp program. Yes, lftp has a lot of interesting features and just got shoved in front of my face when a vendor I deal with changed their download options to be more secure. Using lftp from a Unix command line was one of the choices presented to me going forward. So exactly what is lftp and why it is so much worth using and understanding? For one thing, lftp doesn't just support ftp. It also supports many access methods including FTP, HTTP, FISH, SFTP, HTTPS and FTPS. And it can download an entire website if you do a recursive get with the tool's mirror command. The command also has a great handle on reliability. Non-fatal errors are ignored and operations restarted as needed. My first impression was that it's a solid tool and performs very well. With lftp, you can launch commands in parallel by putting them in the background much as we do on the Unix/Linux command line using & and you can group commands within parentheses and run the whole group in background. To move a running job into the background, just use ^z. You can then bring it back into the foreground with fg. How cool is that? And a jobs command is available to list running jobs. If some of your jobs haven't completed when you exit lftp, lftp will put itself in nohup (no hangup) mode in the background to complete them. That's a lot to get from anything that calls itself an ftp program.

Configuring lftp

What I needed to do was run a series of set commands to prepare my lftp session to work with the server that I needed to grab files from. Fortunately, I quickly determined that I didn't need to type these commands every time I went to the site for updates. Instead, I could drop the needed commands into either a global configuration file (good choice if other accounts on the server would need the same setup) or into a dot file in my personal account (~/.lftprc or ~/.lftp/rc). My ~/.lftprc file now looks like this:

set ftp:ssl-protect-data true
set ftps:initial-prot
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ssl:verify-certificate off

This gives me encrypted connections to the server when I type lftp ftp.targetsite.com.

Using lftp

You can launch lftp by typing just lftp and then using an open command to take you to your target site or you can provide the target's name on the same line as lftp like I did.

$ lftp ftp.targetsite.com


$ lftp
lftp :~> open ftp.targetsite.com

Once you log in, you can list files and use get, mget, put etc. commands as needed:

$ lftp
lftp :~> open ftp.targetsite.com
lftp ftp.targetsite.com:~> user sandrahs NotMyRealPassword
lftp sandrahs@ftp.targetsite.com:~> ls

If you ask for help, you're likely to see something like this, showing all the commands that are available to you:

lftp :~> help
    !<shell-command>                    (commands)
    alias [<name> [<value>]]            anon
    bookmark [SUBCMD]                   cache [SUBCMD]
    cat [-b] <files>                    cd <rdir>
    chmod [OPTS] mode file...           close [-a]
    [re]cls [opts] [path/][pattern]     debug [<level>|off] [-o <file>]
    du [options] <dirs>                 exit [<code>|bg]
    get [OPTS] <rfile> [-o <lfile>]     glob [OPTS] <cmd> <args>
    help [<cmd>]                        jobs [-v]
    kill all|<job_no>                   lcd <ldir>
    lftp [OPTS] <site>                  ls [<args>]
    mget [OPTS] <files>                 mirror [OPTS] [remote [local]]
    mkdir [-p] <dirs>                   module name [args]
    more <files>                        mput [OPTS] <files>
    mrm <files>                         mv <file1> <file2>
    [re]nlist [<args>]                  open [OPTS] <site>
    pget [OPTS] <rfile> [-o <lfile>]    put [OPTS] <lfile> [-o <rfile>]
    pwd [-p]                            queue [OPTS] [<cmd>]
    quote <cmd>                         repeat [OPTS] [delay] [command]
    rm [-r] [-f] <files>                rmdir [-f] <dirs>
    scache [<session_no>]               set [OPT] [<var> [<val>]]
    site <site_cmd>                     source <file>
    user <user|URL> [<pass>]            version
    wait [<jobno>]                      zcat <files>
    zmore <files>                       history -w file|-r file|-c|-l [cnt]

If you want to download an entire directory, you can use the mirror command. Typing mirror images, for example, will copy the images folder from the remote system and create one in your local directory, complete with images and subdirectories if they exist. Use a second argument if you want to give your local folder a different name. Once you've logged off, you can look through your .lftp directory and peruse the history (cwd_history) file that was left there showing the sites you visited and the rl_history file to see what commands you issued and files you downloaded.

cd .lftp
ls -l
-rw------- 1 sandrahs dweebs    43 Sep  1 17:43 cwd_history
-rw------- 1 sandrahs dweebs   252 Sep  1 17:43 rl_history
-rw------- 1 sandrahs dweebs 18554 Sep  1 17:02 transfer_log
$ cat cwd_history
ftp://sandrahs@ftp.targetsite.com   1378071811:/
$ cat rl_history
lftp ftp.targetsite.com
user sandrahs NotMyRealPassword
mirror images
$ cat transfer_log
2013-09-01 17:00:43 ftp://sandrahs:NotMyRealPassword@ftp.targetsite.com
/images/mugshot.jpg 0-629374 557.7K/s
2013-09-01 17:00:44 ftp://sandrahs:NotMyRealPassword@ftp.targetsite.com
/images/myhome.jpg 0-87454 893.1K/s

Moving files around deftly, reliably and securely has just gotten easier. I'm impressed with this limber, lithe, liberal and, ok, sophisticated ftp tool.

Read more of Sandra Henry-Stocker's Unix as a Second Language blog and follow the latest IT news at ITworld, Twitter and Facebook.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: 10 new UI features coming to Windows 10