The Unix command line provides numerous ways to make our work easier. Here are 15 “tricks” that I use often to make quick work of various tasks
Display the last word in every line of a file
This one is pretty easy if you know the meaning of $NF. I came across this many years ago when I was building scripts entirely in awk.
awk '{print $NF}
awk -F: '{print $NF}'
NF for awk represents the number of fields (e.g., 5), so $NF then represents the value of the rightmost field (e.g., $5).
$ cat numbers
1 2 3 4 5
one two three four five
$ awk '{print $NF}' numbers
5
five
This trick makes it easy for you to print the rightmost word in a file even when each line might contain a different number of words. If you want to print some other word relative to the end of each line, you can still use NF as an anchor for your field selection. In this command, we select the next to last field by using NF-1.
$ awk '{print $(NF-1)}' numbers
4
four
Remove the Nth line line of a file
Removing a particular line from a file is a trick that sed handles easily. Assume numbers is a file with the numbers 1 through 11, one number per line. Using the command shown below, line 7 will disappear. The “7 d” command tells sed to remove the 7th line.
$ sed -i '7 d' numbers $ cat numbers 1 2 3 4 5 6 8 9 10 11
You can remove a sequence of lines by putting a range in place of the 7. The command sed -i ‘7,9 d’ numbers removes lines 7 through 9.
Remove blank lines from a file
There are several ways to remove blank lines from a file. One of the easiest is to use sed. In the command below, we use ^$ to represent a blank line. We do this because ^ represents the start of a line and $ represents the end of a line, so a line with a start and an end with nothing between is blank (as opposed to a line containing a sequence of blanks).
$ cat strings This is the last day of your life thus far Mr. Logic $ sed -i '/^$/d' strings $ cat strings This is the last day of your life thus far Mr. Logic
Using a sed command like the one shown below, however, you can both remove blank lines and create a backup of the original file.
$ perl -i.bak -n -e'print if /S/' strings $ ls -l strings* -rw-r----- 1 shs faculty 53 Jan 4 2015 strings -rw-r----- 1 shs faculty 54 Jan 4 18:36 strings.bak
Notice that, with the single blank line missing, the file is one character smaller than the original.
A grep command, similar to one of the sed commands shown above, also uses the ^$ to select blanks lines and the -v to select anything BUT these lines. This command then displays the text without the blank lines but doesn’t change the original file.
grep -v '^$' myfile
You can also use an awk command like the one shown below. This will also display the text and ignore the blank lines but, like the grep command shown above, it won’t change the original file. This works because $0 represents all of the words on the line (i.e., the entire line). You’d have to redirect the output if you want to store it as a new file.
$ awk '$0' strings This is the last day of your life thus far Mr. Logic
Find symbolic links
Finding symbolic links on a Unix system is easy because the find command has an option for finding files by type. The type for a symbolic link is “l” just like the first letter that you see in a long listing for one of these files.
$ find . -type l -ls 12763168 0 lrwxrwxrwx 1 shs staff 6 Jan 27 2013 ./hlinks/5 -> 5beers 12763423 0 lrwxrwxrwx 1 shs staff 4 Nov 23 2013 ./projects/tmp -> /tmp
Find symbolic links that don’t point to current files
It’s possible on Unix system to create a symbolic link that doesn’t point to an actual file or, of course, to create a symbolic link and then remove the original file. It’s also possible to create a pair of symbolic links that point to each other. To track down any of these oddities, you can use a find command that joins forces with a Perl command.
find . -type l -print | perl -nle '-e || print';
In this command, we’re using find to look for symbolic links and sending the results to a perl command that determines whether the target of the link exists or not — and prints the link’s name if it doesn’t.
Determine the architecture (32 or 64 bit) of your system
This one’s very easy. Use the arch command and you should see something like this:
$ arch i686
This works on Linux anyway. When I run this command on Solaris, I get an answer like “sun4”.
Reverse a string
The rev command reverses a string completely — one letter at a time. You can pipe text to it like this:
$ echo "dlroW ,olleH" | rev Hello, World
Alternately, you can provide the rev command with a file name and it reverses the text completely.
$ rev strings yad tsal eht si sihT raf suht efil ruoy fo cigoL .rM
Count how many time a word appears in a file
This is trickier than it might first appear. For one thing, a word might be part of another word — such as “the” being part of “there”. For another, a word might appear more than once on a line. Will your command count each instance or just count the line as one hit? You can use grep -c “word” filename command, but it ‘s basically doing the same thing as grep word filename. It will only tell you how many lines the word appears on, not how many times the word appears as we see here:
$ cat words the the the the the then there they $ grep -c the words 1
Alternatively, you could do something like this:
#!/bin/bash
count=0
look4=$1
file=$2
for word in `cat $file`
do
if [ $word == "$look4" ]; then
((count++))
fi
done
echo $count
In this script, the for loop has us looking at every word and only increments the count if the word exactly matches what we’re looking for. The problem with scripts is that they might not be generic enough to useful very often. At leat this one lets you pass your word and file name as arguments, though you have to pass them in the correct order.
Replace all instances of a word with another word
For many such tasks, I will use a simple command like this one:
sed s/this/that/g filename
That will often work, but it will change any instance of “this” to “that” whether or not the instances of “this” are words or parts of words such as “thistle”. You can get really complicated with sed commands that are particular about the context in which the words they are out to change appear. Here’s an example which is a bit more complicated. It uses the b (word boundary) marker to ensure we only change “the” to “why” when it’s an entire word.
$ cat words the the the the the then there they $ sed -e 's/btheb/why/g' words why why why why why then there they
Note that the string “the” wasn’t changed when it was part of a longer word.
If you want to change “The” as well as “the”, you can add the “ignore case” (i) option to your sed command.
$ sed -e 's/btheb/why/g' words The why why why why then there they $ sed -e 's/btheb/why/gi' words why why why why why then there they
By the way, the sed command can change all instances of a string on a line of text, just the first, or the second or the third, etc. To change only the second, for example, you would use a 2 (2nd) instead of a g (global) in the last position of you s/this/that/g command — s/this/that/2.
NOTE: The sed command can display the modified text or it can modify the original file. The -i option tells sed to edit the file “in place” — rather than requiring you to redirect the output to another file if you want to save it.
Make a variable available from a subshell
Subshells won’t recognize variables that you set up in the shell unless you export them.
$ cat showme #!/bin/bash echo $something $ something="well done" $ ./showme $ export something $ ./showme well done
Explain why a hard link can’t span file systems
A lot of people seem totally confused about the nature of hard links. I sometimes try to explain them as multiple file system incarnations of single files. But why they can’t reach across file system boundaries is due to their nature. They, like all files on a system look to inodes to find and define the files they represent. And inodes are specific to file systems. It’s the symbolic links that are the oddballs. They can point to files in other file systems because their content is nothing but the file system path to the files they “point” to.
Judge how busy a network interface is
This can be a bit tricky. The best solution is probably to install a tool that does this for you. One thing you can do is look at the output of the ifconfig command. Along with all the addressing information, you’ll see some stats that tell you how many packets have been received and transmitted. That will give you some idea how busy the interface has been, but will likely only tell you what you want to know if you also know how long those statistics have been collected — since the system was last booted.
$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:16:35:69:BD:79
inet addr:192.168.0.11 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::216:35ff:fe69:bd79/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11577802 errors:0 dropped:0 overruns:0 frame:0
TX packets:11656938 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2819912413 (2.6 GiB) TX bytes:3530480976 (3.2 GiB)
Interrupt:217 Memory:fdef0000-fdf00000
This probably isn’t all that useful unless you’re comparing interfaces or the system hasn’t been up for very long.
View swap space available and in use
The swapon command can list your swap partitions and files.
$ swapon -s Filename Type Size Used Priority /dev/cciss/c0d0p3 partition 4192956 41152 -1
On Solaris systems, use the swap -l command.
# swap -l swapfile dev swaplo blocks free /dev/dsk/c0t2d0s1 136,1 16 4177904 4177904?
Determine which process is using a file
The lsof command was nothing short of an amazing when I came across it ten years or so ago. Standing for “list open files”, the lsof command comes in extremely handy when you want information about your devices or to find out what a particular user is accessing at some point in time. And it has a plethora of command options:
SYNOPSIS
lsof [ -?abChlnNOPRstUvVX ] [ -A A ] [ -c c ] [ +|-d d ] [
+|-D D ] [ +|-f [cfgGn] ] [ -F [f] ] [ -g [s] ] [ -i [i] ] [
-k k ] [ +|-L [l] ] [ -m m ] [ +|-M ] [ -o [o] ] [ -p s ] [
+|-r [t] ] [ -S [t] ] [ -T [t] ] [ -u s ] [ +|-w ] [ -- ]
[names]
Without options, the lsof command will list all open files (i.e., files that are being accessed by some process).
server# lsof | more COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sched 0 root cwd VDIR 136,0 2048 2 / init 1 root cwd VDIR 136,0 2048 2 / init 1 root txt VREG 136,0 553268 902896 / (/dev/dsk/c0t2d0s0) init 1 root txt VREG 136,0 5008 167126 / -- libdl.so.1 init 1 root txt VREG 136,0 266140 167125 / -- ld.so.1 init 1 root 0u FIFO 136,0 0t0 45285 / (/dev/dsk/c0t2d0s0) pageout 2 root cwd VDIR 136,0 2048 2 / fsflush 3 root cwd VDIR 136,0 2048 2 / syseventd 61 root cwd VDIR 136,0 2048 2 / syseventd 61 root txt VREG 136,0 23272 179817 / (/dev/dsk/c0t2d0s0) syseventd 61 root txt VREG 136,0 12384 185871 / -- modules/picl_slm.so
You can also look to see what processes are using a particular file.
server :>:# lsof /bin/bash COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME monVMweb 456 root txt VREG 136,0 516392 896720 /./usr/bin/bash monTracke 457 root txt VREG 136,0 516392 896720 /./usr/bin/bash monProxy 616 root txt VREG 136,0 516392 896720 /./usr/bin/bash bash 18172 root txt VREG 136,0 516392 896720 /./usr/bin/bash
With the -i (Internet) option, lsof competes with commands like netstat for reporting on network connections.
server :>:# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rpcbind 178 root 3u IPv4 0x3000015b530 0t0 UDP *:sunrpc (Idle) rpcbind 178 root 4u IPv4 0x3000015b3b0 0t0 UDP *:* (Unbound) rpcbind 178 root 5u IPv4 0x3000015b230 0t0 UDP *:32771 (Idle) rpcbind 178 root 6u IPv4 0x3000015b0b0 0t0 TCP *:sunrpc (LISTEN) rpcbind 178 root 7u IPv4 0x3000015af30 0t0 TCP *:* (IDLE) inetd 206 root 11u IPv4 0x3000015adb0 0t0 UDP *:name (Idle) inetd 206 root 12u IPv4 0x3000015aab0 0t0 TCP *:shell (LISTEN) inetd 206 root 13u IPv6 0x3000015a930 0t0 TCP *:shell (LISTEN) inetd 206 root 14u IPv6 0x3000015a7b0 0t0 TCP *:login (LISTEN) inetd 206 root 15u IPv4 0x3000015a630 0t0 TCP *:exec (LISTEN) inetd 206 root 16u IPv6 0x3000015a4b0 0t0 TCP *:exec (LISTEN) ... sendmail 312 root 4r IPv4 0x30002161848 0t0 UDP *:* (Unbound) sendmail 312 root 6u IPv4 0x300021610c8 0t0 TCP *:smtp (LISTEN) sendmail 312 root 7u IPv6 0x30002160f48 0t0 TCP *:smtp (LISTEN) sendmail 312 root 8u IPv4 0x30002160dc8 0t0 TCP *:submission (LISTEN) java 338 root 5u IPv4 0x300025d6350 0t0 UDP *:3052 (Idle) java 338 root 7u IPv4 0x300025d6650 0t0 TCP *:3052 (LISTEN) java 338 root 8u IPv4 0x300025d64d0 0t0 TCP localhost:32798->localhost:32797 (BOUND) perl 618 root 4u IPv4 0x300025d67d0 0t0 TCP *:1555 (LISTEN) sshd 18167 root 5u IPv4 0x3000aa810e8 0t329305 TCP jserver:22->10.2.230.15:53571 (ESTABLISHED)
Turning off aliases that get on your nerves
One of my favorite Unix tricks is the unalias command that turns off an alias that is getting in your way. I often use this when I don’t want my listings to be in color or I don’t want rm to prompt me for every file that I want to delete. The unalias command allows me to turn off any alias that’s in effect now that I may not want to remove entirely from whatever account I’m using, but is not what I want to deal with right now. You can unalias any command and still have the alias working when you next log in or when you resource your “dot files”. Very handy!
