Unix How To: Sed & Awk -- Still friendly after all these years

Even after decades of using Unix on thousands of systems, I find that it's still fun to discover various convolutions of sed and awk commands to perform command line wizardry. There's a lot more to each of these tools than those uses I make of these commands on a routine basis. Let's take a look at some one-liners you might not yet have tried.

One of my long-standing "tricks" for awk is using $NF to print the last field on every line. Since NF represents the number of fields on a line (e.g., 6), $NF represents the value of that last field (e.g., $6). Printing the last field of every line, therefore, might look like this:

$ awk '{print $NF}' myfile

or this:

$ awk -F: '{print $NF}' /etc/passwd

depending on whether the file is white space delimited or not.

You can get the length of every line in a text file by using awk's length command. For example, this command prints the length of every line in the /etc/motd file:

$ awk '{print length($0)}' /etc/motd

You can compute the square root of every fourth field in a file of numbers like so:

$ awk '{print sqrt($4)}' nums

You can display select lines from a chosen file.

$ awk '/nobody/ { print $0 }' /etc/passwd

You can generate an alphabetical listing of your users with an added sort command.

$ awk -F: '{ print $1 }' /etc/passwd | sort

You can count the lines in a file. In this example, we're using the END (only perform this after all lines have been processed) feature of awk to print the line number of the last line in the file.

$ awk 'END { print NR }' myfile

If you want to know how many words are in every line of a file, try this:

$ awk '{print NF}' myfile

You can add line numbers to every line in a file.

$ awk '{print NR, $0}' myfile

Sed, which I often find myself using only for those dead easy text substitutes such as sed "s/this/that/g" myfile also has some more interesting applications.

You can double space a file (appends a newline to every displayed line). Use 'G;G' (don't forget the quotes!) for triple spacing, 'G;G;G' for quadruple and so on.

$ sed G myfile

You can remove double spacing from a file.

$ sed 'n;d'

I've used sed to replace only the first instance of a string or every instance of a string in each line of text, but you can also use it to replace a specific instance -- for example, only the third or only the seventh instance.

$ sed 's/this/that/3'

You can also change multiple strings in a single sed command using semicolons like so:

$ sed 's/this/that/g;s/you/me/g;s/him/her/g' myfile

The sed and awk commands are far more versatile than I had remembered them being and could provide some useful aliases for you to toss into your startup files.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Take IDG’s 2020 IT Salary Survey: You’ll provide important data and have a chance to win $500.