• United States
Unix Dweeb

sed: not just for pipes and one-liners

Aug 26, 20125 mins
Data CenterOpen SourceOperating Systems

Over the years, the primary way that I’ve used sed is to change text on the fly. There are many occassions in which I want to change one string to another the first or every time that it appears in a file or command output.

$ sed 's/2011/2012/' new
$ sed s/tomorrow/today/ msg2
$ sed 's/,/:/g' 

sed is, after all, a stream editor and it does this kind of thing exceptionally well and blindingly fast.

The sed command is, however, also very good for a number of other tasks associated with selecting text from a file. For example, you can select the line in a text file by using a combination of head and tail commands, but this takes a little thought:

$ head -11 myfile | tail -1

Alternately, you can use a sed command and easily pick out the 11th line. Say we have a list of tasks and we want to see just the eleventh task in the list. Use sed like this:

$ sed -n '11 p' tasks
11:  Get the sailboat back in the water

If you want to pick 100 lines from the middle of a file with millions of lines, the difference in how much less time it takes to do the job with sed could be considerable.

You can also do the same thing like this:

$ sed -n 11p tasks
11:  Get the sailboat back in the water

sed can remove a line from a file as well. Once the sailboat is actually back in the water, you might want to remove the 11th line in your tasks list like this (but keep in mind that sed is identifying the line by number not content):

$ sed -i '11 d' tasks


$ sed -i 11d tasks

The -i in this command instructs sed to do its editing in place -- in other words, change the source file. Where the "p" above means "print", the "d" here means "delete". If you want to remove a range of lines rather than a single line, you can specify a range of lines with an expression like 11,13 (lines 11 through 13) or even 11,$ (the eleventh line through to the last line in the file). This command will leave only the first ten lines in your tasks list:

$ sed -i '11,$ d' tasks

The $ in "11,$" stands for the last line in the file.

You can use a similar syntax if you want to display those lines instead of removing them from the file:

$ sed -n '11,$ p' tasks
11:  Get the sailboat back in the water
12:  Drywall the living room
13:  Rebuild the front porch
14:  Donate the llamas to a petting zoo
15:  Take a vacation
16:  Get a new job
17:  Sell the sailboat for a big profit
18:  Hang a photo of the sailboat in the basement of the new home
19:  Live happily ever after

This command will only display the lines. It won't affect the original file. The -n prevents you from seeing all the other lines of the file as well.

If you want to replace task 11 in your tasks list, you can do it in two steps like this:

$ sed -i '11 d' tasks 
$ sed -i'' '11 i 11:  Host a sailboat party on the Bay' tasks

That may look a little more tricky than it is. Keep in mind that the second 11 is part of the text we're inserting. Read this as "at line 11, insert 11..." and so on. You can then check to see if your changes worked as intended with this command:

$ sed -n 11p tasks
11:  Host a sailboat party on the Bay

The sed command can also allow you to make global in-place edits. For example, you could change every instance of "sailboat" to "speedboat" if you decided you were tired of moving at under 10 knots.

$ sed -i s/sailboat/speedboat/ tasks

You can then verify your changes to your tasks list with a sed command that acts a lot like grep:

$ sed -n '/boat/p' tasks
11:  Host a speedboat party on the Bay
18:  Sell the speedboat for a big profile
19:  Hang a picture of the speedboat in the basement

The grep equivalent (grep boat tasks) is, of course, even easier, but it's nice to see that sed has this much flexibility. In fact, sed is not just a tool for one-liners, but can be used for standalone scripts like these:

For double spacing text in a file:

#!/bin/sed -f
#dbl: double-space a text file

For changing one string to another in a specified range of lines (in this case, Linux or linux to "Unix or Linux" in the first three lines:

#!/bin/sed -f

      s/[Ll]inux/Unix or Linux/g

It's been a long time since my first O'Reilly "sed & awk" book, but I still find that it's fun to see what I can do with these commands.

Unix Dweeb

Sandra Henry-Stocker has been administering Unix systems for more than 30 years. She describes herself as "USL" (Unix as a second language) but remembers enough English to write books and buy groceries. She lives in the mountains in Virginia where, when not working with or writing about Unix, she's chasing the bears away from her bird feeders.

The opinions expressed in this blog are those of Sandra Henry-Stocker and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.

More from this author