Sed Command in Linux/Unix with Examples

Linux 'sed' command stands for stream editor. It is used to edit streams (files) using regular expressions. But this editing is not permanent. It remains only in display, but in actual, file content remains the same.

Primarily, it is used for text substitution; additionally, it can be used for other text manipulation operations like insert, delete, search, and more. The sed command allows us to edit files without opening them. Regular expression support makes it a more powerful text manipulation tool.

Overview of sed command

It is a Unix utility that transforms and parses text with a compact and simple programming language. It was integrated from 1973 to 1974 by Lee E. McMahon of Bell Labs and is present today for almost every operating system. The sed command was based on the scripting aspects of the earlier qed (quick editor) and the ed interactive editor. It was the earliest tool for supporting regular expressions, and remains active for text processing, most importantly with the substitution command. Famous alternative tools include Perl and AWK for "stream editing" and plaintext string manipulation.

Operation Mode of sed Command

The sed command is a text processing line-oriented utility: it line by line reads text from a file or input stream into an internal buffer known as the pattern space. All line reads begin a cycle. The sed command uses one or multiple operations which have been described by a sed script to the pattern space. It operates a programming language using about 25 commands that describe the operations over the text.

For all input lines, the sed command ordinarily results in the pattern space and starts the cycle with the next line again after executing the script. Other end-of-script natures are available from script commands and set options, e.g., N to immediately include the next line into the pattern space, q to quit, d to remove the pattern space, and so on. Hence, a sed script relates to a loop body that iterates from a streamline, in which the loop variable and the loop itself are maintained and implicit by sed.

A sed script can be either described on the command line or read through an isolated file. In the sed script, commands may take an extra address in the context of regular expressions or line numbers. The address decides when the command is executed. For instance, 2d will only execute the d (or delete) command on the 2nd input line, while /^ /d will remove every line starting with a space.
The hold space, an isolated special buffer, may be used by some sed commands for holding an accumulating text between cycles. The command language of sed has two variables only (pattern space and hold space) and GOTO-like branching functionality. However, the language is Turing-complete. Esoteric sed scripts are available for several games like tetris, chess, arkanoid, and sokoban.
The main loop runs for all lines of the input stream, checking the sed script on all lines of the input. The sed script lines are all a pattern-action pair, representing what pattern for matching and which action to implement, which can be copied as a conditional statement. The default actions (print pattern space, copy line to the pattern space), input and output streams, and working variables (hold space and pattern space) are implicit; it's possible to specify terse one-liner programs because of the main loop.

Usage of sed Command

Substitution command
The below example represents the most command and typical use of the sed command, i.e., substitution. The usage was the actual motivation for the sed command:
sed 's/regexp/replacement/g' inputFileName > outputFileName
Other commands of sed
Other ways are also possible for simple processing with some 25 sed commands. For instance, below uses the d option for filtering out lines that only include spaces and the line character end:
sed '/^ *$/d' inputFileName
The above example uses a few of the below regular expression metacharacters:
- The caret (^) is the same as the starting of the line.
- The dollar symbol ($) is the same as the completion of the line.
- The asterisk (*) is the same as the more or zero previous character occurrence.
- The plus symbol (+) is the same as the one or multiple previous character occurrences.
- The question mark (?) is the same as the more or zero previous character occurrence.
- The dot symbol (.) is exactly the same as one character.
sed as a filter
Often, the sed command is used as the filter inside a pipeline under Unix:
$ generateData | sed 's/x/y/g'
The generateData program generates data, and the sed command makes a small change by substituting x in place of y.
File-based sed scripts
Often, it's helpful to put many sed commands, a single command per each line, inside a script file and use the -f flag to execute the commands from the file:
sed -f subst.sed inputFileName > outputFileName

Syntax:

Options:

The following are some command line options of the sed command:

-n, --quiet, --silent: It forcefully allows us to print of pattern space.

-e script, --expression=script: It is used to add the script to the commands to be executed.

-f script-file, --file=script-file: It is used to add the contents of script-file to the commands to be executed.

--follow-symlinks: it is used to follow symlinks when processing in place.

-i[SUFFIX], --in-place[=SUFFIX]: it is used to edit files in place (creates backup if SUFFIX option is supplied).

-l N, --line-length=N: It is used to specify the desired line-wrap length for the `l' command.

--posix: it is used to disable all GNU extensions.

-E, -r, --regexp-extended: It allows us to use the extended regular expressions in the script (for portability use POSIX -E).

-s, --separate: it is used for considering files as separate rather than as a single and continues the long stream.

--sandbox: It is used to operate in sandbox mode.

-u, --unbuffered: It is used for loading the minimal amounts of data from the input files and flushes the output buffers more often.

-z, --null-data: It is used to separate lines by NUL characters.

--help: it is used to display the help manual.

--version: It is used to display version information.

Examples of sed Command

Let's see the following examples:

Applying to the STDIN directory
Global Replacement
Removing a Line
Using the Multiple sed Command
Reading Commands From a File
Replacing Characters
Limiting the sed
Inserting and Appending Text
Modifying Lines
Transformation of Characters
Printing the Line Numbers

Applying to the STDIN directory

The sed command is not just limited to manipulate files; also, we can apply it to the STDIN directory.

echo class7 | sed 's/class/jtp/'  
echo class7 | sed 's/7/10/'  
 cat msg.txt | sed 's/learn/study/'  

The above commands will replace the first text with the second text pattern. Consider the below output:

From the above output, first, we have performed 'sed' command on a string 'class7' where 'class' is changed into 'jtp' and 7 into 10. Then we have performed 'sed' command on a stream 'msg.txt' where 'learn' is converted into 'study.'

Global Replacement

In the earlier example, all 'learn' words were not edited into 'study'. To edit every word, we have to use a global replacement 'g'. It will edit all the specified words in a file or string.

Syntax:

Consider the below examples:

echo class7 class9 | sed 's/class/jtp/g'  
cat msg.txt | sed 's/learn/study/g'  

The above commands will replace all the specified text pattern. Consider the below output:

From the above output, by executing the command "echo class7 class9 | sed 's/class/jtp/g'" all the 'class' is converted into 'jtp' and with command "cat msg.txt | sed 's/learn/study/g'" all the 'learn' was converted into 'study'.

Removing a Line

The 'd' option will let us remove a complete line from a file. We only need to specify a word from that line with 'd' option, and that line will be deleted. But, note that all the lines having that same word will be deleted. It will be executed as:

Consider the below command:

The above command will delete the lines having the word 'jtp'. Consider the below output:

From the above output, by executing the command "cat msg.txt | sed '/jtp/d'" all lines containing the word 'jtp' are deleted.

Using the Multiple sed Command

The '-e' option allows us to execute the multiple sed commands at once. We can perform more than one sed operation by executing the command as:

Consider the below command:

The above command will apply all the specified operations in file 'exm.txt'. Consider the below output:

As we can see from the above output, all the 'red' words are replaced with 'blue,' and all the 'yellow' words are replaced with 'black.' We can also separate commands like this:

sed -e '
>s/red/blue/; 
>s/yellow/black/' exm.txt

The result will be the same as the above command.

Reading Commands From a File

We can save the sed commands in a file and apply them at once in any file. It can be done by specifying the '-f' option as follows:

From the above command, the '<sed file>' is a file that has a sed command list. Consider the below command:

The above command will apply all the specified commands in the 'SedCommand' file on 'exm.txt'. Consider the below output:

From the above output, we have used commands applied to the earlier example. So, the output is the same as the previous example.

Replacing Characters

We can use the exclamation mark (!) as a string delimiter. For example, we want to replace bash shell and replace it with csh shell in the "/etc/passwd". To do so, execute the below command:

We can achieve the same result by executing the below command:

Limiting the sed

The basic use of the sed command process the entire file. But, we can limit the sed command and specify any line. There are two ways to limit the sed command:

A range of lines.
A pattern that matches a specific line.

We can provide a number to specify a line as follows:

The above command will apply the specified operation on the third line. Consider the below output:

From the above output, only the line three is modified.

We can also specify a range of lines. To specify a range of lines, execute the command as follows:

The above command will update the specified text in lines 1 and 3. Consider the below output:

Inserting and Appending Text

The 'i' and 'a' flag is used to insert and append the text on a file. The 'i' flag will add the text before the string, and the 'a' flag is used to add text after the string. Consider the below command:

The above command will insert the text before the text "Another Demo". Consider the below output:

To append text, execute the command as follows:

The above command will append the text. Consider the below output:

Modifying Lines

The 'c' flag is used to modify a specific line. To modify a line, execute the command as follows:

The above command will update the line three. Consider the below output:

We can also use a regular expression to update more than one lines having the same pattern. Consider the below command:

The above command will update all the lines having string 'Apple is'. Consider the below output:

Transformation of Characters

The 'y' flag is used to transform the characters. The transformation of characters cannot be limited to specific occurrences. To transform characters, execute the command as follows:

The above command will transform the characters 'a', 'b', 'c' into 'd', 'e', 'f'. consider the below output: