Pipeline in Shell

The content of this article requires the reader to know some of the syntax and role of the shell, the purpose of the shell, and some basic usage.

To learn shell scripting, it is important to understand the concept of pipeline and the concept of input and output of command. Only by mastering pipeline can we write better shell scripts, and this chapter introduces pipeline in detail.

The command in the shell can accept some input and then produce some output, similar to the function expression y = f(x) in mathematics, input the parameter x and get the result y, the command can be regarded as a functional equation.

Standard inputs, outputs and errors

Every program touches three special files (everything is a file in linux): stdin, stdout and stderr

• stdin: abbreviation for standard input, meaning standard input, from which most programs read their input, represented by the number 0
• stdout: abbreviation for standard output, meaning standard output, most programs write output information to this file, represented by the number 1
• stderr: abbreviation for standard error, meaning standard error, most programs need to write error information to this file, indicated by the number 2

The modifier most is used above to mean that not all programs will read and output information according to the above specification, as any program is free to choose where to read input from and write output information to.

These three special files are stored in the dev/ (dev means device) folder.

1
2
3
4

$ ls -al /dev/std*
lrwxrwxrwx 1 root root 15 Nov 26 20:25 /dev/stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Nov 26 20:25 /dev/stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Nov 26 20:25 /dev/stdout -> /proc/self/fd/1

Each time you run a program in the shell, the shell associates the keyboard with the program’s standard input and connects the standard output and standard errors with the terminal display.

From the above diagram we can see the flow of data, keyboard -> stdin file -> program -> stdout/stderr -> screen, where the flow of data is essentially a pipeline.

The pipe (|) operator

The | operator can be used to redirect the output of one command to the input of another command, i.e., stdout of one command is used as stdin of another command.

Hands-on practice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# View the contents of the file.txt file
$ cat file.txt
File
Edit
Selection
view
go
Run
Terminal
help
go
view

# Sort and de-duplicate the contents of a file
$ cat file.txt | sort | uniq
Edit
File
Run
Selection
Terminal
go
help
view

Re-represent the data flow of cat test.txt | sort | uniq as a graph.

Input redirection <

< can redirect a program’s standard input to a file, e.g. rev < /dev/stdin, i.e. the rev command will read input from standard input, so the rev < /dev/stdin carriage return is equivalent to the rev direct carriage return command.

Output redirection to files > and >>

• > Writes the output of a command to a file and overrides (overrides) the contents of the file.
• >> appends (appends) the output of a command to the end of a file, without deleting the original contents of the file.

1
2
3
4
5
6
7

$ echo "Hello Shell" > test.txt
$ cat test.txt
Hello Shell
$ echo "Hello Pipeline" >> test.txt
$ cat test.txt
Hello Shell
Hello Pipeline

2> and 2>> indicate that the standard error is redirected to a file, and 2 is the file descriptor of the standard error.

Standard error

Let’s try to create two identical directories in the same path and see what happens.

1
2
3

$ mkdir js
$ mkdir js
mkdir: cannot create directory 'js': File exists

You can see that if you create the directory with the same name again, the shell will report an error, and the error message here is a standard error, not a standard output. We can try to use the pipe operator to see if we can redirect the standard error.

The test requires the tr (translate characters) command, which converts characters from lowercase to uppercase, for example.

1
2

$ echo 'Be quiet, this is a library!' | tr '[:lower:]' '[:upper:]'
BE QUIET, THIS IS A LIBRARY!

Now we redirect our error message.

1
2

$ mkdir js | tr '[:lower:]' '[:upper:]'
mkdir: cannot create directory 'js': File exists

As you can see the tr command does not seem to receive any standard input, this is because mkdir js is wrong and the error message is output to the standard error, while | only redirects the standard output.

As mentioned above stdin, stdout, and stderr all have a file descriptor of 0, 1, and 2 respectively.

So how do you handle standard error messages? Here are some general practices.

• 2>&1: 2> is used for standard error redirection to a file, while stdin, stdout, stderr are special files, so here it means standard error (2) redirection to standard output (1).
• 2>. /errors.txt: redirects the standard error message to a file, which will overwrite the original content of the file.
• 2>/dev/null: redirects standard error output to /dev/null.
• 2>>. /errors.txt: Adds standard error messages to a file.
• >output.txt 2>&1: redirects both standard errors and standard output to the output.txt file.

Let’s try to redirect the error message of the created folder to the error.txt file.

1
2
3

$ mkdir js 2>error.txt
$ cat error.txt
mkdir: cannot create directory 'js': File exists

2>error.txt indicates a redirect error message, so there is no error message on the screen when mkdir js is executed for the first time.

A great shell learning tutorial is recommended: Effective Shell.

Ref

• https://pengfeixc.com/blogs/developer-handbook/pipelines-in-shell