Redirection in the Linux Shell

I often see other people’s shell scripts followed by a 2>&1, and have never bothered to look deeper, so today this topic will be the starting point to expand and learn about the topic of redirection in the linux shell.

Special things

First, let’s look at some special things in linux to lay the groundwork for the rest of the content.

Special files

• /dev/null empty, you can import garbage content into it and it will disappear
• /dev/zero zero, you can read endless zeros from it
• /dev/urandom random numbers, you can read endless random numbers from it
• /dev/stdin standard input stream
• /dev/stdout Standard output stream
• /dev/stderr standard error output stream

We can see that the last three files are actually links to the kernel’s file descriptors 0\1\2.

1
2
3

lrwxrwxrwx 1 root root         15 Mar 24 16:20 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root         15 Mar 24 16:20 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root         15 Mar 24 16:20 stdout -> /proc/self/fd/1

Special file descriptors

There are three special file descriptors (File descriptor or fd) in the Linux shell:

• fd 0 is the standard input: stdin
• fd 1 is the standard output: stdout
• fd 2 is the standard error output: stderr

With these three special file descriptors we can control the input and output streams.

Redirects

We often come across the symbol >, called a redirect, but there is another symbol >> that has a similar function, with a small difference between them.

• >> is an overwrite method.
• >> is an append method.

The following contents will all be > as an example, >> has no other difference except that the content is appended, so I won’t repeat it.

Using redirects

Redirecting to a file

Let’s first look at the most basic use of redirection, where we redirect the output of the echo command to a file.

1

echo "hello" > a.txt

The result of the execution is as follows.

1
2
3

root@ubuntu:~# echo "hello" > a.txt
root@ubuntu:~# cat a.txt
hello

Here it is redirecting stdout to the file a.txt, which is equivalent to the following command.

1

echo "hello" 1> a.txt

The result of the execution is as follows.

1
2
3
4

root@ubuntu:~# rm a.txt
root@ubuntu:~# echo "hello" 1> a.txt
root@ubuntu:~# cat a.txt
hello

Here we see that the redirection symbol > defaults to redirecting stdout, which is fd 1, to somewhere else.

If we want to redirect the standard error output stderr, we just need to change the file descriptor 1 in the above command to the standard error output file descriptor 2.

Redirecting to file descriptors

In some cases stderr is programmatically controlled to write to the error log, so if we want to display the error on the screen while the command is running, we need to redirect the error output to the standard output stream.

Let’s try it first, here we don’t find a suitable command, so we take the ls command to look at a directory that doesn’t exist, which will generate an error output.

Here the error is output to the screen by default, but I haven’t found a better program yet, so let’s assume it won’t be output to the screen.

1

ls error 2>1

Our guess here is to redirect stderr to stdout , so we write 2>1 , let’s see if that works?

1
2
3
4
5
6

root@ubuntu:~# ls error 2>1
root@ubuntu:~#
root@ubuntu:~# ls
1
root@ubuntu:~# cat 1
ls: cannot access 'error': No such file or directory

We see that instead of output, a file 1 is generated in the current directory, which means that if we just write >1 it will be treated as a redirect to the file 1.

At this point, our & comes into play.

>& is the syntax for redirecting a stream to a file descriptor, so just now we should specify that we want to redirect to fd 1, which is &1.

1

ls error 2>&1

Execution results.

1
2

root@ubuntu:~# ls error 2>&1
ls: cannot access 'error': No such file or directory

Here we can play independently.

Redirects the standard output to the standard error output.

1

echo "hello" 1>&2

or

1

echo "hello" >&2

We can even play a little more complicated.

1
2

root@ubuntu:~# (echo "hello" >&9) 9>&2 2>&1
hello

Here the file descriptor 9 will be generated automatically, but removing the brackets will prompt an error.

1
2

root@ubuntu:~# echo "hello" >&9 9>&2 2>&1
bash: 9: Bad file descriptor

In bash >4.0, a new redirect syntax has been introduced.

1
2
3

$ ls -ld /tmp /tnt 2> >(sed 's/^/E: /') > >(sed 's/^/O: /')
O: drwxrwxrwt 17 root root 28672 Nov  5 23:00 /tmp
E: ls: cannot access /tnt: No such file or directory

I haven’t learned this way of writing yet, I’ll update it later when I learn it

Formatting output

A little bit of high-end usage.

For formatting output, redirecting the standard output and error output streams to different processes, and finally aggregating them.

1
2
3

root@ubuntu:~# ((ls -ld /tmp /tnt |sed 's/^/O: /' >&9 ) 2>&1 |sed 's/^/E: /') 9>&1| cat -n
     1  O: drwxrwxrwt 1 root root 4096 Mar 22 18:59 /tmp
     2  E: ls: cannot access '/tnt': No such file or directory

A new version of the grammar with the same effect.

1
2
3

root@ubuntu:~# cat -n <(ls -ld /tmp /tnt 2> >(sed 's/^/E: /') > >(sed 's/^/O: /'))
     1  O: drwxrwxrwt 1 root root 4096 Mar 22 18:59 /tmp
     2  E: ls: cannot access '/tnt': No such file or directory

Merge files

Merge output files m and n: n >& m.

Merge input files m and n: n <& m.

Input boundaries

Take as input the content between the start tag and the end tag: << tag.

Example.

1
2
3
4
5
6

root@ubuntu:~# wc -l << EOF
    document line 1
    document line 2
    document line 3
EOF
3 //表明收到3行输入

Its function is to pass the content (document) between two EOFs as input to command.

Note.

• The delimiter at the end must be written in top space, with no characters before it and no characters after it, including spaces and tab indents.
• Spaces before and after the starting delimiter are ignored.

About overwriting

If we set bash with set -o noclobber, then bash will not overwrite any files that already exist, but we can bypass this restriction with >|.

Let’s look at the default case first

1
2
3
4
5
6
7

root@ubuntu:~# testfile=$(mktemp /tmp/testNoClobberDate-XXXXXX)
root@ubuntu:~# date > $testfile ; cat $testfile
Tue 24 Mar 2020 05:05:53 PM CST
root@ubuntu:~# date > $testfile ; cat $testfile
Tue 24 Mar 2020 05:05:56 PM CST
root@ubuntu:~# date > $testfile ; cat $testfile
Tue 24 Mar 2020 05:06:13 PM CST

As expected, each redirect overwrites the original file

We set the noclobber flag below

set -o noclobber

Then repeat the above to try it out.

1
2
3
4
5
6

root@ubuntu:~# date > $testfile ; cat $testfile
bash: /tmp/testNoClobberDate-yKVkaY: cannot overwrite existing file
Tue 24 Mar 2020 05:06:13 PM CST
root@ubuntu:~# date > $testfile ; cat $testfile
bash: /tmp/testNoClobberDate-yKVkaY: cannot overwrite existing file
Tue 24 Mar 2020 05:06:13 PM CST

We saw the bash prompt that it cannot overwrite an already existing file, and the actual result is the same.

How can we do the bypass? Let’s try >| instead of >.

1
2
3
4

root@ubuntu:~# date >| $testfile ; cat $testfile
Tue 24 Mar 2020 05:10:45 PM CST
root@ubuntu:~# date >| $testfile ; cat $testfile
Tue 24 Mar 2020 05:10:49 PM CST

We find that we can overwrite the files that already exist at this point, and we look at the current settings.

1
2

root@ubuntu:~# set -o | grep noclobber
noclobber       on

noclobber is indeed turned on, so >| does bypass this restriction.

Use set +o noclobber to turn off this restriction and prevent it from affecting our later use of

1
2
3
4

root@ubuntu:~# set +o noclobber
root@ubuntu:~# set -o | grep noclobber
noclobber       off
root@ubuntu:~# rm $testfile

Other tidbits of knowledge

Redirecting to one place

If we want to redirect stdout and stderr to the same place, how should we write it?

Which of the following two is the right one?

1. ls -ld /tmp /tnt 2>&1 1>a.txt
2. ls -ld /tmp /tnt 1>b.txt 2>&1

Verify it.

The first way to write

1
2
3
4

root@ubuntu:~# ls -ld /tmp /tnt 2>&1 1>a.txt
ls: cannot access '/tnt': No such file or directory
root@ubuntu:~# cat a.txt
drwxrwxrwt 1 root root 4096 Mar 24 17:15 /tmp

The second way to write

1
2
3
4

root@ubuntu:~# ls -ld /tmp /tnt 1>b.txt 2>&1
root@ubuntu:~# cat b.txt
ls: cannot access '/tnt': No such file or directory
drwxrwxrwt 1 root root 4096 Mar 24 17:15 /tmp

We can see that the second way of writing is correct.

Similarly, the following way of writing is also correct.

1

ls -ld /tmp /tnt 2>b.txt 1>&2

Nested

what happens if we redirect stderr to stdout , and at the same time redirect stdout to stderr?

Will such nesting cause a crash?

Try it.

1
2
3

root@ubuntu:~# ls -ld /tmp /tnt 2>&1 1>&2  | sed -e s/^/++/
++ls: cannot access '/tnt': No such file or directory
++drwxrwxrwt 1 root root 4096 Mar 24 17:15 /tmp

We find that it all comes out of the standard output.

And the other way around?

1
2
3

root@ubuntu:~# ls -ld /tmp /tnt 1>&2 2>&1  | sed -e s/^/++/
ls: cannot access '/tnt': No such file or directory
drwxrwxrwt 1 root root 4096 Mar 24 17:15 /tmp

We find that none of it comes out of the standard output, it comes out of the standard error output

That means a>&b b>&a is written in this nested way, and b is the output.

Read more

If you want to understand the functionality, check the official documentation with the following command

man -Len -Pless\ +/^REDIRECTION bash

Reference for this article: stack overflow