Catalog My learning Blog
Linux Command Line Tutorial for Developers · Phase 2 · Lesson 8 · 3 min read

Linux Pipes and Redirection Explained

On this page
  1. stdin, stdout and stderr
  2. Redirection with >, >> and <
  3. Why the order of 2>&1 matters
  4. Pipes: connecting commands
  5. Exit status in pipelines and pipefail
  6. tee, xargs and other pipeline tools
  7. Check yourself

stdin, stdout and stderr

Every process starts with three open file descriptors:

FDNameDefault
0stdinKeyboard
1stdoutTerminal: normal output
2stderrTerminal: errors and diagnostics

A file descriptor is just a numbered handle the process uses to read or write something: a file, a terminal, a pipe. Picture every program born with three hoses attached: one feeding input in (0), one for normal output (1) and one for complaints (2). Redirection is reconnecting a hose somewhere else.

Keeping errors on a separate stream means you can save results to a file while still seeing errors, or the reverse.

Redirection with >, >> and <

Shell
ls /etc/hostname /nope > out.txt 2> err.txt
YAML
out.txt: /etc/hostname
err.txt: ls: cannot access '/nope': No such file or directory
SyntaxEffect
> filestdout to file (overwrite)
>> filestdout to file (append)
2> filestderr to file
> file 2>&1both to file
&> fileboth to file (bash shorthand)
< filestdin from file
> /dev/nulldiscard

Why the order of 2>&1 matters

2>&1 means "make stderr point where stdout points right now". Redirections apply left to right:

Shell
ls /etc/hostname /nope > all.txt 2>&1     # stdout to file, then stderr to the same place
ls /nope 2>&1 > o2.txt                    # stderr to the terminal (where stdout was), then stdout to file
Code
all.txt lines: 2
ls: cannot access '/nope': No such file or directory     <- printed on the terminal
o2.txt size: 0

The first form captures both lines. The second sent the error to the terminal and left the file empty. When you want everything in a log, write > file 2>&1.

Each process has stdin, stdout and stderr; redirections point them at files, and pipes connect one process's stdout to the next process's stdin.
Each process has stdin, stdout and stderr; redirections point them at files, and pipes connect one process's stdout to the next process's stdin.

Pipes: connecting commands

A pipe connects one program's stdout to the next one's stdin, and all of them run at the same time:

Shell
tail -n 5000 /var/log/nginx/access.log \
  | awk '$9 >= 500 {print $1}' \
  | sort | uniq -c | sort -rn | head -5

Top five client IPs producing server errors in recent traffic. Each tool does one job: filter, extract, sort, count, rank. Days 9 and 10 cover grep, sed and awk.

Only stdout goes through a pipe; to pipe errors too, use 2>&1 | (or |& in bash).

Exit status in pipelines and pipefail

A pipeline's exit status is the last command's, which hides failures earlier in the chain:

Code
false | true; echo "without pipefail: $?"
set -o pipefail
false | true; echo "with pipefail: $?"
Code
without pipefail: 0
with pipefail: 1

In scripts, set -o pipefail (part of set -euo pipefail, Day 15) makes a failure anywhere in a pipeline fail the whole thing. Otherwise, curl ... | tar xz can report success even when the download failed.

tee, xargs and other pipeline tools

  • tee writes to a file and passes output on: make 2>&1 | tee build.log.
  • sudo and redirection: sudo echo x > /etc/file fails, because your shell (not sudo) opens the file. Use echo x | sudo tee /etc/file.
  • Here documents feed multi-line input: cat > config <<'EOF' ... EOF (quoting EOF stops variable expansion).
  • xargs turns lines of input into arguments, with -0 for safe filenames (Day 11).

Check yourself

Redirections are processed left to right. 2>&1 pointed stderr at wherever stdout was at that moment, the terminal, and only then was stdout redirected to the file. The error went to the screen; write > o2.txt 2>&1 to capture both.

A pipeline then fails if any command in it fails, returning the last non-zero status, instead of reporting only the last command's status. Without it, false | true counts as success.

The redirection is performed by your unprivileged shell before sudo runs; use sudo tee to write the file as root.

> truncates the file and writes; >> appends to the end. Using > by mistake on a log or config file erases it.

It reads items from stdin and passes them as arguments to a command, such as deleting a list of files produced by find.