Pipes — Chaining Commands

What Is a Pipe — the | Symbol

The pipe | is the symbol that passes one command's output straight to the next command as its input.

By processing one command's results with another, you combine small commands to get the job done.

This is the Unix design philosophy of building up processing by connecting single-purpose tools.

echo apple > fruits.txt      # create line 1
echo banana >> fruits.txt    # append
echo cherry >> fruits.txt    # append
cat fruits.txt | wc -l       # count lines -> 3
ls / | wc -l                 # number of items in root

① Create line 1 with echo apple > fruits.txt, then append with echo banana >> fruits.txt and echo cherry >> fruits.txt to make a 3-line material file.

② Pass the output of cat fruits.txt to wc -l through a pipe, and confirm the line count is shown.

③ Confirm the number shown matches the number of lines in the material. (Run it correctly and an explanation will appear.)

Anything Can Be Connected

Pipes aren't tied to specific commands.

Any command that produces standard output and any command that accepts standard input can be freely connected.

There are countless combinations: ls | wc -l for a file count, cat file | sort for sorting, and so on.

echo log1 > lines.txt    # create line 1
echo log2 >> lines.txt   # append
echo log3 >> lines.txt   # append
cat lines.txt | wc -l    # 3
ls /etc | wc -l          # number of items in /etc

① Create line 1 with echo x > items.txt, then append with echo y >> items.txt and echo z >> items.txt to make a 3-line material.

② Pass the output of cat items.txt to sort through a pipe, and confirm the lines are shown sorted alphabetically.

③ Next, pass the output of cat items.txt to sort, then run a 3-stage pipe that connects the result to wc -l with one more pipe, and confirm the line count is shown.

Sorting — sort

sort is the command that sorts the lines it receives.

By default it sorts alphabetically (in string order).

It can take the previous command's output through a pipe and pass the sorted result on to the screen or the next command.

echo banana > fruits.txt     # create out of order
echo apple >> fruits.txt
echo cherry >> fruits.txt
cat fruits.txt | sort        # sorted to apple / banana / cherry

① Create line 1 with echo banana > fruits.txt, then append with echo apple >> fruits.txt and echo cherry >> fruits.txt to make 3 lines out of order.

② Pass the output of cat fruits.txt to sort through a pipe, and confirm the lines are shown sorted alphabetically.

Collapsing Duplicates — uniq

uniq is the command that collapses consecutive duplicate lines into one.

uniq looks only at adjacent lines, so it can't collapse identical lines that are far apart.

When you want to reliably reduce duplicates to one line, the classic approach is sort | uniq: gather the same lines next to each other with sort first, then connect to uniq.

echo pear > list.txt         # create with duplicates
echo apple >> list.txt
echo pear >> list.txt
cat list.txt | sort          # apple / pear / pear
cat list.txt | sort | uniq   # apple / pear

sort before uniq

uniq treats only adjacent lines as duplicates.

It can't collapse identical lines that are far apart, so when you want to reduce duplicates to one line, sort first with sort and then connect to uniq.

① Create line 1 with echo pear > list.txt, then append with echo apple >> list.txt and echo pear >> list.txt to make 3 lines containing duplicates.

② Pass the output of cat list.txt to sort through a pipe so identical lines end up next to each other.

③ Then connect that output to uniq, and confirm the adjacent duplicates collapse into one line.