Using grep and awk in Scripts — Branching on the Result

Calling commands inside a script — pipes and command substitution

The real power of a shell script is that you can combine general commands like `grep`, `awk`, `sed`, `sort`, and `wc` inside it.

Inside a script, you chain commands with a pipe | just as you do at the prompt, and pull their output in with command substitution, $(cmd).

Write n=$(grep -c error log.txt) and the number of matching lines from grep -c goes into the variable n.

You can also put an aggregation result into a variable, as in total=$(awk '{s+=$1} END{print s}' f).

Once a command's result is in a variable, you can use that value to change how the script behaves.

printf 'INFO ok\nERROR disk\nINFO ok\nERROR cpu\n' > sys.log   # create the sample file
vi count.sh   # open in the editor, write the following, and save with :wq
#   #!/bin/sh
#   n=$(grep -c ERROR sys.log)
#   echo "ERROR count: $n"
chmod +x count.sh
./count.sh                                                      # ERROR count: 2

Build a scanner that counts the error lines in an access-log-style file with grep -c, then prints a warning when there are 2 or more and OK otherwise.

① Run printf 'INFO login\nERROR timeout\nINFO logout\nERROR 500\nINFO ping\n' > access.log to create the sample log file.

② Open scan.sh with vi scan.sh and press i to enter insert mode. Turn the count into a variable with n=$(grep -c ERROR access.log), then print either WARN: $n errors found or OK using if [ "$n" -ge 2 ]; then … else … fi (put #!/bin/sh on the first line). When done, press Esc → :wq to save.

③ Add execute permission with chmod +x scan.sh, run ./scan.sh, and confirm the warning comes out.

④ If you are unsure what to write, you can copy the body from the answer panel and paste it into vi's insert mode.

(Run it correctly and the explanation will appear.)

Total it up, then act on the result — awk and branching

Beyond counts, you can also sum the numbers and decide based on the total.

total=$(awk '{s+=$1} END{print s}' f) adds up column 1 of every line and puts the sum into the variable total.

When you want the number of occurrences per word, sort first with sort f | uniq -c and count, then pull that result in with $(...).

Compare the sum against a threshold, as in if [ "$total" -gt 100 ]; then …, and you can make decisions such as "warn once the total goes over a limit".

Bundle the totalling and the branching into one script and you get a practical tool that reads data and decides what state things are in.

printf '40\n30\n50\n' > sales.txt   # single-column numeric data
vi sum.sh   # open in the editor, write the following, and save with :wq
#   #!/bin/sh
#   total=$(awk '{s+=$1} END{print s}' sales.txt)
#   if [ "$total" -gt 100 ]; then
#     echo "total $total: goal met"
#   else
#     echo "total $total: not met"
#   fi
chmod +x sum.sh
./sum.sh                              # total 120: goal met

printf 'a\na\nb\n' > tags.txt   # sample file with repeated words
sort tags.txt | uniq -c          # sort first, then count
# output:
#   2 a
#   1 b

Build a script that sums numeric data with awk and changes its result depending on whether it goes over a threshold.

① Run printf '12\n8\n5\n3\n' > errors.txt to create single-column numeric data.

② Open sum.sh with vi sum.sh and press i to enter insert mode. Turn the sum into a variable with total=$(awk '{s+=$1} END{print s}' errors.txt), then print either needs action: total $total or within limit: total $total using if [ "$total" -gt 20 ]; then … else … fi (put #!/bin/sh on the first line). When done, press Esc → :wq to save.

③ Add execute permission with chmod +x sum.sh, run ./sum.sh, and confirm you get the result branched on the total.

④ If you are unsure what to write, you can copy the body from the answer panel and paste it into vi's insert mode.

Routing output and errors — > and 2>

Sometimes you want a script's result kept in a file instead of on screen.

./scan.sh > report.txt writes the normal output (standard output) into report.txt.

Send error messages to standard error with echo "..." 1>&2, and ./scan.sh > report.txt 2> error.log lets you keep the good results and the errors in separate files.

vi job.sh   # open in the editor, write the following, and save with :wq
#   #!/bin/sh
#   echo "processing started"          # standard output
#   echo "note: input was empty" 1>&2   # standard error
chmod +x job.sh
./job.sh > out.txt 2> err.txt        # send output and errors to separate files
cat out.txt                          # processing started
cat err.txt                          # note: input was empty

Build a script that routes a normal message and an error message into separate files.

① Open job.sh with vi job.sh and press i to enter insert mode. Send echo "done" to standard output and echo "warning" 1>&2 to standard error (put #!/bin/sh on the first line). When done, press Esc → :wq to save.

② Add execute permission with chmod +x job.sh, then split output and errors into separate files with ./job.sh > out.txt 2> err.txt.

③ Run cat out.txt and cat err.txt, and confirm the normal message and the error message each landed in their own file.

④ If you are unsure what to write, you can copy the body from the answer panel and paste it into vi's insert mode.