Stop 02 · Linux & Command LineTopics · 7/11

Text Processing (grep, sed, awk)

In short

A 2 GB log file and a manager asking 'how many errors since lunch?'. You won't open that in Notepad. Three old tools answer it from the terminal in seconds.

Command-line tools for searching (grep), transforming (sed), and extracting/processing (awk) text — indispensable for parsing logs and configuration files quickly from the terminal.

The mascot runs a finger down a thick register with a magnifying glass and a pencil beside it.

Think of it like this

The class teacher and the result register

To find your roll number, the teacher runs a finger down the list: that's grep. To fix a misspelled surname on every page: sed. To pull out only the maths column and total it: awk. Same register, three different jobs.

Watch it happen

grep | awk | sort | uniq on access.log

access.log

grep

awk

sort | uniq

1/4Start with a web log where each line has an IP, a path and a status code like 200 or 500.

The big ideas

What you need to know

grep finds lines

grep -i ignores case, -n shows line numbers, -v shows lines that do NOT match, -r searches folders.

sed edits streams

sed 's/old/new/g' replaces text. Add -i to edit the file in place, after testing without it.

awk works on columns

awk splits each line into $1, $2 and so on. It can filter, print and add up columns.

Pipes glue them

The | sends one tool's output into the next. Small steps chained together answer big questions.

See it in code

Walk through the code

bash

1# Count 500 errors in the log2grep -c '" 500 ' access.log3 4# Top 5 IPs causing 500s5grep '" 500 ' access.log | awk '{print $1}' \6  | sort | uniq -c | sort -rn | head -57 8# Preview a config change, then apply it9sed 's/listen 80;/listen 8080;/' site.conf10sed -i.bak 's/listen 80;/listen 8080;/' site.conf11 12# Sum the response sizes (10th column)13awk '{sum += $10} END {print sum}' access.log

1/4-c counts matching lines instead of printing them.

sed -i.bak keeps a backup copy as site.conf.bak, so you can undo a bad edit.

Save yourself the pain

Common mistakes

Oops

Running sed -i on a production config without testing the pattern first.

Do this instead

Run sed without -i and check the output. Use -i.bak so a backup is saved automatically.

Oops

Using grep error and missing lines that say ERROR or Error.

Do this instead

Add -i for case-insensitive search, or use a pattern like grep -E 'error|fail' -i.

Your mission

Log detective

~20 min

Quick check

Which tool would you pick to print only the third column of every line in a file?

Go deeper

1Practice grep to search for a pattern across multiple log files.
2Use sed to find-and-replace text in a sample file.
3Write a simple awk command to extract a specific column from structured text output.