Lesson 5 · Text processing
Text processing: get answers out of logs
grep, cut, sort, uniq, tr, wc, awk and sed, chained with pipes, on a real access log. Find the errors, the noisy clients and the broken endpoint in one line.
By the end of this lesson you'll be able to
0 of 6 completeThe log we'll use
Real questions in operations sound like "how many errors since 10 o'clock?", "who's hammering us?"
and "which endpoint is broken?". The answers are in text files, and these small tools, joined with
pipes, get them in seconds. Every example uses one web server access log. Create it with the
mini mission at the bottom, or use any nginx access.log:
203.0.113.7 - - [13/Oct/2026:10:00:04 +0530] "POST /api/orders HTTP/1.1" 500 312 "-" "curl/8.5.0" │ │ │ │ │ │ └ bytes sent └ client │ │ │ │ │ └ status code │ │ │ └ path └ protocol │ │ └ method │ └ time └ client IP
$ head -n 3 access.log # first lines: what does the format look like? $ tail -n 20 access.log # the newest entries $ tail -f access.log # follow live; Ctrl+C stops $ less access.log # scroll; /500 searches, q quits
head -n 3 access.logheadshow the first lines-nhow many lines
tail -n 20 access.logtailshow the last lines
tail -f access.log-fkeep following as new lines arrive
less access.loglessscroll through a file; q quits
grep: find the lines that matter
$ grep '" 500 ' access.log # lines with status 500 (quotes avoid matching "1500") $ grep -c '" 500 ' access.log # how many 3 $ grep -i error app.log # ignore case: Error, ERROR, error $ grep -v /health access.log | wc -l # everything except health checks 10 $ grep -n "timeout" app.log # with line numbers $ grep -r "listen 80" /etc/nginx/ # search every file in a folder $ grep -E '" 5[0-9]{2} ' access.log | wc -l # -E: extended regex, any 5xx status 4 $ grep -oE '"(GET|POST) [^ ]+' access.log | sort | uniq -c | sort -rn | head -3 4 "POST /api/orders 2 "GET /health 2 "GET /
grep '" 500 ' access.loggrepfind lines that match a pattern
grep -c '" 500 ' access.log-ccount matching lines
grep -i error app.log-iignore upper/lower case
grep -v /health access.log | wc -l-vshow lines that do not match|pipe: send output to the next commandwccount lines, words and bytes-lcount lines only
grep -r "listen 80" /etc/nginx/-rsearch every file in a folder
grep -oE '"(GET|POST) [^ ]+' access.log | sort | uniq -c | sort -rn | head -3sortsort linesuniqdrop repeated lines-ccount each one-rreverse-nsort as numbersheadshow the first lines
The regular-expression pieces you'll use most:
| Pattern | Matches |
|---|---|
^ERROR / done$ | at the start / end of the line |
. | any one character |
[0-9] / [a-z] | one character from a set |
a* / a+ / a? | zero or more / one or more / optional (+ and ? need -E) |
5[0-9]{2} | 5 followed by exactly two digits (-E) |
(GET|POST) | either word (-E) |
-o prints only the matching part of each line, which is how you extract values. Test a pattern
on regex101 before you trust it on a production log.
cut and tr: pick and change characters
$ cut -d' ' -f1 access.log | head -n 3 # field 1, splitting on spaces: the client IPs 203.0.113.7 198.51.100.23 203.0.113.7 $ head -n 2 access.log | cut -d'"' -f2 # split on quotes instead: the request GET / HTTP/1.1 GET /login HTTP/1.1 $ cut -d: -f1,7 /etc/passwd # users and their login shells $ echo "Prod-Web-01" | tr 'A-Z' 'a-z' # translate characters prod-web-01 $ echo "a b c" | tr -s ' ' # squeeze repeated spaces a b c $ tr -d '\r' < windows.txt > unix.txt # delete characters (Windows line endings)
cut -d' ' -f1 access.log | head -n 3cutcut out columns|pipe: send output to the next commandheadshow the first lines-nhow many lines
echo "Prod-Web-01" | tr 'A-Z' 'a-z'echoprint texttrtranslate or delete characters
tr -d '\r' < windows.txt > unix.txt<read input from a file>write output to a file, replacing it
sort, uniq and wc: turn lines into numbers
$ wc -l access.log # lines (-w words, -c bytes) 12 access.log $ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -3 # busiest clients 5 203.0.113.7 3 198.51.100.23 2 192.0.2.44 $ sort -t: -k3 -n /etc/passwd # -t separator, -k column, -n as numbers $ sort -u ips.txt # sort and drop duplicates in one go $ du -sh /var/log/* | sort -h # -h understands 4K, 12M, 1.1G
wc -l access.logwccount lines, words and bytes-lcount lines only
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -3cutcut out columns|pipe: send output to the next commandsortsort linesuniqdrop repeated lines-ccount each one-rreverse-nsort as numbersheadshow the first lines
du -sh /var/log/* | sort -hdudisk space used by files-stotal only-hhuman-readable sizes
sort | uniq -c | sort -rn is the most useful pipeline in operations: count each distinct value,
biggest first.
Build a pipeline one step at a time
Don't write the whole pipeline at once. Add one command, look at the output, then add the next. When a long pipeline gives a strange answer, delete commands from the end until the output makes sense again.
tee lets you keep the middle of a pipeline while it carries on:
$ grep '" 500 ' access.log | tee errors.log | wc -l # save the errors AND count them 3
grep '" 500 ' access.log | tee errors.log | wc -lgrepfind lines that match a pattern|pipe: send output to the next commandteewrite input to a file and pass it onwccount lines, words and bytes-lcount lines only
awk and sed
awk reads a line, splits it into fields $1, $2, … ($NF is the last), and runs your
program on it. pattern {action} runs the action only on matching lines:
$ awk '{print $9, $7}' access.log | head -n 2 # status and path 200 / 200 /login $ awk '{print $9}' access.log | sort | uniq -c | sort -rn # requests per status 6 200 3 500 1 502 1 404 1 302 $ awk '$9 >= 500 {print $7}' access.log | sort | uniq -c # which endpoint is failing? 4 /api/orders $ awk '{sum += $10} END {print sum}' access.log # total bytes sent 21730 $ awk -F: '$3 >= 1000 {print $1}' /etc/passwd # -F sets the separator: real users
awk '{print $9, $7}' access.log | head -n 2|pipe: send output to the next commandheadshow the first lines-nhow many lines
awk '{print $9}' access.log | sort | uniq -c | sort -rnsortsort linesuniqdrop repeated lines-ccount each one-rreverse-nsort as numbers
sed edits a stream of lines: substitute, print or delete.
$ echo "port = 8080" | sed 's/8080/9090/' # substitute (first match per line) port = 9090 $ sed 's/http:/https:/g' site.conf # g: every match. Prints, changes nothing $ sed -n '3,4p' access.log # print only lines 3 and 4 $ sed '/^#/d' app.conf # delete comment lines from the output $ sed -i.bak 's/8080/9090/g' app.conf # edit the file in place, keep app.conf.bak
echo "port = 8080" | sed 's/8080/9090/'echoprint text|pipe: send output to the next command
A few more you'll use
$ nl app.conf # number the lines $ paste names.txt emails.txt # join two files side by side $ split -l 100000 huge.csv part_ # cut a big file into 100,000-line pieces $ zcat access.log.2.gz | grep '" 500 ' # read a rotated, compressed log (zgrep works too) $ sha256sum ubuntu.iso # fingerprint a file $ sha256sum -c SHA256SUMS # check downloads against the published list ubuntu.iso: OK
zcat access.log.2.gz | grep '" 500 '|pipe: send output to the next commandgrepfind lines that match a pattern
Mini mission
Find out why checkout is failing
Support says customers can't place orders. Using only the access log, find how many requests failed, which endpoint is failing, and which client is sending those requests. Save the failed requests to a file for the developers.
Input: ~/lab5/access.logOutput:
~/lab5/errors.logReveal one safe solution
$ mkdir -p ~/lab5 && cd ~/lab5 && cat > access.log <<'EOF' 203.0.113.7 - - [13/Oct/2026:10:00:01 +0530] "GET / HTTP/1.1" 200 5120 "-" "Mozilla/5.0" 198.51.100.23 - - [13/Oct/2026:10:00:03 +0530] "GET /login HTTP/1.1" 200 2048 "-" "Mozilla/5.0" 203.0.113.7 - - [13/Oct/2026:10:00:04 +0530] "POST /api/orders HTTP/1.1" 500 312 "-" "curl/8.5.0" 192.0.2.44 - - [13/Oct/2026:10:00:09 +0530] "GET /health HTTP/1.1" 200 2 "-" "kube-probe/1.31" 203.0.113.7 - - [13/Oct/2026:10:00:10 +0530] "POST /api/orders HTTP/1.1" 500 312 "-" "curl/8.5.0" 198.51.100.23 - - [13/Oct/2026:10:00:12 +0530] "GET /api/orders HTTP/1.1" 200 8192 "-" "Mozilla/5.0" 203.0.113.7 - - [13/Oct/2026:10:00:15 +0530] "POST /api/orders HTTP/1.1" 502 157 "-" "curl/8.5.0" 198.51.100.9 - - [13/Oct/2026:10:00:18 +0530] "GET /wp-login.php HTTP/1.1" 404 153 "-" "python-requests/2.32" 192.0.2.44 - - [13/Oct/2026:10:00:19 +0530] "GET /health HTTP/1.1" 200 2 "-" "kube-probe/1.31" 203.0.113.7 - - [13/Oct/2026:10:00:21 +0530] "POST /api/orders HTTP/1.1" 500 312 "-" "curl/8.5.0" 198.51.100.23 - - [13/Oct/2026:10:00:24 +0530] "GET /logout HTTP/1.1" 302 0 "-" "Mozilla/5.0" 203.0.113.50 - - [13/Oct/2026:10:00:30 +0530] "GET / HTTP/1.1" 200 5120 "-" "Mozilla/5.0" EOF $ awk '$9 >= 500' access.log | tee errors.log | wc -l # how many failed 4 $ awk '$9 >= 500 {print $6, $7}' access.log | sort | uniq -c # which endpoint 4 "POST /api/orders $ awk '$9 >= 500 {print $1}' access.log | sort | uniq -c # which client 4 203.0.113.7
mkdir -p ~/lab5 && cd ~/lab5 && cat > access.log <<'EOF'mkdirmake a directory-pcreate missing parents, no error if it exists&&run the next command only if this one workedcdchange directorycatprint a file>write output to a file, replacing it
awk '$9 >= 500' access.log | tee errors.log | wc -l|pipe: send output to the next commandteewrite input to a file and pass it onwccount lines, words and bytes-lcount lines only
awk '$9 >= 500 {print $6, $7}' access.log | sort | uniq -csortsort linesuniqdrop repeated lines-ccount each one
Every failure is a POST /api/orders from one client, so the orders API is failing, and only for
writes: the GET /api/orders at 10:00:12 succeeded. That's a far better bug report than
"checkout is broken", and errors.log gives the developers the exact requests.
Try it yourself
0 of 6 steps doneRun each task in your own account or terminal, then tick it off.
Knowledge check
Question 1 of 5Quick check
Why does uniq -c give wrong counts on an unsorted list?
Official reading for this lesson
Or get every quiz answer right and it completes itself.