Server review: a five-minute health check
Check any Linux server in a fixed order: load, memory, disk, services, ports and logins. Know what each number means and what to do when one is wrong.
By the end of this lesson you'll be able to
0 of 6 completeAlways check in the same order
When a server is "slow" or "acting up", the worst thing you can do is guess. Experienced engineers run the same checks in the same order every time, from the most common causes to the least, and write down what they see.
First, make sure you're on the right machine and see how long it's been up:
$ hostnamectl | head -n 3 # name, and whether it's a VM $ uname -r # kernel version 6.8.0-48-generic $ uptime 10:42:19 up 23 days, 4:11, 2 users, load average: 0.52, 0.48, 0.41
hostnamectl | head -n 3|pipe: send output to the next commandheadshow the first lines-nhow many lines
uname -runamesystem information-rkernel release
uptimeuptimehow long the machine is up, and its load
"Up 3 minutes" when nobody rebooted it is a finding in itself: something restarted the machine.
1. Is it overloaded?
The three numbers at the end of uptime are the load average over 1, 5 and 15 minutes:
roughly, how many tasks were running or waiting to run. It only means something next to the
number of CPU cores:
$ nproc 4
nprocnprocnumber of CPU cores
| Load on 4 cores | Means |
|---|---|
| below 4 | there's spare capacity |
| about 4 | every core is busy, nothing waiting |
| well above 4 | tasks are queueing: users feel it as slowness |
Read the three numbers as a trend: 3.9, 2.1, 0.8 is climbing right now, 0.8, 2.1, 3.9 is
calming down. Linux also counts tasks waiting for the disk, so high load with low CPU use often
means slow storage, not a busy processor. You'll find the busy processes in the next lesson.
What load really counts
Load is not a percentage. It's a count of tasks that want to run, averaged over time: the ones on a core
now, the ones in the run queue waiting for a free core, and the ones stuck waiting for the disk (state
D, "uninterruptible sleep").
Here's a 4-core test machine, before and 45 seconds after starting six CPU-hungry processes:
$ uptime 12:13:59 up 2:06, 0 user, load average: 5.47, 8.99, 7.45 $ nproc 4 $ uptime 12:14:45 up 2:07, 0 user, load average: 7.01, 8.91, 7.50
uptimeuptimehow long the machine is up, and its load
nprocnprocnumber of CPU cores
The 1-minute number moved from 5.47 to 7.01 while the 15-minute one barely changed: the averages smooth things out, so a spike shows in the first number long before the last. And 7 on 4 cores means three tasks, on average, were always waiting their turn.
2. Memory
$ free -h total used free shared buff/cache available Mem: 3.7Gi 1.4Gi 316Mi 3.2Mi 2.1Gi 2.3Gi Swap: 1.0Gi 283Mi 740Mi
free -hfreememory use-hhuman-readable
Look at available, not free. Linux uses spare RAM as a disk cache (buff/cache) and gives
it back as soon as programs need it, so a small "free" is normal and healthy.
Worry when available is close to zero, or swap used keeps growing: the server is moving
memory to disk, which is very slow. If memory runs out completely, the kernel's OOM killer stops a
process to save the machine, and journalctl -k | grep -i oom shows which one.
3. Disk
$ df -h # space per filesystem Filesystem Size Used Avail Use% Mounted on /dev/vda1 40G 38G 2.0G 96% / $ df -i # inodes: the number of files. Can run out with space left $ sudo du -xh --max-depth=1 /var | sort -h | tail -5 # which folders are big $ sudo find / -xdev -type f -size +500M -exec ls -lh {} \; 2> /dev/null # big single files
df -hdfdisk space per filesystem-hhuman-readable sizes
sudo du -xh --max-depth=1 /var | sort -h | tail -5sudorun the next command as rootdudisk space used by files|pipe: send output to the next commandsortsort linestailshow the last lines
sudo find / -xdev -type f -size +500M -exec ls -lh {} \; 2> /dev/nullfindsearch for files-typef for files, d for directories-sizeby size, like +10M
-x (and -xdev) keeps the search on one filesystem, so you don't wander into network shares.
The usual suspects are logs (/var/log), old releases, container images (/var/lib/docker) and
forgotten backups.
Watching it live: vmstat, iostat and sar
uptime, free and df are snapshots. When the server is slow right now, you want numbers every
second, to see whether it's the CPU, the memory or the disk. Three tools from the same family do that.
vmstat comes with every distribution; iostat and sar are in the sysstat package
(sudo apt install sysstat).
vmstat: the one-line overview
vmstat 1 5 prints a line every second, five times. Ignore the first line: it's the average since boot.
This is the same machine with six busy processes on four cores:
$ vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 9 0 1048316 255228 61228 1357400 13 161 625 544 2269 12 42 22 35 1 0 0 13 0 1048316 251576 61228 1357464 0 0 48 0 3706 10481 32 68 1 0 0 0 12 1 1048316 167960 61348 1357376 0 0 0 1360 6195 17821 46 54 0 0 0 0 19 0 1048316 135276 61424 1357712 0 0 0 696 4869 11751 45 55 0 0 0 0 10 0 1048316 140672 61496 1357888 0 0 16 1016 4415 10050 51 49 1 0 0 0
And the same machine writing a large file straight to disk:
$ vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 4 0 1047976 163544 61748 1349000 13 160 621 810 2283 12 42 22 36 1 0 0 2 1 1047960 163292 61764 1349380 32 0 400 568320 4694 8574 10 20 55 16 0 0 4 0 1047952 155228 61780 1350464 12 0 1172 502784 4365 7433 32 24 31 13 0 0 2 0 1047948 141368 61796 1351180 4 0 416 433152 3740 6555 48 22 20 10 0 0 1 1 1043576 143636 61824 1351472 5292 0 5572 481440 4488 7384 15 26 37 22 0 0
| Column | Means | In the runs above |
|---|---|---|
r | tasks running or waiting for a CPU | 10–19 on 4 cores: CPU-bound |
b | tasks blocked on I/O (the D state) | 1 during the disk write |
si, so | memory swapped in and out per second | si 5292 in one second: memory pressure |
bi, bo | blocks read from and written to disk | bo around 500,000 during the write |
us, sy, id | CPU % in user code, in the kernel, idle | id 0–1 in the first run: no spare CPU |
wa | CPU % idle while waiting for I/O | 10–22 during the write, 0 in the CPU-bound run |
Read it in that order: is r above the core count, is b or wa high, is si/so non-zero? That tells
you which of the next tools to open.
iostat: which disk, and how busy
iostat -x shows each disk's work. Its full output is wide, so pick the columns that matter:
$ iostat -dxy sdd 1 1 | awk '/Device|sdd/ {print $1, $2, $8, $9, $12, $(NF-1), $NF}' | column -t Device r/s w/s wkB/s w_await aqu-sz %util sdd 2.00 392.00 401408.00 1.63 0.65 65.20
iostat -dxy sdd 1 1 | awk '/Device|sdd/ {print $1, $2, $8, $9, $12, $(NF-1), $NF}' | column -t|pipe: send output to the next command
| Column | Means |
|---|---|
r/s, w/s | reads and writes per second |
wkB/s | kilobytes written per second: here about 400 MB/s |
w_await | average milliseconds per write, including the wait in the queue |
aqu-sz | average number of requests queued on the disk |
%util | how much of the time the disk was busy |
-y skips the since-boot report and -d hides the CPU part. The number to watch is await: a few
milliseconds is fine for SSDs, tens or hundreds means requests are queueing. %util near 100% used to mean
"disk saturated", but SSDs and cloud volumes serve many requests in parallel, so trust await and
aqu-sz more.
sar: the same numbers, and their history
sar reports CPU (-u), run queue and load (-q), memory (-r), disks (-d) and network (-n DEV):
$ sar -q 1 3 Linux 6.6.87.2-microsoft-standard-WSL2 (web-01) 10/18/26 _x86_64_ (4 CPU) 12:14:49 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 12:14:50 10 704 7.57 9.00 7.54 0 12:14:51 12 694 7.57 9.00 7.54 0 12:14:52 8 695 7.57 9.00 7.54 0 Average: 10 698 7.57 9.00 7.54 0 $ sar -u 1 2 Linux 6.6.87.2-microsoft-standard-WSL2 (web-01) 10/18/26 _x86_64_ (4 CPU) 12:15:40 CPU %user %nice %system %iowait %steal %idle 12:15:41 all 16.83 0.00 33.67 18.34 0.00 31.16 12:15:42 all 36.83 0.00 21.74 13.30 0.00 28.13 Average: all 26.74 0.00 27.76 15.84 0.00 29.66
Live, it overlaps with vmstat. Its real value is history. Turn on collection once
(sudo systemctl enable --now sysstat, and ENABLED="true" in /etc/default/sysstat on Ubuntu), and it
records every 10 minutes into /var/log/sysstat/. Then "the site was slow at 3 a.m." stops being a mystery:
sar -q -s 02:30:00 -e 03:30:00 shows the load at that hour, and sar -f /var/log/sysstat/sa17 reads
another day.
%steal is worth knowing on cloud VMs: time your VM wanted the CPU but the hypervisor gave it to someone
else. Steady steal on a burstable instance (like t3) usually means its CPU credits ran out.
free, once more
$ free -h -w total used free shared buffers cache available Mem: 3.7Gi 2.3Gi 267Mi 93Mi 60Mi 1.3Gi 1.4Gi Swap: 1.0Gi 1.0Gi 476Ki
free -h -wfreememory use-hhuman-readable
-w splits buff/cache into buffers and cache. available (1.4 GiB) is still fine here, but swap
is full: 1.0 GiB used of 1.0 GiB. Together with the si spike in vmstat, that says this machine has been
short of RAM at some point and pushed memory out to disk. free -h -s 2 repeats every two seconds if you want
to watch it change.
| Symptom | Look at |
|---|---|
load high, r high, id near 0 | CPU: top, pidstat 1 to find the process |
load high, b and wa high | disk: iostat -x 1, then sudo iotop |
si/so non-zero, swap growing | memory: free -h, ps aux --sort=-rss |
| everything fine now, slow "earlier" | sar history for that hour |
4. Services and ports
$ systemctl --failed # anything that crashed UNIT LOAD ACTIVE SUB DESCRIPTION ● myapp.service loaded failed failed My App $ systemctl list-units --type=service --state=running # what's running $ sudo ss -tlnp # what's listening, and which process State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=812,fd=6)) LISTEN 0 4096 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=701,fd=3)) LISTEN 0 4096 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=944,fd=7))
systemctl --failedsystemctlcontrol systemd services
sudo ss -tlnpsudorun the next command as root
Read the Local Address column carefully. 0.0.0.0 means the port accepts connections from
anywhere; 127.0.0.1 means only from this machine. A database listening on 0.0.0.0 is a
finding. The Services with systemd lesson shows how to read why a service failed.
5. Who has been here?
$ who # who's logged in now $ w # ... and what they're running $ last -n 5 # recent logins (from /var/log/wtmp) $ sudo lastb -n 5 # recent FAILED logins $ sudo grep -c "Failed password" /var/log/auth.log # brute-force attempts $ sudo grep "Accepted" /var/log/auth.log | tail -n 5 # successful SSH logins $ sudo journalctl -u ssh --since "1 hour ago" # the same, from the journal
sudo lastb -n 5sudorun the next command as root
sudo grep -c "Failed password" /var/log/auth.loggrepfind lines that match a pattern-ccount matching lines
sudo grep "Accepted" /var/log/auth.log | tail -n 5|pipe: send output to the next commandtailshow the last lines-nhow many lines
sudo journalctl -u ssh --since "1 hour ago"journalctlread the systemd journal (logs)-uone service
Authentication events (SSH logins, sudo use, password changes) go to /var/log/auth.log on
Debian and Ubuntu, and /var/log/secure on Red Hat, Rocky and Amazon Linux. Every sudo command
is logged there with the user's name, which is one more reason to use sudo instead of a root shell.
Thousands of Failed password lines from many addresses are normal for any server on the
internet: bots try common passwords all day. They matter when a password login succeeds from
an address you don't know, which is why servers should use SSH keys and disable password logins.
Logs in general
$ sudo journalctl -p err -b # errors since the last boot $ sudo journalctl -u nginx -n 50 # the last 50 lines from one service $ ls -lh /var/log # classic log files $ sudo tail -f /var/log/syslog # everything, live (Debian/Ubuntu)
sudo journalctl -p err -bsudorun the next command as rootjournalctlread the systemd journal (logs)-pminimum priority, like err-bsince this boot
sudo journalctl -u nginx -n 50-uone service-nhow many lines
ls -lh /var/loglslist files-llong format: permissions, owner, size, date-hhuman-readable sizes, like 4.0K
sudo tail -f /var/log/syslogtailshow the last lines-fkeep following as new lines arrive
Old logs are rotated and compressed by logrotate (/etc/logrotate.d/), which is why you see
syslog.1 and syslog.2.gz. If a disk keeps filling with logs, a missing or broken logrotate
rule is the usual cause.
Write a health report for a server
You've just been given SSH access to a server nobody has looked at in months. Run the health check and save a short report with the load next to the core count, available memory, the fullest disk, failed services, listening ports and the last five logins.
~/review.txtReveal one safe solution
$ cat > ~/review.sh <<'EOF' #!/bin/bash echo "== $(hostname) at $(date '+%F %T')" echo "-- load (cores: $(nproc))"; uptime echo "-- memory"; free -h echo "-- disk"; df -h -x tmpfs -x devtmpfs echo "-- failed services"; systemctl --failed --no-legend echo "-- listening"; sudo ss -tlnp echo "-- last logins"; last -n 5 EOF $ bash ~/review.sh > ~/review.txt 2>&1 $ less ~/review.txt
cat > ~/review.sh <<'EOF'catprint a file>write output to a file, replacing it
bash ~/review.sh > ~/review.txt 2>&12>&1send errors to the same place as output
less ~/review.txtlessscroll through a file; q quits
One redirect captures every check, output and errors. Keep both files: next time something feels
wrong, run the script again into a new file and compare the two with diff. What changed is
usually where the problem is.
Try it yourself
0 of 7 steps doneRun each task in your own account or terminal, then tick it off.
Quick check
A 4-core server shows "load average 3.9, 3.5, 2.1". What does that tell you?
Official reading for this lesson
- Docsfree(1) manualUbuntu manpages
- Docsss(8) manualUbuntu manpages
- Docsjournalctl(1) manualUbuntu manpages
- ArticleLinux load averages: solving the mysteryBrendan Gregg
- ArticleLinux performance analysis in 60,000 millisecondsBrendan Gregg (Netflix)
- GuideLPIC-1 exam 102, objectives 108.2 and 110.1Linux Professional Institute
Or get every quiz answer right and it completes itself.