Troubleshooting the network: from "it is down" to the cause
A step-by-step method for network problems: ping and ICMP, traceroute, ss to see what listens, connection refused versus timeout, tcpdump to see real packets, and firewalls with nftables and ufw.
By the end of this lesson you'll be able to
0 of 7 completeA method, not guesses
"The app can't reach the database" has a dozen possible causes. Checking them in a fixed order, from the bottom up, finds the cause quickly and proves it, which matters when you tell another team the problem is on their side.
The first three rungs are in the Networking lesson. This lesson is about the rest: what's reachable, what's listening, and what's in the way.
ping: is the host there?
$ ping -c 3 1.1.1.1 64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=41.8 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=33.6 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=57 time=33.6 ms --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 33.625/36.358/41.818/3.860 ms
ping sends ICMP echo requests. Replies prove the host is reachable; the times show latency
and any packet loss points to a bad link. A failed ping proves much less: AWS security groups
and many firewalls block ICMP by default, so a perfectly healthy server may not answer. Always go on
to test the real port.
traceroute and mtr: where does it stop?
$ traceroute -n 1.1.1.1 # each router (hop) on the way, and its delay $ mtr -rwc 10 1.1.1.1 # traceroute and ping combined: loss per hop
Each line is one router. Where the hops stop answering is roughly where the path breaks, inside your
network, your provider's, or theirs. * * * on one hop in the middle is usually a router that
ignores these probes, not a failure: only loss that continues to the last hop is real. Install them
with sudo apt install traceroute mtr-tiny; tracepath needs no root.
ss: what is listening?
ss shows sockets: what listens and what's connected. It replaced the older netstat.
$ sudo ss -tlnp # t TCP, l listening, n numbers, p process State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",…)) LISTEN 0 5 0.0.0.0:8099 0.0.0.0:* users:(("python3",pid=100308,fd=3)) $ ss -tn state established # current connections $ ss -s # totals: how many connections of each state $ sudo ss -tlnp 'sport = :443' # just one port
sudo ss -tlnpsudorun the next command as root
Read the Local Address column: 0.0.0.0 or * accepts connections on every address,
127.0.0.1 only from this machine. The single most common "works on the server, not from outside"
bug is an app bound to 127.0.0.1.
Refused or timeout?
$ nc -zv -w 2 127.0.0.1 8099 Connection to 127.0.0.1 8099 port [tcp/*] succeeded! $ nc -zv -w 2 127.0.0.1 8098 nc: connect to 127.0.0.1 port 8098 (tcp) failed: Connection refused $ nc -zv -w 3 10.0.2.20 5432 nc: connect to 10.0.2.20 port 5432 (tcp) timed out: Operation now in progress
| Result | Means | Look at |
|---|---|---|
| succeeded | the port is open and something answered | the app itself: logs, curl -v |
| refused, instantly | you reached the host; nothing listens there | ss -tlnp and the service on that host |
| timed out | no answer at all: packets are dropped on the way | firewalls, security groups, routes |
| No route to host | no path, or a firewall rejecting | ip route, the firewall's reject rules |
This one distinction halves most investigations. A timeout is almost never the application; a refusal almost never the network.
tcpdump: see the real packets
When two teams each say "it's not us", tcpdump settles it by showing what actually crosses the wire:
$ sudo tcpdump -ni any port 5432 # all traffic on port 5432 $ sudo tcpdump -ni eth0 host 10.0.2.20 and port 5432 -c 20 $ sudo tcpdump -ni any -w capture.pcap port 443 # save to a file to open in Wireshark
sudo tcpdump -ni any port 5432sudorun the next command as root
10:41:02.113 IP 10.0.1.15.51544 > 10.0.2.20.5432: Flags [S], seq 381… 10:41:03.140 IP 10.0.1.15.51544 > 10.0.2.20.5432: Flags [S], seq 381… 10:41:05.188 IP 10.0.1.15.51544 > 10.0.2.20.5432: Flags [S], seq 381…
[S] is a SYN, the start of a TCP connection. Here it leaves three times and nothing comes back:
proof that the packets leave this server and are dropped further on. A healthy connection shows
[S], then [S.] (SYN-ACK) from the other side, then [.]. A refused one shows [R.] (reset).
Firewalls: nftables and ufw
The Linux kernel filters packets with netfilter. You manage it with nftables (nft), which
replaced iptables. Ubuntu adds ufw, a simple front end; RHEL uses firewalld (firewall-cmd).
In the cloud, the provider's security groups filter before packets reach the server, so a port has
to be open in both.
$ sudo ufw status verbose # Ubuntu: rules and default policy $ sudo ufw allow OpenSSH # ALWAYS first, on a remote server $ sudo ufw allow 443/tcp $ sudo ufw allow from 10.0.1.0/24 to any port 5432 proto tcp # database: app subnet only $ sudo ufw enable $ sudo nft list ruleset # what the kernel actually enforces
sudo ufw status verbosesudorun the next command as root
A minimal nftables policy, /etc/nftables.conf, that drops everything not allowed:
table inet filter { chain input { type filter hook input priority 0; policy drop; ct state established,related accept # replies to our own connections iif lo accept # this machine talking to itself tcp dport 22 accept # SSH tcp dport { 80, 443 } accept # web icmp type echo-request accept # allow ping } }
$ sudo nft -c -f /etc/nftables.conf # check the file without applying it $ sudo systemctl enable --now nftables # load it now and at boot
sudo nft -c -f /etc/nftables.confsudorun the next command as root
sudo systemctl enable --now nftablessystemctlcontrol systemd services
Prove where a connection fails
Write port-check.sh: given a host and a port, test DNS, then the TCP port, and say in plain words what failed and where to look next. Test it on a port that's open, one that refuses, and a name that doesn't exist.
A one-line verdict for each checkReveal one safe solution
$ cat > ~/port-check.sh <<'EOF' #!/usr/bin/env bash host=${1:?usage: port-check.sh <host> <port>}; port=${2:?usage: port-check.sh <host> <port>} ip=$(getent ahostsv4 "$host" | awk 'NR==1 {print $1}') [[ -n $ip ]] || { echo "FAIL dns: $host does not resolve. Check /etc/hosts and resolvectl."; exit 1; } out=$(nc -zv -w 3 "$ip" "$port" 2>&1) case "$out" in *succeeded*) echo "OK $host ($ip) port $port is open. If the app still fails, read its logs." ;; *refused*) echo "FAIL refused: $ip answered, nothing listens on $port. Run ss -tlnp there."; exit 1 ;; *) echo "FAIL timeout: no answer from $ip:$port. Check firewalls and security groups."; exit 1 ;; esac EOF $ python3 -m http.server 8099 > /dev/null 2>&1 & sleep 1 # a test server on 8099 $ bash ~/port-check.sh localhost 8099 OK localhost (127.0.0.1) port 8099 is open. If the app still fails, read its logs. $ bash ~/port-check.sh localhost 8098 FAIL refused: 127.0.0.1 answered, nothing listens on 8098. Run ss -tlnp there. $ bash ~/port-check.sh no-such-host.invalid 80 FAIL dns: no-such-host.invalid does not resolve. Check /etc/hosts and resolvectl. $ kill %1
cat > ~/port-check.sh <<'EOF'catprint a file>write output to a file, replacing it
python3 -m http.server 8099 > /dev/null 2>&1 & sleep 12>&1send errors to the same place as output
kill %1killsend a signal to a process
The exit code is 1 on every failure, so this works in a deploy pipeline or a cron health check. To
see a timeout, point it at an address that drops packets, such as 10.255.255.1 80 on most networks.
Try it yourself
0 of 6 steps doneRun each task in your own account or terminal, then tick it off.
Quick check
curl to port 5432 says "Connection refused" straight away. What does that tell you?
Official reading for this lesson
Or get every quiz answer right and it completes itself.