Networking: addresses, DNS, routes and SSH

What happens when you curl a URL

Every networking problem is a failure at one step of this trip. Learn the steps and you know what to check:

⤢ View

Name, route, port, program. Test the steps in order and the broken one shows itself.© Animation: DevOps Academy

Close

TCP/IP: addresses, ports and layers

An IP address finds the machine. A port finds the program on it. A connection is the pair address:port at each end, such as 172.29.94.205:51544 → 93.184.215.14:443.

TCP sets up a connection and makes sure every byte arrives in order (web, SSH, databases). UDP sends packets with no setup or guarantee (DNS, video calls). Ports you'll meet every day: 22 SSH, 53 DNS, 80 HTTP, 443 HTTPS, 5432 PostgreSQL, 3306 MySQL, 6379 Redis.

$ ip -br addr                    # your interfaces and addresses, one line each
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             172.29.94.205/20 fe80::215:5dff:fe72:6be6/64
$ ip link                        # interfaces, state and MAC addresses

lo is the loopback: 127.0.0.1 is this machine talking to itself. An app listening only on 127.0.0.1 can't be reached from outside, which is the right setting for a database the app shares a machine with.

Subnets and CIDR

172.29.94.205/20 is an address and its network. The /20 says how many leading bits name the network; the rest number the hosts. Machines in the same subnet talk directly; anything else goes through a router.

Each step of one in the prefix halves the size: a /25 is half a /24. The first address is the network and the last the broadcast, so a /24 has 254 usable hosts. Private ranges, which are never routed on the internet, are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16.

DNS: from a name to an address

program → /etc/nsswitch.conf ("hosts: files dns")
        → /etc/hosts               fixed names, checked first
        → /etc/resolv.conf         which DNS server to ask (often 127.0.0.53, systemd-resolved)
        → DNS server               answers, or asks the internet's DNS for you

$ getent ahostsv4 example.com     # look up a name exactly as programs do
172.66.147.243  STREAM example.com
$ cat /etc/resolv.conf            # which DNS server this machine asks
$ resolvectl status               # systemd-resolved: the real upstream servers
$ dig example.com +short          # ask DNS directly (package dnsutils or bind-utils)
$ dig @1.1.1.1 example.com        # ask a specific server: is it my DNS or the record?

Records you'll use: A (name → IPv4), AAAA (→ IPv6), CNAME (name → another name), MX (mail), TXT (verification, SPF). DNS answers are cached for their TTL, which is why a changed record can take minutes or hours to reach everyone.

Use getent to see what your app sees, since it follows /etc/hosts too. Use dig to see what DNS says. When they disagree, something in /etc/hosts is overriding DNS.

Routing: where packets go

$ ip route
default via 172.29.80.1 dev eth0 proto kernel
172.29.80.0/20 dev eth0 proto kernel scope link src 172.29.94.205
$ ip route get 1.1.1.1            # which route this destination takes
1.1.1.1 via 172.29.80.1 dev eth0 src 172.29.94.205 uid 1000

Read it bottom-up: anything in 172.29.80.0/20 is local, sent straight out of eth0. Everything else matches default and goes to the gateway 172.29.80.1. When several routes match, the most specific one (longest prefix) wins. A server with no default route can talk to its subnet and nothing else.

$ sudo ip route add 10.20.0.0/16 via 172.29.80.5   # until reboot: reach another network via .5
$ sudo ip route del 10.20.0.0/16

DHCP and static addresses

Most servers get their address from DHCP: at boot they ask the network, and a DHCP server (your cloud provider, or a router) hands out an address, netmask, gateway and DNS servers. In the cloud, keep DHCP and set fixed private addresses in the console instead.

For a fixed address on Ubuntu, describe it in netplan:

# /etc/netplan/60-static.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: false
      addresses: [10.0.1.15/24]
      routes:
        - to: default
          via: 10.0.1.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]

$ sudo netplan try               # applies it, and rolls back after 120 s unless you confirm
$ sudo netplan apply

SSH with keys

SSH logs you into a remote server over an encrypted connection. Keys beat passwords: there's nothing to guess, and a key never leaves your laptop.

$ ssh-keygen -t ed25519 -C "rajeev@laptop"     # makes ~/.ssh/id_ed25519 (private) and .pub (public)
$ ssh-keygen -lf ~/.ssh/id_ed25519.pub          # its fingerprint
256 SHA256:r9kRUjoTTeZfWC6kJsG7Plwe22vjyNAvM35ng5fPaYo rajeev@laptop (ED25519)
$ ssh-copy-id deploy@10.0.1.15                  # adds your public key to the server's ~/.ssh/authorized_keys
$ ssh deploy@10.0.1.15

The private key stays on your machine, with permissions 600 (SSH refuses keys others can read). The public key goes into ~/.ssh/authorized_keys on each server. The first time you connect, SSH shows the server's fingerprint and saves it in ~/.ssh/known_hosts; if it ever changes, SSH warns you loudly, because someone may be in the middle.

Save typing with ~/.ssh/config:

Host web1
    HostName 10.0.1.15
    User deploy
    IdentityFile ~/.ssh/id_ed25519

Host app-private
    HostName 10.0.2.20
    User deploy
    ProxyJump web1          # reach a private server through web1

Now ssh web1 is enough, and ssh app-private hops through web1 automatically.

Copying files: scp and rsync

$ scp app.tar.gz web1:/tmp/                      # copy a file up
$ scp web1:/var/log/nginx/error.log .            # copy a file down
$ rsync -avn ./site/ web1:/srv/site/             # -n: dry run, shows what would be sent
$ rsync -av --delete ./site/ web1:/srv/site/     # make the server's folder match exactly
$ rsync -avz --progress backup/ web1:/backups/   # -z compress, show progress

rsync sends only what changed, resumes interrupted copies, and keeps permissions and times with -a. Use it for anything bigger than a few files or anything you copy twice. The trailing slash matters: site/ copies what's inside, site creates /srv/site/site. With --delete, files on the server that aren't in your folder are removed, so always run with -n first.

$ cat > ~/net-check.sh <<'EOF'
#!/bin/bash
host=${1:-example.com}
echo "== my addresses"; ip -br addr | grep -v '^lo'
echo "== default gateway"; ip route show default
ip=$(getent ahostsv4 "$host" | awk 'NR==1 {print $1}')
echo "== DNS lookup of $host"; echo "${ip:-FAILED: name does not resolve}"
[ -n "$ip" ] || exit 1
echo "== route to $ip"; ip route get "$ip"
echo "== HTTPS on port 443"; curl -sS -o /dev/null -m 5 -w "HTTP %{http_code} in %{time_total}s\n" "https://$host/" || echo "FAILED: no connection"
EOF
$ bash ~/net-check.sh example.com > ~/net-check.txt
$ cat ~/net-check.txt
== my addresses
eth0             UP             172.29.94.205/20 fe80::215:5dff:fe72:6be6/64
== default gateway
default via 172.29.80.1 dev eth0 proto kernel
== DNS lookup of example.com
172.66.147.243
== route to 172.66.147.243
172.66.147.243 via 172.29.80.1 dev eth0 src 172.29.94.205 uid 1000
    cache
== HTTPS on port 443
HTTP 200 in 0.229564s