Network Monitoring & Troubleshooting

Turning "The Network Is Slow" Into an Answer#

Networking is where vague complaints meet hard evidence. What separates an experienced engineer from a novice is not knowing every protocol. It is knowing which tool answers which question, reading the output correctly, and working through the layers in an order that rules things out quickly.

The Core Toolkit, by Question#

Two distinctions from this table save a lot of time. Refused versus timed out: "connection refused" means a host answered with a TCP RST, so the network path works and nothing is listening on that port. A timeout means packets are being dropped, usually by a firewall or a routing problem. And ping failing proves little: many hosts and cloud security groups block ICMP, so test the actual service port before declaring a host down.

Breaking Down a Slow Request#

A single HTTPS request passes through distinct phases, and each one points to a different owner. curl reports them all:

curl -o /dev/null -s -w '
dns      %{time_namelookup}
connect  %{time_connect}
tls      %{time_appconnect}
ttfb     %{time_starttransfer}
total    %{time_total}
' https://example.com/

The values are cumulative seconds from the start, not per-phase durations, which trips up almost everyone. To get the TCP handshake time, subtract time_namelookup from time_connect; the server's own processing time is roughly time_starttransfer minus time_appconnect.

The same breakdown in Python, which is handy inside monitoring scripts where you want each phase as a separate number:

import socket
import ssl
import sys
import time

def timed_get(host, path="/", port=443):
    t0 = time.perf_counter()
    family, type_, proto, _, addr = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)[0]
    t_dns = time.perf_counter()

    sock = socket.socket(family, type_, proto)
    sock.settimeout(10)
    sock.connect(addr)
    t_tcp = time.perf_counter()

    tls = ssl.create_default_context().wrap_socket(sock, server_hostname=host)
    t_tls = time.perf_counter()

    request = (f"GET {path} HTTP/1.1\r\nHost: {host}\r\n"
               "User-Agent: phase-timer/1.0\r\nConnection: close\r\n\r\n")
    tls.sendall(request.encode())
    first = tls.recv(4096)                  # blocks until the first response bytes arrive
    t_first = time.perf_counter()

    total = len(first)
    while chunk := tls.recv(65536):
        total += len(chunk)
    t_end = time.perf_counter()

    status = first.split(b"\r\n", 1)[0].decode()   # e.g. HTTP/1.1 200 OK
    ms = lambda a, b: (b - a) * 1000
    print(f"{host} -> {addr[0]}  {tls.version()}  {status}")
    print(f"  dns      {ms(t0, t_dns):7.1f} ms")
    print(f"  connect  {ms(t_dns, t_tcp):7.1f} ms")
    print(f"  tls      {ms(t_tcp, t_tls):7.1f} ms")
    print(f"  ttfb     {ms(t_tls, t_first):7.1f} ms")
    print(f"  transfer {ms(t_first, t_end):7.1f} ms  ({total} bytes)")
    tls.close()

timed_get(sys.argv[1] if len(sys.argv) > 1 else "example.com")

example.com -> 2606:4700:10::ac42:93f3  TLSv1.3  HTTP/1.1 200 OK
  dns          7.0 ms
  connect     16.7 ms
  tls         24.2 ms
  ttfb        54.5 ms
  transfer     0.1 ms  (1037 bytes)

How to read it:

• Slow dns (hundreds of ms, or varying wildly): resolver trouble, an unreachable resolver being retried, or a search-domain list causing extra lookups. Retry with dig @1.1.1.1 host to compare against a different resolver.
• Slow connect: roughly one network round trip. If it is much larger than the ping RTT, suspect SYN loss and retransmission (a one-second or three-second pause is the classic signature of a dropped SYN) or a server with a full accept queue.
• Slow tls: typically one or two round trips. Much more suggests packet loss, a large certificate chain hitting an MTU problem, or an overloaded server doing expensive key exchanges.
• Slow ttfb with fast earlier phases: the network is fine and the application is slow. Hand it to whoever owns the service, with this output as evidence.

Reading traceroute and mtr Correctly#

mtr combines ping and traceroute and runs continuously, so intermittent loss shows up. Use report mode to get a shareable snapshot:

mtr -rwzc 200 app.example.com     # report, wide, show AS numbers, 200 probes

HOST                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.1              0.0%   200    1.2   1.4   0.9   5.1   0.4
  2. 100.64.0.1               0.0%   200    8.9   9.3   8.1  21.0   1.6
  3. 203.0.113.9             38.0%   200   11.0  11.4  10.2  30.2   2.0
  4. 198.51.100.17            0.0%   200   24.1  24.8  23.6  41.9   2.2
  5. app.example.com          0.0%   200   25.0  25.3  24.1  39.0   1.9

Hop 3 shows 38% loss, but hops 4 and 5 show none. The loss therefore is not real: that router rate-limits the ICMP replies it generates, while forwarding traffic through it normally. Only loss that continues to the final hop matters. Similarly, a latency jump that persists on every later hop is real (often a long-distance link), while a spike on one hop that disappears afterwards is just a busy router's control plane. Remember that the return path can differ from the forward path, so a trace from the other end is sometimes needed to see the whole picture.

Looking at Sockets#

ss answers questions about the local machine:

ss -tlnp                         # listening TCP sockets and owning processes
ss -tan state time-wait | wc -l  # how many connections are in TIME-WAIT
ss -ti dst 10.0.2.15             # per-connection RTT, cwnd, retransmits

A large Recv-Q on a listening socket means the application is not accepting connections fast enough. Thousands of TIME-WAIT sockets point at a client opening a new connection per request instead of reusing them. Growing retrans counts in ss -ti confirm packet loss on a specific connection. The TCP handshake and flags lesson explains the states.

Measuring Capacity With iperf3#

When the complaint is throughput, separate the network from the application:

iperf3 -s                               # on the server
iperf3 -c server.internal -t 20 -P 4    # client: 20 seconds, 4 parallel streams
iperf3 -c server.internal -R            # reverse: test server-to-client direction
iperf3 -c server.internal -u -b 50M     # UDP at 50 Mbit/s: reports loss and jitter

If a single stream is slow but four parallel streams fill the link, the limit is per-connection (TCP window size combined with high latency, or loss), not raw capacity. Always test both directions, because asymmetric problems are common.

Packet Capture: The Final Word#

When the tools above disagree, capture packets. Capture on the server, or as close to the problem as you can, and filter tightly:

sudo tcpdump -ni eth0 -w slow.pcap 'host 203.0.113.50 and tcp port 443'
sudo tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'   # only SYNs and RSTs

Open the file in Wireshark and use display filters to jump to trouble: tcp.analysis.retransmission, tcp.flags.reset == 1, tcp.analysis.zero_window (the receiver cannot keep up), and dns.flags.rcode != 0 (failed DNS lookups). A SYN with no SYN-ACK means the packet is dropped before the server or the server is not answering; a SYN answered by an immediate RST means nothing is listening; a clean handshake followed by a long silence before the response means the application is slow.

Monitoring at Scale#

Command-line tools are for incidents; production needs continuous telemetry:

• Flow data (NetFlow, IPFIX, sFlow) records who talked to whom, on which ports, and how much. It is essential for capacity planning and for security investigations.
• Device and host metrics such as interface throughput, errors, drops, and CPU, collected over SNMP, streaming telemetry, or exporters for Prometheus.
• Synthetic checks from several regions that run the DNS, TCP, TLS, and HTTP phases above every minute, so you find regional failures before users report them.
• Distributed tracing that follows one request across services to find the slow hop inside your own stack.

Build a baseline before you need it. Record latency percentiles (p50, p95, p99), error rates, and traffic volume per service, with enough history to compare today against the same hour last week. Without a baseline, the first half hour of every incident is spent arguing about whether the current numbers are normal. Alert on deviations from the baseline as well as on fixed thresholds, because gradual latency creep never crosses a fixed line until it is already an outage.

A Method, Not Just Tools#

Troubleshooting is a loop: state a hypothesis, choose the test that could prove it wrong, run it, and narrow down. For "the site is down", work the layers in order: does the name resolve (dig), is the port reachable (nc), does TLS complete (curl -v), what does the application return (the status code and body). Each step either finds the problem or rules out a whole layer. Write down what you tested and what you saw as you go; that log becomes the incident timeline, and it stops the team re-running the same checks. For the underlying protocols, see the DNS lesson and the TLS handshake lesson.