Network Monitoring & Troubleshooting
Turning "The Network Is Slow" Into an Answer
Networking is where vague complaints meet hard evidence. The difference between a novice and a seasoned engineer is not knowing every protocol—it is knowing exactly which tool answers which question, and reading the output correctly.
The Core Diagnostic Toolkit
ping: sends ICMP echo requests to measure reachability and round-trip time. If ping fails, verify it is not just ICMP being blocked before concluding the host is down.traceroute/tracert: shows the path packets take, hop by hop, and where latency or loss begins. Usemtr(My Traceroute) for a live, continuous view that reveals intermittent loss per hop.dig/nslookup: queries DNS directly, independent of the OS cache.dig +tracewalks the delegation chain from the root.ss/netstat: lists open sockets and their states.ss -tan state time-waitis the fastest way to see connection pileups.tcpdump/ Wireshark: capture raw packets. When all else fails, reading the actual handshake and retransmissions resolves mysteries that no log can.curl -v/curl --trace-time: shows DNS lookup, TCP connect, TLS handshake, and HTTP exchange timings for a single request.iperf3: measures raw throughput between two hosts, separating network capacity from application slowness.nc/telnet: the simplest reachability test for a specific port: can I even open a TCP connection?
The Latency Breakdown
When a request feels slow, break it into its stages: DNS resolution, TCP connect, TLS handshake, time to first byte, and transfer time. curl's timing variables (time_namelookup, time_connect, time_appconnect, time_starttransfer, time_total) report each. A slow time_namelookup is a DNS problem; a slow time_connect is network or server overload; a slow time_starttransfer is the application.
Interpreting Packet Loss
Loss has many causes and the tool tells you which. Loss that appears on a middle hop but not on later hops is often just ICMP rate-limiting on that router—not a real problem. Loss that persists to the destination is real and will trigger TCP retransmissions and slow throughput. mtr's per-hop statistics distinguish the two.
Monitoring at Scale
Point-in-time tools are for incidents; production needs continuous telemetry. Modern stacks collect: - Flow data (NetFlow/sFlow) showing who talks to whom, and how much. - Metrics (prometheus-style) for latency, error rate, and throughput per service. - Synthetic checks from multiple regions that catch regional failures before users do. - Distributed tracing that follows a single request across services to find the slow segment.
A Method, Not Just Tools
Good troubleshooting is a loop: form a hypothesis, pick the tool that can falsify it, collect evidence, and narrow the search. "The site is down" branches into DNS, routing, TLS, server, or application—and each branch has a specific test. Chasing a hunch without a test is how hours disappear. Capturing a pcap and following the actual handshake is often faster than any amount of theorizing.
Build a Baseline Before the Incident
The most underrated monitoring tool is a baseline. If you know your normal latency, request mix, and error rate, every anomaly stands out immediately. Teams that lack a baseline spend the first 30 minutes of an incident just deciding whether behavior is normal. Record per-service latency percentiles (p50, p95, p99), error rates by class, and traffic volume on a schedule, and store enough history to compare today against last Tuesday at the same hour. Alerting on deviations from that baseline catches problems that fixed thresholds miss—gradual latency creep, slow memory-driven degradation, and seasonal traffic shifts.