Packet Analysis with Wireshark and tcpdump

When Logs Disagree, the Packets Decide

Logs say what software thinks happened; a packet capture shows what crossed the wire. tcpdump captures on servers, and Wireshark (with its command-line twin tshark) dissects hundreds of protocols and flags TCP problems automatically. The troubleshooting lesson showed when to reach for a capture; this one covers how to capture in the right place, filter it, read what TCP is telling you, and decrypt your own TLS traffic.

Captures contain passwords, cookies, tokens and personal data. Only capture on networks and hosts you own or are authorised to monitor, keep captures short and tightly filtered, and store .pcap files as carefully as you would a database dump.

Where to Capture

A capture only shows what passes the point where you capture it. Choosing that point is half the diagnosis:

The most useful technique is capturing at both ends at once. If the client sends a SYN and the server's capture never shows it, something in between drops it; if the server receives it and replies but the client never sees the SYN-ACK, look at the return path. Container setups add another reason: see container and Kubernetes networking for the veth and NAT hops a packet crosses before it reaches the host's interface.

Capturing Without Drowning

# write to a file, no DNS lookups, only one conversation
sudo tcpdump -ni eth0 -w api.pcap 'host 10.0.3.7 and tcp port 8443'

# ring buffer for an intermittent fault: 10 files of ~100 MB, oldest overwritten
sudo tcpdump -ni eth0 -C 100 -W 10 -w ring.pcap 'tcp port 5432'

# headers only: keep the first 128 bytes of each packet
sudo tcpdump -ni eth0 -s 128 -w headers.pcap

When you stop tcpdump it prints a summary. Check the last line: packets dropped by kernel above zero means the capture itself is incomplete, and Wireshark will later report missing segments that never went missing on the wire. Tighter filters, a larger buffer (-B, in KiB) and writing to a file reduce drops. Wireshark's own capture engine, dumpcap, does the same job with -b filesize:102400 -b files:10.

Two Filter Languages

Wireshark has two filter syntaxes that look alike and are not interchangeable:

Capture broadly enough that you will not need a second attempt, then narrow with display filters. The ones worth memorising:

tcp.flags.syn == 1 && tcp.flags.ack == 0         connection attempts
tcp.flags.reset == 1                             resets
tcp.analysis.flags                               anything TCP analysis flagged
tcp.analysis.retransmission                      retransmitted segments
tcp.analysis.zero_window                         receiver buffer full
tls.handshake.type == 1                          TLS ClientHellos
tls.handshake.extensions_server_name contains "api"   SNI hostname
dns.flags.rcode != 0                             failed DNS lookups
tcp.stream == 7                                  one whole connection

Right-clicking a packet and choosing Follow > TCP Stream applies the tcp.stream filter and shows the conversation's payload as text, which is the quickest way to read a cleartext protocol exchange.

Reading What TCP Is Telling You

Wireshark tracks sequence numbers per connection and labels anomalies in the Info column and under Analyze > Expert Information. The TCP handshake lesson covers the flags; the labels map to causes like this:

Timing is the other half. On a capture taken at the client, the time from SYN to SYN-ACK is the network round trip; on one taken at the server, SYN-ACK to ACK is. The gap between a request and its first response byte is server think-time. Note the capture point in the file name.

Statistics > Conversations sorts connections by bytes and duration, which finds the heavy talker in seconds. Statistics > I/O Graphs with a tcp.analysis.retransmission series shows whether loss is constant or arrives in bursts, which points to congestion or a flapping link respectively.

False Alarms from the Capturing Host

Captures taken on a busy server often show problems that do not exist on the wire:

• Bad checksums on outgoing packets. The NIC computes checksums after the capture point (checksum offload), so the captured copy has a placeholder. Wireshark leaves checksum validation off by default for this reason.
• Packets far larger than the MTU, such as 20,000-byte TCP segments. Segmentation offload (TSO/GSO) and receive coalescing (GRO) mean the kernel handles large chunks that the NIC splits or merges. For analysis that depends on real segment sizes, capture on a mirror port or temporarily disable offloads with ethtool -K eth0 tso off gso off gro off.

Scripting Captures

For large files or automated checks, use tshark to extract fields instead of scrolling:

# every retransmission with time, connection and stream number
tshark -r api.pcap -Y tcp.analysis.retransmission \
       -T fields -e frame.time_relative -e ip.src -e ip.dst -e tcp.stream

# a table of TCP conversations with bytes and durations
tshark -r api.pcap -q -z conv,tcp

For custom logic, read the file in Python. This scapy script summarises each TCP connection: handshake round trip, repeated SYNs, retransmissions and resets:

import sys
from collections import defaultdict
from scapy.all import IP, TCP, PcapReader

streams = defaultdict(lambda: {"syn": None, "rtt": None, "syns": 0, "retrans": 0,
                               "rst": False, "zero_win": False, "seen": set()})

with PcapReader(sys.argv[1]) as capture:              # streams packets, fine for big files
    for pkt in capture:
        if IP not in pkt or TCP not in pkt:
            continue
        ip, tcp = pkt[IP], pkt[TCP]
        ends = sorted([(ip.src, tcp.sport), (ip.dst, tcp.dport)])
        s = streams[tuple(ends)]                      # one key for both directions
        flags = str(tcp.flags)
        if flags == "S":
            s["syns"] += 1
            s["syn"] = s["syn"] or float(pkt.time)
        elif flags == "SA" and s["syn"]:
            s["rtt"] = float(pkt.time) - s["syn"]    # SYN -> SYN-ACK
        s["rst"] |= "R" in flags
        s["zero_win"] |= tcp.window == 0 and "R" not in flags
        size = len(tcp.payload)
        if size:
            segment = (ip.src, tcp.seq, size)         # same bytes sent twice = retransmission
            s["retrans"] += segment in s["seen"]
            s["seen"].add(segment)

for (a, b), s in streams.items():
    client, server = (a, b) if a[1] > b[1] else (b, a)    # higher port = client (heuristic)
    notes = [f"{s['syns']} SYNs" if s["syns"] > 1 else "",
             f"{s['retrans']} retransmitted" if s["retrans"] else "",
             "zero window" if s["zero_win"] else "", "reset" if s["rst"] else ""]
    rtt = f"{s['rtt'] * 1000:.0f} ms" if s["rtt"] else "no SYN-ACK"
    print(f"{client[0]}:{client[1]} -> {server[0]}:{server[1]:<5} {rtt:<11}",
          ", ".join(n for n in notes if n) or "clean")

Run against a small test capture containing a download with one lost segment, a connection to a filtered port and one to a closed port:

192.168.1.20:51514 -> 203.0.113.10:443   42 ms       1 retransmitted
192.168.1.20:51520 -> 203.0.113.10:8443  no SYN-ACK  2 SYNs
192.168.1.20:51522 -> 203.0.113.10:22    no SYN-ACK  reset

These are the three classic outcomes: a working connection with loss, a SYN silently dropped by a firewall, and a SYN refused with a RST. Wireshark's analysis is more thorough, so use scripts like this to triage many files, then open the interesting streams in the GUI.

Decrypting Your Own TLS Traffic

With TLS 1.3, and with any ECDHE key exchange, the server's private key cannot decrypt a capture: the session keys are ephemeral. What works is having the client log its session secrets. Firefox, Chrome and curl write them to the file named in the SSLKEYLOGFILE environment variable, and Python's ssl.create_default_context() honours the same variable:

import os
import ssl
import urllib.request

os.environ["SSLKEYLOGFILE"] = os.path.abspath("keys.log")   # before any context is created
ctx = ssl.create_default_context()                            # picks up SSLKEYLOGFILE

with urllib.request.urlopen("https://example.com/", context=ctx, timeout=10) as r:
    print(r.status, r.headers["Content-Type"])

for line in open("keys.log"):
    if line.startswith("#"):
        continue
    label, client_random, secret = line.split()
    print(f"{label:<32} {client_random[:12]}... {secret[:12]}...")

200 text/html; charset=utf-8
SERVER_HANDSHAKE_TRAFFIC_SECRET  d60d12a9c319... 8af8ab0d90f0...
EXPORTER_SECRET                  d60d12a9c319... 896e97111680...
SERVER_TRAFFIC_SECRET_0          d60d12a9c319... ca0ba87c69be...
CLIENT_HANDSHAKE_TRAFFIC_SECRET  d60d12a9c319... fbc925603ab4...
CLIENT_TRAFFIC_SECRET_0          d60d12a9c319... 865223a5f3a4...

Each line pairs the connection's ClientHello random (the same for all five) with one TLS 1.3 traffic secret; a TLS 1.2 connection logs a single CLIENT_RANDOM line instead. Capture while the script runs, then set Preferences > Protocols > TLS > (Pre)-Master-Secret log filename to keys.log, or pass -o tls.keylog_file:keys.log to tshark. The TLS records turn into readable HTTP, including HTTP/2 frames. To share a capture with the keys for just those sessions, embed them with editcap --inject-secrets tls,keys.log in.pcap out.pcapng. The same key log decrypts QUIC captures of your own browser, which helps when studying HTTP/2 and HTTP/3.

Decryption fails if the capture missed the handshake, so start capturing before the connection. Delete key logs when you finish: anyone with the file and the capture can read everything, and a browser left running with SSLKEYLOGFILE set logs keys for every site you visit. For interactive inspection of HTTP APIs, an intercepting proxy is often simpler, as described in mitmproxy and HAR files.

Practice

Capture on loopback while running python -m http.server 8000, fetch a page from it with curl, then curl a port where nothing listens. Find each connection with tcp.flags.syn == 1 && tcp.flags.ack == 0, follow the successful stream, and run the scapy script on the file. Then repeat the keylog example with your capture running and confirm the decrypted HTTP request appears.