TCP Handshake & Control Flags

The Conversation Before the Conversation#

TCP's reliability is famous, but it starts with a formal negotiation. Before two hosts exchange a byte of application data they perform a three-way handshake to agree that the connection exists and to synchronize sequence numbers. The handshake, the flags that drive it, and the states a connection moves through are the foundation of network debugging: most "the connection hangs" or "connection reset" mysteries can be diagnosed from a handful of packets.

The Three-Way Handshake#

1. SYN: The client sends a segment with the SYN flag and a random initial sequence number (ISN), say SEQ=1000. "I want to talk; my byte numbering starts at 1000."
2. SYN-ACK: The server replies with SYN and ACK set: its own ISN (SEQ=5000) and an acknowledgment of the client's (ACK=1001). "Heard you; I expect your next byte at 1001; my numbering starts at 5000."
3. ACK: The client acknowledges (SEQ=1001, ACK=5001). The connection is ESTABLISHED on both sides.

A SYN consumes one sequence number even though it carries no data, which is why the acknowledgments are ISN + 1. The ISNs are randomized because predictable ones once allowed TCP sequence prediction attacks, where an off-path attacker injected forged segments into a connection.

The SYN and SYN-ACK also negotiate options that shape the whole connection:

• MSS (maximum segment size): the largest payload each side wants to receive, usually 1460 on Ethernet.
• Window scale: multiplies the 16-bit window field so windows larger than 64 KB are possible, essential on fast, long-distance links.
• SACK permitted: allows selective acknowledgment of out-of-order data, so only the missing pieces are resent.
• Timestamps: used for round-trip measurement and to protect against wrapped sequence numbers.

If a middlebox strips these options, performance collapses without any error, so they are worth checking in captures.

Seeing It in tcpdump#

sudo tcpdump -n -i any 'tcp port 443 and host 203.0.113.10'

10:00:00.000100 IP 192.168.1.20.51514 > 203.0.113.10.443: Flags [S], seq 1776230401, win 64240, options [mss 1460,sackOK,TS val 123 ecr 0,nop,wscale 7], length 0
10:00:00.020200 IP 203.0.113.10.443 > 192.168.1.20.51514: Flags [S.], seq 2910002211, ack 1776230402, win 65535, options [mss 1400,sackOK,TS val 456 ecr 123,nop,wscale 9], length 0
10:00:00.020260 IP 192.168.1.20.51514 > 203.0.113.10.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 124 ecr 456], length 0
10:00:00.020900 IP 192.168.1.20.51514 > 203.0.113.10.443: Flags [P.], seq 1:518, ack 1, win 502, length 517

tcpdump's flag letters: S SYN, . ACK, P PSH, F FIN, R RST, U URG, so [S.] is SYN-ACK and [P.] is PSH-ACK. After the handshake tcpdump switches to relative sequence numbers (ack 1, seq 1:518), which is easier to read; add -S for absolute ones. The 20 ms gap between SYN and SYN-ACK is the network round-trip time, the cheapest latency measurement you will ever get. The fourth line, 517 bytes pushed right after the handshake, is a TLS ClientHello. Also note the server advertised mss 1400: something on its side (a tunnel, perhaps) needs smaller segments.

To capture only connection setup and teardown on a busy host:

sudo tcpdump -n 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0'

The Control Flags#

RST deserves special attention because it arrives in several distinct situations: a SYN to a port with no listener, a packet for a connection the receiver does not know (for example, after a server reboot or a NAT timeout), an application closing a socket with unread data in its buffer, or a firewall configured to reject rather than drop. The error an application sees is ECONNREFUSED if the RST answers a SYN, and ECONNRESET ("connection reset by peer") if it arrives mid-connection.

Probing Ports from Python#

The three possible outcomes of a connection attempt map directly onto handshake behaviour:

import socket, time

def probe(host, port, timeout=3.0):
    start = time.monotonic()
    try:
        with socket.create_connection((host, port), timeout=timeout):
            result = "open (handshake completed)"
    except ConnectionRefusedError:
        result = "closed (RST received)"
    except socket.timeout:
        result = "filtered (no reply, SYN retransmitted until timeout)"
    except OSError as e:
        result = f"error: {e}"
    return f"{host}:{port:<5} {result:52} {time.monotonic() - start:.2f}s"

listener = socket.socket()
listener.bind(("127.0.0.1", 0))           # port 0: let the OS choose a free port
listener.listen()
open_port = listener.getsockname()[1]

print(probe("127.0.0.1", open_port))
print(probe("127.0.0.1", 9))              # almost certainly nothing listening
print(probe("10.255.255.1", 80))          # typically unrouted: packets vanish

127.0.0.1:41873 open (handshake completed)                           0.00s
127.0.0.1:9     closed (RST received)                                0.00s
10.255.255.1:80    filtered (no reply, SYN retransmitted until timeout) 3.00s

The timing is the diagnosis. A refusal on Linux is immediate because the kernel answers the SYN with RST; Windows retries a refused SYN a couple of times, so the same refusal takes around two seconds there. A timeout means SYNs vanished: a firewall silently dropping, a wrong route, or a host that is down. Without an explicit timeout, the OS keeps retransmitting the SYN with exponential backoff, which on Linux defaults to over two minutes, which is why every network client should set connect timeouts.

Connection Teardown and TIME_WAIT#

A graceful close is a four-way exchange because each direction closes independently: A sends FIN, B ACKs it, B sends its own FIN when done, A ACKs it. Between the first FIN and the second, the connection is half-closed, and B can still send data.

The side that closes first enters TIME_WAIT for twice the maximum segment lifetime (60 seconds on Linux) so that delayed duplicate segments cannot be mistaken for a new connection reusing the same four-tuple. TIME_WAIT is normal, but it has consequences:

• A client making many short connections to the same server can exhaust ephemeral ports (Linux's default range is 32768-60999), producing Cannot assign requested address errors. The real fix is connection reuse: HTTP keep-alive and connection pools.
• A server restarted while old connections are in TIME_WAIT may fail to bind with Address already in use unless it sets SO_REUSEADDR, which almost every server framework does.

Connection States#

ss -tan                      # all TCP sockets with state
ss -tan state time-wait | wc -l
ss -tni dst 203.0.113.10     # per-connection internals: rtt, cwnd, retransmits
ss -ltn                      # listening sockets

Growing CLOSE-WAIT counts are always an application bug: the peer said goodbye and your code never called close(). Look for connection objects not released on error paths.

For listening sockets, ss -ltn shows Recv-Q (connections completed but not yet accept()ed) and Send-Q (the backlog limit). A Recv-Q pinned at the limit means the application is not accepting fast enough, and new clients will see SYN retries or resets.

SYN Floods and SYN Cookies#

Between SYN-ACK and the final ACK, the server holds a half-open entry in its SYN queue. A SYN flood sends huge numbers of SYNs from spoofed addresses so the queue fills and legitimate clients cannot connect. SYN cookies defend against it: when the queue is full, the server encodes the connection parameters into its ISN and stores nothing, reconstructing the state only when a valid ACK returns. On Linux this is controlled by net.ipv4.tcp_syncookies and is on by default in common distributions. Large floods are handled upstream by providers; see DDoS.

Flow Control and the Window#

The receive window advertises how much buffer space the receiver has. A slow application lets its buffer fill, advertises a smaller window, and the sender slows down. At zero, the sender pauses and sends periodic window probes. In a capture, win 0 from the receiver means the receiving application is the bottleneck, not the network.

Congestion Control#

Separately, the sender estimates what the network can carry with a congestion window (cwnd in ss -tni). Slow start roughly doubles it each round trip until loss or a threshold, then congestion avoidance grows it gradually; loss signalled by duplicate ACKs triggers fast retransmit. Linux defaults to CUBIC; BBR models bandwidth and round-trip time instead of reacting to loss. This is why a download ramps up over its first second and why high-latency links need large windows to be fast.

Two related knobs affect small messages: Nagle's algorithm delays small writes to coalesce them, and delayed ACKs hold acknowledgments briefly. Together they can add tens of milliseconds to request-response protocols, which is why latency-sensitive software sets TCP_NODELAY.

Reading a Handshake While Debugging#

Combined with ss on both ends, this table turns "the network is slow" into a specific fault. See Network Monitoring & Troubleshooting, and TCP vs UDP for what you give up when a protocol skips all of this.

Practice#

Start sudo tcpdump -n -i lo 'tcp port 8000', run python -m http.server 8000, and fetch it with curl. Label every packet (handshake, request, response, FIN/ACK exchange), then note which side ended up in TIME-WAIT with ss -tan state time-wait. Stop the server and curl again to capture the RST.