HTTP and HTTPS

The Language of the Web#

The Hypertext Transfer Protocol (HTTP) is the application-layer protocol that browsers, APIs, mobile apps and scrapers use to ask servers for resources and send data back. It defines how messages are formatted, what the methods and status codes mean, and how intermediaries such as caches and proxies may handle them.

HTTP is a request-response protocol: a client sends a request, a server returns exactly one response. It is also stateless: each request carries everything the server needs, and the server keeps no memory of earlier requests by default. Logins and shopping carts are built on top with cookies and tokens (see Cookies & HTTP Sessions).

What a Request Actually Looks Like#

HTTP/1.1 is plain text over a TCP connection. You can speak it with nothing but a socket:

import socket

request = (
    "GET / HTTP/1.1\r\n"
    "Host: example.com\r\n"
    "User-Agent: raw-socket-demo/1.0\r\n"
    "Accept: text/html\r\n"
    "Connection: close\r\n"
    "\r\n"
)

with socket.create_connection(("example.com", 80), timeout=5) as sock:
    sock.sendall(request.encode("ascii"))
    response = b""
    while chunk := sock.recv(4096):
        response += chunk

head, _, body = response.partition(b"\r\n\r\n")
print(head.decode("iso-8859-1"))
print(f"\n[{len(body)} body bytes]")

Output (header values vary):

HTTP/1.1 200 OK
Date: Wed, 30 Sep 2026 05:32:06 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Last-Modified: Mon, 28 Sep 2026 16:19:32 GMT
Age: 3658

Things this reveals that libraries normally hide:

• Lines end with \r\n (CRLF), and an empty line separates headers from the body. Forgetting the final blank line makes the server wait forever for more headers.
• Host is mandatory in HTTP/1.1. One IP address can serve thousands of sites, and this header tells the server which one you want.
• Connection: close asks the server to close the socket after responding, which is why the read loop ends. Without it, HTTP/1.1 keeps the connection open for reuse (keep-alive is the default), and the loop would block.
• The body length is signalled either by Content-Length or by Transfer-Encoding: chunked, where the body arrives as hex-size-prefixed chunks. The printed "body bytes" above include those chunk-size lines, so a naive socket client would corrupt the content. Getting message framing exactly right is hard enough that disagreements between a proxy and a backend about where one request ends cause HTTP request smuggling.

The fastest way to see the same exchange with any real site is curl -v https://example.com -o /dev/null: lines starting with > are the request, < the response, and * curl's own notes about DNS, TCP and TLS.

Methods#

Safe means the request should not change server state, so crawlers, prefetchers and link previewers may issue it freely. That is why a "delete" link implemented as a GET is a real bug: a crawler following links can wipe data.

Idempotent means sending the request twice leaves the server in the same state as sending it once. It governs retries. A client whose GET or PUT times out can retry safely; a client whose POST times out cannot know whether the order was created. Payment and order APIs solve this with an idempotency key: the client generates a unique ID per logical operation and sends it in a header (commonly Idempotency-Key), and the server stores the result per key and returns the same result for duplicates.

Status Codes#

The first line of the response carries a three-digit status code. The classes:

• 1xx informational: 101 Switching Protocols (used to upgrade to WebSockets).
• 2xx success: 200 OK, 201 Created (with a Location header), 204 No Content.
• 3xx redirection: 301 and 308 permanent, 302 and 307 temporary, 304 Not Modified for cache revalidation.
• 4xx client error: the request is wrong and retrying unchanged will not help.
• 5xx server error: something failed on the server side; retrying later may help.

The distinctions that matter in practice:

When you see 502 or 504, the error came from a proxy or load balancer, not from your application code. Look at the proxy's logs first; see Reverse Proxies and API Gateways.

Important Headers#

• Host: which site the client wants.
• User-Agent: the client's self-description; trivially forged, so never a security control.
• Accept, Accept-Encoding, Accept-Language: content negotiation. The server picks a representation and should say so in Vary.
• Content-Type / Content-Length: what the body is and how long.
• Authorization: credentials, often Bearer <token>.
• Cookie / Set-Cookie: client-side state.
• Cache-Control, ETag, Last-Modified: caching; see HTTP Caching.
• Location: target of a redirect or a newly created resource.
• Retry-After: how long to wait before retrying after a 429 or 503.

Header names are case-insensitive. HTTP/2 and HTTP/3 transmit them in lowercase, so code that looks up headers["Content-Type"] in a plain dict breaks; use the case-insensitive mappings your HTTP library provides.

HTTPS#

Plain HTTP sends everything, including passwords and cookies, in cleartext that anyone on the path can read or modify. HTTPS is HTTP inside a TLS connection. The TLS handshake authenticates the server with a certificate and agrees on symmetric keys, then every HTTP byte, headers included, is encrypted. An observer still sees the destination IP, and usually the hostname through the SNI field, but not the path, query string, headers or body. Details are in The TLS Handshake. Servers enforce HTTPS with a redirect from port 80 and the Strict-Transport-Security header, which tells browsers never to try plain HTTP for that host again.

HTTP/1.1, HTTP/2 and HTTP/3#

• HTTP/1.0 originally opened a new TCP connection per request, paying a handshake each time.
• HTTP/1.1 made connections persistent by default, but each connection carries one request at a time. A slow response blocks everything queued behind it, so browsers open several parallel connections per host (commonly six).
• HTTP/2 keeps the same methods, headers and status codes but uses a binary framing layer that multiplexes many concurrent streams over one TCP connection, with header compression (HPACK). It removes HTTP-level head-of-line blocking, but a single lost TCP segment still stalls every stream, because TCP delivers bytes strictly in order.
• HTTP/3 runs over QUIC, a transport built on UDP with TLS 1.3 integrated. Streams are independent at the transport level, so a lost packet delays only its own stream. The transport and TLS handshakes are combined into one round trip, and resumed connections can send data immediately (0-RTT). Header compression uses QPACK. See TCP vs UDP.

Clients discover HTTP/2 through ALPN during the TLS handshake, and HTTP/3 through the Alt-Svc response header or DNS HTTPS records. You can check what a server negotiates from Python with httpx (install httpx[http2] for HTTP/2 support):

import httpx

with httpx.Client(http2=True, timeout=10) as client:
    r = client.get("https://example.com/")
    print(r.http_version, r.status_code, r.headers.get("content-type"))

This prints HTTP/2 200 text/html; charset=utf-8 for servers that support HTTP/2 and HTTP/1.1 ... otherwise; curl -sI --http2 https://example.com shows the same thing from the shell.

Common Mistakes#

• Retrying non-idempotent requests blindly. Retry GET/PUT/DELETE on timeouts and 502/503/504; retry POST only with an idempotency key.
• Ignoring redirects' method rules. A 301/302 after a POST may turn into a GET without the body; use 307/308 if the method must be kept.
• Treating every non-200 as failure. 201, 204 and 304 are successes.
• No timeouts. Many HTTP libraries wait indefinitely by default. Always set connect and read timeouts.
• Reading the body without checking Content-Type and charset, which produces garbled text or JSON decode errors on error pages.

For the client-side craft of making these requests programmatically, the web-scraping course's HTTP basics lesson builds on everything here.

Practice#

Modify the socket example to remove Connection: close and add a second request on the same socket after the first response. Parse Content-Length or chunked framing yourself to know where the first response ends. Once it works, you will understand why keep-alive and correct framing are the hardest part of HTTP/1.1.