HTTP Basics for Scraping

Understanding the Backbone of the Web#

Every scraper is an HTTP client. A browser is also an HTTP client, one that happens to have a rendering engine and a JavaScript runtime bolted on. When you scrape, you replace the browser with code that sends the same requests and reads the same responses, so the first skill is being able to read and write HTTP messages by hand.

When a server receives your request, it answers with a status code, a set of headers, and usually a body (HTML, JSON, an image). Most scraping bugs are HTTP bugs in disguise: a redirect you did not notice, a compressed body you could not decode, a status code you treated as success.

Anatomy of a Request and a Response#

HTTP/1.1 is a plain-text protocol, and seeing a raw exchange once makes everything else easier to reason about:

GET /products?page=2 HTTP/1.1
Host: shop.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/124.0 Safari/537.36
Accept: text/html,application/xhtml+xml
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate
Cookie: session=abc123

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Set-Cookie: session=abc123; Path=/; HttpOnly
Cache-Control: max-age=60

<!doctype html> ...

A request has three parts:

1. The request line: the method (GET, POST), the path plus query string (/products?page=2), and the protocol version.
2. Headers: key-value metadata about the client and what it will accept. Host is mandatory in HTTP/1.1; the ones that matter most for scraping are User-Agent, Accept, Accept-Language, Accept-Encoding, Referer, and Cookie.
3. The body: empty for GET, used by POST/PUT/PATCH to send form data or JSON.

HTTP/2 carries the same semantics (method, path, headers, body) in a binary, multiplexed framing, and header names are lowercase on the wire. Python's requests speaks only HTTP/1.1; httpx can speak HTTP/2 if you install the h2 extra and pass http2=True. For most scraping the version does not change what data you get, but it does change how your client looks to anti-bot systems, which the TLS fingerprinting lesson covers.

Methods You Will Actually Use#

• GET retrieves a resource. Parameters travel in the query string. Safe to retry.
• POST submits data: login forms, search forms, GraphQL queries, many internal APIs. Parameters travel in the body, either form-encoded (data= in requests) or JSON (json=).
• HEAD returns only headers. Useful for checking Content-Length or Last-Modified before downloading a large file, though some servers handle it badly.
• PUT/PATCH/DELETE modify state. A read-only scraper should almost never send them.

The distinction between idempotent (GET, HEAD, PUT, DELETE) and non-idempotent (POST) methods matters when you add retries: retrying a GET is harmless, retrying a POST that places an order or submits a form may do it twice.

Status Codes as a Decision Table#

The status code is the first thing your code should branch on. Group them by what your scraper should do next:

The error handling lesson turns this table into retry code. The key habit is that "2xx" does not mean "correct data" and "4xx" does not all mean the same thing.

Why Headers Matter#

By default requests sends User-Agent: python-requests/2.x and httpx sends python-httpx/0.x. Many sites reject those outright because they cannot be anything other than a script. You have two honest options: a descriptive bot identity with contact details (MyDataBot/1.0 (+contact: data@mycompany.example)), which the robots.txt and ethics lesson recommends wherever it works, or a browser-like header set when the site only serves normal browsers.

If you send browser-like headers, send a coherent set. A Chrome User-Agent paired with no Accept-Language and Accept: */* looks like nothing a real Chrome ever sends.

import requests

url = "https://example.com/api/data"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.9",
}

response = requests.get(url, headers=headers, timeout=(3, 15))
print("status:", response.status_code)
print("sent headers:", dict(response.request.headers))

response.request.headers shows what was actually sent, including headers the library added for you (Accept-Encoding, Connection). Checking it is the fastest way to find out why your "identical" request behaves differently from the browser's.

Reading the Response Like a Postmortem#

Before writing a single selector, inspect what the server actually sent. The status code tells you the outcome; the headers and redirect chain tell you what your scraper will face next.

import httpx

resp = httpx.get("https://example.com/products", timeout=15.0, follow_redirects=True)
print("status:", resp.status_code)
print("final url:", resp.url)
print("content-type:", resp.headers.get("content-type"))
print("redirect chain:", [(r.status_code, r.headers["location"]) for r in resp.history])
print("elapsed:", resp.elapsed.total_seconds(), "seconds")

Note follow_redirects=True. Unlike requests, httpx does not follow redirects by default, so without it you get the 301 itself, an empty history, and a body that is just a stub. That single default difference explains a lot of "httpx returns nothing" bug reports.

Typical output for a page that has moved looks like:

status: 200
final url: https://www.example.com/catalog/products
content-type: text/html; charset=utf-8
redirect chain: [(301, 'https://www.example.com/products'), (301, '/catalog/products')]
elapsed: 0.412 seconds

The headers worth reading on every new target:

• content-type tells you whether to parse HTML or call .json(), and which charset the text is in.
• content-encoding says the body was compressed (gzip, deflate, br, zstd); the client decompresses for you, but only for encodings it supports.
• set-cookie issues session state; see cookies and state.
• retry-after is the server's precise instruction for how long to wait when rate limiting you.
• cache-control, etag, and last-modified let you make conditional requests and skip unchanged pages.

How HTTP Goes Wrong in Scrapers#

Compressed garbage. If you copy Accept-Encoding: gzip, deflate, br from a browser but the brotli package is not installed, the server may send Brotli and requests hands you undecoded bytes. resp.text looks like random symbols. Either install brotli or advertise only gzip, deflate.

Wrong character encoding. requests takes the charset from Content-Type; if the header omits it for a text/* type, it falls back to ISO-8859-1 and accented characters turn into mojibake like Café. Compare resp.encoding with resp.apparent_encoding, or decode resp.content yourself using the <meta charset> in the page.

Silent redirects. A product URL that redirects to the homepage or a login page returns 200 after following, and your selectors quietly find nothing. Log resp.url and compare it with the URL you asked for.

Hanging requests. Neither library applies a total deadline for you: requests has no timeout at all unless you pass one, and one stalled server can freeze a whole crawl. Always pass timeout.

Debugging With curl#

When Python and the browser disagree, drop down to curl, which shows the raw exchange:

curl -sv -o /dev/null -H "Accept-Language: en-US" https://example.com/products 2>&1 | grep -E "^[<>]"
curl -sI https://example.com/products        # headers only
curl -sL -w "%{http_code} %{url_effective}" -o /dev/null https://example.com/old-path

Lines starting with > are what you sent, < is what came back. The last command follows redirects and prints the final status and URL, a quick way to map redirect behavior across many URLs.

Practice#

Pick any public page you are allowed to crawl. Fetch it with requests and httpx (with and without follow_redirects), print status, final URL, content-type, content-encoding, and the sent headers for each, then reproduce the same request with curl -v. Explain every difference you see in one sentence each.