The Grammar of a Real Browser

Most scrapers obsess over the User-Agent header and stop there. In reality, a WAF can reconstruct an enormous amount of information from how your request is structured—independent of the actual header values. HTTP/2 made this even more granular, because the connection is framed and the frames themselves carry ordering information that HTTP/1.1 did not expose.

Header Ordering

In HTTP/1.1, header names are case-insensitive and order is semantically irrelevant, so Python's requests alphabetizes them (Accept, Accept-Encoding, Connection, User-Agent...). Real browsers send a fixed, non-alphabetical sequence: :authority, :method, :path, :scheme, accept, user-agent, sec-ch-ua, and so on. WAFs fingerprint the sequence. Using a library like curl_cffi or driving a real browser keeps this ordering identical to the real thing.

Blueprint Headers

Modern browsers advertise their capabilities through the User-Agent Client Hints family: - sec-ch-ua: a structured list of browser brand and version ("Chromium";v="124", "Google Chrome";v="124"). - sec-ch-ua-mobile: ?0 for desktop. - sec-ch-ua-platform: the OS ("Windows", "macOS").

These three headers must perfectly agree with the User-Agent and with each other. Claiming Chrome on Android in sec-ch-ua-mobile while posting a desktop resolution is an instant contradiction.

The Full Request Grammar

A believable request aligns every field the WAF can cross-check:

  1. User-Agent and sec-ch-ua family name/major version and platform.
  2. Accept headers appropriate to the content type (an HTML page fetch should send text/html first, then other types; a JSON API call should send application/json first).
  3. Accept-Encoding: modern Chrome advertises gzip, deflate, br, zstd, not just gzip, deflate.
  4. Accept-Language: a single primary language tag with quality weights, consistent with your IP's country.
  5. Referer / Origin: an Origin header on a top-level navigation where a browser would not send one is a red flag; a missing Referer mid-workflow can be another.
  6. Standard headers: Connection, Cache-Control, Upgrade-Insecure-Requests, and the TLS-related hints your browser version normally emits.

Missing Header = Anomaly

While bots usually fake too much, they also fake too little. Real Chrome always sends specific boilerplate that naive scrapers omit—Upgrade-Insecure-Requests: 1, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest, and a correct Priority hint. The Sec-Fetch trio is especially useful to WAFs because it encodes the navigation context: a top-level document navigation sends Sec-Fetch-Dest: document, while a background fetch sends Sec-Fetch-Dest: empty. Mismatching these with the actual HTTP method and referrer can expose a scraper even when everything else is perfect.

Connection Behavior Is Part of the Fingerprint

Browsers multiplex many requests over one HTTP/2 stream and reuse TCP connections. If your scraper opens a fresh TLS connection for every request, your connection-per-request ratio looks nothing like a browser. Real WAF analysis includes connection reuse frequency, the number of parallel streams, and whether you follow redirects with the same TLS session—all reasons to keep a persistent, pooled curl_cffi or Playwright context rather than a fresh connection per call.

Reuse Browser SDKs Instead of Replaying Headers

Manually recreating this grammar is brittle: every browser release changes the header set slightly. The robust approach is to capture the exact request pattern your chosen real browser produces (via Playwright or puppeteer with an intercepted network log), then replicate it verbatim in a fast HTTP client. Intercept, capture, replay—treating the browser's real header grammar as a reusable template keeps your scraper aligned with whatever the browser vendor currently ships.