The Grammar of a Real Browser#

Most scrapers obsess over the User-Agent header and stop there. In reality, a WAF can reconstruct an enormous amount of information from how your request is structured—independent of the actual header values. HTTP/2 made this even more granular, because the connection is framed and the frames themselves carry ordering information that HTTP/1.1 did not expose.

Header Ordering#

In HTTP/1.1, header names are case-insensitive and order is semantically irrelevant, so Python's requests alphabetizes them (Accept, Accept-Encoding, Connection, User-Agent...). Real browsers send a fixed, non-alphabetical sequence: :authority, :method, :path, :scheme, accept, user-agent, sec-ch-ua, and so on. WAFs fingerprint the sequence. Using a library like curl_cffi or driving a real browser keeps this ordering identical to the real thing.

Blueprint Headers#

Modern browsers advertise their capabilities through the User-Agent Client Hints family: - sec-ch-ua: a structured list of browser brand and version ("Chromium";v="124", "Google Chrome";v="124"). - sec-ch-ua-mobile: ?0 for desktop. - sec-ch-ua-platform: the OS ("Windows", "macOS").

These three headers must perfectly agree with the User-Agent and with each other. Claiming Chrome on Android in sec-ch-ua-mobile while posting a desktop resolution is an instant contradiction.

The Full Request Grammar#

A believable request aligns every field the WAF can cross-check:

  1. User-Agent and sec-ch-ua family name/major version and platform.
  2. Accept headers appropriate to the content type (an HTML page fetch should send text/html first, then other types; a JSON API call should send application/json first).
  3. Accept-Encoding: modern Chrome advertises gzip, deflate, br, zstd, not just gzip, deflate.
  4. Accept-Language: a single primary language tag with quality weights, consistent with your IP's country.
  5. Referer / Origin: an Origin header on a top-level navigation where a browser would not send one is a red flag; a missing Referer mid-workflow can be another.
  6. Standard headers: Connection, Cache-Control, Upgrade-Insecure-Requests, and the TLS-related hints your browser version normally emits.

Missing Header = Anomaly#

While bots usually fake too much, they also fake too little. Real Chrome always sends specific boilerplate that naive scrapers omit—Upgrade-Insecure-Requests: 1, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest, and a correct Priority hint. The Sec-Fetch trio is especially useful to WAFs because it encodes the navigation context: a top-level document navigation sends Sec-Fetch-Dest: document, while a background fetch sends Sec-Fetch-Dest: empty. Mismatching these with the actual HTTP method and referrer can expose a scraper even when everything else is perfect.

Connection Behavior Is Part of the Fingerprint#

Browsers multiplex many requests over one HTTP/2 stream and reuse TCP connections. If your scraper opens a fresh TLS connection for every request, your connection-per-request ratio looks nothing like a browser. Real WAF analysis includes connection reuse frequency, the number of parallel streams, and whether you follow redirects with the same TLS session—all reasons to keep a persistent, pooled curl_cffi or Playwright context rather than a fresh connection per call.

Reuse Browser SDKs Instead of Replaying Headers#

Manually recreating this grammar is brittle: every browser release changes the header set slightly. The robust approach is to capture the exact request pattern your chosen real browser produces (via Playwright or puppeteer with an intercepted network log), then replicate it verbatim in a fast HTTP client. Intercept, capture, replay—treating the browser's real header grammar as a reusable template keeps your scraper aligned with whatever the browser vendor currently ships.

The SETTINGS Frame Is Not a Header#

The most durable HTTP-layer fingerprint is not in the request at all. When an HTTP/2 connection is established, the client sends a SETTINGS frame that states its table sizes, window sizes and frame limits. It is emitted by the transport stack, once per connection, before any application code runs, and it is generated by a library rather than by whoever configured the client.

Because it is a frame rather than a header, there is no request-scoped place to override it. You cannot add it, remove it, or set it per profile without touching the HTTP/2 implementation. Hash the frame and you have a stable client identifier that survives every header you might rewrite.

# The SETTINGS frame is a fingerprint that no header can influence, and the
# request header block is a grammar rather than a bag of values. Both are
# cheap for a server to check and cheap for you to verify locally.

import hashlib

FRAME_HEADER = 9           # 3 length, 1 type, 1 flags, 4 stream id

SETTINGS = {
    "chrome-131": [(0x1, 65536), (0x2, 0), (0x4, 6291456), (0x5, 16384),
                   (0x8, 1)],
    "firefox-133": [(0x1, 65536), (0x2, 0), (0x4, 131072), (0x5, 16384)],
    "go-1.23": [(0x1, 4096), (0x2, 0), (0x4, 1073741823), (0x5, 16384)],
    "h2-python": [(0x1, 4096), (0x4, 65535), (0x5, 16384)],
    "nginx-1.27": [(0x1, 4096), (0x2, 0), (0x4, 65535), (0x5, 16384)],
}

SETTINGS_NAME = {
    0x1: "HEADER_TABLE_SIZE", 0x2: "ENABLE_PUSH", 0x3: "MAX_CONCURRENT_STREAMS",
    0x4: "INITIAL_WINDOW_SIZE", 0x5: "MAX_FRAME_SIZE", 0x6: "MAX_HEADER_LIST_SIZE",
    0x8: "ENABLE_CONNECT_PROTOCOL",
}

# Chrome 131 sends ENABLE_CONNECT_PROTOCOL last; nothing else here does
PLAUSIBLE_INITIAL_WINDOW = {
    "chrome-131": 6291456, "firefox-133": 131072, "go-1.23": None,
    "h2-python": None, "nginx-1.27": None,
}

REQUESTS = {
    "chrome-131": {
        "h2": True, "claims": "chrome-131", "locale": "de-DE",
        "authority": "shop.example", "host": "shop.example",
        "pseudo": [":method", ":authority", ":scheme", ":path"],
        "sec-ch-ua": ["Chromium", "Not_A Brand", "Google Chrome"],
        "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"',
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
                  "image/avif,image/webp,*/*;q=0.8",
        "accept-encoding": "gzip, deflate, br, zstd",
        "accept-language": "de-DE,de;q=0.9,en;q=0.8",
        "upgrade-insecure-requests": "1",
        "sec-fetch-dest": "document", "sec-fetch-mode": "navigate",
        "sec-fetch-site": "same-origin", "sec-fetch-user": "?1",
        "sec-fetch-site-none-duplicate": "",
    },
    "curl-8.9-over-h2": {
        "h2": True, "claims": "none", "locale": "",
        "authority": "shop.example", "host": "api.example",
        "pseudo": [":method", ":authority", ":scheme", ":path"],
        "sec-ch-ua": [], "sec-ch-ua-mobile": "", "sec-ch-ua-platform": "",
        "accept": "*/*", "accept-encoding": "", "accept-language": "",
        "upgrade-insecure-requests": "", "sec-fetch-dest": "",
        "sec-fetch-mode": "", "sec-fetch-site": "", "sec-fetch-user": "",
        "sec-fetch-site-none-duplicate": "",
    },
    "curl-over-boring-tls": {
        "h2": True, "claims": "chrome-131", "locale": "de-DE",
        "authority": "shop.example", "host": "shop.example",
        "pseudo": [":authority", ":method", ":scheme", ":path"],
        "sec-ch-ua": ["Chromium", "Not_A Brand", "Google Chrome"],
        "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"',
        "accept": "*/*", "accept-encoding": "", "accept-language": "de-DE,de;q=0.9",
        "upgrade-insecure-requests": "", "sec-fetch-dest": "",
        "sec-fetch-mode": "", "sec-fetch-site": "", "sec-fetch-user": "",
        "sec-fetch-site-none-duplicate": "",
    },
    "spoofed-chrome": {
        "h2": True, "claims": "chrome-131", "locale": "de-DE",
        "authority": "shop.example", "host": "shop.example",
        "pseudo": [":method", ":authority", ":scheme", ":path"],
        "sec-ch-ua": ["Chromium", "Not_A Brand", "Google Chrome"],
        "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"',
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "accept-encoding": "gzip, deflate, br",
        "accept-language": "en-US,en;q=0.9",
        "upgrade-insecure-requests": "", "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors", "sec-fetch-site": "cross-site", "sec-fetch-user": "",
        "sec-fetch-site-none-duplicate": "",
    },
}

# A real Chrome navigation request sends this exact set, in this shape.
CHROME_NAV_FETCH = {
    "sec-fetch-dest": "document", "sec-fetch-mode": "navigate",
    "sec-fetch-site": "same-origin", "sec-fetch-user": "?1",
}
CHROME_PSEUDO_ORDER = [":method", ":authority", ":scheme", ":path"]
CHROME_BRAND_COUNT = 3


def settings_frame(entries):
    # 9-byte frame header then 6 bytes per entry, in the order sent.
    payload = b"".join(
        b"\x00\x04" + v.to_bytes(4, "big") for _i, v in entries)
    frame = (len(payload)).to_bytes(3, "big") + b"\x04" + b"\x00" + \
        b"\x00\x00\x00\x00" + payload
    assert len(frame) == FRAME_HEADER + len(payload)
    return frame


def describe_settings(name, entries):
    frame = settings_frame(entries)
    digest = hashlib.sha256(frame).hexdigest()
    pairs = ["%s=%d" % (SETTINGS_NAME.get(i, "0x%x" % i), v) for i, v in entries]
    return len(entries), len(frame), digest, pairs


def audit(name, r):
    # Assumes every audited request is a top-level document GET. That
    # assumption is the point: the sec-fetch-* triple is only wrong in
    # relation to what the request is for, and a fetch() call legitimately
    # sends dest=empty and mode=cors.
    out = []
    claims_browser = r["claims"].startswith("chrome")
    if claims_browser:
        if len(r["sec-ch-ua"]) != CHROME_BRAND_COUNT:
            out.append("sec-ch-ua brand count is %d, chrome sends %d"
                       % (len(r["sec-ch-ua"]), CHROME_BRAND_COUNT))
        if not r["sec-fetch-dest"]:
            out.append("claims chrome but sends no sec-fetch-* set")
        else:
            for field, want in sorted(CHROME_NAV_FETCH.items()):
                if r[field] != want:
                    out.append("%s is %r on a document request, chrome sends %r"
                               % (field, r[field], want))
        if r["pseudo"] != CHROME_PSEUDO_ORDER:
            out.append("pseudo-header order is %s, chrome sends %s"
                       % (",".join(r["pseudo"]), ",".join(CHROME_PSEUDO_ORDER)))
    if not claims_browser:
        out.append("sends no client hints at all: no browser, whatever the tls")
    if "br" not in r["accept-encoding"]:
        out.append("no brotli in accept-encoding")
    if r["accept"].startswith("*/*"):
        out.append("accept is */*: a client that accepts anything, "
                   "no browser does this")
    if r["locale"] and not r["accept-language"].startswith(r["locale"]):
        out.append("accept-language is %r, profile locale is %r"
                   % (r["accept-language"], r["locale"]))
    if r["authority"] != r["host"]:
        out.append("authority %r disagrees with host %r"
                   % (r["authority"], r["host"]))
    if claims_browser and not r["upgrade-insecure-requests"]:
        out.append("no upgrade-insecure-requests on a claimed chrome navigation")
    return out


def main():
    print("http/2 settings, the frame a server hashes before it reads a header")
    print(f"{'client':<14} {'entries':>7} {'frame':>6}  {'sha256[:16]':<18} order")
    digests = {}
    for name in sorted(SETTINGS):
        count, size, digest, pairs = describe_settings(name, SETTINGS[name])
        digests[name] = digest
        print(f"{name:<14} {count:>7} {size:>6}  {digest[:16]:<18} "
              f"{' '.join(pairs)}")
    print(f"  {len(set(digests.values()))} distinct settings fingerprints "
          f"from {len(digests)} clients")
    print()

    print("the same five clients, and what a window size alone gives away")
    for name in sorted(SETTINGS):
        want = PLAUSIBLE_INITIAL_WINDOW[name]
        found = dict((i, v) for i, v in SETTINGS[name]).get(0x4)
        if want is None:
            note = "library default, and the value tells you which library"
        elif found == want:
            note = "matches the engine"
        else:
            note = "MISMATCH, engine sends %d" % want
        print(f"  {name:<14} initial_window_size={found:<11} {note}")
    print()

    print("request grammar audit: what a server checks before it believes a ua")
    for name in sorted(REQUESTS):
        findings = audit(name, REQUESTS[name])
        print(f"  {name:<22} {'clean' if not findings else str(len(findings)) + ' finding(s)'}")
        for finding in findings:
            print(f"      - {finding}")
    print()

    print("why this beats reading the user agent")
    print("  a user agent is one string, one field, and it is sent by")
    print("  whatever client the operator configured. everything above is")
    print("  either generated by code the operator did not write (the")
    print("  settings frame, the pseudo-header order, the hint defaults)")
    print("  or implied by the transport that carried the request.")
    print("  so agreeing with the ua is necessary and nowhere near")
    print("  sufficient, which is the whole argument for treating these")
    print("  as a grammar rather than as a list of values to copy.")


main()

http/2 settings, the frame a server hashes before it reads a header
client         entries  frame  sha256[:16]        order
chrome-131           5     39  6740445f957e7f07   HEADER_TABLE_SIZE=65536 ENABLE_PUSH=0 INITIAL_WINDOW_SIZE=6291456 MAX_FRAME_SIZE=16384 ENABLE_CONNECT_PROTOCOL=1
firefox-133          4     33  0f8054261e4c7891   HEADER_TABLE_SIZE=65536 ENABLE_PUSH=0 INITIAL_WINDOW_SIZE=131072 MAX_FRAME_SIZE=16384
go-1.23              4     33  fdfefd2340d7911d   HEADER_TABLE_SIZE=4096 ENABLE_PUSH=0 INITIAL_WINDOW_SIZE=1073741823 MAX_FRAME_SIZE=16384
h2-python            3     27  dd925093c81d855c   HEADER_TABLE_SIZE=4096 INITIAL_WINDOW_SIZE=65535 MAX_FRAME_SIZE=16384
nginx-1.27           4     33  adaa26c91072af9b   HEADER_TABLE_SIZE=4096 ENABLE_PUSH=0 INITIAL_WINDOW_SIZE=65535 MAX_FRAME_SIZE=16384
  5 distinct settings fingerprints from 5 clients

the same five clients, and what a window size alone gives away
  chrome-131     initial_window_size=6291456     matches the engine
  firefox-133    initial_window_size=131072      matches the engine
  go-1.23        initial_window_size=1073741823  library default, and the value tells you which library
  h2-python      initial_window_size=65535       library default, and the value tells you which library
  nginx-1.27     initial_window_size=65535       library default, and the value tells you which library

request grammar audit: what a server checks before it believes a ua
  chrome-131             clean
  curl-8.9-over-h2       4 finding(s)
      - sends no client hints at all: no browser, whatever the tls
      - no brotli in accept-encoding
      - accept is */*: a client that accepts anything, no browser does this
      - authority 'shop.example' disagrees with host 'api.example'
  curl-over-boring-tls   5 finding(s)
      - claims chrome but sends no sec-fetch-* set
      - pseudo-header order is :authority,:method,:scheme,:path, chrome sends :method,:authority,:scheme,:path
      - no brotli in accept-encoding
      - accept is */*: a client that accepts anything, no browser does this
      - no upgrade-insecure-requests on a claimed chrome navigation
  spoofed-chrome         6 finding(s)
      - sec-fetch-dest is 'empty' on a document request, chrome sends 'document'
      - sec-fetch-mode is 'cors' on a document request, chrome sends 'navigate'
      - sec-fetch-site is 'cross-site' on a document request, chrome sends 'same-origin'
      - sec-fetch-user is '' on a document request, chrome sends '?1'
      - accept-language is 'en-US,en;q=0.9', profile locale is 'de-DE'
      - no upgrade-insecure-requests on a claimed chrome navigation

why this beats reading the user agent
  a user agent is one string, one field, and it is sent by
  whatever client the operator configured. everything above is
  either generated by code the operator did not write (the
  settings frame, the pseudo-header order, the hint defaults)
  or implied by the transport that carried the request.
  so agreeing with the ua is necessary and nowhere near
  sufficient, which is the whole argument for treating these
  as a grammar rather than as a list of values to copy.

Five Clients, Five Fingerprints, One Cheap Field#

Five SETTINGS descriptions produce five distinct digests, from a total of nineteen settings values across all of them. The reason is that the values are library defaults, and libraries disagree. h2-python sends three entries, the browsers send four or five, and no two clients in the table agree on INITIAL_WINDOW_SIZE.

That one field does most of the work. Chrome sends 6291456, Firefox sends 131072, Go sends 1073741823, and two of the remaining clients send 65535. A server does not need the digest to classify the client: a single INITIAL_WINDOW_SIZE narrows the field to a browser family in most deployments, and it costs one integer comparison.

The practical implication is uncomfortable for anyone planning to run an automated HTTP/2 client against a modern site. The distinctive value is emitted by the library you chose, which means the choice of library is itself a fingerprint. Wrapping an HTTP/1.1 client in TLS termination in front of a real browser, or driving the browser itself, is the only way to inherit a plausible value.

The Request Header Block Is a Grammar#

Headers are usually discussed as a list of values to copy. They are better modelled as a grammar with production rules, because the server is checking production rules. The output's chrome-131 row is clean; the three rows below it are not, and none of them are caught by comparing a user agent string.

curl-8.9-over-h2 is the control case. It is honest: it sends no client hints, no brotli, and Accept: */*. Every one of those is correct for curl. The audit still produces four findings because the audit's job is to answer "would a browser send this", and the answer is no. That distinction matters, because these findings are not evidence of malice, only of absence.

curl-over-boring-tls is the interesting one: someone has added the Chrome client-hint trio to a curl request. It now claims to be a browser, which means the grammar is checked against Chrome's production rules and it fails five of them. Adding hints made the request worse, not better. This is the general shape of the failure: every field you add to impersonate a browser is a field you now have to keep consistent, and consistency is the expensive part.

The Four Findings That Matter Most#

spoofed-chrome is the realistic case and produces six findings, none of which mention the user agent. Two categories.

Cross-request consistency. The Sec-Fetch-* triple was set to dest=empty, mode=cors, site=cross-site, which is a perfectly coherent description of a fetch() call. It is incoherent for a top-level document GET. The server does not need to guess whether this is a navigation: the method, the Sec-Fetch-Dest and the Sec-Fetch-Mode are three fields that have to agree, and here two of them agree with each other and disagree with the third. Note the modelling assumption in the code: the audit assumes a document GET, which is why these are findings. Run the same request as a fetch() and the audit should and would pass it. A production audit has to be told the request type, and an auditor that does not know the request type cannot produce findings.

Cross-profile consistency. Accept-Language is en-US,en;q=0.9 on a de-DE profile. Nothing about this request is wrong in isolation; it is wrong against the profile it claims to belong to. This is the same coherence requirement that governs fingerprint cross-validation and risk scoring, expressed at the header layer, and it is why locale has to be decided once per profile rather than per request.

The remaining finding is the absence of Upgrade-Insecure-Requests: 1 on a claimed browser navigation, which is a field nobody thinks to set and which is present on essentially every real Chrome navigation.

Where the Grammar Ends and the Transport Begins#

Two of the checks above are transport-level rather than header-level, and they are the ones most implementations forget.

Pseudo-header order is part of this. Chrome sends :method, :authority, :scheme, :path; curl-over-boring-tls sends :authority first. The RFC does not require this order, so it is a free choice by the client, and free choices are exactly what a fingerprint is made of. The same applies to every other field the specification left free: header order, the order of Sec-CH-UA brands, whether client hints are sent at all on a given navigation.

Authority agreement is the second. :authority disagreesing with Host is unusual rather than illegal, and HTTP/2 prefers :authority while HTTP/1.1 prefers Host, so a client that populates both from different config files produces a mismatch that looks like credential forwarding. Many clients let you set them independently, which is exactly why they end up inconsistent.

Beyond headers, the server also knows the connection behaviour: stream concurrency, how many streams you opened per connection, window update patterns, and whether you respect the flow-control window it advertised. HTTP/3 and QUIC raises the resolution of all of this, because flow control becomes a per-connection and per-stream negotiation that a client library sets for you.

A Checklist That Survives a Library Upgrade#

The audit above is short on purpose, and it generalises to a set of rules that stay valid when libraries change. Check that Accept-Encoding includes every codec the engine supports, since the supported set is a versioned property of the browser rather than a preference. Check that Accept is a real media-type list with quality values and not */*. Check that the client-hint set is complete for the browser it claims, including the low-entropy Sec-CH-UA-Mobile and Sec-CH-UA-Platform that are absent from most copied examples. Check that Sec-Fetch-* matches the request's actual purpose. Check Accept-Language against the profile locale. Check :authority against Host. And check the SETTINGS frame against a reference capture for the engine version you claim.

The reason to prefer this list over a copied header blob is that it is falsifiable. A copied blob is correct until the browser ships an update, at which point it is wrong in every field at once and there is no signal telling you which. A grammar can be evaluated against the current engine, per version, in CI, which is the same argument made in CDP instrumentation leaks about not hard-coding expectations you cannot re-derive.