The Hidden Header That Says "Trust Me"

Many mobile and single-page apps do not just send an authentication token; they sign each request with a cryptographic digest of its contents. The signature proves the request was produced by the legitimate client and was not tampered with in transit. For a scraper, a signed request is a puzzle: reproduce the signing algorithm exactly or the server rejects you.

Why Signing Exists

A signed request binds the HTTP method, path, query parameters, body, and a timestamp together with a secret or private key. The server recomputes the signature and compares. This prevents:

  • Parameter tampering: changing a price or user ID invalidates the signature.
  • Replay attacks: the timestamp and a nonce make each signature unique and short-lived.
  • Request forgery: without the key, you cannot produce a valid signature.

The Anatomy of HMAC Signing

The most common scheme is HMAC-SHA256. The client builds a canonical string—a precisely specified concatenation of method, path, sorted query parameters, headers, and body hash—then computes HMAC(secret, canonical_string) and sends the hex digest in a header like X-Signature.

import hmac, hashlib, time

def sign(method, path, query, body, secret, timestamp=None):
    ts = timestamp or str(int(time.time()))
    canonical = "\n".join([method.upper(), path, query, body, ts])
    digest = hmac.new(secret.encode(), canonical.encode(), hashlib.sha256).hexdigest()
    return digest, ts

The hard part is never the HMAC; it is discovering the exact canonicalization rules. A single missing newline, a differing parameter sort order, or a URL-encoded value that should be raw will produce a mismatched signature.

AWS SigV4 as a Reference Model

AWS Signature Version 4 is the canonical example of request signing done rigorously: it defines a canonical request (method, canonical URI, canonical query string, canonical headers, signed headers, payload hash), hashes it, builds a string-to-sign with the timestamp and scope, and derives a signing key through a chain of HMACs. Studying SigV4 teaches you the standard structure that most in-house signing schemes imitate.

Reverse Engineering a Signer

When an app signs requests, the algorithm lives in its code. To reproduce it:

  1. Capture several real requests and note which headers vary with the body, the path, and time.
  2. Diff requests where only one input changed to infer which fields participate in the signature.
  3. Decompile the client (see the reverse-engineering lesson) and locate the signing function—search for the crypto imports, the header name, or constants like sha256/hmac.
  4. Extract the key. The secret is usually embedded, obfuscated, or derived; finding it is the crux.
  5. Reimplement and test against captured requests until your signature matches exactly. Byte-level equality is the goal.

Protecting Your Implementation

Once you can sign, keep the key secret, reproduce the timestamp and nonce handling precisely, and watch for short signature windows—if your clock drifts, valid requests start failing. If the signing algorithm changes after an app update, treat it as maintenance debt, because you now own a piece of reverse-engineered logic. And remember the boundary: reproducing a signature to access public data at a polite rate is a scraping technique; forging signatures to impersonate users or bypass authorization is a different act entirely.