Advanced WAF Evasion

Operational Security for Scrapers#

Bypassing a WAF on a single request is easy. Bypassing a WAF for a sustained 10-million-row extraction job over three weeks requires meticulous Operational Security (OpSec) and behavioral discipline.

WAFs utilize machine learning models that analyze aggregated traffic patterns. If you act like a machine, you will be banned like a machine.

1. Header Perfection and Ordering#

We know that spoofing the User-Agent is mandatory. But you must spoof the entire header block flawlessly. If your User-Agent claims to be Chrome 120 on macOS, but you send the Accept-Encoding: gzip, deflate header instead of Chrome's modern Accept-Encoding: gzip, deflate, br, zstd, you create an anomaly score. Furthermore, Header Ordering matters. Python's requests alphabetizes headers. Chrome sends them in a specific, rigid, non-alphabetical order. WAFs check this. Use libraries like curl_cffi that handle order automatically.

2. The Illusion of Human Timing#

Machines operate at the speed of the CPU. If you scrape a site with a hardcoded time.sleep(2) between requests, the WAF analyzes the timestamp deltas: 2.001s, 2.000s, 2.002s. Humans do not click links with millisecond perfection.

Implement Gaussian Jitter: Use a normal distribution to randomize your delays. Most requests will take around 3 seconds, but occasionally one will take 1 second, and another will take 7 seconds (simulating a user getting distracted).

import time
import random

def human_delay(mu=3.0, sigma=1.0):
    # Generates a delay following a bell curve, clamped between 1 and 8 seconds
    delay = max(1.0, min(8.0, random.gauss(mu, sigma)))
    time.sleep(delay)

3. Simulating Human Motor Functions#

If you are using Playwright/Selenium, never use the default .click(x, y) method. It teleports the mouse cursor instantly to the target coordinate. Datadome records mouse telemetry and will ban you for teleporting.

Use specialized libraries that generate Bézier curves to smoothly animate the mouse cursor across the screen, overshooting slightly, and correcting back to the target button—exactly how a human hand behaves.

4. The "Ghost Town" Approach#

If you need to scrape 100,000 pages, do not do it from a single IP at 50 requests a second. Use a massive residential proxy pool and deploy a distributed fleet of slow scrapers. If you have 500 IPs, and each IP only makes 1 request every 60 seconds, your aggregate throughput is extremely high, but from the WAF's perspective, they just see a bunch of distinct, incredibly slow, perfectly normal human users.

5. Session Shape#

Match your session shape to the workflow shape. A logged-in session that makes one call to a search endpoint, then 40 calls to detail pages, then logs out, looks human. A session that makes 400 calls to a single unauthenticated endpoint with no reads in between looks machine-like. Vary the shape naturally: interleave occasional home-page and "about" round trips, and carry cookies for the entire workflow instead of re-creating sessions per request.

6. Avoid Honeypot Traps#

Many WAFs embed invisible honeypots: links hidden with display: none or off-screen positions that real users can never find or click. A scraper that follows every link in the DOM will gleefully navigate into the honeypot and get flagged. Only traverse links on visible, interactive paths, and verify the target is not marked with the standard honeypot attributes.

7. Account for TLS Session Reuse#

Real users browse a site with one TLS connection kept alive across many requests. If your scraper opens a fresh TLS handshake per request, the connection-per-request ratio becomes a statistical outlier even when every packet looks perfect. Keep a persistent pooled connection (and, where possible, an HTTP/2 session) for the lifetime of a crawl batch.

What a Modern WAF Actually Rules On#

The old picture — a rule matches, you get 403 — is only the first layer. Current WAFs run a rule set in a fixed evaluation order, and knowing the order tells you which of your mistakes is even reachable.

Reputation and rate run first because they are nearly free for the origin to evaluate and they remove most traffic before anything expensive happens. The behavioural and model layers run last and are the only ones you can influence without changing your infrastructure. That ordering is why an ops mistake — bursting through an endpoint, or hammering an IP from an ASN with a bad history — burns you before a fingerprint problem ever gets looked at.

Timing Distributions, Not Timing Values#

The single most quoted piece of scraper folklore is "add jitter". Adding a few percent of jitter around a constant is worse than useless, because it converts an obviously synthetic constant into an obviously synthetic near-constant, and it costs you throughput. The distribution is the signature.

You can test this the same way a defender does, with a two-sample Kolmogorov–Smirnov statistic against a captured human session, bounded by the distribution-free DKW critical value:

import math
import random


def gauss(rng, mu=0.0, sigma=1.0):
    u1 = 1.0 - rng.random()
    u2 = rng.random()
    return mu + sigma * math.sqrt(-2.0 * math.log(u1)) * math.cos(2.0 * math.pi * u2)


def ks_two_sample(a, b):
    n, m = float(len(a)), float(len(b))
    biggest = 0.0
    for x in sorted(set(a) | set(b)):
        fa = sum(1 for v in a if v <= x) / n
        fb = sum(1 for v in b if v <= x) / m
        biggest = max(biggest, abs(fa - fb))
    return biggest


def dkw_epsilon(n):
    return math.sqrt(math.log(2.0) / (2.0 * n))


HUMAN = [11.4, 4.2, 0.9, 22.7, 3.6, 41.0, 6.1, 0.4, 9.8, 17.3,
         2.2, 55.6, 7.4, 1.1, 12.9, 30.5, 4.8, 0.6, 19.7, 8.3,
         27.1, 5.5, 0.3, 14.2, 38.8, 6.9, 2.4, 23.1, 10.7, 1.4,
         44.2, 9.1, 3.3, 16.8, 31.4, 7.7, 0.8, 20.6, 11.9, 5.2]

rng = random.Random(20260903)
N = len(HUMAN)
BIG = 400

ARMS = {
    "sleep(2.0) exactly": ([2.0] * N, N),
    "sleep(2.0) plus 2% jitter": ([2.0 * (1.0 + 0.02 * gauss(rng))
                                   for _ in range(N)], N),
    "gauss, mu 9.0 sigma 2.4": ([max(0.5, gauss(rng, 9.0, 2.4))
                                 for _ in range(N)], N),
    "lognormal, mu 1.95 sigma 0.9": ([math.exp(gauss(rng, 1.95, 0.9))
                                     for _ in range(N)], N),
    "lognormal, mu 1.95 sigma 0.9, 10x sample": (
        [math.exp(gauss(rng, 1.95, 0.9)) for _ in range(BIG)], BIG),
}

print("Inter-request gap distribution, two-sample KS against a 40-point capture")
print("")
print("%-36s %5s %8s %8s %6s  %s" % (
    "generator", "n", "eps(0.05)", "D", "D/eps", "verdict"))
for name in sorted(ARMS):
    seq, n = ARMS[name]
    eps = dkw_epsilon(min(n, N))
    d = ks_two_sample(HUMAN, seq)
    print("%-36s %5d %8.4f %8.4f %6.2f  %s" % (
        name, n, eps, d, d / eps, "same distribution" if d <= eps else "rejected"))
print("")
print("sample median / MAD, seconds")
pairs = [("human capture", HUMAN),
         ("lognormal synth", ARMS["lognormal, mu 1.95 sigma 0.9"][0])]
for label, seq in pairs:
    s = sorted(seq)
    med = s[len(s) // 2]
    mad = sorted(abs(v - med) for v in s)[len(s) // 2]
    print("  %-16s median %6.2f   mad %6.2f" % (label, med, mad))
print("")
print("the synthetic arm matches the capture's median to 0.05 s and still fails")
print("epsilon is set by the smaller sample: %.4f at n=10, %.4f at n=40, %.4f at n=400"
      % (dkw_epsilon(10), dkw_epsilon(40), dkw_epsilon(400)))

Inter-request gap distribution, two-sample KS against a 40-point capture

generator                                n eps(0.05)        D  D/eps  verdict
gauss, mu 9.0 sigma 2.4                 40   0.0931   0.3250   3.49  rejected
lognormal, mu 1.95 sigma 0.9            40   0.0931   0.1500   1.61  rejected
lognormal, mu 1.95 sigma 0.9, 10x sample   400   0.0931   0.1700   1.83  rejected
sleep(2.0) exactly                      40   0.0931   0.8250   8.86  rejected
sleep(2.0) plus 2% jitter               40   0.0931   0.8250   8.86  rejected

sample median / MAD, seconds
  human capture    median   9.10   mad   7.70
  lognormal synth  median   9.05   mad   5.61

the synthetic arm matches the capture's median to 0.05 s and still fails
epsilon is set by the smaller sample: 0.1862 at n=10, 0.0931 at n=40, 0.0294 at n=400

Read the table carefully, because the last row is the lesson. A lognormal generator reproduces the human capture's median to within 0.05 s and still gets rejected at every sample size, because matching the median and the spread is not matching the distribution — the tail is what differs. And the 2% jitter row scores identically to the constant row: both are a delta function to within the noise, so a small variance around a constant is indistinguishable from no variance at all.

The actionable conclusions are about effort allocation. You do not need a perfect generator; you need a heavy-tailed one. And the defender's own test gets more sensitive with sample size, so there is no sample size at which "close enough" becomes safe — which is an argument for targeting the whole session's shape rather than trying to win a per-request statistic.

What Generic Evasion Advice Gets Wrong#

Most published advice is written for 2015-era fingerprinting and is now either irrelevant or actively counterproductive.

Rotate user agents. Useless against any WAF that scores sessions, because a UA that changes while the TLS fingerprint, client hints, screen metrics and fonts stay fixed is a contradiction, and contradictions are cheaper to detect than the original sin. Worse, the client-hint headers must agree with the UA string, so rotation now requires maintaining a coherent device profile rather than a string list. The device-probe landscape is catalogued in .

Use headless=false. A launch flag does not remove the headless signals; it changes which set of headless signals you emit. Modern headless Chrome in the "new headless" mode differs from a headed browser in tens of small ways, none of which the flag touches. The framework-level differences are in and the launch-config audit is in .

Add a Referer header. Fine, but the referrer chain is what is checked, and a correct referrer attached to an impossible navigation history is a contradiction. Referrer synthesis is a corollary of session shape, not an independent trick.

Bypass with mobile user agents. Mobile UAs unlock more compatibility paths, not fewer checks — an iPhone UA that reports a 1366x768 viewport, no touch points and a macOS GPU is the single most-flagged profile class there is.

Use a residential proxy and stop. The proxy is one input to a multi-signal score, and it is the input you have least control over at volume, since it is shared with other customers. The ASN and provider classification layer is in .

The pattern across all of these: they treat detection as a checklist when it is a consistency audit. Every one of them makes your profile less coherent, not more.

The Cost Asymmetry Runs Both Ways#

An attacker pays per profile, per browser launch, per solve, per proxy byte, and per engineer-hour to keep a solver current. A defender ships a rule once and it applies to every request from every attacker, forever, at the cost of one line in a config.

This is why WAF capability has grown faster than evasion skill. A new signal costs the defender an afternoon: log it, join it against a labelled set, ship a weight. It costs an attacker days or weeks, because it invalidates an existing layer and forces a re-acquisition. A decade of that arithmetic produced the fingerprint surface documented in and the signal catalogue in .

The asymmetry is not uniform, and the exceptions are where the remaining opportunity lives. Signals that are cheap for the defender but slow for the attacker are the good targets: cross-session correlation, third-party supercookie joins, and behavioural modelling all amortise for the defender and cost the attacker fleet redesigns. Signals that are expensive for the defender are the weak points: proof-of-work at high difficulty, ML retraining, and anything requiring a labelled corpus.

For the defender the asymmetry also produces a discipline worth copying. Ship rules in shadow mode first, measure the false-positive rate against known-good traffic, and only then enforce. The cost curve in is where that shows up quantitatively.

Managed Rules Versus Custom Rules#

Use both, for different purposes. Managed rules for breadth — they are cheap and they cover the long tail of scanners and credential-stuffing tools you will never write a rule for. Custom rules for the thing only you know: an endpoint whose access pattern is abnormal for your site, an ASN that should never reach your checkout, a header combination your own application never produces.

The mistake is the inverse allocation. Teams write bespoke rules for problems a managed rule already covers, then discover the managed rule was flagging their own monitoring and turned it off, losing coverage they never replaced. Audit which managed rules are actually firing before you write anything custom.

There is a third category that the marketing tends to blur: bot-management challenges. They are not rules at all — they are a decision to spend CPU and latency on a request, and they are governed by a score rather than a signature. Configure them with a false-positive budget, not a "detect all bots" goal, because a managed challenge set to maximum sensitivity on a consumer site is an availability incident with extra steps.

Why Evasion Cost Is Rising#

Three structural shifts, all of them in the defender's favour.

Signals became relational. A single-request fingerprint is a value you can look up. Cross-validation between signals — does the JA3 match the claimed browser build, does the client hint match the UA, does the canvas match the GPU string — is a consistency check you can only satisfy by being a consistent client, which is far more work than matching any single value. That approach is laid out in .

State accumulated. WAFs now hold sessions for days, so a profile that was clean in week one is compared against its own history. There is no reset button, and the only cure is a genuinely clean history, which is what the ageing model in buys.

Detection moved to the edge, at a point in the connection where you have not yet chosen a profile. TLS ClientHello, HTTP/2 SETTINGS, ALPN and QUIC transport parameters are sent before you send a request line, so they are graded on a connection you did not choose to make. The transport-layer inventory is in and .

The practical planning consequence: budget for profile maintenance as an ongoing line item, not a one-off hardening pass, and assume each vendor refresh invalidates at least one layer you built. Teams that succeeded at this treated it like keeping a browser build current — continuous, versioned, regression-tested — and the testing discipline is in .

What You Can Measure Without Server Access#

Most of the measurement discipline above assumes you can see the WAF's logs. For an external client you cannot, and the substitute is a black-box probe: send two requests that differ in exactly one property, and classify the pair.

The signal you are building is a differential response, not a status code. Compare body hash, response length, header set, cookie names, and latency between the two responses. Any of these differing on a request that should be identical is the discriminator, and a differentiator is much cheaper to find than a threshold.

• One variable per pair. Rotating the UA and the IP together tells you nothing. The cost of the experiment is the number of pairs, so keep the pairs clean and the properties isolated.
• Establish the null first. Two identical requests from the same client, seconds apart, must be byte-identical. Any variance there is your own noise floor, and every later measurement is meaningless against it.
• Find the cheap discriminators early. Status code, then Server header, then body length, then cookie set. Most sites are separable at one of those four levels and you can stop there.
• Measure the rate, not just the outcome. The threshold itself is a rate threshold. One request succeeding and a hundred requests per second succeeding are different observations, and only the second finds the wall.
• Expect the wall to move. Escalation means the threshold measured at one rate is not the threshold at another. Sweep the rate axis in steps and record the whole curve — the shape of the curve tells you which rule is firing, which a single point does not.

Two artefacts are worth keeping. A property matrix — rows are properties you can change, columns are response fields, cells record whether they differ — tells you what to bother spoofing. A rate curve with transition points marked tells you what to bother pacing. Both are cheap to maintain and neither can be reconstructed from memory six weeks later.

The limitation is worth stating plainly: black-box measurement finds separators, not weights. A discriminator that splits a pair tells you the site reads that property; it does not tell you how much the property contributes to a score, which is what the decay model in requires. External measurement gets you to "passes at N requests per minute"; internal measurement is what gets you to "passes indefinitely".