Four Defenders, Four Personalities#

Bot management is dominated by a handful of vendors, and each builds its defenses around different priorities. Knowing their tendencies tells you where the friction will be, what a challenge looks like, and which countermeasures are worth the effort. None of this is a how-to-bypass guide; it is a map of the terrain.

Cloudflare#

The most widely deployed. Its defenses are layered and configurable by the site owner, ranging from "essentially off" to "under attack". Key behaviors:

  • Managed Challenge: an interstitial that runs JavaScript, probes the environment, and issues a cf_clearance cookie bound to the client IP with a short TTL.
  • Turnstile: a modern, often invisible CAPTCHA replacement that scores the visitor; the site can choose invisible, non-interactive, or interactive modes.
  • Bot Fight Mode / Super Bot Fight Mode: automatic tools that challenge anything matching known bot signatures, including many legitimate crawlers.
  • Strong IP and TLS reputation built from its position in front of a large fraction of the web.

The IP-bound cf_clearance cookie is the defining feature: solve once on an IP, replay on that same IP.

DataDome#

A dedicated bot specialist with a reputation for aggressive, low-latency detection focused on behavior and fingerprinting rather than volume. Its JS challenge sets a datadome cookie and it validates continuously via a separate endpoint, so copying a cookie to another IP or fingerprint usually fails. DataDome tends to be harder on automation than default Cloudflare and frequently gates based on behavioral signals, making consistent, well-aged profiles more valuable than raw throughput.

Akamai#

Enterprise-grade and often found behind large e-commerce and airline sites. Akamai's Bot Manager builds a fingerprint from a _abck sensor payload—an obfuscated script that collects DOM and browser signals and posts them back, later validated on each request. Its detection is sophisticated and its mitigations can include silent throttling and decoy content, not just visible challenges. Pricing and integration mean it is usually seen on high-value, high-abuse targets.

PerimeterX (now HUMAN Security)#

Focused on account takeover, carding, and scraping. Its sensor collects extensive behavioral telemetry—mouse, keyboard, scroll, and timing—and is famously sensitive to synthetic input. It is also known for serving decoy or poisoned data to suspected bots (for example, invented prices or listings), which means a scraper may believe it is succeeding while receiving fabricated data. Verifying data sanity, not just HTTP 200s, is essential against this vendor.

How Their Responses Differ#

  • Visible challenge vs silent scoring: Cloudflare often shows an interstitial; DataDome and HUMAN lean more on invisible scoring and decoys.
  • IP binding vs fingerprint binding: Cloudflare's clearance is strongly IP-bound; DataDome and Akamai bind more to fingerprint and session consistency.
  • Decoy data: HUMAN is the notable user of poisoned responses, so always validate extracted values.
  • Configurability: Cloudflare's protection depends heavily on the site's chosen settings; two Cloudflare sites can be wildly different.

Building a Strategy Matrix#

Effective operations map each target to its vendor and its configured strength, then pick the cheapest adequate approach:

  • Light Cloudflare or none: plain or TLS-impersonating HTTP client.
  • Challenges with clearance cookies: browser solve once, cookie-replay on the same IP.
  • Behavior-heavy (DataDome/HUMAN): consistency, aged profiles, human-shaped behavior, and data-validation checks.
  • Attestation-gated mobile: reassess whether to proceed at all.

Finally, remember that vendor identity is not destiny. A site's configuration matters more than the logo, and the correct response is always to test the actual target, measure what it does, and stay within legal and ethical boundaries.

The first useful question when you hit an interstitial is: which vendor's stack are you looking at? "Cloudflare" and "reCAPTCHA" are product names, but what the detector actually emits is a collection of headers, cookies, challenge flows, timing behaviour, and the structure of the page it returns. Treating a vendor as a logo leads to guessing; treating it as a weighted set of observable markers leads to reproducible debugging.

The relevant axes to compare before you commit to a solve path are:

Dimension What to measure Why it matters
Surface area which endpoints return challenge HTML, and under what conditions determines whether you can detect it early or only after consuming quota
Aggressiveness challenge rate at a fixed request rate per identity see ban recovery and backoff for how challenge failures compound
False-positive profile how often a clean human session is challenged high FP vendors tend to change flows more frequently and break automation harder
Marker stability how often headers/cookies change names or values unstable markers mean your identification code will rot weekly
Bypass surface whether the challenge is browser-bound, token-bound, or session-bound drives whether you must solve in the same browser context
Integration depth edge-only, origin-aware, or JS-heavy affects TLS and JA3/JA4 coupling

A common trap is conflating "uses the same CDN" with "uses the same challenge vendor". Many sites put their WAF at the edge and their own interstitial at origin, or layer multiple products. When you see two different marker families in the same response chain, assume layering until proven otherwise.

Identifying Vendors From Traces#

Do not rely on the HTML title or on error strings alone: those change with localisation, A/B tests, and product renames. Instead, collect a small, weighted evidence set from each trace:

Marker type Example Stability Weight
Response header cf-ray, server, x-hcaptcha-response, x-k-challenge medium high if unique
Set-Cookie names that only that vendor sets high very high
HTML structure well-known form action or script src medium-low medium
Timing first-byte delay characteristic low low (noisy)
Network chain IP/ASN of the responding edge medium medium
JSON shape /v1/challenge responses high high

Weight matters because any single header can appear on a reverse proxy in front of the challenge. The goal is a consistent majority of markers, not a single hit. Also note the direction: some vendors set markers on clearance cookies as well as on challenge pages, so you must classify the response class (challenge, block, success, redirect) before scoring.

The following snippet implements a deliberately small, stdlib-only identification pass over two contrasting traces: a managed challenge trace with a strong vendor footprint, and a generic block that has almost no vendor-specific markers. The verdict is based on a weighted sum with a minimum threshold, which is the right trade-off between false positives (misclassifying a generic nginx block) and false negatives (missing a vendor after a small rename).

VENDORS = ["cloudflare", "akamai", "datadome", "human", "kasada", "imperva", "cequence"]

# Where the product sits. A site can be behind one vendor's edge while running a
# different vendor's bot manager at the origin, and the two set cookies that look
# superficially alike. Clearing one does not clear the other.
LAYER = {
    "cloudflare": "edge, same hop as its CDN",
    "akamai": "origin tier, behind the CDN",
    "datadome": "origin tier",
    "human": "origin tier",
    "kasada": "origin tier",
    "imperva": "edge",
    "cequence": "origin tier",
}

# marker -> {vendor: weight}. Weight is evidence strength, not likelihood: a
# vendor-specific header beats a cookie name, and a lone cookie beats nothing.
MARKERS = {
    "hdr:cf-ray": {"cloudflare": 3},
    "hdr:cf-mitigated": {"cloudflare": 3},
    "hdr:server:cloudflare": {"cloudflare": 2},
    "cookie:cf_clearance": {"cloudflare": 4},
    "cookie:__cf_bm": {"cloudflare": 3},
    "js:/cdn-cgi/challenge-platform/": {"cloudflare": 4},

    "cookie:_abck": {"akamai": 4},
    "cookie:bm_sz": {"akamai": 2},
    "cookie:ak_bmsc": {"akamai": 3},
    "post:/akam/*/sensor_data": {"akamai": 4},

    "cookie:datadome": {"datadome": 4},
    "hdr:x-datadome": {"datadome": 4},
    "js:geo.captcha-delivery.com/captcha/v1/api.js": {"datadome": 3},

    "cookie:_px": {"human": 4},
    "cookie:_pxhc": {"human": 3},
    "hdr:x-px-block": {"human": 4},
    "js:client.perimeterx.com/px.js": {"human": 4},

    "cookie:_kpsdk": {"kasada": 3},
    "hdr:x-kpsdk-ct": {"kasada": 4},
    "js:api.kasada.io/": {"kasada": 4},

    "cookie:incap_ses_": {"imperva": 3},
    "cookie:visid_incap_": {"imperva": 3},
    "hdr:x-iinfo": {"imperva": 4},

    "hdr:x-cep-version": {"cequence": 4},
    "js:/qdcf/": {"cequence": 3},
}

CASES = [
    ("managed challenge", ["hdr:server:cloudflare", "hdr:cf-ray", "hdr:cf-mitigated",
                           "cookie:cf_clearance", "cookie:__cf_bm",
                           "js:/cdn-cgi/challenge-platform/"]),
    ("akamai bot manager", ["cookie:_abck", "cookie:bm_sz", "cookie:ak_bmsc",
                            "post:/akam/*/sensor_data"]),
    ("datadome interstitial", ["cookie:datadome", "hdr:x-datadome",
                               "js:geo.captcha-delivery.com/captcha/v1/api.js"]),
    ("human sensor", ["cookie:_px", "cookie:_pxhc", "hdr:x-px-block",
                      "js:client.perimeterx.com/px.js"]),
    ("one shared cookie only", ["cookie:bm_sz"]),
    ("cdn plus a bot manager", ["hdr:server:cloudflare", "hdr:cf-ray", "cookie:__cf_bm",
                                "cookie:_abck", "cookie:bm_sz"]),
    ("nothing recognisable", ["hdr:server:nginx", "hdr:content-type:text/html"]),
]

THRESHOLD = 6


def identify(observed):
    scores = dict.fromkeys(VENDORS, 0)
    unexplained = []
    for marker in observed:
        hits = MARKERS.get(marker)
        if hits is None:
            unexplained.append(marker)
            continue
        for vendor, weight in hits.items():
            scores[vendor] += weight
    ranked = sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))
    return ranked, sorted(unexplained)


def describe(ranked):
    strong = [(v, s) for v, s in ranked if s >= THRESHOLD]
    if not strong:
        return "unidentified: best evidence {} < threshold {}".format(ranked[0][1], THRESHOLD)
    if len(strong) > 1:
        where = ", ".join("{} at the {}".format(v, LAYER[v]) for v, _ in strong)
        return "two products present: {}; clearing one does not clear the other".format(where)
    if strong[0][1] - ranked[1][1] <= 2:
        return "weak: {} leads by only {} over {}".format(
            strong[0][0], strong[0][1] - ranked[1][1], ranked[1][0])
    return "identified: {}, {} (score {})".format(strong[0][0], LAYER[strong[0][0]], strong[0][1])


def main():
    for label, observed in CASES:
        ranked, unexplained = identify(observed)
        print(label)
        print("  markers observed: {}".format(len(observed)))
        for vendor, score in [(v, s) for v, s in ranked if s > 0]:
            print("    {:<12} {:<6} {}".format(vendor, score, "#" * score))
        print("    claimed by no vendor: {}".format(unexplained or "none"))
        print("    verdict: {}".format(describe(ranked)))
        print()


main()

managed challenge
  markers observed: 6
    cloudflare   19     ###################
    claimed by no vendor: none
    verdict: identified: cloudflare, edge, same hop as its CDN (score 19)

akamai bot manager
  markers observed: 4
    akamai       13     #############
    claimed by no vendor: none
    verdict: identified: akamai, origin tier, behind the CDN (score 13)

datadome interstitial
  markers observed: 3
    datadome     11     ###########
    claimed by no vendor: none
    verdict: identified: datadome, origin tier (score 11)

human sensor
  markers observed: 4
    human        15     ###############
    claimed by no vendor: none
    verdict: identified: human, origin tier (score 15)

one shared cookie only
  markers observed: 1
    akamai       2      ##
    claimed by no vendor: none
    verdict: unidentified: best evidence 2 < threshold 6

cdn plus a bot manager
  markers observed: 5
    cloudflare   8      ########
    akamai       6      ######
    claimed by no vendor: none
    verdict: two products present: cloudflare at the edge, same hop as its CDN, akamai at the origin tier, behind the CDN; clearing one does not clear the other

nothing recognisable
  markers observed: 2
    claimed by no vendor: ['hdr:content-type:text/html', 'hdr:server:nginx']
    verdict: unidentified: best evidence 0 < threshold 6

The output shows the logic in action: the first trace accumulates enough weight to identify Cloudflare and to infer an edge/origin layering hint ("same hop as its CDN"), while the second trace falls below threshold and is returned as "unidentified". That is the correct answer in the second case; guessing would create brittle automation.

Edge Versus Origin, and Layering#

Once you have a candidate vendor, the next question is where the challenge is enforced. This changes both the solve strategy and what you must log.

Enforcement point Typical signals Implication
Edge (CDN/WAF) challenge returns from edge IP, short TTL, cache headers often absent solve must be presented to the same edge; origin sees the clearance
Origin challenge HTML served by the app, server is app, same TLS as app solve path is app-specific; may involve app session
Layered clearance from edge plus a second token from origin must satisfy both in order; see signed API requests for origin tokens
JS-driven initial 200 with JS that triggers challenge fetch need a full browser context, not just HTTP replay

Layering is increasingly common: an edge vendor challenges the TLS/HTTP fingerprint, and the origin validates a secondary token or a mobile attestation signal for mobile clients. If you miss the origin layer, you will pass the edge challenge and be challenged again milliseconds later -- a classic sign that your identification stopped at the CDN logo.

Aggressiveness, False Positives and Change Cadence#

When choosing a strategy, track three operational numbers per vendor and per target:

  1. Challenge rate per 1,000 requests at your target rate, split by new versus warm identities (see account warm-up and identity lifecycle).
  2. Human false-positive rate as reported or inferred. A vendor that challenges 1% of real human traffic will also challenge your automation at unpredictable times and will ship changes defensively.
  3. Marker churn: how many of your markers changed in the last 14 days. A weekly marker churn above ~20% means you should invest in a self-healing identifier (treat markers as a feature set and retrain weights) rather than hardcoding strings.

These numbers decide whether to solve in-process, to outsource solves, or to avoid that vendor entirely. For example, a high-churn, high-FP vendor is often cheapest to route around via official APIs and alternative data if such a channel exists, because the maintenance cost of staying current exceeds the API cost.

Chokepoint Debugging#

When a flow stops working, debug in this order to avoid chasing ghosts:

  1. Confirm the response class. Is it a 200 with challenge HTML, a 403, a redirect loop, or an empty body? Status alone is insufficient.
  2. Collect the full trace. Headers, cookies (request and response), timing, TLS, and the exact HTML/JSON body. Strip PII, keep bytes.
  3. Identify the vendor. Run a weighted marker pass against your known set; record which markers matched and with what weight.
  4. Check for layering. Compare the edge IP to the origin, and look for a second token in cookies/JSON.
  5. Verify the solve boundary. Did you solve in the same browser context that made the request? Was the token bound to the IP/profile that received the challenge? See managed challenge flows for token binding details.
  6. Test with a known-clean identity. Reproduce with a fresh, warmed profile on the same egress to separate "your stack broke" from "that identity is flagged".

A final operational note: vendor identification is for debugging and routing, not for making legal or policy claims. The observable markers are what you see on the wire; the product name is convenient shorthand. Keep your identification tables versioned and tested against captured traces, because the only thing more fragile than a vendor's challenge flow is your assumption that you know its name.