The most popular anti-bot challenges do not permanently block you—they tax you. They issue a clearance cookie that proves you completed their challenge, then they let you through until the cookie expires. Understanding the mechanics of these cookies is the single highest-leverage skill in anti-bot work, because most challenges can be solved once and reused at speed.

When Cloudflare's managed challenge verifies a visitor, it sets cf_clearance—a signed, expiring credential bound to: - The visitor's IP address at issue time. - The target hostname it was issued for. - A lifetime (typically 5 to 30 minutes, configurable by the site).

Any request for a cleared hostname that arrives bearing a valid cf_clearance skips the interstitial entirely. The catch is that the cookie is IP-bound: copy it to a different IP and it is rejected. That is why you must solve the challenge on the same IP you intend to use for scraping, and why a per-IP clearance farm is the classic scaling architecture.

Older DataDome setups used a similar token in a datadome cookie, verified on each subsequent request by a lightweight validator. Modern DataDome moved most of the defense to server-side evaluation of the fingerprint, but where legacy cookies persist, the same solve-once / reuse-until-IP-changes pattern applies.

Proof-of-Work Challenges#

Some challenges skip fingerprinting entirely and require you to compute a proof of work. The page returns inline JS that chains a few thousand SHA-256 operations; the final hash becomes the cookie value. In a browser this takes several seconds and hundreds of milliseconds of CPU. Because the work is pure computation, a native reimplementation in Python or Go solves it in milliseconds—but the solver still needs to produce the correct input format and hash chain, which means you must reverse the exact algorithm. Automated PoW solvers exist for most popular challenge families and are significantly cheaper than human CAPTCHA farms.

The Solve-and-Replay Pipeline#

The production pattern for clearance-cookie targets:

  1. Solve the challenge once inside a real browser (or a native PoW solver) on the working IP.
  2. Snapshot every cookie responsible for clearance (cf_clearance, datadome, plus any session/visitor cookies it depends on).
  3. Hand those cookies to a fast curl_cffi session pinned to the same IP.
  4. Scrape at high speed until the cookie nears expiry (usually 25 minutes of a 30-minute TTL for safety).
  5. Rotate to a new IP and repeat the solve.

This converts a "one browser = ten requests per minute" ceiling into "one solve = thousands of requests." The browser becomes an occasional cookie-making machine.

Treat cookie expiry as a first-class state machine. Track the issue time and TTL for every clearance cookie; schedule the next solve before the previous cookie lapses so the queue never stalls. If a request returns a challenge page mid-stream, fall back to the browseless solve for that IP and continue—never hammer a challenged IP, because challenge pages themselves update WAF signals about your activities.

The value of the cookie is opaque and signed; the bindings are what matter, because they determine the shape of your fleet. In the generic case a clearance cookie asserts: this hostname, this client IP at issue time, this user agent digest, issued at this time, valid for this TTL, computed at this difficulty, with this session score. Every field is a constraint you must hold constant while you use it.

Replay that same cookie against a different IP, a rotated user agent, a sibling hostname, or one second past its TTL, and each one is a separate rejection:

import hashlib

NOW = 1_700_600_000
HASHES_PER_SECOND = 4_800_000.0

COOKIE = {
    "name": "cf_clearance",
    "host": "shop.example",
    "ip": "203.0.113.41",
    "ua_digest": hashlib.sha256(b"Mozilla/5.0 Chrome/126").hexdigest()[:16],
    "issued_at": NOW - 240,
    "ttl_s": 1800,
    "difficulty_bits": 18,
    "score": 0.11,
}

POLICY = {"min_difficulty_bits": 18, "max_score": 0.35, "clock_skew_s": 30}


def verify(cookie, host, ip, ua, now):
    if cookie["host"] != host:
        return "REJECT host_mismatch"
    if cookie["ip"] != ip:
        return "REJECT ip_mismatch"
    if cookie["ua_digest"] != hashlib.sha256(ua.encode("utf-8")).hexdigest()[:16]:
        return "REJECT ua_mismatch"
    if now > cookie["issued_at"] + cookie["ttl_s"]:
        return "REJECT expired"
    if cookie["difficulty_bits"] < POLICY["min_difficulty_bits"]:
        return "REJECT easy_proof"
    if cookie["score"] > POLICY["max_score"]:
        return "REJECT high_score"
    return "ACCEPT"


GOOD_UA = "Mozilla/5.0 Chrome/126"
ALT_UA = "Mozilla/5.0 Chrome/125"

SCENARIOS = [
    ("same ip, same ua, fresh", "shop.example", COOKIE["ip"], GOOD_UA, 240),
    ("same ip, rotated ua", "shop.example", COOKIE["ip"], ALT_UA, 240),
    ("neighbour /24, same ua", "shop.example", "203.0.113.77", GOOD_UA, 240),
    ("datacentre egress, same ua", "shop.example", "198.51.100.9", GOOD_UA, 240),
    ("same ip, sibling hostname", "cdn.example", COOKIE["ip"], GOOD_UA, 240),
    ("same ip, age 1799 s", "shop.example", COOKIE["ip"], GOOD_UA, 1799),
    ("same ip, age 1801 s", "shop.example", COOKIE["ip"], GOOD_UA, 1801),
]

print("cf_clearance replay matrix: ttl %ds, difficulty floor %d bits, skew %ds"
      % (COOKIE["ttl_s"], POLICY["min_difficulty_bits"], POLICY["clock_skew_s"]))
print("")
print("%-27s %-9s %s" % ("scenario", "age", "verdict"))
for label, host, ip, ua, age in SCENARIOS:
    print("%-27s %-9s %s" % (
        label, "%d s" % age, verify(COOKIE, host, ip, ua, COOKIE["issued_at"] + age)))
print("")
print("PoW difficulty calibration at %s hashes/sec" % "{:,.0f}".format(HASHES_PER_SECOND))
print("")
print("  bits   expected hashes   seconds    10k cookies    1M cookies")
for bits in (12, 16, 18, 20, 22, 24, 26):
    expected = 2.0 ** bits
    secs = expected / HASHES_PER_SECOND
    print("  %5d   %15.0f   %8.3f   %10.2f h   %9.1f h" % (
        bits, expected, secs, 10000.0 * secs / 3600.0, 1000000.0 * secs / 3600.0))
print("")
usable = COOKIE["ttl_s"] - POLICY["clock_skew_s"]
print("usable clearance life at 1 req/s   %d s of %d s" % (usable, COOKIE["ttl_s"]))
print("re-solves per ip per hour          %.1f" % (3600.0 / usable))

cf_clearance replay matrix: ttl 1800s, difficulty floor 18 bits, skew 30s

scenario                    age       verdict
same ip, same ua, fresh     240 s     ACCEPT
same ip, rotated ua         240 s     REJECT ua_mismatch
neighbour /24, same ua      240 s     REJECT ip_mismatch
datacentre egress, same ua  240 s     REJECT ip_mismatch
same ip, sibling hostname   240 s     REJECT host_mismatch
same ip, age 1799 s         1799 s    ACCEPT
same ip, age 1801 s         1801 s    REJECT expired

PoW difficulty calibration at 4,800,000 hashes/sec

  bits   expected hashes   seconds    10k cookies    1M cookies
     12              4096      0.001         0.00 h         0.2 h
     16             65536      0.014         0.04 h         3.8 h
     18            262144      0.055         0.15 h        15.2 h
     20           1048576      0.218         0.61 h        60.7 h
     22           4194304      0.874         2.43 h       242.7 h
     24          16777216      3.495         9.71 h       970.9 h
     26          67108864     13.981        38.84 h      3883.6 h

usable clearance life at 1 req/s   1770 s of 1800 s
re-solves per ip per hour          2.0

Two rows are worth pausing on. The neighbour address in the same /24 is rejected, so the binding is to the exact address rather than the prefix — a residential proxy that rotates within its own pool invalidates your own cookie. And the hostname check means clearance does not transfer across a subdomain that a separate WAF rule governs, so a solve for the shop does not clear the CDN.

__cf_bm Is Not Clearance#

The most common operational bug in this area is treating every Cloudflare cookie as the same thing. __cf_bm is a bot-management cookie that carries a device identifier and is set on essentially every visitor, challenged or not. It is not proof of a solved challenge, it does not skip the interstitial, and its presence says nothing about clearance. Scripts that key their health checks on "do we have a Cloudflare cookie" will report success on a session that is still being challenged.

Clearance, where it is issued, is cf_clearance; Akamai's is _abck; DataDome's is datadome; Human/PerimeterX's session cookie carries a different name per deployment. The observable signal you should track is behavioural — does the next request for this host return content or a challenge — rather than the presence of a named cookie. Cookie timelines in captures are the fastest way to tell these apart, which is why treats that timeline as the whole answer to "why did I get challenged".

How Difficulty Is Calibrated#

Difficulty is expressed in expected hashes, so 18 bits means 262,144 hashes on average and 22 bits means 4,194,304 — a 16x jump for four bits. Two forces set the number, and they pull in opposite directions.

The first is the defender's CPU budget, because the origin runs the verification the same way. Raising difficulty to make a scraper expensive makes your own front end expensive for every visitor who lands on a challenge, and for a site behind a shared edge that cost is borne by tenants. In practice this is why a site's difficulty tracks its traffic profile and its customer tier: a low-traffic origin can afford a much higher floor than a high-traffic one serving the same vendor.

The second is reputation. Difficulty is not uniform across visitors. A client with a long clean history, an existing bot-management cookie and a residential IP typically gets 0 bits, no challenge at all. The floor applies to the untrusted case. This is the single most useful fact in the table above: the cheapest cookie is the one the origin decides to give you, and difficulty is what you get when it does not trust you. Difficulty is therefore an output of your risk score, not an input to your budget.

Why Cross-IP Replay Fails and What to Do About It#

The binding exists because clearance is a bearer credential that an attacker would otherwise steal and resell. If cf_clearance worked from any IP, the challenge would be a paywall, not a control. So the design assumes the IP is part of the identity and enforces it strictly.

The architecture that follows is per-IP clearance: one solve per egress, then high-speed reuse from that same egress until the TTL expires, then re-solve. At a 30-minute TTL with a safety margin that is 2.0 solves per IP per hour, and every one of them is a visible event on that IP. Aggregate across 500 residential IPs each making 1 solve per 30 minutes and the origin sees 500 distinct visitors, each clearing once — indistinguishable from a morning of returning readers.

The failure modes to guard are all version-control failures:

  • Pool rotation mid-cookie. If your proxy's sticky session is shorter than the TTL, the cookie outlives its IP. Sticky-session TTL must exceed clearance TTL with margin, not equal it.
  • Cookie sharing across a pool. Handing one clearance to a worker pool is a bug the moment the pool is larger than one, and it looks fine in testing on a single IP.
  • Solve on a different egress from the scrape. The token and the cookie are both bound; a residential solve proxied through a different exit is a silent failure.
  • Clock skew. Expired cookies present as challenges, so a fleet with unsynchronised clocks solves at a higher rate than its TTL implies, which is itself a signal.

Measuring Throughput Before You Buy Egress#

The difficulty table converts directly into the two numbers that decide whether a project is viable: solves per hour per CPU core, and how many cores you need for a target solve rate.

At 18 bits and 4.8M hashes/sec, one core produces 0.055 solves/sec, or about 196 solves per core-hour. Two thousand solves an hour therefore needs roughly ten cores of pure hashing — trivial. But 24 bits needs 3.495 seconds per solve on one core, so the same 2,000 solves an hour needs about nineteen cores, and a million cookies at 24 bits is 971 hours of single-core time no matter how you parallelise it.

Three consequences that fall directly out of those numbers:

Difficulty scaling makes large fleets nonlinear. Going from 20 to 22 bits is a 4x compute increase. Any design that assumes "one core per N solves" breaks the first time a site raises its floor, so size the fleet for the difficulty you are allowed to see rather than the one you currently see.

Parallelism is embarrassingly parallel, which means it is also easy to detect. Thousands of cores each grinding hashes produce a distinctive CPU-time-to-response-latency profile on the challenge endpoint. A solve that returns in exactly the PoW time and never longer is a solve. Treat solve latency as a parameter to randomise within the distribution implied by difficulty, not as a constant.

Verification is the real cost at low difficulty. At 12 bits a solve is essentially free but the resulting cookie is nearly worthless — the site is issuing it because it does not care. Difficulty is a signal about how much the site wants the work to cost, so there is no scenario where buying solve capacity at the lowest difficulty is the cheap option for bulk access.

The same arithmetic run in reverse is a planning tool for the fleet side: with a 30-minute TTL, 100 persistent profiles generate 200 solves an hour on their own, so a target of 2,000 solves an hour is 1,000 persistent profiles at minimum, and you should size for several times that because not every profile survives to the end of its TTL.

Failures That Masquerade as Solve Failures#

When clearance stops working, the cause is usually not the solve. These four present identically — a challenge on the next request — and distinguishing them is the whole diagnostic skill.

The cookie was issued but bound to a client you no longer have. The ip_mismatch and ua_mismatch rows in the matrix above. The tell is that the solve succeeded — you have a cookie — and the failure appears on the next request rather than at solve time.

The cookie was issued for a different hostname. Also a successful solve, failure on the next request. Common when the challenge appears on the API subdomain but the request goes to the www host, or vice versa.

The cookie expired between solve and use. The expired row. With a 30-minute TTL and a solve latency distribution that has a long tail, this is a latency problem: your p99 solve latency was longer than the TTL, so a fraction of your cookies are born expired. The fix is latency budgeting, not more solvers, and the distribution analysis in is the tool for it.

The cookie is valid and you are still challenged. This is the one that sends people looking for a solver bug that does not exist. The score moved above the clearance threshold because of other signals — pool reuse, rate, profile incoherence — so the site now challenges despite holding a valid cookie. Diagnose by changing one input at a time: same cookie, different IP; same IP, different rate. If the cookie works from a clean IP at a low rate, the cookie was never the problem. The escalation model in is the same score seen from the vendor side.

The ordering of that list is worth internalising, because it maps onto how often each cause occurs. Cookie binding errors and expiry together account for most incidents, both are local configuration bugs, and both are fixed in minutes once identified. The last one is a different category of problem entirely.

The Fleet-Shaped Alternative#

There is a second design that avoids per-IP clearing entirely: keep every clearance cookie for the same fixed duration of browsing, low enough that the site never escalates. In practice this means a solve budget of roughly one per several hours per profile, an egress pinned per profile, and a request rate inside the per-IP human envelope. You trade raw throughput for a fleet whose aggregate behaviour is indistinguishable from a small number of heavy readers, and you need far more profiles to reach the same volume. The browser-pool and worker-fleet designs are in and , and the ageing model that makes a profile survive is in .