When a Site Fights Back

Layers, Not a Single Wall#

"Anti-bot" is a label for several separate systems between you and the data. From cheapest to most expensive for the site to run, and for you to deal with:

Most protected sites combine several layers, and most commercial products bundle all of them behind one edge service. The practical skill is identifying which layer rejected you, because each has a different remedy, and for some of them the right remedy is to stop.

A block is also a message. A site that pays for bot management is telling you it does not want automated access to that content. Before engineering around any of these layers, check for an official API or data licence, re-read robots.txt and the terms of service, and consider asking. The robots and ethics lesson sets out the course's stance: the techniques here are for understanding why a legitimate, polite scraper is being misclassified, not for overpowering a site that has said no.

Classify Every Response#

A scraper that only checks status_code == 200 will happily store challenge pages as data. Classify each response into a small set of outcomes and log the result.

import re

VENDOR_MARKERS = {
    "cloudflare": ("cf-chl", "challenge-platform", "cf_clearance", "turnstile"),
    "perimeterx": ("px-captcha", "_pxhd", "perimeterx"),
    "datadome":   ("datadome", "captcha-delivery"),
    "imperva":    ("_incapsula_resource", "incap_ses"),
    "akamai":     ("_abck", "ak_bmsc"),
}

def classify(resp):
    body = resp.text[:20000].lower() if resp.content else ""
    cookies = " ".join(resp.cookies.keys()).lower()
    header_blob = " ".join(f"{k}:{v}" for k, v in resp.headers.items()).lower()
    haystack = body + " " + cookies + " " + header_blob

    if resp.status_code == 429:
        return "rate_limited", None
    if resp.headers.get("cf-mitigated") == "challenge":
        return "challenge", "cloudflare"
    for vendor, markers in VENDOR_MARKERS.items():
        if any(m in haystack for m in markers):
            if resp.status_code in (403, 503) or "captcha" in body:
                return "challenge", vendor
    if resp.status_code == 403:
        return "forbidden", None
    if resp.status_code == 200 and len(resp.content) < 2000 and "<script" in body:
        return "suspicious_tiny_page", None
    if resp.status_code == 200:
        return "ok", None
    return f"http_{resp.status_code}", None

Non-obvious points:

• Markers are evidence, not proof. Cookies like _abck appear on normal pages of sites using that vendor. That is why the vendor markers only count as a challenge when combined with a 403/503 or a CAPTCHA reference. Tune the list against real samples you have saved.
• cf-mitigated: challenge is a response header Cloudflare sets on challenge responses, which is more reliable than body scanning.
• The tiny-page check catches a common silent failure: a 200 response that is only a script tag and a spinner, meaning the real content is gated behind JavaScript. Your parser would see zero products and report success.

Aggregate outcomes per target and per identity (IP, session, account) every few minutes. The shape of the numbers tells you which layer is involved:

How a JavaScript Challenge Works#

Most challenge systems follow the same broad flow. The first request never reaches the origin server; the edge answers with a page containing JavaScript. In a browser, that script collects environment signals (and sometimes performs a small proof-of-work), then posts the result. If the edge is satisfied, it sets a clearance cookie and lets later requests through until the cookie expires. The cookie is typically bound to properties of the client that earned it, such as its IP address and user agent, so copying it to a different client usually fails.

This explains two things you will observe. A plain HTTP client never passes: it does not run the script. And a real browser that passes once can keep working for the lifetime of the clearance, which is why session reuse matters far more than raw request volume. The mechanics are covered in proof-of-work and clearance cookies.

The Honest Strategies, In Order#

Ranked by how sustainable they are:

1. Use a sanctioned channel. An official API, a data export, an RSS feed, a sitemap, or a licensing deal. These never break when the bot vendor updates, and they remove the legal ambiguity. See official APIs and alternative data.
2. Be a less suspicious client. Many blocks hit scrapers that are simply rude or inconsistent: dozens of parallel connections, no delays, mismatched headers and TLS. Slow down, use one coherent client profile, respect Retry-After, and fetch only what you need. This alone resolves a surprising share of rate and reputation blocks.
3. Use a real browser at human pace for the pages that require JavaScript, and reuse its session rather than starting fresh each time (browser automation).
4. Stop. If a site escalates to CAPTCHAs on every page, it has clearly decided against automated access. Continuing turns scraping into an arms race you pay for every week, with growing legal risk.

Writing your own solver for a commercial challenge is the one non-strategy. The scripts are obfuscated, change frequently, and differ per site, and bypassing a technical access control can carry legal consequences in several jurisdictions that ordinary scraping of public pages does not (see the legal considerations lesson).

Back Off Like You Mean It#

When blocks appear, the worst response is to retry faster or rotate through more IPs. Detection systems learn from volume, and hammering a challenge teaches them that your whole address range is automated. Build a circuit breaker per target:

import time
from collections import deque

class TargetBreaker:
    def __init__(self, window=100, max_block_ratio=0.1, cooldown=900):
        self.results = deque(maxlen=window)
        self.max_block_ratio = max_block_ratio
        self.cooldown = cooldown
        self.open_until = 0.0

    def record(self, outcome):
        self.results.append(outcome not in ("ok",))
        ratio = sum(self.results) / len(self.results)
        if len(self.results) >= 20 and ratio > self.max_block_ratio:
            self.open_until = time.time() + self.cooldown
            self.results.clear()
            print(f"breaker open: {ratio:.0%} blocked, pausing {self.cooldown}s")

    def allow(self):
        return time.time() >= self.open_until

Check allow() before each request and skip the target while the breaker is open. When it closes, resume at half the previous concurrency. Doubling the cooldown on each consecutive trip (exponential backoff) prevents a crawl from oscillating between blocked and unblocked. The ban recovery and backoff lesson extends this with per-identity health scores.

Save the Evidence#

Keep the full response (status, headers, first part of the body) for a sample of every non-ok outcome, with the timestamp and identity that received it. When the block rate changes, you can compare today's challenge page with last week's and see whether the site changed vendor, tightened a rule, or started protecting a new path. Without samples, every investigation starts from scratch.

Practice#

Take a scraper you already run against a site that permits it. Add the classifier and log outcome counts per hour for a day. Then double the concurrency for one hour and compare: if the rate_limited or challenge counts rise, you have measured that site's tolerance, and your production setting should sit well below it.