CAPTCHAs, Challenges & Solving Ethically

When the Robot Test Appears#

At some point in a real crawl, a page stops returning data and starts returning a puzzle, a "checking your browser" interstitial, or a quiet 403. CAPTCHAs and challenges are how a site says "we think this traffic is automated, and we want to slow it down or stop it". This lesson is not a guide to defeating them. It covers what the different kinds are, why your scraper triggered one, how to detect challenges automatically so your pipeline does not store garbage, and the legitimate ways to keep getting the data.

The Types You Will Actually Meet#

The key point is that modern challenges are rarely about the puzzle. The visible task is the last step of a risk assessment that already considered your IP address, TLS and HTTP/2 fingerprint, headers, cookies, request rate and navigation pattern. That is why "solving" the puzzle often just produces another one: the risk score that caused it did not change. The machinery behind these scores is covered in how anti-bot systems work.

Why Your Scraper Triggered One#

Diagnose before doing anything else. When the challenge appears tells you what caused it:

Most of these causes are fixed by behaving like a considerate client, not by any solving code: keep cookies in a session, send a coherent set of headers, slow down to a pace a site can absorb (see rate limiting and politeness), add jitter so requests are not perfectly periodic, and avoid crawling at a rate that dwarfs the site's human traffic. A crawler that respects robots.txt and crawl delays meets far fewer challenges than one that does not.

Detect Challenges Automatically#

The dangerous case is not a crash. It is a challenge page returned with status 200 that your parser treats as a product page with no products, so the dataset silently fills with empty records. Classify every response before parsing it:

CHALLENGE_MARKERS = (
    "g-recaptcha", "recaptcha/api.js",
    "hcaptcha.com", "h-captcha",
    "cf-turnstile", "challenges.cloudflare.com", "/cdn-cgi/challenge-platform",
    "captcha", "are you a robot", "verify you are human",
)

def classify(resp, expect_json=False, required_selector_hit=True):
    ctype = resp.headers.get("content-type", "")
    if resp.headers.get("cf-mitigated") == "challenge":
        return "challenge"
    if resp.status_code == 429:
        return "rate_limited"
    if expect_json and "json" not in ctype:
        return "challenge" if resp.status_code in (200, 403, 503) else "error"
    body = resp.text[:20000].lower() if "html" in ctype else ""
    if any(m in body for m in CHALLENGE_MARKERS):
        return "challenge"
    if resp.status_code in (401, 403):
        return "blocked"
    if resp.status_code >= 500:
        return "error"
    if not required_selector_hit:
        return "unexpected_page"           # 200, no markers, but the data is not there
    return "ok"

Notes on the choices:

• Checking content-type first is the cheapest reliable signal for APIs: an endpoint that normally returns JSON and suddenly returns HTML is almost always a challenge or a block page.
• Cloudflare documents a cf-mitigated: challenge response header on challenge pages, which is more reliable than searching the body.
• The generic words ("captcha", "verify you are human") can appear on normal pages, for example in a site's help text. Treat a marker hit as a strong hint and combine it with "the selector I expected is missing".
• unexpected_page catches everything else: consent walls, country selectors, login redirects. Save a sample of each unexpected page to disk so a human can look at it.

Back Off, Don't Push Through#

When challenges start, continuing at the same rate makes the problem worse: the IP or session collects more negative signals and the block becomes longer. Treat challenges as a signal to reduce load on that host, using a per-host circuit breaker:

import time
from collections import defaultdict, deque

class HostBreaker:
    def __init__(self, threshold=3, window=300, base_pause=600, max_pause=6 * 3600):
        self.events = defaultdict(deque)
        self.paused_until = {}
        self.trips = defaultdict(int)
        self.threshold, self.window = threshold, window
        self.base_pause, self.max_pause = base_pause, max_pause

    def allowed(self, host):
        return time.time() >= self.paused_until.get(host, 0)

    def record(self, host, outcome):
        now = time.time()
        if outcome == "ok":
            self.trips[host] = 0
            return
        if outcome not in ("challenge", "blocked", "rate_limited"):
            return
        q = self.events[host]
        q.append(now)
        while q and now - q[0] > self.window:
            q.popleft()
        if len(q) >= self.threshold:
            pause = min(self.max_pause, self.base_pause * 2 ** self.trips[host])
            self.paused_until[host] = now + pause
            self.trips[host] += 1
            q.clear()
            print(f"{host}: {self.threshold} challenges in {self.window}s, pausing {pause // 60:.0f} min")

Three challenges in five minutes pause the host for ten minutes, then twenty, then forty, up to six hours. While paused, the scheduler works on other hosts. Every trip should also raise an alert, because repeated trips mean your crawler's behaviour needs to change, not that it needs more patience. Lower the concurrency for that host when it resumes. The error handling lesson covers where this fits among other retry decisions.

The Legitimate Paths Forward#

A challenge on data you need is a prompt to look for a door the site actually opened:

1. An official API or data export. Many sites offer one, sometimes free with a key and rate limits, sometimes paid. It is stable, documented and legal to use within its terms. See official APIs and alternative data.
2. Ask. A short email explaining who you are, what data you need, how often, and at what rate gets a yes more often than people expect, sometimes with an allowlisted IP or a dedicated feed. Put a contact address in your User-Agent so the site can reach you, too.
3. Other sources for the same data. Sitemaps, RSS feeds, public datasets, government registries and aggregators often carry the same facts without a challenge.
4. A human in the loop, for your own authorised access. For internal tools working on an account you legitimately hold, a person can complete the challenge once in a real browser and the automated session continues at a low rate with that session's cookies. That pattern is described in browser session handoff. It does not change the site's terms: if they forbid automated access, this does not make it permitted.

Commercial CAPTCHA-solving services also exist, where people or models solve challenges for a fee. Outsourcing the solving does not change the legal or terms-of-service position at all; it also means sending the target page's details to a third party. How these services work and why sites detect them is covered in CAPTCHA solving.

The Legal and Ethical Line#

A CAPTCHA is an unambiguous statement of what the site operator wants. Deliberately circumventing it to continue automated collection typically violates the site's terms, and in some jurisdictions circumventing a technical access control can weaken any argument that your access was authorised. Scraping personal data behind a challenge adds data protection obligations on top. Read legal considerations before deciding, and when in doubt, choose a different source.

If You Run the Site: Verify on the Server#

The defensive side has one classic mistake worth knowing because scrapers exploit it: checking the CAPTCHA only in the browser. The widget produces a token; your backend must send that token to the provider's verification endpoint with your secret key and reject the request if verification fails. For reCAPTCHA:

import httpx

def verify_recaptcha(token: str, secret: str, remote_ip: str | None = None) -> bool:
    data = {"secret": secret, "response": token}
    if remote_ip:
        data["remoteip"] = remote_ip
    r = httpx.post("https://www.google.com/recaptcha/api/siteverify", data=data, timeout=10)
    result = r.json()
    return bool(result.get("success"))

Check more than success: tokens are single-use and short-lived, the response includes the hostname the token was issued for (confirm it is yours), and score-based versions return a score and action that you should compare against a threshold and the action you expected. Providers offer test keys that always pass, so automated tests of your own forms do not need to solve anything.

Practice#

Add classify() and HostBreaker to an existing scraper. Replay a saved challenge page and a saved normal page through classify() to confirm both are recognised. Then run the scraper against your own test site with a deliberately aggressive rate limiter in front of it, and confirm the breaker pauses the host instead of storing empty records.