Browsers With a Solid Gold Jailbreak

Before Playwright, the standard advice was "use requests or curl_cffi for speed, use a browser only when forced." The truth is that once you need real JavaScript, you need a real browser—and with the right pipeline, a browser is neither the slow lose-lose tool it used to be, nor a guaranteed ban. A production stealth pipeline treats the browser like an expensive tool that must be warmed, reused, and authenticated before it ever touches a real page.

The Pipeline Stages

1. Profile Selection Every job maps to a long-lived browser profile: a fixed fingerprint (screen size, platform, fonts, timezone), a fixed geolocation, and a stable pool of residential IPs for that region. Creating the profile once—and reusing it across sessions—keeps the WAF's internal device identity constant. Randomizing per request destroys that continuity.

2. Proxy Assignment Assign the profile a residential or mobile proxy before browser startup, and configure the proxy at the browser level (--proxy-server), not through a PAC file that changes mid-session. Check the exit IP, timezone, and language after launch; any mismatch means the session is already suspect.

3. Warm-Up Sequence Navigate first to a neutral page (about:blank, then your provider's homepage), let fonts load, let the media stack initialize, and collect the browser's own TLS session. Only then navigate to the target. This also builds connection reuse so the first "real" request looks like a continuing session rather than a cold open.

4. The Human Workflow Drive the browser with human-shaped interactions: mouse moves along Bézier paths, scroll with variable velocity and occasional stops, type with randomized keystroke timing. Never call page.click() cold on a heavily protected site—give the pointer time to travel.

5. Cookie Extraction & Handoff If the site issues a clearance cookie (cf_clearance, datadome, Datadome's __datadome token) or a session cookie, snapshot it and reuse it in a lightweight curl_cffi session for high-throughput follow-up requests. This is the standard throughput optimization: the browser solves challenges and collects tokens; the fast HTTP client exploits them.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True, args=["--no-sandbox"])
    ctx = browser.new_context(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36",
        viewport={"width": 1920, "height": 1080},
        locale="en-US",
    )
    page = ctx.new_page()
    page.goto("about:blank")
    page.goto("https://nowsecure.nl")
    page.wait_for_timeout(8000)
    cookies = ctx.cookies()  # harvest clearance token
    browser.close()

6. Monitoring & Escape Hatch Every browser session should assert its own health: the presence of an expected DOM element, the absence of a challenge page, and response codes of 200. If a detection page appears, immediately kill the browser, discard the IP, and requeue the URL—fighting a confirmed detection wastes time and burns IPs.

When to Accelerate

A single warmed browser session might handle 30–80 requests a minute on a script-heavy site. When your job needs thousands of pages a minute, the browser pipeline is intentionally the first stage only: solve the challenge, harvest the token, and stream the actual data extraction through curl_cffi. The browser queue is a challenge-solving farm, and the HTTP layer is the bandwidth engine.