Solve Once, Fetch a Thousand Times

Browsers are excellent at the hard part of scraping—executing JavaScript, passing challenges, and collecting cookies—and terrible at the easy part—making thousands of fast requests. The professional pattern combines both: use a browser to establish a session, then hand that session's credentials to a fast HTTP client that reuses it at scale.

What to Capture

A browser session is more than a cookie jar. The minimal handoff set:

  • Cookies: every cookie in the browser context, including HttpOnly ones (which is precisely why you need the browser—JavaScript cannot see them).
  • User-Agent: the exact UA string the browser sent; the HTTP client must send the identical one.
  • Custom request headers: many sites require Accept-Language, Referer, and vendor-specific headers to match.
  • Local/session storage: occasionally an app stores a bearer token in localStorage that is sent as a header on API calls.

Capture these programmatically rather than by hand.

# Playwright: harvest the full context, then replay in httpx
cookies = ctx.cookies()
ua = page.evaluate("navigator.userAgent")
storage = page.evaluate("JSON.stringify(Object.entries(localStorage))")

import httpx
client = httpx.Client(
    headers={"User-Agent": ua, "Accept-Language": "en-US,en;q=0.9"},
    cookies={c["name"]: c["value"] for c in cookies},
    timeout=30,
)

Clearance vs. Session Cookies

Two distinct kinds of cookies matter:

  • Session cookies (auth tokens, cart IDs) identify a logged-in user and usually survive IP changes.
  • Clearance cookies (cf_clearance, datadome) prove you passed an anti-bot challenge and are typically bound to the IP that solved it.

If a clearance cookie is IP-bound, your HTTP client must use the same exit IP as the browser that solved the challenge. Solve and fetch through the same proxy, and keep them pinned together for the cookie's lifetime.

Keeping the Account Alive

When replaying, mimic the browser's request shape: send the same headers in the same order, reuse a persistent connection, and pace requests humanly. A perfect cookie with a mismatched User-Agent or a burst of 500 requests per second will still get the session flagged. Think of the handoff as inheriting an identity—everything about that identity must stay consistent.

Detecting Expiry and Falling Back

Sessions expire. Build detection into the client: if a response is a login page, a challenge page, or a 401/403, the session is dead. On expiry, transparently re-run the browser flow to obtain fresh cookies, then resume. Automate the full lifecycle: solve → snapshot → fetch → detect expiry → re-solve. A session manager that handles this loop is the difference between a script and a scraper that runs for weeks.

Security and Hygiene

Treat captured cookies as secrets. Store them encrypted, scope them to the specific target, and never log token values. Persist them to a small secure store rather than checking them into code. If a session is compromised or shared, it can be abused—treat it with the same care as an API key.

The handoff pattern is the single biggest throughput win available to a browser-based scraper. One browser solve can fuel a thousand fast requests, turning an expensive, slow pipeline into a cheap, fast one without abandoning the browser's unique capabilities.