A Production Stealth Playwright Pipeline
On this page
Browsers With a Solid Gold Jailbreak#
Before Playwright, the standard advice was "use requests or curl_cffi for speed, use a browser only when forced." The truth is that once you need real JavaScript, you need a real browser—and with the right pipeline, a browser is neither the slow lose-lose tool it used to be, nor a guaranteed ban. A production stealth pipeline treats the browser like an expensive tool that must be warmed, reused, and authenticated before it ever touches a real page.
The Pipeline Stages#
1. Profile Selection Every job maps to a long-lived browser profile: a fixed fingerprint (screen size, platform, fonts, timezone), a fixed geolocation, and a stable pool of residential IPs for that region. Creating the profile once—and reusing it across sessions—keeps the WAF's internal device identity constant. Randomizing per request destroys that continuity.
2. Proxy Assignment
Assign the profile a residential or mobile proxy before browser startup, and configure the proxy at the browser level (--proxy-server), not through a PAC file that changes mid-session. Check the exit IP, timezone, and language after launch; any mismatch means the session is already suspect.
3. Warm-Up Sequence
Navigate first to a neutral page (about:blank, then your provider's homepage), let fonts load, let the media stack initialize, and collect the browser's own TLS session. Only then navigate to the target. This also builds connection reuse so the first "real" request looks like a continuing session rather than a cold open.
4. The Human Workflow
Drive the browser with human-shaped interactions: mouse moves along Bézier paths, scroll with variable velocity and occasional stops, type with randomized keystroke timing. Never call page.click() cold on a heavily protected site—give the pointer time to travel.
5. Cookie Extraction & Handoff
If the site issues a clearance cookie (cf_clearance, datadome, Datadome's __datadome token) or a session cookie, snapshot it and reuse it in a lightweight curl_cffi session for high-throughput follow-up requests. This is the standard throughput optimization: the browser solves challenges and collects tokens; the fast HTTP client exploits them.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True, args=["--no-sandbox"])
ctx = browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36",
viewport={"width": 1920, "height": 1080},
locale="en-US",
)
page = ctx.new_page()
page.goto("about:blank")
page.goto("https://nowsecure.nl")
page.wait_for_timeout(8000)
cookies = ctx.cookies() # harvest clearance token
browser.close()
6. Monitoring & Escape Hatch Every browser session should assert its own health: the presence of an expected DOM element, the absence of a challenge page, and response codes of 200. If a detection page appears, immediately kill the browser, discard the IP, and requeue the URL—fighting a confirmed detection wastes time and burns IPs.
When to Accelerate#
A single warmed browser session might handle 30–80 requests a minute on a script-heavy site. When your job needs thousands of pages a minute, the browser pipeline is intentionally the first stage only: solve the challenge, harvest the token, and stream the actual data extraction through curl_cffi. The browser queue is a challenge-solving farm, and the HTTP layer is the bandwidth engine.
A Pipeline Is a Sequence of Constraints#
Every Playwright stealth setup is a list of calls, and almost every one of them has an ordering constraint. The proxy credentials must be known before the context is created. The user-agent metadata and the timezone and locale must be context options, so they must exist before the context is created. Init scripts, route handlers and permissions must all be registered before the first navigation, or the first document escapes them.
None of these constraints are enforced by the API. Playwright will happily let you create a context, navigate, and then add an init script, and the call will succeed and apply to the next document. The result is a stack that works on every page except the first one, which is exactly the hardest failure to notice in a test suite and the easiest to hit in production.
# A stealth pipeline is a sequence with ordering constraints, and most of
# the bugs are ordering bugs. Model the constraints explicitly and a bad
# plan becomes a list of violations instead of a mysterious early leak.
STAGES = [
"launch-args",
"proxy-auth",
"user-agent-metadata",
"timezone-locale",
"context-options",
"init-script-preload",
"network-interception",
"permissions",
"page-navigation",
]
# first must be applied before second, or the second has no effect on the
# first document the client ever sees
RULES = [
("launch-args", "context-options"),
("proxy-auth", "context-options"),
("user-agent-metadata", "context-options"),
("timezone-locale", "context-options"),
("context-options", "permissions"),
("init-script-preload", "page-navigation"),
("network-interception", "page-navigation"),
]
PROFILE = {
"id": "de-chrome-131-004",
"timezone": "Europe/Berlin",
"locale": "de-DE",
"geolocation": (52.5200, 13.4050),
"viewport": {"width": 1536, "height": 864},
"device_scale_factor": 1.25,
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/131.0.0.0 Safari/537.36",
"ua_metadata": {
"brands": [{"brand": "Chromium", "version": "131"},
{"brand": "Not_A Brand", "version": "24"},
{"brand": "Google Chrome", "version": "131"}],
"platform": "Windows", "platform_version": "15.0.0",
"architecture": "x86", "bitness": "64", "model": "",
"mobile": False,
},
"accept_language": "de-DE,de;q=0.9,en;q=0.8",
}
CALL = {
"launch-args": "launch(args=[...])",
"proxy-auth": "context(proxy={'server':..., 'username':...})",
"user-agent-metadata": "context(user_agent=..., user_agent_metadata=...)",
"timezone-locale": "context(locale=..., timezone_id=...)",
"context-options": "context(viewport=..., device_scale_factor=...)",
"init-script-preload": "context.add_init_script(...)",
"network-interception": "context.route('**/*', ...)",
"permissions": "context.grant_permissions([...], origin=...)",
"page-navigation": "page.goto(url)",
}
def check_order(plan):
position = dict((stage, i) for i, stage in enumerate(plan))
violations = []
for first, second in RULES:
if first in position and second in position:
if position[first] > position[second]:
violations.append((first, second,
position[first], position[second]))
return violations
def coherence(profile):
errors = []
ua = profile["user_agent"]
if "Chrome/131" not in ua:
errors.append("user agent does not name the claimed major version")
brands = [b["brand"] for b in profile["ua_metadata"]["brands"]]
if "Not_A Brand" not in brands:
errors.append("ua metadata is missing the Not_A Brand entry")
if "Google Chrome" not in brands:
errors.append("ua metadata does not claim Google Chrome")
if not profile["accept_language"].startswith(profile["locale"]):
errors.append("accept-language %r disagrees with locale %r"
% (profile["accept_language"], profile["locale"]))
if profile["geolocation"] and not profile["timezone"].startswith("Europe/"):
errors.append("geolocation is european, timezone is %r"
% profile["timezone"])
if profile["viewport"]["width"] % 2 or profile["viewport"]["height"] % 2:
errors.append("viewport has an odd dimension, which real screens "
"essentially never do")
return errors
def render(label, plan):
print(f"{label}:")
position = dict((stage, i) for i, stage in enumerate(plan))
for stage in plan:
print(f" {position[stage] + 1:>2}. {stage:<22} {CALL[stage]}")
violations = check_order(plan)
print(f" ordering rules checked: {len(RULES)}, "
f"violations: {len(violations)}")
for first, second, a, b in violations:
print(f" - {first} (step {a + 1}) must precede "
f"{second} (step {b + 1})")
return violations
# the well-formed plan is exactly the constraint graph's topological order
GOOD = list(STAGES)
# a plan a tired engineer writes: patch after navigating, intercept after
# navigating, metadata after the context exists, permissions too early
SLOPPY = [
"context-options",
"launch-args",
"timezone-locale",
"proxy-auth",
"permissions",
"user-agent-metadata",
"page-navigation",
"init-script-preload",
"network-interception",
]
def main():
print("profile bundle under test")
for key in ["id", "timezone", "locale", "accept_language",
"device_scale_factor"]:
print(f" {key:<22} {PROFILE[key]!r}")
print(f" {'viewport':<22} {PROFILE['viewport']}")
print(f" {'geolocation':<22} {PROFILE['geolocation']}")
print(f" {'ua brand count':<22} {len(PROFILE['ua_metadata']['brands'])}")
print()
errors = coherence(PROFILE)
print("coherence check on the profile")
if not errors:
print(" clean: user agent, brand list, locale and timezone agree")
for error in errors:
print(f" - {error}")
print()
good = render("well-formed plan", GOOD)
print()
sloppy = render("plan written at 2am", SLOPPY)
print()
print("what the violations cost")
for first, second, _a, _b in sloppy:
if (first, second) == ("init-script-preload", "page-navigation"):
cost = "the first document renders with no overrides at all"
elif (first, second) == ("network-interception", "page-navigation"):
cost = "the first request escapes the route handler unmodified"
elif (first, second) == ("launch-args", "context-options"):
cost = "launch flags are ignored because the browser is already up"
elif (first, second) == ("user-agent-metadata", "context-options"):
cost = "metadata is attached to a context that already exists"
elif (first, second) == ("timezone-locale", "context-options"):
cost = "locale falls back to the host, so the profile is not the profile"
elif (first, second) == ("proxy-auth", "context-options"):
cost = "the context connects directly, with no proxy credentials"
elif (first, second) == ("context-options", "permissions"):
cost = "permissions granted before the context they apply to exists"
else:
cost = "the dependent stage runs too late to apply"
print(f" {first} after {second}: {cost}")
print()
print("the constraint graph, and the only orders that satisfy it")
for first, second in RULES:
print(f" {first:<22} -> {second}")
print()
print(" every rule is an edge, and the only safe plan is a topological")
print(" order of the edge set. a pipeline that renders from a list of")
print(" stages without checking the edges is a pipeline that will leak")
print(" on the first document whenever somebody reorders the list for")
print(" readability, which is the most common cause of a regression in")
print(" a stack that was working yesterday.")
main()
profile bundle under test
id 'de-chrome-131-004'
timezone 'Europe/Berlin'
locale 'de-DE'
accept_language 'de-DE,de;q=0.9,en;q=0.8'
device_scale_factor 1.25
viewport {'width': 1536, 'height': 864}
geolocation (52.52, 13.405)
ua brand count 3
coherence check on the profile
clean: user agent, brand list, locale and timezone agree
well-formed plan:
1. launch-args launch(args=[...])
2. proxy-auth context(proxy={'server':..., 'username':...})
3. user-agent-metadata context(user_agent=..., user_agent_metadata=...)
4. timezone-locale context(locale=..., timezone_id=...)
5. context-options context(viewport=..., device_scale_factor=...)
6. init-script-preload context.add_init_script(...)
7. network-interception context.route('**/*', ...)
8. permissions context.grant_permissions([...], origin=...)
9. page-navigation page.goto(url)
ordering rules checked: 7, violations: 0
plan written at 2am:
1. context-options context(viewport=..., device_scale_factor=...)
2. launch-args launch(args=[...])
3. timezone-locale context(locale=..., timezone_id=...)
4. proxy-auth context(proxy={'server':..., 'username':...})
5. permissions context.grant_permissions([...], origin=...)
6. user-agent-metadata context(user_agent=..., user_agent_metadata=...)
7. page-navigation page.goto(url)
8. init-script-preload context.add_init_script(...)
9. network-interception context.route('**/*', ...)
ordering rules checked: 7, violations: 6
- launch-args (step 2) must precede context-options (step 1)
- proxy-auth (step 4) must precede context-options (step 1)
- user-agent-metadata (step 6) must precede context-options (step 1)
- timezone-locale (step 3) must precede context-options (step 1)
- init-script-preload (step 8) must precede page-navigation (step 7)
- network-interception (step 9) must precede page-navigation (step 7)
what the violations cost
launch-args after context-options: launch flags are ignored because the browser is already up
proxy-auth after context-options: the context connects directly, with no proxy credentials
user-agent-metadata after context-options: metadata is attached to a context that already exists
timezone-locale after context-options: locale falls back to the host, so the profile is not the profile
init-script-preload after page-navigation: the first document renders with no overrides at all
network-interception after page-navigation: the first request escapes the route handler unmodified
the constraint graph, and the only orders that satisfy it
launch-args -> context-options
proxy-auth -> context-options
user-agent-metadata -> context-options
timezone-locale -> context-options
context-options -> permissions
init-script-preload -> page-navigation
network-interception -> page-navigation
every rule is an edge, and the only safe plan is a topological
order of the edge set. a pipeline that renders from a list of
stages without checking the edges is a pipeline that will leak
on the first document whenever somebody reorders the list for
readability, which is the most common cause of a regression in
a stack that was working yesterday.
The Profile Bundle Is the Contract#
The other half of pipeline correctness is that the values it sets have to agree with each other. The profile above is a coherent bundle: a German timezone, a German locale, an Accept-Language that starts with that locale, a Berlin geolocation, an even viewport dimension, a device scale factor of 1.25, and a Chrome 131 user agent whose userAgentData brands include Not_A Brand and Google Chrome.
Each of those is a field someone could set independently and break. The coherence check is deliberately simple and checks agreement rather than plausibility: the user agent and the metadata brand list, the locale and the accept-language, the geolocation and the timezone, the viewport parity. Those are the four classes of error that actually occur when a profile is assembled from configuration fragments written by different people. Plausibility against a real device population is the more expensive check described in device profile consistency, and it belongs at profile creation time, not at request time.
The reason to check at creation is cost. A profile is created once and used for thousands of requests; a coherence failure is a property of the bundle, not of the request, so checking it per request is pure waste. The reason to check at all is that a bundle assembled from parts is coherent by accident, and the accident only holds until somebody changes one part.
Ordering Violations and What They Cost#
The well-formed plan produces zero violations against seven ordering rules. The plan written at 2am produces six, and the six are worth reading as a catalogue of how this goes wrong.
Four of them are the same mistake in different clothing: launch-args, proxy-auth, user-agent-metadata and timezone-locale all run after context-options. That is what happens when setup is factored into helper functions and someone decides the context should be created first "so we have something to configure". The context is created with no proxy, no locale, no timezone and no metadata, and every later attempt to set them applies to a context that already exists. Depending on the option, the call either throws, silently applies only to future pages, or is ignored entirely. Proxy credentials are the worst of the four, because the observable symptom is a direct connection rather than an error.
The other two are the classic first-document bugs. init-script-preload and network-interception run after page-navigation, so the first document renders with no overrides and the first request bypasses the route handler. Every subsequent document is fine. A test suite that navigates, asserts, and navigates again will never see it.
The Async Footgun: Whoever Navigates First#
There is a second version of the same bug that ordering checks cannot catch, because the plan is correct and the execution is not. It appears when more than one task shares a browser.
# The most common playwright stealth bug is not a missing override. It is
# an override registered after the first navigation, so the first document
# the server ever sees is the unpatched one. Model the interleaving.
STOCK = {
"navigator.webdriver": True,
"window.chrome": "absent",
"navigator.languages": ("en-US",),
}
PATCHED = {
"navigator.webdriver": False,
"window.chrome": "present",
"navigator.languages": ("de-DE", "de"),
}
class Page:
def __init__(self, name):
self.name = name
self.patched = False
self.documents = 0
def read(self):
self.documents += 1
return self.name, self.documents, PATCHED if self.patched else STOCK
def reader(page, label):
# first document, then a reload: a real flow navigates, patches, reloads
yield ("read", label, page.read())
yield ("read", label + "+reload", page.read())
def patcher(page, label):
page.patched = True
yield ("patch", label, None)
def schedule(coros):
# round-robin over generators. a generator that yields is suspended at
# the yield and resumed next pass, which is the smallest honest model of
# cooperative interleaving and involves no clock at all.
pending = list(coros)
events = []
while pending:
rest = []
for coro in pending:
try:
events.append(next(coro))
rest.append(coro)
except StopIteration:
pass
pending = rest
return events
def show(label, events):
leaks = [e for e in events
if e[0] == "read" and e[2][2]["navigator.webdriver"]]
print(f" {label}")
for kind, who, detail in events:
if kind == "patch":
print(f" patch {who}")
else:
_page, number, values = detail
state = "stock" if values["navigator.webdriver"] else "patched"
print(f" read {who:<18} document {number} -> {state}")
print(f" first-document leaks: {len([e for e in leaks if e[2][1] == 1])}"
f", any-document leaks: {len(leaks)}")
return len(leaks)
def run(label, order):
pages = {}
coros = []
for kind, page_name in order:
page = pages.setdefault(page_name, Page(page_name))
if kind == "read":
coros.append(reader(page, page_name))
else:
coros.append(patcher(page, page_name))
return show(label, schedule(coros))
def main():
print("two documents per page: an initial navigation and a reload")
print("an override is only effective for documents that follow it")
print()
results = []
print("scenario 1: one page, navigate then patch")
results.append(run("navigate, patch, reload",
[("read", "A"), ("patch", "A")]))
print()
print("scenario 2: one page, patch then navigate")
results.append(run("patch, navigate",
[("patch", "A"), ("read", "A")]))
print()
print("scenario 3: two pages sharing one browser, patch last")
results.append(run("A reads, B reads, then A patches",
[("read", "A"), ("read", "B"), ("patch", "A")]))
print()
print("scenario 4: two pages, each patches before it navigates")
results.append(run("A patches, A reads, B patches, B reads",
[("patch", "A"), ("read", "A"),
("patch", "B"), ("read", "B")]))
print()
print(" scenario 3 is the one that ships. the code reads correctly as")
print(" sequential, a second page is added later, and the round-robin")
print(" order means the patch for A is only applied after B has already")
print(" fetched its first document. the fix is not to move the patch")
print(" call earlier in A's function. the fix is to make the patch part")
print(" of context setup, so no page can exist before it is applied.")
print()
print(" %d leaks across the four scenarios, all of them on a document"
% sum(results))
print(" whose page had not been patched yet. none of them are visible")
print(" in a synchronous reading of the code, which is why this class")
print(" of bug survives review and shows up as a small challenge rate")
print(" that only the first navigation of each page triggers.")
main()
two documents per page: an initial navigation and a reload
an override is only effective for documents that follow it
scenario 1: one page, navigate then patch
navigate, patch, reload
read A document 1 -> stock
patch A
read A+reload document 2 -> patched
first-document leaks: 1, any-document leaks: 1
scenario 2: one page, patch then navigate
patch, navigate
patch A
read A document 1 -> patched
read A+reload document 2 -> patched
first-document leaks: 0, any-document leaks: 0
scenario 3: two pages sharing one browser, patch last
A reads, B reads, then A patches
read A document 1 -> stock
read B document 1 -> stock
patch A
read A+reload document 2 -> patched
read B+reload document 2 -> stock
first-document leaks: 2, any-document leaks: 3
scenario 4: two pages, each patches before it navigates
A patches, A reads, B patches, B reads
patch A
read A document 1 -> patched
patch B
read B document 1 -> patched
read A+reload document 2 -> patched
read B+reload document 2 -> patched
first-document leaks: 0, any-document leaks: 0
scenario 3 is the one that ships. the code reads correctly as
sequential, a second page is added later, and the round-robin
order means the patch for A is only applied after B has already
fetched its first document. the fix is not to move the patch
call earlier in A's function. the fix is to make the patch part
of context setup, so no page can exist before it is applied.
4 leaks across the four scenarios, all of them on a document
whose page had not been patched yet. none of them are visible
in a synchronous reading of the code, which is why this class
of bug survives review and shows up as a small challenge rate
that only the first navigation of each page triggers.
Reading the Interleaving#
Four scenarios make the failure mode clear. Patching before navigating is safe. Navigating before patching leaks the first document and is repaired by the reload. The interesting rows are the two-page scenarios, because they show the difference between a per-page fix and a shared one.
In scenario 3, page A navigates, page B navigates, and then A patches. Every first document in the run is unpatched, and B's reload is still unpatched, because A's patch applied to A's context and B never got one. Four leaks across the run, and none of them are visible in a synchronous reading of the code: both functions look correct in isolation, and the bug exists only in the interleaving.
Scenario 4 fixes it by patching each page before it navigates, but that is the fragile fix, because it depends on every future author remembering to do it in the right order. The durable fix is the one the ordering check implies: make the patch part of context setup, so that no page can exist before the patch is registered.
That is the same conclusion as the synchronous plan, reached from a different direction. addInitScript, route and grant_permissions all belong at context creation, not at page setup, and once they are there the interleaving cannot produce an unpatched document because there is no window in which an unpatched page exists. This is the practical difference between hardening a Playwright stack through context configuration and bolting overrides onto pages: the first removes an ordering bug by construction, the second removes it by discipline.
Detecting It in Production#
The signature of a first-navigation leak is specific enough to monitor for. A small fraction of sessions fail a challenge on the very first request to a host and succeed on every request after it. The rate is low, it is stable, and it never reproduces locally, because a development loop reloads more often than it navigates fresh.
Three things make it visible. Log the challenge outcome by navigation ordinal per page, so the first-document rate is a number rather than an anecdote. Run the audit from headless detection leaks against the first document of every session in a canary, not against a warmed-up page. And assert the pipeline order in a unit test over the plan, so a refactor that moves a stage fails in CI rather than in a challenge rate three days later.
The ordering check is the cheap half of this and the one people skip, because a plan that runs correctly looks correct. The generator model is the other half: it makes the interleaving deterministic and reproducible, so a bug that depends on task scheduling can be turned into a failing test with no timing, no sleeps and no flakiness. That property is worth more than any single override, which is why automation framework comparison and browser pool architecture both spend their time on setup structure rather than on stealth primitives: a pipeline assembled in the right order needs far fewer primitives to hide.