Real Browser Automation with Playwright

When Only a Browser Will Do#

Every earlier lesson steered you toward the fastest path: plain HTTP, and wherever possible the site's own internal API. Some targets make that impossible. The page is rendered entirely by JavaScript and the data is not in any JSON response you can replay, the API requests are signed by obfuscated code, or the site requires a login flow that only works in a real browser. For those cases you stop imitating a browser and start driving one. Playwright is the standard tool: it controls real Chromium, Firefox, and WebKit engines, so the server sees an actual browser while your code decides where it goes and what it reads.

Running a real engine removes a whole class of problems. The TLS handshake is genuine, headers come out in the browser's order, JavaScript executes, and the site's own code signs its own requests. You pay in resources: a browser uses hundreds of megabytes of RAM and seconds per page, against milliseconds and kilobytes for an HTTP call. Treat it as the precise tool for pages that need it.

Install once with pip install playwright followed by playwright install chromium to download the engine.

Browser, Context, Page#

A Browser is one engine process. A Page is a tab. Between them sits the object you will care about most: the BrowserContext, an isolated profile with its own cookies, localStorage, cache, and permissions. Contexts are cheap (they start in milliseconds, unlike a browser), so one browser process can host many independent sessions.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        locale="en-US",
        timezone_id="America/New_York",
        viewport={"width": 1440, "height": 900},
    )
    context.set_default_timeout(15_000)   # ms; applies to every locator action
    page = context.new_page()
    page.goto("https://example.com/search", wait_until="domcontentloaded")
    print(page.title())
    context.close()
    browser.close()

Set identity (locale, timezone, viewport) on the context, not with ad-hoc headers, so everything the page can observe stays consistent. Leaving the user agent at Playwright's default is usually better than pasting in a string from another browser version, which creates a mismatch between what the headers claim and what the engine actually is. headless=False is useful while developing so you can watch what happens.

wait_until controls when goto returns. "domcontentloaded" is the right default for scraping; "load" waits for every image; "networkidle" waits for 500 ms without network traffic, which never happens on pages with analytics beacons or polling, so avoid it except on simple pages.

Locators Replace Sleeping#

The classic brittle script does time.sleep(3) and hopes the data has loaded. Playwright's Locator auto-waits: each action retries until the element is attached, visible, stable, and enabled, or the timeout passes. Your code describes the target instead of the timing.

from playwright.sync_api import expect

page.get_by_placeholder("Search").fill("keyboards")
page.get_by_placeholder("Search").press("Enter")

cards = page.locator(".results .card")
expect(cards.first).to_be_visible()

for card in cards.all():
    title = card.locator("h3").inner_text()
    price = card.locator(".price").text_content()
    print(title, price)

A locator is lazy: it is a query, re-evaluated each time you use it, so it never goes stale when the framework re-renders the list. get_by_role, get_by_text, get_by_placeholder, and get_by_label match what a user sees and survive CSS refactors better than class names. inner_text() returns rendered text (hidden elements excluded, CSS whitespace applied); text_content() returns raw DOM text, including hidden nodes.

One subtlety: cards.all() snapshots how many matches exist at that instant. Wait for the content first (as expect does above), otherwise you may iterate over an empty list while results are still loading.

Extracting in Bulk#

Calling inner_text() per element is a round-trip to the browser each time. For grids of hundreds of items, run one JavaScript expression in the page and return plain data:

rows = page.locator(".results .card").evaluate_all('''
  cards => cards.map(c => ({
    title: c.querySelector("h3")?.textContent.trim(),
    price: c.querySelector(".price")?.textContent.trim(),
    url:   c.querySelector("a")?.href,
  }))
''')
print(len(rows), rows[0])

evaluate_all passes the matched elements to the function; the return value must be JSON-serializable. Note a.href returns the absolute URL resolved by the browser, which saves you a urljoin. Often the fastest route of all is to skip the DOM and let the page render, then parse page.content() with lxml or BeautifulSoup as you would any HTML.

Watch the Page's Own Network Traffic#

The page's own API calls are frequently cleaner than its DOM. Capture them while triggering the action that causes them:

with page.expect_response(
    lambda r: "/api/items" in r.url and r.request.method == "GET" and r.ok
) as resp_info:
    page.get_by_role("button", name="Next").click()

payload = resp_info.value.json()
print(len(payload["results"]))

Register the expectation before the click, which is what the with block does; waiting after the click can miss a fast response. For passive collection across a whole session, page.on("response", handler) fires for every response.

You can also cut the page's weight by blocking resources you do not need:

BLOCK = {"image", "media", "font"}

def block_heavy(route):
    if route.request.resource_type in BLOCK:
        return route.abort()
    return route.continue_()

context.route("**/*", block_heavy)

Blocking images and fonts often halves load time. Do not block scripts or XHR, and be careful with stylesheets: lazy-loading and visibility checks depend on layout.

Shadow DOM and Iframes#

Web components hide their internals in a shadow root. Playwright's CSS and text locators pierce open shadow roots automatically, so page.locator("price-widget .amount") works even though .amount lives inside the component. XPath does not pierce shadow roots, so stick to CSS or role locators there. Closed shadow roots are deliberately inaccessible to page scripts.

Iframes are separate documents and need an explicit hop:

frame = page.frame_locator("iframe#reviews")
print(frame.locator(".review").count())

Keep Sessions Between Runs#

Logging in on every run is slow and draws attention. Save the context's cookies and localStorage once, and start later contexts from that file:

context.storage_state(path="state.json")          # after logging in
# next run:
context = browser.new_context(storage_state="state.json")

Treat state.json like a password: it contains live session tokens. The authentication and sessions lesson covers exporting these cookies into a plain HTTP client, so the browser only handles the login.

Debugging a Script That Works Locally and Fails in Production#

Most browser-scraping bugs come from timing and environment differences. The tools that find them:

• Tracing. Wrap a run in context.tracing.start(screenshots=True, snapshots=True) and context.tracing.stop(path="trace.zip"), then open it with playwright show-trace trace.zip. You get a timeline of every action, network request, and DOM snapshot. Record traces only on failure in production to keep disk use down.
• Inspector. Setting the environment variable PWDEBUG=1 runs headed with the Playwright Inspector, pausing before each action.
• Codegen. playwright codegen https://example.com records your clicks into a script and suggests robust locators.
• Screenshots on error. page.screenshot(path="fail.png", full_page=True) inside an except block shows whether you hit a consent banner, a login wall, or a block page.

Common failures and their causes:

The Cost of Running a Browser#

A browser is an operating-system-sized process. Launch it once and reuse it for the whole job; create a fresh context per task or identity and close it when the task finishes, which frees its memory. Cap concurrency by memory, not CPU: a handful of pages per gigabyte is a realistic starting point, measured on your own targets. In containers, Chromium's shared-memory use can exceed Docker's small default /dev/shm; raising it (for example --shm-size=1g) prevents random crashes. For many pages at once, the async API (playwright.async_api) with a semaphore works the same way as the patterns in the concurrency lesson.

Politeness rules do not change because a browser is doing the fetching: honour robots.txt and the site's terms, and keep request rates as low as with HTTP. A browser downloads every script and stylesheet too, so each page view costs the server more than a single API call.

Practice#

Pick a JavaScript-rendered page on a site that permits scraping. Write the scraper three ways: DOM extraction with evaluate_all, response capture with expect_response, and page.content() parsed with lxml. Time each for 20 pages with images blocked, and record a trace for one failed run to practise reading it.