Scraping JavaScript-Rendered Pages

Overcoming the Single Page Application (SPA)#

The classic stack of requests plus BeautifulSoup assumes the data is in the HTML the server sends. Sites built with React, Vue, Angular, or Svelte often break that assumption. The server returns a thin shell and a JavaScript bundle; the browser runs the bundle, the bundle calls an API, and only then is the DOM filled in. Fetch such a page with requests and you get something like:

<body>
  <div id="root"></div>
  <script src="/static/js/main.4f9c2a.js"></script>
</body>

Your selectors match nothing, not because they are wrong, but because the data never arrived in that response. The good news is that "rendered by JavaScript" does not automatically mean "needs a browser". The data had to reach the page somehow, and usually you can fetch it the same way.

Rendering Models and What They Mean for You#

• Server-side rendering (SSR): the server sends complete HTML. Plain HTTP scraping works.
• Client-side rendering (CSR): the server sends a shell; JavaScript fetches JSON and builds the DOM. Find the JSON API.
• SSR with hydration: frameworks such as Next.js and Nuxt send full HTML and embed the same data as a JSON blob so the client can "hydrate" without refetching. This is the best case: the data is in the first response, already structured. Next.js puts it in <script id="__NEXT_DATA__" type="application/json"> (pages router), and Nuxt commonly assigns window.__NUXT__. Other apps use names like window.__INITIAL_STATE__ or window.__APOLLO_STATE__.
• Streaming and islands: newer frameworks may stream HTML in chunks or embed data in several script tags. Still fetchable, just messier to extract.

Step 1: Diagnose Before You Spend Browser RAM#

Pick a value you can see on screen, a price or a product name that is unlikely to appear elsewhere, and check whether it is in the raw response:

import re
import httpx

def diagnose(url: str, needle: str) -> None:
    html = httpx.get(url, timeout=15.0, follow_redirects=True).text
    print("bytes:", len(html))
    print("needle in raw HTML:", needle in html)
    print("empty root div:", bool(re.search(r'<div id="(root|app|__next)"></div>', html)))
    print("__NEXT_DATA__:", '<script id="__NEXT_DATA__"' in html)
    print("window state vars:", sorted(set(re.findall(r"window[.](__[A-Z_]+__)", html))))
    print("json-ld blocks:", html.count("application/ld+json"))

diagnose("https://shop.example.com/product/123", "Mechanical Keyboard")

Read the output as a decision:

• Needle present, no state blob: it is ordinary HTML. Use selectors.
• Needle present inside a script blob: extract the JSON blob; no browser needed.
• Needle absent: the data comes from a later request. Go to the Network tab.

Step 2: Extract Embedded State#

When the data sits in a hydration blob, parse it as JSON rather than scraping the rendered markup:

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
tag = soup.select_one("script#__NEXT_DATA__")
data = json.loads(tag.string)
props = data["props"]["pageProps"]
print(list(props.keys()))

The structure under pageProps is whatever the site's developers chose, so explore it interactively once (print(json.dumps(props, indent=2)[:2000])) and then write a small function that maps it to your record shape. Blobs assigned to window.X = {...}; inside a larger script need a regex to locate the start plus a real JSON decoder to find the end; the regular expressions lesson shows a safe way to do that.

Step 3: Replay the API the Page Calls#

If the data is not in the first response, open DevTools, go to Network, filter to Fetch/XHR, and reload. The request that returns your data as JSON is the one to replicate. Calling it directly gives you clean, structured records with no rendering at all, and it is typically one or two orders of magnitude cheaper per page than a browser. The full workflow (copying requests, minimizing headers, paginating, handling tokens) is in finding the hidden JSON APIs.

Step 4: Headless Browsers, When They Are Worth It#

Use a real browser when the API requests carry signatures or tokens generated by obfuscated code, when content only appears after interaction, or when replicating the request chain would take longer than it is worth. Playwright is the modern default: one API drives Chromium, Firefox, and WebKit, and it auto-waits for elements.

from playwright.sync_api import sync_playwright

BLOCK = {"image", "media", "font"}

def scrape_dynamic(url: str):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(locale="en-US", viewport={"width": 1366, "height": 900})
        page = context.new_page()

        # Skip heavy resources we never read
        page.route("**/*", lambda route: route.abort()
                   if route.request.resource_type in BLOCK else route.continue_())

        # Capture the JSON the page itself fetches, instead of scraping the DOM
        with page.expect_response(lambda r: "/api/products" in r.url and r.ok) as resp_info:
            page.goto(url, wait_until="domcontentloaded")
        api_data = resp_info.value.json()

        # Or read the rendered DOM once a specific element exists
        page.locator(".product-row").first.wait_for(timeout=15_000)
        titles = page.locator(".product-row .title").all_inner_texts()

        context.close()
        browser.close()
        return api_data, titles

The non-obvious choices:

• Wait for the thing you need, not for "idle". wait_until="networkidle" waits for 500 ms with no network activity. Pages with analytics beacons, polling, or websockets may never go idle, and Playwright's own documentation discourages it. Waiting on a concrete locator or response is faster and deterministic.
• expect_response gives you the API payload that the page fetched with all its tokens and signatures already applied. This hybrid, a browser for authentication and JSON for data, is often the most robust setup.
• Blocking images, fonts, and media cuts bandwidth and memory substantially on media-heavy pages. Do not block scripts or XHR, or the page will not render.
• Contexts are cheap, browsers are not. Launch one browser and create a fresh new_context() per job or identity; each context has isolated cookies and storage. Always close contexts, or memory grows until the process is killed.

How Browser Scraping Fails#

• Timeouts on elements that never appear. Often the page served a consent banner, a login redirect, or a block page instead. Save page.screenshot(path="fail.png") and page.content() on every failure; they show you what the browser actually saw.
• Content below the fold. Lazy-loaded lists only fetch more items on scroll. Scroll in steps with page.mouse.wheel(0, 2000) and wait for the item count to grow, or better, call the pagination API directly.
• Iframes. Content inside an iframe is a separate document. Use page.frame_locator("iframe#reviews").locator(".review").
• Shadow DOM. Playwright CSS locators pierce open shadow roots automatically; BeautifulSoup on page.content() does not see shadow content at all, so query with Playwright instead.
• Headless differences. Some sites serve headless browsers differently. That is an anti-bot question, covered in browser automation and the anti-bot course, and the same rules about permission and rate limits apply there as everywhere else.

Choosing the Tool#

Treat the list as a budget, not a preference: each step down costs substantially more CPU, memory, and time per page, and a crawl of a million pages feels every step. Try the cheap routes first on every new target, even if the site "looks like" a heavy SPA.

Practice#

Pick three sites you are allowed to crawl: one plain server-rendered site, one Next.js or Nuxt site, and one client-rendered SPA. Run diagnose() on each, then extract the same five fields with the cheapest method that works. Record the wall time and bytes transferred per page for each method, including a Playwright run on all three for comparison.