Infinite Scroll and Lazy Loading

Pages That Are Never Really Done#

Many modern pages do not paginate with links. They load more as you scroll: infinite scroll appends items as you approach the bottom, images load only when they enter the viewport, and virtualized lists (built with libraries such as react-window or TanStack Virtual) render only the rows currently visible, removing the rest from the DOM. A naive scraper reads the initial HTML, finds a handful of items, and stops. The right first question is: where does the data actually come from?

Work through the options in this order, cheapest first:

Check the Initial HTML First#

Server-rendered frameworks often ship the first batch of data, and sometimes all of it, as JSON inside the page so the app can start without an extra request. Next.js uses <script id="__NEXT_DATA__" type="application/json">, Nuxt uses a window.__NUXT__ assignment, and many shops include application/ld+json structured data. You can read these with a plain HTTP request:

import json
import httpx
from lxml import html as lh

resp = httpx.get("https://example.com/feed", timeout=15.0)
tree = lh.fromstring(resp.content)

blob = tree.xpath('string(//script[@id="__NEXT_DATA__"])')
if blob:
    data = json.loads(blob)
    print(list(data["props"]["pageProps"].keys()))   # explore to find the items

The structure under props is app-specific, so print the keys level by level until you find the list. Even when only the first batch is embedded, the JSON often contains the cursor or page parameters for the next request, which tells you exactly how the scroll endpoint works.

Replay the Scroll Endpoint#

Open DevTools, go to Network, filter by Fetch/XHR, clear the log, and scroll. Each time new items appear you should see a request such as api/feed?cursor=... or search?page=3&size=24. That endpoint is the real data source, and the scroll is just the trigger. Replaying it directly is faster and more reliable than any browser, as covered in XHR and API reverse engineering.

import time

client = httpx.Client(timeout=15.0, headers={"Accept": "application/json"})

def fetch_all(query, max_pages=500):
    cursor, seen = None, set()
    for _ in range(max_pages):
        params = {"q": query, "limit": 24}
        if cursor:
            params["cursor"] = cursor
        data = client.get("https://example.com/api/feed", params=params).json()
        new = [it for it in data["items"] if it["id"] not in seen]
        for item in new:
            seen.add(item["id"])
            yield item
        cursor = data.get("next_cursor")
        if not cursor or not new:
            return
        time.sleep(1.0)

The stop conditions are the important part. Feeds signal the end in different ways: an empty items list, a missing or null cursor, a has_more: false flag, or, in badly built APIs, a repeat of the last page. Checking both "no cursor" and "nothing new" catches all of them. The max_pages cap protects you from feeds that never end, such as recommendation streams that keep generating items. Page- or offset-based endpoints are replayed the same way; see pagination for the variants.

If the endpoint requires a signature or token you cannot reproduce, you can still let a browser make the requests and capture the responses rather than scraping the DOM, as described in reversing JavaScript signatures.

Drive the Scroll in a Browser#

When there is no usable endpoint, scroll a real browser with Playwright. Waiting a fixed time after each scroll works, but it is either too slow or too short. A better signal is the network response that delivers the next batch:

from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1280, "height": 900})
    page.goto("https://example.com/feed", wait_until="domcontentloaded")
    items = page.locator(".feed-item")
    items.first.wait_for()

    stalls = 0
    for _ in range(200):                              # hard cap on batches
        before = items.count()
        try:
            with page.expect_response(lambda r: "/api/feed" in r.url, timeout=8000):
                items.last.scroll_into_view_if_needed()
                page.mouse.wheel(0, 800)
        except PWTimeout:
            pass                                      # no request fired this time
        page.wait_for_timeout(300)                    # let the app render the batch
        if items.count() == before:
            stalls += 1
            if stalls >= 3:
                break
        else:
            stalls = 0

    rows = items.evaluate_all(
        "els => els.map(e => ({id: e.dataset.id, title: e.querySelector('h3')?.innerText}))")
    print("collected", len(rows))
    browser.close()

What each part does:

• scroll_into_view_if_needed() on the last item. Most infinite scroll implementations use an IntersectionObserver on a sentinel element near the end of the list. Bringing the last item into view triggers it reliably, whatever the page height.
• mouse.wheel produces real wheel events, for the implementations that listen to scroll events instead. Both together cover almost every loader.
• expect_response waits exactly as long as the next batch takes to arrive. The timeout is not an error: on the final scroll no request fires.
• Three consecutive stalls before stopping. A single stall can be a slow network or a debounce delay; three in a row almost always means the end. This is more robust than stopping on the first unchanged count.
• The 200-batch cap. Endless feeds exist; a scraper without a cap runs until the browser runs out of memory.

Many pages also show a "Load more" button instead of, or after, a number of automatic loads. Check for it on each iteration (page.get_by_role("button", name="Load more")) and click it when visible.

Virtualized Lists: Collect As You Go#

Virtualization breaks the counting approach. The list keeps roughly the same number of DOM nodes however far you scroll, reusing them for new rows, so items.count() never grows and items you scrolled past no longer exist in the DOM. Extract each visible batch as you go, keyed by a stable id, and stop when a scroll adds nothing new:

collected = {}
stalls = 0
while stalls < 3 and len(collected) < 10_000:
    batch = page.locator("[role=row]").evaluate_all(
        "els => els.map(e => [e.dataset.rowId || e.innerText, e.innerText])")
    added = 0
    for key, text in batch:
        if key not in collected:
            collected[key] = text
            added += 1
    stalls = stalls + 1 if added == 0 else 0
    page.mouse.wheel(0, 600)                  # less than a screen, so no rows are skipped
    page.wait_for_timeout(400)
print(len(collected), "unique rows")

Scroll by less than one viewport per step so no row appears and disappears between two snapshots. If rows lack an id attribute, fall back to a combination of visible fields. Virtualized grids often render inside their own scrollable container rather than the window; in that case hover the mouse over the container first (page.locator(".grid").hover()) so the wheel events scroll it, not the page.

Lazy-Loaded Images#

Images that load on scroll usually keep their real URL in the HTML under a different attribute, so you rarely need to scroll for them. Look for data-src, data-srcset, data-lazy, or a srcset alongside a tiny placeholder src. Native lazy loading (loading="lazy") keeps the real URL in src from the start.

for img in tree.xpath("//img"):
    url = img.get("data-src") or img.get("src")
    srcset = img.get("data-srcset") or img.get("srcset")
    if srcset:                                # pick the widest candidate
        url = srcset.split(",")[-1].strip().split(" ")[0]
    print(url)

Taking the last srcset candidate assumes the list is sorted by width, which is common but not guaranteed; parse the w descriptors if accuracy matters. Downloading the images themselves is covered in media scraping.

How It Fails in Practice#

Scrolling to the end of a long feed can mean hundreds of requests to the site's API in a few minutes. Pace the loop, stop at the depth you actually need, and prefer the embedded JSON or the endpoint approach, which loads far less on the server than a browser fetching every image and script along the way.