Handling Pagination

Crawling Across Multiple Pages#

Listings, search results, archives, and API collections are almost never delivered in one response. To build a complete dataset your scraper must recognize how a site paginates, walk every page exactly once, and know with confidence when it has reached the end. Getting the loop to run is easy; getting it to stop correctly and prove it collected everything is the actual skill.

The Pagination Styles You Will Meet#

Identify the style before writing code: click to page 2 and 3 with DevTools open, and watch both the address bar and the Network tab.

1. Page Numbers and Offsets#

For query-parameter pagination, loop until the site stops giving you new rows. The naive "stop when the page is empty" loop has two weaknesses: some sites never return an empty page (page 999 silently shows the last page again), and a transient error page with zero items ends your crawl early. Guard both.

import requests
from bs4 import BeautifulSoup

def scrape_numbered(session, base_url, max_pages=500):
    seen_ids = set()
    for page in range(1, max_pages + 1):
        resp = session.get(base_url, params={"page": page}, timeout=15)
        if resp.status_code == 404:
            break
        resp.raise_for_status()
        soup = BeautifulSoup(resp.text, "html.parser")
        cards = soup.select(".product-item")
        ids = [c.get("data-id") for c in cards]

        if not cards:
            break
        if all(i in seen_ids for i in ids):
            print(f"page {page} repeats earlier rows; stopping")
            break
        seen_ids.update(ids)
        for card in cards:
            yield {"id": card.get("data-id"), "title": card.select_one("h3").get_text(strip=True)}
    else:
        print(f"hit max_pages={max_pages}; check whether the listing is really that long")

The for ... else branch runs only if the loop was never broken, which is exactly the "we hit the cap" case worth logging. Offset pagination is the same loop with params={"offset": page * limit, "limit": limit} and an extra stop rule: a page with fewer than limit items is the last one.

2. Following the Next Link#

When page URLs are not predictable, follow the site's own "next" link. Always resolve it against the current URL, because href="?page=3" or href="/list/3" are relative.

from urllib.parse import urljoin

def follow_next(session, start_url, max_pages=500):
    url, visited = start_url, set()
    while url and url not in visited and len(visited) < max_pages:
        visited.add(url)
        resp = session.get(url, timeout=15)
        resp.raise_for_status()
        soup = BeautifulSoup(resp.text, "html.parser")
        yield from soup.select(".product-item")

        nxt = soup.select_one('a[rel="next"], a.next:not(.disabled), li.next > a')
        url = urljoin(resp.url, nxt["href"]) if nxt and nxt.get("href") else None

urljoin(resp.url, ...) uses the final URL after redirects, not the one you requested, so relative links resolve correctly. The visited set catches sites whose last page links back to itself.

APIs often put the next link in a response header instead. requests parses it for you:

resp = session.get("https://api.example.com/items", params={"per_page": 100}, timeout=15)
print(resp.headers.get("Link"))
# <https://api.example.com/items?page=2&per_page=100>; rel="next", <...page=37...>; rel="last"
next_url = resp.links.get("next", {}).get("url")

3. Cursors#

Cursor APIs return an opaque token that points at the position after the last item you received. You send it back unchanged until the server stops returning one.

def walk_cursor(client, url, page_size=100):
    cursor = None
    while True:
        params = {"limit": page_size}
        if cursor:
            params["cursor"] = cursor
        body = client.get(url, params=params, timeout=15).json()
        yield from body["items"]
        cursor = body.get("next_cursor")
        if not cursor or not body["items"]:
            break

Cursors are more robust than offsets because they are anchored to a record rather than a position, so inserts during your crawl do not shift the window. Do not try to decode or construct them yourself; even when they look like base64 JSON, the server may validate or sign them. Save the latest cursor to disk if the crawl is long, since resuming from a cursor is how you survive a crash (see caching and resume), but expect some cursors to expire after minutes or hours.

4. Infinite Scroll and "Load More"#

Infinite scroll is cursor or offset pagination with a scroll listener on top. Open the Network tab, filter to Fetch/XHR, scroll, and the request that fires each time you near the bottom is your paginated endpoint. Replicating it directly is far faster and more stable than driving a browser to scroll. The finding hidden JSON APIs and infinite scroll lessons cover the details, including the cases where a browser really is needed.

How Pagination Fails in Practice#

Shifting offsets. On a "newest first" listing, if five items are published while you are on page 3, everything moves down five slots: page 4 now starts with items you already saw, and five items slip past the boundary you already crossed. Mitigations: sort by a stable key if the site allows it (sort=id_asc or oldest first), deduplicate by ID, and prefer cursor endpoints.

Result caps. Many sites stop at a fixed depth, such as 50 pages or 10,000 results, regardless of how many items match. Page 51 returns an error or repeats page 50. The fix is to partition: split the query by a filter (category, price range, date range, region) until each slice fits under the cap, then paginate each slice.

def partitioned_ranges(count_for, lo, hi, cap=1000):
    '''Yield (lo, hi) price ranges whose result count fits under the cap.'''
    n = count_for(lo, hi)
    if n <= cap or hi - lo <= 1:
        yield (lo, hi)
        return
    mid = (lo + hi) // 2
    yield from partitioned_ranges(count_for, lo, mid, cap)
    yield from partitioned_ranges(count_for, mid, hi, cap)

count_for requests page 1 of the filtered listing and reads the displayed total. Watch the boundaries: if the site's price filter is inclusive on both ends, an item priced exactly at mid appears in both halves, which your ID deduplication absorbs.

Unstable order. Listings sorted by "relevance" or "recommended" may be personalized or randomized per request, so page 2 of one request and page 2 of another are different sets. Choose an explicit deterministic sort whenever the site offers one.

Session-bound pages. Some sites store the search in your session and paginate with ?page=2 relative to it. Fetching page 2 in a fresh session returns the wrong results or an error; use one session per listing.

Oversized pages. Raising per_page=20 to per_page=100 cuts requests fivefold and is usually fine. Servers typically clamp large values silently, so check the actual item count returned rather than assuming the parameter was honored, and do not push it to extremes that make each request expensive for the server.

Proving Completeness#

Most listings display a total ("1,284 results"). Parse it on page 1, and at the end compare it to the number of unique IDs you collected. A small gap is normal on live data; a large one means you hit a cap, a stop condition fired early, or offsets shifted. Log pages fetched, items per page, duplicates skipped, and the stop reason for every run. When a crawl that yielded 1,284 items yesterday yields 240 today, those four numbers tell you why in seconds.

Practice#

Write one generator per style (numbered, next link, cursor) that shares a common interface: for item in paginate(...). Test each against saved pages or a small mock server where you control the data, including a last page that repeats, a listing capped at five pages, and an item inserted mid-crawl. Each generator must stop, log its stop reason, and produce no duplicate IDs.