Finding the Hidden JSON APIs

Scrape the Data, Not the DOM#

Modern frontends rarely hold the data themselves. A React or Vue page is usually a shell: the browser runs a JavaScript bundle, and that bundle makes fetch() and XHR calls to an internal API that returns JSON. Every row you see on screen travelled over the network as a structured payload before it became HTML.

If you find that API and call it directly, a whole category of problems disappears. No selectors to maintain, no waiting for rendering, no headless browser. You get the same records the site's own frontend gets, already typed and nested, ready to load into a database.

Why the API Beats the DOM#

The DOM is a presentation layer full of noise: navigation, ads, tracking scripts, hidden elements, formatting that changes with every redesign. The internal API is a contract between the site's frontend and backend. It changes less often, because changing it means changing the frontend too, and its payloads are smaller and cleaner.

The cost difference is large. A JSON request costs tens of milliseconds and a few kilobytes. A headless browser costs hundreds of megabytes of memory and seconds of CPU per page. Across a million pages, that is the difference between a job that finishes overnight and one that needs a cluster.

APIs also often expose fields the page never displays: stable IDs, exact timestamps, unrounded numbers, stock counts, category trees. Those are frequently the most valuable columns in the final dataset.

Step 1: Capture the Network Traffic#

Open DevTools, select the Network tab, then set it up properly:

• Tick Preserve log so requests survive redirects and page navigations.
• Filter to Fetch/XHR to hide images, scripts, and stylesheets.
• Reload the page, then perform the action that loads the data: scroll, click "next", change a filter, open a detail view.

Candidate requests usually have a few tells:

• The response is JSON (check the Preview tab for a browsable tree).
• The path contains api, v1/v2, graphql, search, or a resource name like products.
• The query string or body carries values that match what is on screen: page number, category, search term.

When you cannot spot the right request, search for the data itself. The Network panel's search (Ctrl+F with the panel focused, or the magnifier icon) looks inside response bodies, so searching for a product name you can see on screen leads straight to the request that delivered it. The Initiator column shows which script made the call, which helps later if you need to understand how a parameter is computed.

Step 2: Replay the Request Outside the Browser#

Right-click the request and choose Copy as cURL. Run it once in a terminal to confirm it works outside the browser at all. Then translate it to Python, but do not keep all fifty headers: most are noise, and some (like cookies that expire in minutes) make your scraper fragile.

import httpx

url = "https://store.example.com/api/products"
params = {"category": "keyboards", "page": 1, "per_page": 50}
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0 Safari/537.36",
    "Accept": "application/json",
}

with httpx.Client(headers=headers, timeout=10.0) as client:
    resp = client.get(url, params=params)
    resp.raise_for_status()
    data = resp.json()

print(data.get("total"))
print(data["results"][0])

Step 3: Find the Minimum Request#

Which headers and cookies does the endpoint actually require? Find out empirically. Start from the full copied header set, remove one at a time, and check whether the response is still equivalent:

def minimal_headers(client, url, params, full_headers, check):
    '''Return the smallest header set for which check(resp) stays True.'''
    needed = dict(full_headers)
    for name in list(full_headers):
        trial = {k: v for k, v in needed.items() if k != name}
        resp = client.get(url, params=params, headers=trial)
        if check(resp):
            needed = trial                  # header was not required
        else:
            print(f"required: {name} (status {resp.status_code})")
    return needed

ok = lambda r: r.status_code == 200 and "results" in r.json()

Run it with a fresh client that does not carry extra default headers, or the defaults will mask the result. Typical findings: the endpoint needs Accept: application/json, sometimes a Referer or Origin, sometimes X-Requested-With, a session cookie, or an API key header that is the same for every visitor and embedded in the page's JavaScript. Keep that list short, and note why each header is there; when the scraper breaks months later, that note is the first thing you check. Space the trial requests out; this is a few dozen requests, not a burst.

Step 4: Explore the Parameters#

Once you can replay the request, treat the parameters as an interface to explore:

• Page size. Try raising per_page or limit. Servers usually clamp it at some maximum; check the number of items actually returned instead of assuming the value was honored.
• Fields. Some APIs accept fields= or include= to add or remove attributes.
• Sorting. A stable sort such as by ID or creation date makes pagination reliable.
• Filters. Category, date range, and price range parameters let you partition large result sets that hit pagination caps; see pagination.

Change one parameter at a time, compare responses, and write down what you learn. When the UI adds a new filter or column, watch the Network tab again and a new parameter usually appears.

Step 5: Walk Pagination Through the API#

APIs paginate by offset (page, offset, start) or by cursor (an opaque next_cursor token passed back unchanged). Cursors are anchored to a record rather than a position, so items added mid-crawl do not shift the window.

import httpx

def save(record):
    print("storing:", record["id"])

with httpx.Client(timeout=10.0, headers={"Accept": "application/json"}) as client:
    url = "https://store.example.com/api/products"
    cursor = None
    while True:
        params = {"per_page": 100}
        if cursor:
            params["cursor"] = cursor
        resp = client.get(url, params=params)
        resp.raise_for_status()
        body = resp.json()
        for item in body["results"]:
            save(item)
        cursor = body.get("next_cursor")
        if not cursor or not body["results"]:
            break

With offsets, stop when a page returns fewer items than requested, and guard against servers that ignore the offset and keep returning the same page by tracking the IDs you have seen.

Step 6: Tokens and Authentication#

For endpoints behind a login you are authorized to use, the browser holds either a session cookie or a bearer token. A cookie-based session works naturally with a persistent client:

import os
import httpx

with httpx.Client(timeout=10.0) as client:
    login = client.post(
        "https://store.example.com/api/login",
        json={"email": os.environ["STORE_USER"], "password": os.environ["STORE_PASS"]},
    )
    login.raise_for_status()
    orders = client.get("https://store.example.com/api/orders",
                        headers={"Accept": "application/json"}).json()

Token-based APIs send Authorization: Bearer .... Tokens usually expire, often within an hour, so a token pasted from DevTools works in testing and fails overnight. Find the endpoint that issues or refreshes tokens, call it from your code, and refresh when you receive a 401. The authentication sessions lesson covers refresh flows. Some sites also use GraphQL, a single endpoint with queries in the request body; the GraphQL scraping lesson covers its specifics.

Diagnosing a Replay That Fails#

Tools such as mitmproxy make this comparison easier by recording both the browser's and your script's traffic in one place; see mitmproxy and HAR files.

When the API Is Locked Down#

Some sites sign each request with a hash of the parameters and a timestamp, or require a token minted by obfuscated JavaScript. Reverse engineering those schemes is a treadmill: the site redeploys and your implementation breaks. A pragmatic middle ground is to let a real browser load the page and capture the JSON responses it receives, which the JavaScript-rendered pages lesson shows with Playwright. Whether you should be scraping that data at all is a separate question: an undocumented API is not a licence, and the same robots.txt, terms, and rate-limit considerations apply as for HTML pages.

Practice#

Pick a site you are allowed to crawl that loads its listings via XHR. Capture the request, reduce it to the minimal headers, find the largest page size the server honors, and write a paginator that collects every item. Compare the item count with the total the page displays and with a Playwright run that scrapes the same listing from the DOM.