From Raw JSON to Clean Records#

Many targets are not really HTML. The page loads, a hidden API returns JSON, and the page's JavaScript turns it into what you see. Once you have found that API (see XHR and API reverse engineering), the remaining work is turning a deep, inconsistent JSON tree into flat, typed rows. JSON is easier than HTML in one way (the structure is explicit) and harder in another: fields appear and disappear per record, types change between records, and one-to-many relationships hide inside nested lists. This lesson is about extracting from it defensively.

Read the Shape Before Writing Code#

Save one real response to disk and inspect it before writing any extractor. Working from a saved file also means you are not hitting the API every time you adjust a line of code.

import json
import httpx

resp = httpx.get("https://api.example.com/search", params={"q": "lamps", "page": 1}, timeout=15)
resp.raise_for_status()
with open("sample_search.json", "w", encoding="utf-8") as f:
    json.dump(resp.json(), f, indent=2, ensure_ascii=False)

Printing one record shows you what a typical record looks like. It does not show you what varies across records, and variation is what breaks extractors. A short walker that lists every path, its type, and how often it appears answers that in one pass:

from collections import Counter

def walk(node, path="$", out=None):
    out = Counter() if out is None else out
    if isinstance(node, dict):
        for key, value in node.items():
            walk(value, f"{path}.{key}", out)
    elif isinstance(node, list):
        out[(path, "array")] += 1
        for item in node:
            walk(item, f"{path}[]", out)
    else:
        out[(path, type(node).__name__)] += 1
    return out

payload = json.load(open("sample_search.json", encoding="utf-8"))
for (path, kind), n in sorted(walk(payload).items()):
    print(f"{n:5}  {kind:8} {path}")

    1  array    $.results
   48  str      $.results[].id
   48  str      $.results[].title
   41  float    $.results[].price.amount
    4  str      $.results[].price.amount
    3  NoneType $.results[].price.amount
   48  str      $.results[].price.currency
   31  array    $.results[].variants
   77  str      $.results[].variants[].sku
    1  int      $.total
    1  str      $.next_cursor

This output (trimmed) tells you things a single record would hide: 48 results, but price.amount is a float in 41, a string in 4 and null in 3; only 31 records have variants; and pagination uses a cursor. Every one of those facts turns into a line of defensive code.

On the command line, jq gives the same quick look: jq '.results[0] | keys' sample_search.json lists one record's keys, and jq -c '.results[] | {id, price: .price.amount}' sample_search.json prints a compact projection of every record.

Traverse Defensively#

Chains like rec["price"]["current"]["amount"] raise KeyError or TypeError the first time a record lacks a level, and one bad record should not kill a crawl of thousands. A small helper that walks dict keys and list indexes, and returns a default for anything missing, removes that whole class of crash:

def dig(node, *path, default=None):
    for key in path:
        if isinstance(key, int):
            if not isinstance(node, list) or not -len(node) <= key < len(node):
                return default
        elif not isinstance(node, dict) or key not in node:
            return default
        node = node[key]
    return default if node is None else node

dig(rec, "price", "amount")                  # 24.99, or None
dig(rec, "images", 0, "url")                 # first image URL, or None
dig(rec, "seller", "rating", default=0)      # 0 if missing or null

Be deliberate about defaults. Returning 0.0 for a missing price makes missing data look like a free product. Return None for "unknown" and let validation decide what to do with it.

One Parser Function per Record Type#

Put all knowledge of one record's shape in one function that returns a flat dict. It is easy to unit-test against saved samples, and when the API changes, there is exactly one place to fix.

from decimal import Decimal, InvalidOperation
from datetime import datetime, timezone

def to_decimal(value):
    if value is None or value == "":
        return None
    try:
        return Decimal(str(value))
    except InvalidOperation:
        return None

def to_datetime(value):
    if isinstance(value, (int, float)):               # epoch seconds or milliseconds
        seconds = value / 1000 if value > 1e11 else value
        return datetime.fromtimestamp(seconds, tz=timezone.utc)
    if isinstance(value, str) and value:
        return datetime.fromisoformat(value.replace("Z", "+00:00"))
    return None

def parse_product(rec: dict) -> dict:
    cents = dig(rec, "price", "amount_minor")
    return {
        "id": str(dig(rec, "id")),
        "title": (dig(rec, "title", default="")).strip(),
        "price": to_decimal(cents) / 100 if cents is not None else to_decimal(dig(rec, "price", "amount")),
        "currency": dig(rec, "price", "currency"),
        "in_stock": dig(rec, "availability", "status") in ("IN_STOCK", "in_stock", True),
        "updated_at": to_datetime(dig(rec, "updatedAt")),
        "brand": dig(rec, "brand", "name"),
    }

Type problems you will meet in real payloads:

  • Numbers as strings. "24.99" in some records, 24.99 in others. Convert everything through one function.
  • Money in minor units. Many APIs send 2499 meaning 24.99. Check against the price shown on the page before assuming.
  • Floats for money. json.loads turns 24.99 into a binary float. For financial data, parse with json.loads(text, parse_float=Decimal) so values stay exact.
  • Timestamps. ISO strings, epoch seconds and epoch milliseconds all occur, sometimes in the same API. The 1e11 threshold in to_datetime separates seconds from milliseconds for any date after 1973.
  • Large IDs. JavaScript loses precision on integers above 2 to the power of 53, so APIs designed for browsers often send large IDs as strings, or send both a number and a string version. Store IDs as strings and prefer the string field.

Validate the Output, Not Just the Input#

A schema library turns "the API changed" from silent bad data into a loud error. With pydantic, declare what a good record looks like and validate each parsed dict:

from pydantic import BaseModel, ValidationError

class Product(BaseModel):
    id: str
    title: str
    price: Decimal | None
    currency: str | None
    in_stock: bool

good, bad = [], []
for rec in payload["results"]:
    try:
        good.append(Product.model_validate(parse_product(rec)))
    except ValidationError as exc:
        bad.append((dig(rec, "id"), exc.errors()[0]["msg"]))
print(len(good), "valid;", len(bad), "rejected")

Track the rejection rate per run. A jump from 0% to 30% after a deploy on the site's side is your earliest warning of a schema change. More on this in data cleaning and validation.

Flatten, or Split Into Tables?#

Storage wants flat rows; JSON wants trees. pandas.json_normalize flattens nested objects into dotted columns and can unwrap a nested list while carrying parent fields along:

import pandas as pd

variants = pd.json_normalize(
    payload["results"],
    record_path="variants",
    meta=["id", "title", ["brand", "name"]],
    meta_prefix="product_",
    errors="ignore",
)
print(variants.columns.tolist())

['sku', 'color', 'size', 'stock', 'product_id', 'product_title', 'product_brand.name']

errors="ignore" fills missing meta fields with NaN instead of raising. Records without a variants key raise an error with record_path, so filter those out first or give them an empty list.

The real decision is structural. Nested objects (a product's price, brand) flatten naturally into columns. Nested lists (a product's variants, reviews, images) are one-to-many relationships. Flattening them into columns like variants.0.sku, variants.1.sku produces a sparse, unqueryable table. Store them as a child table keyed by the parent ID instead, as in the example above: one row per variant, with product_id as the link. The storing data lesson covers the schema side.

JSON Hidden Inside HTML#

Many server-rendered pages embed the full data object in a script tag, which is often cleaner than scraping the rendered HTML:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")

# Framework data blocks and structured data are plain JSON
tag = soup.select_one('script#__NEXT_DATA__, script[type="application/json"]')
data = json.loads(tag.string) if tag else None

products_ld = [json.loads(s.string) for s in soup.select('script[type="application/ld+json"]') if s.string]

Some pages assign the data to a variable instead: window.__INITIAL_STATE__ = {...};. The trailing semicolon and any code after it make json.loads fail. JSONDecoder.raw_decode parses one JSON value starting at a position and ignores what follows:

text = next(s.string for s in soup.find_all("script") if s.string and "__INITIAL_STATE__" in s.string)
start = text.index("{", text.index("__INITIAL_STATE__"))
state, end = json.JSONDecoder().raw_decode(text, start)

If the object is a JavaScript literal rather than strict JSON (unquoted keys, single quotes, undefined), raw_decode will fail; the chompjs library parses those.

Large Responses#

A 500 MB JSON export loaded with json.load can take several gigabytes of memory as Python objects. Stream it instead with ijson, which yields items one at a time from a path:

import ijson

with open("export.json", "rb") as f:
    for rec in ijson.items(f, "results.item"):
        handle(parse_product(rec))

Some APIs return JSON Lines (one object per line, often application/x-ndjson). Parse those line by line with json.loads on each; never try to load the whole body as one document.

Deduplicate and Paginate by Stable Keys#

Paginated APIs return overlapping records when data changes between page requests, so the same product can appear on pages 1 and 2. Deduplicate on the API's own stable ID, not on title or URL, and prefer cursor pagination (next_cursor, after) over page numbers when the API offers both, because cursors do not skip or repeat items when records are inserted mid-crawl. Pass cursors back exactly as received. The loop mechanics are in pagination and consuming REST APIs.

Practice#

Save three pages of a JSON API response. Run walk over all three combined, write down every path whose type varies, and make parse_product handle each case. Then validate with the pydantic model and confirm the rejection count is zero, or that every rejection is genuinely bad data.