Dataset Versioning & Lineage

Scraping Is Easy; Trusting the Data Is Hard#

A scraper that runs once produces a file. A scraper that runs for years produces a dataset, and a dataset is only useful if you can answer three questions about any row in it: where did this come from, when was it captured, and what changed between then and now. That is the job of versioning (keeping past states retrievable) and lineage (recording how each value was produced). Get them right and a scraped dataset becomes evidence; get them wrong and it is an unfalsifiable assertion that quietly rots.

Provenance: Store the Story With Every Record#

When someone disputes a number, you must trace it to the exact page and moment it came from. That means storing, alongside the extracted value:

• the source URL and HTTP status;
• the capture timestamp in UTC;
• a content hash of the raw payload (and the raw payload itself, addressed by that hash);
• the scraper version and the extraction logic version that produced the record.

import hashlib
import json
from datetime import datetime, timezone

def envelope(url, status, raw: bytes, record: dict, *,
             scraper_version, parser_version):
    return {
        "record": record,
        "meta": {
            "url": url,
            "status": status,
            "fetched_at": datetime.now(timezone.utc).isoformat(),
            "content_sha256": hashlib.sha256(raw).hexdigest(),
            "scraper_version": scraper_version,   # e.g. a git commit short hash
            "parser_version": parser_version,
        },
    }

The two version fields are what let you explain a change in the data as a change in your code rather than a change in the world. When a field's values shift on a given date, the first question is always "did we deploy a new parser then?", and the metadata answers it in seconds.

Snapshots vs Diffs#

Two storage strategies, with different trade-offs:

Most teams keep immutable raw snapshots and derive current and historical views from them. Storage is cheap relative to the cost of losing the ability to answer "what did this page say last March?" Partition raw data by date and source so queries touch only relevant files:

raw/source=shopA/date=2026-09-29/part-000.jsonl.gz
raw/source=shopA/date=2026-09-30/part-000.jsonl.gz

The parsing layer never mutates or deletes raw data. That single rule is what makes everything below possible.

Content-Addressed Raw Storage#

Store raw bytes under a hash of their content (SHA-256) rather than under the URL. Two benefits follow. Identical content fetched from many URLs is stored once, which deduplicates automatically. And you can verify integrity years later: re-hash the bytes and compare. Keep a small index mapping URL and timestamp to hash so you can find and replay any specific page:

from pathlib import Path

def put_raw(root: Path, raw: bytes) -> str:
    h = hashlib.sha256(raw).hexdigest()
    path = root / h[:2] / h[2:4] / h
    if not path.exists():                 # dedupe: identical bytes stored once
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_bytes(raw)
    return h

# index row: (url, fetched_at, content_sha256) in a database or Parquet file

Because the same unchanged page fetched daily hashes to the same value, your index shows exactly when content actually changed, independent of when you visited.

Storage Formats and Zones#

Organise the lake as zones, each rebuildable from the one before it:

• Raw zone: immutable snapshots of fetched bytes, content-addressed. Never edited.
• Cleaned zone: normalised, typed records, usually in a columnar format.
• Curated zone: entity-resolved golden records (see entity resolution and record linkage) ready for analysis.

For the cleaned and curated zones, Parquet is the default: columnar, compressed, typed, and fast to scan by column. Partition by date and source so a query for one day reads a handful of files instead of the whole history:

import pyarrow as pa
import pyarrow.parquet as pq

def write_partition(records, base, source, date):
    table = pa.Table.from_pylist(records)
    out = f"{base}/source={source}/date={date}"
    Path(out).mkdir(parents=True, exist_ok=True)
    pq.write_table(table, f"{out}/part-000.parquet", compression="zstd")

Reading back with a partition filter (pq.read_table(base, filters=[("source", "=", "shopA")])) prunes files it does not need, which is why partitioning pays off as the dataset grows.

Schema Evolution and Backfills#

Your extraction improves over time: you add a field, or fix a parser bug. Two rules keep that safe:

• Add columns; never silently change the meaning of an existing one. If price changes from "dollars" to "cents", make a new column and version the schema, or every historical comparison becomes wrong without warning.
• Backfill from raw, not from the web. Because you kept raw snapshots, you can re-run the new parser over old bytes and regenerate history without re-fetching a single page.

def backfill(raw_index, parser, from_date, to_date):
    for url, fetched_at, sha in raw_index.between(from_date, to_date):
        raw = read_raw(sha)               # bytes you already stored
        record = parser(raw, url)         # new/fixed extraction logic
        yield envelope(url, 200, raw, record,
                       scraper_version="backfill", parser_version=parser.version)

This is the decisive argument for keeping raw payloads: the expensive, un-repeatable part (the network fetch, capturing the site as it was that day) happens once, and every future parsing improvement applies retroactively for free. A pipeline that discards raw HTML can never fix a past extraction bug; it can only start collecting correctly from today.

Retention and Cost#

Raw data accumulates quickly, so set retention deliberately rather than letting storage grow without limit. A common shape: keep full raw snapshots hot for a recent window (say the last few months), then compress and move older snapshots to cheap archival storage, keeping the content-hash index forever so you can still prove what you had. Deleting raw data is irreversible and forecloses future backfills, so compress and archive long before you consider dropping anything. Content-addressing helps here too: dedupe means the storage bill grows with the amount of distinct content, not with the number of crawls.

Auditing and Trust#

Make the dataset explain itself. Record per-crawl metrics (record counts, extraction failure rates, fill rates per field) so anomalies are visible, and connect them to the raw layer so any spike is traceable:

def crawl_report(records):
    n = len(records)
    return {
        "records": n,
        "fill_rates": {f: sum(1 for r in records if r.get(f)) / max(n, 1)
                       for f in ("title", "price", "sku")},
        "distinct_hashes": len({r["meta"]["content_sha256"] for r in records}),
    }

When a downstream analyst sees Tuesday's average price jump, they should be able to ask "which pages produced this, and when were they captured?" and get an answer from the index in minutes, not a shrug. That traceability, paired with the drift checks from layout drift detection and the record-level rules from data cleaning and validation, is what turns a pile of scraped rows into a dataset people are willing to make decisions on.

A Common Anti-Pattern#

The failure this lesson guards against usually looks responsible from the outside. A team scrapes each night and writes the result straight into one table, upserting by key, keeping only the current value. It works, the dashboard is green, and nobody stores raw HTML because "we already extracted what we need". Then three things happen in the same quarter: a parser bug is found that mislabelled a field for weeks, an analyst asks what a price was on a specific past date, and a stakeholder disputes a number and wants proof. None can be answered. The bug cannot be back-fixed because the source bytes are gone; the past value is unrecoverable because only the current one was kept; and there is no provenance to show where the number came from. Every one of these is cheap to prevent up front and impossible to fix after the fact, which is why raw retention and provenance are decisions to make on day one, not after the first incident.

Checklist#

A dataset is versioned and lineage-tracked when: every record carries source URL, UTC capture time, content hash, and scraper and parser versions; raw payloads are stored immutably and content-addressed; current views are derived from raw, never edited in place; schema changes add columns and are versioned; you can re-run today's parser over last month's raw bytes and reproduce a past state; and per-crawl metrics let anyone trace an anomaly back to the pages that caused it.