Scraping Is Easy; Trusting the Data Is Hard

A scraper that runs once produces a file. A scraper that runs for years produces a dataset—and datasets are only useful if you can answer: where did this row come from, when was it captured, and what changed between then and now? That is the domain of versioning and lineage.

Reproducibility as a Requirement

If someone questions a number in your dataset, you must be able to trace it back to the exact page and the exact moment it was captured. That requires storing, alongside every record:

  • The source URL and the HTTP status.
  • The capture timestamp (UTC).
  • The raw payload (or a content-addressed hash of it).
  • The scraper version and the extraction logic version.

Without provenance, your dataset is an assertion; with it, it is evidence.

Snapshots vs. Diffs

Two complementary storage strategies:

  • Immutable snapshots: append every crawl as a new partition (e.g., date=2026-09-17/hour=03/). Storage is cheap and you can always reconstruct any past state. This is the safest default.
  • Diffs / change logs: store only what changed plus a running current state. Far smaller, but dependency-heavy—you need the full history to reconstruct any past view.

Most teams keep immutable raw snapshots and derive current views from them. Raw data is never mutated or deleted by the parsing layer.

Content-Addressed Raw Storage

Store raw fetched bytes under a hash of their content (SHA-256) rather than by URL. Identical content fetched from many URLs dedupes automatically, and you can verify integrity years later. Keep a small index mapping URL, timestamp, and hash so you can find and replay any page.

Storage Formats and Layout

  • Columnar formats (Parquet): the default for analytics—compressed, typed, and fast to scan.
  • Partitioning: by date and source so queries prune irrelevant files.
  • Schema evolution: add new columns as your extraction improves; never silently change the meaning of an existing column. Version your schema and document changes.
  • Data lake layout: raw zone (immutable snapshots) → cleaned zone (normalized records) → curated zone (entity-resolved golden records). Each zone is rebuildable from the one before it.

Schema Evolution and Backfills

When you add a field or fix a parser bug, you will want to backfill history. Because you stored raw snapshots, you can re-run the extraction over old data without re-scraping. This is a decisive argument for keeping raw payloads: the expensive part (network) only happens once, and every parsing improvement can be applied retroactively.

Retention and Cost

Raw data accumulates fast. Set retention policies—keep all raw snapshots for, say, 90 days, then keep only deduped content hashes plus a monthly archive. Balance storage cost against the value of re-deriving history. Deleting raw data is irreversible; compress and archive before you ever consider dropping it.

Auditing and Trust

Finally, make the dataset explainable. Record counts per crawl, extraction failure rates, and anomaly alerts. When a downstream analyst sees a spike, they should be able to ask "which pages produced this, and when?" and get an answer in minutes. Versioning and lineage are what turn a pile of scraped rows into a system people can rely on.