Trust Nothing, Verify Everything#

A scraper has one job: produce records as clean as if someone typed them by hand. Raw HTML hands you strings with currency symbols, stray whitespace, HTML entities, non-breaking spaces, and missing attributes. If those strings flow straight into a database, the whole dataset becomes a liability at analysis time. Cleaning and validation are the difference between a dataset you trust and a dataset you quietly avoid.

The pipeline has a fixed order, and each step has one responsibility:

  1. Extract raw strings from the page, with no interpretation.
  2. Normalize each string into a canonical type.
  3. Validate the whole record against a schema.
  4. Deduplicate on a stable key when storing.
  5. Monitor the batch for drift.

Keeping extraction and normalization separate is what makes the pipeline debuggable: when a price is wrong, you can look at the raw string that produced it.

Clean the Text Before Anything Else#

Scraped text carries invisible junk that breaks equality checks and joins. The usual suspects are HTML entities that were not decoded, non-breaking spaces (\xa0), zero-width characters, and Unicode forms that look identical but compare differently.

import html
import re
import unicodedata

ZERO_WIDTH = dict.fromkeys(map(ord, "​‌‍"))

def clean_text(raw):
    if raw is None:
        return None
    text = html.unescape(raw)                    # & -> &,   -> \xa0
    text = unicodedata.normalize("NFKC", text)   # \xa0 -> space, full-width digits -> ASCII
    text = text.translate(ZERO_WIDTH)
    text = re.sub(r"\s+", " ", text).strip()
    return text or None

print(repr(clean_text("  Café Latte​\n  ")))
# 'Café Latte'

NFKC normalization is aggressive: it also turns ligatures like "fi" into "fi" and superscripts into plain digits. That is what you want for matching and search fields. For text you intend to display verbatim, store the original alongside the cleaned version. Mojibake such as café is an encoding bug, not a cleaning problem, and must be fixed at decode time (see parsing engines and encoding).

Normalize Into Canonical Types#

Decide on canonical forms and convert every scraped value into one. Prices become Decimal (never float, which cannot represent 0.1 exactly and will produce totals like 19.990000000000002), dates become ISO 8601, booleans become real booleans. Do the conversion close to the parse, once, so every downstream consumer sees the same shape.

The trap with prices is locale. 1,299.00 and 1.299,00 are the same number written by different countries, and a naive "strip everything except digits and dots" turns the second into 1.29900.

from decimal import Decimal, InvalidOperation

def parse_price(raw):
    if not raw:
        return None
    s = re.sub(r"[^\d.,-]", "", raw)
    if not s:
        return None
    last_dot, last_comma = s.rfind("."), s.rfind(",")
    if last_comma > last_dot:
        # comma is the decimal separator if one or two digits follow it: 1.299,00 or 12,5
        if len(s) - last_comma - 1 in (1, 2):
            s = s.replace(".", "").replace(",", ".")
        else:
            s = s.replace(",", "")          # 1,299 -> thousands separator
    else:
        s = s.replace(",", "")
    try:
        return Decimal(s)
    except InvalidOperation:
        return None

for raw in ["$1,299.00", "1.299,00 €", "£12", "1,299", "Call for price"]:
    print(f"{raw!r:>18} -> {parse_price(raw)}")

       '$1,299.00' -> 1299.00
      '1.299,00 €' -> 1299.00
             '£12' -> 12
           '1,299' -> 1299
  'Call for price' -> None

The heuristic is not perfect: 1,299 could mean one-point-two-nine-nine in a comma-decimal locale. When you scrape a single site, you usually know its locale, so pass it in explicitly instead of guessing. Store the currency as its own field ("EUR"), taken from the page or the site's region, not inferred from the symbol, because $ alone is used by several currencies.

Dates follow the same rule. Try the formats the site actually uses, and return None rather than a guess:

from datetime import datetime

DATE_FORMATS = ("%Y-%m-%d", "%d %b %Y", "%b %d, %Y", "%d/%m/%Y")

def parse_date(raw):
    raw = clean_text(raw)
    if not raw:
        return None
    for fmt in DATE_FORMATS:
        try:
            return datetime.strptime(raw, fmt).date().isoformat()
        except ValueError:
            continue
    return None

Ambiguous formats such as 03/04/2025 (March 4 or 3 April?) are the classic silent error. Only list one of %d/%m/%Y and %m/%d/%Y per site. Relative dates like "2 hours ago" must be resolved against the fetch time, which is another reason to store fetched_at on every record.

A value that cannot be parsed becomes None, not a silent string. You want to see the gap between how many records you scraped and how many parsed cleanly.

Validate the Shape With a Schema#

After normalization, assert invariants before saving. Hand-written if checks work for three fields; beyond that, a schema library keeps the rules in one readable place and reports every problem at once. Pydantic (v2) is the common choice in Python:

from datetime import date
from decimal import Decimal
from pydantic import BaseModel, Field, HttpUrl, ValidationError, field_validator

class Product(BaseModel):
    sku: str = Field(min_length=1, max_length=64)
    title: str = Field(min_length=2)
    price: Decimal = Field(ge=0, le=1_000_000)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    url: HttpUrl
    in_stock: bool
    released: date | None = None

    @field_validator("title")
    @classmethod
    def not_placeholder(cls, v):
        if v.lower() in {"loading...", "untitled", "n/a"}:
            raise ValueError("placeholder title")
        return v

raw = {"sku": "KB-104", "title": "Loading...", "price": "-5",
       "currency": "usd", "url": "not a url", "in_stock": "yes"}
try:
    Product.model_validate(raw)
except ValidationError as e:
    for err in e.errors():
        print(err["loc"], err["msg"])

('title',) Value error, placeholder title
('price',) Input should be greater than or equal to 0
('currency',) String should match pattern '^[A-Z]{3}$'
('url',) Input should be a valid URL, relative URL without a base

Two details worth noticing. Pydantic coerces in its default "lax" mode: "yes" became True and "-5" became a Decimal before the range check failed. That is convenient, but if you want the schema to catch a normalizer that forgot to run, use model_config = ConfigDict(strict=True). Second, the placeholder check catches a real scraping failure: the page was captured before JavaScript filled it in.

Do not throw invalid records away. Write them to a quarantine table or file with the error list and the source URL. A spike in quarantined records is often the first sign of a site change, and the stored errors tell you which field broke.

Deduplicate on a Stable Key#

Pages repeat records. The same product appears on ten listing pages, the same article under two categories. Deduplicate on a business key that is independent of the page: a SKU, an external id, a canonical URL. Push the constraint into the database so duplicates cannot slip in even when your code forgets to check.

CREATE TABLE products (
    sku        TEXT PRIMARY KEY,
    title      TEXT NOT NULL,
    price      TEXT,            -- Decimal stored as text to keep precision
    currency   TEXT NOT NULL,
    fetched_at TEXT NOT NULL
);

INSERT INTO products (sku, title, price, currency, fetched_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(sku) DO UPDATE SET
    title = excluded.title,
    price = excluded.price,
    fetched_at = excluded.fetched_at;

The upsert means a re-scrape updates the price and the row is still one row. If you need price history, write to a separate append-only table keyed by (sku, fetched_at) as well. When no clean key exists (people, companies, addresses), deduplication becomes a fuzzy matching problem; that is the subject of entity resolution and record linkage.

Detect Layout Drift Before It Silences You#

The most expensive failure in scraping parses without error but returns empty fields. The site renames a class, your selector returns nothing, and the pipeline stores hundreds of records of None. Catch it with per-field completeness checks on every batch:

FIELDS = ("title", "price", "sku", "in_stock")
THRESHOLDS = {"title": 0.99, "price": 0.95, "sku": 0.99, "in_stock": 0.90}

def completeness_report(records):
    n = max(1, len(records))
    report = {f: sum(r.get(f) is not None for r in records) / n for f in FIELDS}
    broken = {f: pct for f, pct in report.items() if pct < THRESHOLDS[f]}
    return report, broken

report, broken = completeness_report(batch)
if broken:
    raise RuntimeError(f"parser drift suspected: {broken}")

Per-field thresholds matter because fields have different natural fill rates: a discount price may legitimately be missing on most products, while a title never should be. Set each threshold from a week of healthy runs. Pair it with a record-count check (today's batch is less than half of yesterday's) and a distribution check (median price moved by 10x, which usually means cents were parsed as units). The layout drift detection lesson builds this into a full monitoring loop.

A failed scrape you hear about is a ten-minute fix. A scrape that quietly returns garbage for a week poisons everything built on top of it.

Keep Samples Instead of Trusting Memory#

When a target changes, you want the old markup to debug against, not a memory of it. Keep a rolling handful of raw HTML samples per page type (for example, the last five product pages that parsed with full completeness). When the check fires, diff an old sample against a fresh page and you will see the exact class rename or moved attribute. Store them next to the fixtures in your test suite and you get regression tests for free: every selector change must still parse the old samples correctly.

Practice#

Take 50 records from a scraper you have written and run them through a Pydantic model with strict types and range checks. Count how many fail and group the failures by field. Then deliberately break one selector, rerun, and confirm the completeness report names the right field before anything is written to the database.