Catching Layout Drift with Visual Diffs

The Silent Death of a Scraper#

A site redesign rarely returns a 403. It quietly serves new HTML in which your classes no longer exist, your parser returns None for every field, and the pipeline keeps running, writing empty or wrong rows for days. That is layout drift. Defending against it takes layers, from cheap assertions inside the parser to structural fingerprints and, for the cases markup cannot reveal, visual diffs of screenshots. Each layer catches something the others miss.

Layer 1: Fail Loudly Inside the Parser#

The cheapest detector lives where extraction happens. A common beginner pattern is worse than useless:

title = card.select_one(".title").get_text(strip=True)

When .title disappears, select_one returns None and this line raises AttributeError: 'NoneType' object has no attribute 'get_text', a crash that says nothing about drift. The opposite habit, wrapping everything in try/except and emitting None, is how pipelines fill with blank rows. Make missing required fields an explicit, named error:

class LayoutDrift(Exception):
    pass

def need(node, selector, url):
    el = node.select_one(selector)
    if el is None:
        raise LayoutDrift(f"{selector!r} missing on {url}")
    text = el.get_text(strip=True)
    if not text:
        raise LayoutDrift(f"{selector!r} empty on {url}")
    return text

def parse_card(card, url):
    return {
        "title": need(card, ".title", url),
        "price": need(card, ".price", url),
        "rating": (card.select_one(".rating") or {}).get("data-value"),  # optional
    }

Required fields go through need; genuinely optional ones are allowed to be missing. Deciding which is which is itself useful: it documents what "a valid record" means for this source. Pair this with value checks that catch a selector matching the wrong element, for example a price that must match ^[\d.,]+$ after stripping the currency. Selectors that still match but grab the wrong node are drift too.

Layer 2: Counts and Fill Rates#

Before parsing details, count the repeating units the parser depends on. A listing page that always renders 24 cards suddenly rendering 0 means the selector died, not that the catalogue emptied:

cards = soup.select("div.product-card")
if not cards:
    raise LayoutDrift(f"no product cards on {url}")
no_price = sum(1 for c in cards if c.select_one(".price") is None)
if no_price > 0.5 * len(cards):
    raise LayoutDrift(f"{no_price}/{len(cards)} cards lost .price on {url}")

Per-page checks miss slow drift, where one template out of five changes and 20% of records go blank. Catch that at the run level by comparing each field's fill rate with a trailing baseline:

def fill_rates(records, fields):
    n = len(records) or 1
    return {f: sum(1 for r in records if r.get(f) not in (None, "")) / n
            for f in fields}

today = fill_rates(records, ["title", "price", "rating", "image"])
for field, rate in today.items():
    base = baseline[field]  # e.g. median of the last 7 runs
    if rate < base - 0.15:
        alert(f"fill rate for {field} fell {base:.0%} -> {rate:.0%}")

A fall from 98% to 71% for price is a strong drift signal even though no single page failed. Keep the baseline per source and per page type; a search page and a product page have different normal rates. The scheduling and monitoring lesson shows how to route these alerts.

Layer 3: Structural Fingerprints#

Sometimes nothing breaks yet but the page has changed: a new wrapper div, renamed classes on elements you do not parse today, a moved section. A structural fingerprint notices before fields go blank. Reduce the page to its skeleton of tag and class paths, ignoring text, then compare skeletons with Jaccard similarity:

from bs4 import BeautifulSoup

def skeleton(html, root="main"):
    soup = BeautifulSoup(html, "lxml")
    base = soup.select_one(root) or soup.body
    paths = set()
    for el in base.find_all(True):
        chain = []
        for node in [el, *el.parents]:
            if node is base:
                break
            cls = ".".join(sorted(node.get("class", [])))
            chain.append(f"{node.name}.{cls}" if cls else node.name)
        paths.add(" > ".join(reversed(chain)))
    return paths

def similarity(a, b):
    return len(a & b) / len(a | b) if a | b else 1.0

Store the skeleton of a known-good page per template. On each run, compute the skeleton of one or two sample pages and compare. Scores that stay around the usual level (product pages vary a little with different numbers of reviews or images) are normal; a sudden drop signals a template change. Record the paths that disappeared: the diff (baseline - current) often points straight at the renamed class. Hashed or generated class names such as css-1x2y3z change on every deploy of some front-end frameworks; strip them with a regex before fingerprinting, or fingerprint on tags and stable attributes (data-testid, itemprop) only.

Layer 4: Visual Diffs#

Some drift leaves the DOM plausible but changes what users see: a price rendered as an image, content moved behind a "show more" button, a cookie wall covering the page, or a completely different page served to your scraper. Pixel evidence catches these. Take a screenshot with Playwright, masking regions that change legitimately:

from playwright.sync_api import sync_playwright

def screenshot(url, path, masks=(".ad-slot", ".live-timestamp")):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(viewport={"width": 1280, "height": 900})
        page.goto(url, wait_until="networkidle")
        page.screenshot(
            path=path,
            full_page=False,           # above-the-fold is stabler
            animations="disabled",     # freeze CSS animations and transitions
            mask=[page.locator(sel) for sel in masks],
        )
        browser.close()

Then compare with a perceptual hash rather than raw pixels. Raw equality flags anti-aliasing, a rotated banner or one-pixel shifts as changes; a perceptual hash summarises the overall structure of the image, so small noise barely moves it while a different layout moves it a lot:

import imagehash
from PIL import Image

def visual_drift(shot, baseline, threshold=10):
    distance = imagehash.phash(Image.open(shot)) - imagehash.phash(Image.open(baseline))
    return distance > threshold, distance

Subtracting two ImageHash objects gives the Hamming distance between them: 0 for identical structure, higher for more difference. The threshold is not universal. Calibrate it: screenshot the same unchanged page several times over a few days, note the largest distance you see, and set the threshold comfortably above that. Store baselines per route and per viewport, commit them alongside the scraper, and update them deliberately after reviewing a change, never automatically because a check failed.

Keep the Visual Baseline Honest#

Visual checks produce false positives when the page is not in a stable state at capture time. The usual causes and fixes:

• Dynamic regions (ads, carousels, "5 minutes ago" timestamps, personalised recommendations): mask them with mask=.
• Late-loading content: wait for a specific locator that marks the main content as ready, not just networkidle, which some pages with long polling never reach.
• Fonts and images still loading: web fonts swapping in shift text; wait for document.fonts.ready via page.evaluate if needed.
• Consent banners: capture after dismissing, or mask the banner area.

Treat a visual diff as a signal to inspect, not a verdict. It says the page changed; your parser's assertions say whether the data broke.

Choosing the Layers#

Most scrapers need the first two on every run, a fingerprint on a sample, and visual diffs as a weekly audit of a handful of representative URLs.

When Drift Is Detected#

Save the offending HTML and screenshot with the alert; a failure you cannot reproduce is hard to fix once the site changes again. Keep a folder of saved pages per template as regression fixtures and run the parser against them in tests, so a fix for the new layout does not break the old one still served to some users. For the gap between detection and fix, route affected pages to a fallback such as LLM-assisted extraction and flag those records, or pause the source; both beat filling the database with blanks. The data cleaning and validation lesson covers the record-level rules that make a final gate before storage.