Incremental Scraping and Change Detection

Stop Re-Scraping Everything#

The naive crawler re-downloads the entire site on every run. That is fine at two hundred pages and wasteful at two million: most of the day goes to fetching pages that did not change, the target pays for bandwidth it did not need to serve, and your storage fills with duplicate snapshots. An incremental crawler flips the question. Instead of "fetch everything", it asks "what is the cheapest check that tells me whether this page changed?" and only does the expensive work when the answer is yes.

Four signals cooperate, from cheapest to most precise: the site's own change claims (sitemaps and feeds), HTTP conditional requests, content hashes, and field-level diffs of the parsed record. Each layer catches what the one above misses.

Keep Crawl State Somewhere Durable#

Every signal needs memory of the previous run, and a Python dict dies with the process. A small SQLite table is enough for millions of URLs and survives restarts:

import sqlite3

db = sqlite3.connect("crawl_state.db")
db.execute('''
CREATE TABLE IF NOT EXISTS pages (
    url            TEXT PRIMARY KEY,
    etag           TEXT,
    last_modified  TEXT,
    content_hash   TEXT,
    last_checked   TEXT,
    last_changed   TEXT,
    status         TEXT DEFAULT 'live',
    misses         INTEGER DEFAULT 0     -- consecutive 404/410 responses
)''')
db.commit()

last_checked and last_changed are separate on purpose. Their gap tells you how often a page really changes, which later drives how often you revisit it.

Signal One: Sitemaps and Feeds#

The sitemaps and feeds lesson showed that many sites publish a lastmod date per URL. It costs one request to fetch the sitemap, and it lets you skip every URL whose lastmod is older than your last successful run. Treat it as a hint, not the truth: some sites set lastmod to the current time on every page, some never update it, and some omit it. Use it to decide the order of work and to skip obvious non-changes, and let the next layer act as the safety net.

Signal Two: HTTP Conditional Requests#

HTTP has change detection built in. A response may carry an ETag (an opaque version string) and/or a Last-Modified date. On the next request you send them back in If-None-Match and If-Modified-Since. If nothing changed, a server that supports it answers 304 Not Modified with an empty body, so you pay for headers only.

from datetime import datetime, timezone
import httpx

MAX_MISSES = 3      # one 404 plus two re-checks before a page counts as gone

def now():
    return datetime.now(timezone.utc).isoformat()

def fetch_if_changed(client: httpx.Client, url: str):
    row = db.execute(
        "SELECT etag, last_modified FROM pages WHERE url = ?", (url,)
    ).fetchone()
    headers = {}
    if row:
        etag, last_mod = row
        if etag:
            headers["If-None-Match"] = etag        # send back exactly as received
        if last_mod:
            headers["If-Modified-Since"] = last_mod

    resp = client.get(url, headers=headers)

    if resp.status_code == 304:
        db.execute("UPDATE pages SET last_checked = ?, misses = 0 WHERE url = ?",
                   (now(), url))
        db.commit()
        return None

    if resp.status_code in (404, 410):
        # count consecutive misses; only mark gone once they reach MAX_MISSES
        # (SET expressions see the old row, so misses + 1 is the new count)
        db.execute(
            '''UPDATE pages SET misses = misses + 1, last_checked = ?,
                 status = CASE WHEN misses + 1 >= ? THEN 'gone' ELSE status END
               WHERE url = ?''',
            (now(), MAX_MISSES, url),
        )
        db.commit()
        return None

    resp.raise_for_status()
    db.execute(
        '''INSERT INTO pages (url, etag, last_modified, last_checked)
           VALUES (?, ?, ?, ?)
           ON CONFLICT(url) DO UPDATE SET
             etag = excluded.etag,
             last_modified = excluded.last_modified,
             last_checked = excluded.last_checked,
             misses = 0, status = 'live' ''',
        (url, resp.headers.get("etag"), resp.headers.get("last-modified"), now()),
    )
    db.commit()
    return resp

Details that matter in practice:

• Send the ETag back byte-for-byte, including quotes and a W/ prefix if present. W/"abc" is a weak validator (semantically equivalent content), and it is still valid for If-None-Match.
• When a server supports both headers, If-None-Match takes precedence per the HTTP spec. ETag is also more reliable, because some servers regenerate Last-Modified on every request.
• A 404 or 410 never deletes the row. The code counts consecutive misses and only marks the page gone after MAX_MISSES of them, because transient errors and half-finished deploys happen; any successful response resets the count. Keep the gone row as a tombstone so downstream consumers learn the item disappeared.

Verify that a site honours conditional requests before relying on them: fetch a URL twice with curl -sI, then repeat with the header set.

curl -sI https://example.com/products/42 | grep -i -E "etag|last-modified"
curl -sI -H 'If-None-Match: "5f3a-1a2b"' https://example.com/products/42 | head -1

If the second call returns HTTP/2 304, you are set. If it returns 200 every time, the server ignores validators and you fall through to signal three.

Signal Three: Content Hashing#

Without server help, compute the fingerprint yourself. The trap is hashing the raw HTML: pages embed CSRF tokens, request ids, timestamps, rotating ads and "people also viewed" widgets that differ on every load. Hash the raw bytes and every page "changes" every run. Hash only the part you care about, after normalising it:

import hashlib
from bs4 import BeautifulSoup

def content_fingerprint(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")
    main = soup.select_one("main, #content, article") or soup.body
    for tag in main.select("script, style, noscript, .ad, .recommendations"):
        tag.decompose()
    text = " ".join(main.get_text(" ").split())   # collapse all whitespace
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def page_changed(url: str, html: str) -> bool:
    new_hash = content_fingerprint(html)
    old = db.execute("SELECT content_hash FROM pages WHERE url = ?", (url,)).fetchone()
    if old and old[0] == new_hash:
        return False
    db.execute("UPDATE pages SET content_hash = ?, last_changed = ? WHERE url = ?",
               (new_hash, now(), url))
    db.commit()
    return True

How to tell your hash is too noisy: run the crawler twice, a minute apart, on the same pages. The second run should report close to zero changes. If it reports 80%, diff two saved copies of one page and remove whatever differs from the hashed region.

Signal Four: Field-Level Diff#

A whole-page hash says something changed. Often you need to know what: did the price move, or did someone fix a typo in the description? Compare the parsed, cleaned record (see data cleaning and validation) against the stored one, and let the database do the comparison. SQLite's upsert accepts a WHERE clause on the update branch, so unchanged rows are left untouched:

INSERT INTO items (sku, title, price, stock, updated_at)
VALUES (:sku, :title, :price, :stock, :updated_at)
ON CONFLICT(sku) DO UPDATE SET
    title = excluded.title,
    price = excluded.price,
    stock = excluded.stock,
    updated_at = excluded.updated_at
WHERE items.title IS NOT excluded.title
   OR items.price IS NOT excluded.price
   OR items.stock IS NOT excluded.stock;

IS NOT is used instead of != because of NULL. In SQL, any comparison with NULL using != yields NULL, which the WHERE treats as false, so != would silently miss a field changing from NULL to a value or from a value to NULL (a price appearing, or disappearing). IS NOT treats NULL as an ordinary value: NULL vs a value counts as different, and two NULLs count as equal, so an always-missing field still does not register as a change on every run. After executing, cursor.rowcount is 1 for a new or changed item and 0 for an unchanged one, which gives you change counts per run for free.

When the history itself is valuable (price tracking, stock monitoring), also write a row to an append-only item_changes table with the old value, new value and timestamp. Overwriting the current row loses exactly the information your users will ask for later.

Revisit Pages at the Rate They Change#

Checking every URL daily wastes requests on pages that change yearly and misses changes on pages that update hourly. An adaptive schedule fixes both: shorten the interval when a page changed, lengthen it when it did not.

from datetime import timedelta

MIN_INTERVAL = timedelta(hours=1)
MAX_INTERVAL = timedelta(days=30)

def next_interval(current: timedelta, changed: bool) -> timedelta:
    if changed:
        return max(MIN_INTERVAL, current / 2)
    return min(MAX_INTERVAL, current * 2)

Store next_check_at per URL and have the scheduler pull only rows that are due. This plugs directly into the priority queue from crawl strategy and the frontier. New URLs discovered on listing pages go in with a short interval so they are fetched quickly.

How Incremental Crawls Go Wrong#

A useful safety valve: once a week (or every N runs), do a full fetch of a random sample and compare against what the incremental logic claimed. If the sample finds changes the cheap signals missed, you have a bug in a layer, not a quiet site.

Listen Instead of Poll#

For fast-moving sources, polling is always a compromise between freshness and load. When the site offers a push channel, such as an RSS or Atom feed, a webhook, or a WebSocket stream (see WebSockets and real-time data), consuming it is incremental by design: you store what arrives and poll nothing. Keep a slow full-crawl fallback anyway, because push channels drop messages and you need a way to reconcile.

Practice#

Take a crawler you already have and add the pages table above. Run it twice in a row and record how many pages each layer skipped: sitemap filter, 304 responses, unchanged hashes, and unchanged records. Then tighten the hashed region until the second run reports near-zero changes.