Caching and Crash-Resume

A Crawl Is a Long-Running Process#

Scrapes fail. The target goes down, the container restarts, the network blips, the disk fills, the IP gets throttled. If every failure means restarting from page one, you have built a job that punishes you on schedule. The fix is to make the crawl idempotent: cache what you have already fetched, dedupe your work queue, and persist enough state that the next run picks up exactly where the last one died.

Three layers do the work, and each protects something different:

Cache the Raw HTML#

The first and cheapest layer is a raw-HTML cache. Parse against a local copy during development, and at runtime never hit the network twice for the same URL. A file-per-URL layout is simple and debuggable: the cache key is a hash of the URL, the value is the body, and the file system is the index.

import hashlib
import json
import os
import pathlib

CACHE = pathlib.Path("html_cache")
FINAL_STATUSES = {200, 404, 410}   # anything else is retried on the next run

def cache_path(url: str) -> pathlib.Path:
    key = hashlib.sha256(url.encode("utf-8")).hexdigest()
    # shard into 256 subfolders so no directory holds millions of files
    folder = CACHE / key[:2]
    folder.mkdir(parents=True, exist_ok=True)
    return folder / key

def write_atomic(path: pathlib.Path, data: bytes):
    # write to a temp file, then rename: a crash never leaves half a file
    tmp = path.with_name(path.name + ".tmp")
    tmp.write_bytes(data)
    os.replace(tmp, path)

def get_page(client, url: str) -> tuple[int, bytes]:
    body_path = cache_path(url).with_suffix(".body")
    meta_path = cache_path(url).with_suffix(".json")
    if body_path.exists() and meta_path.exists():
        meta = json.loads(meta_path.read_text())
        if meta["status"] in FINAL_STATUSES:
            return meta["status"], body_path.read_bytes()

    resp = client.get(url, timeout=15.0)
    write_atomic(body_path, resp.content)
    write_atomic(meta_path, json.dumps({          # metadata last, and atomic too
        "url": url,
        "status": resp.status_code,
        "content_type": resp.headers.get("content-type"),
        "etag": resp.headers.get("etag"),
        "last_modified": resp.headers.get("last-modified"),
    }).encode("utf-8"))
    return resp.status_code, resp.content

The non-obvious lines matter more than the obvious ones:

• Store bytes, not resp.text. Decoding is a separate decision (see parsing engines and encoding). If you cache an already-mangled string, every re-parse inherits the bug.
• Write the metadata last. The body exists without its .json only if the process died between the two writes, and the exists() check treats that as a miss.
• os.replace is atomic on the same file system. Without the temp-and-rename step, a crash mid-write leaves a truncated file that the cache will happily serve forever. That applies to the .json too: a half-written metadata file would make json.loads fail on every later run, so both writes go through write_atomic.
• Shard the directory. Many file systems slow down badly with hundreds of thousands of entries in one folder; two hex characters give 256 buckets.

Cache every response, including errors, and keep the status code in the metadata. A page that 404ed is data too: without it you will retry the URL forever. That is what FINAL_STATUSES encodes: a cached 200 or 404 is final, while a cached 429 or 503 is treated as a miss and fetched again.

If you want a ready-made layer for requests, the requests-cache package wraps a session with a SQLite-backed cache (requests_cache.CachedSession("crawl_cache", expire_after=86400)). It is excellent for development. The hand-rolled version above wins when you need to inspect, diff, or ship the raw pages to another process.

Revalidate Instead of Refetching#

A cache that never expires is only right for immutable pages. For pages that change, keep the ETag and Last-Modified headers you stored and send them back as a conditional request. An unchanged page comes back as 304 Not Modified with an empty body, which costs the server almost nothing.

def revalidate(client, url, meta):
    headers = {}
    if meta.get("etag"):
        headers["If-None-Match"] = meta["etag"]
    if meta.get("last_modified"):
        headers["If-Modified-Since"] = meta["last_modified"]
    resp = client.get(url, headers=headers, timeout=15.0)
    return resp.status_code == 304, resp

Not every server honours these headers, so verify once: fetch a page, repeat with its validators, and check that you actually get a 304. The HTTP caching lesson covers the header semantics in detail.

Dedupe the Work Queue#

The cache stops network waste. A separate dedupe stops processing waste. Keep a record of every URL you have started, and filter against it before you add new links to the queue. Normalize URLs first or the same page sneaks in under five spellings: strip the fragment, drop tracking parameters, sort the remaining query parameters, and lowercase the host.

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def normalize(url: str) -> str:
    parts = urlsplit(url)
    query = sorted((k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
                   if not k.startswith("utm_"))
    netloc = (parts.hostname or "").lower()
    if parts.port and parts.port not in (80, 443):
        netloc += f":{parts.port}"
    return urlunsplit((parts.scheme.lower(), netloc,
                       parts.path or "/", urlencode(query), ""))

print(normalize("HTTPS://Shop.Example.com/p?b=2&a=1&utm_source=x#reviews"))
# https://shop.example.com/p?a=1&b=2

Be careful not to over-normalize. Lowercasing the path, or dropping a parameter that actually selects content (?page=2, ?id=), merges distinct pages and silently loses data. The URL handling lesson goes deeper on which rules are safe.

Persist the Queue#

In-memory queues vanish on a crash. For any crawl bigger than a few thousand pages, persist the queue in a table with states: pending, in_progress, done, failed. A UNIQUE constraint on the normalized URL makes the dedupe survive restarts for free.

import sqlite3
import time

con = sqlite3.connect("crawl.db", isolation_level=None)
con.execute("PRAGMA journal_mode=WAL")   # readers do not block the writer
con.execute('''
CREATE TABLE IF NOT EXISTS queue (
    url        TEXT PRIMARY KEY,
    state      TEXT NOT NULL DEFAULT 'pending',
    attempts   INTEGER NOT NULL DEFAULT 0,
    claimed_at REAL,
    error      TEXT
)''')

def enqueue(urls):
    con.executemany("INSERT OR IGNORE INTO queue (url) VALUES (?)",
                    [(normalize(u),) for u in urls])

def claim_batch(size=50):
    now = time.time()
    rows = con.execute('''
        UPDATE queue SET state='in_progress', claimed_at=?, attempts=attempts+1
        WHERE url IN (
            SELECT url FROM queue WHERE state='pending' LIMIT ?
        )
        RETURNING url''', (now, size)).fetchall()
    return [url for (url,) in rows]

The single UPDATE ... RETURNING statement (SQLite 3.35 or newer) claims rows atomically. The common alternative, a SELECT followed by a loop of UPDATEs, lets two workers claim the same rows between the two statements.

On restart, rows stuck in in_progress belonged to a process that died. Return them to the queue, and park URLs that keep failing instead of retrying them forever:

def recover(stale_after=600, max_attempts=5):
    cutoff = time.time() - stale_after
    con.execute('''UPDATE queue SET state='pending'
                   WHERE state='in_progress' AND claimed_at < ?''', (cutoff,))
    con.execute('''UPDATE queue SET state='failed'
                   WHERE state='pending' AND attempts >= ?''', (max_attempts,))

The table doubles as a progress report: SELECT state, COUNT(*) FROM queue GROUP BY state tells you at a glance how far the crawl has got and how much is failing.

Journal Extracted Records#

When a crash loses your in-memory result list, it also loses the work that already succeeded. Write each parsed record the moment it is produced, and mark the URL done only after the record is safely stored.

import json

out = open("out.jsonl", "a", encoding="utf-8")

def save_record(url, record):
    out.write(json.dumps(record, ensure_ascii=False) + "\n")
    out.flush()
    os.fsync(out.fileno())
    con.execute("UPDATE queue SET state='done', error=NULL WHERE url=?", (url,))

The order is the whole point. Record first, then state. If the process dies between the two, the URL is re-processed and you get a duplicate line, which is harmless if you dedupe downstream on a stable key (covered in data cleaning and validation). The opposite order loses records permanently. This is "at least once" delivery, and it is the right default for scrapers.

If you would rather avoid duplicates entirely, store records in the same SQLite database and update the queue row in the same transaction. Then the record and the state change commit together or not at all.

The Resumable Main Loop#

With the three layers in place, the crawl itself becomes short and boring:

while True:
    recover(stale_after=600, max_attempts=5)
    batch = claim_batch()
    if not batch:
        break
    for url in batch:
        try:
            status, body = get_page(client, url)
            if status in (404, 410):
                con.execute("UPDATE queue SET state='done', error=? WHERE url=?",
                            (f"HTTP {status}", url))
                continue
            if status != 200:
                # back to pending; claim_batch counts attempts, recover() parks it
                con.execute("UPDATE queue SET state='pending', error=? WHERE url=?",
                            (f"HTTP {status}", url))
                continue
            record, links = parse(body, url)
            enqueue(links)
            save_record(url, record)
        except Exception as exc:
            con.execute("UPDATE queue SET state='pending', error=? WHERE url=?",
                        (repr(exc)[:500], url))

Calling recover() at the top of every batch parks URLs that keep failing, so a permanently broken page cannot loop forever. Kill the process halfway through and run it again: recover() puts the orphaned rows back once they go stale, the cache serves everything already fetched without touching the network, and the journal keeps what was saved. A "crash" is now just a pause.

How It Fails in Practice#

• Poisoned cache entries. A block page or a login redirect arrives with status 200 and gets cached as if it were real content. Validate the body (expected selector present, size above a floor) before writing it to the cache, and keep a way to purge by pattern.
• Parser changes invalidate records, not the cache. When you fix a selector, re-parse from the cache instead of re-crawling. This is the biggest practical payoff of storing raw bytes.
• Unbounded disk use. Compressed pages are much smaller; wrapping the body write in gzip.compress is usually worth it on large crawls. Check free space before the crawl, not after.
• Clock-based expiry across machines. If several workers share a cache, store fetch time as UTC and never compare local wall-clock strings.

Practice#

Run a crawl of a site you control (or a local test server) for 200 pages, kill the process at a random point with kill -9, and restart it. Verify three things: the queue table has no rows left in in_progress, the number of network requests on the second run equals only the pages that were not cached, and out.jsonl has no missing URLs when you compare it to the done rows.