Scheduling and Monitoring

The Crawl Is Not a One-Liner#

A scraper that has to be started by hand eventually stops being started. The difference between a script and a system is that the system runs without you, refuses to trip over itself, notices when its output is wrong, and tells someone. Scrapers fail in a particular way: they rarely crash loudly. More often the site changes a class name, the selector returns nothing, and the job "succeeds" every night with zero rows. Monitoring for scrapers is therefore mostly about checking the data, not the process.

Pick a Scheduler That Fits the Job#

Start with the simplest one that meets your needs. Moving to an orchestrator is covered in data pipelines and orchestration, and container schedulers in serverless and container scraping. The patterns below apply to all of them.

Cron, and Its Traps#

# m  h  dom mon dow  command
30 6 * * * cd /srv/scraper && ./venv/bin/python run.py products >> logs/products.log 2>&1

Most "it works when I run it, but not from cron" problems come from four traps:

• Environment. Cron runs with a minimal PATH and none of your shell profile. Use absolute paths to the interpreter and files, and load secrets from a file the script reads, not from variables exported in .bashrc.
• Working directory. Cron starts in the user's home directory. The cd in the entry above is not optional if your code opens relative paths.
• Percent signs. In a crontab line, % is treated as a newline unless escaped, which silently breaks commands like date +%F. Put anything complex in a shell script and call that.
• Time zones. Cron uses the system time zone, so jobs scheduled during a daylight saving transition can run twice or not at all. Run servers in UTC, or avoid the 01:00 to 03:00 window.

Always redirect output (>> file 2>&1). Without it, errors go to local mail that nobody reads.

systemd Timers: Cron With Memory#

A timer is two small units. The service describes what to run, the timer when.

# /etc/systemd/system/scraper-products.service
[Unit]
Description=Products scraper

[Service]
Type=oneshot
User=scraper
WorkingDirectory=/srv/scraper
ExecStart=/srv/scraper/venv/bin/python run.py products
TimeoutStartSec=3h

# /etc/systemd/system/scraper-products.timer
[Unit]
Description=Run products scraper daily

[Timer]
OnCalendar=*-*-* 06:30:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

sudo systemctl enable --now scraper-products.timer
systemctl list-timers scraper-products.timer     # next and last run
journalctl -u scraper-products.service -n 100     # output of recent runs

Persistent=true runs a missed job after the machine comes back up. RandomizedDelaySec spreads start times, so ten jobs scheduled at 06:30 do not all hit the network (or the same target) in the same second. TimeoutStartSec kills a hung run instead of letting it live forever. And because starting an already-running oneshot service does nothing, the timer cannot launch a second copy while the first is still going.

Never Let Runs Overlap#

With cron, nothing prevents overlap. A crawl that takes longer than its interval collides with the next run, doubling load on a site you promised to treat gently and letting two writers race over the same tables. On Linux, flock wraps the command in a lock:

30 * * * * flock -n /tmp/scraper-products.lock /srv/scraper/run-products.sh

-n means "give up immediately if locked" rather than queueing behind the running job. For a portable version inside Python, the filelock package does the same:

from filelock import FileLock, Timeout

lock = FileLock("/tmp/scraper-products.lock")
try:
    with lock.acquire(timeout=0):
        run_crawl()
except Timeout:
    print("previous run still active; skipping")

A skipped run costs nothing if your crawl is resumable and incremental, as in caching and resume and incremental scraping.

Record Every Run#

Give the crawler a memory of its own behaviour. Wrap the entry point so every run writes a row, whether it succeeds, fails, or raises:

import logging
import sqlite3
import sys
import uuid
from datetime import datetime, timezone

log = logging.getLogger("scraper")

def utcnow():
    return datetime.now(timezone.utc).isoformat()

def run_job(source: str, crawl_fn) -> int:
    db = sqlite3.connect("/srv/scraper/ops.db")
    db.execute('''CREATE TABLE IF NOT EXISTS crawl_runs (
        run_id TEXT PRIMARY KEY, source TEXT, started_at TEXT, finished_at TEXT,
        status TEXT, items INTEGER, errors INTEGER, http_429 INTEGER)''')
    run_id = uuid.uuid4().hex[:12]
    db.execute("INSERT INTO crawl_runs (run_id, source, started_at, status) VALUES (?, ?, ?, 'running')",
               (run_id, source, utcnow()))
    db.commit()

    stats = {"items": 0, "errors": 0, "http_429": 0}
    status = "failed"
    try:
        crawl_fn(stats, run_id)        # the crawler increments the counters
        status = "ok"
    except Exception:
        log.exception("run_id=%s source=%s crashed", run_id, source)
    finally:
        db.execute('''UPDATE crawl_runs SET finished_at = ?, status = ?, items = ?,
                      errors = ?, http_429 = ? WHERE run_id = ?''',
                   (utcnow(), status, stats["items"], stats["errors"], stats["http_429"], run_id))
        db.commit()
    return 0 if status == "ok" else 1

if __name__ == "__main__":
    sys.exit(run_job("products", crawl_products))

The non-zero exit code matters: systemd, Kubernetes and orchestrators all use it to mark a run as failed. A row stuck at running with no finished_at means the process was killed (out of memory, timeout, reboot), which is a failure mode that exception handling alone never sees.

Check the Data, Not Just the Process#

A separate checker, run every 15 minutes or after each job, turns the runs table and the data tables into alerts. These four checks catch almost every real scraper failure:

import os
import statistics
from datetime import datetime, timedelta, timezone
import httpx

EXPECTED_HOURS = {"products": 26, "reviews": 8}     # cadence plus slack

def check(db) -> list[str]:
    problems = []
    now = datetime.now(timezone.utc)
    for source, max_age in EXPECTED_HOURS.items():
        (last_ok,) = db.execute(
            "SELECT MAX(finished_at) FROM crawl_runs WHERE source = ? AND status = 'ok'",
            (source,)).fetchone()
        if last_ok is None or now - datetime.fromisoformat(last_ok) > timedelta(hours=max_age):
            problems.append(f"{source}: stale, last good run {last_ok}")

        rows = db.execute(
            "SELECT items, errors, http_429 FROM crawl_runs WHERE source = ? AND status = 'ok' "
            "ORDER BY finished_at DESC LIMIT 8", (source,)).fetchall()
        if len(rows) >= 4:
            latest_items, latest_errors, latest_429 = rows[0]
            baseline = statistics.median(r[0] for r in rows[1:])
            if latest_items < 0.5 * baseline:
                problems.append(f"{source}: {latest_items} items vs median {baseline:.0f}")
            if latest_items and latest_errors / latest_items > 0.05:
                problems.append(f"{source}: error rate {latest_errors / latest_items:.1%}")
            if latest_429 > 0:
                problems.append(f"{source}: {latest_429} rate-limit responses; slow down")
    return problems

def alert(problems: list[str]) -> None:
    if problems:
        httpx.post(os.environ["ALERT_WEBHOOK_URL"], json={"text": chr(10).join(problems)}, timeout=10)

• Staleness catches jobs that stopped running or keep failing.
• Volume collapse against a rolling median catches the silent selector break. The median, not the mean, keeps one unusual day from skewing the baseline.
• Error rate catches partial breakage: the listing pages work but half the detail pages fail.
• 429 responses are a signal to reduce your rate, not something to retry harder. See rate limiting and politeness.

Add one data-quality check per important field: the share of rows where price is null. A jump from 0% to 40% means a template change on part of the site, which layout drift detection covers in depth.

Watch the Watcher#

All of the above runs on the same machine as the scraper. If that machine is down, or cron is misconfigured, nothing runs, including the checker, and no alert fires. The fix is a dead man's switch: at the end of each successful run, send a request to an external heartbeat monitor that alerts you when the expected ping does not arrive on time.

if status == "ok" and os.environ.get("HEARTBEAT_URL"):
    httpx.get(os.environ["HEARTBEAT_URL"], timeout=10)

Many uptime and cron-monitoring services offer this, and it can also be self-hosted. It is the only check that catches "the job never started".

Logs You Can Search#

When an alert fires, the log should answer "what happened" without re-running anything. Write one line per event with consistent key=value fields:

2026-09-30T06:31:02Z level=INFO run_id=4f1c9a2e7b10 source=products event=fetch url=/p/1841 status=200 ms=412 bytes=48213
2026-09-30T06:31:03Z level=WARN run_id=4f1c9a2e7b10 source=products event=parse url=/p/1842 missing=price
2026-09-30T06:31:09Z level=ERROR run_id=4f1c9a2e7b10 source=products event=fetch url=/p/1850 status=503 retries=3

Including run_id on every line lets you pull exactly one run's history with grep run_id=4f1c9a2e7b10. Rotate files with logrotate, or log to stdout and let journald or your container platform keep them.

Alert Hygiene#

An alert channel that fires every day gets muted, and then the real failure is missed. Alert on the second consecutive failure for flaky sources, include the source, the number and the threshold in the message, and send a single "recovered" message when the check passes again. Every alert should lead to an action; if it does not, delete it or turn it into a dashboard number.

Practice#

Put an existing scraper on a systemd timer (or cron with flock), wrap it with run_job, and write the checker. Then break the main selector on purpose and confirm the volume-collapse alert fires on the next run, while the process itself still exits successfully.