Webhooks & Event-Driven Ingestion

Stop Asking, Start Receiving#

Polling is a scraper's default posture: wake up, ask "anything new?", usually hear "no", repeat. When a source changes constantly, polling wastes bandwidth, burns rate limits and adds delay between an event and your knowledge of it. Webhooks invert the model: the source sends each event to you when it happens. Many platforms you might otherwise scrape (payment providers, code hosts, e-commerce platforms, form builders, CRMs) offer them for data you are entitled to, typically your own account's data. This lesson covers building a receiver that is secure, idempotent and impossible to overwhelm, plus the reconciliation job that catches what webhooks miss.

How a Webhook Works#

You register an HTTPS endpoint with the provider and choose event types. When an event occurs (an order is placed, a repository is starred, a payment settles), the provider sends an HTTP POST to your endpoint with a JSON body:

POST /hooks/shop HTTP/1.1
Host: ingest.example.org
Content-Type: application/json
X-Event-Id: evt_01J9ZK3M8Q
X-Signature-256: sha256=5d2f0c...
X-Timestamp: 1790000000

{"id": "evt_01J9ZK3M8Q", "type": "order.updated", "created": 1790000000,
 "data": {"order_id": "A-1042", "status": "shipped", "updated_at": "2026-09-30T08:12:44Z"}}

You answer with any 2xx status to acknowledge. Header names differ by provider (GitHub, for instance, sends X-Hub-Signature-256 and X-GitHub-Delivery), but the shape is universal. Everything difficult lies in four operational facts: anyone can POST to your URL, deliveries are retried, they arrive out of order, and some never arrive at all.

Verify Every Delivery#

Your endpoint is public, so the provider proves authorship with a signature. The most common scheme is an HMAC: the provider computes HMAC-SHA256 over the raw request body with a shared secret and sends the hex digest in a header. You recompute and compare. Some providers instead sign with a private key and publish a public key you verify against.

Two details decide whether verification actually works:

• Verify the raw bytes. Compute the HMAC over the body exactly as received, before any JSON parsing. Parsing and re-serialising changes whitespace and key order, and the signature no longer matches.
• Compare in constant time. hmac.compare_digest avoids leaking, through response timing, how many leading characters matched.

Providers that include a timestamp usually sign timestamp + "." + body and expect you to reject old timestamps, which stops an attacker from replaying a captured valid request days later. A receiver in FastAPI:

import hashlib
import hmac
import os
import time
from fastapi import FastAPI, Header, HTTPException, Request

app = FastAPI()
SECRET = os.environ["WEBHOOK_SECRET"].encode()
TOLERANCE = 300  # seconds

def valid_signature(raw: bytes, timestamp: str, header: str) -> bool:
    signed = timestamp.encode() + b"." + raw
    expected = hmac.new(SECRET, signed, hashlib.sha256).hexdigest()
    received = header.removeprefix("sha256=")
    return hmac.compare_digest(expected, received)

@app.post("/hooks/shop")
async def receive(request: Request,
                  x_signature_256: str = Header(...),
                  x_timestamp: str = Header(...)):
    raw = await request.body()              # bytes, untouched
    if abs(time.time() - int(x_timestamp)) > TOLERANCE:
        raise HTTPException(400, "stale timestamp")
    if not valid_signature(raw, x_timestamp, x_signature_256):
        raise HTTPException(401, "bad signature")
    enqueue(raw)                            # durable queue, see below
    return {"ok": True}

Follow your provider's documentation for exactly what is signed and how the header is formatted; the structure above is typical, and details such as the prefix or the separator vary. Keep the secret in a secrets manager or environment variable, never in code (see secrets management).

You can exercise the endpoint locally with a signature computed the same way:

SECRET=test-secret
TS=$(date +%s)
BODY='{"id":"evt_1","type":"order.updated","data":{"order_id":"A-1"}}'
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.*= //')
curl -i -X POST http://localhost:8000/hooks/shop \
  -H "Content-Type: application/json" \
  -H "X-Timestamp: $TS" -H "X-Signature-256: sha256=$SIG" \
  --data "$BODY"

Expect 200 with the correct secret, 401 after changing one character of the body, and 400 after setting TS to an hour ago. Those three cases belong in your automated tests. To receive real deliveries on a laptop during development, providers generally require a public HTTPS URL; a tunnelling tool gives you one temporarily.

Acknowledge Fast, Process Later#

Providers wait only a few seconds for your 2xx before counting the delivery as failed and retrying. If you do real work inline (fetching related resources, writing to a slow database), a traffic spike makes you slow, slowness causes retries, and retries add more load. The endpoint should do three things only: verify, persist to a durable queue, return 2xx. Workers consume the queue at their own pace.

Any durable queue works: SQS, Redis Streams, RabbitMQ, Kafka, or a plain database table for modest volumes. What matters is that once you have returned 2xx, the event cannot be lost if a worker crashes.

Idempotency: Deliveries Come At Least Once#

Webhooks are delivered at least once. A timeout, a network blip or a slow response causes a retry of an event you may already have processed. Deduplicate on the provider's event ID with a unique constraint:

import json
import sqlite3

con = sqlite3.connect("events.db")
con.execute('''CREATE TABLE IF NOT EXISTS events (
    event_id TEXT PRIMARY KEY, type TEXT, received_at TEXT, payload TEXT)''')

def record_event(raw: bytes) -> bool:
    event = json.loads(raw)
    cur = con.execute(
        "INSERT OR IGNORE INTO events VALUES (?, ?, datetime('now'), ?)",
        (event["id"], event["type"], raw.decode()))
    con.commit()
    return cur.rowcount == 1   # False means duplicate: skip processing

INSERT OR IGNORE plus rowcount makes the check-and-insert atomic, so two workers handling the same retried event cannot both process it. Keeping the raw payload also gives you an audit log and something to replay after fixing a bug.

Ordering: Newer Must Win#

Events can arrive out of order: order.updated (shipped) may land before the earlier order.updated (paid) after a retry. If each event blindly overwrites the record, the order ends up "paid" forever. Guard the write with the resource's own timestamp or version:

con.execute('''
    INSERT INTO orders (order_id, status, updated_at) VALUES (?, ?, ?)
    ON CONFLICT(order_id) DO UPDATE SET
        status = excluded.status, updated_at = excluded.updated_at
    WHERE excluded.updated_at > orders.updated_at''',
    (d["order_id"], d["status"], d["updated_at"]))

The WHERE clause turns a stale event into a no-op. Some providers deliberately send thin events containing only the resource ID; the worker then fetches the current state from the provider's REST API, which sidesteps ordering entirely at the cost of one API call per event (see REST API scraping).

Reconciliation: Webhooks Are Not a Complete Record#

Deliveries fail permanently: your endpoint was down longer than the provider's retry window, a deploy broke verification for an hour, or the provider had an incident. Webhooks give you speed; they do not guarantee completeness. Production systems pair them with a periodic reconciliation poll: every hour or day, list resources changed since the last reconciliation through the provider's API and upsert them through the same idempotent path. Most events arrive in seconds via webhook; the poll catches the few that never did. The incremental scraping lesson's "changed since" techniques apply directly.

Operating the Endpoint#

• Monitor delivery rate, verification failures and queue depth. A sudden drop to zero deliveries is as suspicious as a spike of 401s.
• Dead-letter events that fail processing repeatedly, so one poisonous payload does not block the queue.
• Build a replay tool that re-enqueues stored events by ID or time range. Many providers also let you redeliver from their dashboard.
• Rotate secrets without downtime by accepting both the old and new secret during the switch.
• Limit the endpoint: accept only POST, cap body size, and rate-limit per source. It is a public API and deserves the same care (see API security).

When Webhooks Beat Scraping#

If a platform documents webhooks for your use case, prefer them. They are faster, lighter, officially supported, and avoid the anti-bot game entirely. Scraping remains the fallback for sources that offer no push channel.