Stop Asking, Start Receiving

Polling is a scraper's default posture: wake up, ask "anything new?", get nothing, repeat. When a source is high-volume and updates constantly, polling wastes bandwidth, burns rate limits, and introduces latency between an event and your knowledge of it. Webhooks invert the model—the source pushes each event to you the moment it happens.

How a Webhook Works

You register an HTTPS endpoint with the provider. When an event occurs (an order is placed, a repository is starred, a payment settles), the provider sends an HTTP POST to your endpoint with a JSON payload describing the event. You acknowledge with a 2xx response and process it asynchronously.

The contract is simple, but the operational details decide whether an ingestion pipeline is reliable.

Verifying Authenticity

Anyone on the internet can POST to your endpoint, so the provider signs each delivery. The two common schemes are:

  • HMAC signatures: the provider computes an HMAC (e.g., sha256) over the raw request body using a shared secret and puts the hex digest in a header like X-Hub-Signature-256. You recompute it and compare in constant time.
  • Public-key signatures: the provider signs with a private key and exposes a public key (or JWKS endpoint) you use to verify.

Critical detail: verify against the raw bytes of the body before any JSON parsing or framework middleware mutates them. Frameworks that parse JSON for you can silently change whitespace and invalidate the signature.

import hmac, hashlib

def verify(raw_body: bytes, signature_header: str, secret: bytes) -> bool:
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    got = signature_header.split("=")[-1]
    return hmac.compare_digest(expected, got)

Idempotency and Replay

Webhooks are delivered at least once, not exactly once. Any delivery can be retried after a timeout, a network blip, or a non-2xx response. Your consumer must be idempotent: processing the same event twice should not create two records. The standard approach is to extract a stable event ID from the payload, store it with a unique constraint, and drop duplicates.

Ordering and Retries

Events are not guaranteed to arrive in order. A provider may deliver event 42 before event 41 after a retry. Design consumers to be order-independent, or carry a monotonic sequence/timestamp and reconcile out-of-order arrivals. Providers typically retry failed deliveries with exponential backoff for hours or days; return a 2xx quickly and do the real work in a queue so a slow downstream does not cause unwanted retries.

Acknowledge Fast, Process Later

The golden rule: your endpoint should validate the signature, enqueue the event, and return 2xx in milliseconds. Never do heavy work inline—if processing takes longer than the provider's timeout, it will retry and you will process the event multiple times. A durable queue (SQS, Redis, Kafka) between the endpoint and the workers decouples delivery spikes from processing capacity.

Operating the Endpoint

Production webhook endpoints need the same care as any API: TLS, authentication, rate limiting, monitoring, and alerting on non-2xx responses. Log every delivery with its event ID, maintain a dead-letter queue for events that repeatedly fail, and build a replay tool so a bug can be fixed and the affected events reprocessed.

When Webhooks Beat Scraping

If a platform offers documented webhooks, prefer them. They are faster, lighter, officially supported, and avoid the entire cat-and-mouse game of anti-bot evasion. Treat webhooks as the premium data source and scraping as the fallback for the platforms that lack them.