Retries, Errors & Timeouts

Building Resilient Scrapers#

Over a long crawl, everything that can fail will: DNS lookups time out, connections reset mid-body, gateways return 502, rate limiters answer 429, and a server that returned perfect HTML yesterday sends a block page with status 200 today. A scraper that assumes success will crash at hour three, or worse, finish "successfully" with a dataset full of holes. Resilience comes from classifying failures correctly and giving each class the right response.

A Taxonomy of Failures#

The worst retry policies treat all of these as "an exception happened, try again". Retrying a 403 hammers a server that already decided to refuse you and can turn a temporary block into a permanent one. Retrying a parse error fetches the same page again and fails the same way. Catching every exception around your own code hides real bugs.

Timeouts: Always, and Split#

requests has no default timeout: a server that accepts the connection but never sends bytes will hang your process forever. httpx defaults to five seconds, which is too short for some slow pages and still not a total deadline.

import requests

try:
    resp = requests.get("https://example.com/report", timeout=(3.05, 30))
    resp.raise_for_status()
except requests.exceptions.ConnectTimeout:
    print("could not connect within 3s: host down or network problem")
except requests.exceptions.ReadTimeout:
    print("connected, but the server stalled for more than 30s between bytes")
except requests.exceptions.HTTPError as exc:
    print("HTTP error:", exc.response.status_code)
except requests.exceptions.RequestException as exc:
    print("other transport error:", type(exc).__name__, exc)

The tuple is (connect, read). The connect timeout should be short, because a healthy server accepts connections quickly. The read timeout is the maximum gap between bytes, not the total download time, so a server that trickles one byte every 20 seconds never trips a 30-second read timeout. When you need a hard ceiling per request, run the fetch under an overall deadline such as asyncio.wait_for in async code, or stream the body and check elapsed time as you read.

Order the except clauses from specific to general: ConnectTimeout and ReadTimeout are subclasses of Timeout, and everything is a subclass of RequestException.

Retries With Tenacity, Retry-After Included#

The tenacity library keeps retry policy declarative. The version below retries only transient failures, uses the server's Retry-After when it sends one, and falls back to randomized exponential backoff otherwise.

import logging
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

import httpx
from tenacity import (retry, stop_after_attempt, wait_random_exponential,
                      retry_if_exception_type, before_sleep_log)

log = logging.getLogger("scraper")
RETRYABLE = {429, 500, 502, 503, 504}

class RetryableStatus(Exception):
    def __init__(self, status, retry_after=None):
        super().__init__(f"HTTP {status}")
        self.status, self.retry_after = status, retry_after

def parse_retry_after(value):
    if not value:
        return None
    if value.strip().isdigit():
        return float(value)
    try:   # HTTP-date form, e.g. "Wed, 21 Oct 2026 07:28:00 GMT"
        when = parsedate_to_datetime(value)
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None

backoff = wait_random_exponential(multiplier=1, max=60)

def wait_policy(retry_state):
    exc = retry_state.outcome.exception()
    if isinstance(exc, RetryableStatus) and exc.retry_after is not None:
        return min(exc.retry_after, 300)
    return backoff(retry_state)

@retry(
    stop=stop_after_attempt(5),
    wait=wait_policy,
    retry=retry_if_exception_type((RetryableStatus, httpx.TransportError)),
    before_sleep=before_sleep_log(log, logging.WARNING),
    reraise=True,
)
def fetch(client: httpx.Client, url: str) -> httpx.Response:
    resp = client.get(url)
    if resp.status_code in RETRYABLE:
        raise RetryableStatus(resp.status_code, parse_retry_after(resp.headers.get("Retry-After")))
    resp.raise_for_status()   # other 4xx/5xx raise HTTPStatusError, which is NOT retried
    return resp

What each piece does:

• httpx.TransportError covers timeouts, connection errors, and protocol errors, the network layer of the taxonomy.
• resp.raise_for_status() raises httpx.HTTPStatusError, which is not in the retry list, so 404 and 403 fail immediately.
• Retry-After may be a number of seconds or an HTTP date; both forms appear in the wild. The cap of 300 seconds prevents a misconfigured header from stalling a worker for a day.
• wait_random_exponential adds jitter. Without it, fifty workers that failed together retry together, and the synchronized wave triggers the next failure.
• before_sleep_log writes a warning for every retry, so a surge of retries is visible in logs.
• reraise=True raises the original exception after the last attempt instead of tenacity's RetryError wrapper, which keeps your error handling readable.

For simple cases, transport-level retries through urllib3.Retry mounted on a requests session do much of this with no extra code; the requests and sessions lesson shows that setup. Application-level retries like the above are better when you need custom logic or logging.

Only Retry Idempotent Requests#

A GET can be retried safely. A POST that submits a form, places an order, or sends a message may have succeeded on the server even though your client saw a timeout. Retrying it can do the action twice. For scrapers that submit forms, retry POST only if the endpoint is a pure query (search, GraphQL read) or you can check whether the first attempt took effect.

Soft Failures: When 200 Is Not Success#

Status codes only tell you what the server claims. Validate the content before you trust it:

BLOCK_MARKERS = ("unusual traffic", "verify you are a human", "access denied", "captcha")

def check_page(resp, min_items=1, items=None):
    text = resp.text.lower()
    if any(m in text for m in BLOCK_MARKERS):
        raise RuntimeError(f"block page at {resp.url}")
    if "/login" in str(resp.url):
        raise RuntimeError(f"redirected to login from {resp.request.url}")
    if items is not None and len(items) < min_items:
        raise RuntimeError(f"only {len(items)} items at {resp.url}; layout change or empty result?")

A block page should pause the crawl, not trigger five quick retries. Treat it as a signal for the whole host and back off for minutes, as covered in rate limiting and politeness.

Circuit Breakers and Dead Letters#

Two patterns keep one bad host or one bad URL from poisoning a run.

A circuit breaker counts consecutive failures per host and stops sending requests to that host for a cooldown once a threshold is reached:

import time
from collections import defaultdict

class HostBreaker:
    def __init__(self, threshold=5, cooldown=300):
        self.fails = defaultdict(int)
        self.open_until = {}
        self.threshold, self.cooldown = threshold, cooldown

    def allow(self, host):
        return time.monotonic() >= self.open_until.get(host, 0)

    def record(self, host, ok):
        if ok:
            self.fails[host] = 0
            return
        self.fails[host] += 1
        if self.fails[host] >= self.threshold:
            self.open_until[host] = time.monotonic() + self.cooldown
            self.fails[host] = 0

A dead-letter file holds URLs that exhausted their retries, with the reason:

import json, time

def dead_letter(url, reason, path="failed.jsonl"):
    with open(path, "a", encoding="utf-8") as f:
        print(json.dumps({"url": url, "reason": reason, "at": time.time()}), file=f)

At the end of a run you re-queue the dead letters once, after the incident has passed, instead of losing them or blocking the crawl on them.

Parse Errors: Keep the Evidence#

When extraction fails, save the raw response next to the error. A parse failure you cannot reproduce is a parse failure you cannot fix, and the page may have changed by the time you look. Store the HTML keyed by URL hash and log the path. The caching and resume lesson makes this automatic.

Watch the Rates, Not the Individual Errors#

Individual errors are normal. Patterns are not. Log every attempt with URL, status, exception type, attempt number, and latency, then watch ratios: a 403 rate jumping from 0.1% to 20%, or retries per success climbing steadily, means something changed. Alerting on those trends catches blocks and layout changes hours before a human would notice missing data.

Practice#

Run your fetcher against a local test server (Flask or http.server) with endpoints that return 429 with Retry-After: 2, 503 twice then 200, 404, a 200 block page, and one that sleeps 40 seconds. Confirm from the logs that each case gets the intended behavior and that the total run time matches what you expect from the waits.