Throttle Like the Site Owns the Server#

The web is other people's infrastructure. A scraper that fires requests as fast as the network allows may work for an hour, then meet a WAF rule, a struggling database, or a blocked subnet. Crawlers that last pace themselves, listen to what the server tells them, and slow down the moment it complains. Politeness is not separate from results: it is what keeps the pipeline alive long enough to produce them.

What Server-Side Limits Look Like#

Sites protect themselves at several layers, and you will see different symptoms from each:

  • Explicit rate limits return 429 Too Many Requests, often with a Retry-After header, and sometimes informational headers such as X-RateLimit-Remaining or RateLimit-Remaining that tell you how much budget is left.
  • Overload shows up as 503, 502, 504, or steadily rising response times. The server is not angry at you; it is struggling, and you are part of the load.
  • Soft blocks return 200 with a challenge page, empty results, or degraded data.
  • Hard blocks return 403 for every request from your IP or subnet, sometimes for days.

The first two are signals you can respond to. The last two are what happens when you did not.

Choosing a Rate#

For a site you have not measured, start around one request per second per host, one request in flight at a time, and a delay no shorter than any Crawl-delay in robots.txt. Then let evidence move you: if response times stay flat and errors stay near zero, you can increase gradually; if either rises, you are already too fast.

Two numbers relate rate and concurrency, a form of Little's law: requests per second is roughly the number of requests in flight divided by the average response time. With 500 ms responses, two concurrent requests already give about four per second. That is why a concurrency setting alone is not a rate limit, and why a rate limit needs to be enforced explicitly.

Scale the rate to the site. A large e-commerce platform serving millions of visitors will not notice a few requests per second. A small forum on a single shared server can. Run large backfills during the site's off-peak hours and spread them across days.

The Token Bucket#

A token bucket refills at a steady rate and holds a small burst. Each request takes a token and waits if none is available, giving a smooth average rate with a little room for bursts.

import asyncio
import time
from collections import defaultdict

class TokenBucket:
    def __init__(self, rate: float, burst: int = 1):
        self.rate = rate              # tokens added per second
        self.burst = burst            # bucket capacity
        self.tokens = float(burst)
        self.updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        async with self._lock:        # waiters queue up in order
            while True:
                now = time.monotonic()
                self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                await asyncio.sleep((1 - self.tokens) / self.rate)

buckets = defaultdict(lambda: TokenBucket(rate=2.0, burst=3))   # one bucket per host

async def demo():
    start = time.monotonic()
    async def one():
        await buckets["shop.example.com"].acquire()
        return round(time.monotonic() - start, 2)
    print(await asyncio.gather(*(one() for _ in range(8))))

asyncio.run(demo())

Output (timings vary by a few milliseconds):

[0.0, 0.0, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5]

Three requests go out immediately (the burst), then one every half second, exactly two per second. Details that matter:

  • time.monotonic() never jumps when the system clock is adjusted, unlike time.time().
  • The lock makes waiters line up, so the bucket stays correct with many concurrent workers.
  • It sleeps for exactly the time until the next token instead of polling in a busy loop.
  • One bucket per host, shared by all workers. A bucket per worker multiplies your real rate by the number of workers, which is the most common way "two per second" turns into forty.

If you prefer a library, aiolimiter provides the same idea: limiter = AsyncLimiter(2, 1) then async with limiter: around each request.

Honor Retry-After#

When a server returns 429 or 503 with Retry-After, it tells you exactly how long to wait. The value is either a number of seconds or an HTTP date, and both appear in practice:

import time
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(value, default=60.0):
    if not value:
        return default
    if value.strip().isdigit():
        return float(value)
    try:
        return max(0.0, (parsedate_to_datetime(value) - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return default

def polite_get(client, url, attempts=4):
    for attempt in range(attempts):
        resp = client.get(url, timeout=15.0)
        if resp.status_code not in (429, 503):
            return resp
        wait = min(retry_after_seconds(resp.headers.get("Retry-After")), 600)
        time.sleep(wait * (1 + 0.25 * attempt))
    return resp

In a concurrent crawler, a 429 is a signal about the whole host, not one URL. Pause every worker for that host until the window passes, rather than letting the other workers keep spending the budget you just exhausted. The error handling lesson combines this with retries and circuit breakers.

Adaptive Throttling#

A fixed rate is either too slow for a strong server or too fast for a weak one. Adaptive throttling adjusts from feedback, using the same idea as TCP congestion control: increase slowly while things are healthy, cut sharply when they are not (additive increase, multiplicative decrease).

class AdaptiveRate:
    def __init__(self, bucket: TokenBucket, floor=0.2, ceiling=5.0, slow_s=2.0):
        self.bucket, self.floor, self.ceiling, self.slow_s = bucket, floor, ceiling, slow_s

    def observe(self, status: int, latency_s: float) -> None:
        rate = self.bucket.rate
        if status == 429 or status >= 500:
            rate = rate / 2                  # back off hard
        elif latency_s > self.slow_s:
            rate = rate * 0.8                # server is straining
        else:
            rate = rate + 0.05               # creep up slowly
        self.bucket.rate = max(self.floor, min(self.ceiling, rate))

Call observe after every response for that host. Latency is the early warning: response times usually climb before errors start, so reacting to latency lets you back off before you cause the errors. Keep a ceiling even when everything looks healthy; "the server can take it" is not the same as "the operator wants it".

Scrapy ships this as AutoThrottle. A conservative configuration:

# settings.py
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 1.0                 # base delay, randomized by default
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 2.0
AUTOTHROTTLE_MAX_DELAY = 60.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
HTTPCACHE_ENABLED = True             # during development

See the Scrapy framework lesson for the rest of its politeness settings.

The Cheapest Request Is the One You Do Not Send#

Throttling limits how fast you ask; you can also ask less:

  • Conditional requests. Send If-None-Match with the stored ETag, or If-Modified-Since with the stored Last-Modified. An unchanged page returns 304 Not Modified with no body, cheap for both sides.
  • Sitemaps with lastmod tell you which pages changed without fetching them.
  • Larger API pages replace many small requests with a few bigger ones, within limits the server accepts.
  • Skip what you do not parse. Do not download images, fonts, or scripts from a plain HTTP scraper, and block them in browser automation.
  • Cache during development so every parser tweak does not refetch the site.

Put the Throttle at the Right Layer#

Place the limiter immediately around the network call, not around parsing or database writes. Throttling too high slows down work that never touches the target while leaving actual requests uncoordinated. In a distributed crawl with several machines, a per-process bucket is not enough: either assign each host to exactly one worker, or keep the shared budget in a central store such as Redis. The distributed crawling lesson covers both.

When the Site Says Stop, Stop#

A 200 response containing "unusual traffic" or a challenge page is a block in disguise. Detect those markers, pause the whole crawl for that host for a long window, and alert a human. Resuming at the same rate after the pause usually re-triggers the block immediately, so resume at a fraction of the old rate. A crawl that pauses for an hour has cost you an hour. A crawl that keeps hammering a blocked target has cost you the target.

Practice#

Put a local test server behind a simple rate limiter (for example five requests per second per client, returning 429 with Retry-After). Point your crawler at it with the token bucket at ten per second and AdaptiveRate enabled. Log the rate after every response and confirm that it converges below five, that no worker keeps sending during a Retry-After window, and that the total number of 429 responses stays small.