Parallel Scraping with asyncio

Throughput without Chaos#

If one request takes 0.3 seconds, a sequential crawl of 100,000 pages takes over eight hours no matter how fast your parser is. Almost all of that time is spent waiting: for DNS, for the TLS handshake, for the server to think, for bytes to arrive. Concurrency lets you wait on many requests at once. The catch is that unbounded concurrency is also the fastest way to overload a site and get blocked, so the real skill is combining concurrency with the rate controls from rate limiting and politeness.

The Arithmetic#

Throughput is roughly concurrency divided by latency. Ten requests in flight with 0.3 s responses gives about 33 requests per second; one in flight gives about three. That relation tells you two things. You need surprisingly little concurrency to go fast, and concurrency is not a rate limit: if the server gets faster, the same ten workers send more requests per second. Decide the rate you want per host first, then use just enough concurrency to reach it.

Threads, asyncio, or Processes?#

Scraping is mostly I/O-bound, which is why asyncio is the usual choice: one thread can hold hundreds of in-flight requests far more cheaply than hundreds of threads. Parsing, however, is CPU work, and a slow parser can become the bottleneck once the network is no longer the problem.

A Worker Pool on a Queue#

A common first attempt is asyncio.gather(*(fetch(u) for u in urls)) with a semaphore. It works for a few thousand URLs, but it creates a task object for every URL up front, which uses a lot of memory for a million-URL crawl and gives you no natural place to add new URLs discovered during the crawl. A fixed pool of workers pulling from a queue scales better and is easier to reason about.

The example below uses httpx.MockTransport to simulate a server with 200 ms responses and occasional 503s, so you can run it without touching any real site:

import asyncio
import random
import time
import httpx

async def fake_server(request: httpx.Request) -> httpx.Response:
    await asyncio.sleep(0.2)                       # simulated network + server time
    if random.random() < 0.05:
        return httpx.Response(503)
    return httpx.Response(200, json={"path": request.url.path})

async def worker(name, client, queue, results):
    while True:
        url = await queue.get()
        try:
            resp = await client.get(url)
            results.append((url, resp.status_code))
        except httpx.HTTPError as exc:
            results.append((url, type(exc).__name__))
        finally:
            queue.task_done()

async def crawl(urls, concurrency=10):
    queue: asyncio.Queue[str] = asyncio.Queue()
    for u in urls:
        queue.put_nowait(u)
    results = []
    limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency)
    transport = httpx.MockTransport(fake_server)   # remove for real requests
    async with httpx.AsyncClient(transport=transport, limits=limits, timeout=15.0) as client:
        workers = [asyncio.create_task(worker(i, client, queue, results)) for i in range(concurrency)]
        await queue.join()                         # wait until every URL is processed
        for w in workers:
            w.cancel()
        await asyncio.gather(*workers, return_exceptions=True)
    return results

urls = [f"https://shop.example.com/item/{i}" for i in range(100)]
for n in (1, 10, 25):
    start = time.perf_counter()
    res = asyncio.run(crawl(urls, concurrency=n))
    ok = sum(1 for _, s in res if s == 200)
    print(f"concurrency={n:>2}  {time.perf_counter() - start:5.2f}s  ok={ok}/{len(res)}")

Typical output (the ok counts vary because failures are random):

concurrency= 1  20.44s  ok=97/100
concurrency=10   2.10s  ok=94/100
concurrency=25   0.85s  ok=99/100

How it works:

• Each worker loops forever: take a URL, fetch it, record the result, and call task_done() in finally so the queue's count stays correct even when a request fails.
• queue.join() returns once every item has been marked done. The workers are then idle, waiting on queue.get(), so they are cancelled, and gather(..., return_exceptions=True) waits for the cancellations to complete.
• Exceptions are caught inside the worker. If one escaped, that worker would die silently, the queue would never drain, and join() would hang forever. That is the most common deadlock in queue-based crawlers.
• New URLs discovered while parsing (next pages, detail links) can simply be put on the same queue, which makes this shape a natural fit for crawling, as the crawl strategy and frontier lesson develops.

For real targets, remove the mock transport and add await bucket.acquire() for the URL's host right before client.get, using the per-host token bucket from the rate limiting lesson. The queue bounds how much work is in flight; the bucket bounds how fast it reaches each server.

Connection Limits#

httpx.Limits controls the connection pool. By default an httpx client allows up to 100 connections and keeps up to 20 alive between requests. Keep max_connections in line with your worker count. If 200 workers share a pool of 10 connections, most of them sit waiting for a connection and your apparent concurrency is fake, while the waiting counts against your timeouts (httpx.PoolTimeout). If you leave the pool at 100 while intending to be gentle, a burst can open far more sockets to one server than you meant. Bound both, and reuse one client for the whole crawl so connections are actually kept alive.

Per-Host Fairness#

In a multi-site crawl, a single queue lets one site with thousands of URLs occupy every worker. Add a per-host limit so no host gets more than one or two concurrent requests:

from collections import defaultdict
from urllib.parse import urlsplit

host_slots = defaultdict(lambda: asyncio.Semaphore(2))

async def fetch_fair(client, url):
    async with host_slots[urlsplit(url).hostname]:
        await buckets[urlsplit(url).hostname].acquire()
        return await client.get(url)

The global worker count sets your total throughput; the per-host semaphore and bucket set what each server experiences.

Things That Silently Block the Event Loop#

asyncio only helps if nothing blocks the loop. These do, and they make an async crawler run one request at a time without any error:

• time.sleep() instead of await asyncio.sleep().
• Calling requests.get() or any other synchronous HTTP library inside a coroutine.
• Heavy parsing of large documents directly in the coroutine.
• Synchronous database writes for every record.

For unavoidable blocking work, move it off the loop: await asyncio.to_thread(parse, html) for moderate work, or a ProcessPoolExecutor via loop.run_in_executor when parsing is genuinely CPU-bound. Running with PYTHONASYNCIODEBUG=1 (or asyncio.run(main(), debug=True)) logs callbacks that take longer than 100 ms, which points straight at the blocking call.

The Thread Alternative#

For an existing synchronous scraper, threads are the smallest change. Give each thread its own session, since requests does not guarantee that a Session is safe to share across threads:

import threading
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

local = threading.local()

def session():
    if not hasattr(local, "s"):
        local.s = requests.Session()
    return local.s

def fetch(url):
    return url, session().get(url, timeout=(3, 15)).status_code

with ThreadPoolExecutor(max_workers=8) as pool:
    futures = [pool.submit(fetch, u) for u in urls]
    for f in as_completed(futures):
        url, status = f.result()

f.result() re-raises any exception from the thread, so handle errors there or inside fetch. The rate limiter still applies; a thread-safe version uses threading.Lock instead of asyncio.Lock.

When Parallelism Backfires#

Concurrency helps when your side is the bottleneck. It hurts when the target is: a per-IP rate limit, a login endpoint, a slow search backend, or a small server. Symptoms are rising latency, more 429s and 5xxs, and throughput that stops improving as you add workers. Measure throughput and error rate at 1, 2, 4, and 8 workers; the point where throughput flattens and errors appear is your ceiling, and the right setting is below it. One careful worker with a token bucket is often enough, and it is the least likely to get you blocked.

Memory Discipline#

Holding every response body in a list is fine for a thousand pages and a crash for a million. Parse inside the worker, keep only the extracted record, and hand records to a single writer task that batches them into storage, as in storing data. Use a bounded queue (asyncio.Queue(maxsize=1000)) between fetchers and the writer, so that if storage slows down, fetching slows down with it instead of buffering unbounded data in memory.

Practice#

Run the example above, then add a per-host token bucket at 5 requests per second and rerun with 1, 10, and 25 workers. Explain why the timings converge. Finally, replace await asyncio.sleep(0.2) in the fake server with time.sleep(0.2) and observe what happens to the 25-worker run.