httpx, Async Clients & HTTP/2

The Modern Requests Library#

requests is the default HTTP library for scrapers, and for a single script it is still fine. httpx offers a very similar API with three upgrades that matter once a crawl grows: an async client with the same interface as the sync one, optional HTTP/2, and stricter defaults (timeouts on by default, explicit redirects). This lesson covers what changes when you switch, how to run many requests concurrently without overwhelming anyone, and the failure modes that appear only at scale.

Same Shape, Different Defaults#

The basic calls look almost identical to what you learned in requests and sessions:

import httpx

with httpx.Client(
    headers={"User-Agent": "price-research-bot/1.0 (+contact@example.org)"},
    timeout=httpx.Timeout(10.0, connect=5.0),
    follow_redirects=True,
) as client:
    r = client.get("https://example.com/api/items", params={"page": 1})
    r.raise_for_status()
    print(r.status_code, r.http_version, r.headers["content-type"])
    data = r.json()

The differences that bite people during migration:

The redirect default is the most common surprise: a migrated scraper suddenly receives 301 and 302 responses with empty bodies and parses nothing. Set follow_redirects=True on the client, or handle redirects deliberately if you want to record them.

Both libraries send their own identifying User-Agent (python-requests/..., python-httpx/...) unless you set one. Set it once on the client so every request inherits it.

Why the Client Object Matters#

A Client keeps a pool of open connections. The second request to the same host reuses the TCP connection and TLS session from the first, skipping one or two network round trips. It also stores cookies between requests and applies shared headers, base URLs and timeouts. Calling the module-level httpx.get() in a loop creates and tears down a connection each time: slower for you and more connection churn for the server. One client per crawl (or per target host) is the rule.

base_url removes repetition when a scraper talks to one API:

client = httpx.Client(base_url="https://api.example.com/v2", timeout=10.0)
client.get("/items", params={"page": 2})     # https://api.example.com/v2/items?page=2

Async: Many Requests in Flight#

A synchronous client waits idle for each response. Most of a scraper's time is exactly that wait. With httpx.AsyncClient, while one request waits for the network, others can be sent and received on the same thread. The API is the same with await in front.

import asyncio
import time
import httpx

async def fetch(client, sem, url):
    async with sem:                       # cap in-flight requests
        r = await client.get(url)
        r.raise_for_status()
        return r.json()

async def main():
    urls = [f"https://api.example.com/items/{i}" for i in range(1, 201)]
    sem = asyncio.Semaphore(10)
    limits = httpx.Limits(max_connections=10, max_keepalive_connections=10)
    async with httpx.AsyncClient(timeout=15.0, limits=limits, follow_redirects=True) as client:
        start = time.perf_counter()
        results = await asyncio.gather(*(fetch(client, sem, u) for u in urls),
                                       return_exceptions=True)
        elapsed = time.perf_counter() - start

    ok = [r for r in results if not isinstance(r, Exception)]
    failed = [(u, r) for u, r in zip(urls, results) if isinstance(r, Exception)]
    print(f"{len(ok)} ok, {len(failed)} failed in {elapsed:.1f}s")

asyncio.run(main())

With responses that take about 200 ms each, 200 sequential requests take about 40 seconds; with 10 in flight, about 4. The speed-up comes from overlapping waiting time, not from faster requests, so it is bounded by how much parallelism the target should reasonably receive. The concurrency and rate limiting lessons explain how to pick that number; for most sites it is low single digits.

What the key lines do:

• The semaphore limits how many coroutines are actively requesting. Without it, gather starts all 200 at once.
• httpx.Limits caps the connection pool. Tasks that cannot get a connection wait for one, and if they wait longer than the pool timeout they fail with httpx.PoolTimeout. Keeping the semaphore at or below max_connections avoids that error entirely.
• return_exceptions=True makes gather return exceptions as results instead of raising the first one, so one bad URL does not discard 199 good responses. Zipping results back with URLs gives you a retry list.

On Python 3.11 and later, asyncio.TaskGroup is the alternative when you want the opposite behaviour: if one task fails, cancel the rest.

Mistakes That Only Show Up in Async Code#

• Blocking the event loop. Calling requests.get(), time.sleep() or a slow CPU-bound parser inside a coroutine freezes every other request. Use await asyncio.sleep(), the async client, and await asyncio.to_thread(parse, html) for heavy parsing.
• A client per request. async with httpx.AsyncClient() inside fetch defeats pooling and can exhaust sockets. Create it once and pass it in.
• Forgetting to close. An AsyncClient created without async with must be closed with await client.aclose(), or you get resource warnings and leaked connections.
• Unbounded task creation. Creating a million coroutines up front uses memory even if the semaphore holds most of them back. For very large URL lists, use a fixed pool of worker tasks reading from an asyncio.Queue.

Retries With Backoff#

HTTPTransport(retries=3) only retries failed connection attempts. Rate limits and server errors need your own logic, which should respect Retry-After when the server sends it:

import asyncio
import random
import httpx

RETRY_STATUS = {429, 500, 502, 503, 504}

async def get_with_retry(client, url, attempts=5):
    for attempt in range(attempts):
        try:
            r = await client.get(url)
        except (httpx.ConnectError, httpx.ReadTimeout) as exc:
            error = exc
        else:
            if r.status_code not in RETRY_STATUS:
                r.raise_for_status()
                return r
            error = httpx.HTTPStatusError(f"{r.status_code}", request=r.request, response=r)
            retry_after = r.headers.get("retry-after")
            if retry_after and retry_after.isdigit():
                await asyncio.sleep(min(int(retry_after), 300))
                continue
        await asyncio.sleep(min(60, 2 ** attempt) + random.random())
    raise error

Exponential backoff with random jitter stops many workers from retrying in lockstep. Only transient conditions are retried: a 404 or 403 raises immediately, because retrying it just repeats the same answer. The broader taxonomy of which errors to retry is in error handling.

Event Hooks for Logging#

Hooks run on every request or response without touching your scraping code, which makes them the right place for logging and metrics. In an AsyncClient the hooks must be async functions.

async def log_request(request: httpx.Request):
    print(f"-> {request.method} {request.url}")

async def log_response(response: httpx.Response):
    request = response.request
    print(f"<- {request.method} {request.url} {response.status_code} {response.http_version}")

client = httpx.AsyncClient(event_hooks={"request": [log_request], "response": [log_response]})

A response hook runs before the body has been read, so response.elapsed and response.text are not available there yet. If you need them, await response.aread() inside the hook first, at the cost of always buffering the body.

HTTP/2 Where the Server Offers It#

HTTP/2 multiplexes many requests over one connection and compresses repeated headers. For a crawler making many requests to one host, that means fewer connections and less overhead. Support is optional:

pip install "httpx[http2]"

with httpx.Client(http2=True) as client:
    r = client.get("https://example.com/")
    print(r.http_version)       # "HTTP/2" if the server negotiated it, otherwise "HTTP/1.1"

The protocol is negotiated during the TLS handshake, so http2=True is a request, not a guarantee; always check r.http_version if it matters. With HTTP/2, one connection can carry many concurrent streams, so max_connections no longer equals the number of requests in flight. Keep limiting concurrency with the semaphore.

Enabling HTTP/2 does not make httpx look like a browser. Servers that fingerprint clients look at TLS handshake details and HTTP/2 settings frames, and httpx's values differ from Chrome's with or without HTTP/2. That topic belongs to TLS fingerprinting.

Timeouts, Explained#

httpx.Timeout has four parts: connect (establishing the connection), read (waiting for each chunk of data), write (sending each chunk) and pool (waiting for a free connection from the pool). httpx.Timeout(10.0, connect=5.0) sets 5 seconds for connect and 10 for everything else. None of them is a total deadline for the whole request. If you need one, wrap the call: await asyncio.wait_for(client.get(url), timeout=30).

For large files, stream instead of loading the body into memory, as covered in downloading files and streaming.

Sync or Async?#

Use the sync Client for scripts, notebooks, and crawls where a handful of threads is enough. Use AsyncClient when you need dozens or hundreds of requests in flight, when the scraper is already part of an async application, or when you combine HTTP with other async I/O like WebSockets. Keep one style per module: mixing sync calls into async code is the most common source of mysteriously slow async scrapers.

Practice#

Take a sequential requests scraper of about 100 URLs from a site you are allowed to crawl. Port it to httpx.AsyncClient with a semaphore of 3, add get_with_retry, and log each response through an event hook. Measure the run time before and after, and confirm that the maximum number of simultaneous requests in your logs never exceeds 3.