The Modern Requests Library

requests is the scraper's default hammer, but httpx is the same API with three upgrades that matter at scale: a real async client, built-in HTTP/2 support, and the same interface for both. When your crawl graduates from a script to a pipeline, most of the jobs requests handled are replaced by httpx without changing how the code reads.

Drop-In Compatibility

httpx.get(url, headers=..., params=...) looks like requests.get, returns an object with .status_code, .headers, .text and .json(), and follows redirects by default. The most common newcomer surprise is that httpx does not send a User-Agent by default — same trap as requests, set one explicitly on every request or, better, on a shared client so every call inherits it.

import httpx

with httpx.Client(
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Safari/537.36"},
    timeout=10.0,
    follow_redirects=True,
) as client:
    r = client.get("https://example.com/api/items")
    data = r.json()

The reusable Client is the first real productivity win. It keeps a connection pool open, reuses TCP/TLS handshakes across requests, rotates cookies automatically, and can apply a single headers dict to every call. Opening a new requests.get per URL throws away that warm connection and looks less like a browser, so prefer one client for the whole run.

Async: Concurrency Without Threads

A sync client waits for each response; an async client pauses while the network is busy and lets other requests proceed. httpx.AsyncClient with asyncio.gather is the same familiar API, and it turns a 60-request crawl from serial minutes into parallel seconds. The key detail: pass the client explicitly so each task reuses the same pool instead of creating its own.

import asyncio, httpx

async def fetch(client, url):
    r = await client.get(url)
    return url, r.status_code

async def main():
    async with httpx.AsyncClient(
        headers={"User-Agent": "Mozilla/5.0 ..."}, timeout=15.0
    ) as client:
        results = await asyncio.gather(
            *(fetch(client, f"https://example.com/item/{i}") for i in range(60))
        )
    return results

Pairs of functions exist because there is no get that serves both worlds: sync clients use client.get, async clients use await client.get. Keep one style per module so you never mix the two by accident.

HTTP/2 Where the Server Offers It

HTTP/2 multiplexes many requests over one connection and compresses headers, which is precisely what a high-volume crawl wants. httpx speaks HTTP/2 when http2=True and the server supports it. Some anti-bot stacks fingerprint only on HTTP/1.1 and Connection header oddities, but modern backends expect http2 from browsers — matching it is one more way your client looks like the browsers it replaces.

with httpx.Client(http2=True) as client:
    r = client.get("https://example.com/")
    print(r.http_version)   # "HTTP/2"

Timeouts, Streaming and Binary

Set an explicit timeout tuple — (connect, read) — so a hung socket cannot stall the pool; httpx also ships httpx.Timeout and httpx.Limit for controlling pool size. For huge downloads use client.stream("GET", url) and iterate chunks instead of buffering the whole body into memory (see the file-downloads lesson). For binary content, r.content gives you raw bytes — decoding happens only when you ask for .text.

Migrating a scraper from requests to httpx is a zero-risk refactor: identical verbs, better pooling, async for free, and http2 to match modern servers. It is the single highest-leverage library swap a growing crawl can make.