The Part That Isn't HTML#

Data does not end at HTML. Organisations publish CSV exports, zipped archives of filings, PDF reports, image sets and multi-gigabyte data dumps. Downloading these well is a different skill from parsing pages. The failure modes are "the process ran out of memory", "the connection dropped at 87%", "we saved a login page and named it report.pdf", and "the file is corrupt and nobody noticed for a month". Four habits cover almost every binary download a scraper does: stream to disk, write atomically, verify what you got, and resume instead of restarting.

Stream in Chunks, Never in Full#

resp.content, resp.text and resp.json() load the entire body into memory. That is fine for a 50 KB page and a crash for a 3 GB archive. Streaming hands you the body in pieces as it arrives, so memory stays flat regardless of file size. In httpx, streaming is a context manager on the client:

import httpx

def download(client: httpx.Client, url: str, dest: str) -> int:
    written = 0
    with client.stream("GET", url) as resp:
        resp.raise_for_status()
        with open(dest, "wb") as f:
            for chunk in resp.iter_bytes(chunk_size=65536):
                f.write(chunk)
                written += len(chunk)
    return written

with httpx.Client(timeout=httpx.Timeout(10.0, read=60.0), follow_redirects=True) as client:
    print(download(client, "https://example.com/data/export.csv", "export.csv"))

The equivalent in requests is requests.get(url, stream=True) followed by resp.iter_content(chunk_size=65536). A chunk size of 64 KiB is a reasonable default: small enough to keep memory trivial, large enough that Python loop overhead does not dominate. Never collect the chunks in a list and join them at the end; that just rebuilds resp.content more slowly.

Two non-obvious points about timeouts. httpx timeouts apply to each network operation, not to the whole transfer, so a 2 GB download that keeps receiving data will not be killed by a 60-second read timeout. That same read timeout is what rescues you from a stalled connection that stops sending bytes but never closes.

One gotcha with requests: shutil.copyfileobj(resp.raw, f) is fast, but resp.raw does not decode Content-Encoding by default, so a gzip-encoded response lands on disk still compressed. iter_content and httpx's iter_bytes decode for you; iter_raw in httpx does not.

Write to a Temporary Name, Then Rename#

If the process dies mid-download, a half-written report.pdf looks exactly like a finished one to the next step of the pipeline. Write to report.pdf.part and rename only after the file is complete and verified. os.replace is atomic on the same filesystem, so readers see either the old file or the complete new one, never a fragment.

import os

tmp = dest + ".part"
download(client, url, tmp)
os.replace(tmp, dest)

The .part file then doubles as the resume point for the next section.

Verify What You Got#

Corruption and wrong content arrive silently. Three cheap checks catch nearly all of it:

  1. Size. If the response had Content-Length and no Content-Encoding, the bytes written must equal it. A mismatch means a truncated transfer. With compression, Content-Length counts compressed bytes and will not match the decoded size.
  2. Type. Servers that fail often return an HTML error or login page with status 200. Check the first bytes of the file, not just the Content-Type header.
  3. Checksum. When the publisher lists a SHA-256 next to the download, compare against it. Hash while writing so you never re-read the file.
import hashlib

MAGIC = {".pdf": b"%PDF-", ".zip": b"PK", ".xlsx": b"PK", ".png": b"PNG"}

def looks_like_html(first_bytes: bytes) -> bool:
    head = first_bytes.lstrip().lower()
    return head.startswith(b"<!doctype html") or head.startswith(b"<html")

def download_verified(client, url, dest, expected_sha256=None):
    h = hashlib.sha256()
    first = b""
    tmp = dest + ".part"
    with client.stream("GET", url) as resp:
        resp.raise_for_status()
        with open(tmp, "wb") as f:
            for chunk in resp.iter_bytes(chunk_size=65536):
                if not first:
                    first = chunk[:16]
                f.write(chunk)
                h.update(chunk)
    if looks_like_html(first):
        raise ValueError(f"{url} returned an HTML page, not a file")
    for ext, magic in MAGIC.items():
        if dest.endswith(ext) and magic not in first[:8]:
            raise ValueError(f"{url}: expected {ext} signature, got {first[:8]!r}")
    if expected_sha256 and h.hexdigest() != expected_sha256.lower():
        raise ValueError(f"checksum mismatch for {url}")
    os.replace(tmp, dest)
    return h.hexdigest()

PNG files start with a non-printable byte followed by PNG, which is why the table checks for PNG within the first eight bytes rather than an exact prefix. Store the returned digest with the file's metadata: it lets you deduplicate identical files published under different URLs, and later detect when a file at the same URL changed.

Resume Interrupted Transfers With Range#

Long downloads get interrupted. HTTP's answer is the Range header: a server that supports it (it usually advertises Accept-Ranges: bytes) will send only the bytes from an offset you name, with status 206 Partial Content. A resumable downloader checks how much of the .part file already exists and asks for the rest.

import hashlib
import os

def resumable_download(client, url, dest, expected_sha256=None):
    tmp = dest + ".part"
    done = os.path.getsize(tmp) if os.path.exists(tmp) else 0
    headers = {"Accept-Encoding": "identity"}
    if done:
        headers["Range"] = f"bytes={done}-"

    with client.stream("GET", url, headers=headers) as resp:
        if resp.status_code == 206:
            mode = "ab"                        # append the remaining bytes
        elif resp.status_code == 200:
            mode = "wb"                        # server ignored Range: start over
        elif resp.status_code == 416:
            mode = None                        # nothing left to send
        else:
            resp.raise_for_status()
            raise RuntimeError(f"unexpected status {resp.status_code}")

        if mode:
            with open(tmp, mode) as f:
                for chunk in resp.iter_bytes(chunk_size=65536):
                    f.write(chunk)

    # hash the whole file: bytes from earlier runs count too
    h = hashlib.sha256()
    with open(tmp, "rb") as f:
        for block in iter(lambda: f.read(1 << 20), b""):
            h.update(block)
    if expected_sha256 and h.hexdigest() != expected_sha256.lower():
        os.remove(tmp)
        raise ValueError("checksum mismatch after resume; partial file discarded")
    os.replace(tmp, dest)
    return h.hexdigest()

Why each line is there:

  • Accept-Encoding: identity asks for the file uncompressed. Byte ranges refer to the encoded representation, so mixing on-the-fly gzip with ranges can splice incompatible bytes together.
  • A 200 in reply to a range request means the server does not support ranges, or decided to send the whole thing. Appending it to the partial file would corrupt it, so the code truncates and starts over.
  • 416 Range Not Satisfiable usually means the partial file is already complete. It can also mean the remote file got smaller, which the checksum then catches.
  • The hash is computed over the finished file on disk. Hashing only the bytes received in this run (a common bug) produces a digest of the tail, which never matches the published one.

If the remote file might change between attempts, also send If-Range with the ETag you saw on the first attempt. The server then returns a range only if the file is unchanged, and a full 200 otherwise, which the code above already handles correctly.

Choosing a Safe Filename#

The server's suggested name arrives in Content-Disposition, sometimes in the RFC 2231 form filename*=UTF-8''... for non-ASCII names. The standard library's email parser handles both forms. Never use that value as a path directly: a malicious or buggy server can send ../../.bashrc, the classic path traversal problem.

from email.message import Message
from pathlib import PurePosixPath
from urllib.parse import unquote, urlsplit

def safe_filename(resp, url: str) -> str:
    msg = Message()
    msg["content-disposition"] = resp.headers.get("content-disposition", "")
    name = msg.get_filename() or unquote(PurePosixPath(urlsplit(url).path).name)
    name = PurePosixPath(name.replace(chr(92), "/")).name   # drop any directory part
    name = "".join(c for c in name if c.isalnum() or c in "._- ").strip(" .")
    return name or "download.bin"

chr(92) is the backslash character, converted to / so Windows-style paths are also stripped. For large collections, a better scheme is to name files by their SHA-256 and keep the original name in metadata: it removes collisions and gives you deduplication for free.

Many Files at Once#

For hundreds of independent files, fan out with asyncio and a semaphore, as in the concurrency lesson. Each task streams to its own file, and the semaphore caps how many transfers run at the same time.

import asyncio
import os
import httpx

async def fetch_one(client, sem, url, dest):
    async with sem:
        async with client.stream("GET", url) as resp:
            resp.raise_for_status()
            with open(dest + ".part", "wb") as f:
                async for chunk in resp.aiter_bytes(65536):
                    f.write(chunk)
        os.replace(dest + ".part", dest)

async def fetch_all(jobs):
    sem = asyncio.Semaphore(4)
    async with httpx.AsyncClient(timeout=httpx.Timeout(10.0, read=60.0)) as client:
        results = await asyncio.gather(
            *(fetch_one(client, sem, u, d) for u, d in jobs), return_exceptions=True
        )
    for (url, _), r in zip(jobs, results):
        if isinstance(r, Exception):
            print("failed:", url, r)

Keep the concurrency low for large files. A few parallel streams usually saturate your link, and many parallel multi-gigabyte downloads are exactly the kind of load that gets a scraper blocked; the rate limiting and politeness rules apply to bytes as much as to requests. return_exceptions=True keeps one failed file from cancelling the rest, and the failures list becomes the retry queue for the next run.

Checklist#

  • Stream with a read timeout; never call .content on unknown-size responses.
  • Write to .part, verify, then os.replace.
  • Check magic bytes so an HTML error page cannot pass as a PDF.
  • Resume with Range, handle 200/206/416 separately, and hash the complete file.
  • Sanitise filenames, or name files by hash.