Cost & Bandwidth Optimization

Scraping Has a Bill#

Every scrape costs something: proxy bandwidth, compute time, cloud egress, storage, and the engineering hours to maintain it. At ten pages a day the bill is invisible. At ten million it becomes the constraint that decides whether the project is viable. Optimising cost is a core scraping skill, and it starts with one habit: measure cost per successful record, not per request. A run that makes a million requests and produces two hundred thousand records is spending four-fifths of its money on failures and duplicates, and no amount of micro-tuning fixes that until you can see it.

Know Where the Money Goes#

For most production scrapers the costs rank roughly like this:

Instrument each bucket per target: bytes downloaded per record, requests per record, browser-seconds per record, retry rate. The rest of this lesson is the set of levers those metrics point you toward.

Escalate Transports Reluctantly#

The single largest cost decision is which tool fetches a page. Each step up this ladder costs far more CPU and bandwidth than the last, yet teams routinely start at the top:

1. Plain HTTP client (httpx, requests): cheapest, handles most static pages and JSON APIs.
2. TLS-impersonating client (curl_cffi): same cost order, adds a browser-like fingerprint for sites that check it (see TLS fingerprinting).
3. Headless browser (Playwright): tens to hundreds of times more expensive; only for pages that genuinely need JavaScript or challenge solving.

Route each target to the cheapest tool that reliably works, and re-test periodically; a page that needed a browser last year may serve its data as JSON today. Before rendering anything, check whether the data arrives from an API you can call directly (see XHR and API reverse engineering); a single JSON request often replaces a full browser render.

Cut Bytes on the Wire#

Since bandwidth is usually the top line, transfer less.

Accept modern compression. Advertise every algorithm your client can decode:

import httpx

# httpx and requests handle gzip/deflate automatically; add brotli and zstd
# by installing the optional packages so the server may send them.
client = httpx.Client(headers={"Accept-Encoding": "gzip, br, zstd"})
r = client.get("https://api.example/products")
print(r.headers.get("content-encoding"), len(r.content))

Brotli and zstd typically shrink HTML and JSON far more than gzip. Many libraries negotiate gzip by default but not brotli or zstd unless the supporting package is present, so a plain install can leave the biggest saving on the table. Verify with the content-encoding response header that the server actually compressed the body.

Block sub-resources in the browser. When you must render, the biggest single win is refusing to download images, fonts, media and stylesheets you never read:

def render(page, url):
    def route(r):
        if r.request.resource_type in {"image", "media", "font", "stylesheet"}:
            return r.abort()
        return r.continue_()
    page.route("**/*", route)
    page.goto(url, wait_until="domcontentloaded")
    return page.content()

On an image-heavy page this can cut bandwidth and render time dramatically, because images are usually most of the bytes and CSS/font fetches block rendering. Keep whatever the page needs to produce the data you want; block the rest.

Fetch only the bytes you need. If a server supports range requests, you can read a file's header without downloading it whole, which is useful for sniffing a large PDF's metadata or checking a file type:

r = client.get("https://files.example/report.pdf", headers={"Range": "bytes=0-1023"})
print(r.status_code)  # 206 Partial Content if honoured, 200 if ignored

A 206 means the server honoured the range; a 200 means it ignored it and sent the whole file, so check before assuming you saved anything.

Do Not Fetch What Has Not Changed#

The cheapest request is the one you skip. If you re-scrape sources that change rarely, store each response's ETag and Last-Modified, and send them back as validators:

def conditional_get(client, url, etag=None, last_modified=None):
    headers = {}
    if etag:
        headers["If-None-Match"] = etag
    if last_modified:
        headers["If-Modified-Since"] = last_modified
    r = client.get(url, headers=headers)
    if r.status_code == 304:
        return None                      # unchanged: near-zero bandwidth
    return r                             # 200: new content, store new validators

A 304 Not Modified has an empty body, so on an incremental run over mostly-unchanged pages this eliminates most of the transfer. Sitemaps with lastmod do the same job one level up: skip a URL entirely when the sitemap says it has not changed since your last visit. The incremental scraping lesson goes deeper, and sitemaps and feeds covers the prefilter.

Cache Raw Responses for Free Re-runs#

During development and reprocessing you re-parse the same pages repeatedly. Cache raw bodies on disk keyed by URL so parser changes cost nothing to test:

import hashlib
from pathlib import Path

CACHE = Path("raw_cache")

def cached_get(client, url):
    key = hashlib.sha256(url.encode()).hexdigest()
    path = CACHE / key[:2] / key
    if path.exists():
        return path.read_bytes()
    path.parent.mkdir(parents=True, exist_ok=True)
    body = client.get(url).content
    path.write_bytes(body)
    return body

The two-character prefix directory keeps any one folder from holding millions of files, which some filesystems handle poorly. A local raw store means you fetch each page once and re-run extraction endlessly for free; it also doubles as crash-resume (see caching and resume).

Stop Paying for Failures#

Retries that fail are pure cost: bandwidth and compute spent for zero data.

• Set realistic timeouts. A default of no timeout means one hung connection ties up a worker indefinitely.
• Honour Retry-After, and back off exponentially with jitter so you stop hammering a struggling server (which would only produce more failures).
• Cap retries. After a few attempts, move a URL to a dead-letter list instead of retrying it forever; a URL that 404s will 404 tomorrow too.
• Do not retry non-retryable errors. A 404 or 410 is permanent; retrying it wastes requests. Retry only transient failures (timeouts, 429, 5xx).

Compress stored raw data too. Gzipping HTML at rest typically shrinks it by a large factor, and columnar formats like Parquet compress extracted records further (see dataset versioning and lineage).

Measure, Then Cut the Biggest Line#

from dataclasses import dataclass

@dataclass
class Cost:
    requests: int = 0
    bytes: int = 0
    browser_seconds: float = 0.0
    retries: int = 0
    records: int = 0

    def report(self):
        n = max(self.records, 1)
        return {
            "bytes_per_record": round(self.bytes / n),
            "requests_per_record": round(self.requests / n, 2),
            "browser_s_per_record": round(self.browser_seconds / n, 3),
            "retry_rate": round(self.retries / max(self.requests, 1), 3),
        }

Log these per target and review them regularly. The pattern is consistent: the big wins come from routing work to a cheaper transport and eliminating waste (blocked sub-resources, skipped unchanged pages, dropped dead URLs), not from shaving bytes off individual requests. Fix the biggest line first, re-measure, and repeat.

A Worked Estimate#

The numbers make the priorities concrete. Suppose a daily crawl of 500,000 pages averaging 400 KB each through a residential proxy. That is 200 GB per day on the wire. Three changes, applied in order of impact:

• Enable brotli/zstd. HTML and JSON compress heavily, so the transferred bytes drop to a fraction of the raw size. This alone is usually the largest single saving because it costs nothing but a package install.
• Skip unchanged pages. If most pages change rarely, conditional requests turn the majority of fetches into empty 304 responses, so on a typical incremental day you pay full price for only the pages that actually changed.
• Right-size the transport. If 90% of pages work with a plain HTTP client and you were rendering all of them in a browser, moving that 90% off the browser removes most of the compute bill and a chunk of bandwidth (no sub-resources), leaving browsers for the 10% that need them.

Notice the order: compression and change-skipping cut bandwidth (the top line for proxy-billed crawls), while transport routing cuts compute (the top line for browser-heavy crawls). Which matters more depends on your own per-record metrics, which is exactly why you measure before optimising rather than guessing.

Checklist#

Per target, confirm: cost is tracked per accepted record, not per request; brotli and zstd are negotiated and confirmed via content-encoding; the browser blocks images, fonts, media and CSS it does not read; unchanged pages are skipped via 304 or sitemap lastmod; raw responses are cached so re-parsing is free; retries are capped with a dead-letter list and never fire on permanent errors; and the cheapest transport that works is doing each job.