Scraping Has a Bill

Every scrape costs something: proxy bandwidth, compute time, egress fees, retries, and the engineering hours spent maintaining it. At small scale the bill is invisible; at production scale it becomes the dominant constraint. Optimizing cost is therefore a core scraper skill, not an afterthought.

Know Where the Money Goes

Typical cost buckets:

  • Residential/mobile proxy bandwidth: usually billed per gigabyte and by far the largest line item for evasive scraping.
  • Compute: headless browsers burn CPU and RAM; a fleet of Chromium instances can rival the cost of the data it produces.
  • Egress and storage: moving and storing raw HTML adds up over millions of pages.
  • Retries and wasted requests: every 429, timeout, and CAPTCHA is money spent for no data.

Measure cost per successful record, not per request. That single metric reveals which optimizations actually help.

Compress Everything

Always request and accept compression: Accept-Encoding: gzip, deflate, br, zstd. Brotli and zstd routinely cut HTML and JSON transfer size by 70–90% versus plain text. Most HTTP libraries enable gzip by default but not Brotli—turn it on. Since bandwidth is often the biggest cost, compression is often the single largest saving.

Conditional Requests

If you re-scrape sources that rarely change, use validators. Store the ETag and Last-Modified from the last fetch and send If-None-Match / If-Modified-Since. When the content is unchanged, the server returns 304 Not Modified with an empty body—you learn the page is unchanged for a fraction of the bandwidth. For sitemaps and feeds this can eliminate most of the cost on incremental runs.

Fetch Only What You Need

  • Use Range requests to download only the bytes you need when a server supports it (e.g., the header of a large PDF).
  • Prefer JSON API endpoints over rendering full HTML when the API returns the same data with less markup.
  • Skip images, fonts, CSS, and scripts when rendering: configure the browser to block resource types you do not need. This is the single biggest win for headless-browser bandwidth and CPU.
  • Request the smallest representation that works; avoid fetching high-resolution media unless the job requires it.

Choose the Cheapest Transport That Works

Escalate reluctantly: plain HTTP client → TLS-impersonating client (curl_cffi) → headless browser. Each step up costs far more CPU and bandwidth than the last. Many teams jump to browsers by default when a plain or impersonating HTTP client would succeed for 90% of pages. Route each target to the cheapest tool that reliably works, and reserve browsers for pages that genuinely require JavaScript or challenge solving.

Cache Aggressively

Cache raw responses on disk keyed by URL plus a content hash. During development and reprocessing, never re-fetch what you already have. A local raw-HTML store lets you re-run parsing logic endlessly for free, and a resume mechanism avoids re-downloading completed work after a crash.

Reduce Retries

Retries are pure cost when they fail. Tune timeouts to realistic values, honor Retry-After, and use exponential backoff with jitter so you stop hammering a struggling server. Drop permanently failing URLs to a dead-letter list instead of retrying them forever. Every avoided retry is bandwidth and compute saved.

Measure and Iterate

Instrument per-target metrics: bytes downloaded per record, requests per record, browser-seconds per record, retry rate. Review them monthly. Cost optimization is an ongoing discipline, and the biggest wins usually come from routing more work to cheaper transports and eliminating waste, not from micro-tuning individual requests.