Running Scrapers in the Cloud

From Laptop Script to Production Service#

A scraper on your laptop dies when you close the lid, forgets to run when you are on holiday, and leaves no trace when it fails. Moving it to the cloud makes it scheduled, monitored and durable. Two deployment shapes dominate, serverless functions and containers, and they suit different scraping jobs. Choosing wrong is expensive: serverless for a browser crawl means fighting cold starts and time limits, while a container for a tiny hourly poll wastes a machine sitting idle. This lesson covers the trade-offs, the shared concerns (state, secrets, observability) that decide reliability, and how mature systems combine both shapes.

Serverless Functions#

AWS Lambda, Google Cloud Functions, Azure Functions and Cloudflare Workers run your code on demand with no server to manage, scaling from zero to hundreds of concurrent invocations instantly. They fit:

• Short, scheduled jobs: fetch a set of pages every hour and write to storage.
• Event handlers: react to a queue message or a webhook delivery (see webhooks and event-driven ingestion).
• Bursty, embarrassingly parallel work: fan out one function per URL.

The constraints bite quickly for scraping:

• Execution time limits (commonly up to about 15 minutes on the big providers) rule out a long crawl in one invocation. Split the work into many short invocations instead.
• Ephemeral, shared, datacenter IPs are exactly the IPs anti-bot systems distrust most, and they rotate outside your control. Serverless scraping of protected sites almost always needs an external proxy (see proxies and rotation).
• No persistent local disk. The filesystem is temporary and not shared between invocations, so state must live in object storage or a database.
• Cold starts and package size. A headless browser inflates the deployment package and adds seconds of startup; some platforms cannot run one at all without a special layer or container image.

A Lambda-style handler for a short scrape keeps each invocation small and idempotent:

import json
import os
import boto3
import httpx

s3 = boto3.client("s3")
BUCKET = os.environ["RAW_BUCKET"]

def handler(event, context):
    url = event["url"]
    proxy = os.environ.get("PROXY_URL")           # injected, not hard-coded
    with httpx.Client(proxy=proxy, timeout=20, follow_redirects=True) as c:
        r = c.get(url)
    key = f"raw/{event['run_id']}/{event['url_id']}.html"
    s3.put_object(Bucket=BUCKET, Key=key, Body=r.content)   # idempotent: same key
    return {"url": url, "status": r.status_code, "key": key}

Writing to a deterministic S3 key makes re-invocation safe: a retried event overwrites the same object instead of creating a duplicate.

Containers#

Docker containers on ECS/Fargate, Cloud Run, Kubernetes or a plain VPS remove the time and size limits. You control the runtime, install browser dependencies, and keep long-lived state. This is the right choice for browser automation, large crawls and anything needing a real filesystem or a process that runs for hours. A Playwright image looks roughly like:

FROM mcr.microsoft.com/playwright/python:v1.55.0-noble
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-m", "crawler.worker"]

Using the vendor's Playwright base image saves you assembling Chromium's system libraries by hand, which is the fiddliest part of containerising a browser scraper. Keep the image slim otherwise: large images slow every deploy and cold start, and on Cloud Run or Fargate a fat image means longer scale-up.

Serverless vs Containers#

Scheduling#

• cron on a VPS is the simplest scheduler for a single container.
• Managed schedulers (Cloud Scheduler, EventBridge Scheduler) trigger functions or containers reliably, with their own retries.
• Workflow orchestrators (Airflow, Prefect, Temporal) model dependencies between stages when a scrape is a pipeline, not a single step (see data pipelines and orchestration).

Whatever triggers the job, make runs idempotent and resumable: a run started twice, or crashed halfway and restarted, must not duplicate or lose data. In the cloud, double-triggers are common (a scheduler retry, an at-least-once queue), so this is not optional.

State and Coordination#

The moment two instances run at once, they will scrape the same URL twice and blow past a shared rate limit unless something coordinates them. Two pieces solve it:

• A central work queue (SQS, Redis, RabbitMQ) hands out URLs with a visibility timeout or lease. A worker that takes a URL, crashes, and never acknowledges it lets the message reappear for another worker after the lease expires. The queue is also your progress tracker: unacknowledged messages are exactly the work left to do.
• A shared rate limiter so the aggregate request rate across all workers stays within the target's tolerance. A per-worker limit does not help when you scale from two workers to twenty. A common implementation is a token bucket in Redis that every worker draws from (see rate limiting and politeness and distributed crawling).

import redis, time

r = redis.Redis()

def allow(host, rate_per_sec):
    '''Fixed-window limiter shared across all workers for one host.'''
    key = f"rl:{host}:{int(time.time())}"
    n = r.incr(key)
    if n == 1:
        r.expire(key, 2)
    return n <= rate_per_sec

This fixed-window counter is simple and good enough for keeping an aggregate rate polite; a token-bucket variant smooths bursts at window edges if you need it.

Secrets#

Never bake credentials, session cookies or proxy passwords into the image or code; an image is not a secret store and layers can be extracted. Inject them at runtime from a secrets manager (AWS Secrets Manager, Google Secret Manager, HashiCorp Vault), scope each secret to the job that needs it, and rotate on a schedule. Secrets leak through debug logs more often than through any other channel, so redact them before logging (see secrets management).

Observability#

You cannot tail -f a fleet, so build observability in from the start:

• Structured logs with a run_id on every line, shipped to a central store, so you can reconstruct one run across many workers.
• Metrics: records scraped, errors, bytes, retries, queue depth.
• Alerts when a scheduled run stops producing data. A scraper that silently fails for a week is worse than one that crashes loudly, because you keep trusting stale data.

The staleness and volume-collapse checks from scheduling and monitoring are what turn those metrics into a page at 4am instead of a surprise at the next board meeting.

The Combined Shape#

Mature systems rarely pick one. A common architecture uses both for what each does best:

1. A scheduled function wakes on a timer, discovers URLs (sitemap, feed) and enqueues them.
2. A pool of containers drains the queue, running browsers where needed, writing raw payloads to object storage and records to a database, coordinated by the shared queue and rate limiter.
3. A function triggered on completion publishes or notifies downstream.

Match the shape of the tool to the shape of the job: serverless for the small, frequent, stateless edges, containers for the heavy, stateful middle. Get state, secrets and observability right across both, and a laptop script becomes a service that runs unattended for months.

Cloud-Specific Failure Modes#

Some bugs only appear once the code leaves your laptop:

• The double-trigger. A scheduler retries a run it thinks failed, or an at-least-once queue redelivers a message, and a non-idempotent job duplicates rows. Test it deliberately: invoke the same job twice and assert the output is identical.
• The timeout cliff. A crawl that grew past the function's time limit is killed mid-write, leaving partial data with no error. Move long work to a container, or split it into short invocations that each checkpoint.
• The silent IP ban. Serverless IPs are shared and reused, so another tenant's abuse can get your function blocked through no fault of yours. An external proxy pool isolates you from that.
• The vanished filesystem. Code that wrote to local disk works in testing and loses everything between invocations in production. Route all state to object storage or a database.
• The invisible failure. Without shipped logs and a staleness alert, a broken scheduled job looks exactly like a working one that found nothing. Alert on "no data produced", not only on crashes.

Checklist#

Before a cloud scraper is production-ready: runs are idempotent and survive a double-trigger; long jobs run in containers, short stateless ones in functions; all state lives in object storage or a database, never local disk; a shared queue hands out work with leases and a shared limiter caps the aggregate rate; secrets are injected at runtime and kept out of logs; and structured logs plus a staleness alert make a silent failure visible.