From Laptop Script to Production Service

A scraper on your laptop dies when you close the lid. Running it in the cloud makes it scheduled, monitored, and durable. The two dominant deployment shapes are serverless functions and containers, each with distinct tradeoffs for scraping workloads.

Serverless Functions

AWS Lambda, Google Cloud Functions, and Cloudflare Workers run your code on demand with no server management. They are ideal for:

  • Scheduled, short jobs: scrape a set of pages every hour and write to storage.
  • Event handlers: react to a queue message or a webhook delivery.
  • Bursty workloads: scale to hundreds of concurrent invocations instantly.

The constraints bite quickly, though:

  • Execution time limits (often 15 minutes) rule out long crawls in one invocation.
  • Ephemeral, shared IPs are immediately flagged by anti-bot systems; serverless scraping almost always requires an external proxy.
  • No persistent filesystem: state must live in object storage or a database.
  • Cold starts add latency, and headless browsers inflate package size and runtime.

Containers

Docker containers on ECS, Cloud Run, Kubernetes, or a plain VPS remove the time and size limits. You control the runtime, can install browser dependencies, and can keep long-lived state. This is the right choice for browser-based scraping, large crawls, and anything that needs a real filesystem.

A production container image typically bundles: the application, the HTTP/browser libraries, a headless Chromium (or Playwright's browser build), and the certificate store. Keep it slim—large images slow every deploy and cold start.

Scheduling

  • cron on a VPS is the simplest scheduler.
  • Managed schedulers (Cloud Scheduler, EventBridge) trigger functions or containers reliably with retries.
  • Workflow orchestrators (Airflow, Prefect, Temporal) model dependencies between stages: fetch → parse → resolve → publish. Use them when a scrape is a pipeline, not a single step.

Whatever you choose, make runs idempotent and resumable: if a run is triggered twice or crashes midway, it should not duplicate or lose data.

State and Coordination

Two independent instances must not scrape the same URL twice or exceed a shared rate limit. Use a central queue (SQS, Redis, RabbitMQ) to hand out work with leases, and a shared rate limiter so the aggregate request rate stays within tolerance. The queue is also your progress tracker: unacked messages are work to resume.

Secrets Management

Never bake credentials, session cookies, or proxy passwords into the image or code. Inject them from a secrets manager (AWS Secrets Manager, HashiCorp Vault) at runtime, scope them to the job, and rotate them regularly. Log carefully—secrets leak through debug logs more often than through any other channel.

Observability

In the cloud you cannot tail -f easily, so build observability in: structured logs shipped to a central store, metrics for records scraped and errors encountered, and alerts when a scheduled run stops producing data. A scraper that silently fails for a week is worse than one that crashes loudly.

Choosing

Use serverless for small, frequent, stateless jobs with external proxies. Use containers for browser automation, long crawls, and complex pipelines. Many mature systems combine both: a scheduled function enqueues work, containers consume it, and a function handles the final publish. Match the shape of the tool to the shape of the job.