Distributed Worker Fleets at Scale

A Thousand Browsers, One Orchestrated Human#

When a single IP or a single browser profile is no longer enough, scraping architecture shifts from "a scraper with a proxy pool" to "a message-driven fleet of worker machines, each impersonating one consistent human identity." This is the same architecture improvements you would apply to any distributed system—queues, leases, graceful shutdown, monitoring—with fingerprint-management constraints layered on top.

The Fleet Anatomy#

• A central job queue (Redis or a managed MQ) holds URLs with metadata: expected priority, retry budget, geo-region requirement, and minimal session affinity.
• Workers (k8s pods or a pool of VMs) dequeue a job, claim it with a lease, run their browser workflow, and push results to a store.
• An orchestrator supervises leases: it detects dead workers, requeues unacked jobs, and throttles the global request rate below the target's tolerance.

Per-Worker Fingerprint Discipline#

Every worker owns one or more fixed browser profiles and one geo-aligned residential/mobile IP bundle. The critical rule: a worker never borrows another worker's profile, and a profile never leaves its region. Cross-region reuse of one fingerprint is the most reliable way to expose the entire fleet at once—the WAF sees the same device identity hopping continents in minutes.

Coordinated Slow Crawling (The Ghost Town)#

The "ghost town" pattern scales beautifully: distribute the daily request budget across IPs so each IP emits a request every 30–120 seconds. From the target's view, they see hundreds of slow, unique, human-shaped visitors. Use a shared token-bucket rate limiter (Redis) so no worker bursts beyond the fleet's aggregate cap even during retry storms.

Failure Domains#

Design failure domains so a single ban event cannot cascade. Profile-level isolation: a banned profile is retired, its queue drained to maturing profiles. IP-pool-level isolation: never let two workers share one proxy subnet for the same target. Worker-level isolation: a worker that triggers a non-200/block response transitions to "cooling off" before taking new jobs. Target-level isolation: separate the queue per target so a ban on one site cannot stall an unrelated crawl. Every boundary you can draw is one less way for one incident to become a fleet-wide outage, and the boundaries are cheapest to add while you are first designing the fleet.

Scheduling Across Time Zones#

A globally distributed fleet should look like a globally distributed audience. Route jobs for a given target through IPs whose local clocks are in daytime or early evening—not 3 a.m., when real traffic for a consumer site collapses. A sudden spike of traffic at a single site's dead of night is a subtle but real detection vector that coordinated fleets often overlook. Assign workers to targets based on region-appropriate activity windows.

Instrumentation for the Fleet#

Log per-worker metrics—requests/s, success rate, ban rate, average risk score if exposed—and alert when any worker's success rate dips. The fleet's subagents: - Requeue failed URLs with respect to per-profile backoff. - Automatically retire profiles on N consecutive challenges. - Rebalance workers between jobs when one target begins rejecting.

The calmark of a mature fleet is that it survives any single worker, IP, or profile failing, because every component is disposable and every failure is requeued.

State That Must Never Cross a Worker Boundary#

The fleet anatomy gives you a queue and workers. What it does not tell you is which pieces of state are allowed to move between workers, and the answer is a much shorter list than most first implementations assume. Everything below the line is worker-local for the entire lifetime of a profile, and a design that lets any of it cross is a design that will hand the target a device that changes identity mid-session.

The one that bites hardest in practice is the challenge token. A clearance cookie minted on a worker's residential exit in Osaka is worthless on the same profile from Lisbon, and replaying it there is worse than not using it: the replay attempt is itself a signal that someone is moving clearance cookies between addresses. If a worker fails a challenge, the token dies with the attempt. Requeue the job, never the token. The token lifecycle is spelled out in managed challenge flows, and the discipline there -- one solve, one host, one action, then discard -- is exactly the discipline a fleet needs to inherit.

Exactly-Once Is a Claim You Have to Choose#

Queue systems give you at-least-once delivery, because the alternative requires a distributed transaction between the queue broker and your side-effect store. So the question is not whether duplicates happen; it is what a duplicate costs you. For a scrape-and-store pipeline the cost is a duplicate row. For a pipeline that consumes metered quota, spends a paid API credit, or advances an account-warmup account lifecycle counter, the cost is real money and real burn rate.

The answer is an idempotency key, and the discipline is choosing it so that it is semantically the dedupe key rather than the job id. A job id is unique per delivery attempt, so it deduplicates nothing. What you want is the tuple that means "this unit of work has already been done":

• For a page fetch: (profile_id, url, canonical_query) -- the same profile visiting the same URL again is the same observation.
• For a paid API call: the provider's own idempotency key, generated once and carried through every retry.
• For an account-warmup action: (account_id, action, local_date), so a retried "browse three category pages" cannot silently become four.

Store the key with a TTL longer than your maximum retry horizon plus the maximum queue visibility timeout. A dedupe table with a 60-second TTL against a queue that retries over 30 minutes will deduplicate the first retry and none of the rest, which is worse than not having one, because you will believe the problem is handled.

The table below walks one job through the three failure modes that produce duplicates -- a stolen lease, a lost ack, and an overrun -- with the dedupe key on and off. The number that matters is over-apply ratio: side effects divided by distinct jobs. Anything above 1.00 means you acted on the same logical work more than once.

import itertools

LEASE_TTL_S = 30.0
MAX_ATTEMPTS = 3
SKEW_S = {"w-north": 0.0, "w-south": -37.0, "w-west": 12.0}
PROFILE = {"w-north": "p-osaka", "w-south": "p-lisbon", "w-west": "p-aarhus"}


class Coordinator:
    '''Authoritative clock, lease registry, dedupe table and dead-letter queue.'''

    def __init__(self, dedupe_on):
        self.t = 0.0
        self.dedupe_on = dedupe_on
        self.fences = itertools.count(1)
        self.lease = {}
        self.row_fence = {}
        self.side_effects = []
        self.accepted = {}
        self.attempts = {}
        self.dead = []
        self.quarantined = set()
        self.m = dict.fromkeys([
            "grants", "grants_refused_quarantine", "grants_refused_profile",
            "reaped", "commits", "accepted",
            "suppressed_dedupe", "rejected_fenced", "rejected_expired",
            "requeued", "quarantines"], 0)

    def grant(self, job, worker, profile=None):
        if worker in self.quarantined:
            self.m["grants_refused_quarantine"] += 1
            return None
        if profile is not None and PROFILE[worker] != profile:
            self.m["grants_refused_profile"] += 1
            return None
        fence = next(self.fences)
        self.lease[job] = {"worker": worker, "fence": fence, "expire": self.t + LEASE_TTL_S}
        self.m["grants"] += 1
        return fence

    def reap(self, job):
        held = self.lease.pop(job)
        self.m["reaped"] += 1
        return held

    def commit(self, job, worker, fence, key):
        self.m["commits"] += 1
        held = self.lease.get(job)
        if not held or self.t >= held["expire"]:
            self.m["rejected_expired"] += 1
            return "rejected:lease-expired"
        if self.row_fence.get(job, 0) > fence:
            self.m["rejected_fenced"] += 1
            return "rejected:fenced-out"
        if self.dedupe_on and key in self.accepted:
            self.m["suppressed_dedupe"] += 1
            return "suppressed:duplicate-key"
        self.accepted[key] = job
        self.row_fence[job] = fence
        self.side_effects.append((job, key, worker))
        self.m["accepted"] += 1
        return "committed"

    def fail(self, job, reason):
        self.attempts[job] = self.attempts.get(job, 0) + 1
        if self.attempts[job] >= MAX_ATTEMPTS:
            self.dead.append((job, reason))
        else:
            self.m["requeued"] += 1

    def quarantine(self, worker):
        self.quarantined.add(worker)
        self.m["quarantines"] += 1
        held = sorted(j for j, h in self.lease.items() if h["worker"] == worker)
        for job in held:
            self.lease.pop(job)
            self.fail(job, "worker-quarantined")
        return held


def run(dedupe_on):
    co = Coordinator(dedupe_on)
    log = []

    # w-south sizes its lease from its own clock, which runs 37 s behind the
    # coordinator's, so it plans for 67 s of headroom on a 30 s lease.
    believes = LEASE_TTL_S - SKEW_S["w-south"]
    log.append(("skew", "w-south plans {:.0f}s on a {:.0f}s lease".format(
        believes, LEASE_TTL_S)))

    # --- j-101: the lease is stolen while the original holder is still working
    co.t = 0.0
    f1 = co.grant("j-101", "w-south")
    co.t = 5.0
    co.grant("j-102", "w-west")
    co.t = 20.0
    co.reap("j-101")
    f2 = co.grant("j-101", "w-north")
    log.append(("lease stolen", "fence {} superseded by fence {}".format(f1, f2)))
    co.t = 24.0
    log.append(("new holder", co.commit("j-101", "w-north", f2, "j-101")))
    co.t = 34.0
    log.append(("stale holder", co.commit("j-101", "w-south", f1, "j-101")))
    co.t = 35.0
    log.append(("redelivery", co.commit("j-101", "w-north", f2, "j-101")))

    # --- j-105: nobody steals it, the holder simply runs past the TTL
    co.t = 40.0
    f3 = co.grant("j-105", "w-north")
    co.t = 40.0 + LEASE_TTL_S + 5.0
    log.append(("overran", co.commit("j-105", "w-north", f3, "j-105")))

    # --- w-west is quarantined after consecutive bans
    co.t = 80.0
    held = co.quarantine("w-west")
    log.append(("quarantine", "w-west, {} in-flight job(s) requeued".format(len(held))))
    co.t = 81.0
    co.grant("j-106", "w-west")
    co.t = 82.0
    co.grant("j-106", "w-south")

    # --- the scheduler asks w-north to carry a job tagged for another profile
    co.t = 90.0
    co.grant("j-107", "w-north", profile="p-lisbon")
    co.t = 90.5
    f8 = co.grant("j-107", "w-south", profile="p-lisbon")
    log.append(("profile pin", "j-107 refused on w-north, granted on w-south (fence {})".format(f8)))

    # --- j-103 is a poison message: the reason it fails is not retryable
    for _ in range(MAX_ATTEMPTS):
        co.t += 1.0
        co.fail("j-103", "selector-removed")

    return co, log


print("lease_ttl={}s  max_attempts={}  worker clock offset vs coordinator: {}".format(
    int(LEASE_TTL_S), MAX_ATTEMPTS,
    ", ".join("{} {:+d}s".format(w, int(v)) for w, v in sorted(SKEW_S.items()))))
print()

for label, flag in [("at-least-once, no dedupe key", False), ("at-least-once, deduped", True)]:
    co, log = run(flag)
    print("=== {} ===".format(label))
    for what, detail in log:
        print("  {:<14} {}".format(what, detail))
    print("  {:<14} {}".format("dead letter", sorted(j + ":" + r for j, r in co.dead)))
    print()
    print("  {:<22} {}".format("METRIC", "VALUE"))
    print("  " + "-" * 42)
    for k in sorted(co.m):
        print("  {:<22} {}".format(k, co.m[k]))
    print("  {:<22} {}".format("side_effects", len(co.side_effects)))
    print("  {:<22} {}".format("distinct jobs", len(set(s[0] for s in co.side_effects))))
    print("  {:<22} {}".format("over-apply ratio",
                              "{:.2f}".format(len(co.side_effects) /
                                              float(len(set(s[0] for s in co.side_effects)) or 1))))
    print()

lease_ttl=30s  max_attempts=3  worker clock offset vs coordinator: w-north +0s, w-south -37s, w-west +12s

=== at-least-once, no dedupe key ===
  skew           w-south plans 67s on a 30s lease
  lease stolen   fence 1 superseded by fence 3
  new holder     committed
  stale holder   rejected:fenced-out
  redelivery     committed
  overran        rejected:lease-expired
  quarantine     w-west, 1 in-flight job(s) requeued
  profile pin    j-107 refused on w-north, granted on w-south (fence 6)
  dead letter    ['j-103:selector-removed']

  METRIC                 VALUE
  ------------------------------------------
  accepted               2
  commits                4
  grants                 6
  grants_refused_profile 1
  grants_refused_quarantine 1
  quarantines            1
  reaped                 1
  rejected_expired       1
  rejected_fenced        1
  requeued               3
  suppressed_dedupe      0
  side_effects           2
  distinct jobs          1
  over-apply ratio       2.00

=== at-least-once, deduped ===
  skew           w-south plans 67s on a 30s lease
  lease stolen   fence 1 superseded by fence 3
  new holder     committed
  stale holder   rejected:fenced-out
  redelivery     suppressed:duplicate-key
  overran        rejected:lease-expired
  quarantine     w-west, 1 in-flight job(s) requeued
  profile pin    j-107 refused on w-north, granted on w-south (fence 6)
  dead letter    ['j-103:selector-removed']

  METRIC                 VALUE
  ------------------------------------------
  accepted               1
  commits                4
  grants                 6
  grants_refused_profile 1
  grants_refused_quarantine 1
  quarantines            1
  reaped                 1
  rejected_expired       1
  rejected_fenced        1
  requeued               3
  suppressed_dedupe      1
  side_effects           1
  distinct jobs          1
  over-apply ratio       1.00

Read the two runs against each other. Without the dedupe key, four commit attempts produce two side effects on a single job, and nothing in the system notices. With it, the redelivery is suppressed and the ratio is 1.00. The fencing rejection is doing separate and equally important work: it is what stops the slow original holder from writing after its lease was taken, which in a scrape pipeline means one profile submitting two copies of the same session to the target.

Fencing Tokens, and Why Clock Skew Matters#

Leases expire. A worker that finishes late, or that loses its lease to a reaper, can still be holding a live browser session and will still try to commit. If your storage accepts that write, you have two workers owning one profile, and the target sees the same fingerprint from two addresses within the lease window -- a very short-lived but very distinctive anomaly.

The fix is a monotonically increasing fencing token, handed out by the coordinator with every lease grant. The consumer records the highest token it has accepted per row and rejects anything lower. Two properties matter and are frequently confused:

1. Fencing is order-dependent. If the stale holder writes first, the fresh holder's write is the one rejected. Tokens prevent the later stale write from landing; they do not order writes by themselves. Shorter leases and holder-side checks are what reduce how often a stale write wins the race.
2. Tokens require a monotonic resource. The check fence < stored_fence only works on storage that never loses an update. A eventually-consistent store, or a cache you can evict, will eventually accept a token you already rejected.

Now the skew. A worker that computes its own lease remaining time as expires_at - local_now() is reading a different clock from the one that wrote expires_at. Thirty-seven seconds of skew on a thirty-second lease means the worker plans for sixty-seven seconds of headroom and is still working when the coordinator has already given the job away. This is not exotic: NTP on a busy container host drifts, a suspended laptop resumes, a VM gets migrated between hypervisors with a different time source. The mitigation is to never let the worker's local clock decide a liveness question. The coordinator's clock expires leases; the worker renews by asking, and treats a failed renewal as "someone else owns this now" -- stop, discard the profile's in-flight state, and requeue.

One trap: never encode a lease deadline as a wall-clock timestamp in a job payload. It is stale the moment the payload is serialised. Encode a duration, and let the receiving side compute the deadline against the coordinator.

Queue Semantics and Poison Messages#

Four queue behaviours decide whether a fleet degrades gracefully or becomes an incident.

Visibility timeout and redelivery. The broker hides a message for vt seconds after delivery, then redelivers if no ack arrived. Set vt from the 95th percentile of job duration multiplied by three, not from the mean. A vt shorter than your slowest job guarantees that every slow job gets delivered twice, which on a browser workflow means two browsers, one profile, and a duplicated challenge solve.

Redelivery count and the dead-letter queue. Cap attempts at three and route the fourth failure to a dead-letter queue with the failure reason attached. The important part is the reason, because it distinguishes two failures that look identical in a metric: a network timeout, which may succeed on a different worker, and a selector that no longer exists, which will never succeed anywhere. Retrying the second one three times costs you three challenge solves and three strikes against the identity for nothing.

Priority and fairness. A high-priority queue that drains against a shared pool of profiles will starve the bulk queue and, worse, will concentrate your highest-volume work on your best profiles -- which is exactly the population a risk-scoring system will have modelled most thoroughly. Reserve capacity per queue tier rather than letting priority win on arrival order.

Backpressure over buffering. When the target's Retry-After or your own rate limiter says slow down, the correct behaviour is to stop dequeuing, not to dequeue faster and sleep inside the worker. Buffering turns a rate limit into a memory problem and then into a burst when the workers finish their sleeps in unison.

Config Drift Is a Fingerprint Nobody Designed#

You already know that one worker with a different impersonation target is a problem. At fleet scale it is the default failure mode, because configuration arrives from three places: the image, the environment, and a config service. A worker that started before a config push keeps the old values until it is recycled, so your fleet is a mixture of versions for as long as the rollout takes -- and often longer, because long-lived browser workers are exactly the kind you do not want to recycle often.

Make the version tuple part of every request's telemetry: (browser_target, tls_backend, profile_bundle_version, code_revision). Then you can ask the question that matters -- "did the failures cluster on one version?" -- instead of guessing. A per-version success-rate chart is the cheapest drift detector you will ever build, and it turns "things got worse last Tuesday" into "the chrome131 cohort fell from 0.94 to 0.61 and the chrome124 cohort did not move."

Pin everything. Image tags, dependency locks, profile bundle versions, the TLS backend build. No floating tags in a worker image, ever, because a fleet that resolves its own dependencies at start time is a fleet whose workers do not know what they are. Keeping current covers the rollout discipline; the fleet-specific rule is that a worker advertises its version tuple on every job and the coordinator refuses to schedule a job onto a worker whose tuple is not in the currently approved set.

One Worker's Ban Must Not Become the Fleet's Fate#

Containment is easier to get right at design time than to retrofit, so treat these as four separate mechanisms rather than one "ban handling" feature.

Scope the quarantine to a target, not to the worker. A worker banned by one site is still a perfectly good worker for another. Quarantining the whole worker because of a third party's WAF wastes your capacity and, worse, tempts you to raise concurrency elsewhere to compensate.

Requeue in-flight work off the quarantined identity. The jobs it was holding have already had part of their session state built on a profile the target has flagged. Requeue them against a maturing profile, never against the same one and never against a brand-new one: the warm-up and identity lifecycle material explains why a first-sight identity and a flagged identity are both bad candidates.

Never let the quarantine response increase the fleet's rate somewhere else. This is the classic cascade: worker A gets banned, its jobs move to workers B through C, their rates rise, they get banned too, and you have converted one detection into a fleet-wide event. The aggregate token bucket in Redis is what prevents this, provided the requeue path respects it. If your requeue logic bypasses the limiter, you have built the cascade.

Separate "this identity is flagged" from "this egress is flagged." They have different blast radii and different recovery times, and the difference is worth carrying in your data model. A profile ban means retire the profile; an egress ban means quarantine the subnet, which is a much larger operation and one you will want to see on a dashboard before you do it on a Tuesday night.

The metric to watch is not bans per day. It is the second-order rate: how many identities were retired because a neighbouring identity was retired. That number should be near zero. When it is not, your containment is leaking and the blast radius is larger than the dashboard claims.

What a Fleet Runbook Needs to Say#

Five things, written down before you need them:

1. The lease contract. TTL, renewal cadence, and the exact rule for what a worker does when renewal fails mid-job.
2. The dedupe key format, per job type, with its TTL and where it is stored.
3. The quarantine trigger, stated as a count over a window rather than as a feeling ("three challenges on one profile in ten minutes"), because a threshold you cannot evaluate is not a threshold.
4. The version tuple, and the approved set it must belong to.
5. The manual escape hatch, and who may pull it. A fleet that cannot be paused by a human during an incident will finish the incident on its own schedule, and the schedule will be the target's, not yours.

None of this is exotic distributed-systems material. That is the point: the fingerprint constraints are what make it urgent, but the machinery you need is the same machinery any queue-backed worker pool needs, and you should not be inventing it under pressure.