Anti Bot

Detecting Bans & Graceful Recovery

26 min read · lesson 49 of 62

On this page

The Ban Is a Controllable Event
Detecting a Ban Precisely
The Recovery Sequence
Backoff Algebra
Cooldown and Warm Return
The Cost Model
Prefer Prevention to Recovery
Document the Ban Playbook
The Central Skill Is Naming the Failure
Ban Duration and Why It Is Hard to Guess
Rotating Egress: Honest and Dishonest Use
IP Reputation Recovery Timescales
The Retry-Amplification Bug
When to Stop and Re-evaluate

The Ban Is a Controllable Event#

Getting banned is not a failure state—it is an expected, budgeted condition of operating at scale. The professionals differentiate themselves by detecting bans fast, isolating the affected identity, and restarting cleanly instead of compounding damage.

Detecting a Ban Precisely#

A wholesale 403 is easy. Real bans are subtler: - Soft block: the site returns 200 but serves a CAPTCHA page or an empty shell with no product data. - Honeypot silent block: the page loads but is laced with decoy links, inflated prices, or duplicated listings. - Rate-limit 429: distinct from a ban; treat with backoff, not identity change. - Session poisoning: cookies that were valid flip to "session expired" repeatedly. - Empty results on high-frequency endpoints: a leading indicator that your requests are being null-rounded.

Detect these by validating content, not just status codes: assert the presence of expected DOM landmarks or JSON keys before trusting a 200.

The Recovery Sequence#

When a ban is detected: 1. Freeze the identity (IP + profile). No more requests from that combination; continued hits only escalate the block and can extend its duration. 2. Cool down the IP. Reputation scoring decays over hours; keep the IP idle for a cooldown period before any successor traffic on the same subnet. 3. Requeue the job for a fresh profile/IP, with an exponential backoff on the initial delay. 4. Drain the profile. Mark affected profile as "cooling", let legacy cookies expire naturally, and stop reusing its clearance tokens.

Backoff Algebra#

The classic pattern is exponential backoff with full jitter: for retry n, sleep random(0, min(cap, base * 2^n)). With a background of distributed-queue leases, the targeted cap keeps recovery request rates from peaking when many workers recover simultaneously.

import random
import time

def backoff(n, base=5.0, cap=300.0):
    return random.uniform(0, min(cap, base * 2 ** n))

for attempt in range(6):
    ok = try_request()
    if ok:
        break
    time.sleep(backoff(attempt))

Cooldown and Warm Return#

IPs and profiles do not recover instantly. Re-introduce a cooled identity via low-risk traffic: a homepage visit, one detail page, a slow crawl at a tenth of the normal rate. If the recovered identity is re-rejected within the first minute, the block has not lifted—cool it again. Going hot too fast after a ban is how "cooled" operations get permanently blacklisted.

The Cost Model#

Every ban has a price: the IP, the profile, the CLV of that identity, and the retry cost. Model bans as a line item, tune your request rate and proxy mix to keep the ban rate under a budgetary threshold, and stop optimizing for raw speed when diminishing returns push ban rates through the roof. The fleet that survives for months is the one that treats burn rate as a first-class metric.

Prefer Prevention to Recovery#

The cheapest ban is the one that never happens. Because reputation decays and accounts age, prevention beats recovery by a wide margin: run below the tolerance threshold, warm profiles before high-volume jobs, and retire a profile at the first sign of soft-blocking rather than waiting for a hard ban. Recovery is the fallback when prevention fails, not the primary strategy.

Document the Ban Playbook#

Write down, per target, what a ban looks like, which recovery step works, and what triggered it. Targets differ: one responds to cooldowns, another to new profiles, another to a rate drop. Encoding that institutional knowledge as an automated playbook means an on-call engineer does not have to rediscover it during an incident at 3 a.m.

The Central Skill Is Naming the Failure#

Everything in this lesson hangs off one discrimination: given a response, decide whether you have been rate-limited, challenged, banned, poisoned, or have simply written a bug. The four recovery responses are incompatible. Rate-limiting says slow down and keep the identity. A ban says stop, and rotating egress immediately extends the ban to the new address. A challenge failure says your stack is wrong, not that you were blocked. And your own parser bug looks, from the outside, exactly like a soft block, because both produce a 200 with no usable data.

Get the taxonomy concrete and the discriminator becomes mechanical:

Verdict	Distinguishing evidence	What it is not	First response
Rate limit	429, or `Retry-After`, or a `200` with an explicit quota field	a ban; the identity is untouched	honour the advertised interval
Challenge failure	200 with an interstitial body, or a vendor cookie that never gets set	a ban; you were asked to prove something	fix the solve path, keep the profile
Poisoned	200 with the right shape and implausible values	a block; you are being served data	validate values, never trust the row
Soft block	200 with the right shape and no rows	a rate limit; nobody told you to stop	retire the profile, do not retry it
Hard ban	403/451, or a repeated refusal across a fresh profile on the same egress	your bug; it survives an identity change	freeze both identity and egress
Your bug	5xx, a parser exception, a row count of zero where zero is impossible	anything above	do not touch the identity

The row that costs people the most is the last one. A 500 from the target, a selector that stopped matching because the site shipped a redesign, or a paginated endpoint whose page-two marker you missed will each look like "the site is refusing me" if you only look at the volume of extracted rows. The tell is that it is deterministic: the same selector fails on every page, on every profile, from every egress. A real block is probabilistic and identity-linked; a parser bug is absolute.

Build the classifier as code, not as a runbook paragraph, because the runbook is what you reach for at 3 a.m. and the thing you need at 3 a.m. is a single function returning a label per response. The trace below is one identity degrading over 45 seconds through four distinct failure modes, and the counts make the point that a status-code-only monitor sees two of them: the 200 empty_shell rows are a soft block wearing a success code, and the honeypot rows are a decoy wearing a product-page code.

import random

One identity, one session, degrading over 45 s. body_class is what the page actually contained, which is the only signal separating a soft block from a success; the status code alone hides two of the four failure modes. OBSERVED = [ (0, 200, "product_page", {}), (4, 200, "product_page", {}), (9, 429, "retry_after", {"retry-after": "30"}), (11, 429, "retry_after", {"retry-after": "30"}), (15, 200, "challenge_interstitial", {"cf-mitigated": "challenge"}), (18, 200, "product_page", {}), (22, 200, "honeypot_listing", {}), (26, 200, "honeypot_listing", {}), (31, 403, "hard_block", {}), (34, 403, "hard_block", {}), (38, 200, "empty_shell", {}), (41, 200, "empty_shell", {}), (45, 500, "traceback", {}), ]

EPECTED = "product_page"

verdict -> (severity, identity_damage, what it actually means) FINGERPRINTS = { "ok": (0, False, "the data we asked for"), "rate_limit": (1, False, "slow down; the identity is fine"), "challenge": (2, True, "prove it is a browser; not a ban"), "poisoned": (3, True, "served a decoy on purpose"), "soft_block": (4, True, "200 with no data; the identity is being null-rounded"), "ban": (5, True, "identity is refused outright"), "our_bug": (6, False, "a 5xx, or our parser asking the wrong question"), }

def classify(status, body_class, headers): if status == 429 or "retry-after" in headers: return "rate_limit" if status >= 500: return "our_bug" if status == 403: return "ban" if body_class == EPECTED: return "ok" if "challenge" in body_class: return "challenge" if "honeypot" in body_class: return "poisoned" return "soft_block"

def honouring_client(first_429, retry_after, max_requests=40): '''A client that obeys Retry-After emits nothing inside the window.''' sent, t = 0, first_429 for _ in range(max_requests): t += retry_after if t < first_429 + retry_after: sent += 1 return sent

def jitter_client(first_429, retry_after, base=0.5, max_requests=40): '''The bug: exponential backoff on a local counter, ignoring Retry-After.''' rng = random.Random(4242) sent, t, attempt = 0, first_429, 0 while sent < max_requests: attempt += 1 t += base * (2 ** min(attempt, 8)) sent += 1 if t >= first_429 + retry_after: break return sent, int(t - first_429)

def ladder(): return [ ("0", "hold identity, obey Retry-After", 30, "429"), ("1", "freeze identity, halve global rate", 300, "one challenge, no 403"), ("2", "retire profile, keep the egress", 1800, "decoy or empty shell"), ("3", "retire profile and egress subnet", 7200, "explicit 403"), ("4", "pause the target, page a human", 86400, "ban recurring inside 24h"), ]

def main(): rows, counts, streak = [], {}, 0 for t, status, body, headers in OBSERVED: verdict = classify(status, body, headers) counts[verdict] = counts.get(verdict, 0) + 1 if FINGERPRINTS[verdict][1]: streak += 1 wait = 0 if verdict in ("ok", "our_bug") else int( random.Random(1000 + streak).uniform(0.0, min(1800.0, 2.0 * (2 ** streak)))) rows.append((t, status, body, verdict, wait))

print("one identity, {} responses over {} s, {} identity-damaging".format(
    len(OBSERVED), OBSERVED[-1][0], streak))
print()
print("{:>3}  {:>4}  {:<20} {:<11} {:>6}".format("t", "code", "body", "verdict", "wait"))
print("-" * 56)
for t, status, body, verdict, wait in rows:
    print("{:>3}  {:>4}  {:<20} {:<11} {:>6}".format(t, status, body, verdict, wait))
print()

print("{:<11} {:>5}  {}".format("VERDICT", "COUNT", "MEANING"))
print("-" * 78)
for verdict in sorted(counts, key=lambda v: FINGERPRINTS[v][0]):
    print("{:<11} {:>5}  {}".format(verdict, counts[verdict], FINGERPRINTS[verdict][2]))
print()

# The retry-amplification bug, measured rather than asserted.
first_429 = OBSERVED[2][0]
served = int(OBSERVED[2][3]["retry-after"])
honest = honouring_client(first_429, served)
sent, elapsed = jitter_client(first_429, served)
print("server advertised Retry-After={}s at t={}s".format(served, first_429))
print("requests inside that window, obeying client:   {}".format(honest))
print("requests inside that window, local-counter:    {}".format(sent))
print("elapsed before the local-counter client gave up: {}s".format(elapsed))
print("extra strikes created inside one cooldown:     {}".format(sent - honest))
print()

print("{:<4} {:<36} {:>6}  {}".format("RUNG", "ACTION", "WAIT", "TRIGGER"))
print("-" * 92)
for rung, action, wait, trigger in ladder():
    print("{:<4} {:<36} {:>5}s  {}".format(rung, action, wait, trigger))

main()


```text
one identity, 13 responses over 45 s, 7 identity-damaging

  t  code  body                 verdict       wait
--------------------------------------------------------
  0   200  product_page         ok               0
  4   200  product_page         ok               0
  9   429  retry_after          rate_limit       1
 11   429  retry_after          rate_limit       1
 15   200  challenge_interstitial challenge        3
 18   200  product_page         ok               0
 22   200  honeypot_listing     poisoned         4
 26   200  honeypot_listing     poisoned         7
 31   403  hard_block           ban             13
 34   403  hard_block           ban             30
 38   200  empty_shell          soft_block      44
 41   200  empty_shell          soft_block     219
 45   500  traceback            our_bug          0

VERDICT     COUNT  MEANING
------------------------------------------------------------------------------
ok              3  the data we asked for
rate_limit      2  slow down; the identity is fine
challenge       1  prove it is a browser; not a ban
poisoned        2  served a decoy on purpose
soft_block      2  200 with no data; the identity is being null-rounded
ban             2  identity is refused outright
our_bug         1  a 5xx, or our parser asking the wrong question

server advertised Retry-After=30s at t=9s
requests inside that window, obeying client:   0
requests inside that window, local-counter:    5
elapsed before the local-counter client gave up: 31s
extra strikes created inside one cooldown:     5

RUNG ACTION                                 WAIT  TRIGGER
--------------------------------------------------------------------------------------------
0    hold identity, obey Retry-After         30s  429
1    freeze identity, halve global rate     300s  one challenge, no 403
2    retire profile, keep the egress       1800s  decoy or empty shell
3    retire profile and egress subnet      7200s  explicit 403
4    pause the target, page a human       86400s  ban recurring inside 24h

import random

EPECTED = "product_page"

print("one identity, {} responses over {} s, {} identity-damaging".format(
    len(OBSERVED), OBSERVED[-1][0], streak))
print()
print("{:>3}  {:>4}  {:<20} {:<11} {:>6}".format("t", "code", "body", "verdict", "wait"))
print("-" * 56)
for t, status, body, verdict, wait in rows:
    print("{:>3}  {:>4}  {:<20} {:<11} {:>6}".format(t, status, body, verdict, wait))
print()

print("{:<11} {:>5}  {}".format("VERDICT", "COUNT", "MEANING"))
print("-" * 78)
for verdict in sorted(counts, key=lambda v: FINGERPRINTS[v][0]):
    print("{:<11} {:>5}  {}".format(verdict, counts[verdict], FINGERPRINTS[verdict][2]))
print()

# The retry-amplification bug, measured rather than asserted.
first_429 = OBSERVED[2][0]
served = int(OBSERVED[2][3]["retry-after"])
honest = honouring_client(first_429, served)
sent, elapsed = jitter_client(first_429, served)
print("server advertised Retry-After={}s at t={}s".format(served, first_429))
print("requests inside that window, obeying client:   {}".format(honest))
print("requests inside that window, local-counter:    {}".format(sent))
print("elapsed before the local-counter client gave up: {}s".format(elapsed))
print("extra strikes created inside one cooldown:     {}".format(sent - honest))
print()

print("{:<4} {:<36} {:>6}  {}".format("RUNG", "ACTION", "WAIT", "TRIGGER"))
print("-" * 92)
for rung, action, wait, trigger in ladder():
    print("{:<4} {:<36} {:>5}s  {}".format(rung, action, wait, trigger))

main()


```text
one identity, 13 responses over 45 s, 7 identity-damaging

  t  code  body                 verdict       wait
--------------------------------------------------------
  0   200  product_page         ok               0
  4   200  product_page         ok               0
  9   429  retry_after          rate_limit       1
 11   429  retry_after          rate_limit       1
 15   200  challenge_interstitial challenge        3
 18   200  product_page         ok               0
 22   200  honeypot_listing     poisoned         4
 26   200  honeypot_listing     poisoned         7
 31   403  hard_block           ban             13
 34   403  hard_block           ban             30
 38   200  empty_shell          soft_block      44
 41   200  empty_shell          soft_block     219
 45   500  traceback            our_bug          0

VERDICT     COUNT  MEANING
------------------------------------------------------------------------------
ok              3  the data we asked for
rate_limit      2  slow down; the identity is fine
challenge       1  prove it is a browser; not a ban
poisoned        2  served a decoy on purpose
soft_block      2  200 with no data; the identity is being null-rounded
ban             2  identity is refused outright
our_bug         1  a 5xx, or our parser asking the wrong question

server advertised Retry-After=30s at t=9s
requests inside that window, obeying client:   0
requests inside that window, local-counter:    5
elapsed before the local-counter client gave up: 31s
extra strikes created inside one cooldown:     5

RUNG ACTION                                 WAIT  TRIGGER
--------------------------------------------------------------------------------------------
0    hold identity, obey Retry-After         30s  429
1    freeze identity, halve global rate     300s  one challenge, no 403
2    retire profile, keep the egress       1800s  decoy or empty shell
3    retire profile and egress subnet      7200s  explicit 403
4    pause the target, page a human       86400s  ban recurring inside 24h

The escalation ladder at the bottom of that output is the part to encode. Rungs are ordered by how much you give up, not by how bad the response was: at rung 0 you still hold the identity, and by rung 3 you have burned both a profile and a subnet. The mistake teams make is skipping rungs. A single 403 on a target you have never seen before does not justify discarding a two-week-old profile and a residential exit you paid a month for. A third 403 on the same egress inside an hour does. Make the trigger a count over a window, so the decision is a function you can evaluate rather than a judgement call.

Ban Duration and Why It Is Hard to Guess#

Ban durations are not published, and the reason is that a published duration is an invitation to schedule a retry at exactly the moment the defender wants you to. What you can observe instead is a distribution, and it is not the distribution people assume:

Explicit refusals with a stated duration (Retry-After, a "try again in 10 minutes" page) are exact and trustworthy. Use them.
Silent soft blocks have no stated duration and frequently clear on the target's own schedule rather than yours. Empirically these sit in the tens of minutes, and they often clear while you are still backing off.
Explicit hard bans are the ones that matter. The distribution has a long tail: some clear in an hour, some are effectively permanent for a residential address because the address was already on a list before you used it. That last case is not a ban you recover from; it is an exit you should have screened for.
Subnet and ASN-level blocks are much longer than address-level ones and are usually not about you at all. If your egress is a mobile carrier CGNAT range, one subscriber's behaviour can put the whole /24 in a bad state for days.

The practical consequence is that you should not schedule your own recovery attempt time. You should schedule a probe: a single low-cost, low-risk request at increasing intervals, and a decision rule for what each probe result means. Probe at 15 minutes, then 45, then 2 hours, then 6, then 24. A probe that returns a challenge means you are past the ban -- solve and return. A probe that returns a soft block means you are not, and the interval doubles. Crucially, a probe that returns 429 means you have found the site's actual rate ceiling, which is more useful than the ban was: write it down.

Rotating Egress: Honest and Dishonest Use#

There are two reasons to change your egress address, and they have opposite characters.

The honest reason is scheduling. Your current exit is wrong for the work: wrong region, congested, on a list you did not know about, or simply mispriced for the volume you are running. You rotate because the next job is a better fit for that address. This is ordinary capacity planning, it does not depend on having been blocked, and it is available whether or not you were banned.

The dishonest reason is shedding a block. You were refused, so you obtain a new address and continue. This is where the operational damage is, and the mechanism is not obvious. Ban systems at the serious end of the market are not keyed on IP alone. The scoring that matters is the tuple of address plus account plus fingerprint plus session history, and the address is the cheapest thing for you to change and the most expensive thing for them to lose. When you rotate immediately after a refusal, you convert one scoped decision into a pattern: the same fingerprint, the same session cookie, the same request shape, arriving from an address that has never contacted this site before. A brand-new address used by an established profile is less plausible than an established address used for a fresh profile, and it is a signature you can only avoid by not doing it.

There is a third pattern that is worse than both: rotate the address but keep the profile, on the theory that the profile is the valuable asset. Now the profile has a history at three addresses in an hour and none of them agree, and the risk ledger the target keeps for it is a record of incoherence. If you must change something after a ban, change the pair -- profile and address together, and accept that you are starting a new identity rather than recovering an old one -- or change nothing and wait.

IP Reputation Recovery Timescales#

Reputation is not a boolean and it is not a timer you can look up. Treat it as a slow-moving aggregate with roughly these timescales, which are worth having in a comment in your own config because they justify why your cooldowns are as long as they are:

Signal	Typical decay	Practical consequence
Request-rate contribution to a score	minutes to an hour	halving your rate helps within the hour
Challenge failure contribution	hours	a failed challenge is worse than a high rate
Address-level abuse flags	days to weeks	do not wait this out; screen new exits before use
Behavioural history bound to a fingerprint	weeks	the profile is not recoverable, retire it
Account-level history	months	the account is the asset, not the address

The row that surprises people is the second one. A high request rate from a clean profile is a throttle problem and decays in minutes. A failed challenge is a statement about the client, and it stays on the record for hours. So a burst that happens to include five challenge failures leaves you in a worse position than a burst of two hundred clean 200s, and the recovery is much longer. This is why challenge solves belong on the low-value path and bulk fetching does not.

The fifth row is the argument for separating identities by construction. If your address and your profile history are entangled, a behavioural flag on the profile cannot be recovered while the profile is in use. If you can retire the profile and keep a warm address, you recover the address and rebuild the profile; if they are welded together, you rebuild both and start from zero history.

The Retry-Amplification Bug#

This is the single most common way a scraper turns a rate limit into a ban, and it is a bug rather than a decision. The shape of it:

RETRY_AFTER = 30 WORKERS = ["w-a", "w-b", "w-c"] FRESH_WORKER = "w-d" # a new pod, replacing one that died SCRIPT = [("j-1", "w-a"), ("j-2", "w-b"), ("j-3", "w-c")]

class LocalCounterBackoff: '''Retry state lives on the stack, keyed by process. Every worker that hits the same 429 independently decides to retry almost immediately.'''

def __init__(self):
    self.attempt = {w: 0 for w in WORKERS + [FRESH_WORKER]}
    self.next_at = {w: 0 for w in WORKERS + [FRESH_WORKER]}

def fire_at(self, worker):
    self.attempt[worker] += 1
    self.next_at[worker] = 1.0 * (2 ** min(self.attempt[worker], 6))
    return self.next_at[worker]

def may_send(self, worker):
    return 0.0 >= self.next_at[worker]

class IdentityBackoff: '''Retry state lives beside the egress, so every worker and every process waits out one shared cooldown and a replacement pod inherits it.'''

def __init__(self):
    self.cooldown_until = 0.0
    self.strikes_avoided = 0

def fire_at(self, worker):
    self.cooldown_until = RETRY_AFTER
    return self.cooldown_until

def may_send(self, worker):
    return 0.0 >= self.cooldown_until

def drive(policy): log, inside = [], 0 for job, worker in SCRIPT: at = policy.fire_at(worker) inside += 1 if at < RETRY_AFTER else 0 log.append((job, worker, int(at))) escaped = policy.may_send(FRESH_WORKER) log.append(("j-1", FRESH_WORKER + " (requeue)", int(policy.fire_at(FRESH_WORKER)))) return log, inside, escaped

def main(): print("one residential exit, three workers, one 429 with Retry-After={}s".format(RETRY_AFTER)) print("all three workers share the exit, so the cooldown belongs to the exit") print() for name, factory in [("backoff keyed to the process", LocalCounterBackoff), ("backoff keyed to the egress", IdentityBackoff)]: policy = factory() log, inside, escaped = drive(policy) print("=== {} ===".format(name)) for job, worker, at in log: print(" {:<6} on {:<22} retries at t={:>2}s".format(job, worker, at)) print(" retries inside the server's window: {}".format(inside)) print(" replacement pod escapes the cooldown: {}".format(escaped)) print()

main()


```text
one residential exit, three workers, one 429 with Retry-After=30s
all three workers share the exit, so the cooldown belongs to the exit

=== backoff keyed to the process ===
  j-1    on w-a                    retries at t= 2s
  j-2    on w-b                    retries at t= 2s
  j-3    on w-c                    retries at t= 2s
  j-1    on w-d (requeue)          retries at t= 2s
  retries inside the server's window:   3
  replacement pod escapes the cooldown: True

=== backoff keyed to the egress ===
  j-1    on w-a                    retries at t=30s
  j-2    on w-b                    retries at t=30s
  j-3    on w-c                    retries at t=30s
  j-1    on w-d (requeue)          retries at t=30s
  retries inside the server's window:   0
  replacement pod escapes the cooldown: False

RETRY_AFTER = 30 WORKERS = ["w-a", "w-b", "w-c"] FRESH_WORKER = "w-d" # a new pod, replacing one that died SCRIPT = [("j-1", "w-a"), ("j-2", "w-b"), ("j-3", "w-c")]

class LocalCounterBackoff: '''Retry state lives on the stack, keyed by process. Every worker that hits the same 429 independently decides to retry almost immediately.'''

def __init__(self):
    self.attempt = {w: 0 for w in WORKERS + [FRESH_WORKER]}
    self.next_at = {w: 0 for w in WORKERS + [FRESH_WORKER]}

def fire_at(self, worker):
    self.attempt[worker] += 1
    self.next_at[worker] = 1.0 * (2 ** min(self.attempt[worker], 6))
    return self.next_at[worker]

def may_send(self, worker):
    return 0.0 >= self.next_at[worker]

class IdentityBackoff: '''Retry state lives beside the egress, so every worker and every process waits out one shared cooldown and a replacement pod inherits it.'''

def __init__(self):
    self.cooldown_until = 0.0
    self.strikes_avoided = 0

def fire_at(self, worker):
    self.cooldown_until = RETRY_AFTER
    return self.cooldown_until

def may_send(self, worker):
    return 0.0 >= self.cooldown_until

main()


```text
one residential exit, three workers, one 429 with Retry-After=30s
all three workers share the exit, so the cooldown belongs to the exit

=== backoff keyed to the process ===
  j-1    on w-a                    retries at t= 2s
  j-2    on w-b                    retries at t= 2s
  j-3    on w-c                    retries at t= 2s
  j-1    on w-d (requeue)          retries at t= 2s
  retries inside the server's window:   3
  replacement pod escapes the cooldown: True

=== backoff keyed to the egress ===
  j-1    on w-a                    retries at t=30s
  j-2    on w-b                    retries at t=30s
  j-3    on w-c                    retries at t=30s
  j-1    on w-d (requeue)          retries at t=30s
  retries inside the server's window:   0
  replacement pod escapes the cooldown: False

REMOVED_STRAY2

Three rules, all mechanical:

The server's Retry-After overrides your backoff, always. Not "the larger of the two" -- the server's. If it is longer, wait longer. If it is shorter, still wait, because a shorter-than-your-backoff server hint is not permission to hammer.
Backoff state is per identity, not per process. Backoff stored in a worker's memory is lost when the job is requeued to a different worker, and the retry arrives immediately on a fresh process with a zeroed counter. Store next-allowed-at alongside the identity, not on the stack.
Jitter must not be able to exceed the server's window. Full-jitter backoff on your own counter is correct for unknown-failure retries and wrong here. Jitter is for spreading your own load, not for negotiating with the server's timer.

A fleet makes a fourth version of this bug possible: fifty workers all retrying after their own independent counters expire at roughly the same moment, producing a synchronised burst that is both larger and more structured than any single worker's would have been. This is the case for a shared, identity-keyed "next allowed at" in Redis, checked at dequeue time as well as inside the worker.

When to Stop and Re-evaluate#

A ban response is information about the economics of what you are doing, and there is a point where continuing costs more than the data is worth. Stop and re-evaluate when any of these hold:

The same target has ended your work three times in a fortnight. Three independent recoveries means the site has a durable memory of you, and each recovery you performed taught it something about your recovery pattern.
The ban is on the account, not the address. If your authenticated session is refused, the identity you care about is at risk, and no amount of infrastructure work fixes it.
The data is available through a licensed channel at a price that is now below your cost per successful fetch. This is a real number and you should compute it rather than assert it; the framework for doing that is in official APIs and alternative data.
Your retention is the problem. If you keep getting caught because your profile history is thin, the honest answer is a slower, older, smaller fleet, which is a design decision and not a technical obstacle.

Two specific signals that you are past ordinary tuning: if your recovery requires more than one identity rotation to succeed, your profile handling is the problem; and if the site challenges you on a request rate well below what you believe is its ceiling, your shape is the problem rather than your volume. In both cases the fix is slower and fewer, not faster and cleverer.

The uncomfortable but useful framing is that a ban you cannot recover from cheaply is information the target gave you for free. Treating it as a puzzle to solve converts a signal into a tax on your time. Treating it as a measurement -- this target, this egress class, this rate -- converts it into a parameter you can plan around, which is the same discipline you would apply to any other constrained resource.

by Muhammad Usman · updated Oct 04, 2026

Lesson 49 of 62

See where this fits: Anti-Bot Evasion Roadmap open roadmap ->