Anti Bot
Red-Teaming Your Own Defences
Red-Teaming Your Own Defences
You attack your own stack ethically and with written authorisation. Link to defending against scrapers, challenge vendor deep dive, testing stealth pipelines, legal and compliance boundaries.
Scope and Rules of Engagement
Define boundaries: test only what you own, avoid harm to others, get permission in writing.
Reproducing What Matters
Focus on flows relevant to your defences.
Building a Regression Suite
Convert findings to deterministic tests.
Coverage Reporting
Measure tested versus untested areas.
Ethics and Reporting
Report responsibly without creating an attack guide.
Operational Constraints
Respect limits and shared infrastructure.
Continuous Improvement
Close gaps through tracked remediations.
from collections import defaultdict
# Your control set as the ticket tracker describes it, and a synthetic probe
# set you can run against your own origin with written authorisation.
#
# A control is a thing you have deployed. A probe is a synthetic emulation of
# a technique, expressed as data so this report stays reproducible. Nothing
# here reaches the network; it is a coverage ledger, not an attack tool.
#
# scope whose infrastructure the probe runs against. Only
# own-* scopes are authorised; a probe pointed at shared
# third-party infrastructure tests somebody else's
# contract, not yours.
# needs_auth the probe requires a valid credential of your own issuing
# severity 3 = full dataset or account compromise, 2 = partial
# dataset or sustained nuisance, 1 = nuisance only
CONTROLS = [
("C01", "network", "edge token bucket per IP and ASN"),
("C02", "network", "datacenter and hosting ASN deny list"),
("C03", "network", "verified crawler allowlist by reverse DNS"),
("C04", "network", "origin reachable only behind the edge, mTLS to CDN"),
("C05", "transport", "TLS ClientHello allowlist, JA4 pinned"),
("C06", "transport", "ALPN and HTTP/2 frame-order match"),
("C07", "protocol", "header grammar and Sec-Fetch cross-check"),
("C08", "protocol", "cache key separates query from session"),
("C09", "session", "clearance cookie bound to session and IP"),
("C10", "session", "server-side device record joined to cookie"),
("C11", "runtime", "obfuscated JS environment challenge"),
("C12", "runtime", "proof-of-work before the protected route"),
("C13", "runtime", "app attestation required on mobile API"),
("C14", "behaviour", "cadence and navigation-graph model"),
("C15", "behaviour", "dwell-time and journey-coherence model"),
("C16", "account", "per-account daily quota with overage throttle"),
("C17", "account", "breached-credential check at sign-in"),
("C18", "account", "MFA step-up on anomalous sign-in"),
("C19", "legal", "abuse desk with 24-hour response commitment"),
]
PROBES = [
("P01", "network", "datacenter burst", "own-origin", False, 2,
["C01", "C02"]),
("P02", "network", "residential proxy rotation", "own-origin", False, 2,
["C01", "C02"]),
("P03", "network", "direct origin bypass", "own-origin", False, 3,
["C04"]),
("P04", "transport", "JA4 impersonation of a current browser",
"own-origin", False, 1, ["C05", "C07"]),
("P05", "transport", "HTTP/2 frame forgery and pseudo-header order",
"own-origin", False, 2, ["C06", "C07"]),
("P06", "protocol", "header order canonicalised by the client library",
"own-origin", False, 1, ["C07"]),
("P07", "protocol", "range-request id enumeration", "own-origin", False,
2, ["C01", "C08"]),
("P08", "protocol", "shared CDN cache hit harvested without challenge",
"own-cdn-zone", False, 3, []),
("P09", "session", "clearance cookie replayed on a second egress",
"own-origin", False, 3, ["C09"]),
("P10", "session", "cloned device fingerprint across two sessions",
"own-origin", False, 2, ["C10"]),
("P11", "runtime", "headless leak in navigator and CDP artefacts",
"own-origin", False, 1, ["C11"]),
("P12", "runtime", "JavaScript disabled, static fetch", "own-origin",
False, 2, ["C11"]),
("P13", "runtime", "proof-of-work farm, many cores, one clearance",
"own-origin", False, 2, ["C12"]),
("P14", "runtime", "emulated device, no hardware attestation",
"own-origin", False, 3, []),
("P15", "behaviour", "fixed 2.000 s cadence across a whole run",
"own-origin", False, 2, ["C14"]),
("P16", "behaviour", "one long warm session then a flat-rate burst",
"own-origin", True, 2, ["C14", "C15"]),
("P17", "behaviour", "perfect navigation graph, no dead ends",
"own-origin", False, 1, []),
("P18", "account", "credential stuffing across 4,200 addresses",
"own-origin", True, 3, ["C17", "C18"]),
("P19", "account", "burn accounts, harvest, discard, repeat",
"own-origin", True, 3, ["C16"]),
("P20", "account", "single quota exhausted, pool rotated to new signups",
"own-origin", True, 2, []),
("P21", "legal", "sustained abusive crawl, ignored for 40 hours",
"own-origin", False, 2, ["C19"]),
("P22", "vendor", "third-party scraper replayed through a bot-as-a-service",
"vendor-saas", False, 3, []),
("P23", "vendor", "burst from a CDN POP shared with 40,000 tenants",
"third-party-shared", False, 2, []),
]
AUTHORISED = ("own-origin", "own-cdn-zone")
CONTROL_NAMES = dict((c[0], c[2]) for c in CONTROLS)
CONTROL_LAYER = dict((c[0], c[1]) for c in CONTROLS)
SEVERITY = {1: "nuisance", 2: "partial dataset", 3: "full compromise"}
in_scope, withheld = [], []
for p in PROBES:
(in_scope if p[3] in AUTHORISED else withheld).append(p)
def out(line=""):
print(line.rstrip())
out("authorised scope: %s" % ", ".join(AUTHORISED))
out("control inventory: %d controls across %d layers"
% (len(CONTROLS), len(set(c[1] for c in CONTROLS))))
out("probe inventory: %d probes, %d in scope, %d withheld"
% (len(PROBES), len(in_scope), len(withheld)))
out()
out("scope gate: probes withheld before execution")
if withheld:
for p in withheld:
out(" %s %-46s scope=%s" % (p[0], p[2], p[3]))
else:
out(" none")
exercised = set()
gaps = []
out()
out("%-5s %-11s %-46s %-7s %-6s %s"
% ("probe", "layer", "technique", "auth", "result", "catching controls"))
for p in sorted(in_scope):
pid, layer, name, _scope, auth, sev, catches = p
hit = [c for c in catches if c in CONTROL_NAMES]
exercised.update(hit)
verdict = "detected" if hit else "GAP"
if not hit:
gaps.append((pid, layer, name, sev))
out("%-5s %-11s %-46s %-7s %-6s %s"
% (pid, layer, name, "yes" if auth else "no", verdict,
" ".join(hit) or "-"))
untested = [c for c in CONTROLS if c[0] not in exercised]
by_layer = defaultdict(lambda: [0, 0, 0, 0])
for p in in_scope:
pid, layer, _n, _s, _a, sev, catches = p
by_layer[layer][0] += 1
if [c for c in catches if c in CONTROL_NAMES]:
by_layer[layer][1] += 1
else:
by_layer[layer][2] += 1
by_layer[layer][3] += sev
out()
out("coverage by layer")
out("%-12s %7s %9s %5s %11s" % ("layer", "probes", "detected", "gaps",
"risk units"))
total_risk = 0
for layer in sorted(by_layer):
total, det, gap, risk = by_layer[layer]
total_risk += risk
out("%-12s %7d %9d %5d %11d" % (layer, total, det, gap, risk))
out("%-12s %7d %9d %5d %11d"
% ("ALL", len(in_scope),
len(in_scope) - len(gaps), len(gaps), total_risk))
out()
out("coverage gaps, worst first")
for pid, layer, name, sev in sorted(gaps, key=lambda g: (-g[3], g[0])):
out(" %s %-46s %-11s severity %d (%s)"
% (pid, name, layer, sev, SEVERITY[sev]))
out()
out("controls nobody has probed: you think you have coverage here")
if untested:
for cid, layer, name in sorted(untested):
out(" %s %-11s %s" % (cid, layer, name))
else:
out(" none")
gap_layers = defaultdict(int)
for pid, layer, _n, sev in gaps:
gap_layers[layer] += sev
out()
out("next work, ranked by unmitigated severity then untested surface")
work = sorted(gap_layers.items(), key=lambda kv: (-kv[1], kv[0]))
for layer, risk in work:
extra = " (and %d untested control%s here)" % (
len([c for c in untested if c[1] == layer]),
"" if len([c for c in untested if c[1] == layer]) == 1 else "s")
out(" %-11s %d risk unit%s%s" % (layer, risk, "" if risk == 1 else "s",
extra))
out()
out("report discipline: attach the reproduction steps, the observed")
out("response, and the remediation to each gap, then close it as a ticket.")
authorised scope: own-origin, own-cdn-zone
control inventory: 19 controls across 8 layers
probe inventory: 23 probes, 21 in scope, 2 withheld
scope gate: probes withheld before execution
P22 third-party scraper replayed through a bot-as-a-service scope=vendor-saas
P23 burst from a CDN POP shared with 40,000 tenants scope=third-party-shared
probe layer technique auth result catching controls
P01 network datacenter burst no detected C01 C02
P02 network residential proxy rotation no detected C01 C02
P03 network direct origin bypass no detected C04
P04 transport JA4 impersonation of a current browser no detected C05 C07
P05 transport HTTP/2 frame forgery and pseudo-header order no detected C06 C07
P06 protocol header order canonicalised by the client library no detected C07
P07 protocol range-request id enumeration no detected C01 C08
P08 protocol shared CDN cache hit harvested without challenge no GAP -
P09 session clearance cookie replayed on a second egress no detected C09
P10 session cloned device fingerprint across two sessions no detected C10
P11 runtime headless leak in navigator and CDP artefacts no detected C11
P12 runtime JavaScript disabled, static fetch no detected C11
P13 runtime proof-of-work farm, many cores, one clearance no detected C12
P14 runtime emulated device, no hardware attestation no GAP -
P15 behaviour fixed 2.000 s cadence across a whole run no detected C14
P16 behaviour one long warm session then a flat-rate burst yes detected C14 C15
P17 behaviour perfect navigation graph, no dead ends no GAP -
P18 account credential stuffing across 4,200 addresses yes detected C17 C18
P19 account burn accounts, harvest, discard, repeat yes detected C16
P20 account single quota exhausted, pool rotated to new signups yes GAP -
P21 legal sustained abusive crawl, ignored for 40 hours no detected C19
coverage by layer
layer probes detected gaps risk units
account 3 2 1 2
behaviour 3 2 1 1
legal 1 1 0 0
network 3 3 0 0
protocol 3 2 1 3
runtime 4 3 1 3
session 2 2 0 0
transport 2 2 0 0
ALL 21 17 4 9
coverage gaps, worst first
P08 shared CDN cache hit harvested without challenge protocol severity 3 (full compromise)
P14 emulated device, no hardware attestation runtime severity 3 (full compromise)
P20 single quota exhausted, pool rotated to new signups account severity 2 (partial dataset)
P17 perfect navigation graph, no dead ends behaviour severity 1 (nuisance)
controls nobody has probed: you think you have coverage here
C03 network verified crawler allowlist by reverse DNS
C13 runtime app attestation required on mobile API
next work, ranked by unmitigated severity then untested surface
protocol 3 risk units (and 0 untested controls here)
runtime 3 risk units (and 1 untested control here)
account 2 risk units (and 0 untested controls here)
behaviour 1 risk unit (and 0 untested controls here)
report discipline: attach the reproduction steps, the observed
response, and the remediation to each gap, then close it as a ticket.
Lesson 55 of 62
See where this fits: Anti-Bot Evasion Roadmap
open roadmap ->