IP Reputation & Proxies

Your Digital Disguise#

Your IP address is the foundational layer of your online identity. If a website's anti-bot system flags your IP as malicious or highly suspicious, no amount of advanced browser fingerprint spoofing will save you.

When scraping at scale, you must distribute your traffic across thousands of different IP addresses to avoid rate limits and bans. This is where Proxies come in.

1. Datacenter Proxies#

These are IPs owned by massive cloud providers like AWS, Google Cloud, Linode, and DigitalOcean. - The Good: Extremely fast (10Gbps+ links), cheap, and stable. - The Bad: Anti-bot systems maintain static lists of all Datacenter IP subnets (ASN databases). If a request comes from an AWS IP, the WAF instantly knows it is a server, not a real human at home. - Use Case: Scraping unprotected, simple websites, or internal B2B APIs that don't employ aggressive bot mitigation.

2. Residential Proxies#

These are IP addresses assigned by local Internet Service Providers (ISPs like Comcast, AT&T, Vodafone) to actual, physical homeowners. Proxy providers acquire these by paying users to install background apps that share their bandwidth. - The Good: Highly trusted. When you scrape using a residential proxy, the WAF sees the traffic coming from a standard residential neighborhood. Blocking residential IPs risks blocking legitimate customers, so WAFs are extremely hesitant to do so. - The Bad: Expensive (billed by the Gigabyte, not by the IP), slower, and connection drops are common if the homeowner turns off their router. - Use Case: E-commerce scraping, sneaker botting, and bypassing moderate-to-strict Cloudflare/Datadome protections.

3. Mobile Proxies#

The gold standard of stealth. These are IPs assigned by cellular carriers (Verizon, T-Mobile) to 4G/5G mobile devices. - The Good: Complete immunity to IP bans. Because cellular networks use CGNAT (Carrier-Grade NAT), thousands of real mobile phones share the exact same public IP address simultaneously. If a WAF bans a single mobile IP, they might accidentally ban 5,000 legitimate users on the same cell tower. Thus, Mobile IPs almost always get a free pass. - The Bad: Exorbitantly expensive and generally much slower than broadband. - Use Case: Instagram/TikTok automation, high-security financial scraping.

How Reputation Gets Built#

Reputation is not static. Every request you (or anyone else on the same IP) make feeds a behavioral history: how many distinct sites has this IP touched today, how fast did it hammer associated domains, was it ever seen succeeding at a CAPTCHA, and how old is the IP lease. Freshly allocated ranges from cheap VPS vendors carry an instant penalty, while subnets that have been assigned to the same customer for years score far better. When you choose a proxy pool, you are effectively renting this aggregated history.

ASN and Subnet Intelligence#

Reputation operates at three granularities: the exact IP, the /24 subnet, and the owning ASN (Autonomous System Number). WAFs map every ASN to a category—residential ISP, mobile carrier, cloud provider, hosting company, or anonymizer. A single bad ray in an otherwise clean residential ASN will not poison the whole ASN, but a datacenter ASN is poisoned from the start. This is why "which ASN owns my proxy" matters more than any single IP address.

IPv6 Considerations#

Most consumer proxy pools are IPv4 because IPv4 addresses are scarce and valuable. IPv6, where a single residential connection can hold a /64 subnet of trillions of addresses, is much harder to ban precisely but much easier to block wholesale. Many WAFs simply penalize any IPv6 traffic that does not look like a dual-stack residential connection, so treat IPv6 proxies as a separate reputation pool rather than a free source of endless addresses.

Proxy Rotation Strategies#

When buying a proxy pool, you typically route your code through a single "Gateway" endpoint provided by the proxy vendor. - Per-Request Rotation: Every single HTTP request you make is routed to a brand new IP. Best for stateless scraping (like downloading 1,000 distinct product pages). - Sticky Sessions: You maintain the same IP address for 5 to 15 minutes. Best for flows that require logging in, solving a CAPTCHA, and maintaining a session cookie. If your IP changes mid-session, the WAF will immediately flag the session as hijacked.

Reputation Is Built, Not Looked Up#

Nobody maintains a table of "bad addresses". They maintain four aggregates and join them at request time, and the join is what you are trying to influence. The four:

• Static list membership. Published cloud ranges, provider proxy egress lists, Tor exit nodes. A set lookup, no scoring, and the only one of the four you can enumerate yourself. See ASN and datacenter detection for how these lists are published and how stale they are.
• ASN category. Every address resolves to an autonomous system, every ASN is classified as access, mobile, cloud, hosting, business, or anonymiser. This is the strongest single term, because the classification changes slowly and is expensive to spoof.
• Address-level behavioural history. Requests per minute, distinct hosts per day, challenge outcomes, whether this address ever solved a CAPTCHA. Recomputed on a rolling window.
• Lease and tenure. How long the prefix has been allocated to this customer, and whether the address is dynamically assigned. A freshly announced cloud range is a penalty; a residential range in the same ASN is not.

Those four are not equally attackable. You cannot change your ASN without changing your operator. You can change your address within a pool. You can influence your request rate. You cannot change the ASN category of the network you are paying for, which is why datacenter egress is a dead end and every commercial tool on the market is built around residential and mobile supply.

Collateral Decides Block Granularity#

Here is the calculation that explains why blocklists look the way they do. A defender choosing between blocking a prefix and blocking an address has to weigh the customer cost of the block. For cloud and hosting ranges that cost is zero, so those ranges get blocked wholesale at the ASN. For access networks it is enormous, so those ranges get blocked only at fine granularity, and usually only when a specific behaviour is confirmed.

# Reputation is a number somebody chose, not a lookup.
# Worth computing for yourself: the collateral cost of a prefix block,
# which is what decides whether a vendor blocks granularly or wholesale.

import ipaddress

# asn -> (holder, class, broadband subscribers served)
ASN = {
    13335: ("CLOUDFLARENET", "cloud", 0),
    14061: ("LINODE", "cloud", 0),
    15169: ("GOOGLE", "cloud", 0),
    16509: ("EC2", "cloud", 0),
    3320: ("DEUTSCHE-TELEKOM", "access", 39_000_000),
    680: ("VODAFONE", "access", 17_000_000),
    7018: ("ATT-INTERNET4", "access", 30_000_000),
    7922: ("COMCAST-CABLE", "access", 32_000_000),
    2001: ("HEANET", "hosting", 0),
    9009: ("M247", "hosting", 0),
    10416: ("TIGER", "hosting", 0),
    6167: ("CELLCO-PART", "mobile", 145_000_000),
}

V6_PER_CUSTOMER = 2 ** 64          # a residential customer holds a whole /64

FEATURES = [
    ("on_abuse_feed", 0.55),
    ("asn_class_cloud", 0.60),
    ("asn_class_hosting", 0.50),
    ("asn_class_access", -0.25),
    ("asn_class_mobile", -0.30),
    ("lease_age_lt_30d", 0.25),
    ("egress_tenants_gt_40", 0.45),
    ("req_rate_10min_gt_30", 0.35),
    ("hosts_per_day_gt_500", 0.20),
    ("solved_captcha_before", -0.15),
]


def risk(flags):
    total = sum(w for name, w in FEATURES if flags.get(name))
    return max(0.0, min(1.0, total))


def bucket(value):
    for cut, label in [(0.10, "clean"), (0.30, "watched"), (0.60, "suspect"),
                       (1.01, "prebanned")]:
        if value < cut:
            return label
    return "prebanned"


def households(net, subscribers):
    # Concurrent subscribers a block on this prefix takes offline.
    if subscribers == 0:
        return 0, 0.0
    per = V6_PER_CUSTOMER if net.version == 6 else 1
    people = net.num_addresses / float(per)
    return int(people), 100.0 * people / subscribers


def main():
    print("collateral of a prefix block: one row per published range")
    print(f"{'prefix':<21} {'asn':<16} {'class':<8} {'subscribers':>12} {'of base':>8}")
    rows = [
        ("151.101.0.0/22", 13335), ("104.16.0.0/13", 13335),
        ("140.82.112.0/20", 16509), ("5.9.0.0/16", 2001),
        ("71.0.0.0/12", 7018), ("203.0.113.0/24", 7922),
        ("2001:610:240::/42", 3320), ("2a02:c9f0:4000::/44", 680),
        ("2620:0:860::/44", 15169),
    ]
    for text, asn in rows:
        net = ipaddress.ip_network(text)
        holder, klass, subs = ASN[asn]
        people, share = households(net, subs)
        print(f"{text:<21} {holder:<16} {klass:<8} {people:>12,} "
              f"{(f'{share:6.2f}%' if subs else '    n/a')}")
    print()

    print("so the v6 blocklist is blunt, and the v4 one is a scalpel")
    for text, asn in [("203.0.113.0/24", 7922), ("2001:610:240::/42", 3320)]:
        net = ipaddress.ip_network(text)
        people, share = households(net, ASN[asn][2])
        unit = "one public address at a time" if net.version == 4 else "one /64 per household"
        print(f"  {text:<21} {net.num_addresses:>26,} addresses, {unit}")
        print(f"  {'':<21} {people:>26,} concurrent subscribers, {share:.2f}% of the base")
    print()

    print("what a residential exit is worth, scored the way a risk engine scores it")
    print(f"{'case':<34} {'risk':>5}  {'bucket':<10} driven by")
    cases = [
        ("leased line, 3yr tenant", {"asn_class_access": True}),
        ("leased line, 3yr tenant", {"asn_class_access": True,
                                     "solved_captcha_before": True}),
        ("residential, 11 co-tenants", {"asn_class_access": True,
                                        "egress_tenants_gt_40": True}),
        ("mobile CGNAT exit", {"asn_class_mobile": True}),
        ("mobile CGNAT exit", {"asn_class_mobile": True,
                               "req_rate_10min_gt_30": True}),
        ("residential, fresh lease", {"asn_class_access": True,
                                      "lease_age_lt_30d": True,
                                      "hosts_per_day_gt_500": True}),
        ("cloud egress", {"asn_class_cloud": True, "lease_age_lt_30d": True}),
        ("hosting reseller", {"asn_class_hosting": True,
                              "on_abuse_feed": True}),
    ]
    for label, flags in cases:
        adverse = [n for n, _w in FEATURES
                   if flags.get(n) and not n.startswith("asn_class")]
        credit = [n for n, _w in FEATURES
                  if flags.get(n) and n.startswith("asn_class")]
        driven = ", ".join(adverse + credit)
        print(f"{label:<34} {risk(flags):0.2f}  {bucket(risk(flags)):<10} {driven}")
    print()

    print("the cgnat ceiling: one public address, n subscribers, one shared verdict")
    for label, n in [("cell sector, busy hour", 4096), ("cell sector, 03:00", 610),
                     ("business fibre behind nat", 64),
                     ("free wifi hotel chain", 3000)]:
        print(f"  {label:<28} {n:>5,} subscribers share one address; "
              f"one ban costs all of them")


main()

collateral of a prefix block: one row per published range
prefix                asn              class     subscribers  of base
151.101.0.0/22        CLOUDFLARENET    cloud               0     n/a
104.16.0.0/13         CLOUDFLARENET    cloud               0     n/a
140.82.112.0/20       EC2              cloud               0     n/a
5.9.0.0/16            HEANET           hosting             0     n/a
71.0.0.0/12           ATT-INTERNET4    access      1,048,576   3.50%
203.0.113.0/24        COMCAST-CABLE    access            256   0.00%
2001:610:240::/42     DEUTSCHE-TELEKOM access      4,194,304  10.75%
2a02:c9f0:4000::/44   VODAFONE         access      1,048,576   6.17%
2620:0:860::/44       GOOGLE           cloud               0     n/a

so the v6 blocklist is blunt, and the v4 one is a scalpel
  203.0.113.0/24                               256 addresses, one public address at a time
                                               256 concurrent subscribers, 0.00% of the base
  2001:610:240::/42     77,371,252,455,336,267,181,195,264 addresses, one /64 per household
                                         4,194,304 concurrent subscribers, 10.75% of the base

what a residential exit is worth, scored the way a risk engine scores it
case                                risk  bucket     driven by
leased line, 3yr tenant            0.00  clean      asn_class_access
leased line, 3yr tenant            0.00  clean      solved_captcha_before, asn_class_access
residential, 11 co-tenants         0.20  watched    egress_tenants_gt_40, asn_class_access
mobile CGNAT exit                  0.00  clean      asn_class_mobile
mobile CGNAT exit                  0.05  clean      req_rate_10min_gt_30, asn_class_mobile
residential, fresh lease           0.20  watched    lease_age_lt_30d, hosts_per_day_gt_500, asn_class_access
cloud egress                       0.85  prebanned  lease_age_lt_30d, asn_class_cloud
hosting reseller                   1.00  prebanned  on_abuse_feed, asn_class_hosting

the cgnat ceiling: one public address, n subscribers, one shared verdict
  cell sector, busy hour       4,096 subscribers share one address; one ban costs all of them
  cell sector, 03:00             610 subscribers share one address; one ban costs all of them
  business fibre behind nat       64 subscribers share one address; one ban costs all of them
  free wifi hotel chain        3,000 subscribers share one address; one ban costs all of them

The /42 row is the whole argument for treating IPv6 as its own reputation pool. A single published block at Deutsche Telekom covers 4.19 million households. No vendor can ship that in a blocklist, so the practical granularity for IPv6 collapses to "the whole /32" or "the whole ASN", which means an IPv6 exit is judged on the reputation of an enormous shared block rather than on anything it did. A /24 in the same table covers 256 addresses and costs the operator essentially nothing to block, which is why IPv4 address-level reputation is sharp and IPv6 prefix-level reputation is not.

IPv4 and IPv6 Are Different Products#

The right-hand column is the part that surprises people. An IPv6 exit has no history because the address space is too large for anyone to have observed it, so the vendor cannot have an opinion about it. An address with no opinion gets treated as either neutral or suspicious depending on the vendor's default, and if you serve localized content, the default is suspicious. Geo-grading and timezone must therefore be built deliberately rather than left to the OS: see DNS resolver signals for the leak that undoes an otherwise perfect v6 exit.

Why a Residential Proxy Is Not Automatically Trusted#

The residential, 11 co-tenants row scores 0.20 and lands in watched. That is the trap. A residential proxy is a shared address by construction, and the vendor knows the sharing ratio, because the vendor sold it. Worse, the tenant count is a public commercial fact: proxy resellers publish pool sizes, and a WAF that has been told "this address is shared by 40 tenants of a proxy service" scores it accordingly no matter how clean your stack is.

Two more things work against a naive residential exit. The address may be a published VPN or proxy exit, which is a list lookup rather than a judgement call. And the household may not behave like a household: a proxy subscriber in a residential country who requests 500 distinct hosts in a day has a usage profile no ISP subscriber has, and that profile is attached to the address regardless of which tenant produced it.

What actually earns a residential address a good score is tenure and consistency: a stable address across a long window, a request rate inside the household distribution, a small number of distinct hosts, and a coherent session profile that looks like one person reading one site. None of those are properties of the proxy product. They are properties of how you use it, which is why warming an identity over days beats buying a "trusted" pool.

CGNAT and the Shared Verdict#

Mobile egress is not trusted because it is clean. It is tolerated because blocking it is catastrophic. A busy-hour cell sector can carry 4,096 subscribers behind one public address, and the moment a vendor bans that address every one of them is collateral. This produces the perverse result that the least trustworthy addresses, from a behaviour standpoint, get the least scrutiny, and it is exactly why mobile proxies cost what they cost.

The consequence for your own planning is that a mobile exit cannot be treated as a scarce resource to be rotated away from when challenged. You cannot get another one, and if the sector is shared with other tenants your behaviour is being scored alongside theirs. Read the last four lines of that output as an availability statement: a 3,000-subscriber hotel wifi address is functionally equivalent to a busy cell sector, so if you need an address that a defender will not block, the mobile tier and the enterprise wifi tier are the two places to look, and both are priced accordingly.