Proxy Pools & Session Stickiness

Managing the Geyser of IPs#

Buying a proxy pool is easy; operating it is a discipline. Most providers give you a gateway URL you can plug into requests, then all of the pool complexity is hidden behind that single endpoint. Understanding how that gateway works—and how to steer which IP you get—separates scrapers that survive months from scrapers that get banned in a day.

Backconnect Gateways#

A backconnect gateway is a load balancer in front of the pool. Your HTTP request hits the gateway, and the gateway forwards it through a specific exit IP chosen by its rotation logic.

• Rotating (Every Request): The gateway gives you a fresh exit IP per request. Perfect for wholesale page downloads, but useless when you need a login session.
• Sticky (Timed): The gateway keeps you on the same exit IP for a configurable window (typically 5–30 minutes). Within that window every request shares cookies, TLS session, and reputation continuity.
• Sticky (Bounded): The gateway holds the IP until it has served N requests or until the target site bans it, then rotates. This is the classic "sneaker bot" mode.

The choice of rotation mode must match the shape of your job. A job that hits 10,000 distinct product pages with no cookies has no reason to be sticky. A job that signs in, adds items to a cart, checks out, and repeats absolutely must be sticky—otherwise a mid-session IP swap looks like account hijacking to the WAF.

Session Continuity Is Data#

Think of a session as a bundle of correlated signals: IP, TLS parameters, cookies, fingerprint, and visit history. WAFs model sessions the same way you do. If your cookies say "same user", your fingerprint says "same device", but your IP suddenly jumps from California to Germany, the correlation breaks. Staying sticky on one IP for the entire logical workflow keeps all those signals mutually consistent.

Geo-Alignment#

Residential providers let you filter exits by country, region, and sometimes city. Always align geography with the content you are fetching: a scraping job reading localized Japanese retail sites from a US residential IP will trigger anomaly signals that a local Japanese IP would never see. When a site legitimately uses geo-gating, a mislocated IP is a guaranteed red flag.

Pool Health and Recycling#

Provider pools recycle. When a homeowner unplugs the router, that IP vanishes from the pool and reappears elsewhere. Your code must treat rotating exits as the norm: never hardcode an IP, always expect connect timeouts, and structure retries so that a dead exit simply means "try the next IP". Also monitor your own ban rate per pool segment. If 10% of one provider's subnets return 403s while another provider stays clean, rebalance your traffic toward the clean pool.

Pricing Models#

• By GB: residential and mobile proxies bill on bandwidth. High-throughput HTML scraping becomes expensive fast.
• By IP count: datacenter proxies bill per IP slot. Ideal for many long-lived socks.
• By concurrent session: some providers bill by simultaneous gateway sessions, which fits headless-browser workloads better than raw bandwidth.

Choose the billing model that matches your traffic profile. Paying per gigabyte for a text-only scrape wastes money; renting 10,000 IPs to run 50 slow browsers wastes money the other way.

Stickiness Is a Lease, Not a Reservation#

The single most expensive misunderstanding in proxy operations is reading "sticky session" as "you keep this IP". What a provider actually offers is a lease: for the duration of the window, or for the count you have paid for, your requests route through one address. At the moment the lease ends, that address returns to the pool and may be leased to somebody else, including you, including not you. There is no callback and no error. The next request simply leaves from somewhere else, and the only symptom is a WAF scoring your session as incoherent.

That is why the lease boundary has to be a design input rather than an operational detail. Either your flow completes well inside the window, or you renew deliberately and accept the discontinuity, or you hold a fixed address on a dedicated port. The middle option is the dangerous one: it looks like stickiness and behaves like rotation.

A Gateway, and What It Decides#

Here is the decision logic of a backconnect gateway, reduced to something you can reason about. Fourteen exits with country, class and tenant count, two five-step flows that straddle the window boundary on purpose, and a thirty-six request crawl after them.

# What a backconnect gateway actually decides, and what it costs.
# Three rotation modes, one workload, one logical clock. No sleeps, no
# network: the only state is the assignment table.

import random

MODES = ["rotate", "sticky:6min", "sticky:4req"]

POOL = [
    ("de-01", "DE", "residential", 3),
    ("de-02", "DE", "residential", 11),
    ("de-03", "DE", "residential", 2),
    ("de-04", "DE", "residential", 18),
    ("de-05", "DE", "residential", 5),
    ("de-06", "DE", "residential", 9),
    ("fr-01", "FR", "residential", 40),
    ("fr-02", "FR", "residential", 6),
    ("fr-03", "FR", "residential", 14),
    ("us-01", "US", "mobile", 900),
    ("us-02", "US", "hosting", 1),
    ("us-03", "US", "residential", 7),
    ("nl-01", "NL", "residential", 25),
    ("se-01", "SE", "residential", 4),
]

INDEX = {e[0]: e for e in POOL}

# checkout-DE deliberately straddles the six-minute window boundary
FLOWS = {
    "checkout-DE": ("DE", [("login", 0), ("home", 1), ("cart", 2),
                           ("shipping", 7), ("confirm", 8)]),
    "checkout-FR": ("FR", [("login", 9), ("home", 9), ("cart", 10),
                           ("shipping", 11), ("confirm", 12)]),
}

CRAWL = [("page-%02d" % i, 14 + i // 6, "crawl") for i in range(36)]


class Gateway:
    def __init__(self, mode, seed):
        self.mode = mode
        self.window = 6
        self.cap = 4
        self.rng = random.Random(seed)
        self.assign = {}
        self.quarantined = set()
        self.use = dict((e[0], 0) for e in POOL)
        self.banned = set()

    def sticky(self):
        return self.mode.startswith("sticky")

    def pick(self, flow, now, country):
        hold = self.assign.get(flow)
        if hold is not None and self.sticky():
            exit_id, since, count = hold
            in_window = now - since < self.window
            under_cap = count < self.cap
            if (self.mode == "sticky:6min" and in_window) or \
               (self.mode == "sticky:4req" and under_cap):
                return exit_id
        live = [e for e in POOL if e[0] not in self.quarantined]
        if self.sticky():
            # A timed hold is a lease, not a reservation: the moment it expires
            # the address goes back into the pool and may be re-leased. That is
            # why the gateway cannot promise you your old exit back.
            taken = set(v[0] for k, v in self.assign.items() if k != flow)
            if hold is not None:
                taken.add(hold[0])
            free = [e for e in live if e[0] not in taken]
            live = free or live
        if self.sticky() and country:
            pinned = [e for e in live if e[1] == country]
            if pinned:
                live = pinned
        exit_id = live[self.rng.randrange(len(live))][0]
        self.assign[flow] = (exit_id, now, 0)
        return exit_id

    def note(self, flow, exit_id):
        exit_id, since, count = self.assign[flow]
        self.assign[flow] = (exit_id, since, count + 1)
        self.use[exit_id] += 1
        if exit_id in self.banned:
            self.quarantined.add(exit_id)


def run(mode, banned=frozenset()):
    gw = Gateway(mode, seed=20260803)
    gw.banned = set(banned)
    breaks, geo_miss, trace = 0, 0, []
    for flow, (country, steps) in sorted(FLOWS.items()):
        prev = None
        for name, minute in steps:
            exit_id = gw.pick(flow, minute, country)
            exit_id, holder, klass, tenants = INDEX[exit_id]
            if prev is not None and prev != exit_id:
                breaks += 1
                trace.append((minute, flow, name, prev, exit_id, "egress changed"))
            if holder != country:
                geo_miss += 1
                trace.append((minute, flow, name, prev, exit_id,
                              "geo %s wanted, got %s" % (country, holder)))
            if exit_id in gw.banned:
                trace.append((minute, flow, name, prev, exit_id, "quarantined exit"))
            prev = exit_id
            gw.note(flow, exit_id)
    crawl_exits = set()
    for name, minute, country in CRAWL:
        key = name if gw.sticky() else name + str(minute)
        exit_id = gw.pick(key, minute, country)
        crawl_exits.add(exit_id)
        gw.note(key, exit_id)
    return gw, breaks, geo_miss, crawl_exits, trace


def main():
    print("workload: two five-step checkout flows, then 36 crawl fetches")
    print(f"{'mode':<12} {'egress breaks':>13} {'geo misses':>11} "
          f"{'crawl exits':>12} {'quarantined':>12}")
    for mode in MODES:
        gw, breaks, geo, crawl, _t = run(mode, banned={"fr-01"})
        print(f"{mode:<12} {breaks:>13} {geo:>11} {len(crawl):>12} "
              f"{len(gw.quarantined):>12}")
    print()

    print("the sticky:6min trace: the window expires mid-flow and nothing warns you")
    for mode in ["sticky:6min", "sticky:4req"]:
        _gw, _b, _g, _c, trace = run(mode)
        print(f"  {mode}")
        if not trace:
            print("    (no discontinuity in this workload)")
        for minute, flow, name, prev, exit_id, note in trace:
            print(f"    {minute:>3}min  {flow:<14} {name:<9} {note}")
    print()

    print("the same run with fr-01 banned: quarantine removes it from the pool")
    for mode in MODES:
        gw, _b, _g, crawl, _t = run(mode, banned={"fr-01"})
        print(f"  {mode:<12} quarantined={sorted(gw.quarantined)} "
              f"crawl exits={sorted(crawl)}")
    print()

    print("exit utilisation under sticky:4req: the cap is what spreads load")
    gw, _b, _g, _c, _t = run("sticky:4req")
    for exit_id, count in sorted(gw.use.items(), key=lambda kv: (-kv[1], kv[0])):
        _id, holder, klass, tenants = INDEX[exit_id]
        print(f"  {exit_id}  {holder}  {klass:<12} tenants={tenants:<4} requests={count}")
    print()
    print("  worst exit by tenant count, and what a single 403 costs it")
    worst = max(POOL, key=lambda e: e[3])
    print(f"    {worst[0]} carries {worst[3]} tenants; "
          f"quarantining it removes 1/{len(POOL)} of capacity")


main()

workload: two five-step checkout flows, then 36 crawl fetches
mode         egress breaks  geo misses  crawl exits  quarantined
rotate                   8           8           13            1
sticky:6min              1           0           13            1
sticky:4req              2           0           14            1

the sticky:6min trace: the window expires mid-flow and nothing warns you
  sticky:6min
      7min  checkout-DE    shipping  egress changed
  sticky:4req
      8min  checkout-DE    confirm   egress changed
     12min  checkout-FR    confirm   egress changed

the same run with fr-01 banned: quarantine removes it from the pool
  rotate       quarantined=['fr-01'] crawl exits=['de-01', 'de-02', 'de-03', 'de-04', 'de-05', 'de-06', 'fr-01', 'fr-03', 'nl-01', 'se-01', 'us-01', 'us-02', 'us-03']
  sticky:6min  quarantined=['fr-01'] crawl exits=['de-01', 'de-02', 'de-04', 'de-05', 'de-06', 'fr-01', 'fr-02', 'fr-03', 'nl-01', 'se-01', 'us-01', 'us-02', 'us-03']
  sticky:4req  quarantined=['fr-01'] crawl exits=['de-01', 'de-02', 'de-03', 'de-04', 'de-05', 'de-06', 'fr-01', 'fr-02', 'fr-03', 'nl-01', 'se-01', 'us-01', 'us-02', 'us-03']

exit utilisation under sticky:4req: the cap is what spreads load
  de-02  DE  residential  tenants=11   requests=8
  fr-03  FR  residential  tenants=14   requests=8
  us-02  US  hosting      tenants=1    requests=6
  fr-01  FR  residential  tenants=40   requests=4
  de-01  DE  residential  tenants=3    requests=3
  fr-02  FR  residential  tenants=6    requests=3
  us-01  US  mobile       tenants=900  requests=3
  de-03  DE  residential  tenants=2    requests=2
  de-04  DE  residential  tenants=18   requests=2
  nl-01  NL  residential  tenants=25   requests=2
  us-03  US  residential  tenants=7    requests=2
  de-05  DE  residential  tenants=5    requests=1
  de-06  DE  residential  tenants=9    requests=1
  se-01  SE  residential  tenants=4    requests=1

  worst exit by tenant count, and what a single 403 costs it
    us-01 carries 900 tenants; quarantining it removes 1/14 of capacity

Read the first table carefully. rotate produces eight egress breaks and eight geo misses inside two short flows: it is unusable for anything stateful, and the geo misses are the tell that it ignores your country pin entirely. sticky:4req produces two breaks, and they land on the fifth request of a flow, because a request-count lease is a fixed budget and a five-step flow costs five. sticky:6min produces one, at minute seven of a flow that started at minute zero, which is the lease boundary and nothing else.

There is no mode in that table with zero breaks, and that is the correct answer. A timed or counted lease always ends. The only ways to get zero are to finish inside the window every time, or to stop using a shared gateway.

What Rotating Egress Breaks at the Transport Layer#

Rotating the address mid-flow breaks more than the cookie binding. Four things happen at once, and only one of them is about cookies.

The TLS session dies. A new address means a new TCP connection, so a new handshake, so a new session ticket. A session ticket is scoped to a client and a server, and a client that resumes from a different source address still resumes, but a client that suddenly stops resuming on a connection that should resume is itself a signal. The resumption rate of your fleet drops the moment you rotate, and it drops in a way that correlates with the address churn.

The HTTP/2 stream count resets. Multiplexed streams cannot survive a new connection, so request-per-connection goes up, and connection reuse frequency is one of the cheapest transport-level statistics to compute on the server side.

HTTP/3 makes it worse. If the client negotiates QUIC and then changes address mid-connection, the server sees a connection migration event, and migration is a legitimate feature that a fleet should not be performing routinely.

The cookie jar's continuity becomes a claim rather than a fact. The WAF sees one sessionid presented from three source addresses in ninety seconds. That is the signature of a stolen cookie, and it is scored as one.

The rule: treat the address as part of the session key. If you cannot hold the address, do not hold the cookies, and if you cannot hold the cookies, do not hold the account.

Geo-Consistency Is Four Fields, Not One#

Asking the provider for a German exit gets you a German address and nothing else. Four other fields have to agree with it, and each one is separately observable.

Geo-grading covers why the first row is a mismatch detector rather than a lookup, and the DNS path covers the leak that undoes all four when the resolver is not proxied. The operational rule is that the country pin has to be applied at profile creation, not per request, because a profile whose timezone changes when you change exits is a profile whose cookies and language list are now the odd ones out.

Health Checks and Quarantine#

The last table shows a subtlety that catches most implementations: quarantine is reactive. fr-01 is banned, and in every mode it still appears in the crawl exits, because the crawl reached it before the ban was observed and only the subsequent note() quarantined it. Under sticky:6min it also disappears from two other crawl pages, because once quarantined it is excluded from new picks. So a single 403 removes an address from your pool one request late, and the request that discovered it already went out.

The fix is to make quarantine a pre-pick filter fed by a health signal, not a post-hoc annotation. Three signals feed it: a transport-level 403 from the edge, an application-level challenge interstitial that persists across two attempts, and a sudden drop in successful fetches per exit. Quarantine should be sticky for the rest of the process lifetime, and the exit should be reported back to the provider's own pool if your contract allows it, because the address is still shared with whoever leased it next. Note also the last line: us-01 carries 900 tenants, so quarantining it removes one address but protects you from inheriting the aggregate risk of 900 other users' behaviour. That trade is usually worth taking, and it is the practical argument for spreading a workload across several providers and several request schedulers rather than one large pool, which is the scheduling concern browser pool architecture is built around.