Proxies and IP Rotation

What a Proxy Actually Changes#

A proxy changes one thing: the source IP address the target server sees. It does not change your TLS fingerprint, your header order, your cookies, your request timing, or how your client runs JavaScript. When a site blocks you, first ask whether the cause is your IP or your behavior. Paying for proxies to fix a fingerprint or rate problem only moves the same blocked traffic to more addresses.

So when does a proxy help? When a server enforces limits or decisions per IP, spreading requests across several addresses gives you several independent budgets. Proxies also let you see a site as users in another country see it (prices, availability, language), which the geo-targeted scraping lesson covers. If the limit is per account, per session cookie, or per fingerprint, rotation does nothing for you.

How Proxying Works on the Wire#

• HTTP proxies receive your request and forward it. For an HTTPS target, your client first sends CONNECT shop.example.com:443 to the proxy, and the proxy opens a raw TCP tunnel. TLS then runs end to end between you and the target, so the proxy sees the hostname but not the URL path, headers, or body.
• The proxy URL scheme describes the proxy, not the target. Most proxies are http://user:pass@host:port even when every target is HTTPS. Writing https:// for an HTTP proxy is a frequent cause of TLS errors at connection time.
• SOCKS5 proxies tunnel TCP at a lower level. With requests (install requests[socks]), socks5:// resolves the target's hostname on your machine, while socks5h:// lets the proxy resolve it. Use socks5h when DNS lookups from your own network should not reveal the targets, or when the target must be resolved from the proxy's location.
• 407 Proxy Authentication Required comes from the proxy, not the target. It means bad or missing proxy credentials. A 502 or 503 can also come from the proxy itself; check the response body or headers to see who answered before blaming the target.

Proxy Types#

Start with the cheapest type that works, measure the block rate on a sample, and move up only when data shows hosting IPs are being rejected. Residential networks deserve a second look on sourcing: the IPs belong to real people, and reputable providers obtain explicit consent from them. Ask a provider how their network is sourced before you route traffic through it.

Providers deliver proxies in two shapes. A list gives you specific addresses you manage yourself. A gateway (sometimes called backconnect) gives you one hostname that picks an exit IP for you, rotating per request by default; sticky sessions are requested through a session parameter, usually embedded in the proxy username, with syntax that varies by provider.

Using a Proxy in Python#

With requests, proxies can be set per request or per session:

import requests

proxy = "http://user:pass@dc-proxy-01.example.net:8080"
s = requests.Session()
s.proxies = {"http": proxy, "https": proxy}
s.trust_env = False     # do not silently merge HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the environment

resp = s.get("https://ip-echo.example.com/json", timeout=(5, 20))
print(resp.json())      # confirm the exit IP is the proxy's, not yours

httpx configures the proxy on the client, not on individual requests, so the clean pattern is one client per proxy:

import httpx

clients = {}

def client_for(proxy_url: str) -> httpx.Client:
    if proxy_url not in clients:
        clients[proxy_url] = httpx.Client(proxy=proxy_url, timeout=15.0, follow_redirects=True)
    return clients[proxy_url]

Always verify the exit IP once per proxy at startup with an IP-echo endpoint. Misconfigured environment variables, a forgotten trust_env, or a NO_PROXY entry can send traffic out of your real address while you believe it is proxied.

Keep Sessions Sticky#

The classic mistake is rotating on every request. Sites tie state to an IP: a login session, a shopping cart, an anti-bot clearance cookie, a CSRF token. Change the IP mid-session and you look like a stolen cookie being replayed from elsewhere, which is a strong signal for fraud systems.

A policy that works: assign one proxy per target host (or per logical session, such as one logged-in account), keep it for the whole crawl of that host, and switch only when it fails. The pool below adds health tracking so a dead proxy is benched for a while instead of burning a timeout on every request:

import itertools
import time

class ProxyPool:
    def __init__(self, proxies, max_fails=3, cooldown=600):
        self.proxies = list(proxies)
        self.fails = {p: 0 for p in self.proxies}
        self.benched_until = {p: 0.0 for p in self.proxies}
        self.assigned = {}                      # host -> proxy
        self.rr = itertools.cycle(self.proxies)
        self.max_fails, self.cooldown = max_fails, cooldown

    def _healthy(self, p):
        return time.monotonic() >= self.benched_until[p]

    def for_host(self, host):
        p = self.assigned.get(host)
        if p and self._healthy(p):
            return p
        for _ in range(len(self.proxies)):
            candidate = next(self.rr)
            if self._healthy(candidate):
                self.assigned[host] = candidate
                return candidate
        raise RuntimeError("no healthy proxies; slow down or add capacity")

    def report(self, proxy, ok):
        if ok:
            self.fails[proxy] = 0
            return
        self.fails[proxy] += 1
        if self.fails[proxy] >= self.max_fails:
            self.benched_until[proxy] = time.monotonic() + self.cooldown
            self.fails[proxy] = 0
            self.assigned = {h: p for h, p in self.assigned.items() if p != proxy}

Using it:

pool = ProxyPool([
    "http://user:pass@dc-proxy-01.example.net:8080",
    "http://user:pass@dc-proxy-02.example.net:8080",
    "http://user:pass@dc-proxy-03.example.net:8080",
])

def fetch(url):
    host = httpx.URL(url).host
    proxy = pool.for_host(host)
    try:
        resp = client_for(proxy).get(url)
    except (httpx.ProxyError, httpx.ConnectError, httpx.TimeoutException):
        pool.report(proxy, ok=False)
        raise
    pool.report(proxy, ok=resp.status_code not in (407, 502))
    return resp

Only proxy-level failures count against a proxy. A 404 or a parse error says nothing about the proxy's health. A 429 or block page from the target is about your request rate or behavior, and the right response is usually to slow down for that host, as described in rate limiting and politeness, not to hop to a fresh IP and continue at the same pace.

Rotate on Evidence, Not on a Timer#

Rotate when a proxy is dead, when it returns proxy errors, or when a sticky session has naturally ended (a job finished, a login expired). Rotating on a schedule multiplies bandwidth costs, breaks sessions, and does not help if the block is fingerprint-based. And rotating specifically to keep going after a site has clearly told you to stop, through repeated blocks or a direct request, is the pattern the legal considerations lesson warns about: it turns a technical disagreement into evidence of intent.

Where Proxy Setups Go Wrong#

• Inconsistent identity. An exit IP in Germany with Accept-Language: en-US, a US time zone in the browser, and US-dollar cookies is an inconsistent profile. Match locale settings to the exit location when geography matters; the anti-bot course's lesson on proxies and session stickiness goes deeper.
• Burned shared IPs. Cheap shared pools are used by many customers at once, so an address may be blocked before you ever use it. Track success rate per proxy and per subnet, not just per request.
• Bandwidth surprises. Residential traffic is usually billed per GB. Fetching images, fonts, and full HTML through it when a small JSON endpoint would do can multiply the bill. Route only the requests that need the proxy through it.
• Latency. Proxies add a hop, often a slow one. Raise read timeouts moderately and measure per-proxy latency; drop proxies that are consistently slow.

Practice#

Set up three proxies (or three local forwarding proxies such as mitmproxy instances on different ports). Implement the pool, verify each exit IP at startup, and crawl two hosts. Kill one proxy mid-run and confirm from the logs that it is benched after three failures, its hosts move to a healthy proxy, and no request goes out directly.