Geo-Targeted and Localized Scraping

The Page You See Depends on Where You Look#

Two visitors sending identical requests can get different pages because their IP addresses sit in different countries. Sites localise for language, currency, tax, shipping, licensing and legal reasons, so a scraper has to understand which signals it is sending, control the locale it presents, and record that locale alongside every record. Otherwise you end up with a dataset that silently mixes euros and dollars, or one that describes a market you never meant to measure.

How Servers Decide Where You Are#

Sites combine several signals, roughly from strongest to weakest:

1. Explicit choice. A path segment (/fr-fr/), a country-code domain (.de), a query parameter (?country=CA) or a region cookie set by a country picker. These usually override everything else.
2. IP geolocation. A geo-IP database maps your address to a country and often a city. CDNs do this at the edge and may pass the result to the origin in a request header.
3. Accept-Language. Mainly drives UI language; some sites also use it to guess the market.
4. Browser signals. Time zone, navigator.language and, if the page asks and you grant it, the Geolocation API. These only matter when JavaScript runs.
5. Account settings. A logged-in profile's country often wins over everything else.

When a page looks "wrong" (prices in dollars instead of euros, a product missing, a different layout), check geography before debugging selectors. A currency bug is often a location bug.

Prefer the Explicit Switch#

Before paying for proxies in ten countries, look for the site's own locale mechanism. Many international sites publish hreflang alternates that list every regional version of a page:

from bs4 import BeautifulSoup

def locale_alternates(html):
    soup = BeautifulSoup(html, "lxml")
    return {
        link["hreflang"]: link["href"]
        for link in soup.select('link[rel="alternate"][hreflang]')
    }

# {'en-us': 'https://shop.example/en-us/p/42',
#  'fr-fr': 'https://shop.example/fr-fr/p/42',
#  'x-default': 'https://shop.example/p/42'}

If regional URLs exist, fetch them directly; no proxy is needed to see the French catalogue if the site serves it at /fr-fr/ to everyone. Likewise, if the country picker just sets a cookie such as country=FR, set that cookie in your session. Check the response's Vary header too: Vary: Accept-Language tells you the header genuinely changes the content, which also means any cache in your pipeline must include it in the key.

Test Whether a Site Geo-Targets#

Measure before you engineer. Fetch the same URL through two exits and compare the fields that matter:

import requests

def fetch(url, proxy=None, lang="en-US,en;q=0.9"):
    proxies = {"http": proxy, "https": proxy} if proxy else None
    r = requests.get(url, proxies=proxies, timeout=30,
                     headers={"Accept-Language": lang})
    return r

def exit_country(proxy=None):
    proxies = {"http": proxy, "https": proxy} if proxy else None
    return requests.get("https://ipinfo.io/json", proxies=proxies,
                        timeout=15).json().get("country")

for label, proxy in [("local", None), ("fr", FR_PROXY)]:
    r = fetch("https://shop.example/p/42", proxy)
    print(label, exit_country(proxy), r.status_code, r.url,
          r.headers.get("Content-Language"), len(r.text))

Typical signs of geo-targeting: a redirect to a different regional path (r.url differs), a different Content-Language, a different currency symbol in the price element, or a 451/403 from one exit only. HTTP 451 ("Unavailable For Legal Reasons") is the explicit status for legally restricted content, though many sites just show a generic block page with a 200 or 403. If the two responses are identical, the site does not geo-target this page, and you can skip regional proxies entirely.

Controlling the Exit With Regional Proxies#

When content really does follow IP geolocation, you need an exit in the target country: a residential, ISP or datacenter proxy from a pool you can filter by country (the proxies and rotation lesson covers pool types). Two rules keep the results trustworthy.

Verify every exit. Providers' geo labels and the target's geo-IP database do not always agree; a "Belgian" exit that the target places in the Netherlands silently gives you the wrong currency. Check the exit once per proxy session against a geo-echo endpoint and log the answer with the data:

session_country = exit_country(proxy)
if session_country != "BE":
    raise RuntimeError(f"wanted BE exit, got {session_country}")

A geo-echo service uses its own database, so it approximates rather than guarantees what the target sees. The strongest check is on the target itself: assert that the currency or regional path matches what you expected before storing a record.

Keep the signals consistent. An exit in France with Accept-Language: en-US and a browser time zone of America/New_York is an unusual combination. It may produce a mixed page (French prices, English text) and it is a common anomaly signal for bot detection. Align language and time zone with the exit. With Playwright, the browser context carries all of it:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": FR_PROXY_SERVER,
                                       "username": USER, "password": PASS})
    context = browser.new_context(
        locale="fr-FR",
        timezone_id="Europe/Paris",
        geolocation={"latitude": 48.8566, "longitude": 2.3522},
        permissions=["geolocation"],
    )
    page = context.new_page()
    page.goto("https://shop.example/p/42")
    print(page.locator(".price").inner_text())

locale sets navigator.language and the Accept-Language header, timezone_id changes what Intl and Date report, and geolocation plus the permission answers the Geolocation API. Only grant geolocation when the page actually needs it; most sites decide by IP alone.

Persist the Locale With the Data#

Geo-targeted data is regional data, so each record must say which region produced it:

from babel.numbers import parse_decimal

record = {
    "sku": "42",
    "country": "FR",                 # requested market
    "exit_country": session_country, # what the geo check reported
    "locale": "fr-FR",
    "url": r.url,                    # final URL after regional redirects
    "price_raw": "1 234,56 €",
    "amount": parse_decimal("1234,56", locale="fr_FR"),
    "currency": "EUR",
    "fetched_at": "2026-09-30T08:00:00Z",
}

Locale-aware parsing matters because 1.234,56 in German means one thousand two hundred thirty-four, while in English the same string is nonsense or a small number. Parse with the locale that produced the text, and store amount and currency in separate columns so "did the price differ by region?" is a simple query later. The data cleaning and validation lesson covers the general normalisation step; the geo-specific rule is that the locale is part of the record's key. (sku, country) is unique, sku alone is not.

How Geo Scrapes Fail#

• Cache poisoning in your own pipeline. Your HTTP cache stores the French response under the URL alone; the German run reads it back. Include country and language in the cache key.
• Sticky redirects. The first visit sets a region cookie; later requests through a different exit keep the old region. Use a fresh session per region.
• Mixed exits in one session. Rotating proxies across countries mid-session produces pages assembled from different markets. Pin one country per session (see proxies and session stickiness).
• City-level differences. Delivery availability and local prices can vary by city or postcode, not just country. If they do, the postcode (often set via a cookie or form) becomes part of the key too.

The Legal Line#

Seeing the localised version a site serves to people in a region is ordinary market research; many sites even publish those versions openly through hreflang. Presenting a foreign location to get past a deliberate regional block is different. Streaming and media services geo-block to honour licences, and some regulated goods are restricted by law in certain countries. Before routing around a block, re-read the legal considerations lesson and the target's terms. Detect and document geo-gating first, then collect the regional data that is openly published for that region, and record faithfully where every row came from.