URL Parsing, Canonicalization & Redirects

Stop Storing the Same Page Six Times#

Websites expose one resource under many URLs. The product page lives at /product/42, but the site also links /product/42/, /product/42?utm_source=newsletter, /product/42#reviews, /Product/42;jsessionid=A1B2 and an old http:// version. Treat URLs as opaque strings and your crawler fetches the same page six times, your frontier grows without bound, and your database fills with duplicates that later need entity resolution to undo. URL handling has three separate jobs: resolving links correctly, normalising equivalent URLs to one key, and following redirects and canonical hints to the address the site considers authoritative.

The Parts of a URL#

urllib.parse.urlsplit breaks a URL into the pieces you will manipulate:

from urllib.parse import urlsplit

p = urlsplit("https://User@Shop.Example.com:8443/a/b.html?page=2&sort=price#reviews")
print(p.scheme, p.netloc, p.hostname, p.port, p.path, p.query, p.fragment, sep=" | ")

https | User@Shop.Example.com:8443 | shop.example.com | 8443 | /a/b.html | page=2&sort=price | reviews

Note that hostname is already lowercased and excludes the port and user info, while netloc is the raw string. Build normalised URLs from hostname and port, never by editing netloc with string replacements.

Resolve Relative Links the Right Way#

The most common bug is concatenating strings, base_url + href. It breaks on absolute links, protocol-relative links, ../ paths and query-only links. Every link in a page is relative to that page's own URL, and urljoin implements the resolution rules browsers use:

Three traps sit around urljoin:

• The base is the final URL, not the one you requested. If /lamps redirected to /shop/lamps/, relative links resolve against the redirect target. Use str(response.url).
• A <base href="..."> tag overrides the page URL as the base for every relative link. Check for it once per page: base = urljoin(str(response.url), soup.base["href"]) if soup.base and soup.base.get("href") else str(response.url).
• Junk hrefs. Strip whitespace (hrefs often contain newlines from templates), and discard javascript:, mailto:, tel: and data: links before they reach your frontier.

from urllib.parse import urljoin, urlsplit

SKIP_SCHEMES = ("javascript:", "mailto:", "tel:", "data:")

def extract_links(soup, page_url):
    base = page_url
    if soup.base and soup.base.get("href"):
        base = urljoin(page_url, soup.base["href"].strip())
    for a in soup.select("a[href]"):
        href = a["href"].strip()
        if not href or href.lower().startswith(SKIP_SCHEMES):
            continue
        url = urljoin(base, href)
        if urlsplit(url).scheme in ("http", "https"):
            yield url

Normalise Equivalent URLs to One Key#

Normalisation means rewriting URLs that point to the same resource into one identical string. Some rules are safe everywhere, because the standards say the variants are equivalent:

• Lowercase the scheme and host. (Paths are case-sensitive on most servers, so leave them alone.)
• Remove the default port (:443 for https, :80 for http).
• Drop the fragment; #reviews is handled by the browser and never sent to the server.
• Use uppercase hex in percent-escapes (%c3%a9 and %C3%A9 are the same) and percent-encode raw spaces and non-ASCII characters.
• Encode international domain names to their ASCII (punycode) form.
• Use / for an empty path.

Other rules are usually right but site-specific: removing tracking parameters, sorting query parameters, stripping session IDs, and collapsing www., trailing slashes or index.html. Apply them deliberately.

import re
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode, quote

DEFAULT_PORTS = {"http": 80, "https": 443}
TRACKING_KEYS = {"fbclid", "gclid", "msclkid", "mc_cid", "mc_eid", "ref_src"}
SAFE_PATH = "/%:@!$&'()*+,;=-._~"

def upper_escapes(s):
    return re.sub(r"%[0-9a-fA-F]{2}", lambda m: m.group(0).upper(), s)

def canonicalize(url, keep_params=None):
    parts = urlsplit(url.strip())
    scheme = parts.scheme.lower()
    host = (parts.hostname or "").rstrip(".")
    try:
        host = host.encode("idna").decode("ascii")      # bücher.example -> xn--bcher-kva.example
    except UnicodeError:
        pass
    port = parts.port
    netloc = host if port in (None, DEFAULT_PORTS.get(scheme)) else f"{host}:{port}"

    path = re.sub(r";jsessionid=[^/?]*", "", parts.path, flags=re.I)
    path = upper_escapes(quote(path or "/", safe=SAFE_PATH))

    query = []
    for k, v in parse_qsl(parts.query, keep_blank_values=True):
        if k.lower().startswith("utm_") or k.lower() in TRACKING_KEYS:
            continue
        if keep_params is not None and k not in keep_params:
            continue
        query.append((k, v))
    query.sort()
    return urlunsplit((scheme, netloc, path, urlencode(query), ""))

Test it with the variants you actually see. These all pass:

cases = {
    "HTTPS://Example.COM:443/a/b?utm_source=x&b=2&a=1#top": "https://example.com/a/b?a=1&b=2",
    "https://example.com/caf%c3%a9": "https://example.com/caf%C3%A9",
    "https://example.com/café": "https://example.com/caf%C3%A9",
    "https://example.com/p/42;jsessionid=AB12?color=": "https://example.com/p/42?color=",
    "https://example.com": "https://example.com/",
    "http://example.com:8080/x": "http://example.com:8080/x",
    "https://bücher.example/": "https://xn--bcher-kva.example/",
}
for raw, expected in cases.items():
    assert canonicalize(raw) == expected, (raw, canonicalize(raw))

Details worth knowing:

• keep_blank_values=True matters. Without it, parse_qsl silently drops color=, and a server that distinguishes "empty" from "absent" would get a different URL.
• quote with a safe set that includes % leaves existing escapes alone, so already-encoded URLs are not double-encoded. Decoding and re-encoding everything would turn an encoded slash (%2F) into a real path separator, which changes the meaning.
• Python's built-in idna codec implements the older IDNA 2003 rules. For unusual international domains, the third-party idna package implements the current standard.

Allowlist or Denylist for Query Parameters?#

For a crawl of one site you have studied, the allowlist gives the cleanest frontier. For broad crawls across many sites, use the denylist and rely on content hashing to catch remaining duplicates, as in incremental scraping. Either way, verify each rule empirically: fetch both variants once and compare the redirect target or a hash of the main content. If /product/42 and /product/42/ return the same content, collapsing the slash is safe on that site. Do not assume it for all sites.

Dedupe Key Versus Fetch URL#

Keep two URLs per page. The fetch URL is what you request, and it should be as close as possible to what the site linked, because some servers care about parameter order or exact encoding. The dedupe key is canonicalize(url), used only for "have I seen this?" checks in the frontier and as the storage key. Sorting query parameters is safe for the key and occasionally unsafe for fetching. Scrapy's duplicate filter works the same way, using w3lib.url.canonicalize_url to compute fingerprints while still requesting the original URL.

Follow Redirects to Their End#

A redirect is the server telling you the authoritative address. Record the whole chain, and key the stored record on the final URL:

import httpx

client = httpx.Client(follow_redirects=True, max_redirects=5, timeout=15)
try:
    r = client.get("http://example.com/product/42")
except httpx.TooManyRedirects:
    print("redirect loop; quarantine this URL")
else:
    chain = [str(h.url) for h in r.history] + [str(r.url)]
    print(" -> ".join(chain))
    for h in r.history:
        print(h.status_code, h.headers.get("location"))

http://example.com/product/42 -> https://example.com/product/42 -> https://example.com/products/42-desk-lamp
301 https://example.com/product/42
301 /products/42-desk-lamp

301 and 308 are permanent: add the old URL to a mapping so future crawls skip straight to the target. 302, 303 and 307 are temporary, and the original URL should stay in the frontier. Watch for redirects that lead to a login page, a country selector or the homepage; they mean the resource is unavailable to you, not that it moved, and storing the homepage content under a product key is a common data bug. The Location header can be relative (the second hop above), which the client resolves for you.

The Canonical Tag: Trust, Then Verify#

Many pages declare their preferred URL with <link rel="canonical" href="...">. It is the best available hint when several URLs show the same content, especially for pages with filter and sort parameters. Resolve it with urljoin (it can be relative) and sanity-check it before using it as the storage key:

def canonical_from_page(soup, page_url):
    tag = soup.select_one('link[rel="canonical"][href]')
    if not tag:
        return None
    canon = urljoin(page_url, tag["href"].strip())
    page_host = urlsplit(page_url).hostname
    canon_parts = urlsplit(canon)
    if canon_parts.hostname != page_host and not (canon_parts.hostname or "").endswith("." + (page_host or "")):
        return None                       # cross-site canonical: do not trust blindly
    if canon_parts.path in ("", "/") and urlsplit(page_url).path not in ("", "/"):
        return None                       # every page pointing at the homepage is a misconfiguration
    return canonicalize(canon)

Misconfigured canonicals are common: templates that point every page to the homepage, or paginated pages that all claim page 1 as canonical. The second case would make your crawler think pages 2 to 50 are duplicates and skip them, which is why paginated listings are usually better keyed by their own normalised URL. See pagination.

Where This Plugs In#

Canonicalise every discovered link before it enters the frontier from crawl strategy and the frontier, store the dedupe key alongside each record, and log how many discovered links were rejected as duplicates per run. A sudden change in that number usually means the site started adding a new tracking or session parameter.

Practice#

Crawl 200 pages of a site you are allowed to crawl and log every raw link. Count distinct raw URLs versus distinct canonicalize() keys. Then inspect the ten most common query parameter names and decide, by fetching both variants, which belong in the allowlist.