Stop Storing the Same Page Six Times

Web pages expose one entity under many URLs. The canonical page lives at /product/42.html, but the site also links /product/42, product/42?utm_source=x, product/42#reviews and an http:// version from an old page. If you treat URLs as opaque strings, your database fills with duplicates and your crawl wastes bandwidth re-fetching the same entity. URL normalization — parsing, resolving and canonicalizing — fixes it at the frontier.

Resolve Relative Links the Right Way

The most common bug in this area is building absolute URLs with string concatenation:

# Wrong: breaks when href is an absolute URL, //-protocol, or ../ path
full = base_url + href

Every link in an HTML document is relative to the page that contains it, so ../catalog/42.html must resolve against the current page URL, not the site root. Use urljoin, which handles ../, absolute paths, protocol-relative //host/path, and fragments correctly in one call:

from urllib.parse import urljoin

current = "https://example.com/a/b/page.html"
full = urljoin(current, "../../catalog/42.html")
# https://example.com/catalog/42.html

Store the urljoin result, because the page's own links are the contract you scrape; the browser resolves them, and you should too.

Normalize One Entity to One URL

Write a canonicalize(url) that turns equivalent variants into the same string. Strip the fragment (#reviews means the same document), drop tracking query params (utm_*, fbclid, ref), remove the default port, lowercase only the hostname (paths are case-sensitive on most servers), and decide whether to collapse a trailing slash — a light /2 vs 2/ both being fine if the server does not care. urllib.parse.urlsplit plus a small allowlist of the query keys the site actually uses is all you need:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
import re

KEEP = {"page", "sort", "id"}

def canonicalize(url):
    parts = urlsplit(url)
    host = parts.hostname.lower()
    params = [(k, v) for k, v in parse_qsl(parts.query) if k in KEEP]
    params.sort()
    return urlunsplit(("https", host, parts.path.rstrip("/") or "/", urlencode(params), ""))

Now the dedupe key is canonicalize(url) instead of the raw string, and product/42?utm=1 collapses into product/42 automatically.

Follow Redirects to Their End

Servers move pages, and every redirect doubles as a canonicalization hint: if /product/42 answers with 302 → /product/42.html, the final URL is the one to store. Your HTTP client already follows redirects by default — the pipeline must store response.url (the last hop), not the URL you requested:

r = client.get("https://example.com/product/42", follow_redirects=True)
stored_url = str(r.url)          # final destination after the 302
canonical = canonicalize(stored_url)

Store both the canonical URL and the immediate URL you discovered; a future crawl can re-check old links by resolving them anew. Pages that never settle (redirect loops) are crawl bugs, not features — cap max_redirects and drop the URL into a quarantine log.

Honor the Canonical Tag and Normalize the Frontier

Some sites declare the canonical entity themselves via <link rel="canonical" href="...">. Read it if present and use it as the storage key over anything the URL string implies — the site's own declaration beats your guesses. Then feed canonical URLs into the frontier from the crawl-strategy lesson: visit a page once, enqueue canonical forms, and let the dedupe set reject the rest. A crawl with a normalization layer fetches each resource once, stores it once, and never re-runs a redirected old link again — the cheapest bandwidth saving in all of scraping.