When the Same Thing Wears Many Names#

Scrape the same product from three retailers and you get "iPhone 15 Pro 128GB", "Apple iPhone 15 Pro (128 GB)" and "iPhone 15 Pro 128 GB - Natural Titanium". They are the same phone. Entity resolution is recognising that different records refer to the same real-world thing; record linkage is joining them across sources. Without it your dataset counts one phone three times, compares prices of different products as if they were the same, and reports a company as two competitors. This lesson builds a working resolver step by step: normalise, match exact identifiers, block, score, cluster, and measure.

Step 0: Use Real Identifiers First#

Before any fuzzy matching, look for identifiers that settle the question outright. Products often carry a GTIN/EAN/UPC barcode or a manufacturer part number in JSON-LD or the spec table. Books have ISBNs, companies have registration numbers and website domains, and articles have DOIs. Scrape these fields deliberately; one exact key match beats any similarity score.

import re

def gtin(value):
    '''Normalise a barcode: digits only, zero-padded to GTIN-14.'''
    digits = re.sub(r"\D", "", value or "")
    return digits.zfill(14) if 8 <= len(digits) <= 14 else None

Zero-padding matters: the same product can appear as a 12-digit UPC on one site and a 13-digit EAN with a leading zero on another. Padding both to 14 digits makes them equal. Records that share a normalised identifier are linked immediately; fuzzy matching is only for what remains.

Step 1: Normalise#

A large share of duplicates disappear with careful normalisation alone. Lowercase, strip punctuation, collapse whitespace, unify units, expand abbreviations, and drop words that carry no identity (marketing text, colour names when you track models rather than variants):

import unicodedata

UNITS = [(r"(\d+)\s*(gb|gigabytes?)\b", r"\1 gb"),
         (r"(\d+)\s*(tb|terabytes?)\b", r"\1 tb"),
         (r"(\d+)\s*(ml|millilit(?:re|er)s?)\b", r"\1 ml")]
NOISE = {"apple", "new", "unlocked", "the", "-", "natural", "titanium"}

def normalise(title):
    t = unicodedata.normalize("NFKD", title).encode("ascii", "ignore").decode()
    t = t.lower()
    t = re.sub(r"[()\[\],/|]", " ", t)
    for pattern, repl in UNITS:
        t = re.sub(pattern, repl, t)
    tokens = [tok for tok in t.split() if tok not in NOISE]
    return " ".join(tokens)

for s in ["iPhone 15 Pro 128GB", "Apple iPhone 15 Pro (128 GB)",
          "iPhone 15 Pro 128 GB - Natural Titanium"]:
    print(normalise(s))

All three print iphone 15 pro 128 gb. The noise list is domain-specific and should be built by looking at your data, not guessed. Be careful what you delete: removing colour words is right if you track models and wrong if you track stock per variant. For people and organisations, normalisation means handling titles ("Dr"), suffixes ("Ltd", "Inc", "GmbH"), and ordering ("Smith, Jane" versus "Jane Smith").

Step 2: Blocking#

Comparing every record with every other is O(n²): 100,000 records means about five billion pairs. Blocking groups records by a cheap key and compares only within groups. Good keys are ones true matches almost always share:

from collections import defaultdict
from itertools import combinations

def block_key(rec):
    toks = rec["norm"].split()
    return (rec.get("brand") or "", toks[0] if toks else "")

def candidate_pairs(records):
    blocks = defaultdict(list)
    for r in records:
        blocks[block_key(r)].append(r)
    for group in blocks.values():
        yield from combinations(group, 2)

Blocking trades a little recall for an enormous speedup: if two true matches land in different blocks, they will never be compared. Reduce that risk by using several blocking keys (brand plus first token, and separately a key built from all numbers in the title) and taking the union of candidate pairs. Measure how many true matches your blocks keep, using the labelled sample described below.

Step 3: Score Pairs#

For each candidate pair, compute several similarity signals and combine them. The common functions, all available in the rapidfuzz library:

  • Edit distance (Levenshtein): number of single-character edits; good for typos.
  • Jaro-Winkler: rewards shared prefixes; good for short strings and personal names ("jonathan" and "jonathon" score 0.95).
  • Token set ratio: compares sets of words, ignoring order and duplicates; robust for product titles.
  • TF-IDF cosine: weights rare words more than common ones across the whole corpus, so a model number counts more than "pro".
  • Phonetic codes (Soundex, Metaphone): match names that sound alike.

A single fuzzy score has a dangerous blind spot. Token set ratio rates iphone 15 pro 128 gb against iphone 15 pro 256 gb natural titanium at about 89 out of 100, yet they are different products. The fix is hard constraints: fields that must agree exactly, no matter how similar the rest looks. Numbers in titles (capacities, sizes, model years) are the classic case:

from rapidfuzz import fuzz

def numbers(s):
    return set(re.findall(r"\d+", s))

def score(a, b):
    if a.get("gtin") and b.get("gtin"):
        return 1.0 if a["gtin"] == b["gtin"] else 0.0
    if numbers(a["norm"]) != numbers(b["norm"]):
        return 0.0                      # hard constraint: 128 gb is not 256 gb
    title = fuzz.token_set_ratio(a["norm"], b["norm"]) / 100
    brand = 1.0 if a.get("brand") == b.get("brand") else 0.0
    pa, pb = a.get("price"), b.get("price")
    price_close = 1.0 if pa and pb and abs(pa - pb) / max(pa, pb) < 0.25 else 0.0
    return 0.7 * title + 0.2 * brand + 0.1 * price_close

The weights here are a starting point, not a truth. Tune them against labelled data; with enough labelled pairs, a logistic regression over the same features learns them for you, which is essentially what libraries such as dedupe, recordlinkage and Splink provide, along with blocking and clustering.

Step 4: Decide and Cluster#

Use two thresholds. Above the high one (say 0.9) is a match; below the low one (say 0.6) is a non-match; the band in between goes to a review queue or a stricter rule. Then merge pairwise matches into entities with union-find, which handles chains of links efficiently:

class UnionFind:
    def __init__(self):
        self.parent = {}
    def find(self, x):
        self.parent.setdefault(x, x)
        while self.parent[x] != x:
            self.parent[x] = self.parent[self.parent[x]]  # path halving
            x = self.parent[x]
        return x
    def union(self, a, b):
        self.parent[self.find(a)] = self.find(b)

uf = UnionFind()
review = []
for a, b in candidate_pairs(records):
    s = score(a, b)
    if s >= 0.9:
        uf.union(a["id"], b["id"])
    elif s >= 0.6:
        review.append((a["id"], b["id"], round(s, 2)))
clusters = defaultdict(list)
for r in records:
    clusters[uf.find(r["id"])].append(r["id"])

Transitive merging has a known failure: A matches B, B matches C, but A and C are clearly different, and suddenly one generic record ("iPhone case") glues dozens of products into one entity. Watch for it by flagging clusters that are unusually large or whose members fail a pairwise check against each other, and split them for review.

Step 5: Stable IDs and Golden Records#

Give each resolved entity a stable internal ID that survives re-runs; downstream tables join on it, so it must not change every time clustering reruns. A practical approach: when a new cluster overlaps an existing entity, it inherits that entity's ID. Then build a golden record per entity, choosing each field by a rule (most recent, most complete, most trusted source) and recording which source supplied it:

golden = {
    "entity_id": "ent_000812",
    "title": {"value": "iPhone 15 Pro 128 GB", "source": "retailer_a", "seen": "2026-09-29"},
    "gtin": {"value": "00012345678905", "source": "retailer_b", "seen": "2026-09-28"},
    "members": ["retailer_a:8812", "retailer_b:A-77", "retailer_c:xp-15p-128"],
}

Never overwrite or delete the source records during resolution. Thresholds will change, and you will need to re-resolve from raw inputs (see dataset versioning and lineage).

Step 6: Measure Precision and Recall#

Label a few hundred candidate pairs by hand as match or non-match, deliberately including hard cases: near-identical titles with different capacities, and true matches with very different wording. Then measure:

def evaluate(predicted: set, truth: set):
    tp = len(predicted & truth)
    precision = tp / len(predicted) if predicted else 0.0
    recall = tp / len(truth) if truth else 0.0
    return round(precision, 3), round(recall, 3)

Precision asks: of the pairs we merged, how many are truly the same entity? Recall asks: of the true matches, how many did we find? Precision errors (false merges) corrupt analytics silently, for example averaging the prices of two different models. Recall errors leave duplicates, which inflate counts. Price comparison and financial data usually favour precision; deduplicating a mailing list can tolerate lower precision in exchange for recall. Re-run the evaluation after every rule change, because a tweak that raises recall often lowers precision.

Keeping It Running#

New records arrive with every crawl. Match them against existing entities rather than re-clustering everything: block the new record against the entity index, score against each candidate entity's members or golden record, and attach, create or send to review. Entities also split (a product line is renamed, a company divests a brand), so keep an audit trail of every merge with its score and rule, and make merges reversible. Resolution quality depends on upstream cleaning, so pair this lesson with data cleaning and validation, and capture identifiers like GTINs at scrape time from structured data where the page offers it (see JSON payloads).