When the Same Thing Wears Many Names

Scrape the same product from three retailers and you get "iPhone 15 Pro 128GB", "Apple iPhone 15 Pro (128 GB)", and "iPhone 15 Pro 128 GB - Natural Titanium". They are the same phone. Entity resolution is the discipline of recognizing that different records refer to the same real-world entity, and record linkage is joining them across sources. Without it, your dataset counts one phone three times.

Normalize First

Before comparing, normalize. Lowercase, strip punctuation and whitespace, expand common abbreviations, convert units, and standardize formatting (dates, sizes, currency). For names, handle titles, suffixes, and nicknames. A large fraction of duplicates collapse at this stage with no fuzzy matching required.

Blocking to Make It Tractable

Comparing every record to every other is O(n²) and impossible at scale. Blocking (or canopy clustering) groups records into candidate blocks that are likely to match—by first letter of a name, a shared domain, a zip code, or a MinHash bucket—and only compares within blocks. This trades a little recall for an enormous speedup.

Similarity Functions

For records that survive normalization, measure similarity:

  • Levenshtein / edit distance: how many edits to transform one string into another.
  • Jaro-Winkler: favors common prefixes; excellent for personal names and short strings.
  • Token set / Jaccard: compares the set of words, ignoring order—robust for product titles.
  • TF-IDF cosine similarity: weights rare words more heavily, so "Pro" matters more than "the".
  • Phonetic encodings (Soundex, Metaphone): match names that sound alike but are spelled differently.

Combine several signals into a weighted score rather than trusting one metric.

Matching and Clustering

A score above a high threshold is a match; below a low threshold is a non-match; the gray zone between needs a decision rule or human review. Once pairwise matches exist, transitive clustering merges connected records into one entity. Beware of chaining errors—A≈B and B≈C does not always mean A≈C. Using a stricter threshold for transitive links, or a graph algorithm that detects weak bridges, prevents runaway merges.

Stable Keys and Golden Records

Assign each resolved entity a stable internal ID and maintain a golden record: the best, most complete representation of the entity, with per-field provenance to record which source supplied each value. Keep the original records too—never destroy raw data during resolution, because your match thresholds will change and you may need to re-resolve.

Ongoing Maintenance

Resolution is not a one-time job. New records arrive with every scrape, and old entities split when a company rebrands or a product line is renamed. Build an incremental pipeline: match new records against the existing entity graph, flag merges and splits for review, and keep an audit trail so you can explain why two records were linked. The entity graph becomes a durable asset—the thing your downstream analytics actually depend on.

Evaluating Resolution Quality

You cannot improve what you do not measure. Build a small labeled sample: a few hundred pairs a human has marked match or no-match. Use it to tune thresholds and similarity weights, and to report two standard metrics. Precision answers "of the pairs we merged, how many were truly the same entity?" Recall answers "of the true matches, how many did we find?" Precision errors create false merges that corrupt analytics; recall errors leave duplicates. The right balance depends on the cost of each mistake—financial data usually favors precision, while deduping a marketing list can tolerate more recall. Re-measure after every rule change, because a tweak that raises recall often quietly lowers precision. Without a labeled sample, threshold tuning is guesswork dressed up as engineering.