Entity Resolution for Scraped Company Data — One Company, Every Source, Zero Lost Evidence

The Problem Multiplies With Every Source

For the market intelligence platform I've written about before, company records flow in from 27 distinct sources: directory scrapers, an OCR'd print directory, trade-show exhibitor lists, OpenStreetMap, government registries, enrichment APIs, the client's own CRM export. Every source sees the same market through its own keyhole. The same manufacturer shows up as "Acme Baking Co.", "Acme Baking Company, Inc.", and "ACME BAKING" — at its headquarters in one source, at a plant address in another, under a brand name in a third.

If you don't solve this, nothing downstream works. Your lead count is fiction, your sales team calls the same company three times under three names, and enrichment spend triples because every duplicate looks like a fresh target. The platform's current store holds 97,258 raw sightings resolved into 19,100 company rows — roughly five sightings per real company — and that five-to-one ratio is what entity resolution is fighting.

The Core Model: Sightings Are Not Companies

The architecture decision everything else hangs on: separate what a source said from what you believe. Every scrape result is a sighting — an immutable row recording that source X saw company-shaped data Y, with the full raw record preserved as JSON. Sightings resolve onto canonical company rows — one per real-world company — but the sighting itself is never modified, merged away, or deleted.

This buys you three things. Auditability: any field on any company can be traced to every source that ever asserted it. Reversibility: a bad merge can be unwound because the evidence still exists. Corroboration as signal: a company seen independently in six sources is real in a way a single-source row isn't. When the OCR'd print directory ran, one bakery came out carrying provenance rows from both the 2024 and 2026 editions on a single company row — same company, two years of independent confirmation, zero duplicates.

The merge rule is deliberately conservative: fill blanks only. A new sighting adds fields the canonical row is missing; it never overwrites a field that's already populated. Overwrite-on-merge means your best data gets clobbered by your worst source, and you won't even know which source did it.

The Match Ladder

Matching runs at ingest time — every incoming sighting walks a ladder of rules, strictest first:

Normalized domain. The strongest identity signal a company has. Two records pointing at the same website are the same company, full stop — after stripping www, protocol, and tracking junk. One critical exclusion: social and directory domains (facebook.com, linkedin.com, yelp.com) are never accepted as identity. Hundreds of small companies list their Facebook page as their website; treating that as identity would merge them all into one monster record.
Normalized name + state. Legal-suffix stripping, case folding, punctuation normalization — then match within the same state. With a tripwire: if the name and state match but the two records claim different domains, that's not a match, it's a conflict — the pair goes to a review queue instead of merging. Same-name different-domain is exactly what franchises and coincidentally-named companies look like.
Name-only, heavily guarded. Only for records with no domain and no usable location — and if one name spans two or more domains across the store, it's flagged as a corporate/brand split and never auto-merged.
Fuzzy name + ZIP. The last automated rung, for near-miss spellings anchored to the same place.
Create. Nothing matched — a genuinely new company enters the store.

Anything ambiguous lands in a review state rather than guessing. Review rows are quarantined from every downstream surface — they don't appear in lead lists, exports, scoring, or the map until a human promotes them. An uncertain merge that ships to a sales team is worse than a missing record.

Contacts Deserve the Same Discipline

People records duplicate even harder than companies — the same plant manager arrives from a directory listing, an enrichment API, and a website scrape, with three different email fields. The same no-loss principle applies with person-shaped keys: named contacts merge on name + title; anonymous rows merge on email, then phone. And verification status only moves upward — a contact marked valid by a verification pass can never be demoted back to unverified by a later, dumber source re-asserting the same email. One cleanup pass under these rules removed 2,668 duplicate contact rows from the store without losing a single verified email.

The War Story: Counting Is Genuinely Hard

Here's the bug class nobody warns you about. After ingesting 906 records from the print-directory run, I asked the obvious question — how many new companies did this source find? — and got three different answers from three reasonable queries. The ingest counter said 11. A "companies with more than one source" query said 51. The correct answer was 53.

All three were "right" about something. The ingest counter only credits the first record of a multi-record new company, so it undercounts. The multi-source query overcounts in the other direction, because a company sighted twice in the same run looks multi-source — one manufacturer appeared five separate times in a single edition, once per state page it was listed under. The correct query asks: does this company have any provenance from a different source? That's the one that separates "614 companies we already knew" from "53 genuinely new discoveries."

The lesson isn't the specific queries. It's that in any dedup system, every aggregate number has a precise definition hiding under it, and adjacent definitions diverge by 5x. Reconcile the counts against each other before you put any of them in front of a client — and when they disagree, the disagreement is usually pointing at a real property of the pipeline, not a bug in the query.

A Sharp Edge Worth Stealing the Fix For

One subtle defect survived multiple audits: the dedup key for a sighting was computed from the company name as originally parsed — before a later verification pass repaired OCR damage in it. Same company, printed cleanly on one page and garbled on another, produced two different keys and two sightings in one run. The store stayed correct, because ingest's own name-and-state matching folded both sightings onto one company — but the run counters were inflated, and inflated counters are how you end up trusting a number that's quietly wrong. The general rule: compute identity keys after repair, not before — or accept that your in-run counters are estimates and treat the store as the only authority.

Why This Pays For Itself

Every downstream system inherits the guarantees. Credit-gated enrichment never pays twice for a company it already knows under another name. Scoring operates on complete records assembled from six partial views. The sales team's map shows each company once. And when someone asks "why does the system think this company has 53 plants?" the answer is a provenance row with a source and a raw record, not a shrug.

The pattern applies to any multi-source data problem — companies, products, properties, people. Keep sightings immutable, resolve onto canonical rows, match down a ladder from strong identity to fuzzy, fill blanks instead of overwriting, quarantine ambiguity, and audit your counts against each other. The data collection service page covers how I scope these systems; the case study shows this one in production.