GoldenAnalysis

<!-- mcp-name: io.github.benseverndev-oss/goldenmatch --> <div align="center">

Golden Suite

Splink-beating entity resolution — Arrow-native, Rust-fast, zero-tuning — feeding a durable identity layer, so messy records from every source become stable golden entities with whole-record, Customer-360 provenance.

Zero-config matching that beats expert-tuned Splink head-to-head on messy customer records, in an Arrow-native, Rust-authoritative engine verified from a laptop CSV to a 100M-row dedupe in 9.2 minutes. The identities it produces live in a transaction-native control plane — stable entity_ids, per-field provenance, merge/split, and a tamper-evident audit log — one call away as a Customer 360. It even owns its primitives: byte-identical, faster-than-rapidfuzz / jellyfish / FAISS Rust kernels, not rented dependencies.

Python · TypeScript · SQL, at 4-decimal parity · native in Postgres + DuckDB · edge WASM · 70+ MCP tools · beats hand-tuned Splink · 100M rows in 9.2 min

<br>

<!-- Headline package: goldenmatch -->     

<!-- Quality / proof -->     

<!-- Reach -->    

<!-- Ecosystem -->     

</div>


<p align="center">_{<em>Pair drilldown in the web workbench: cluster members, field-level diff, and a one-line NL explanation per pair. <code>pip install goldenmatch[web]</code> then <code>goldenmatch serve-ui &lt;project&gt;</code>. <a href="https://github.com/benseverndev-oss/goldenmatch/wiki/Web-UI">More screenshots →</a></em>}</p>

<!-- README-callouts:start (auto-synced from packages/python/goldenmatch/CHANGELOG.md by scripts/sync_readme_callouts.py — edit the CHANGELOG, not this block) --> > v3.5.0 — New date scorer for date fields (#1858). jaro_winkler scores unrelated ISO birthdays 0.80+ (the fixed YYYY-MM-DD shape + shared digit alphabet dominate), so it can't tell a typo from a different person. The date scorer compares dates by Damerau-Levenshtein over the canonical digits — a typo scores 0.90, an unrelated date 0.00 — with a levenshtein fallback for non-ISO input. Cross-surface (Python, native kernel, TypeScript), and a preflight check warns when a name-oriented scorer sits on a date field. > > v3.4.0 — Embeddings are first-class on Fellegi-Sunter matchkeys. embedding and record_embedding field scorers now train (EM) and score end-to-end on the probabilistic path via the vectorized matrix — previously they raised Unknown scorer on both training and scoring. They are matrix-only, so a matchkey carrying one always runs vectorized, and the TUI now routes FS through the same native/vectorized selector. > > v3.3.0 — 3.3.0 — negative evidence on Fellegi-Sunter matchkeys. negative_evidence now works on type: probabilistic matchkeys as EM-learned __ne__ dimensions (no labels needed; penalty_bits as a fixed override), and the Splink migration upgrade pass gains a fan-out lever — a risk-gated NE suggestion plus cluster-guard tuning from your reference clusters. goldenmatch-native 0.1.15 scores NE in the Rust kernels (FS_SUPPORTS_NE; older wheels keep the pure-Python fallback automatically). <!-- README-callouts:end -->

---

What you get: the golden entity

Most entity-resolution tools hand you clusters and stop. GoldenMatch keeps going: it resolves messy records into a durable golden entity — one per real-world customer — that survives re-runs, carries provenance on every field, and answers "who is this, and where did each value come from?" in a single call.

• A stable identity, not a throwaway cluster id. Every entity gets a stable entity_id (UUIDv7) that persists across runs as new data arrives — records are absorbed, entities merge or split, but the id an entity earns is the id downstream systems can rely on. Run-local cluster numbers reshuffle on every run; these don't.
• Whole-record provenance. Every field of the golden record traces back to the source record that won it — which source, when it was last seen, and which survivorship strategy picked it — with the values it didn't pick kept visible, not silently dropped. Provenance is on the whole unified record, not just the match decision.
• Governed by construction. Conflicting values collapse to one best value by an explicit survivorship policy (most-complete · source-priority · most-recent · majority-vote); every identity change lands in an append-only event timeline; and the log is sealed with a hash-chained, tamper-evident audit that a reviewer (or the other language port) can independently verify.

customer_360(entity_id) composes it into one read — golden record, per-field provenance, every linked source record, the event timeline, and the entity's relationship neighborhood:

// customer_360("018f...c2a1")  — trimmed
{
  "entity_id": "018f2b7e-...-c2a1", "confidence": 0.97, "record_count": 3,
  "sources": ["salesforce", "billing", "support"],
  "golden_record": { "name": "Ada Lovelace", "email": "ada@analytical.io", "phone": "+1-555-0100" },
  "field_provenance": [
    { "field": "email", "value": "ada@analytical.io",
      "winning_source": "billing", "winning_record_id": "billing:8821",
      "conflicting_values": [ { "value": "ada@ada.dev", "source": "salesforce" } ] },
    { "field": "phone", "value": "+1-555-0100", "winning_source": "salesforce" }
  ],
  "timeline": [ { "kind": "created", "actor": "pipeline", "recorded_at": "2026-07-30T..." },
                { "kind": "absorbed_record", "reason": "matched billing:8821" } ],
  "relationships": [ { "other_entity_id": "018f...9d0e", "kind": "shares_address" } ]
}

What ships today vs. what's emerging. The identity spine is production-grade and in main: stable entity_ids, per-field provenance, survivorship, merge/split, the append-only log + audit chain, cross-channel stitching, the relationship overlay, and incremental resolution against a persisted index (a new record resolves without a full re-run). The customer_360() serving view above and the source-registry layer that keeps it fresh from live systems are the newer, actively-landing pieces (the source connectors — Snowflake, BigQuery, Salesforce, HubSpot — ship today; the registry that wires them into the spine is emerging) — see the Customer 360 design + ADR. We label the seam rather than blur it.

The golden entity lives in the control plane; the matching that builds it runs in the compute engine. That split is the next section.

---

One product, two engines

The golden entity above is produced by two engines that optimize for genuinely different things — and keeping them distinct is the architecture, not an implementation detail (ADR 0047).

flowchart LR
    src([source records])
    e360([golden entities · Customer 360])
    subgraph compute ["Identity Compute Engine — Arrow-native, Rust-authoritative"]
        match[block · score · cluster]
    end
    subgraph control ["Identity Control Plane — transaction-native state machine"]
        spine[stable ids · survivorship · merge/split · provenance · audit]
    end
    src --> compute -->|resolution batch + evidence| control --> e360
    control -.->|persisted index| compute

| | Identity Compute Engine | Identity Control Plane | |---|---|---| | Shape | Arrow at bulk boundaries, Rust-authoritative kernels | Transaction-native state machine (SQLite default · Postgres) | | Job | Block, score, cluster — throughput, vectorized, deterministic per run | Stable ids, survivorship, merge/split, provenance, append-only audit | | State | Stateless per call; measurement-driven kernelization | Durable, transactional, replayable, auditable | | Backends | DataFusion · Ray · Sail are replaceable execution backends, none synonymous with GoldenMatch | Storage backends conform to one externally-observable semantics |

Many surfaces, one answer. The same capabilities reach Python, edge-safe TypeScript (with an opt-in WASM backend running the same Rust kernels), SQL inside PostgreSQL and DuckDB, and MCP / REST / A2A — governed by specification + conformance, not copy-paste. There is one authoritative owner per capability; pure-Python / standalone-TS paths are classified, conformance-tested fallbacks. Where a boundary can't cross byte-for-byte, we measure and label it rather than claim parity.

Why a platform engineer should care:

• The compute layer isn't framework lock-in. It's Arrow-native and backend-replaceable — push the heavy matching to a query engine that plans, spills, and distributes (verified to 100M rows), without the identity state coming along for the ride.
• The identity layer is a real state machine, not a columnar rebuild-every-time batch — durable ids, transactional merge/split, provenance and audit as first-class operations you can integrate against.
• Behavior is consistent where it's shared — SQL, Python, and TypeScript track the same answers to a conformance spec, so the surface you build on isn't quietly inventing its own semantics.

---

Resolution that beats the expert

The identity layer is only as good as the matching underneath it — and the matching starts at zero config. dedupe_df(df) runs with no rules and no training data: it profiles the data, picks a defensible configuration, and returns golden records immediately. The config it chose comes back on result.config — inspectable, diffable, versionable. Never a black box.

• Beats the expert, out of the box. On messy customer records, the opt-in Fellegi-Sunter engine beats hand-tuned Splink head-to-head — historical_50k pairwise F1 0.778 vs 0.757, cluster B³ 0.844 vs 0.789, one shared evaluator, reproducible bake-off. Fuzzy, exact, probabilistic (Fellegi-Sunter), and LLM scorers, with EM-trained weights and calibrated scores.
• A healing loop, not a one-shot. Zero-config gets you most of the way; then every run checks a free unsupervised signal and, when there's headroom, attaches ranked, self-verified config tweaks to result.suggestions. Each is kept only if it doesn't worsen a health proxy — so a suggestion never makes results worse. dedupe_df(df, heal=True) applies and re-runs in one call. You close the gap to expert-tuned without being the expert.
• Privacy-preserving record linkage. Match across organizations without sharing raw data — Bloom-filter PPRL, 92.4% F1 on FEBRL4, with HMAC-salted encodings.
• Self-verifying. Every step runs preflight + postflight checks and returns an inspectable report instead of failing silently — the "advanced, never black-box" contract that makes an automated identity layer safe to build on.

Runs on unstructured input, too: extract records from PDFs and images, then resolve them like any other source (pip install goldenmatch[documents]).

---

Runs where your stack is

The engine and the identity layer reach your stack through the surface you already use — the same capabilities, governed by conformance (one product, two engines), not a thin re-implementation per surface.

• SQL-native, at parity. The same functions run inside PostgreSQL (pgrx extension) and DuckDB: dedupe · match · score · auto-config + telemetry · identity-graph reads · profiling · evaluate · Fellegi-Sunter scoring · GoldenFlow transforms. Resolve without moving data out of the warehouse.
• Python and edge-safe TypeScript. The full suite ships on npm alongside PyPI. The TS cores are dependency-free and node:-free (browsers, Cloudflare Workers, Vercel Edge, Deno); an opt-in WebAssembly backend (await enableWasm()) swaps in the same* pyo3-free Rust kernels the Python wheels and SQL UDFs use, with pure-TS as the byte-identical default.
• AI-native by default. Every package ships an MCP server, a REST API, and an A2A agent surface (70+ MCP tools across the suite), all exposing the same JSON telemetry shape across web, TUI, CLI, Postgres, DuckDB, and MCP.
• Pipeline-native. A dbt package (dedupe/match materializations, quality tests, identity-graph reads), a GitHub Action (fail PRs on data-quality regressions), and 13 drop-in Airflow DAGs (Deploy).
• Production paths. Postgres sync, daemon mode, lineage tracking, review queues.

Cross-language parity

Surface parity is not the same as handing any pipeline phase from one language to the other byte-for-byte. Each verdict below is measured by a conformance harness, not assumed:

| Boundary | Verdict | | --- | --- | | Identity graph DB | ✅ byte-safe + cryptographically cross-verifiable (a seal written by one toolkit validates under the other) | | score → cluster and the end-to-end split-run | ✅ byte-safe — reproduces the single-language run | | Cluster JSON · config YAML · Learning Memory · record_fingerprint | ✅ portable | | String scoring | 🟡 4-decimal tolerance — a pair on a threshold can flip (byte-identical only with the shared WASM scorer) | | Standardize / dates · embeddings · auto-config controller | 🟠 divergent — not byte-portable | | Distributed / Ray · document (VLM) ingest | ⛔ Python-only by architecture |

Rule of thumb: hand off at the cluster or identity boundary and it's seamless; don't split across standardize/dates, embeddings, or the controller and expect bit-exact reproduction. Full detail + the runnable harness that keeps these verdicts honest: Cross-language parity & phase-handoff limits.

---

The suite — the pipeline into the spine

GoldenMatch is the headline, but resolution is only as good as what feeds it. Five sibling tools clean, standardize, and map records before they reach the identity layer — each stands alone, but they compose into one pipeline, orchestrated declaratively by GoldenPipe:

flowchart LR
    raw([raw rows])
    golden([golden entities])
    subgraph orchestration ["GoldenPipe orchestrates"]
        direction LR
        infermap[InferMap] --> goldencheck[GoldenCheck] --> goldenflow[GoldenFlow] --> goldenmatch[GoldenMatch]
    end
    raw --> infermap
    goldenmatch --> golden

| Package | Lang | Role in the pipeline | Install | |---|---|---|---| | InferMap | Python · TS | Schema mapping — auto-aligns columns across heterogeneous sources | pip install infermap · npm i infermap | | GoldenCheck | Python · TS | Data-quality scanning — encoding, format validation, anomaly detection | pip install goldencheck · npm i goldencheck | | GoldenFlow | Python · TS | Transforms & standardizers — phone, date, address, categorical | pip install goldenflow · npm i goldenflow | | GoldenMatch | Python · TS | Zero-config entity resolution → the identity spine. Headline package. | pip install goldenmatch · npm i goldenmatch | | GoldenAnalysis | Python · TS | Analysis & reporting — any stage's artifacts → a unified AnalysisReport + cross-run regression detection | pip install goldenanalysis · npm i goldenanalysis | | GoldenPipe | Python · TS | Orchestrator — declarative YAML wiring the steps | pip install goldenpipe · npm i goldenpipe | | golden-suite | Python | One-line meta-install: the whole suite + native acceleration | pip install golden-suite |

The deepest docs live in packages/python/goldenmatch/README.md (~1,300 lines: full feature list, CLI, architecture, benchmarks).

Owned libraries (standalone)

The suite owns its string-matching primitives instead of renting them — byte-identical drop-in replacements, published on their own so they're usable outside the suite too.

| Library | Replaces | What it is | Install | |---|---|---|---| | goldenfuzz | rapidfuzz | Fuzzy-string scorers + the full fuzz.* composite family + one-vs-many extract/cdist. Byte-identical (oracle-fuzzed), faster on short strings. | pip install goldenfuzz · cargo add goldenfuzz-core | | goldenphonetic | jellyfish | Phonetic encoders — soundex / metaphone / nysiis / match-rating. Byte-identical (6,000-input + 2,500-pair fuzz corpus), pure-Rust zero-dep. | pip install goldenphonetic · cargo add goldenphonetic-core | | goldenmatch-hnsw | FAISS IndexHNSWFlat | Pure-Rust HNSW approximate-nearest-neighbor index (zero C deps) — powers embedding-based blocking across Python, Rust, and TS/WASM. | pip install goldenmatch-hnsw |

Knowledge graphs

Entity resolution is the stage most GraphRAG pipelines do worst — duplicate surface forms of one entity scatter across documents. Two packages put GoldenMatch's resolution there:

| Package | What it does | Status | |---|---|---| | goldenmatch-kg | Drop-in GoldenMatch resolution as the ER stage of existing KG frameworks (neo4j-graphrag, LlamaIndex, Graphiti). | in-repo · not published (by design) | | goldengraph | Build-your-own-KG from text — text → LLM extraction → GoldenMatch resolution → durable bi-temporal store. Rust engine; ER is the differentiator. | in-repo · first PyPI release pending |

Measured, not asserted (ER-KG-Bench): resolution scores F1 0.602 on the labelled set, ahead of Neo4j-KGBuilder (0.456), neo4j-graphrag (0.403), and MS-GraphRAG / LightRAG / Cognee / mem0 (0.066). A resolved graph also does two things passage-window RAG structurally can't — exact aggregation (size-invariant where RAG recall collapses 0.99 → 0.64 across cluster-size buckets) and temporal as-of (1.000 vs 0.002 on past-date queries).

---

Scale & benchmarks

Every headline number maps back to a single committed runner (scripts/run_benchmarks.py); see docs/reproducing-benchmarks.md for per-number commands, dataset URLs, and expected output with tolerance.

• Accuracy on customer-shaped data — NC Voter 0.9719 F1, Febrl3 0.9443 F1, DQbench composite 91.04; the opt-in Fellegi-Sunter path beats hand-tuned Splink head-to-head on every dataset Splink scores (bake-off). (Bibliographic DBLP-ACM lands 96.4% F1 for the record-linkage crowd, but customer identity is the focus.)
• Privacy-preserving — PPRL 92.4% F1 on FEBRL4, matching across parties with no shared raw data.
• Scale envelope (docs/scale-envelope.md) — per-backend ranges (in-memory/bucket to a few M · DuckDB out-of-core to ~50M · Ray distributed ≥ 50M), block-size failure modes, and a decision tree for picking a backend.

Verified at the top end: a full 100M-row dedupe on a 5-node Ray cluster in 9.2 min (554 s), 20,000,000 golden records recovered exactly, driver peak 0.36 GB RSS. The default distributed path is recall-complete — duplicates merge correctly no matter how the input is partitioned (blocking-key shuffle scoring + distributed randomized-contraction WCC), and it stays driver-collect-free end to end. Recipe: configs/distributed-100m.yaml.

Three reproducible real-world pipelines run this on public data at scale:

• shell-company-network — investigative ER across ICIJ Offshore Leaks + OpenSanctions + GLEIF + UK PSC. −62.5% analyst-hours to triage vs single-source baselines.
• vuln-attribution — 6,126,895 OSS-vulnerability records → 847,475 canonical vulns across 40 sources in ~5 minutes on one 64GB runner.
• sanctions-reconciliation — cross-list coverage on 85 public sanctions lists across 50+ jurisdictions.

---

Install & quick start

Dedupe a CSV in 30 seconds — zero config, writes <timestamp>_golden.csv:

pip install goldenmatch && goldenmatch dedupe customers.csv

import goldenmatch as gm

result = gm.dedupe("customers.csv")               # zero-config
print(result)                                     # DedupeResult(records=5000, clusters=847, match_rate=12.0%)
result.golden.write_csv("deduped.csv")

result = gm.dedupe("customers.csv",               # or be explicit
    exact=["email"], fuzzy={"name": 0.85, "zip": 0.95}, blocking=["zip"], threshold=0.85)

import { dedupe } from "goldenmatch";           // edge-safe: browsers, Vercel Edge, Workers, Deno
const result = dedupe(rows, { fuzzy: { name: 0.85 }, blocking: ["zip"], threshold: 0.85 });

The whole suite, configured for speed — golden-suite pulls in every package plus the native (Rust) kernels, pinned and defaulted to the perf-optimized config. Native wheels are hard dependencies on purpose: a platform without a wheel fails loudly rather than silently running the slow pure-Python path.

pip install golden-suite
golden-suite doctor        # verify every package + native kernel is importable and healthy
golden-suite optimize      # repair / re-enable the perf-optimized config

pip install golden-suite[mcp]     # + aggregator MCP server (every tool, one endpoint)
pip install golden-suite[all]     # everything

Just GoldenMatch — fat optional extras, pay only for what you use (native acceleration is default on common platforms):

pip install goldenmatch                    # core (CSV in, CSV out) + native
pip install goldenmatch[documents]         # + PDF/image ingest (resolve unstructured input)
pip install goldenmatch[embeddings]        # + sentence-transformers, FAISS
pip install goldenmatch[llm]               # + Claude / OpenAI for LLM boost
pip install goldenmatch[ray]               # + Ray distributed backend (50M+ rows)
pip install goldenmatch[postgres]          # + Postgres sync  (also: [snowflake] [bigquery] [databricks] [salesforce])
pip install goldenmatch[mcp]               # + MCP server     (also: [agent] A2A, [web] browser workbench)

Web workbench — pip install 'goldenmatch[web]' then goldenmatch serve-ui my-project (opens http://localhost:5050): edit rules with live validation, preview against a sampled slice, label pairs (mirrored into Learning Memory), compare runs.

More: examples/ — Python (quickstart, full pipeline, customer 360, PPRL, MCP client) · TypeScript (quickstart, Vercel Edge, MCP client) · Airflow.

---

Deploy

Remote MCP (nothing to install) — hosted on Smithery; connect any MCP client:

{ "mcpServers": { "goldenmatch": { "url": "https://goldenmatch-mcp-production.up.railway.app/mcp/" } } }

Containers — every package ships as a multi-arch image (linux/amd64 + arm64) on GHCR, pull anonymously:

docker run -p 8300:8300 ghcr.io/benseverndev-oss/goldensuite-mcp:latest   # one container, every tool
docker run -p 8200:8200 ghcr.io/benseverndev-oss/goldenmatch-mcp:latest   # per-package (also goldencheck/goldenflow/goldenpipe/infermap)
docker run -e POSTGRES_PASSWORD=secret ghcr.io/benseverndev-oss/goldenmatch-extensions:latest   # Postgres + extension

Airflow — 13 drop-in DAGs at examples/airflow/ (TaskFlow API, Airflow 2.7+ / 3.x; idempotent, marker-protected), grouped by lifecycle stage:

| Group | DAGs | |---|---| | Core pipeline | daily_dedupe, incremental_match, warehouse_native (Snowflake), customer_360, identity_graph | | Privacy | pprl_linkage (two-party PPRL) | | Onboarding & monitoring | schema_align_and_load, schema_drift_alarm, quality_gate | | Feedback loop | review_worker, active_learning | | Operationalize | reverse_etl (Salesforce/HubSpot), backfill |

---

Repository layout

goldenmatch/
├── packages/
│   ├── python/        goldenmatch · goldencheck · goldenflow · goldenpipe · infermap · goldenanalysis
│   │                  goldensuite-mcp (aggregator) · golden-suite (meta) · goldengraph · goldenmatch-kg
│   ├── typescript/    full TS ports (edge-safe cores + WASM) · goldencheck-types
│   ├── rust/extensions/  Postgres pgrx + DuckDB UDFs + native kernels + owned libraries (own Cargo workspace)
│   ├── dbt/goldensuite/  dbt materializations, tests, macros
│   └── actions/goldencheck/  GitHub Action
├── examples/          python · typescript · airflow (drop-in DAGs)
├── context-network/   architecture decisions + design docs (ADRs, the two-engine frame, Customer 360)
├── docs/superpowers/  design specs and implementation plans
└── justfile · pyproject.toml (uv workspace) · pnpm-workspace.yaml (Turborepo) · .github/workflows/ci.yml

• Cargo — no root workspace. packages/rust/extensions/ is itself a Cargo workspace (the postgres crate is excluded for pgrx); Cargo commands run from inside it.
• TypeScript — one pnpm workspace. packages/typescript/* form a single pnpm + Turborepo workspace.

just install   # uv sync + per-package npm install + cargo fetch
just test      # all languages   ·   just lint   ·   just build

Contributing

• Feature work on feature/<name> branches; merge via squash PR. Titles: feat: / fix: / docs:.
• Tests must pass on all three languages where the change applies; the parity harness in packages/typescript/goldenmatch/tests/parity/ enforces 4-decimal Python ↔ TypeScript scorer parity.
• Architecture changes conform to (or amend) the one-product-two-engines frame in the same PR. Design rationale lives in context-network/decisions/ and docs/superpowers/specs/.

corepack enable                               # one-time, picks up pnpm@9.15.0
pnpm install
pnpm turbo run build test typecheck           # full pipeline (cached after first run)

Windows: enable Developer Mode so pnpm install can create symlinks; if corepack enable needs admin, npm i -g pnpm@9.15.0 is equivalent.

---

_{This repo was formed on 2026-05-01 by folding 8 sibling repos into goldenmatch via git filter-repo (full history preserved). Built by Ben Severn. MIT — see LICENSE.}