LinkageMaker
Resource center for ChoiceMaker practitioners and the expert-in-the-loop approach to record linkage.
What is LinkageMaker?
LinkageMaker is an independent, educational resource center for the people who put ChoiceMaker to work: the analysts who build its models, the reviewers who decide uncertain pairs, and the data stewards who answer for the quality of the result.
In ChoiceMaker, human judgment is part of the system. Models are trained on expert decisions, and uncertain pairs are routed to people for review. That work is a craft of its own, and LinkageMaker brings together clear explanations of how ChoiceMaker works, how organizations use it, and the practices that make record linkage accurate and responsible.
It is a community effort, built and funded independently, and modeled on the community sites that have grown up around open-source projects such as PostgreSQL.
ChoiceMaker at a glance
ChoiceMaker is machine-learning software for record linkage: it determines when records in one database, or across several, refer to the same person. It learns from the decisions of expert reviewers, runs inside an organization's own secure environment, and has served government agencies and research organizations for more than two decades.
Since 1998. First developed with the New York City Department of Health and Mental Hygiene to remove duplicate records from the Citywide Immunization Registry.
Trained by experts. Its matching models are trained on record pairs that an organization's own reviewers have marked as match, differ, or unsure, and are tested on hand-marked pairs they have never seen before going into use.
Human review where it counts. Two thresholds, set by each organization, sort every pair into Match, Differ, or Hold. Pairs on Hold are routed to people for review as part of everyday operation, not only during testing.
Built for scale. Blocking narrows the comparisons to plausible pairs, so matching stays practical on very large datasets (see How it works).
Works with existing systems. Connects to common SQL databases, including Oracle, Microsoft SQL Server, and PostgreSQL, as well as XML and flat files.
Open source. Released under the Eclipse Public License in 2010, so its methods are open to study, and its record-matching patents were placed with the Open Invention Network for royalty-free use.
The organization's own environment. Runs inside an organization's own secure infrastructure, so identifying information never has to leave it.
Transparent, not a black box. Every decision can be traced to the evidence behind it: which clues applied to a pair, and how they shaped the result, whether that result was a match or not. Teams do more than inspect the logic; they write it. Through ClueMaker, ChoiceMaker's built-in scripting language for writing comparison rules, they define how records are compared, and through the plugin framework they can add comparison methods of their own. Unlike many machine-learning systems, ChoiceMaker makes both the process and the result of record linkage fully explainable, auditable, and adjustable.
How it works
ChoiceMaker matches records the way an expert would, at a scale no person could. Records about the same person rarely look identical: names are misspelled, addresses change, and fields go missing. ChoiceMaker uses machine learning to judge when two records probably refer to the same person, and it sends the uncertain cases to people for review.
Step 1: Blocking. Comparing every record with every other quickly becomes impossible: one million records yield about 500 billion possible pairs. Blocking solves this by first sorting records into smaller groups, called blocks, whose members share a likely trait, such as a similar-sounding surname or the same birth year. ChoiceMaker then compares records only within each block, rather than across the entire dataset. A well-designed blocking scheme keeps nearly all true matches together while setting aside the vast majority of pairs that could never match.
Step 2: Scoring. Each pair of records that blocking puts side by side is then compared field by field. The comparisons are made by clues: small rules, each testing one kind of evidence. One clue might ask whether the first names agree exactly or only sound alike; another, whether the birth dates match or differ by a single transposed digit; another, whether the two addresses share a street. No single clue decides the outcome. The trained model combines all of them, giving each the weight it has learned from expert reviewers, and produces a probability that the two records describe the same person.
Step 3: Transitivity analysis. Scoring judges records two at a time, and decisions made one pair at a time can contradict each other. Suppose record A matches record B, and record B matches record C, but A and C look like two different people. Those three decisions cannot all be right. ChoiceMaker therefore checks the decisions within each group of related records against one another and, where they conflict, can place the records in a queue for human review. The goal is that every final cluster, the set of records meant to represent one person, is internally consistent, so that one person ends up as one person, and two people are never merged into one.
Three decisions. Each scored pair receives one of three recommendations. A pair with a high probability is recommended as a Match, one with a low probability as Differ, and one in between is placed on Hold for a person to review. The boundaries between them are two thresholds, a match threshold and a differ threshold, which each organization sets. ChoiceMaker Analyzer helps find the thresholds that keep errors within the organization's limits. A wider Hold range means more human review and fewer errors; a narrower one means more automation.
Learning from experts. Reviewers mark record pairs as match, differ, or unsure, and ChoiceMaker trains its model, a maximum entropy model, on those decisions. The model learns which evidence the reviewers rely on and how much weight each piece deserves, so it reproduces an organization's own judgment about its own data. Before a model is put into use, it is tested on hand-marked pairs it was not trained on, and Analyzer helps teams examine the pairs it gets wrong. As data sources or populations change, the model can be retrained on fresh decisions.
ClueMaker and comparison methods. Clues are written in ClueMaker, ChoiceMaker's scripting language. They draw on established comparison methods, each suited to a different kind of variation:
Soundex, NYSIIS, Metaphone, and Double Metaphone encode names by how they sound, so that "Smith" and "Smyth" are recognized as alike.
Edit distance counts the single-character changes needed to turn one value into another, which catches typing errors.
Jaro-Winkler scores spelling similarity, giving extra weight to values that share their first letters.
Value-frequency weighting gives more credit to agreement on rare values: two records sharing an unusual surname are stronger evidence of a match than two sharing a common one.
Built for very large datasets. ChoiceMaker Server applies trained models at high speed against an organization's own SQL databases, and a plugin framework lets developers add comparison methods and components. Benchmarks on datasets of millions of records will be published with their methods and test data.
Practitioners
ChoiceMaker's results depend on the people who train and review it, yet the hands-on side of record linkage receives little practical guidance. LinkageMaker is built to fill that gap.
Labeling for training. Choosing which record pairs to mark, so that a model learns from the cases that matter most.
Review guidelines. Writing decision rules clear enough that two reviewers looking at the same pair reach the same answer.
Managing the Hold queue. Keeping review workloads manageable, and knowing when a growing queue signals a model that needs retraining.
Measuring quality. Estimating missed matches and false matches, and reporting linkage quality to the people who rely on the results.
Lessons from the field. Accounts from practitioners of what worked, what did not, and why.
[Guides on each of these topics are in preparation.]
ChoiceMaker in practice
ChoiceMaker is used wherever records about people must be linked accurately and the cost of an error is high. A missed match leaves one person's history split across several records; a false match merges two people's histories into one. The uses below show where each kind of error matters most.
Public health registries. A registry collects records from hospitals, clinics, schools, and pharmacies over many years, and the same person arrives many times under slightly different names, addresses, or dates. Duplicates distort counts and can lead to services being missed or repeated; in an immunization registry, a person's history split across two records can look incomplete. ChoiceMaker began in exactly this setting, with New York City's Citywide Immunization Registry.
Child and family research. Children and families appear in records held by separate agencies, such as health, education, child welfare, and social services, each with its own identifiers. Linking those records lets researchers follow outcomes over time and across systems, which no single agency's data can show.
Population health studies. Policy questions often require joining datasets that were never designed to work together, such as health records with housing, income support, or education data. Accurate linkage at the individual level lets such studies examine outcomes person by person, rather than relying on aggregate totals.
Large database cleanup. Long-lived databases accumulate duplicates through data entry, system migrations, and the merging of record sources. Deduplication restores one record per person, in SQL, XML, or flat-file databases, at scale.
Frequently asked questions
What is ChoiceMaker?
Record linkage software that determines when records in a database, or across databases, probably refer to the same person. It uses machine learning trained on the decisions of an organization's own expert reviewers, and it sends uncertain cases to people rather than guessing.
Is it open source?
Yes. ChoiceMaker has been released under the Eclipse Public License since 2010, and its record-matching patents were placed with the Open Invention Network for royalty-free use. Anyone can study the methods behind its decisions.
Does an organization's data have to leave its own environment?
No. ChoiceMaker typically runs inside an organization's own secure infrastructure. That matters because the records it links usually contain identifying information, such as names, birth dates, and addresses.
What kinds of data can it work with?
SQL databases such as Oracle, Microsoft SQL Server, and PostgreSQL, as well as XML and flat files. Through ClueMaker and the plugin framework, comparisons can be tailored to the fields a dataset actually contains.
What do Match, Differ, and Hold mean?
They are the three recommendations ChoiceMaker gives each pair of records: the same person, different people, or uncertain. Two thresholds set by each organization decide which is which, and pairs on Hold are routed to a person for review.
Can anyone see why two records were matched?
Yes. Every decision can be traced to the clues that applied to the pair and how they shaped the result, whether the outcome was a match or not.
How does ChoiceMaker differ from other open-source linkage tools?
Many linkage tools estimate their models without labeled examples and use human review mainly to check accuracy. ChoiceMaker is built around expert judgment from the start: its models are trained on decisions an organization's reviewers have made, and uncertain cases are routed to people as part of everyday operation.
Community
LinkageMaker is a place for practitioners, and for anyone curious about ChoiceMaker, to ask questions and share what they have learned. Two programs are planned:
Discussion forum. Questions and answers on methods, models, and real linkage problems, kept searchable so that each answer helps the next person with the same question.
Talks and events. Presentations and webinars from people who use ChoiceMaker in their work, covering the problems they faced and how they solved them.
Members will also be able to contribute what they learn, such as worked examples, clues for common kinds of data, and lessons from their own projects, so that the resource center grows with the field.