2

This question is related to Tools for matching name/address data. There is a number commercial tools provided by SAS, Oracle, Microsoft, etc., that allow to de-duplicate or merging names of individuals or companies coming from multiple sources.

However, after reading the answers to the question mentioned before, I wondered why a seemingly interesting problem didn't receive any answers mentioning open source projects that could tackle the problem.

Are you aware of any open source projects or algorithms to implement the so called "record linking", "record merging", or "clustering"?

Community
  • 1
  • 1
luiscolorado
  • 1,525
  • 2
  • 16
  • 23
  • @luis: you get what you pay for. – John Saunders May 10 '10 at 18:56
  • 1
    John, In this particular case it may be true (and that's what I'm trying to find out). However, in general I have to disagree. After all, ain't some of the best things in life free? I.e., sun, air, etc.? 20 years ago, who would have dreamed of a powerful operating system that would cost you nothing? – luiscolorado May 10 '10 at 20:34
  • 2
    @luis: it may not have cost _you_ anything, but it cost _somebody_ something. – John Saunders May 10 '10 at 21:30
  • 1
    Try DataExplorer from appliedalgo.com, support inexact record matching. While key that links data from left side and right side must be exact, Data Explorer allow user to specify inexact field criteria/threshold when it try to determine one cell on the left side is "Different" from corresponding cell on right side. For numerical values, it's simple % numerical diff. For string, they run some sort of fuzzy string similarity measures in the back. – Swab.Jat Jan 02 '14 at 09:34
  • Here, http://www.youtube.com/watch?v=Se0wt7mSkxM – Swab.Jat Jan 02 '14 at 11:41

3 Answers3

4

I'd recommend Google Refine as an open source (New BSD license) tool for parsing and fixing crufty data. It also allows clustering and reconciling of duplicate data, as well as having data-mining features.

I've used it to import and fix a lot of data in various formats, .csv, .tsv, .xls, .xml, .json, .rdf etc. with success. It can be used in-house without sending any data externally, which seemed to be a concern of the question "tools for matching name/address data"

NB. Google Refine was previously called Freebase Gridworks.

Community
  • 1
  • 1
Iain Sproat
  • 5,210
  • 11
  • 48
  • 68
0

Try OSDQ open source data quality and profiling project on sourceforge

aashisha
  • 1
0

I stumble upon the following article: "Merge/Purge and Duplicate Detection".

By looking at http://www.semaphorecorp.com I found some extremely low prices.

This is not what I'm looking for, but at least is a bit of help, and a step on the right direction.

luiscolorado
  • 1,525
  • 2
  • 16
  • 23