Best method to deduplicate people records in Rails

Question

I am writing a rails app with a Person model that looks something like this:

  create_table "people", :force => true do |t|
    t.string   "first_name"
    t.string   "last_name"
    t.string   "email"
    t.datetime "created_at", :null => false
    t.datetime "updated_at", :null => false
  end

I have a two step process as follows:

1. Fill out person records, with the names of people. The names of people may have unkown duplicates, due to nicknames, etc. For example, "tim smith" and "timothy smith"
2. Query an API to get potential email address matches for those people.

After doing that processing, I could have data like:

record 1: first_name: tim last_name: smith email: tim.smith@sampleemail.com

record 2: first_name: timothy last_name: smith email: tim.smith@sampleemail.com

What's the best way in rails to model that those are duplicates?

UPDATE: CLARIFICATION

After step 2, I know how to find out that those two records are duplicates (i.e. the same person), my question is how to represent that in the model? Should I add a "duplicate_of_person_id" type field and put the id of the first record in that field in the second record? Is there a better way?

Answer 1

You could link all the records together. The first scheme that comes to mind is to keep the record with the lowest id as the winner and make all the dupes point to it. You could also do a has_and_belongs_to_many, which would involve a separate table where each record says that these two people are the same. The latter grows quadratically with the number of people, though.

Or, just copy all the information from the second into the first and delete the second.

Answer 2

Not 100% sure what you're asking for. If you just want to find duplicates, and, say, list them in an array, you could create a method like this:

# This isn't particularly efficient, but it should return an array in which
# each element is a list of duplicated people (assuming we define duplicates
# by doubled email addresses). 
def self.find_duplicates
  array = []
  self.each do |person|
   similar = self.find_by_email
   if similar.count > 1
    array << similar
   end
  end
  return array
end

If you don't want to allow for duplicates, just create a validation in your model:

validates :email, :uniqueness => true

be sure, before that, though, to make sure that emails are all in the same case. You could do something like this, again in the model:

before_validation :format_emails

def format_emails
  self.email = self.email.downcase
end