-4

I have a vCard file with records for thousands of contacts. This file has been corrupted and copies of personal phone, work, and extra records have been added for each of the users.

How could I clean up duplicates?

BEGIN:VCARD
VERSION:3.0
N:Doe;John;Q.,Public
FN;CHARSET=UTF-8:John Doe
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=HOME,VOICE:(404) 555-1212
TEL;TYPE=HOME,VOICE:(404) 555-1212
TEL;TYPE=HOME,VOICE:(404) 555-1212
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=HOME,VOICE:(404) 555-1212
TEL;TYPE=HOME,VOICE:(404) 555-1212
TEL;TYPE=HOME,VOICE:(404) 555-1212
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
ADR;TYPE=HOME:;;42 Plantation St.;Baytown;LA;30314;United States of America
URL:https://www.google.com/
PHOTO;VALUE=URL;TYPE=PNG:http://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/Example_svg.svg/200px-Example_svg.svg.png
AGENT:BEGIN:VCARD
 VERSION:3.0
 N:Doe;John;Q.,Public
 FN:John Doe
 TEL;TYPE=WORK,VOICE:(111) 555-1212
 TEL;TYPE=HOME,VOICE:(404) 555-1212
 TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
 EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
 EMAIL;TYPE=INTERNET:example@example.com
 PHOTO;VALUE=URL;TYPE=PNG:http://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/Example_svg.svg/200px-Example_svg.svg.png
 END:VCARD
END:VCARD

I have used the following solution seen in StackOverflow, but it has not solved the problem since not all duplicates appear consecutively.

perl -ne 'print unless (defined($prev) && ($_ eq $prev)); $prev=$_'

resulting in:

...
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
TEL;TYPE=WORK,VOICE:(111) 555-1212
TEL;TYPE=HOME,TYPE=VOICE:(404) 555-1213
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=INTERNET:example@example.com
EMAIL;TYPE=PREF,INTERNET:forrestgump@example.com
EMAIL;TYPE=INTERNET:example@example.com
Borodin
  • 126,100
  • 9
  • 70
  • 144
Ravai
  • 1
  • 1

1 Answers1

3

The easiest way to delete all duplicate lines is

perl -ne 'print if !$seen{$_}++'

If you want to treat each BEGIN:VCARD section separately,

perl -ne '%seen = () if /\bBEGIN:VCARD\b/; print if !$seen{$_}++'
melpomene
  • 84,125
  • 8
  • 85
  • 148
  • Thanks melpomene, the problem is that there is n BEGIN:VCARD, therefore, it will erase all the BEGIN:VCARD and other labels that are repeated. How could I execute the command for each BEGIN. – Ravai Jun 10 '18 at 20:02
  • Thanks, although with this script it deletes all the tags "BEGIN:VCARD" – Ravai Jun 11 '18 at 12:57
  • @Ravai Oh, hmm. Try the new version. – melpomene Jun 11 '18 at 16:30