3

I would like to import my contacts into Gnome Contacts via Python or command-line. I am willing to do some programming, but I am unable to find the necessary documentation.

Gnome Contacts uses the libfolks library, which includes a folks-import utility, but there is no man page and the output of folks-import --help is not very helpful:

Usage:
  folks-import [OPTION…] — import meta-contact information to libfolks

Help Options:
  -h, --help            Show help options

Application Options:
  -s, --source=name     Source backend name (default: ‘pidgin’)
  --source-filename     Source filename (default: specific to source backend)

Contacts are stored locally in ~/.local/share/evolution/addressbook/system/contacts.db which is a SQLite3 database, so I could write directly to this file. However I would prefer not to without good documentation, particularly as the database appears to embed binary and vCard data.

On Ubuntu 18.04, you can install Gnome Contacts and libfolks tools via:

sudo apt install gnome-contacts folks-tools

You can inspect the local contact database via:

sqlite3 ~/.local/share/evolution/addressbook/system/contacts.db .dump

Related answers that did not solve my problem:

bitinerant
  • 1,168
  • 7
  • 24
  • I was just wondering, from where do the contacts coming from? What is the format of the contact information? – Kevin Ng Nov 02 '19 at 23:22
  • @KevinNg - as described above, the contacts are stored in `~/.local/share/evolution/addressbook/system/contacts.db` which is a SQLite database with what appears to be embedded vCard data. Run the above `sqlite3` command to view it. – bitinerant Nov 03 '19 at 10:18
  • From a quick google search I found that Gnome Contacts shares information with Evolution Data Server, so you can update it via Evolution email client and let Contacts sync themselves. Here is the [link](https://askbot.fedoraproject.org/en/question/83711/how-to-import-contacts-to-gnome-contacts/). I personally haven't tested this so I'm not sure if it'll work. – curiousMinded Nov 03 '19 at 21:59
  • Also [here](https://developer.gnome.org/eds/stable/) is the reference manual for the Evolution Data server which you might find useful as it contains documentation for programming interfaces that interact with contacts among other things. – curiousMinded Nov 03 '19 at 22:13
  • @curiousMinded - thanks! I'll check those out. By the way, what was your Google search? I was apparently not searching for the same thing. – bitinerant Nov 04 '19 at 08:43
  • I searched for _import data into gnome contacts_ and then for _evolution data server_. – curiousMinded Nov 04 '19 at 08:51

1 Answers1

0

In my eyes, besides using Evolution there are two ways to import the contacts. There is no automatic tool to do this without the Evolution.

Make a backup of your contacts.db before making any changes!

  1. The first way is the easy way using folks-import to import your contacts. I have checked the source and it appears to be a importer from pidgin's blist.xml file.

    • blist.xml - A local copy of your buddy lists, for use in keeping locally applied aliases and group and buddy ordering between. I have tried it, out of curiosity, on my blist.xml and it worked for some contacts.

For you to create a blist.xml (here is a blist.h and the source for it) you could use somebody's work like vcftoxml.py and adjust it.

Edit - trying to create a dummy xml to be imported

I have tried to create simple xml (blist.xml) to be imported via the the importer.

The xml inspired by blist.xml example:

<?xml version="1.0" encoding="UTF-8" ?>
<purple version="1.0">
<blist>
  <group name='GroupName'>
  <contact>
    <buddy account='test' proto='prpl-hangouts'>
      <name>1131313213446</name>
      <alias>Jakto Tomos</alias>
      <phone>+6568309312</phone>
      <email>testing@yahoo.com</email>
      <address>Berlin</address>
    </buddy>
  </contact>
  </group>
</blist>
<privacy>
</privacy>
</purple>

When I checked the created contact I have found out that only the alias name was imported.

So I have create blist.xml which mimics the real-life one:

<?xml version="1.0" encoding="UTF-8" ?>
<purple version="1.0">
<blist>
  <group name='GroupName'>
  <contact>
    <buddy account='test' proto='prpl-hangouts'>
      <name>1131313213446</name>
      <alias>Jakto Tomos</alias>
      <setting name='phone' type='string'>+6568309312</setting>
      <setting name='email' type='string'>testing@yahoo.com</setting>
      <setting name='address' type='string'>Lesser strasse, Berlin</setting>
    </buddy>
  </contact>
  </group>
</blist>
<privacy>
</privacy>
</purple>

When I imported it again I got again only the alias name. So I went to the import-pidgin.vala source code and I have found out that it indeed imports only the name (contact ID) and alias as name:

      ...
      /* Parse the <name> and <alias> elements beneath <buddy> */
      for (Xml.Node *subiter = iter->children; subiter != null;
          subiter = subiter->next)
        {
          if (subiter->type != ElementType.ELEMENT_NODE)
            continue;

          if (subiter->name == "alias")
            alias = subiter->get_content ();
          else if (subiter->name == "name")
            {
              /* The <name> element seems to give the contact ID, which
               * we need to insert into the Persona's im-addresses property
               * for the linking to work. */
              string im_address = subiter->get_content ();
              im_addresses.set (tp_protocol,
                  new ImFieldDetails (im_address));
              im_address_string += "    %s\n".printf (im_address);
            }
        }
       ...

So this way is feasible but you have to add parsing of the nodes you want to import. You could make a PR for such patch so others after you could use it (I recommend using the <setting name... notation as this is how the blist.xml file is formatted.

  1. The second way is way harder and you have to correctly populate the tables in the sqlite. This is the DB structure:
    BEGIN TRANSACTION;
    CREATE TABLE IF NOT EXISTS "folder_id_phone_list" (
        "uid"   TEXT NOT NULL,
        "value" TEXT,
        FOREIGN KEY("uid") REFERENCES "folder_id"("uid")
    );
    CREATE TABLE IF NOT EXISTS "folder_id_email_list" (
        "uid"   TEXT NOT NULL,
        "value" TEXT,
        FOREIGN KEY("uid") REFERENCES "folder_id"("uid")
    );
    CREATE TABLE IF NOT EXISTS "folder_id" (
        "uid"   TEXT,
        "Rev"   TEXT,
        "file_as"   TEXT,
        "file_as_localized" TEXT,
        "nickname"  TEXT,
        "full_name" TEXT,
        "given_name"    TEXT,
        "given_name_localized"  TEXT,
        "family_name"   TEXT,
        "family_name_localized" TEXT,
        "is_list"   INTEGER,
        "list_show_addresses"   INTEGER,
        "wants_html"    INTEGER,
        "x509Cert"  INTEGER,
        "vcard" TEXT,
        "bdata" TEXT,
        PRIMARY KEY("uid")
    );
    CREATE TABLE IF NOT EXISTS "keys" (
        "key"   TEXT,
        "value" TEXT,
        "folder_id" TEXT,
        PRIMARY KEY("key"),
        FOREIGN KEY("folder_id") REFERENCES "folders"
    );
    CREATE TABLE IF NOT EXISTS "folders" (
        "folder_id" TEXT,
        "version"   INTEGER,
        "multivalues"   TEXT,
        "lc_collate"    TEXT,
        "countrycode"   VARCHAR(2),
        PRIMARY KEY("folder_id")
    );
    INSERT INTO "keys" VALUES ('eds-reserved-namespace-is-populated','1','folder_id');
    INSERT INTO "keys" VALUES ('revision','2019-11-04T07:28:25Z(0)','folder_id');
    INSERT INTO "folders" VALUES ('folder_id',11,'email;prefix:phone','en_US.UTF-8','US');
    CREATE INDEX IF NOT EXISTS "UID_INDEX_phone_folder_id" ON "folder_id_phone_list" (
        "uid"
    );
    CREATE INDEX IF NOT EXISTS "INDEX_email_folder_id" ON "folder_id_email_list" (
        "value"
    );
    CREATE INDEX IF NOT EXISTS "UID_INDEX_email_folder_id" ON "folder_id_email_list" (
        "uid"
    );
    CREATE INDEX IF NOT EXISTS "SINDEX_family_name_folder_id" ON "folder_id" (
        "family_name_localized"
    );
    CREATE INDEX IF NOT EXISTS "INDEX_family_name_folder_id" ON "folder_id" (
        "family_name"
    );
    CREATE INDEX IF NOT EXISTS "SINDEX_given_name_folder_id" ON "folder_id" (
        "given_name_localized"
    );
    CREATE INDEX IF NOT EXISTS "INDEX_given_name_folder_id" ON "folder_id" (
        "given_name"
    );
    CREATE INDEX IF NOT EXISTS "INDEX_full_name_folder_id" ON "folder_id" (
        "full_name"
    );
    CREATE INDEX IF NOT EXISTS "INDEX_nickname_folder_id" ON "folder_id" (
        "nickname"
    );
    CREATE INDEX IF NOT EXISTS "SINDEX_file_as_folder_id" ON "folder_id" (
        "file_as_localized"
    );
    CREATE INDEX IF NOT EXISTS "INDEX_file_as_folder_id" ON "folder_id" (
        "file_as"
    );
    CREATE INDEX IF NOT EXISTS "keysindex" ON "keys" (
        "folder_id"
    );
    COMMIT;

You would then need to create a bunch of inserts that follow the structure logic like this:

INSERT INTO "folder_id_phone_list" VALUES ('pas-id-5DBFDF5A00000000','+0000000000');
INSERT INTO "folder_id_phone_list" VALUES ('pas-id-5DBFDF5A00000000','+131545678');
INSERT INTO "folder_id_phone_list" VALUES ('pas-id-5DBFE7F200000001','+45646546565465');
INSERT INTO "folder_id_email_list" VALUES ('pas-id-5DBFDF5A00000000','first_mail@gmal.com');
INSERT INTO "folder_id_email_list" VALUES ('pas-id-5DBFDF5A00000000','home_email@gmail.com');
INSERT INTO "folder_id_email_list" VALUES ('pas-id-5DBFE7F200000001','asdf@fasdf.com');
INSERT INTO "folder_id" VALUES ('pas-id-5DBFDF5A00000000','2019-11-04T08:28:52Z(15)','a header mail, this','001-)71)/1KA)9?O79M��',NULL,'john smith','this','020-O79M�','a header mail','001-)71)/1KA)9?',0,0,0,0,'BEGIN:VCARD

VERSION:3.0

UID:pas-id-5DBFDF5A00000000

X-URIS:www.testing.com

FN:John Smith

N:a header mail;This;is;;

X-EVOLUTION-FILE-AS:a header mail\, This

REV:2019-11-04T08:28:52Z(15)

TEL;TYPE=VOICE,HOME:+0000000000

TEL;TYPE=CELL:+131545678

EMAIL;TYPE=PERSONAL:first_mail@gmal.com

EMAIL;TYPE=HOME:home_email@gmail.com

ADR;TYPE=WORK:p.o. box;456;Stree work;New York;New York;456465;USA

ADR;TYPE=HOME:p.o. box;456;Street;City;New York State;13676;USA

BDAY:2019-11-04

PHOTO;VALUE=uri:file:///home/osboxes/.local/share/evolution/addressbook/sys

 tem/photos/pas_id_5DBFDF5A00000000_photo-file0.image%252Fpng

END:VCARD',NULL);
INSERT INTO "folder_id" VALUES ('pas-id-5DBFE7F200000001','2019-11-04T08:57:22Z(17)','sdaf','019-M/)3',NULL,'sdaf','sdaf','019-M/)3','','000-',0,0,0,0,'BEGIN:VCARD

VERSION:3.0

EMAIL;TYPE=PERSONAL:asdf@fasdf.com

TEL;TYPE=CELL:+45646546565465

ADR;TYPE=HOME:asdf;;asdfsdfasdf;;sadf;;asdf

FN:sdaf

N:;sdaf;;;

X-EVOLUTION-FILE-AS:sdaf

UID:pas-id-5DBFE7F200000001

REV:2019-11-04T08:57:22Z(17)

END:VCARD',NULL);
INSERT INTO "folder_id" VALUES ('pas-id-5DBFE7F800000002','2019-11-04T08:57:28Z(19)','next','014-C1WO',NULL,'next','next','014-C1WO','','000-',0,0,0,0,'BEGIN:VCARD

VERSION:3.0

FN:next

N:;next;;;

X-EVOLUTION-FILE-AS:next

UID:pas-id-5DBFE7F800000002

REV:2019-11-04T08:57:28Z(19)

END:VCARD',NULL);
INSERT INTO "keys" VALUES ('eds-reserved-namespace-is-populated','1','folder_id');
INSERT INTO "keys" VALUES ('revision','2019-11-04T08:57:28Z(20)','folder_id');
INSERT INTO "folders" VALUES ('folder_id',11,'email;prefix:phone','en_US.UTF-8','US');

What is important to note is the uid which is e.g. for the first record: pas-id-5DBFDF5A00000000 (which is defined as new HashMap<string, Field?>). I see it as fixed string pas-id-, then hex number 5DBFDF5A (appears a random one) and last is a counter 00000000 (next would 00000001).

I would personally go for the first option, but second is there if you would have issues creating the xml file. If you go with the manual inserts don't forget to be consistent with the data.

Edit - comment on the data exported out

It appears so attributes like file_as_localized, given_name_localized, and family_name_localized contain some "rubbish" data. However, these attributes are defined as TEXT so there should be some kind of readable data, but it is not (tried with different encoding but it did not help). I recommend reading the source code for these attributes. There maybe a possibility to enter empty value like it appears to be done on some account at family_name_localized with the 000- only. You can also try it with NULL value at the beginning and when you know it you can update it later on.

Conclusion

After performing the tests with the importer, I think the easier option is actually the option 2. The option 1 would include some programming in vala, if you do not want to import only names that is.

As for the SQLite tool I recommend DB Browser for SQLite where you can query your SQLite database, run queries and even edit by hand if needed.

tukan
  • 17,050
  • 1
  • 20
  • 48
  • 1
    I very much appreciate your time and research. It's too bad that importing data requires digging into the source code, but at least that *is* an option (thank you, open source). Option 1 may be what I end up with, but I would like to know more about your comment, "it worked for some contacts". Do you have any information about why it *didn't* work for some? – bitinerant Nov 05 '19 at 16:53
  • 1
    Option 2 makes me really uncomfortable because doing it imperfectly could lead to corruption which wasn't detected until later. As I said in my OP, the database seems embed binary data (which you didn't mention) and vCard data (which you mentioned, but I think vCard is a lot more complex than just substituting text into a template). – bitinerant Nov 05 '19 at 16:53
  • @bitinerant ad 1) I have used my real-life `blist.xml` which is over 10 years old (the previous one I have only back off) and I've used to connect to many different networks so there is plenty of addtional information. I went back to the `import-pidgin.vala` source code and read it once more - I'll edit the answer with the findings. – tukan Nov 06 '19 at 08:58
  • @bitinerant as for Option 2 - Based on the database structure there should not be any binary data (BLOB) but there is only a TEXT ones. You are right that some attributes like `given_name_localized` appear to have a binary value. However, there should not be any issue if it is `NULL` or ''. You can check the source code what is done there. If there would be any issue you can always create `UPDATE`. – tukan Nov 06 '19 at 09:01