How can I read a python pickle database/file from C?

Question

I am working on integrating with several music players. At the moment my favorite is exaile.

In the new version they are migrating the database format from SQLite3 to an internal Pickle format. I wanted to know if there is a way to access pickle format files without having to reverse engineer the format by hand.

I know there is the cPickle python module, but I am unaware if it is callable directly from C.

Not really, since exaile only runs on linux as far as I know. — Phillip Whelan, Aug 18 '09 at 21:25

score 32 · Accepted Answer · edited Mar 30 '11 at 04:00

http://www.picklingtools.com/

There is a library called the PicklingTools which I help maintain which might be useful: it allows you to form data structures in C++ that you can then pickle/unpickle ... it is C++, not C, but that shouldn't be a problem these days (assuming you are using the gcc/g++ suite).

The library is a plain C++ library (there are examples of C++ and Python within the distribution showing how to use the library over sockets and files from both C++ and Python), but in general, the basics of pickling to files is available.

The basic idea is that the PicklingTools library gives you "python-like" data structures from C++ so that you can then serialize and deserialize to/from Python/C++. All (?) the basic types: int, long int,string, None, complex, dictionarys, lists, ordered dictionaries and tuples are supported. There are few hooks to do custom classes, but that part is a bit immature: the rest of the library is pretty stable and has been active for 8 (?) years.

Simple example:

#include "chooseser.h"
int main()
{
  Val a_dict = Tab("{ 'a':1, 'b':[1,2.2,'three'], 'c':None }");
  cout << a_dict["b"][0];  // value of 1

  // Dump to a file
  DumpValToFile(a_dict, "example.p0", SERIALIZE_P0);

  // .. from Python, can load the dictionary with pickle.load(file('example.p0'))

  // Get the result back
  Val result;
  LoadValFromFile(result, "example.p0", SERIALIZE_P0);
  cout << result << endl;
}

There is further documentation (FAQ and User's Guide) on the web site.

Hope this is useful:

Gooday,

Richie

http://www.picklingtools.com/

This looks great, but it's not clear here (or on the website) if the library can open files which have previously been saved with python's pickle module – or if a custom picklingtools python pickler needs to be used. — Bill Cheatham, Oct 29 '14 at 11:45
As long as your data is in the standard data types (int, long, dict, list, string, etc), and not a class, it should work just fine. (I.e., something pickled from Python should be able to read from C++ using the techniques above). If your data does contain a class, you may be able to get it to work: recall that all Python classes implement their namespaces as dicts, so you can get the __dict__ from a class and pickle that! — rts1, Mar 13 '15 at 18:11

score 5 · Answer 2 · edited Jun 01 '18 at 12:57

5

Like Cristian told, you can rather easily embed python code in your C code, see the example here.

Using cPickle is dead easy as well on python you could use something like:

import cPickle

f = open('dbfile', 'rb')
db = cPickle.load(f)
f.close()
# handle db integration
f = open('dbfile', 'wb')
cPickle.dump(db, f)
f.close()

edited Jun 01 '18 at 12:57

George Sovetov

4,942
5
36
57

answered Aug 18 '09 at 21:25

hhurtta

104
2

as far as I can tell and as far as I've been told the only option is to embed a python interpreter. Since I won't need to port it to win32, this isn't such a bad solution. – Phillip Whelan Aug 19 '09 at 09:23

score 3 · Answer 3 · answered Aug 18 '09 at 20:07

3

You can embed a Python interpreter in a C program, but I think that the easiest solution is to write a Python script that converts "pickles" in another format, e.g. an SQLite database.

answered Aug 18 '09 at 20:07

Cristian Ciupitu

20,270
7
50
76

How can I read a python pickle database/file from C?

3 Answers3

Linked