Pickling routines accessible through the C API?

Question

I would like to call Python's pickling routines (dumps and loads) from within c++ code. Are they exposed in the official API? I am currently calling those via boost::python from c++, looking for a simpler way perhaps.

Answer 1

You can call any Python code through the C API:

static PyObject *module = NULL;
PyObject *pickle;

if (module == NULL &&
    (module = PyImport_ImportModuleNoBlock("pickle")) == NULL)
    return NULL;

now, you can either call it like:

python-2.x

pickle = PyObject_CallMethodObjArgs(module,
                                    PyString_AS_STRING("dumps"),
                                    py_object_to_dump,
                                    NULL);

python-3.x

pickle = PyObject_CallMethodObjArgs(module,
                                    PyUnicode_FromString("dumps"),
                                    py_object_to_dump,
                                    NULL);

or like:

picle = PyObject_CallMethod(module, "dumps", "O", py_object_to_dump);

and then do the error checking and clean up:

if (pickle != NULL) { ... }
Py_XDECREF(pickle);

but in the case of pickle you can just use the cPickle functions directly. The only problem there is that the cPickle module (or _pickle in Python 3) is statically compiled into the Python binary, or needs to be loaded separately. Using the Python import mechanisms is simply easier here.

Answer 2

Many years later, not quite answering the question, and based on Martijn Pieters' answer, here is a drop in replacement for Marshal:

static PyObject * PyPickle=NULL;
static PyObject * pDumps=NULL;
static PyObject * pLoads=NULL;

static void init_pickle()
{
    if(!pDumps)
        pDumps = PyUnicode_FromString("dumps");
    if(!pLoads)
        pLoads = PyUnicode_FromString("loads");

    if(!PyPickle)
        PyPickle = PyImport_ImportModule("pickle");
}

static PyObject *PyPickle_ReadObjectFromString(const char *data, Py_ssize_t len)
{
    PyObject *pstr = PyBytes_FromStringAndSize(data, len);
    PyObject *ret=NULL;

    ret = PyObject_CallMethodOneArg(PyPickle, pLoads, pstr);
    Py_XDECREF(pstr);

    return ret;
}

static PyObject *PyPickle_WriteObjectToString(PyObject *value, int version)
{
    //version currently ignored
    return PyObject_CallMethodOneArg(PyPickle, pDumps, value);
}

Perhaps someone will find it useful.

Answer 3

Complementing the other C-API-based answer by @aflin, this would be in c++ using pybind11 (and the C-API underneath):

Header file:

#include<string>
#include<pybind11/pybind11.h>
namespace py=pybind11;

class Pickler{
    static bool initialized;
    static py::handle cPickle_dumps;
    static py::handle cPickle_loads;
    static void ensureInitialized();
    public:
        static std::string dumps(py::object o);
        static py::object loads(const std::string&);
};

Implementation:

py::handle Pickler::cPickle_dumps;
py::handle Pickler::cPickle_loads;
bool Pickler::initialized=false;

void Pickler::ensureInitialized(){
    if(initialized) return;
    py::gil_scoped_acquire pyLock;
    py::object cPickle=py::module::import("pickle");
    cPickle_dumps=cPickle.attr("dumps");
    cPickle_loads=cPickle.attr("loads");
    initialized=true;
}

std::string Pickler::dumps(py::object o){
    ensureInitialized();
    py::gil_scoped_acquire pyLock;
    py::object minus1=py::cast(-1);
    // watch out! the object might have NULL pointer (uninitialized??) so make sure to pass None in that case
    PyObject *b=PyObject_CallFunctionObjArgs(cPickle_dumps.ptr(),o.ptr()?o.ptr():Py_None,minus1.ptr(),NULL);
    if(!b){
        if(PyErr_Occurred()) throw py::error_already_set();
        else throw std::runtime_error("Pickling failed but no python error was set??");
    }
    assert(PyBytes_Check(b));
    return std::string(py::reinterpret_steal<py::bytes>(b));
}
py::object Pickler::loads(const std::string& s){
    ensureInitialized();
    py::gil_scoped_acquire pyLock;
    return cPickle_loads(py::handle(PyBytes_FromStringAndSize(s.data(),(Py_ssize_t)s.size())));
}

And then, obviously, calling Pickler::dumps(...) or Pickler::loads(...) in your code.