Python C API free() errors after using Py_SetPath() and Py_GetPath()

Question

I'm trying to figure out why I can't simply get and set the python path through its C API. I am using Python3.6, on Ubuntu 17.10 with gcc version 7.2.0. Compiling with:

gcc pytest.c `python3-config --libs` `python3-config --includes`

#include <Python.h>

int main()
{
    Py_Initialize(); // removes error if put after Py_SetPath

    printf("setting path\n"); // prints
    Py_SetPath(L"/usr/lib/python3.6"); // Error in `./a.out': free(): invalid size: 0x00007fd5a8365030 ***
    printf("success\n"); // doesn't print
    return 0;
}

Setting the path works fine, unless I also try to get the path prior to doing so. If I get the path at all, even just to print without modifying the returned value or anything, I get a "double free or corruption" error.

Very confused. Am I doing something wrong or is this a bug? Anyone know a workaround if so?

Edit: Also errors after calling Py_Initialize();. Updated code. Now errors even if I don't call Py_GetPath() first.

Why do you use a C++ compiler to compile C code? Doing so makes the C code be C++ code, which in turn invalidates your questions' tagging. — alk, May 06 '18 at 09:13
I typed g++ by accident. Same result with gcc. Yes `Python.h` includes `stdio.h`. — cowemoji, May 06 '18 at 09:22
You may very well edit your question and correct any "typos". — alk, May 06 '18 at 09:25
Most Python C API functions cannot be used before `Py_Initialize()`. The docs say `Py_SetPath()` can be, but they do not say that `Py_GetPath()` can be called before `Py_Initialize()`. Do you have any reason to think it can? — John Zwinck, May 06 '18 at 09:31
I assumed it could be called before Py_Initialize() because I called it and it printed my correct python3 paths. Is there another way to get the same path? — cowemoji, May 06 '18 at 09:34
Hmmm, changing your question significantly after comments (and answers) had been given is not a good idea as it renders the latter ununderstandable. — alk, May 06 '18 at 09:48
It's getting messy now. The title does not fit the code any more ... but well — alk, May 06 '18 at 09:55
https://bugs.python.org/issue31532 which version/build do you use? — alk, May 06 '18 at 10:03
Thank you alk I think that is the issue.. unforunately I do use python 3.6 and gcc. But also Py_SetPath won't even work. I think I found a workaround I will post in a sec — cowemoji, May 06 '18 at 10:27

score 1 · Accepted Answer · answered May 06 '18 at 10:41

From alk it seems related to this bug: https://bugs.python.org/issue31532

Here is the workaround I am using. Since you can't call Py_GetPath() before Py_Initialize(), and also seemingly you can't call Py_SetPath() after Py_Initialize(), you can add to or get the path like this after calling Py_Initialize():

#include <Python.h>

int main()
{
    Py_Initialize();

    // get handle to python sys.path object
    PyObject *sys = PyImport_ImportModule("sys");
    PyObject *path = PyObject_GetAttrString(sys, "path");

    // make a list of paths to add to sys.path
    PyObject *newPaths = PyUnicode_Split(PyUnicode_FromWideChar(L"a:b:c", -1), PyUnicode_FromWideChar(L":", 1), -1);

    // iterate through list and add all paths
    for(int i=0; i<PyList_Size(newPaths); i++) {
        PyList_Append(path, PyList_GetItem(newPaths, i));
    }

    // print out sys.path after appends
    PyObject *newlist = PyUnicode_Join(PyUnicode_FromWideChar(L":", -1), path);
    printf("newlist = %ls\n", PyUnicode_AsWideCharString(newlist, NULL));
    return 0;
}

alk · Answer 2 · 2018-05-06T10:31:59.010

0

[the below answer refers to this version of the question.]

From the docs:

void Py_Initialize()

Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the exception of Py_SetProgramName(), Py_SetPythonHome() and Py_SetPath().

But the code you show does call Py_GetPath() before it calls Py_Initialize();, which it per the above paragraph implicitly should not.

edited May 06 '18 at 10:31

answered May 06 '18 at 09:35

alk

69,737
10
105
255

Thanks. If I call `Py_Initialize();` first it works without crash on `Py_SetPath()`. Although either way the printf correctly outputs: `Py_GetPath(): /usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload`. Edit: nevermind it still errors – cowemoji May 06 '18 at 09:40

Python C API free() errors after using Py_SetPath() and Py_GetPath()

2 Answers2