Sporadic segfault in c++ python extension

Question

I have a python object which accesses and downloads some text via HTTP. I'm running this python object, and processing that text, using a c++ code. I.e.

/* CPPCode.cxx */
int main(...) {
    for(int i = 0; i < numURLs; i++) {
        // Python method returns a string
        PyObject *pyValue = PyObject_CallMethod(pyObjectInstance, pyFunctionName, par1, par2....);
        string valString = PyString_AsString(pHistValue);   
        // ... process string ... 
    }
} 

/* PyObject.py */
class PyClass:
    def PyFunction(...):
        try: urlSock = urllib.urlopen(urlName)
        except ...

        while(...) :
             dataStr = urlSock.readline()
             # do some basic string processing....

        return dataStr

Most URLs work fine---the c++ code gets the proper string, I can process it, all is happy and well. A few particular URLs which look (basically) the same as the others on a browser, lead to a segfault in the PyString_AsString() method:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x00000000000000b2
0x000000010007716d in PyString_AsString ()

If I print out the string that should be returned by the python method ('dataStr' in the pseudo-code above), it looks fine! I have no idea what could be causing this problem---any tips on how to procede would be appreciated! Thanks

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SOLUTION:

The template code I was using had a call to

Py_DECREF(pyValue)

before I called

PyString_AsString(pyValue)

Why it was being deallocated for certain particular function calls, I have no idea. As 'Gecco' says in the comments below,

'PyString_AsString documentation says: "The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). It must not be deallocated." '

http://www.wunderground.com/history/airport/PAFM/2012/01/05/DailyHistory.html?HideSpecis=0&format=1 — DilithiumMatrix, Jan 05 '12 at 21:24
unexplicable segfault: This is quite often due to some memory-allocation errors. Check your _entire_ code to ensure you correctly allocate and free used memory... — gecco, Jan 05 '12 at 21:26
`PyString_AsString` documentation says: "The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). **It must not be deallocated**." _Please ensure you do not deallocate this buffer._ — gecco, Jan 05 '12 at 21:32
@gecco That was my first thought, but I'm not using *any* dynamically allocated memory; and because the segfault is happening for the same URL every time, regardless of how many I examine before or after, it seems there might be another issue. I'm not manually allocating or deallocating any memory. — DilithiumMatrix, Jan 05 '12 at 21:32
`address: 0x00000000000000b2` looks like data rather than a pointer. Check the pointer being passed into PyString_AsString. — cdarke, Jan 05 '12 at 21:42
@gecco you were right, there was a Py_DECREF(pyValue) stuck in there from the template code that I originally copied. Removing it fixed the problem! Thanks! — DilithiumMatrix, Jan 05 '12 at 21:50
@zhermes I posted my comment as an answer in case you want to reward me ;-) — gecco, Jan 05 '12 at 22:54

score 1 · Accepted Answer · answered Jan 05 '12 at 22:53

PyString_AsString documentation says: "The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). It must not be deallocated."

Please ensure you do not deallocate this buffer

score 0 · Answer 2 · answered Jan 05 '12 at 21:32

0

If you compile your C code with the -g debug flag (in GCC at least) then you can run your python code using the gnu debugger gdb:

$ gdb /path/to/python/compiled/against 
... blah ...
(gdb) run PyObject.py

and you should catch your segfault.

answered Jan 05 '12 at 21:32

danodonovan

19,636
10
70
78

Here is my full backtrace from GDB `code` Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: 13 at address: 0x0000000000000000 0x00007fff80237c00 in strlen () (gdb) bt #0 0x00007fff80237c00 in strlen () #1 0x0000000100088e1e in PyString_FromFormatV () #2 0x00000001000d3fdc in PyErr_Format () #3 0x000000010008510a in PyString_AsStringAndSize () #4 0x0000000100085183 in PyString_AsString () #5 0x000000010000f6f3 in callPythonWHistoryMethod () at stl_vector.h:137 #6 0x0000000100010555 in main () at stl_vector.h:137 – DilithiumMatrix Jan 05 '12 at 21:36
So you can now prod the code to see what was breaking everything - for example, what is "pHistValue" : try "(gdb) print pHistValue" and see if it is what you expected – danodonovan Jan 05 '12 at 21:53

Demolishun · Answer 3 · 2012-01-10T07:22:59.213

0

My guess is the Py_DECREF is somehow getting a NULL value.

edited Jan 10 '12 at 07:22

answered Jan 06 '12 at 02:52

Demolishun

1,592
12
15

My understanding is that once the references to an object are zero, it is deallocated (like in cocoa, e.g.). – DilithiumMatrix Jan 06 '12 at 21:46
Yes, when it reaches zero it will deallocate. Good catch, I will fix it. – Demolishun Jan 10 '12 at 07:20

Sporadic segfault in c++ python extension

3 Answers3