Python calling Cython-function result in core dump

Question

I'm currently working on a project the requires Python C-API
So I created a class in the C side and inherited from it python side.
one of the C functions in the future should return either None or dict.
the function:

static PyObject* handle_data(BaseClassObject* self){
    struct data *data;
    static int counter= 0;

    // get next data from queue
    if ((data= data_queue_pop(self->data_queue))) {
        // TODO: build a Python dict from data and return it
        free_data(data);
        counter++;
    }
    return Py_None;
}

At this development stage the function data_queue_pop return nothing, as a result, this code always get to return Py_None;
In the Python side I have the flowing class:

from concurrent.futures import ThreadPoolExecutor
from threading import Thread
from time import sleep

from utils.logger import getLogger
logger = getLogger(__name__)

try:
    from samplelib import BaseClass
except ImportError:
    from uploadlib import BaseClass


class SomeClass(BaseClass):
    def __init__(self, args):
        super().__init__()
        self.log = logger.getChild(self.__class__.__name__)
        self.version = self.version.rstrip()  # self.version is intilaied in the C side
        self.args = args
        self.wait = True
        self._pool = ThreadPoolExecutor(max_workers=1)
        self.worker = None

    def __enter__(self):
        self.log.info(f"set worker")
        # self.worker = Thread(target=self._handle_data)
        self.worker = self._pool.submit(self._handle_data)
        # self.log.info(f"worker start")
        # self.worker.start()
        self.log.info(f"return self")
        return self

    def __exit__(self, a, b, c):
        self.log.info(f"exit {self.__class__.__name__}")
        # self.log.info('stop thread')
        self.wait = False
        if self.worker:
            self.log.info('wait for thread to stop')
            self.log.info(self.worker.cancel())
            # self.worker.join()
        self.log.info('exiting...')

    def _handle_data(self):
        self.log.info(f"thread started")
        while self.wait:
            data = self.handle_data()
            if data:
                self.log.warning(data)
        self.log.info(f"thread stopped")


def main():
    args = _parse_args()
    with SomeClass(args) as e:
        sleep(2)
        print(e.version)
        # e.wait = False


if __name__ == "__main__":
    main()

It doesn't matter what I try to do I get the following error:

Fatal Python error: deallocating None

Current thread 0x00007f8c37b17700 (most recent call first):
  File "/home/ramih/SomeClass-automation/utils/some_module.py", line 50 in _handle_alert
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56 in run
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 69 in _worker
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f8c3aedc740 (most recent call first):
  File "./test.py", line 124 in main
  File "./test.py", line 130 in <module>
Aborted (core dumped)

Why this is happening?!
How can I debug it?!
Can I make the threading safer?!

Thanks.

It's a ref-counting error somewhere - you've somehow managed to decref `None` more times than you've increfed it, meaning the global `None` object has been destroyed. Unfortunately your example is nowhere close to an [mre] so it's hard to give more details than that. — DavidW, Jul 29 '20 at 20:31
@DavidW maybe my example is nowhere close to a minimal reproducible example, but your response helped me to find the solution, I just replaced the line `return Py_None;` to `Py_RETURN_NONE;` and the issue was solved. the difference between both lines is that the macro `Py_RETURN_NONE` also increases the reference of the None object, therefore, thanks a lot :) — Rami Hassan, Jul 30 '20 at 00:20

score 0 · Answer 1 · answered Jul 30 '20 at 00:32

Thanks to @DavidW response, the issue was solved and I will share it so if anyone else encounters this issue can find the solution fast. In such a case you have two options:

before return Py_None; do Py_INCREF(Py_None)
call the macro Py_RETURN_NONE which already include the ref increment

Python calling Cython-function result in core dump

1 Answers1