How to avoid RuntimeWarning memory leak in callback function (with Python callback returning a string to C)?

Question

C++ library

CallbackTestLib.hpp

#pragma once

using callback_prototype = const char* __cdecl();

extern "C" __declspec(dllexport) int __cdecl do_something(callback_prototype*);

CallbackTestLib.cpp

#include "CallbackTestLib.hpp"
#include <iostream>

using namespace std;

__declspec(dllexport) auto __cdecl do_something(callback_prototype* cb) -> int
{
    if (!cb) { return 5678; }
    const auto* str = cb();
    cout << "Hello " << str << endl;
    return 1234;
}

Python script

CallbackTest.py

import os
import sys
from ctypes import CDLL, CFUNCTYPE, c_char_p, c_int32

assert sys.maxsize > 2 ** 32, "Python x64 required"
assert sys.version_info.major == 3 and sys.version_info.minor == 8 and sys.version_info.micro == 4, "Python 3.8.4 required"

callback_prototype = CFUNCTYPE(c_char_p)

@callback_prototype
def python_callback_func() -> bytes:
    return "from Python".encode("utf-8")

dll_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "CallbackTestLib.dll")
testlib = CDLL(dll_path)
testlib.do_something.restype = c_int32
testlib.do_something.argtypes = [callback_prototype]

null_return_code = testlib.do_something(callback_prototype())
assert null_return_code == 5678, "Null return code failed"

ok_return_code = testlib.do_something(python_callback_func)
assert ok_return_code == 1234, "Ok return code failed"

print("Done.")

Python output

d:/path/to/CallbackTest.py:22: RuntimeWarning: memory leak in callback function.
ok_return_code = testlib.do_something(python_callback_func)
Hello from Python
Done.

As the output shows, Python (somehow) seems to have detected a memory leak when python_callback_func is executed, which returns bytes (UTF-8 encoded string) back to C++ where the string is being printed out.
My question is all about this: what is going on around with this warning, how to avoid/solve it?

You probably need to free `str` before returning: `delete [] str;` — Mike67, Aug 16 '20 at 23:55
It's `ctypes`, not `cpptypes` so perhaps `free()` would work assuming the DLL and Python were built with the same C runtime library. But the real solution is not to return a `char*`. `ctypes` can't know how to decrement the reference count of the Python object the callback returned, if the value is returned to C/C++ code. — Mark Tolonen, Aug 17 '20 at 03:11
@MarkTolonen I can't find anything about `cpptypes` where did you get this from? Why would Python care about decrementing refcount on CData (c_char_p/bytes)? If I am not going to return `const char*` then what am I...? — Abbecuby, Aug 17 '20 at 15:19
I'm saying `cpptypes` doesn't exist...so it is likely `ctypes` (a "C" wrapper) would use `malloc/free` internally not "new/delete". When you return an object from a Python function, internally Python increments the reference count to the object returned; otherwise, the local object would be freed when the function returns. But you return the object to C code, not Python, so `ctypes` is obtaining a `char*` pointer to the Python object's buffer, and the C code has no ability to decrement the Python ref, hence the leak. `ctypes` can't decrement the ref either, or the `char*` would be invalid. — Mark Tolonen, Aug 17 '20 at 16:48

score 1 · Answer 1 · answered Aug 17 '20 at 17:18

Having your Python callback return a char* is similar to a C++ function returning:

char* callback() {
    return new char[10];
}

You've got a memory leak unless you free it. Since Python allocated the bytes object, C++ can't free it correctly, hence the leak.

Instead, pass a C++-managed buffer to the callback:

test.cpp

#include <iostream>
using namespace std;

typedef void (*CB)(char* buf,size_t len);

extern "C" __declspec(dllexport) int func(CB cb) {
    char buf[80];
    if(cb) {
        cb(buf,sizeof buf);
        cout << "Hello " << buf << endl;
        return 1234;
    }
    return 5678;
}

test.py

from ctypes import *

CALLBACK = CFUNCTYPE(None,POINTER(c_char),c_size_t)

@CALLBACK
def callback(buf,size):
    # Cast the pointer to a single char to a pointer to a sized array
    # so it can be accessed safely and correctly.
    arr = cast(buf,POINTER(c_char * size))
    arr.contents.value = b'world!'

dll = CDLL('./test')
dll.func.argtypes = CALLBACK,
dll.func.restype = c_int

print(dll.func(callback))

Output:

Hello world!
1234

Problem with this approach is that `buf` is fixed-size, and I need to scale things in a real application where Python callback must be able to return an arbitrary-size string to C/C++ code. Hardcoding and/or guessing a maximum size is... not an option. Odds are on potential buffer overflow if done wrong. Sorry for not mentioning this more explicitly. — Abbecuby, Aug 17 '20 at 17:56
If you allocate the buffer from Python, then you have to free it from Python. Your void* answer could work, if you store the object in Python for the lifetime of its use and release it when it is no longer needed by C++ code. — Mark Tolonen, Aug 17 '20 at 18:03

Abbecuby · Answer 2 · 2020-08-17T20:32:33.437

This is still somewhat vague/unclear to me. But here is a stupid fix (I am not happy with it), it makes the memleak warning message go away:

Version 1

CallbackTestLib.hpp

#pragma once

using callback_prototype = void* __cdecl(); // Changed 'const char*' to 'void*'.

extern "C" __declspec(dllexport) int __cdecl do_something(callback_prototype*);

CallbackTestLib.cpp

#include "CallbackTestLib.hpp"
#include <iostream>

using namespace std;

__declspec(dllexport) auto __cdecl do_something(callback_prototype* cb) -> int
{
    if (!cb) { return 5678; }
    const auto* str = cb();
    cout << "Hello " << static_cast<const char*>(str) << endl; // Added 'static_cast'.
    return 1234;
}

CallbackTest.py

import os
import sys
from ctypes import CDLL, CFUNCTYPE, cast, c_void_p, c_int32

assert sys.maxsize > 2 ** 32, "Python x64 required"
assert sys.version_info.major == 3 and sys.version_info.minor == 8 and sys.version_info.micro == 4, "Python 3.8.4 required"

callback_prototype = CFUNCTYPE(c_void_p)  # Changed restype to 'c_void_p'.

@callback_prototype
def python_callback_func():
    return cast("from Python :)".encode("utf-8"), c_void_p).value  # Added casting.

dll_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "CallbackTestLib.dll")
testlib = CDLL(dll_path)
testlib.do_something.restype = c_int32
testlib.do_something.argtypes = [callback_prototype]

null_return_code = testlib.do_something(callback_prototype())
assert null_return_code == 5678, "Null return code failed"

ok_return_code = testlib.do_something(python_callback_func)
assert ok_return_code == 1234, "Ok return code failed"

print("Done.")

Output

Hello from Python :)
Done.

Version 2

CallbackTestLib.hpp

#pragma once

using callback_prototype = void __cdecl();

static char* do_something_buffer;

extern "C" __declspec(dllexport) int __cdecl do_something(callback_prototype);

extern "C" __declspec(dllexport) void __cdecl receive_do_something_buffer(const char*);

CallbackTestLib.cpp

#include "CallbackTestLib.hpp"
#include <iostream>

using namespace std;

auto do_something(callback_prototype cb) -> int
{
    if (!cb) { return 5678; }
    cb();
    cout << "Hello " << do_something_buffer << endl;
    cb();
    cout << "Hello again " << do_something_buffer << endl;
    return 1234;
}

void receive_do_something_buffer(const char* str)
{
    // Create a copy of the given string and save it into buffer.
    if (do_something_buffer) { free(do_something_buffer); }
    do_something_buffer = _strdup(str);
}

CallbackTest.py

import os
import sys
from ctypes import CDLL, CFUNCTYPE, c_int32, c_char_p

assert sys.maxsize > 2 ** 32, "Python x64 required"
assert sys.version_info.major == 3 and sys.version_info.minor == 8 and sys.version_info.micro == 4, "Python 3.8.4 required"

callback_prototype = CFUNCTYPE(None)

dll_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "CallbackTestLib.dll")
testlib = CDLL(dll_path)
testlib.do_something.restype = c_int32
testlib.do_something.argtypes = [callback_prototype]

testlib.receive_do_something_buffer.restype = None
testlib.receive_do_something_buffer.argtypes = [c_char_p]

@callback_prototype
def python_callback_func() -> None:
    testlib.receive_do_something_buffer("from Python :D".encode("utf-8"))

null_return_code = testlib.do_something(callback_prototype())
assert null_return_code == 5678, "Null return code failed"

ok_return_code = testlib.do_something(python_callback_func)
assert ok_return_code == 1234, "Ok return code failed"

print("Done.")

Output

Hello from Python :D
Hello again from Python :D
Done.

In this case, you are passing back a number (a `void*` address`). Python is likely freeing the memory for the string, and you are getting lucky that the address still contains the data (undefined behavior invoked!). Instead, pass a C-managed buffer to the callback, and return the string in that. I'll post an example. — Mark Tolonen, Aug 17 '20 at 17:15
@MarkTolonen I have come up with alternative approach (V2), can you please look at it? (Still not happy with it.) — Abbecuby, Aug 17 '20 at 19:48
Other than not being thread-safe, it looks good. `_strdup()` should be freed with `free()` not `delete[]` per its documentation. — Mark Tolonen, Aug 17 '20 at 20:11
Ok, so far V2 would be the "best" approach in my case, because everything is running on a single thread. — Abbecuby, Aug 17 '20 at 20:34

How to avoid RuntimeWarning memory leak in callback function (with Python callback returning a string to C)?

C++ library

Python script

Python output

2 Answers2

Version 1

Output

Version 2

Output

Linked