Python multithreading- memory leak when use an shared object(so)

Question

I have a python programs that gets memory leaks when use an third-party SO. I simplify my code like this:

import time
import sys
import threading
import codecs
import ctypes
sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())


class TestThirdPartySo(object):
    def __init__(self):
        # this so uses thread-specific data
        self.call_stat_so = ctypes.CDLL("./third_party_fun.so")
        self.handle = self.call_stat_so._handle

    def test_fun(self):
        self.call_stat_so.fun_xxx()

def thread_fun():
    TestThirdPartySo().test_fun()


def test_main(num):
    count = 0
    while True:
        # create 3 * num threads
        thread_num = 3
        thread_list = []
        for _ in range(thread_num):
            thread_list.append(threading.Thread(target=thread_fun))
        for thread in thread_list:
            thread.start()
        for thread in thread_list:
            thread.join()
        count += thread_num
        time.sleep(0.01)
        if count % 100 == 0:
            print("finied %s" % count)
        if count > num:
            break
    print("end !!!!")


if __name__ == '__main__':
    num = sys.argv[1]
    test_main(int(num))

Now, I know this shared object uses thread-specific data.And I have tried to close the SO after called it like this:

class TestThirdPartySo(object):
    def __init__(self):
        # this so uses thread-specific data
        self.call_stat_so = ctypes.CDLL("./third_party_fun.so")
        self.handle = self.call_stat_so._handle

    def test_fun(self):
        self.call_stat_so.fun_xxx()


    def __del__(self):
        dlclose_func(self.handle)


def dlclose_func(_handle):
    dlclose_func_tmp = ctypes.cdll.LoadLibrary('libdl.so').dlclose
    dlclose_func_tmp.argtypes = [ctypes.c_void_p]
    dlclose_func_tmp(_handle)

But I failed to close the so. And I'm also not sure if the leaked memory will be freed after closing the so.

If the program not uses multi-threads or creates a fixed number of threads(threadpool), it works ok.

For some reason,I need create threads constantly in my program. What can I do to prevent this memory leaks?

Python multithreading- memory leak when use an shared object(so)

0 Answers0