Memory Leak in Threaded COM Object with Python

Question

I am creating a COM client within a thread and performing several operations with this client. Each thread is spawned from a server that uses Python's socketserver module which has built-in threading support.

When I am loading and using this COM object there is an expected spike in memory usage by python.exe. With 10 concurrent threads there is a peak in memory usage of about 500Mb. However when the operations are finished and the COM object is apparently released, there are 50Mb of additional memory used by the process than before. If I then spawn 10 additional threads using the same server there are 13Mb additional used by python.exe after those COM objects are closed. Eventually every 10 additional concurrent threads adds approximately 6Mb after they are done. When I end the entire python.exe process, all the memory is released.

I have simplified the following code to mimic how the socketserver uses threadding and the problem is the exact same.

import win32com.client
import threading
import pythoncom

def CreateTom():
    pythoncom.CoInitialize()
    tom = win32com.client.Dispatch("TOM.Document")
    tom.Dataset.Load("FileName")
    tom.Clear()
    pythoncom.CoUninitialize()

for i in range(50):
    t = threading.Thread(target = CreateTom)
    t.daemon = False
    t.start()

I understand that I will unlikely get any support here around the specific COM library (it is an IBM product used in Market Research known as the TablesObjectModel). However I want to know if there is anything, ANYTHING, additional that I can do to release this memory. I have read about Apartments in COM but it sounds like pythoncom.CoInitialize should take care of this for me. Any help would be appreciated.

This might not be entirely relevant, but try use a thread pool instead of spawning new thread without defining any constraints like the example you posted. — woozyking, May 03 '13 at 21:06
Is the TOM.Document in-process or out-of-process? Are you sure the memory is not released after "some time"? This can happen for example with COM Objects written in technologies with a garbage collector (such as .NET, though I doubt TOM.Document is written in .NET) — Simon Mourier, May 03 '13 at 21:41
TOM.Document is in-process. It does not fork another process outside python.exe — Michael David Watson, May 03 '13 at 21:51
I do not believe that this will work with sockets. In order to use the multiprocessing module all arguments need to be 'pickleable' and sockets are not. — Michael David Watson, May 06 '13 at 18:36
@Aya- However I could thread my socket requests as I currently am, and then create a separate supprocess within each thread where the COM instance lives. I am going to look into this right now. — Michael David Watson, May 06 '13 at 18:58
Creating a separate subprocess for the COM object did not work out. Because of the amount of the cross-process communication, I could not get this to work with pipes on Windows. — Michael David Watson, May 13 '13 at 00:36
Perhaps you could be a more active relay/proxy between the inbound socket and the one you maintain to your subprocesses? Use a socket on localhost rather than a Windows pipe? Or even a segment of shared memory? (Does the mmap on MS Windows support shared memory)? — Jim Dennis, May 13 '13 at 03:04
Can you add the end part of this `t.start()` before ending the thread, or it ends automatically..!!??. — MarmiK, May 13 '13 at 12:20
@Jim, this is a very very interesting approach. I am going to test it out and get back to you. — Michael David Watson, May 13 '13 at 14:26

score 6 · Accepted Answer · answered May 17 '13 at 20:08

As it turns out this increase in Memory was in fact due to the COM object written in .NET and had nothing to do with threading. Here is a detailed description of Task Manager giving misleading information about memory usage for .NET apps. To resolve this issue I added the following to my code and I am all set. Hopefully someone else reads this response before they start tearing their hair out trying to find a memory leak in their code.

from win32process import SetProcessWorkingSetSize
from win32api import GetCurrentProcessId, OpenProcess
from win32con import PROCESS_ALL_ACCESS

import win32com.client
import threading
import pythoncom

def CreateTom():
    pythoncom.CoInitialize()
    tom = win32com.client.Dispatch("TOM.Document")
    tom.Dataset.Load("FileName")
    tom.Clear()
    pythoncom.CoUninitialize()
    SetProcessWorkingSetSize(handle,-1,-1) #Releases memory after every use

pid = GetCurrentProcessId()
handle = OpenProcess(PROCESS_ALL_ACCESS, True, pid)

for i in range(50):
    t = threading.Thread(target = CreateTom)
    t.daemon = False
    t.start()

score 0 · Answer 2 · answered May 13 '13 at 04:22

0

here it is link may help you release COM in python win32

answered May 13 '13 at 04:22

xwlan

554
3
5

Thank you for that, it was very helpful in confirming that there are no remaining COM references in my script as pythoncom._GetInterfaceCount() = 0 – Michael David Watson May 13 '13 at 14:20

score 0 · Answer 3 · answered May 03 '17 at 09:15

for me it help (based)::

from comtypes.automation import IDispatch
from ctypes import c_void_p, cast, POINTER, byref

def release_reference(self, obj):
    logger.debug("release com object")
    oleobj = obj._oleobj_
    addr = int(repr(oleobj).split()[-1][2:-1], 16)

    pointer = POINTER(IDispatch)()
    cast(byref(pointer), POINTER(c_void_p))[0] = addr
    pointer.Release()

Memory Leak in Threaded COM Object with Python

3 Answers3

Linked