1

I have a Python script like this:

import module
globalvariable a
def read(key):
    x = funcA(a, key)
    ret = funcB(x)
    if ret == True:
        return "Success"
    else:
        return "Failure"

I am calling this function from my C++ program repeatedly from time to time via the Python-C-API. I noticed that my C++ process memory kept increasing each time I called this function.

After a lot of investigation by:

  1. Running Valgrind
  2. Checking all the Py_DECREFs of all the variables/PyObjects in my C++ code
  3. Ensuring my code is not leaking memory elsewhere
  4. Replacing my raw Python-C-API with boost python and checking again

I have found out that the C++ memory is increasing because of the Python script itself and not the Python-C-API. I confirmed this by replacing the above script with a dummy script(that has no variables and doesn't do anything) and calling the same function repeatedly. The process memory was constant and did not increase this time.

How do I force the Python Garbage collector to free all the variables after each function call? I don't know if/when the Python GC will free memory. I noticed some suggestions in other stackoverflow answers that forcing the Garbage collector to collect is a bad thing to do.

How do I ensure that no additional memory is getting used up by my c++ process each time I call the python script? Is it as simple as calling a del() on all the local python variables that I declare? Or is there some other way?

I am also calling quite a few functions from the imported module and I don't know if that will affect the garbage collection as well.

Venkat Krishnan
  • 107
  • 1
  • 2
  • 8
  • You are decrefing the return value from when you call the function in C++? Because that's an obvious place for memory to be lost. – DavidW Oct 16 '18 at 13:40
  • You can probably ignore my comment if you've tested with Boost Python - that should handle it automatically. The next question would be "are you caching data in `funcA` or `funcB`, for example by writing to your global variable `a`?" – DavidW Oct 16 '18 at 16:49
  • @DavidW No all the global variables are constants and their values are not changed in any of my functions. But the functions do have a lot of local variables. – Venkat Krishnan Oct 16 '18 at 17:58
  • Mostly the local variables should be freed automatically. It's only if they have cyclic references to each other that you need to call the GC. I think you'd just use the `gc` module as a standard Python module if you want to force the garbage collection – DavidW Oct 16 '18 at 19:23
  • @DavidW I'm sorry I don't understand. Can I simply call gc.collect() at the end of my functions? – Venkat Krishnan Oct 16 '18 at 21:48
  • 1
    How are you calling the Python code (either with or without Boost)? – Davis Herring Oct 17 '18 at 02:27
  • Yes, that would work. Or you could call `gc.collect()` from c++. – DavidW Oct 17 '18 at 05:48
  • @DavisHerring First call Py_Initialize(), then With the Python-C-API(without Boost) importing my script then calling the functions via PyObject_CallObject(), with boost similar but using inbuilt boost ways of calling functions. – Venkat Krishnan Oct 17 '18 at 20:58
  • @DavidW how do I call gc.collect() from C++? using PyRun_SimpleString() ..? – Venkat Krishnan Oct 17 '18 at 20:58
  • The same way you call any other Python function from C++: `PyImport_ImportModule`, get the attribute `collect`, then one of the `PyObject_Call*` functions. `PyRun_SimpleString()` would probably work too. Anyway, this is my last comment on this question - I don't think we have enough information to identify the problem or offer a useful solution. – DavidW Oct 17 '18 at 21:29
  • @VenkatKrishnan: Are you calling `Py_Initialize` and importing each time, or is it just the `PyObject_CallObject` that’s repeated? How are you constructing the `key` argument? – Davis Herring Oct 18 '18 at 00:06
  • @DavisHerring it is just the `PyObject_CallObject()` that is getting repeated. I am constructing by using `PyTuple_Set()` function. I have checked all the variable references and Py_DECREFs. I am confident that I am not leaking any memory there. Only when I actually do something in the script, the memory increases. – Venkat Krishnan Oct 18 '18 at 00:09
  • 1
    @VenkatKrishnan: I guess we have to see the actual script, then. – Davis Herring Oct 18 '18 at 12:59

0 Answers0