4

I have written a c++ module that I want to use in Python, using pybind11. It contains, amongst other, this code:

class Foo{
    ...

    void Foo::Bar(BigObject& returnBigObject)
    {    
        for (int i=0; i<4; i++)
        { 
            BigObject tempObject;
            //doStuffWithTempObject
            returnBigObject += tempObject;
        }
    }
};

Now, I have exposed this code to python and it functions as expected. However when I look at the memory usage, I see that when the BigObject tempobject goes out of scope the memory is not always freed. Meaning that If I look at psutil.Process(os.getpid()).memory_info().rss the memory does not go down to what I expect. After some debugging I traced it back to this temporary BigObject. Also I see this behaviour on Linux, but not on Windows, and on Linux it happening depends a lot on the code that I write before the call to this function.

My question is, is this normal behaviour of a managed language like Pyhton? Should I worry about memory usage in this case or trust the garbagecollector to clean up the memory when the application needs it to? Does the garbage collector even handle this kind of stuff, or there a different mechanism at play?

Frank
  • 2,446
  • 7
  • 33
  • 67
  • Not completely sure, but the gc module in python might help you look inside: https://docs.python.org/3/library/gc.html – adamkgray Nov 25 '19 at 09:32
  • OT: _Managed_ is MS / .NET vocabulary. Let's call it _garbage collected_ instead please. – nada Nov 25 '19 at 10:05
  • Is `BigObject` dynamically big (e.g. `std::vector(1000000)`) or statically big (e.g. std::array`)? – Caleth Nov 25 '19 at 11:10
  • The memory not going down isn't the right question, as it depends on the allocator implementation for example. E.g. if the BigObject consists of many SmallObjects, they may well be allocated in arena's to reduce admin overhead. These arena's may be kept around and be available to your application. Rather, the question should be: is the memory going up when you call the bound method multiple times? – Wim Lavrijsen Nov 25 '19 at 17:59
  • 1
    Aside, in answer to your last question: since the function returns nothing to python, neither it or its garbage collector is in involved here. – Wim Lavrijsen Nov 25 '19 at 18:02
  • This question could do with some more details, because answering it depends on how you have wrapped the function with pybind11. Also, RSS is not a reliable indicator of whether/when memory has been freed, for a variety of reasons (see eg https://unix.stackexchange.com/questions/34795/correctly-determining-memory-usage-in-linux). – Isaiah Norton May 10 '20 at 02:29

0 Answers0