0

These days, I'm trying to determine whether my C++ method (still) has a memory leak. To this end, I wrote a special main that runs the actual method infinitely over and over and record the memory footprint of the process every minute. One run of the method takes less than a second. The method itself is a little complicated, it internally does some pre-processing, invokes an ONNX model and then does some post-processing on the output. The method is written in a way that one call to it shouldn't increase the memory persistently (i.e., it should free all its local data after exit) and for except one instance, I didn't made use of C-style pointers. valgrind reports some memory issues in onnxruntime space, but no memory issues in my code.

If I remove the ONNX part, the RSS memory only ever decreases (x-axis: minutes after start, y-axis: RSS in bytes):

graph of RSS memory in the program without ONNX over the span of 8143 minutes

Whereas with the ONNX part, the memory increases slowly over days, until it suddenly sharply falls off:

graph of RSS memory in the program with ONNX over the span of 8143 minutes.

What are the reasons for this kind of memory behavior? It seems that suddenly, relatively big chunks of memory are freed. Shouldn't memory be freed immediately after a block is exited, so shouldn't the graph be more smooth over time?

Green绿色
  • 1,620
  • 1
  • 16
  • 43
  • I bet you haven't tried analyzing your C++ program with `-fsanitize=address,undefined`. The memory behavior is likely your OS doing its best to accomodate all applications. – Ted Lyngmo Aug 11 '22 at 01:46
  • You're right, I haven't. Is this a `valgrind` or `g++` option and what does it? – Green绿色 Aug 11 '22 at 01:52
  • No, if you're using `g++` or `clang++` they are easy options. `-g -fsanitize=address,undefined` - unless you want threading help, then `-fsanitize=thread` - and it's really helpful. – Ted Lyngmo Aug 11 '22 at 01:55
  • Do you use `new` or do you use containers and pointer wrappers for memory management? If you use `new` the only thing that can release memory is `delete`, so you'll need to track where that happens. – tadman Aug 11 '22 at 02:02
  • No `new` or `malloc` in my code. As I said, I don't use pointers except for input/output names for ONNX models and I made sure those names are freed with the appropriate methods afterwards. – Green绿色 Aug 11 '22 at 02:26
  • @TedLyngmo Thanks, I'll have a try. ONNX internally uses threading, but I'll go for your proposed options first. – Green绿色 Aug 11 '22 at 02:26
  • @TedLyngmo When I compile my code with the flag `-fsanitize=address,undefined` and run it, it doesn't output anything. Does it mean, it didn't detect any issues? – Green绿色 Aug 11 '22 at 03:00
  • @Green绿色 That's a surprise. If you provided a [mre], I would give it a shot. – Ted Lyngmo Aug 11 '22 at 03:02
  • Providing a minimal reproducible example is maybe hard because I'd need to provide the ONNX model as well. As I said, if I remove the ONNX part (graph above), the memory doesn't increase. – Green绿色 Aug 11 '22 at 03:13
  • @Green绿色 Then using suppression files may help. It may not be _your_ code that leaks etc. – Ted Lyngmo Aug 11 '22 at 03:16
  • Thanks, I'll try that. However, do you have some explanation for why there's so much memory suddenly released? Is it program-specific or is there a usual explanation for this on Linux? – Green绿色 Aug 11 '22 at 03:28
  • @Green绿色 Oh, that's just your operating system, operating. Don't worry about that. It's optimized to deal with this kind of behavior in programs. – Ted Lyngmo Aug 11 '22 at 03:50
  • So, would it be correct to factor out those sudden drops and only focus on the slow increase over time? Because my current approach to this problem is to remove some code, compile and see how the memory develops over time. Some test runs are increasing, some aren't. In a binary-search fashion, I slowly approach the location(s) that are associated to an increase. However, I'm not sure whether there's other stuff going on that might be misleading. – Green绿色 Aug 11 '22 at 06:49

0 Answers0