0

We have a golang process that internally uses CGO libraries. We are observing OOM kills for this process after some time.

kernel: [322533.632311] Memory cgroup out of memory: Killed process 1895550 (grpc_latest_srvc) total-vm:3072488kB, anon-rss:634580kB, file-rss:17192kB, shmem-rss:0kB, UID:0 pgtables:1812kB oom_score_adj:936

I am debugging the memory usage of my process by printing runtime.MemStats and also using pprof package and getting heap file. The heap file is showing just 60 to 100 MB but top command RES is showing around 620MB.

In some documents I read that go GC will not release the freed memory to OS to fix that we are calling runtime.debug.FreeOSMemory() at some intervals.

But still, top RES is keep on increasing, and after few hours process gets killed with OOM. runtime.MemStats/pprof-heap data is showing consistent memory usage of around 60 to 100MB

can someone help me to understand why there was a lot of difference between the top RES and runtime.MemStats/pprof-heap?

runtime.MemStats are like:

Alloc: 104 MiB
TotalAlloc: 366909 MiB
StackInuse: 1152 KiB
Sys: 155 MiB
NumGC: 84458

Top command results:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
  69917 root      20   0 3072488 620116  31160 S 141.8   3.8  88:59.63
samba
  • 323
  • 1
  • 4
  • 14
  • 2
    Memory is complicated. These two values are just different. Nothing to see here. Stop comparing absolute values measured from inside and outside. Memory (allocation) is _fucking_ complicated. Memory leaks are _unlikely_ **unless** you leak goroutines. Make sure you do not leak goroutines. OOM can happen even without memory**leaks** if you simply use (read waste) too much memory. – Volker Aug 15 '23 at 06:51
  • I am checking a number of go routines before each request, there are no leak goroutines. – samba Aug 15 '23 at 06:57
  • 1
    Just because your heap is 60 to 100 MB at that moment does not mean the process didn't require 620MB at one point to continue operating. Calling `debug.FreeOSMemory ` isn't going to change that (in the long run it does nothing, it's in the "debug" package after all). You can read about the details of the GC [here](https://tip.golang.org/doc/gc-guide) if you want to make it work more aggressively, but no matter how low you keep average memory usage, it's only peak memory usage that matters when you are running out of memory. – JimB Aug 15 '23 at 12:46
  • 1
    Also note that the Go runtime cannot tell what is happening in the C code. Any memory allocated in C must be also be freed correctly. – JimB Aug 15 '23 at 12:58
  • @JimB thanks for the response. I collect heap files every 5 minutes until the process encounters an Out of Memory (OOM) error, and the memory usage in all the heap files is relatively consistent. – samba Aug 15 '23 at 14:02
  • 1
    @samba, if you are certain that the Go memory usage is not the problem, then the only other possibility is that the memory is allocated by your C code. – JimB Aug 15 '23 at 14:06
  • «But still, top RES is keep on increasing, and after few hours process gets killed with OOM. runtime.MemStats/pprof-heap data is showing consistent memory usage»–that's it: Go runtime does not in any way control memory used by C code (except when Go code calls `C.malloc` and `C.free` directly–these at least could be accounted for, but they are not, bear with me), and, what is more important, does not consider any memory allocated by C code as allocated by the running process… – kostix Aug 15 '23 at 14:21
  • …That's simply because 1) the calls `C.malloc` and `C.free` merely call the corresponding functions in the C library linked to the running Go process; this makes memory allocated this way "compatible" with the C code (let's not dig deeper on what this means), but 2) That's not the only way to allocate memory in C code, and 3) Any called C function is free to do memory management by itself–including allocating more memory. The cases covered by 2 and 3 above are completely hidden from the Go runtime, and won't affect its memory management metrics at all, there's simply no way to achieve that. – kostix Aug 15 '23 at 14:24
  • «the calls C.malloc and C.free merely call the corresponding functions in the C library linked to the running Go process»– to make it more clear, I mean, that these calls do not interact with the memory manager provided by the Go runtime, and which is tasked with managing memory on the running program. IOW, they are just calls to a specific C library (`libc.so` on *BSD- and Linux-based systems, `msvcrtXY.dll` on Windows etc) which are provided "automagically" by `cgo` because it's convenient to have them when `cgo` is being used. – kostix Aug 15 '23 at 14:37
  • Try something like [this](https://kirshatrov.com/posts/finding-memory-leak-in-cgo): to me, that `memleak` thing looks like a way forward. You will probably need to make all your C deps be built with debug symbols included, though. – kostix Aug 15 '23 at 17:08

0 Answers0