6

I have a service written in go that takes 6-7G memory at runtime (RES in top). So I used the pprof tool trying to figure out where the problem is.

go tool pprof --pdf http://<service>/debug/pprof/heap > heap_prof.pdf

But there are only about 1-2G memory in result ('Total MB' in pdf). Where's the rest ?

And I've tried profile my service with GOGC=off, as a result the 'Total MB' is exactly the same as 'RES' in top. It seems that memory is GCed but haven't been return to kernel won't be profiled.

Any idea?

P.S, I've tested in both 1.0.3 and 1.1rc3.

Tianran Shen
  • 921
  • 2
  • 9
  • 11

1 Answers1

9

This is because Go currently does not give memory of GC-ed objects back to the operating system, to be precise, only for objects smaller then predefined limit (32KB). Instead memory is cached to speed up future allocations Go:malloc. Also, it seems that this is going to be fixed in the future TODO.

Edit: New GC behavior: If the memory is not used for a while (about 5 min), runtime will advise the kernel to remove the physical mappings from the unused virtual ranges. This process can be forced by calling runtime.FreeOSMemory()

Gvozden
  • 154
  • 1
  • 5
  • Thanks a lot, I've come across that thread too. Actually, I've tested on FreeOSMemory() from "runtime/debug" in Go1.1rc3. Not much help. Any recipe in current Go version available to fix this problem? Maybe manipulate some critical memory with cgo? – Tianran Shen May 14 '13 at 02:41
  • Only sane thing to do is to try to minimize temp object creation. Maybe even profiling with pprof to see what are the biggest offenders. Things like specifying capacity of slice etc. can't hurt. All other things, IMHO, should be handeld by Go's GC. – Gvozden May 14 '13 at 09:43
  • 1
    Do you know what's the current status of this issue? – Tomasz Kalkosiński Jan 30 '14 at 22:11
  • @Gvozden what if the app consumes 5xx MB as per pprof, however htop is showing a VIRT of 10.6 G, a res of 5820M? My app does not consume a whole lot per itself, but keeps too much memory, and sometimes crashes in a 8GB server. – Melardev Apr 17 '20 at 10:20
  • 1
    @Melardev, try running your app with the following environment variable: GODEBUG=madvdontneed=1 . By default, the go runtime gives back the memory back to the OS in a lazy way, using MADV_FREE. This will delay the actual reuse until the system is under memory pressure. Using this env variable will force the freeing (MADV_DONTNEED). – Gvozden Apr 20 '20 at 13:07
  • @Gvozden Thanks, I will give a try, can you elaborate more why is this happening? I understand that it caches memory for performance, but why not release it under pressure automatically instead of pushing too hard up to the point of a crash? Also do you know if madvdontneed will completly disable caching, or just disable it partially? the docs does not seem to explain that further. Thanks – Melardev Apr 20 '20 at 21:44
  • @Melardev With MADV_FREE kernel is supposed to reclaim memory under pressure. However, the memory stays accounted to the process RSS which might be picked up by the OOM. With MADV_DONTNEED application RSS should reflect the actual memory use more closely, however you incur higher latency when releasing and accessing that same memory again. I cannot tell why the default option is not working for you, if you don't exceed the available size. – Gvozden May 14 '20 at 12:57
  • Not always, there are instances where the heap used is higher in Grafana (read mem stats) than the heap pprof. – Jamel Toms Jul 07 '22 at 20:05