3

I have been into a little trouble lately: The memory used by GenServer processes is super high, probably because of large binary leaks.

The problem comes from here: we receive large binaries through the GenServer and we pass them to the consumer, which then interacts with that data. Now, these large binaries are never assigned to a variable and the GC doesn't go over them.

I have tried hibernating the processes after managing the data, which partially worked because the memory used by processes lowered a lot, but since binaries were not getting GC'd, the amount of memory used by them increased slowly but steadily, from 30 MBs without hibernating to 200MBs with process hibernation in about 25 minutes.

I have also tried to set :erlang.system_flag(:fullsweep_after, 0), which has also worked and lowered the memory used by processes by around 20%.

Before and after. I must say it goes down to 60-70MB used by processes from time to time.

Edit: Using :recon.bin_leak(15) frees a lot of memory -- result of :recon.bin_leak(15)

Anyhow the memory used is still high and I'm completely sure it can be fixed.

Here you have a screenshot taken from the observer in the Processes tab. As you can see, GenServer is the one eating the memory like the cookie monster.

I have researched a lot about this topic, tried all the suggestions and possible solutions that were given out there, and nevertheless, I am still in this position.

Any help is welcome.

The code is in this Github Repository

Code of interest that is probably causing this + Applications tree. 3 out of 4 processes there (<0.294.0>, <0.295.0>, <0.297.0> are using 27MB of memory.

Thank you beforehand for reading.

ShaH
  • 170
  • 1
  • 9
  • Just looking at this, but this has a bit of code smell to it: https://github.com/satom99/coxir/blob/master/lib/coxir/stage/middle.ex#L99-L104 Have you been able to isolate where the leak is coming from? Perhaps time the execution of all your functions? Its hard to diagnose the entire pipeline, but to me it seems to be coming from the producer consumer. – Botonomous Sep 12 '18 at 17:11
  • Did you try copying messages with `:binary.copy/1`? – Milan Jaric Sep 13 '18 at 12:50
  • I have seen that lowering the amount of created Middles lowered the memory usage a lot. About copying messages with `:binary.copy/1`, how would I apply that in my situation? Botonomous, I added a `buffer_size` to the [producer_consumer](https://github.com/satom99/coxir/pull/25/files#diff-2ff44e3f492cc3dd6cd17d282cfbd9ffR15) and that has lowered the memory used by a little. – ShaH Sep 13 '18 at 13:28

1 Answers1

1

You can try to add the :hibernate atom to your handle_events return values in your GenStage related modules. For example:

def handle_events(events, _from, %{handler: handler, public: public} = state) do
  public = handle(handler, events, public)
  {:noreply, [], %{state | public: public}, :hibernate}
end

Another option is to record the PIDs after :recon.bin_leak() and then pass them to Process.info(PID) to get some more information about the offending GenServers.

Some additional resources: https://elixirforum.com/t/extremely-high-memory-usage-in-genservers/4035/23

https://www.erlang-in-anger.com/ (Specifically Chapter 7 on Memory Leaks)

Alex Koutmos
  • 326
  • 2
  • 4