2

Edit: The issue seems to be with SSL acccpeting and a memory leak.

I have noticed if you have long lived Processes(its a server), and clients send data to the server(recv), the Erlang garbage collection never gets called (or rarely)

Servers need data (to preform actions), and the data can be variable length (due to a message like "Hello" or "How are you doing"). Because of this, it seems like the Erlang process will accumulate garbage.

How can you properly handle this, the Erlang process has to touch the recv data, so is it unavoidable? Or do you have to come up with designs that touches the variable length data the less amount of times (like immediately passing it to a port driver).

Spawning a worker to process the data is a bad solution(millions of connections ...), and using workers would basically be the same thing, right? So that leaves me with very few options.

Thanks ...

Mike5050
  • 625
  • 1
  • 7
  • 22
  • 1
    You need more details about erlang GC, i recommend to read https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html – Pouriya Aug 20 '17 at 20:05

3 Answers3

3

If the server holds on to the received message longer than it needs to, it's a bug in the server implementation. Normally, the server should forget all or most references to the data in a request when that request has finished processing, and the data will then become garbage and will eventually get collected. But if you stick the data from each request in a list in the process state, or in an ets table or similar, you will get a memory leak.

There is a bit of an exception with binaries larger than 64 bytes, because they are handled by reference counting, and to the memory allocator it can look like there's no need to perform a collection yet, although the number of bytes used off-heap by such binaries can be quite large.

RichardC
  • 10,412
  • 1
  • 23
  • 24
0

Just incase anyone finds themselves in the same boat. This is known/happening when hammering the server with many connections at once. I think.

Without ssl, my sessions are at ~8KB roughly, and resources are being triggered for GC as expected. With SSL its an increase to ~150KB, and the memory keeps growing and growing and growing.

http://erlang.org/pipermail/erlang-questions/2017-August/093037.html

Mike5050
  • 625
  • 1
  • 7
  • 22
  • You could try to experiment with forcing GC by calling erlang:garbage_collect/0/1/2; see http://erlang.org/doc/man/erlang.html#garbage_collect-0. If GC:ing after each request works, try to cut it down to every n:th request and figure out a good value for n. – RichardC Aug 23 '17 at 11:07
  • I dont know if it fixes the issue or not, but even with all those fixes using SSL increases user process by 142+KB (and that is without any logic, just a ssl:recv). This is unacceptable. The solution on my end will be to just SRP login credentials and to enable some type of encryption (maybe AES) for the data. – Mike5050 Aug 23 '17 at 12:41
0

You might be having a problem with large SSL/TLS session tables. We (OTP team) have historically had some issues with those tables as they may grow very large if you do not have some limiting mechanisms. Alas in the latest ssl version one of the limiting mechanism was broken, however it is easily fixed by this patch. As a workaround you can also sacrifice some efficiency and disable session reuse.

diff --git a/lib/ssl/src/ssl_manager.erl b/lib/ssl/src/ssl_manager.erl
index ca9aaf4..ef7c3de 100644
--- a/lib/ssl/src/ssl_manager.erl
+++ b/lib/ssl/src/ssl_manager.erl
@@ -563,7 +563,7 @@ server_register_session(Port, Session, #state{session_cache_server_max = Max,

do_register_session(Key, Session, Max, Pid, Cache, CacheCb) ->
     try CacheCb:size(Cache) of
-   Max ->
+   Size when Size >= Max ->
    invalidate_session_cache(Pid, CacheCb, Cache);
_ ->    
    CacheCb:update(Cache, Key, Session),
Ingela Andin
  • 1