0

I have an erlang gen_server which receives messages from clients asynchronously(using cast). On each message handling, the server inserts them into a ordered ETS table and deletes some of them based on conditions (Because of the conditional delete, the number of records in the ETS table is in most of the time around 5000. But there are rare scenarios which can increase this size up to 200k, introducing overhead for both inserts and deletes).

This works fine until messages per second is 100k but above this it gives me the process memory high watermark and starts consuming a lot of memory. When I comment out the insert and delete part to the ordered ets, it can process more than 100k per second. Can you please give me some hints on how to solve this?

Is there any maximum limit of memory that a process can allocate? I'm using 35 GB of Memory and I set the watermark threshold by memsup:set_procmem_high_watermark(0.6). I also tried gbtree and ordered dictionary but they didn't solve the memory error.

Chathuri Gunawardhana
  • 171
  • 1
  • 10
  • I think the message queue is too long, your gen_server can't handle the message quickly. Could u share your code? – BlackMamba Apr 01 '16 at 01:24

2 Answers2

1

I have a suggestion. When it comes more than 100k/s, gen_server can not process in time message queue, so it starts to grow => gen_server gets slower and memory leak . Try to monitor erlang:process_info(Pid, message_queue_len) if it grows.

nikit
  • 153
  • 8
  • 1
    There is no memory leak in Erlang, however that's also my suspicion, that the message queue grows eating u process memory. `process_info` can also be used to get many more process information, not only queue length, i.e. allocated heap/stack. Another possibility would be to install `recon` to allow better monitoring of the node, also see this answer: http://stackoverflow.com/questions/36184551/how-to-scale-ejabberd-server-machine-on-centos-to-handle-200-k-connections/36185206#36185206 – Greg Mar 31 '16 at 12:21
0

Process memory limit

There's no such option in Erlang. memsup:set_procmem_high_watermark/1 simply sets the threshold for process memory allocation. If the threshold is passed, the alarm will be raised.

What may be the cause of the problem?

At the beginning you have to find out why your process slows down. As already stated here, check characteristics of your message queue: is it growing over time? If that's the case, your process isn't able to handle the messages at the rate they are arriving. erlang:process_info/2 will be helpful.

Another thing to keep in mind is when you insert/look up data in ETS, the objects are copied around. So, if the conditions that have to be met when deleting the objects involve retrieving the data from your ETS, that also might be possible cause of your problem.

How to tackle this?

There're several ways to approach this problem. The obvious one is to simply use back pressure mechanism by, for example, making the clients sending the messages synchronously.

Altenatively you could try to split what your one process is doing into several of them. To achieve that you could have one process distributing the messages to others based on some information contained within the messages.

Finally, you could try to apply some optimisation regarding your ETS. For instance, if you're storing big objects, you could check whether you're not retrieving them back unnecessarily (by using ets:lookup/X instead of ets:match/X or ets:match_object/X to get exactly the bits you want from the stored message).

mentels
  • 196
  • 4
  • I think ets:insert and delete takes some time which causes message queues to grow incrementally (I do not read from ets, and as I'm using an ordered ets for the deletion I delete the matching objects starting from root until the condition is satisfied). Monitored a heavy message queue and a large heap. I will try separating inserts and deletes into 2 processes. Thank you very much for your suggestions. – Chathuri Gunawardhana Apr 01 '16 at 17:57
  • Here is the code for gen_server including inserts and deletes. Please let me know If I'm doing something inefficient [ https://github.com/chathurilanchana/riak_kv/blob/rik-sequencer-load-test-batch-noft/src/riak_kv_ord_service_ets_ordered.erl ]. – Chathuri Gunawardhana Apr 01 '16 at 18:07