0

I have implemented an amqplib consumer that is experiencing some issues related to memory overconsumption.

I'm unsure whether to classify it as a memory leak since, on one hand, the process seems to stabilize at a lower memory level, but on the other hand, it never returns to the initial level.

The amqplib consumer performs the following steps:

  1. It fetches messages from a RabbitMQ server, and these messages contain a file path.
  2. It downloads the file from the specified path using buffer streams. At this point, the buffer object in the consumer holds the entire file data.
  3. It then uploads the file to Amazon S3 using the AWS SDK library.

I have already exposed and executed the Garbage Collector, which has significantly improved memory usage by cleaning up numerous dead objects. However, if I attempt to download/upload a large file, the memory decreases but still remains significantly high.

Using the GC, the profile goes from this: profil before GC

To this: enter image description here

Now I've also tried closing all stream objects, finding the buffers and setting them to null at the end of every upload, still, the GC does not go as far as bringing MEM use low enough.

Short of restarting the process every now and then, what are the possible mitigation strategies I can implement ?

Akheloes
  • 1,352
  • 3
  • 11
  • 28
  • 2
    It's only a leak if repeating the process over and over continues to raise the total memory consumed each time. Otherwise, it's just the heap manager deciding not to return everything back to the OS (possibly due to heap fragmentation) and holds onto memory that is not in use, but is available for future use. – jfriend00 Jul 06 '23 at 15:44
  • You're probably right, I found out by accident that once I try downloading/uploading a small file the process releases the memory for some reason. – Akheloes Jul 07 '23 at 07:51
  • That may relate to the Buffer heap which is different than the regular JS object heap. I'm not aware of any way to specifically control that from JS. You could try allocating a small Buffer object and then dereferencing it to simulate the small file upload/download. – jfriend00 Jul 07 '23 at 16:33

0 Answers0