Python threading causes memory leak using pm2

Question

What's going wrong?

I have done some test and realized that when using threading with pm2 it causes memory leaks where it stacks up the threading instead of dumping them. I have tested both with pm2 vs normal terminal. When using threading I could see that using normal terminal would take around 60mb~ usage. While using PM2 could stack up to be over 160MB~ with the same exact code.

What I do is that when the threading is done with its task, it should be killed with a return/or sys.exit()

How could we reproduce this issue?

This is a example I have done, I'm not sure if this could replicate it but same code should give two different RAM usage even thought its the same exact code.

EDIT:

After testing on Windows. The issue still remains. I can see that it takes around 11MB~ with PM2 while with terminal/cmd it takes around 4.5MB.

#!/usr/bin/python3
# -*- coding: utf-8 -*-

import random
import threading
import time


class Monitoring(object):

    def parseNew(self):

        ListsNumber = []

        while True:

            newtLists = random.sample(range(1, 1000), 999)

            for numbers in newtLists:
                if numbers not in ListsNumber:
                    ListsNumber.append(numbers)

                    threading.Thread(
                        target=self.threadingTest,
                        args=(numbers,)
                    ).start()
            else:
                print("sleeping")
                time.sleep(random.randint(2, 4))

    def threadingTest(self, numbers):
        print(numbers)
        return


if __name__ == '__main__':
    Monitoring().parseNew()

Supporting information

Python 3.8.2, Ubuntu 20.04.1 LTS

--- PM2 report ----------------------------------------------------------------
Date                 : Fri Sep 11 2020 15:45:24 GMT+0200 (Central European Summe                                                                                                                                                             r Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.4.0
node version         : 14.6.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : testing
uid                  : 1000
gid                  : 1000
uptime               : 50586min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.4.0
node version         : 14.6.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : testing
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Core(TM) i7-8700T CPU @ 2.40GHz
cpus nb              : 12
freemem              : 1734098944
totalmem             : 16617046016
home                 : /home/tester
===============================================================================

Another issue can be the way I am using threading but I don't think so due to it gives two different RAM usage using pm2 & cmd

instead of random sample did you try will the sample in both cases? — aRvi, Sep 20 '20 at 10:00
@aRvi Yes, just tried and it seemed to be the same situation for me if I change instead of random, a list of 100 names. The one with pm2 always had higher RAM usage than without pm2. — PythonNewbie, Sep 21 '20 at 06:45
Have you tried the [link](https://docs.python.org/3/library/gc.html) Garbage Collector module. It will help you to find out where the leak is happening. — SaaSy Monster, Sep 21 '20 at 09:21
@AresZephyr Im not sure what I should do? Is there an example you can provide - Should I just implement `gc.enable()` in the code? — PythonNewbie, Sep 21 '20 at 12:54
This article covers some good examples of implementation [link] https://www.geeksforgeeks.org/garbage-collection-python/ — SaaSy Monster, Sep 21 '20 at 14:04
@AresZephyr as I understood, it would be enough to just call gc.collect()? — PythonNewbie, Sep 21 '20 at 18:27
@AresZephyr After testing, gc.collect() did not work. The memory leak is still there. — PythonNewbie, Sep 26 '20 at 16:36

Python threading causes memory leak using pm2

What's going wrong?

How could we reproduce this issue?

Supporting information

0 Answers0