quit 1000 of QThread - pyqt5

Question

I have python gui app using Qt

I'm using pyqt5

the app should creating 1000 or more Qthreads, each thread will use pycurl to open external URL

I'm using this code to open the threads:

self.__threads = []
# workerClass thread
for i in range(1000):
    workerInstance =  workerClass()
    workerInstance.sig.connect(self.ui.log.append)
    thread = QThread()
    self.__threads.append((thread, workerInstance))
    workerInstance.moveToThread(thread)
    thread.started.connect(workerInstance.worker_func)
    thread.start()

WorkerClass will use a simple pycurl to visit an external URL and emit signal to add some info on the log, here is the code:

class workerClass(QObject):
    x = 1
    sig = pyqtSignal(str)

    def __init__(self, linksInstance, timeOut):
        super().__init__()
        self.linksInstance = linksInstance
        self.timeOut = timeOut

    @pyqtSlot()
    def worker_func(self):
        while self.x:
            # run the pycurl code
            self.sig.emit('success!') ## emit success message

Now the problem is with stopping all this, I used the below code for stop button:

def stop_func(self):
    workerClass.x = 0
    if self.__threads:
        for thread, worker in self.__threads:
            thread.quit()
            thread.wait()

changing the workerClass.x to 0 will stop the while self.x in the class workerClass and quit and wait should close all threads

all this is working properly, BUT only in the low amount of threads, 10 or may be 100 but if I rung 1000 threads, it takes much time, I wait for 10 minutes but it didn't stopped (mean threads not terminated), however the pycurl time out is only 15 seconds

I also used: self.thread.exit() self.thread.quit() self.thread.terminate() but it gives no difference

if any one have any idea about this, this will be really great :)

Do you have any evidence that using 1000's of threads provides any benefits? If not, you really should do some testing before assuming that adding more threads will improve performance. I suspect you will find that, in python, using a relatively small pool of threads can be much more efficient. For example, see: [Optimal Number of Threads in Python](http://edmundmartin.com/optimal-number-of-threads-in-python/). — ekhumoro, Jul 03 '18 at 13:30
yes it makes a huge difference using `pycurl`, also this app is a general app, the user should choose how many threads he want to use. — Bassem Shahin, Jul 04 '18 at 13:16
I don't see why the user should care about such implementation details. — ekhumoro, Jul 04 '18 at 13:27
PS: have you considered using [pycurl multi](https://stackoverflow.com/a/8311408/984421)? — ekhumoro, Jul 04 '18 at 13:43
these details gives the user the full control to choose the speed and/or other factors, so he have the option to choose the timeout and the threads amount, also thanks for your advice to try pycurl multi, I will test it as it also may gives some better performance! — Bassem Shahin, Jul 06 '18 at 12:05

score 0 · Accepted Answer · answered Jul 06 '18 at 12:10

after some tries, I removed the pycurl code and test to start 10k QThreads and stopped it, and this working perfect without any errors. so I define that the error is with the pycurl code.

as per the advice of ekhumoro in the question comments, I may try to use pycurl multi

but for now I added this line to the pycurl code:

c.setopt(c.TIMEOUT, self.timeOut)

I was using only this:

c.setopt (c.CONNECTTIMEOUT, self.timeOut)

so now with the both 2 parameters all is properly working, starting and stopping bulk qthreads

quit 1000 of QThread - pyqt5

1 Answers1