Update progressbar inside multiprocessor functions (Ray) simultaneously

Question

I'm writing a program that uses ray package for multiprocessing programming. In the program, there is a function that would be called 5 times at the same time. During the execution, I want to show a progress bar using PyQT5 QprogressBar to indicate how much work is done. My idea is to let every execution of the function updates the progress bar by 20%. So I wrote the code like the following:

 running_tasks = [myFunction.remote(x,y,z,self.progressBar,QApplication) for x in myList]
 Results = list(ray.get(running_tasks))

Inside myFunction, there is a line to update the sent progress bar as the following:

QApplication.processEvents()
progressBar.setValue(progressBar.Value()+20)

But, when I run the code, I got the following error:

TypeError: Could not serialize the argument <PyQt5.QtWidgets.QProgressBar object at 0x000001B787A36B80> for a task or actor myFile.myFunction. Check https://docs.ray.io/en/master/serialization.html#troubleshooting for more information.

I searched through the internet (The URL returns 404) and I understand that this error is because multiprocessing in ray doesn't have shared memory between the processors, and sending a class attribute (like self.prgressBar) will lead each processor to have its own copy where it will modify it locally only. I also tried using the multiprocessing package instead of ray but it throws a pickling error, and I assume it is due to the same reason. So, Can anyone confirm if I'm right? or provide a further explanation about the error? Also, how can I achieve my requirement in multiprocessing (i.e. updating the same progress bar simultaneously) If multiprocessing doesn't have shared memory between the processors?

Andew · Answer 1 · 2022-06-23T20:13:16.980

1

I am unfamiliar with ray, but you can do this in the multiprocessing library using the multiprocessing.Queue().

The Queue is exactly as it's named, a queue where you can put data for other multiprocesses to read. In my case I usually put a dictionary in the Queue with a Command (Key) and what to do with that command (Value).

In one multiprocess you will do Queue.put() and in the other you can do Queue.get(). If you want to pass in one direction. In the example below I emulate what you may be looking to do.

I usually use a QTimer to check if there is any data in the queue, but you can also check whenever you feel like by calling a method to do so.

from multiprocessing import Process, Queue
myQueue = Queue()

class FirstProcess():
    ...

    def update_progress_percentage(self, percentage):
        self.progresss_percentage = percentage

    def send_data_to_other_process(self):
        myQueue.put({"UpdateProgress":self.progresss_percentage})

class SecondProcess():
    ...
    
    def get_data_from_other_process(self):
        while not myQueue.empty():
            queue_dict = myQueue.get()
            for key in queue_dict :
                if key == "UpdateProgress":
                    percentage = queue_dict["UpdateProgress"]
                    progressBar.setValue(percentage)

edited Jun 23 '22 at 20:13

answered Jun 21 '22 at 14:21

Andew

321
1
9

The pickling error you see is because you are trying to pass a PyQt5 object through a queue. Pickling is when you basically save the state of an entire class and PyQt5 objects don't pickle well. You can pass most basic data structures through this queue though. So I would keep the data you send to more simple python objects like strings, lists, dictionaries, etc. – Andew Jun 21 '22 at 14:24
Thanks for your answer. As you mentioned, once I pass a PyQt5 object it will give me an error. The problem is that I want to reflect the progress bar change during the multiprocessing function execution, so I need to pass it as an argument. The way you do it is to manipulate the percentage only during the multiprocessing, but this will not be reflected on the actual progress bar until the processing is finished. Am I correct? – abdullatif Jun 22 '22 at 12:55
1

No. What my answer is suggesting you do is this: Start your multiprocess. Within that multiprocess, send information which can be used to figure out what the progress percentage should be back to the main thread using the multiprocessing queue. In the main thread with a timer, check the multi-process queue for that information and whenever there is data there update the progressbar – Andew Jun 22 '22 at 15:40
You could also use signals and slots instead of a timer for a faster less cpu intensive update. – Andew Jun 22 '22 at 15:41
now i got you, i will try that and update you with the results, thanks! – abdullatif Jun 23 '22 at 15:04

Update progressbar inside multiprocessor functions (Ray) simultaneously

1 Answers1