How do I implement multiprocessing on different hosts using Pyro4

Question

I'm trying to implement Pyro4 on different hosts.

On a worker node, I implemented multiprocessing in this manner:

    import Pyro4
    from multiprocessing import Pool, Manager

    @Pyro4.expose
    class PyroClass(object):


    def parallel_calls():
        pool = Pool()
        try:
            pool.map(self.function, enumerate(self.p.results()))
        finally:
            pool.close()
            pool.join()

    def function_name(self):
        print("test")

However, this multiprocessing doesn't seem to work.

The concept here is that on each host, cores will be maximized to their fullest.

I am still new to this and I don't know the workarounds.

Does "doesn't seem to work" mean you're getting errors, or is something else happening? — larsks, Sep 27 '20 at 12:30
"doesn't seem to work" means that it assigns a new process to another host instead of just simply creating a new process within the object. — Sidnetopia, Sep 27 '20 at 16:17

score 0 · Answer 1 · answered Sep 28 '20 at 09:17

Apparently, multiprocessing.Pool is not supported in Pyro4.

For workarounds, I implemented the multiprocessing in the ff manner:

try:
    processes = [Process(target=self.__function_name, args=(result,)) for result in enumerate(self.p.results())]

    for p in processes:
        p.start()

    for p in processes:
        p.join()
except Exception as e:
    print(e)

How do I implement multiprocessing on different hosts using Pyro4

1 Answers1