Remote calls are blocking when used on methods in an Actor Object?

Question

Executing the following will not work concurrently, instead it will first execute Run1 and block until it's completed, before it will execute Run2.

@ray.remote
class Test:
    def __init__(self):
        pass

    def Run1(self):
        print('Run1 Start')
        sleep(5)
        print('Run1 End')

    def Run2(self):
        print('Run2')

ray.init()
test = Test.remote()
test.Run1.remote()
test.Run2.remote()

sleep(10)

Output:

(pid=8109) Run1 Start
(pid=8109) Run1 End
(pid=8109) Run2

This is a bit unexpected. How can I enforce that the methods get executed concurrently?

EDIT TO ADDRESS THE FOLLOW UP COMMENTS:

Doing a dual threaded approach doesn't seem to work. The below code consistently results in broken pipes from PyArrow. I'd like to run both, the self.PreloadSamples method as well as the self.Optimize methods consistently in parallel. The BufferActor class collects and provides batched samples through the @ray.remote decorated GetSamples() method. Since data on GPU is not serializable, this needs to be done on the Optimizer object side, and I want to make sure that this gets done in parallel and not sequentially with respect to the optimization.

See below for a fully isolated version of the problem that replicates the issues after about 1 minute of running:

import torch
import ray
import threading
from time import sleep


def Threaded(fn):
    def wrapper(*args, **kwargs):
        thread = threading.Thread(target=fn, args=args, kwargs=kwargs)
        thread.start()
        return thread
    return wrapper

@ray.remote
class BufferActor():
    def __init__(self):
        pass

    def GetSamples(self):
        return torch.randn(32, 100)


@ray.remote(num_gpus=1)
class OptimizerActor():
    def __init__(self, bufferActor):
        self.bufferActor = bufferActor
        self.samplesOnGPU = list()

        self.PreloadSamples()
        self.Optimize()

    @Threaded
    def PreloadSamples(self):
        #this retrieves a batch of samples (in numpy/torch format on CPU)
        if (len(self.samplesOnGPU) < 5):
            samples = ray.get(self.bufferActor.GetSamples.remote())

            self.samplesOnGPU.append(samples.to('cuda'))

            print('Samples Buffer: %s' % len(self.samplesOnGPU))
        else:
            sleep(0.01)

        self.PreloadSamples()

    @Threaded
    def Optimize(self):
        if (len(self.samplesOnGPU) > 0):
            samples = self.samplesOnGPU.pop(0)
            print('Optimizing')

            #next we perform loss calc + backprop + optimizer step (not shown)

        sleep(0.01)
        self.Optimize()



ray.init()

bufferActor = BufferActor.remote()
optimizerActor = OptimizerActor.remote(bufferActor)

sleep(60*60)

score 1 · Accepted Answer · answered Jun 13 '19 at 04:47

1

Actors will execute one method at a time to avoid concurrency issues. If you want parallelism with actors (which you normally do), the best way is to start two (or more) actors and submit tasks to them both.

answered Jun 13 '19 at 04:47

Robert Nishihara

3,276
16
17

The problem is I need the methods to pass data that's not serializable, that's why I am trying to execute within the same class. In this specific case, I want to pre-load data from a BufferActor, so that it can be put onto the GPU, ready for the Backpropagation/Optimizer to process. 70% of the time is spent on getting the data onto the GPU, so I'd really like this to be done in the background continuously as opposed to sequentially. I tried using threads (with `ray.get()` inside) but that frequently led to Broken Pipes. – Muppet Jun 13 '19 at 15:43
In that case, I think something like having the actor method kick off a background thread that does some work makes sense. Do you need to call `ray.get` in the background thread? – Robert Nishihara Jun 13 '19 at 18:30
Yes, I do. It needs to pull in samples for the BufferActor. I have adapted my above code so that you can see how I'm handling it right now. Unfortunately the code always breaks with "broken pipe" at some (random) point, even if it runs well for a few minutes. – Muppet Jun 13 '19 at 18:47

Remote calls are blocking when used on methods in an Actor Object?

1 Answers1

Linked