How do I stop a Python Ray Cluster

Question

I've been looking through the documentation, and I can't find any way to stop Ray after an answer is calculated within a tolerance level. Right now I'm writing the answer to a csv with a print('Ok to stop'), and then I kill the process manually. I'd like to stop all of the workers, and then have it automatically move on to another problem. Is there an error that I can raise that will make all of the workers stop?

Thanks.

score 3 · Answer 1 · answered Jan 16 '21 at 03:26

3

If you start a ray cluster with ray.init, the cluster should be killed when your python program exists. If you start it with ray start, you can use the ray stop command to shutdown the cluster. You can also use ray stop --force to forcefully kill all processes left.

answered Jan 16 '21 at 03:26

Sang

885
5
4

1

I don't want to wait for the python program to finish if it has found an acceptable value. I'm starting the cluster with ray.init(), and I was hoping there way a way to stop ray from within the python program. – Jan 16 '21 at 04:00
1

Same issue here, from within a single python process I need to start and stop clusters at several point in the workflow and there doesn't seem to be a mechanism to do this? ray.shutdown() doesn't seem to fully remove the cluster. – user3062260 Oct 11 '21 at 09:48

score 3 · Answer 2 · answered May 18 '21 at 13:46

3

Have a line in your Python code to break the process if a condition is met and then shutdown your Ray instance:

ray.init()

....
if tolerance_level > 50:
    break

ray.shutdown()

answered May 18 '21 at 13:46

Rohan George Thampi

61
4

score 2 · Answer 3 · answered Jan 16 '21 at 18:14

2

To stop a Ray cluster from Python, you can use ray.shutdown().

answered Jan 16 '21 at 18:14

keej

121
2

Before running `ray.shutdown()` you can check if ray is running at all with `if ray.is_initialized():` – Elijas Dapšauskas Dec 31 '22 at 05:54

How do I stop a Python Ray Cluster

3 Answers3