2

I am using Ray Tune to tune the hyper-paramters of a pytorch model. The storage capacity where the default ray session directory is located (/tmp/ray) is limited, thus I want to specify the session directory (i.e. /tmp/ray → /my_directory) when working with tune.Tuner(). I found that I could specify a parameter --temp-dir or pass it somehow to ray.init().

However, I neither call ray nor ray.init() and simply do not know where to set the session directory. Any simple tips? To tune the hyper-parameters I am basically passing an object of a tune.trainable to tune.Tuner() and call tuner.fit().

Thanks for any advise! stillsen

So far I identified the error "No space left on device" to be connected to limited capacity on /tmp. I found that there is a default ray session directory, that setting it to different location would solve the issue. But I simply to not now how and where to pass it to tune.Tuner()

stillsen
  • 41
  • 4

1 Answers1

2

if you run Ray Tune in a script without calling ray.init() yourself, it will be called for you with the default settings (it will just start a local ray cluster).

What you can do is to call it yourself before you run tuner.fit() and pass the _temp_dir argument:

import ray
from ray import tune

# ...

ray.init(_temp_dir="/path/to/tmp/ray")
tune = tune.Tuner(
    # ...
)

Alternatively, you can start Ray from the command line. Then you'd have to use ray.init("auto") in your script instead:

Terminal:

ray start --head --temp-dir /path/to/tmp/ray

Script:

import ray
from ray import tune

# ...

ray.init("auto")
tune = tune.Tuner(
    # ...
)

Lastly, you could also just symlink your /tmp/ray directory to somewhere else

Terminal:

ln -s /tmp/ray /path/to/tmp/ray
Kai
  • 241
  • 1
  • 3