How can I sync the complete `local_dir` (`~/ray_results`) with the head node?

Question

I am creating custom result files in my jobs and want to sync them from the worker nodes to the head nodes (To rsync them down to my local computer later on). I tried to write them all into the local_dir e.g. ~./ray_results but unfortunately it seems that ray tune is only synching the individual trial folders in the local_dir. Is this correct?

score 1 · Accepted Answer · answered Apr 12 '20 at 20:04

1

Yes; try writing them to the self.logdir for each trainable.

answered Apr 12 '20 at 20:04

richliaw

1,925
16
14

Thanks for the answer. Is there a way to get the logdir also when using the Function-based API? – Denis Apr 14 '20 at 02:07
2

tune.get_trial_dir() is the new API – richliaw Aug 14 '20 at 04:46
Thanks for following up on this. – Denis Aug 28 '20 at 11:50

How can I sync the complete `local_dir` (`~/ray_results`) with the head node?

1 Answers1