'No such file or directory' error after submitting a training job

Question

I execute:

gcloud beta ml jobs submit training ${JOB_NAME} --config config.yaml

and after about 5 minutes the job errors out with this error:

Traceback (most recent call last): 
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) 
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals 
File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 232, in <module> tf.app.run() 
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run sys.exit(main(sys.argv[:1] + flags_passthrough)) 
File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 228, in main run_training() 
File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 129, in run_training data_sets = input_data.read_data_sets(FLAGS.train_dir, FLAGS.fake_data) 
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 212, in read_data_sets with open(local_file, 'rb') as f: IOError: [Errno 2] No such file or directory: 'gs://my-bucket/mnist/train/train-images.gz'

The strange thing is, as far as I can tell, that file exists at that url.

score 3 · Answer 1 · answered Sep 29 '16 at 16:23

3

This error usually indicates you are using a multi-region GCS bucket for your output. To avoid this error you should use a regional GCS bucket. Regional buckets provide stronger consistency guarantees which are needed to avoid these types of errors.

For more information about properly setting up GCS buckets for Cloud ML please refer to the Cloud ML Docs

answered Sep 29 '16 at 16:23

Amir Hormati

329
2
6

1

I do have exactly the same problem and I have a regional bucket : EUROPE-WEST1. I am using the python `open` function, which might not work. – Thomas Reynaud Oct 18 '16 at 15:38
I have the same problem. – Konstantin Solomatov Apr 22 '17 at 04:48
@ThomasReynaud You should use tf.file_io instead. They wrap gs:// correctly. – Konstantin Solomatov Apr 22 '17 at 15:17
For all future readers, Amir Hormati's solution worked for me. Note that I was not using the `open` function though but still had OP's error. – Exa Apr 20 '21 at 14:34

score 1 · Answer 2 · answered Jun 25 '17 at 20:16

Normal IO does not know how to deal with GCS gs:// correctly. You need:

first_data_file = args.train_files[0]
file_stream = file_io.FileIO(first_data_file, mode='r')

# run experiment
model.run_experiment(file_stream)

But ironically, you can move files from the gs://bucket to your root directory, which your programs CAN then actually see:

with file_io.FileIO(gs://presentation_mplstyle_path, mode='r') as input_f:
    with file_io.FileIO('presentation.mplstyle', mode='w+') as output_f:
        output_f.write(input_f.read())

mpl.pyplot.style.use(['./presentation.mplstyle'])

And finally, moving a file from your root back to a gs://bucket:

with file_io.FileIO(report_name, mode='r') as input_f:
    with file_io.FileIO(job_dir + '/' + report_name, mode='w+') as output_f:
        output_f.write(input_f.read())

Should be easier IMO.

'No such file or directory' error after submitting a training job

2 Answers2