0

I have have trained from local use command like this:

gcloud ml-engine jobs submit training task43 --module-name=train.train --config=config.yaml --job-dir=gs://root-album-8512 --package-path=train --region=asia-east1 --staging-bucket=gs://root-album-8512

After training, where can I find my output training directory? In my log job written "Saved model checkpoint to /user_dir/runs/1503579423/checkpoints/model-227400"

But I don't know where is it. When I check my Storage or Bucket there is no such directory.

Moch. Ari Nasichuddin
  • 31
  • 1
  • 4
  • Looks like it is saved on the container. Are you sure that you're pointing it to the gs bucket? – Dat Tran Aug 25 '17 at 10:14
  • In my command, I only defined job directory "--job-dir=gs://root-album-8512" and staging bucket "--staging-bucket=gs://root-album-8512". But I'm not defined the directory for my output training. – Moch. Ari Nasichuddin Aug 25 '17 at 10:31

1 Answers1

2

The value of --job-dir gets passed as the value of the command line argument "job-dir" to your code. So your code needs to get the value from the command line argument and use that value with the model saver to save your checkpoints to that location.

If you don't set a location for the model saver, then it will end up saving to a local directory and not your GCS bucket and won't be available after the job ends.

Jeremy Lewi
  • 6,386
  • 6
  • 22
  • 37