4

I currently have Google Cloud ML Engine setup to train models created in Keras. When using Keras, it seems ML Engine does not automatically save the logs to a storage bucket. I see the logs in the ML Engine Jobs page but they do not show in my storage bucket and therefore I am unable to run tensorboard while training.

You can see the job completed successfully and produced logs:enter image description here

But then there are no logs saved in my storage bucket:enter image description here

I followed this tutorial when setting up my environment: (http://liufuyang.github.io/2017/04/02/just-another-tensorflow-beginner-guide-4.html)

So, how do I get the logs and run tensorboard when training a Keras model on ML Engine? Has anyone else had success with this?

hellowill89
  • 1,538
  • 2
  • 15
  • 26

1 Answers1

7

You will need to create a callback keras.callbacks.TensorBoard(..) in order to write out the logs. See Tensorboad callback. You can supply GCS path as well (gs://path/to/my/logs) to the log_dir argument of the callback and then point Tensorboard to that location. You will add the callback as a list when calling model.fit_generator(...) or model.fit(...). 

tb_logs = callbacks.TensorBoard(
            log_dir='gs://path/to/logs',
            histogram_freq=0,
            write_graph=True,
            embeddings_freq=0)

model.fit_generator(..., callbacks=[tb_logs])
Puneith Kaul
  • 364
  • 2
  • 4
  • Awesome! That totally worked. I was using the Tensorboard callback but didn't realize I could put a GCS path in it. – hellowill89 Jul 13 '17 at 22:21