0

I'm trying to use google cloud ml with GPU mode.

When I train BASIC_GPU mode, I have many error log.
But, It works training well.
I am not sure whether the learning was good working in GPU mode.

This is error log history.
enter image description here

This is the some part of print config.log_device_placement.
enter image description here

Also, I tried training complex_model_m_gpu mode. I also have error log like BASIC_GPU.
But, I can't see gpu:/1, gpu:/2, gpu:/3 when i print config.log_device_placement. Only gpu:/0 i can see.

The important thing is that BASIC_GPU and complex_model_m_gpu have same speed for running time.

I wonder whether the learning was good working in GPU mode or there is something wrong.

Sorry for my english, anyone knows the problem then help me.
thank you.

kanghyuk
  • 49
  • 5

1 Answers1

0

Please refer to TensorFlow's performance guide for optimizing for GPUs for tips on how to make the most of your GPUs.

A couple things to note

  • You can turn on logging of device placement to see which ops get assigned to which Devices. This is a great way to check that ops are actually assigned to GPUs and that you are using all GPUs when you have multiple GPUs.
  • TensorBoard should also provide information about device placement so that is another way to check that you are using all GPUs.
  • When using multiple GPUs, you need to make sure you are assigning ops to all GPUs. The TensorFlow guide provides more information on this topic.
Jeremy Lewi
  • 6,386
  • 6
  • 22
  • 37