I am training a Tensorflow model with, unfortunately, many CPU operations. On my local machine, I compiled Tensorflow from source with support for SSE4.2/AVX/FMA to make training run faster. When I train on gcloud via their ML engine service, I am getting 10x slow down compared to local. I suspect that Tensorflow on gcloud ML engine wasn't compile with CPU optimizations. I was wondering if what are ways around this.
Asked
Active
Viewed 353 times
0
-
What machine type are you using on Cloud ML Engine? The service should have most of the CPU optimizations built in. It might be your local machine has higher specs than the machine you selected on the service. – Jing Jing Long Oct 11 '17 at 17:56
-
The lag was caused by no GPU instead of CPU. I made the wrong assumption that the default machine tier is BASIC_GPU. After manually defining it the per instance time is more or less the same as my local machine. Thanks! – exe163 Oct 13 '17 at 23:23