0

I don't have a GPU on my machine, since most of the performance recommondations on tensorflow mention only GPU, can someone confirm that e.g.

tf.data.prefetch
tf.distribute.mirroredstrategy
tf.distribute.multiworkerstrategy

Will only work with multi GPU ? I tried it on my PC and most of the functions realy slow down the process instead of increasing it. Therefore multi CPU is no benefit here?

Max
  • 1

2 Answers2

0

I did not understand exactly what you are asking but let me give you a 10,000ft explanation of those. It might help you to understand what/when you should use it.

  1. tf.data.prefetch : let's suppose that your have 2 steps while training your model. a) read data, b) process the data. While you are processing the data, you could be reading more data to make sure it is available once 'training' is done with the current batch of data. Just think about a producer/consumer model. You don't want your consumer idle while you are producing more data.

  2. tf.distribute.mirroredstrategy : this one helps if you have a single machine with more than one GPU. It allows to train a model in "parallel" on the same machine.

  3. tf.distribute.multiworkerstrategy : let's suppose now that you have a cluster with 5 machines. You could train your model in a distributed fashion using all of them.

This is just a simple explanation of those 3 items you mentioned here.

srishtigarg
  • 1,106
  • 10
  • 24
CrazyBrazilian
  • 1,030
  • 1
  • 11
  • 15
  • I was just surprised, cause I have no GPU and I thought e.g. the .prefetch would bring any benifit. But in reality it slows down the process (in my exmaple around 10%). Therefore my question was, if I have no GPU at all, prefetch seems not to forward the task to free capacity of any other CPU. I guess similar for the other tasks, they only adress it to GPU's but not to free CPU's – Max Dec 29 '20 at 09:27
  • It is not a rule set in stone. It depends on each case. Let's suppose you are fetching batches of data from a remote location and pushing it thru your model for training. Once the first batch is done, it might take sometime for the next batch to be available. Again, think about it as a producer/consumer model. – CrazyBrazilian Dec 29 '20 at 15:19
0

In case you haven't solved your problem yet, you can use Google Colab (https://colab.research.google.com) to get a GPU - there you can change a runtime to GPU or TPU.

Ena
  • 106
  • 7