0

Reading the following from TPU FAQ: https://cloud.google.com/tpu/docs/faq

Can I train a Recurrent Neural Network (RNN) on Compute Engine?

In certain configurations, tf.static_rnn() and tf.dynamic_rnn() are compatible with the current TPU execution engine. More generally, the TPU supports both tf.while_loop() and TensorArray, which are used to implement tf.dynamic_rnn(). Specialized toolkits such as CuDNN are not supported on the TPU, as they contain GPU-specific code. Using tf.while_loop() on the TPU does require specifying an upper bound on the number of loop iterations so that the TPU execution engine can statically determine the memory usage.

How can i make my SimpleRNN static or valid for running on a Colab TPU?

Colab TPU code

import tensorflow as tf
import os
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, SimpleRNN

resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
print("All devices: ", tf.config.list_logical_devices('TPU'))

strategy = tf.distribute.TPUStrategy(resolver)

with strategy.scope():  
  model = Sequential()
  model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
  model.add(Dense(16, activation="relu"))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='rmsprop')

model.fit(X,y, epochs=50, batch_size=16, verbose=0)
Bob Smith
  • 36,107
  • 11
  • 98
  • 91
Machine Learning
  • 485
  • 6
  • 15
  • looks like the answer isn't quite as simple as I would like it to be:https://www.hpcwire.com/2020/07/27/hardware-acceleration-of-recurrent-neural-networks-the-need-and-the-challenges/ – Machine Learning Aug 25 '20 at 18:13
  • "Further, in their ISCA 2017 paper, Google researchers have compared the performance of MLP (multi-layer perceptron), CNN and LSTM (long short term memory) models on Google’s TPU version 1 which can provide a peak-performance of 92 TOPS/sec. They observed that while CNN0 achieved a performance 86 TOPS/sec, LSTM0 and LSTM1 achieved only 3.7 and 2.8 TOPS/sec, respectively. Evidently, the hardware acceleration of RNNs is more challenging than that of CNNs" – Machine Learning Aug 25 '20 at 18:16
  • microsoft brainwave is the way to accelerate RNN because it is designed for matrix-vector multiplication which RNN is based on – Machine Learning Aug 25 '20 at 18:29
  • 2
    "Here is an example Colab that uses RNN + TPU: https://colab.research.google.com/github/GoogleCloudPlatform/tensorflow-without-a-phd/blob/master/tensorflow-rnn-tutorial/02_Keras_RNN_TPU_temperatures_playground.ipynb" - [Zachary Cain](https://stackoverflow.com/users/13385794/zachary-cain?tab=profile) – jysohn Sep 22 '20 at 15:50

0 Answers0