I was surprised to find out that there is a significant performance difference in terms of speed and model accuracy when using different backends for the same Deep Learning problem, namely the famous MNIST hand-written digit recognition. The code below has completely different outputs depending on which backend I used. Since my MacBook comes with an AMD Radeon Pro 560x GPU, I used PlaidML backend for GPU-based training. Then I switched the backend back to Tensorflow CPU, and both the speed and the accuracy dropped significantly.
#!/usr/bin/env python
import os
#os.environ["KERAS_BACKEND"] = "tensorflow"
#import keras
# import tensorflow
# from tensorflow import keras
# from tensorflow.keras.datasets import mnist
# from tensorflow.keras.models import Sequential
# from tensorflow.keras.layers import Dense, Dropout, Flatten
# from tensorflow.keras.layers import Conv2D, MaxPooling2D
# from tensorflow.keras import backend as K
os.environ["KERAS_BACKEND"] = "plaidml.keras.backend"
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras import backend as K
batch_size = 128
num_classes = 10
epochs = 12
# input image dimensions
img_rows, img_cols = 28, 28
# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()
if K.image_data_format() == 'channels_first':
x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
input_shape = (1, img_rows, img_cols)
else:
x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test))
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
The results for the PlaidML backend is as follows:
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
INFO:plaidml:Opening device "metal_amd_radeon_pro_560x.0"
Train on 60000 samples, validate on 10000 samples
Epoch 1/12
60000/60000 [==============================] - 21s 356us/step - loss: 0.2686 - acc: 0.9160 - val_loss: 0.0551 - val_acc: 0.9826
Epoch 2/12
60000/60000 [==============================] - 17s 290us/step - loss: 0.0900 - acc: 0.9732 - val_loss: 0.0538 - val_acc: 0.9828
Epoch 3/12
60000/60000 [==============================] - 18s 295us/step - loss: 0.0678 - acc: 0.9796 - val_loss: 0.0357 - val_acc: 0.9887
Epoch 4/12
60000/60000 [==============================] - 17s 288us/step - loss: 0.0554 - acc: 0.9830 - val_loss: 0.0462 - val_acc: 0.9853
Epoch 5/12
60000/60000 [==============================] - 18s 294us/step - loss: 0.0466 - acc: 0.9854 - val_loss: 0.0312 - val_acc: 0.9899
Epoch 6/12
60000/60000 [==============================] - 18s 296us/step - loss: 0.0415 - acc: 0.9877 - val_loss: 0.0299 - val_acc: 0.9893
Epoch 7/12
60000/60000 [==============================] - 17s 285us/step - loss: 0.0362 - acc: 0.9889 - val_loss: 0.0310 - val_acc: 0.9904
Epoch 8/12
60000/60000 [==============================] - 17s 290us/step - loss: 0.0337 - acc: 0.9900 - val_loss: 0.0254 - val_acc: 0.9920
Epoch 9/12
60000/60000 [==============================] - 17s 287us/step - loss: 0.0314 - acc: 0.9905 - val_loss: 0.0284 - val_acc: 0.9911
Epoch 10/12
60000/60000 [==============================] - 38s 635us/step - loss: 0.0288 - acc: 0.9911 - val_loss: 0.0282 - val_acc: 0.9909
Epoch 11/12
60000/60000 [==============================] - 28s 466us/step - loss: 0.0270 - acc: 0.9918 - val_loss: 0.0267 - val_acc: 0.9915
Epoch 12/12
60000/60000 [==============================] - 17s 291us/step - loss: 0.0234 - acc: 0.9929 - val_loss: 0.0249 - val_acc: 0.9919
Test loss: 0.024859706979990005
Test accuracy: 0.9919
The results for the Tensorflow backend is as follows:
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
2020-06-26 11:19:40.480676: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-06-26 11:19:40.522835: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fee96fa2510 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-26 11:19:40.522857: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Epoch 1/12
469/469 [==============================] - 52s 110ms/step - loss: 2.2788 - accuracy: 0.1488 - val_loss: 2.2448 - val_accuracy: 0.3032
Epoch 2/12
469/469 [==============================] - 53s 112ms/step - loss: 2.2231 - accuracy: 0.2634 - val_loss: 2.1760 - val_accuracy: 0.4383
Epoch 3/12
469/469 [==============================] - 62s 133ms/step - loss: 2.1507 - accuracy: 0.3570 - val_loss: 2.0862 - val_accuracy: 0.5263
Epoch 4/12
469/469 [==============================] - 69s 147ms/step - loss: 2.0570 - accuracy: 0.4280 - val_loss: 1.9673 - val_accuracy: 0.5936
Epoch 5/12
469/469 [==============================] - 62s 133ms/step - loss: 1.9348 - accuracy: 0.4843 - val_loss: 1.8131 - val_accuracy: 0.6555
Epoch 6/12
469/469 [==============================] - 56s 120ms/step - loss: 1.7855 - accuracy: 0.5313 - val_loss: 1.6273 - val_accuracy: 0.7145
Epoch 7/12
469/469 [==============================] - 58s 125ms/step - loss: 1.6176 - accuracy: 0.5739 - val_loss: 1.4250 - val_accuracy: 0.7579
Epoch 8/12
469/469 [==============================] - 61s 131ms/step - loss: 1.4518 - accuracy: 0.6086 - val_loss: 1.2300 - val_accuracy: 0.7858
Epoch 9/12
469/469 [==============================] - 63s 134ms/step - loss: 1.3029 - accuracy: 0.6394 - val_loss: 1.0629 - val_accuracy: 0.8057
Epoch 10/12
469/469 [==============================] - 59s 125ms/step - loss: 1.1806 - accuracy: 0.6632 - val_loss: 0.9307 - val_accuracy: 0.8176
Epoch 11/12
469/469 [==============================] - 69s 147ms/step - loss: 1.0802 - accuracy: 0.6850 - val_loss: 0.8292 - val_accuracy: 0.8270
Epoch 12/12
469/469 [==============================] - 63s 135ms/step - loss: 1.0001 - accuracy: 0.7041 - val_loss: 0.7502 - val_accuracy: 0.8357
Test loss: 0.7502195835113525
Test accuracy: 0.8356999754905701
real 12m15.867s
user 59m51.727s
sys 30m20.694s
I tried to run the same code on Google Colab Notebook with GPU Enabled with the following results:
CPU times: user 2 µs, sys: 1e+03 ns, total: 3 µs
Wall time: 5.72 µs
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Epoch 1/24
235/235 [==============================] - 6s 25ms/step - loss: 2.2962 - accuracy: 0.1233 - val_loss: 2.2759 - val_accuracy: 0.1710
Epoch 2/24
235/235 [==============================] - 6s 24ms/step - loss: 2.2674 - accuracy: 0.1693 - val_loss: 2.2421 - val_accuracy: 0.2451
Epoch 3/24
235/235 [==============================] - 6s 24ms/step - loss: 2.2344 - accuracy: 0.2293 - val_loss: 2.2036 - val_accuracy: 0.3520
Epoch 4/24
235/235 [==============================] - 6s 24ms/step - loss: 2.1973 - accuracy: 0.2848 - val_loss: 2.1584 - val_accuracy: 0.4723
Epoch 5/24
235/235 [==============================] - 6s 24ms/step - loss: 2.1523 - accuracy: 0.3398 - val_loss: 2.1037 - val_accuracy: 0.5635
Epoch 6/24
235/235 [==============================] - 6s 24ms/step - loss: 2.0966 - accuracy: 0.4012 - val_loss: 2.0366 - val_accuracy: 0.6309
Epoch 7/24
235/235 [==============================] - 6s 24ms/step - loss: 2.0308 - accuracy: 0.4533 - val_loss: 1.9543 - val_accuracy: 0.6766
Epoch 8/24
235/235 [==============================] - 6s 24ms/step - loss: 1.9489 - accuracy: 0.4958 - val_loss: 1.8547 - val_accuracy: 0.7109
Epoch 9/24
235/235 [==============================] - 6s 25ms/step - loss: 1.8527 - accuracy: 0.5337 - val_loss: 1.7375 - val_accuracy: 0.7357
Epoch 10/24
235/235 [==============================] - 6s 24ms/step - loss: 1.7465 - accuracy: 0.5607 - val_loss: 1.6061 - val_accuracy: 0.7532
Epoch 11/24
235/235 [==============================] - 6s 24ms/step - loss: 1.6292 - accuracy: 0.5872 - val_loss: 1.4659 - val_accuracy: 0.7702
Epoch 12/24
235/235 [==============================] - 6s 24ms/step - loss: 1.5146 - accuracy: 0.6075 - val_loss: 1.3261 - val_accuracy: 0.7851
Epoch 13/24
235/235 [==============================] - 6s 24ms/step - loss: 1.4024 - accuracy: 0.6272 - val_loss: 1.1938 - val_accuracy: 0.7976
Epoch 14/24
235/235 [==============================] - 6s 24ms/step - loss: 1.3001 - accuracy: 0.6442 - val_loss: 1.0753 - val_accuracy: 0.8071
Epoch 15/24
235/235 [==============================] - 6s 24ms/step - loss: 1.2117 - accuracy: 0.6586 - val_loss: 0.9745 - val_accuracy: 0.8172
Epoch 16/24
235/235 [==============================] - 6s 24ms/step - loss: 1.1344 - accuracy: 0.6747 - val_loss: 0.8900 - val_accuracy: 0.8249
Epoch 17/24
235/235 [==============================] - 6s 24ms/step - loss: 1.0698 - accuracy: 0.6881 - val_loss: 0.8203 - val_accuracy: 0.8292
Epoch 18/24
235/235 [==============================] - 6s 24ms/step - loss: 1.0108 - accuracy: 0.7033 - val_loss: 0.7624 - val_accuracy: 0.8370
Epoch 19/24
235/235 [==============================] - 6s 24ms/step - loss: 0.9621 - accuracy: 0.7141 - val_loss: 0.7140 - val_accuracy: 0.8441
Epoch 20/24
235/235 [==============================] - 6s 24ms/step - loss: 0.9267 - accuracy: 0.7212 - val_loss: 0.6742 - val_accuracy: 0.8498
Epoch 21/24
235/235 [==============================] - 6s 24ms/step - loss: 0.8904 - accuracy: 0.7322 - val_loss: 0.6397 - val_accuracy: 0.8543
Epoch 22/24
235/235 [==============================] - 6s 24ms/step - loss: 0.8588 - accuracy: 0.7395 - val_loss: 0.6105 - val_accuracy: 0.8579
Epoch 23/24
235/235 [==============================] - 6s 24ms/step - loss: 0.8309 - accuracy: 0.7464 - val_loss: 0.5850 - val_accuracy: 0.8607
Epoch 24/24
235/235 [==============================] - 6s 24ms/step - loss: 0.8031 - accuracy: 0.7561 - val_loss: 0.5624 - val_accuracy: 0.8643
Test loss: 0.5624315142631531
Test accuracy: 0.864300012588501
As you can see, the differences are significant. The accuracy with the PlaidML backend is above 99%, while on TensorFlow it is less than 80%. The execution time is also higher for the TensorFlow backends both with a GPU and a CPU device. What am I doing wrong here? How can I run Tensorflow efficiently on my MacBook?
Unfortunately, Keras version 2.3 is the last version with the support for multiple backends. So, in the future, Keras API will also be available for TensorFlow. So, I want to be able to use Keras efficiently on my MacBook in the future as well.
Device and Software specifications:
Python version: 3.7.6
Tensorflow version: 2.2.0
Keras version: 2.2.4
Tensroflow.keras version: 2.3.-tf
Radeon Pro 560X:
Chipset Model: Radeon Pro 560X
Type: GPU
Bus: PCIe
PCIe Lane Width: x8
VRAM (Total): 4 GB
Vendor: AMD (0x1002)
gMux Version: 5.0.0
Metal: Supported, feature set macOS GPUFamily2 v1
CPU:
Processor Name: 6-Core Intel Core i7
Processor Speed: 2,6 GHz
