Keras activation layer is not working well

Question

I made a model as below at first:

from tensorflow.keras.layers import Dense, Flatten, Conv2D, Dropout, BatchNormalization, 
     AveragePooling2D, ReLU, Activation
from tensorflow.keras import Model

class MyModel(Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv = Conv2D(4, (3,3), padding = 'same', activation = 'linear'
                           ,input_shape = x_train.shape[1:])
        self.bn = BatchNormalization()
        self.RL = ReLU()
        self.FL = Flatten()
        self.d1 = Dense(4, activation = 'relu')
        self.d2 = Dense(100, activation = 'softmax')
    def call(self,x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.RL(x)
        x = self.FL(x)
        x = self.d1(x)
        return self.d2(x)

However, this model did not work well. The accuracy is just 1% which means it learned nothing. (I trained this model with CIFAR100 - simplicity is for just checking the code) But as I changed the code as below, it worked.

class MyModel(Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv = Conv2D(4, (3,3), padding = 'same', activation = 'linear'
                           ,input_shape = x_train.shape[1:])
        self.bn = BatchNormalization()

        # The below code is changed from ReLU() -> Activation('relu')
        self.RL = Activation('relu')

        self.FL = Flatten()
        self.d1 = Dense(4, activation = 'relu')
        self.d2 = Dense(100, activation = 'softmax')
    def call(self,x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.RL(x)
        x = self.FL(x)
        x = self.d1(x)
        return self.d2(x)

Why is it happened? I don't know the problem. Thank you for reading.

It means that the model learns nothing. I edited the question. Thank you! — Geonsu Kim, May 09 '20 at 04:08

Zabir Al Nazi · Answer 1 · 2020-05-11T08:13:52.233

They are exactly equivalent, there is no difference which should make your network instable. Here is an example:

from tensorflow.keras.layers import Dense, Flatten, Conv2D, Dropout, BatchNormalization, AveragePooling2D, ReLU, Activation, Input
from tensorflow.keras import Model
import numpy as np

ip = Input(shape = 5)
rl = ReLU()(ip)

model1 = Model(ip, rl)

ip = Input(shape = (5,))
rl = Activation('relu')(ip)

model2 = Model(ip, rl)

i = np.array([[1., 2., 3., 4., 5.], [-1., -100., -123213., 0., 100000.], [-1., 100234323423., -123213., 0., 100000.]])

print(model1(i))
print(model2(i))

Out:

tf.Tensor(
[[1.0000000e+00 2.0000000e+00 3.0000000e+00 4.0000000e+00 5.0000000e+00]
 [0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.0000000e+05]
 [0.0000000e+00 1.0023432e+11 0.0000000e+00 0.0000000e+00 1.0000000e+05]], shape=(3, 5), dtype=float32)
tf.Tensor(
[[1.0000000e+00 2.0000000e+00 3.0000000e+00 4.0000000e+00 5.0000000e+00]
 [0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.0000000e+05]
 [0.0000000e+00 1.0023432e+11 0.0000000e+00 0.0000000e+00 1.0000000e+05]], shape=(3, 5), dtype=float32)

The two models have identical outputs.

Keras activation layer is not working well

1 Answers1