Dying ReLU problem when initial training data normalized

Question

I have used training data which is normalised and consist of n features. The number of training examples are m. I have implemented Deep Learning model Keras with the first layer as

model.add(layers.Dense(32,input_shape=(n,),activation='relu')

As my training data is normalised with mean 0 and std 1 should the neural network suffers from dying Relu problem as many data points have value less than 0 during training.

Should Relu be used in the first layer when the training data is normalised with mean 0 and std 1?

score 0 · Answer 1 · answered Apr 17 '20 at 16:38

0

As long as you don't apply the relu directly on the input it should work!

answered Apr 17 '20 at 16:38

rmeertens

4,383
3
17
42

Thanks for the answer – Sarthak Bansal Apr 17 '20 at 16:39
Thanks for your help. Can you please help me on this one https://stackoverflow.com/q/62307806/12361444 – Sarthak Bansal Jun 10 '20 at 17:17

desertnaut · Accepted Answer · 2020-04-17T23:35:13.063

As I have explained elsewhere, in

model.add(Dense(32,input_shape=(n,),activation='relu')

when used as a first layer in the Keras sequential API (i.e. with an input_shape argument), there is an implicit input layer; this is more clearly shown when we write the same thing using the Keras functional API, i.e.:

inputs = Input(shape=(n,))                 # input layer
x = Dense(32, activation='relu')(inputs)   # 1st hidden layer

So, from this viewpoint, it is apparent that ReLU is not applied directly on your inputs.

Dying ReLU problem when initial training data normalized

2 Answers2