Why we must drop out in Tensorflow?

Question

I have read this article, https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout. Dropout will help prevent overfitting by make non-active neutron in ANN. But the next question...

why we must dropout neutron since we can adjust how much neutron in ANN?. For example, what is different of this code?.

FIRST

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(100, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])

SECOND

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(80, activation='relu'),
  tf.keras.layers.Dense(10)
])

We use 80 neutron instead of 100 which will dropout 20 of that neutron

score 1 · Accepted Answer · answered Apr 05 '20 at 03:45

In each training phase using dropout, some neurons are randomly selected and removed. In test phase, every neuron is used.

Thus, the first one is to use 100 neurons but they are trained likethis. [First time] H1, H2, H3, ..., H80 are trained [Second time] H61, H62, ..., H100 are trainedm ....

The second one is to use 80 neurons, and they are trained in every time.

score 1 · Answer 2 · answered Apr 05 '20 at 03:46

Dropout probabilistically removes few neurons on Training to reduce overfitting. In the first code, during training 20% neuron will be dropped out which means weights linked to those neurons will not be updated during training. During Testing all the neurons will be there and the network will see all the 100 neurons. For the second case, during training and testing, the network will see 80 neurons in both training and testing face.

Why we must drop out in Tensorflow?

2 Answers2