0

Lets say, I am constructing a neural network like so:

x = tf.nn.conv2d(input, ...)
x = tf.nn.max_pool(x, ...)
x = tf.nn.dropout(x, keep_prob=1.)
x = tf.nn.thislayershallhavedropout(x,...)
x = tf.nn.dropout(x, keep_prob=.5)

Would this be an effective technique to tell TensorFlow to just dropout the layer thislayershallhavedropout?

Basically, what I am trying to do is to tell TensorFlow to use dropout only on a single layer and not cascade back into earlier layers.

stop-cran
  • 4,229
  • 2
  • 30
  • 47
Ulu83
  • 523
  • 9
  • 20
  • 1
    I'm unclear on the semantics of what you're trying to do. `dropout` is a layer operation, much like ReLU, pool, or activation. Dropping (resetting) half the weights doesn't directly affect the weights in previous or successive layers; it's applied at this one point in the forward data flow. There is no effect on the back-propagation. – Prune May 19 '17 at 23:03
  • In short, a dropout with 100% keep is a null operation. – Prune May 19 '17 at 23:03
  • @Prune dropout drops **activations**. dropconnect drops weights. – MWB May 19 '17 at 23:30
  • Yes what I intend to do is preventing dropout from the earlier layers. But as you and Thomas explain, dropout is applied only at the layer it is applied to. – Ulu83 May 22 '17 at 21:09

1 Answers1

1

Dropout sets activations that pass through to 0 with a given chance. It's hard to give a 'layer' dropout, as you are only setting connections to 0 or to 1 with a given chance.

If you want to give the outgoing connections from a certain layers dropout, you should do:

x = tf.nn.thislayershallhavedropout(x,...)
x = tf.nn.dropout(x, keep_prob=.5)

Which you have basically done. So 50% percent of the activations coming from thislayershallhavedropout will be disactivated.

By the way, as was pointed out in the comments, setting keep_prob to 1 has no effect at all: this will let all activations pass through like normal.

x = tf.nn.dropout(x, keep_prob=1.)

Keep in mind: dropout might not directly interfere with previous layers, however, during backpropagation, weights of previous and succesive layers will adapt to half the activations being disabled. Thus, there is no way you can prevent dropout having an (indirect) effect on previous layers.

Thomas Wagenaar
  • 6,489
  • 5
  • 30
  • 73