-2

I was reading all that fancy articles about neural networks. I know I have to use it, but I'm having a problem with understanding what does Activation Layer actually do.

Could someone explain it in the easiest possible way ?

Correct me if I am wrong, but I perceive it as a weight(?) or a importance of a single neuron for a given observation. Is this kinda close ?

kwotsin
  • 2,882
  • 9
  • 35
  • 62
sebb
  • 1,956
  • 1
  • 16
  • 28
  • The activation causes the non-linearity in your model's computation, which I'd think as making it more random in its computation, and this 'randomness' is somehow better in dealing with complex problems where the answer is not as straight forward as a linear summation of variables. Another way I think of it is because of the complexity the activations introduce, you get a very complex 'formula' (which we do not really know its exact form) that answers a very specific problem, in contrast to what a simple linear formula like (w1x1 + w2x2 +w3x3...) would give. – kwotsin Feb 03 '17 at 13:37

1 Answers1

4

Activation functions are meant to introduce non-linear behavior into the network. A network without any activations is completely linear and this way it cannot learn many interesting problems.

The basic idea of activations in neural networks is that of function composition, each layer has one activation and this "stacks" as the network gets deeper, increasing the complexity of the functions that the network can represent.

Dr. Snoopy
  • 55,122
  • 7
  • 121
  • 140
  • Great explanation :) I only add, nonlinear transformation transform features into the another space and features after transformation may be separable within new space. – viceriel Feb 03 '17 at 14:19
  • @viceriel Yes I forgot to mention feature transformations, but separability is quite specific for classification. – Dr. Snoopy Feb 03 '17 at 16:44