1

I'm a bit confused by activation functions and blogs/posts that continuously mention that neurons are not activated or not fired.

But mathematically speaking, if whatever activation function (whether it's sigmoid, tanh, relu) calculates an output of 0, isn't that value still given to all connected neurons in the next layer? And if so, doesn't that mean that this neuron still firing/activating?

Or am I simply wrong and the neuron is really not firing and it really doesn't provide any value at all to any connected neurons in the next layer? And how does this work mathematically?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
Bob de Graaf
  • 2,630
  • 1
  • 25
  • 43

1 Answers1

1

Expressions such as not activated and not fired, as well as the term neuron itself, are just metaphorical depictions, and they should not be taken at face value. They are used just to verbally describe the (very) loose analogy between the (artificial) neural networks used in machine learning and the actual neuronal networks of living beings, but that's all.

As you correctly suspect, in such cases an output value of 0 is indeed produced by the "neuron" and propagated in the net. Because, actually, there are not any neurons there, just variables in a computer program, which must have a value at all times, for both mathematical and computational reasons.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
  • 2
    Ah thank god! I already thought it was because of the biological comparison but words like 'not activating' kept throwing me off. Thanks for clearing up my confusion! I was thinking of deleting this question now but if there's one confused person, there might be more :) Thanks! – Bob de Graaf Mar 31 '20 at 10:58