-1

I have been studying Neural Networks for a couple of weeks and noticed that all guides and documentation either never mentioned the Bias unit and/or always assumed it to be 1. Is there any reason or cases where we want a bias unit not to be 1? Or have it as an adjustable parameter in the network?

Edit:I'm sorry, i'm new to stack overflow and found similar questions so I thought this was a good place to ask, thank you for correcting me.

Edit: When people refer to bias they are in most cases referring to the bias_weight:

Bias&BiasUnit

The bias unit is also the reason we get the equation for the bias Δb in back-propagation as: Δb = ΔY * 1 (the * 1 is just normally left out as it has no effect on the equation)

Hope that clears thinks up.

ThomasB
  • 11
  • 2
  • It would be better to ask non-programming questions on other SE websites. – Amir Jan 24 '19 at 11:58
  • Can you please point to an example. The bias is usually a learned parameter just like the other weights in NNs. – SaiBot Jan 24 '19 at 12:07
  • https://medium.com/deeper-learning/glossary-of-deep-learning-bias-cf49d9c895e2 – Sreeram TP Jan 24 '19 at 12:51
  • 2
    Welcome to Stack Overflow! I'm voting to close this question as off-topic because [it is specific to machine learning rather than software development](https://meta.stackoverflow.com/q/291009/1233251). You may consider visiting [Cross Validated](//stats.stackexchange.com) or [DataScience.SE](//datascience.stackexchange.com) instead. – E_net4 Jan 24 '19 at 14:42

1 Answers1

2

This question is better suited for cross-validation or maybe data-science (not about code at all).

I think you have a misunderstanding, the bias term is a trainable parameter that is also learned and updated during training.

I think I know what is the source to your confusion (correct me if I'm wrong). In many places, the bias term is incorporated into the input vector x as a constant 1 element.

So if we have the following input:

The output for some operation can be written as:

Where the trained parameters are:

But it can also be written in the following way:

But, despite the fact that we have the constant 1 in the input, since is still one of the trainable parameters, the bias can still be anything.

Mark.F
  • 1,624
  • 4
  • 17
  • 28
  • it is that Constant 1 i'm interested in, this constant is referred to as the Bias unit. i made an edit in my question that hopefully clarifies things a little. – ThomasB Jan 25 '19 at 07:54
  • okay with dark mode it isn't possible to read the equations. But yeah after switching i could read them :D maybe use better pictures for the future – Khan May 07 '20 at 14:42