1

If yes, how? Also what are some examples of simple applications that can be implemented by using linear activation functions.

Jack
  • 41
  • 2
  • 6

2 Answers2

2

The answer is quite short: no.

If you use linear activation function, then your neural network is not stronger then a simple perceptron (lack of hidden layer).

In other words there is no point in using hidden layer with linear activation function.

lejlot
  • 64,777
  • 8
  • 131
  • 164
  • How would I use linear activation function for AND/OR? – Jack Nov 07 '15 at 12:35
  • Again, for linearly separable problems, such as AND/OR you **do not nead hidden layer at all** (you can use it, but it is redundant, and shows lack of understanding of basics of machine learning). just linear model, for example cl_AND(x1,x2) = sign(x1+x2-1.5), cl_OR(x1,x2) = sign(x1+x2-0.5). – lejlot Nov 07 '15 at 12:41
  • Example (AND): Input/weight =>1/0.3 1/-0.4 Bias=> 1/0.3 What is the next step using linear function? How do you set a threshold? Where is the difference between linear and step? – Jack Nov 07 '15 at 13:33
  • you seem to confuse model as such with some training methods. There is no such thing as "next step"; I gave you final weights. For AND => 1 (first input neuron), 1 (second input neuron), -1.5 (bias); thus on the output neuron (without hidden layer) f(x1, x2) = x1 + x2 - 1.5. If you insist on having hidden layer, just put a single hidden neuron, with weight "1" to output one - it will do nothing but pass the signal. – lejlot Nov 07 '15 at 21:55
  • I understand it would have to be a single neuron. Give me a simple example of where one would use a linear activation function. – Jack Nov 08 '15 at 09:12
  • As I said - **you never use a linear activation function**, you use a linear model then (lack of hidden layer). For an artificial problem - AND is a good example, you can perfectly classify it with the weights I gave you. For more "real" example - in NLP, when you have extremely high dimensional spaces, using just linear models is often enough (despite modern love to deep techniques; look for example to works of Mikolov) – lejlot Nov 08 '15 at 09:27
  • Thanks a lot for all the feedback/information. One last thing, in that case would it be something like if f(x1, x2)>0 then output=1 else output=0 – Jack Nov 08 '15 at 12:23
  • For a binary classification you add a sign rule at a top, so yes. – lejlot Nov 08 '15 at 12:28
1

XOR can't be separated linearly. if you print the output of the function in a plane of 2D and you tried to find a way to separate it linearly you won't find. that was the breakdown of NN at the beginning until they found different Activation method that allow to separate point non-linearly like sigmoid function. 

a   b   a XOR b
1   1      0
0   1      1
1   0      1
0   0      0

XOR PLOT

Activation function

H. Pauwelyn
  • 13,575
  • 26
  • 81
  • 144
Feras
  • 834
  • 7
  • 18