Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.
Questions tagged [activation-function]
343 questions
2
votes
3 answers
How to customize Keras layer names and also have it automatically increment layer.name
I am currently trying to have multiple layers with a customized activation with the name cust_sig. But when I try to compile the model, I get a ValueError raised as multiple layers have the same name cust_sig. I am aware that I can manually change…
DVK
- 475
- 2
- 6
- 27
2
votes
2 answers
Can't get PELU or SineRelu activation function to work in keras with contribute module
When I try to replace LeakyRELU or relu in a working coding with either SineRELU or PELU. I keep getting this error:
ValueError: Unknown activation function:PELU
I'm using the keras.contrib. I attached example code. I have tried it in several…
Gadi Licht
- 21
- 1
2
votes
1 answer
Is there a simple way to extend an existing activation function? My custom softmax function returns: An operation has `None` for gradient
I want to implement an attempt to make softmax faster by using only the top k values in the vector.
For that I tried implementing a custom function for tensorflow to use in a model:
def softmax_top_k(logits, k=10):
values, indices =…
JtheB
- 98
- 6
2
votes
1 answer
Faster implementation for ReLu derivative in python?
I have implemented ReLu derivative as:
def relu_derivative(x):
return (x>0)*np.ones(x.shape)
I also tried:
def relu_derivative(x):
x[x>=0]=1
x[x<0]=0
return x
Size of X=(3072,10000).
But it's taking much time to compute. Is there any…
Talha Yousuf
- 301
- 2
- 12
2
votes
3 answers
How does ReLu work with zero-centered output domain?
In the problem i am trying to solve, my output domain is zero centered, between -1 an 1. When looking up activation functions i noticed that ReLu outputs values between 0 and 1, which basically would mean that your output is all negative or all…
learningthemachine
- 574
- 7
- 18
2
votes
1 answer
Getting the output prior to non-linear activation in Keras
How can I get the value prior to activation when I use the following syntax to define a layer in Keras:
model.add(Convolution2D(128, 5, 5, activation='relu'))
I know that I can simply use:
model.add(Convolution2D(128, 5,…
Mark.F
- 1,624
- 4
- 17
- 28
2
votes
1 answer
What is the difference between keras.activations.softmax and keras.layers.Softmax?
What is the difference between keras.activations.softmax and keras.layers.Softmax? Why are there two definitions of the same activation function?
keras.activations.softmax: https://keras.io/activations/
keras.layers.Softmax:…
Amir Saniyan
- 13,014
- 20
- 92
- 137
2
votes
1 answer
using tanh as activation function in MNIST dataset in tensorflow
I am working on simple MLP neural network for MNIST dataset using tensorflow as my homework. in the question we should implement a multilayer perceptron with tanh as activation function. I should use the data label with [-1,+1].For example for…
m.ar
- 33
- 6
2
votes
0 answers
XOR NN not learnable with 2 hidden nodes and sigmoid activation?
I felt like my backpropagation intuition wasn't crystal clear, so I wrote a neural network class to train/predict on XOR. It has 2 inputs, 1 output, a variable number of hidden nodes, and bias nodes for the hidden and output layer.
What I noticed is…
Austin
- 6,921
- 12
- 73
- 138
2
votes
1 answer
how to define the derivative of a custom activation function in keras
I have a custom activation function and its derivative, although I can use the custom activation function I don't know how to tell keras what is its derivative.
It seems like it finds one itself but I have a parameter that has to be shared between…
Tissuebox
- 1,016
- 3
- 14
- 36
2
votes
0 answers
Differing results for MNIST autoencoder due to different placement of activation function
I stumbled across a strange phenomenon while playing around with variational autoencoders. The problem is quite simple to describe:
When defining the loss function for the VAE, you have to use some kind of reconstruction error. I decided to use my…
DocDriven
- 3,726
- 6
- 24
- 53
2
votes
1 answer
ReLU activation function with neuralnet package in R
Due to the neuralnet package doesn't have ReLU function, so I try to write the code for ReLU function. But there is an error I don't understand. Please see my code and error information below.
relu<-function(x){ifelse(x>=0,x,0)}
nn <-…
Jeffrey
- 41
- 2
- 8
2
votes
1 answer
Does having multiple activation function type neurons in a single layer make sense?
I am wondering if there exists any case or needs for having multiple type of neurons which have different activation functions to each other, mixed within a single layer, and if so, how to implement that using tensorflow Estimator framework.
I can…
nursmaul
- 33
- 7
2
votes
4 answers
Advanced Custom activation function in keras + tensorflow
def newactivation(x):
if x>0:
return K.relu(x, alpha=0, max_value=None)
else :
return x * K.sigmoid(0.7* x)
get_custom_objects().update({'newactivation': Activation(newactivation)})
I am trying to use this activation…
Pavithran Ravichandiran
- 1,711
- 1
- 17
- 20
2
votes
1 answer
Interpretation of prelu weights
What is the interpretation of prelu weights, if weights of prelu in a layer are close to 1, and in some other layer they are close 0?
Not much prelu literature around, any help would be really helpful!
Ryan
- 8,459
- 14
- 40
- 66