Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.
Questions tagged [activation-function]
343 questions
1
vote
1 answer
Learnable LeakyReLU activation function with Pytorch
I'm trying to write a class for Invertible trainable LeakyReLu in which the model modifies the negative_slope in each iteration,
class InvertibleLeakyReLU(nn.Module):
def __init__(self, negative_slope):
super(InvertibleLeakyReLU,…
Nikoo_Ebrahimi
- 63
- 7
1
vote
2 answers
After training, is the prediction of the neural network achieved by just forward propagating from input to output layer?
I made a simple feedforward neural net in matlab as follows:
mynet = feedforwardnet(5)
mynet.layers{1}.transferFcn = 'poslin'; % one hidden layer(5 neurons) with poslin = ReLU activation function
mynet.layers{2}.transferFcn = 'purelin'; % last layer…
1
vote
0 answers
How to replace ReLU activation function with Mish?
I have trained ResNext50 for activity recognition. The original architecture of network includes ReLU. The test accuracy I achieved with original architecture is %85. When I replace all ReLU activation functions with Mish, accuracy is going down…
Vitriol
- 11
- 2
1
vote
1 answer
what is the best choice for an activation function in case of small sized neural networks
I am using pytorch and autograd to build my neural network architecture. It is a small 3 layered network with a sinngle input and output. Suppose I have to predict some output function based on some initial conditions and I am using a custom loss…
unstableEquilibrium
- 157
- 1
- 7
1
vote
1 answer
How can I get my neural net to correctly do linear regression?
I used the code for the first neural net from in the book neural nets and deep learning by Michael Nielsen, which was used for recognising handwritten digits. It uses stochastic gradient descent with mini batches and the sigmoid activation function.…
thebasqueinterdisciplinarian
- 27
- 1
- 9
1
vote
0 answers
Combining activation functions together - PyTorch
I'm working on a research to evolve activation functions (using genetic algorithms) and I want to combine several activation functions together.. meaning if I have ReLU and Sin function, I could have Sin(ReLU(x)) or Sin(x)+Relu(x) as an activation…
JojoHalastra
- 23
- 1
- 7
1
vote
0 answers
Why PyTorch MultiheadAttention is considered as activation function?
When scrolling all activation functions available on PyTorch package (here) I found that nn.MultiheadAttention is described there. Can you please explain why it's considered activation function? Maybe I understand something wrong, but Multihead…
demo
- 421
- 5
- 22
1
vote
1 answer
ReLU inconsistency/randomized behaviour
i wrote a simple nn (it should add two numbers) and i tried different activation functions, this is my code
class Layer:
def __init__(self):
self.inputs = None
def forward(self, inputs):
pass
def backward(self, error_gradient,…
EpsilonCode
- 29
- 8
1
vote
1 answer
Custom Activation function in tensorflow with learnable parameter for tanh
I would like to implement a custom activation function in tensorflow. The idea of this activation function is that it should learn how linear it will be. Using the following function.
tanh(x*w)/w for w!= 0
x for w = 0
The parameter w…
codeprof
- 13
- 2
1
vote
1 answer
Implementing PRelu activation as a function in TensorFlow 2.4.1
I am trying to implement PReLU activation in tensorflow 2.4.1 as given here How to implement PReLU activation in Tensorflow?
Got following error
ValueError: Variable alpha already exists, disallowed. Did you mean to set reuse=True or…
spb
- 165
- 1
- 9
1
vote
2 answers
How to use tanh instead of sigmoid in sklearn logistic regression
Is there a way to run sklearn's logistic regression with tanh?
I know tanh is better when labels are {-1,1} and sigmoid is better when labels are {0,1}
if I can't implement logistic regression would converting labels from {-1,1} -> {0, 1} would…
haneulkim
- 4,406
- 9
- 38
- 80
1
vote
2 answers
What is the activation layer used for TensorFlow text classification example
I am trying to understand the TensorFlow text classification example at https://www.tensorflow.org/tutorials/keras/text_classification. They define the model as follows:
model = tf.keras.Sequential([
layers.Embedding(max_features + 1,…
Venkat Papana
- 4,757
- 13
- 52
- 74
1
vote
0 answers
Softmax function returning nan or all zeros in a neural network
I am designing a neural network to predict MNIST dataset from scratch with layers 784, 500, 500, 10.
In forward propagation function my softmax function is returning nan values, I tried solving it bu subtracting maximum value as below
def…
Umme Ammarah
- 23
- 4
1
vote
1 answer
Bug in Neural Network with cost function rising
I have been working on my first neural net, building it completely from scratch. However when printing the cost function to track the models progress it only rises, the data I am using is just 1s,0s I wanted something simple for my first model. It…
kuzlo
- 33
- 4
1
vote
1 answer
Add custom activation function to be used with a string
I followed this response doing:
get_custom_objects().update(act_dispatcher)
Where act_dispatcher is a dictionary with all the activation functions I want to add like {'fun_1':fun_1, 'fun_2': fun_2}.
The first thing that caught my attention is that…
J Agustin Barrachina
- 3,501
- 1
- 32
- 52