Stack Overflow
Stack Overflow

Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

343 questions
1
vote
1 answer

What are the differences between type of layer and its activation function in PyTorch?

I am trying to write some simple neural network using pytorch. I am new to this library. I faced with two ways of implementing the same idea: a layer with some fixed activation function (e.g. tanh). The first way to implement it: l1 = nn.Tanh(n_in,…
MefAldemisov
  • 867
  • 10
  • 21
1
vote
1 answer

Why do both tf.nn.relu and tf.nn.sigmoid work the same in this custom estimator

This is the guide to make a custom estimator in TensorFlow: https://www.tensorflow.org/guide/custom_estimators The hidden layers are made using tf.nn.relu: # Build the hidden layers, sized according to the 'hidden_units' param. for units in…
Dee
  • 7,455
  • 6
  • 36
  • 70
1
vote
1 answer

How to change to sigmoid to learn multi-label classification

I'm trying to use 'inception resnet v2.py' to do a multi-label classification. I used sigmoid, but the result is not good. Do you know exactly where to…
AZDDKKS
  • 41
  • 1
  • 8
1
vote
0 answers

Randomized ReLU in Keras

It is proved that Randomized ReLU, as well as Parametric ReLU, perform better compared to ReLU in CIFAR-10, CIFAR-100 and NDSB datasets. While Keras has provided documentation on how to use PReLU, I was not able to find one for RReLU. Can someone…
Anirudh M
  • 19
  • 5
1
vote
0 answers

Sigmoid Activation Function: ValueError data type not inexact / RuntimeWarning: overflow encountered in exp

I encountered " RuntimeWarning: overflow encountered in exp " while implementing Sigmoid in my code. I tried to solved it by using this solution sigmoid RuntimeWarning: overflow encountered in exp . Nonetheless, now a new error occurs on the…
user5431233
  • 61
  • 1
  • 8
1
vote
1 answer

Why does multi-class classification fails with sigmoid?

MNIST trained with Sigmoid fails while Softmax works fine I am trying to investigate how different activation affects the final results, so I implemented a simple net for MNIST with PyTorch. I am using NLLLoss (Negative log likelihood) as it…
Brandon Lee
  • 695
  • 1
  • 10
  • 22
1
vote
0 answers

Tensorflow apply different activation functions in output layer

I want to build a network like this The hidden layer is not important, I want to know how can I write the code in my output layer, and the following is my code, am I right? Parameters: state_dim = 13 layer1_size, layer2_size = 400, 300 action_dim…
Tan Tan Tan
  • 11
  • 2
1
vote
1 answer

Using a custom step activation function in Keras results in "An operation has `None` for gradient." error. How to resolve this?

I am building auto-encoder and I want to encode my values into a logical matrix. However, when I'm using my custom step activation function in one of the intermediate layers (all other layers are using 'relu'), keras raises this error: An operation…
Matas Minelga
  • 70
  • 9
1
vote
2 answers

Piecewise activation function in tensorflow and broadcasting math operation

I am trying to implement and test an activation function that I have read in a paper. I am using Keras with tensorflow backend and I want to feed the activation function to the fit method of my model. Here is the mathematical form of the…
Reza
  • 13
  • 3
1
vote
1 answer

Block Activation Function Realization in Tensorflow

I was trying to reproduce a DNN where a block activation function called BlockRelu is used. It is defined as BlockRelu I tried to write this function according to some example codes about self-defined activation functions, but these functions are…
Bowen Hou
  • 13
  • 3
1
vote
2 answers

Can I use a Sigmoid activation for my output layer, even if my CNN model is doing a regression?

Final objective: Object Midpoint calculation. I have a small dataset (around 120 images), which has an object (the same in all cases), and the labels are the normalized x,y coordinates of the midpoint of the object in the image (always between 0…
Ambareesh
  • 324
  • 1
  • 4
  • 18
1
vote
1 answer

Tensorflow, square root activation function implementation (shaping error)

For the purpose of implementing a classification NN I found some really useful tutorials, like this one (2 hidden layer, one-hot-encoding output, dropout regularization, normalization etc.) which helped me with a bit of the learning curve behind…
Paul
  • 756
  • 1
  • 8
  • 22
1
vote
2 answers

what activation function should I use to enforce rounding like behaviour

I need an activation function that rounds my tensors. the derivative(gradients) of the function round() is 0 (or None in tensorflow) which makes it unusable as an activation function. I am looking for a function that enforce rounding-like behaviour…
Tissuebox
  • 1,016
  • 3
  • 14
  • 36
1
vote
2 answers

How does Keras optimize weights on layers which have no activation?

Background: If I am not mistaken, when training a network we feed forward performing sigmoid(sum(W*x)) for every layer then in back-propagation we calculate the error and the deltas (change) then we calculated the gradients and update the…
Kevin
  • 3,077
  • 6
  • 31
  • 77
1
vote
1 answer

Why pytorch has two kinds of Non-linear activations?

Why pytorch has two kinds of Non-linear activations? Non-liner activations (weighted sum, nonlinearity): https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity Non-linear activations (other):…
nkhuyu
  • 840
  • 3
  • 9
  • 23
1 2 3
…
22 23