Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.
Questions tagged [activation-function]
343 questions
0
votes
1 answer
Why is softmax not used in hidden layers
I have read the answer given here. My exact question pertains to the accepted answer:
Variables independence : a lot of regularization and effort is put to keep your variables independent, uncorrelated and quite sparse. If you use softmax layer as…
figs_and_nuts
- 4,870
- 2
- 31
- 56
0
votes
1 answer
Tensorflow: Incompatible shapes when making a custom activation function?
I am trying to build a neural network using custom activation functions. I followed the solution given here, and it works when the input and output vectors have the same size, but not when using different sizes (like in a pooling function). Here is…
ghyur7
- 119
- 2
- 9
0
votes
1 answer
How to use softmax activation function at the output layer, but relus in the middle layers in TensorFlow?
I have a neural net of 3 hidden layers (so I have 5 layers in total). I want to use Rectified Linear Units at each of the hidden layers, but at the outermost layer I want to apply Softmax on the logits. I want to use the DNNClassifier. I have read…
Bishwajit Purkaystha
- 1,975
- 7
- 22
- 30
0
votes
1 answer
What activation function is used in the nce_loss?
I am stuck with the nce_loss activation function in the word2vec model. I want to figure out what activation function it uses among all these listed here:
These include smooth nonlinearities (sigmoid, tanh, elu, softplus,
and softsign), continuous…
Lerner Zhang
- 6,184
- 2
- 49
- 66
0
votes
0 answers
How to model scalar values with a neural network if besides direction the magnitude matters too
Say you want to predict temperature changes based on some input data. Temperature changes are positive or negative scalars with a mean of zero. If only the direction matters one could just use tanh as an activation function in the output layer. But…
JohnDeeDoe
- 27
- 4
0
votes
1 answer
Training a TensorFlow model for regression when labels are probabilities
I am going to train a neural network (e.g., a feed-forward network) in which the output is just a real value representing a probability (and thus in the [0, 1] interval). Which activation function shall I use for the last layer (i.e., the output…
boomz
- 657
- 5
- 21
0
votes
1 answer
Torch- why don't people use activation explicitly?
Whenever I saw neural networks implemented in Torch, nn, they only plug modules together. For example, there is a sequencer module with LookupTable, Splittable, FasLSTM, Linear, LogSoftMax.
Why don't people use activation functions in between, such…
hypnoticpoisons
- 342
- 4
- 11
-1
votes
0 answers
How to turn softmax jacobian matrix to a vector gradient so it can be used in backpropegation?
I am learning how to differentiate the softmax function, I am using the article: https://towardsdatascience.com/derivative-of-the-softmax-function-and-the-categorical-cross-entropy-loss-ffceefc081d1
So in the example the (4x1) matrix is turned into…
Subeen Regmi
- 44
- 1
- 5
-1
votes
1 answer
How to initialize parameters of an activation function?
I'm studying the basics of neural networks with pytorch, and I'm having a hard time understanding how the activation function should work.
I don't understand what shape should the trainable parameters of my activation function have.
Should they have…
parwal
- 21
- 4
-1
votes
1 answer
is this the right way to apply softmax?
self.classifier = nn.Sequential(
nn.Flatten(),
nn.Linear(in_features = 32*8*8, out_features = 26),
nn.ReLU(),
nn.Linear(in_features = 26, out_features = output_shape),
nn.Softmax(dim=1)
)
and my loss fn…
UrDailyCS
- 17
- 6
-1
votes
1 answer
I want my Neural network output to be either 0 or 1, and not probabilities between 0 and 1, with customised step function at output layer
I want my Neural network output to be either 0 or 1, and not probabilities between 0 and 1.
for the same I have designed step function for output layer, I want my output layer to just roundoff the output of previous(softmax) layer i.e. converting…
Himani Deshpande
- 1
- 1
-1
votes
1 answer
How an Activation Function Works?
I'm still a little confused on how activation functions work in Neural Networks (well, confused to not be able to explain them in layman's terms). So far I have:
The activation function in a hidden layer determines whether the neuron is switched ON…
AndyC
- 59
- 5
-1
votes
1 answer
Which activation function do I use in this Neural Network, that does not have a predefined output class?
I want the neural network to guess a number as close to the actual output as possible.
What activation function best suits this scenario?
the_begging_beginner
- 159
- 1
- 14
-1
votes
1 answer
Is Leaky Relu with alpha = 1 a linear activation function?
if I set alpha = 1, then I simply extend the normal Relu function down. Should this function then be viewed as a linear activation function?
janwe
- 11
- 2
-1
votes
1 answer
Neural network for AI playing Connect Four: how to encode inputs and outputs, and what kind of NN setup?
Trying to understand how to build and train neural network based AI for games, and struggling to get some details straight.
Not even concerned yet with whether to use TensorFlow or something else or build my own. First I'm trying to grasp some…
RocketNuts
- 9,958
- 11
- 47
- 88