Stack Overflow
Stack Overflow

Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

343 questions
5
votes
2 answers

Considerations for using ReLU as activation function

I'm implementing a neural network, and wanted to use ReLU as the activation function of the neurons. Furthermore, I'm training the network with SDG and back-propagation. I'm testing the neural network with the paradigmatic XOR problem, and up to…
tulians
  • 439
  • 5
  • 23
5
votes
2 answers

Neural Network composed of multiple activation functions

I am using the sknn package to build a neural network. In order to optimize the parameters of the neural net for the dataset I am using I am using an evolutionary algorithm. Since the package allows me to build a neural net where each layer has a…
benj rei
  • 329
  • 2
  • 5
  • 12
4
votes
1 answer

Details about alpha in tf.nn.leaky_relu( features, alpha=0.2, name=None )

I am trying to use leaky_relu as my activation function for hidden layers. For parameter alpha, it is explained as: slope of the activation function at x < 0 What does this means? What effect will the different values of alpha have on the results…
Anthony0202
  • 43
  • 1
  • 4
4
votes
2 answers

when do we not need activation function?

I wrote a very basic tensorflow model where I want to predict a number: import tensorflow as tf import numpy as np def HW_numbers(x): y = (2 * x) + 1 return y x = np.array([1.0,2.0,3.0,4.0,5.0,6.0,7.0], dtype=float) y =…
Aditya
  • 113
  • 1
  • 7
4
votes
2 answers

ValueError: Unknown activation function:swish_activation

I am trying to load the save weights using keras load_model(). from keras.models import load_model model=load_model("weights.hdf5") This is error i am getting. ValueError Traceback (most recent call…
Gaurav Sahadev
  • 51
  • 1
  • 4
4
votes
1 answer

How To specify model.compile for binary_crossentropy, activation=sigmoid and activation=softmax?

I am trying to figure out how to match activation=sigmoid and activation=softmax with the correct model.compile() loss parameters. Specifically those associated with binary_crossentropy. I have researched related topics and read the docs. Also I…
Jon
  • 343
  • 3
  • 9
4
votes
1 answer

How to make a custom activation function in tensorflow

I need to make an activation function which is not exist in tensorflow.How should I do? I ever saw this link, How to make a custom activation function with only Python in Tensorflow? but I still don't know how to implement the new type of activation…
YFye
  • 77
  • 2
  • 4
4
votes
2 answers

Matrix multiplication in pyTorch

I'm writing a simple neural network in pyTorch, where features and weights both are (1, 5) tensors. What are the differences between the two methods that I mention below? y = activation(torch.sum(features*weights) + bias) and yy =…
Fariman Kashani
  • 856
  • 1
  • 16
  • 29
4
votes
1 answer

Custom activation with parameter

I'm trying to create an activation function in Keras that can take in a parameter beta like so: from keras import backend as K from keras.utils.generic_utils import get_custom_objects from keras.layers import Activation class Swish(Activation): …
user7867665
  • 852
  • 7
  • 25
4
votes
3 answers

What if we do not apply activation to the hidden layers and only to the output layer of a feed forward neural network?

Are there any hidden gotchas here? Even popular ReLU is max(0,x) where we will allow the maximum value to pass and clip the negative as zero. What is the problem if we allow both positive and negative values? Or why are we clipping the negative…
Sumith
  • 161
  • 7
4
votes
3 answers

Activation functions - Neural Network

I am working with neural network in my freetime. I developed already an easy XOR-Operation with a neural network. But I dont know when I should use the correct activations function. Is there an trick or is it just math logic?
david-schor
  • 260
  • 2
  • 11
4
votes
2 answers

Why adding one more layer to the Tensorflow simple neural net example breaks it?

Here is a basic Tensorflow network example (based on MNIST), complete code, that gives roughly 0.92 accuracy: import numpy as np import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data mnist =…
Massyanya
  • 2,844
  • 8
  • 28
  • 37
3
votes
2 answers

In Tensorflow, why add an activation function to a model only when preparing to export it?

In the Tensorflow ML Basics with Keras tutorial for making a basic text classification, when preparing the trained model for export, the tutorial suggests including the TextVectorization layer into the Model so it can "process raw strings". I…
jac
  • 141
  • 4
3
votes
1 answer

How can you get the activations of the neurons during an inference in tensorflow?

I would like to specifically know how to get the neurons in a neural network are getting activated (the outputs of each neuron after the activation function) How can I get the activations of all the neurons of a sequential model when I give input…
RabbitBadger
  • 539
  • 7
  • 19
3
votes
2 answers

Activation functions: Softmax vs Sigmoid

I've been trying to build an image classifier with CNN. There are 2300 images in my dataset and two categories: men and women. Here's the model I used: early_stopping = EarlyStopping(min_delta = 0.001, patience = 30, restore_best_weights =…
bearthum
  • 107
  • 2
  • 10
1 2
3
…
22 23