Stack Overflow
Stack Overflow

Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

343 questions
2
votes
1 answer

multiple object detection in an image

I wanted to ask you a question about image classification. Actually I am making a image classifier and I am using convolutuional neural networks with keras and tensorflow as backend. my question is how to identify multiple objects in an image. I've…
Dexter
  • 630
  • 1
  • 11
  • 24
2
votes
1 answer

Impact of using relu for gradient descent

What impact does the fact the relu activation function does not contain a derivative ? How to implement the ReLU function in Numpy implements relu as maximum of (0 , matrix vector elements). Does this mean for gradient descent we do not take…
blue-sky
  • 51,962
  • 152
  • 427
  • 752
2
votes
1 answer

Why is ReLU is used as activation unit in Convolutional Neural Network?

I'm trying to use CNN to classify images and as far as I can see, ReLu is a popular choice for activation unit in each convolutional layer. Based on my understanding, ReLU would keep all positive image intensities and convert the negative ones to…
MinhNguyen
  • 816
  • 1
  • 11
  • 26
2
votes
1 answer

visualizing activations of layers in TensorBoard

I have a layer, layer3, that is of type: Tensor("vgg_16/conv3/conv3_3/Relu:0", shape=(1, 500, 700, 120), dtype=float32, device=/device:GPU:0) I'd like to visualize the activations of this layer. How can I process layer3 to do that? What would I…
haxtar
  • 1,962
  • 3
  • 20
  • 44
2
votes
2 answers

dropout with relu activations

I am trying to implement a neural network with dropout in tensorflow. tf.layers.dropout(inputs, rate, training) From the documentation: "Dropout consists in randomly setting a fraction rate of input units to 0 at each update during training time,…
pegazik
  • 115
  • 2
  • 9
2
votes
2 answers

Different weight functions for neurons

I have been playing around in TensorFlow and made a generic fully connected model. At each layer I'm applying sigmoid(WX + B) which as everybody knows, works well. I then started messing around with the function that is applied at each layer and…
Michael Hackman
  • 491
  • 2
  • 6
  • 21
2
votes
0 answers

Total network error get stuck in XOR gate with ReLU transfer function

I tried to use ReLU activation function on XOR problem to see its performance because I see a lot of post and page said it's better than sigmoid and others. I used this code: /** * Copyright 2010 Neuroph Project http://neuroph.sourceforge.net * *…
Kế Hoàng Văn
  • 21
  • 1
2
votes
2 answers

How Precise Does an Activation Function Need to Be and How Large Will Its Inputs Be?

I am writing a basic neural network in Java and I am writing the activation functions (currently I have just written the sigmoid function). I am trying to use doubles (as apposed to BigDecimal) with hopes that training will actually take a…
Dylan Siegler
  • 742
  • 8
  • 23
1
vote
0 answers

Usage of activation function in node classification with GraphNets

I am currently trying to learn how Graph Neural Networks work with Deep Minds Graph-Nets-Library, but I am stuck for days with my understanding of this topic. Maybe someone of you can help me out. I am using Zacharys Karate Club as graph dataset,…
NickT2606
  • 33
  • 3
1
vote
1 answer

Is the sklearn MLPRegressor linear activation producing a linear model?

Can anyone help me understand the linear activation option in the sklearn.neural_network.MLPRegressor? From the documentation: ‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x However a neural network with all…
Harry Salmon
  • 187
  • 9
1
vote
1 answer

What is negative_slope argument of tf.keras.layers.ReLU?

tf.keras.layers.ReLU has negative_slope argument which is explained as Float >= 0. Negative slope coefficient. Default to 0. tf.keras.layers.ReLU( max_value=None, negative_slope=0.0, threshold=0.0, **kwargs ) Is this to make it…
mon
  • 18,789
  • 22
  • 112
  • 205
1
vote
0 answers

I get a loss : nan when implementing mish from the scratch

I'm currently working on making a custom activation function using tf2 on python. model architecture: VGG 16, on CIFAR-10 epochs: 100 lr: 0.001 for initial 80 epochs, 0.0001 for 20 epochs optimizer: Adam loss: categorical cross entropy batch_size:…
pnpsuM
  • 11
  • 2
1
vote
0 answers

Keras Tuner and Specifying Advanced Activation Layers

I have the following Keras (Python) code for using the Keras Tuner for a single hidden layer neural network. def build_model(hp): # Initialize sequential model = keras.Sequential() # Tune the number of units in the first Dense layer …
pdhami
  • 187
  • 1
  • 9
1
vote
0 answers

How to add activation histogram in tensorboard pytorch?

I am using histogram in tensorboard pytorch to visualize weight in my model. Currently, this is how I use tensorboard to visualize my layers's weight. for name, weight in model.named_parameters( tb.add_histogram(name, weight, epoch) …
killermama98
  • 45
  • 5
1
vote
0 answers

Should I do the reverse-normalisation after pass the last activation function

I have been working on frame interpolation on Pytorch, and I have one question to ask about image normalization. Before train a model, I normalized the image dataset, with mean=[0.3852, 0.3699, 0.3618] and std=[0.2444, 0.2472, 0.2391] (this is the…
killermama98
  • 45
  • 5
1 2 3
…
22 23