Stack Overflow
Stack Overflow

Questions tagged [backpropagation]

Backpropagation is a method of the gradient computation, often used in artificial neural networks to perform gradient descent.

Backpropagation is a method of the gradient computation, often used in artificial neural networks to perform gradient descent. It led to a “renaissance” in the field of artificial neural network research.

In most cases, it requires a teacher that knows, or can calculate, the desired output for any input in the training set. The term is an abbreviation for "backward propagation of errors".

1267 questions
23
votes
1 answer

When to use in-place layers in Caffe?

By setting the bottom and the top blob to be the same we can tell Caffe to do "in-place" computation to preserve memory consumption. Currently I know I can safely use in-place "BatchNorm", "Scale" and "ReLU" layers (please let me know if I'm wrong).…
dontloo
  • 10,067
  • 4
  • 29
  • 50
22
votes
2 answers

How does pytorch backprop through argmax?

I'm building Kmeans in pytorch using gradient descent on centroid locations, instead of expectation-maximisation. Loss is the sum of square distances of each point to its nearest centroid. To identify which centroid is nearest to each point, I use…
jammygrams
  • 358
  • 1
  • 2
  • 9
22
votes
1 answer

Why do sigmoid functions work in Neural Nets?

I have just started programming for Neural networks. I am currently working on understanding how a Backpropogation (BP) neural net works. While the algorithm for training in BP nets is quite straightforward, I was unable to find any text on why the…
Anshul Porwal
  • 223
  • 1
  • 2
  • 4
21
votes
3 answers

Multi-layer neural network won't predict negative values

I have implemented a multilayer perceptron to predict the sin of input vectors. The vectors consist of four -1,0,1's chosen at random and a bias set to 1. The network should predict the sin of sum of the vectors contents. eg Input = <0,1,-1,0,1>…
B. Bowles
  • 764
  • 4
  • 9
  • 21
20
votes
3 answers

numpy : calculate the derivative of the softmax function

I am trying to understand backpropagation in a simple 3 layered neural network with MNIST. There is the input layer with weights and a bias. The labels are MNIST so it's a 10 class vector. The second layer is a linear tranform. The third layer is…
Sam Hammamy
  • 10,819
  • 10
  • 56
  • 94
17
votes
3 answers

What is the difference between backpropagation and reverse-mode autodiff?

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through each layer in reverse to measure the error…
rrz0
  • 2,182
  • 5
  • 30
  • 65
17
votes
1 answer

How to correctly train my Neural Network

I'm trying to teach a neural network to decide where to go based on its inputted life level. The neural network will always receive three inputs [x, y, life]. If life => 0.2, it should output the angle from [x, y] to (1, 1). If life < 0.2, it should…
Thomas Wagenaar
  • 6,489
  • 5
  • 30
  • 73
17
votes
1 answer

Trouble Understanding the Backpropagation Algorithm in Neural Network

I'm having trouble understanding the backpropagation algorithm. I read a lot and searched a lot but I can't understand why my Neural Network don't work. I want to confirm that I'm doing every part the right way. Here is my Neural Network when it is…
Annie Caron
  • 437
  • 1
  • 5
  • 15
15
votes
1 answer

Backpropagation for Max-Pooling Layers: Multiple Maximum Values

I am currently implementing a CNN in plain numpy and have a brief question regarding a special case of the backpropagation for a max-pool layer: While it is clear that the gradient with respect to non-maximum values vanishes, I am not sure about the…
x3t2h
  • 151
  • 5
15
votes
5 answers

How to implement the Softmax derivative independently from any loss function?

For a neural networks library I implemented some activation functions and loss functions and their derivatives. They can be combined arbitrarily and the derivative at the output layers just becomes the product of the loss derivative and the…
danijar
  • 32,406
  • 45
  • 166
  • 297
14
votes
4 answers

XOR neural network error stops decreasing during training

I'm training a XOR neural network via back-propagation using stochastic gradient descent. The weights of the neural network are initialized to random values between -0.5 and 0.5. The neural network successfully trains itself around 80% of the time.…
Vivin Paliath
  • 94,126
  • 40
  • 223
  • 295
14
votes
2 answers

how is backpropagation the same (or not) as reverse automatic differentiation?

The Wikipedia page for backpropagation has this claim: The backpropagation algorithm for calculating a gradient has been rediscovered a number of times, and is a special case of a more general technique called automatic differentiation in the…
Brannon
  • 5,324
  • 4
  • 35
  • 83
14
votes
1 answer

Backpropagation training stuck

I am trying to implement a neural network in Javascript and the specifications of my project would prefer the implementation to have separate objects for each node and layer. I am rather new at programming neural networks, and I have run into a few…
Adam
  • 303
  • 3
  • 11
13
votes
1 answer

In pytorch how do you use add_param_group () with a optimizer?

The documentation is pretty vague and there aren't example codes to show you how to use it. The documentation for it is Add a param group to the Optimizer s param_groups. This can be useful when fine tuning a pre-trained network as frozen layers…
Inkplay_
  • 561
  • 2
  • 5
  • 18
13
votes
1 answer

Backpropagation in Pooling Layer (Subsamplig layer) in CNN

My doubt is how do I backpropagate error in the Pooling layer, because when I calculate the derivative, there is only 1 element of 4 (for example, when using a 2x2 pooling kernel) that affects the result of the feedforward.
Malvrok
  • 369
  • 4
  • 16
1
2
3
…
84 85