Stack Overflow
Stack Overflow

Questions tagged [stochastic-gradient]

Stochastic Gradient Descent (or SGD) is an algorithm used to find a minima (local or global) of a differentiable function.

35 questions
0
votes
0 answers

How to formulate the SGD (Stochastic Gradient Descent) method to solve for the maximum "energy" in a dynamical system?

What I face: I would like to maximize the "energy" of variable x governed by a dynamical system equation dx/dt=g(x) at time T. The "energy" of x is defined as E(t)=x^2/2, since x is a function of time. x could be a vector or a scalar. The…
jengmge
  • 77
  • 1
  • 5
0
votes
0 answers

How to get Stochastic Gradient Descent result in contour plot

import numpy as np from matplotlib import pyplot as plt xk = np.linspace(-1,1,100) yk= 2 * xk + 3 + np.random.rand(len(xk)) x1,x2 = np.meshgrid(xk,yk) F = (x1 - 2) ** 2 + 2 * (x2 - 3) ** 2 fig=plt.figure() surf = fig.add_subplot(1,1,1,…
user20697471
  • 1
  • 1
0
votes
0 answers

Tensorflow gradient descent algorithm

I'm currently working in python with tensorflow and would like to train my model with a gradient descent model and not a stochastic gradient descent model. The reason is that I want to train my model on all data points instead of a subset.…
Arka
  • 1
0
votes
1 answer

how to set the gradient for a network in pytorch

I have a model in pytorch. The model can take any shape but lets assume this is the model torch_model = Sequential( Flatten(), Linear(28 * 28, 256), Dropout(.4), ReLU(), BatchNorm1d(256), ReLU(), Linear(256, 128), …
Yedidya kfir
  • 1,419
  • 3
  • 17
  • 32
0
votes
1 answer

How variable alpha changes SGDRegressor behavior for outlier?

I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior. Below is the sample code I have: from…
Ross_you
  • 881
  • 5
  • 22
0
votes
0 answers

pytorch: implementing a custom optimizer

I'm trying to implement a slightly different version of SGD with pytorch and test it on some datasets. I need to write a custom optimizer on which to train my model, however I cannot find any guide which explains how to do so. Is anyone able to tell…
Francesco Borg
  • 15
  • 6
0
votes
1 answer

Does SGD in Tensorflow make a move with each data point?

I assumed the "stochastic" in Stochastic Gradient Descent came from the random selection of samples within each batch. But the articles I have read on the topic seem to indicate that SGD makes a small move (weight change) with every data point. …
Mastiff
  • 2,101
  • 2
  • 23
  • 38
0
votes
1 answer

GAN - Generator loss decreasing but Discriminator fake loss increase after a initial drop, why?

I'm learning GAN and was trying to run the pix2pix GAN model on a custom dataset, my average generator loss per epoch and average Discriminator Fake and Real loss are as follows - and I just can't understand, how come my Generator loss decrease…
Soumyajit Sarkar
  • 33
  • 1
  • 5
0
votes
1 answer

What is the problem with this SGD loss graph?

I've been trying to train audio classification model. When i used SGD with learning_rate=0.01, momentum=0.0 and nesterov=False i get the following Loss and Accuracy graphs: I can't figure out what what causes the instant decrease in loss at around…
Alp Sarıca
  • 76
  • 3
0
votes
0 answers

Accumulating a variable while updating dict values? (Trying to implement SGD + momentum)

So I have the following dictionary implemented for vanilla SGD: update_weights = dict(zip(weight_keys, [grad_weight[key] - lr * convert_to_tensor(dx[key])]) for key in…
skidjoe
  • 493
  • 7
  • 16
0
votes
1 answer

Using SGD on MNIST dataset with Pytorch, loss not decreasing

I tried to use SGD on MNIST dataset with batch size of 32, but the loss does not decrease at all. I checked my model, loss function and read documentation but couldn't figure out what I've done wrong. I defined my neural network as below class…
ILoveC
  • 35
  • 4
0
votes
1 answer

Random sampling from multiple vectors in python

So I have an assignment to code Stoachastic gradient decent and basically i am finding it a bit of a problem to randomly sample from multiple vectors while keeping the order intact. My code follows: import numpy as np import matplotlib.pyplot as…
Yousef Kaddoura
  • 31
  • 5
0
votes
1 answer

How to interpret gradient and the partial derivative when updating the weights of Neural Network?

I have just started studying nueral networks and I managed to figure out how to derive the equations necessary for back propagation. I've spent nearly 3 days asking all of my professors and googling everything I can find. My math skills are…
Danny Zhou
  • 9
  • 1
0
votes
1 answer

regression with stochastic gradient descent algorithm

I am studying regression with Machine Learning in Action book and I saw a source like below : def stocGradAscent0(dataMatrix, classLabels): m, n = np.shape(dataMatrix) alpha = 0.01 weights = np.ones(n) #initialize to all ones for…
semenbari
  • 725
  • 1
  • 8
  • 22
0
votes
0 answers

Online Learning for Yolo Network?

I want to use the Yolo Network v3 for let's say detect 5 custom object classes, for which I already have data. So I'm going to use my train data of these classes to retrain the yolo network with pre-trained weights. Now Imagine the case: After some…
f_3464gh
  • 162
  • 3
  • 11
1
2
3