Autograd can automatically differentiate native Python and Numpy code and is also used by the deep learning framework PyTorch. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can also take derivatives of derivatives of derivatives. The main intended application of Autograd is gradient-based optimization.
Questions tagged [autograd]
362 questions
2
votes
1 answer
Taking the derivative of the zero function in pytorch giving Runtime error
I have the following simple code:
def f(x):
return x[:,0] + x[:,1]
def g(x):
return torch.zeros_like(x[:,0])
def main():
x = torch.tensor([[0.3, 0.3],
[0.6, 0.3],
[0.3, 0.6],
[0.6, 0.6]])
…
BBB
- 55
- 4
2
votes
0 answers
How do I manually set partial derivatives for a multi-input function in pytorch?
I am writing a machine learning program for my PhD which finds poles of a rational function which approximates the solution to a given differential equation. To calculate the loss, I need to calculate this estimate which is given as a function with:…
shmirrkk
- 51
- 2
2
votes
0 answers
Computing Gradients of Weighted State Averages in PyTorch Models
I have two trained models with the same architecture and different performances. I want to construct a new model by taking the weighted average of their states (s1, s2, s1+s2=1) and calculate the gradients of the new model's loss with respect to s1…
Amin Rezaei
- 376
- 2
- 11
2
votes
2 answers
How does automatic differentiation with respect to the input work?
I've been trying to understand how automatic differentiation (autodiff) works. There are several implementations of this that can be found in Tensorflow, PyTorch and other programs.
There are three aspects of automatic differentiation that currently…
Abe Brandsma
- 139
- 1
- 1
- 7
2
votes
1 answer
Performance gap between `batch_size==32` and `batch_size==8, gradient_accumulation==4`
I tried to use gradient accumulation in my project. To my understanding, the gradient accumulation is the same as increasing the batch size by x times. I tried batch_size==32 and batch_size==8, gradient_accumulation==4 in my project, however the…
namespace-Pt
- 1,604
- 1
- 14
- 25
2
votes
2 answers
PyTorch Autograd for Regression
another PyTorch newbie here trying to understand their computational graph and autograd.
I'm learning the following model on potential energy and corresponding force.
model = nn.Sequential(
nn.Linear(1, 32),
nn.Linear(32, 32), nn.Tanh(),
…
ArkadyBogdanov
- 21
- 2
2
votes
3 answers
Pytorch bincount with gradient
I am trying to get gradient from sum of some indexes of an array using bincount. However, pytorch does not implement the gradient. This can be implemented by a loop and torch.sum but it is too slow. Is it possible to do this efficiently in pytorch…
Roy
- 65
- 2
- 15
- 40
2
votes
1 answer
Plot derivatives of sin(x) using pytorch
Am unsure why my code does not plot cos(x) (yes, am aware pytorch has cos(x) function)
import math
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
import random
x = torch.linspace(-math.pi, math.pi, 5000,…
zed111
- 49
- 1
- 4
2
votes
0 answers
Designing a custom loss function in pytorch
The custom loss function which we are designing does contain few non differentiable operations, e.g. histogram creation, and counting values greater than a threshold. We were hoping that Pytorch Autograd can automatically generate approximate…
Parth
- 65
- 5
2
votes
1 answer
Why are the gradients not equivalent when using loss.backward() v.s torch.auto.grad?
I ran into this weird behavior when trying to "manually" optimize a network's parameters via SGD. When attempting to update the model's parameters using the following way, it works just fine:
for _ in trange(epochs):
for x, y in train_loader:
…
Omar AlSuwaidi
- 1,187
- 2
- 6
- 26
2
votes
2 answers
How does batch’s element are processed by Pytorch?
I have a generic network without random element in his structure (e.g. no dropout) so that if I forward a given image input through the network, I put gradient to zero and repeat again the forward with the same image input I get the same result…
user1172131
- 103
- 7
2
votes
2 answers
Get grads of parameters w.r.t a loss term in pytorch
I my Pytorch training i use a composite loss function defined as :
.
In order to update the weights alpha and beta, i need to compute three values : which are the the means of the gradients of the loss terms w.r.t to all the weights in the…
Valentin Mercier
- 385
- 1
- 4
- 16
2
votes
1 answer
Why torch.autograd.grad() returns None with torch.cat?
I know the torch.autograd.grad() returns None if the gradient is stopped somehow, however, I am wondering what is wrong with the following snippet?
x = torch.rand(6, requires_grad=True)
y = x.pow(2).sum()
z = torch.cat([x])
grad1 =…
Tengerye
- 1,796
- 1
- 23
- 46
2
votes
3 answers
One of the variables modified by an inplace operation
I am relatively new to Pytorch. Here I want to use this model to generate some images, however as this was written before Pytorch 1.5, since the gradient calculation has been fixed then, this is the error message.
RuntimeError: one of the variables…
Linux Penguin
- 23
- 1
- 4
2
votes
1 answer
Pytorch sum jacobian over inputs instead of outputs
Suppose I have a tensor Y that is (directly or indirectly) computed from a tensor X.
Normally when I apply torch.autograd.grad(Y, X, grad_outputs=torch.ones_like(Y)), I get a gradient mask that is of the same shape as X. This mask is actually a…
Jonas De Schouwer
- 755
- 1
- 9
- 15