Autograd can automatically differentiate native Python and Numpy code and is also used by the deep learning framework PyTorch. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can also take derivatives of derivatives of derivatives. The main intended application of Autograd is gradient-based optimization.
Questions tagged [autograd]
362 questions
5
votes
1 answer
Do all variables in the loss function have to be tensor with grads in pytorch?
I have the following function
def msfe(ys, ts):
ys=ys.detach().numpy() #output from the network
ts=ts.detach().numpy() #Target (true labels)
pred_class = (ys>=0.5)
n_0 = sum(ts==0) #Number of true negatives
n_1 = sum(ts==1)…
CutePoison
- 4,679
- 5
- 28
- 63
5
votes
1 answer
How to get the full Jacobian of a derivative in PyTorch?
Lets consider a simple tensor x and lets define another one which depends on x and have multiple dimension : y = (x, 2x, x^2).
How can I have the full gradient dy/dx = (1,2,x) ?
For example lets take the code :
import torch
from torch.autograd…
Kilian Hersent
- 93
- 1
- 7
5
votes
2 answers
Using automatic differentiation libraries to compute partial derivatives of an arbitrary tensor
(Note: this is not a question about back-propagation.)
I am trying so solve on a GPU a non-linear PDE using PyTorch tensors in place of Numpy arrays. I want to calculate the partial derivatives of an arbitrary tensor, akin to the action of the…
BenjaminDSmith
- 170
- 1
- 9
5
votes
1 answer
Pytorch - why does preallocating memory cause "trying to backward through the graph a second time"
Suppose I have a simple one-hidden-layer network that I'm training in the typical way:
for x,y in trainData:
optimizer.zero_grad()
out = self(x)
loss = self.lossfn(out, y)
loss.backward()
optimizer.step()…
dkv
- 6,602
- 10
- 34
- 54
4
votes
2 answers
Pytorch's autograd issue with joblib
There seems to be a problem mixing pytorch's autograd with joblib. I need to get gradient in parallel for a lot of samples. Joblib works fine with other aspects of pytorch, however, when mixing with autograd it gives errors. I made a very small…
Roy
- 65
- 2
- 15
- 40
4
votes
1 answer
PyTorch backward() on a tensor element affected by nan in other elements
Consider the following two examples:
x = torch.tensor(1., requires_grad=True)
y = torch.tensor(0., requires_grad=True)
z = torch.full((2, ), float("nan"))
z0 = x * y / y
z1 = x + y
print(z0, z1) # tensor(nan, grad_fn=) tensor(1.,…
Jerry
- 107
- 7
4
votes
0 answers
How to use a for loop length as an optimization parameter in Pytorch
I am trying to register my loop's length as an optimization parameter so that the optimizer will automatically adjust the length of the loop.
Here is an example code (the real problem is more complex but you will get the idea):
import…
Etienne D
- 111
- 4
4
votes
1 answer
Explanation of the tuple grad_fn.next_functions in PyTorch
The torch computation graph is composed of grad_fn. For the terminating node, the grad_fn object has an attribute called next_functions which is a tuple. I understand that using the first element (0th index) of the tuple, I can reconstruct the…
pecey
- 651
- 5
- 13
4
votes
1 answer
Efficient way to compute Jacobian x Jacobian.T
Assume J is the Jacobian of some function f with respect to some parameters. Are there efficient ways (in PyTorch or perhaps Jax) to have a function that takes two inputs (x1 and x2) and computes J(x1)*J(x2).transpose() without instantiating the…
Milad
- 4,901
- 5
- 32
- 43
4
votes
1 answer
Using autograd to compute Jacobian matrix of outputs with respect to inputs
I apologize if this question is obvious or trivial. I am very new to pytorch and I am trying to understand the autograd.grad function in pytorch. I have a neural network G that takes in inputs (x,t) and outputs (u,v). Here is the code for G:
class…
arijit_the_game
- 53
- 1
- 5
4
votes
2 answers
How to wrap PyTorch functions and implement autograd?
I'm working through the PyTorch tutorial on Defining new autograd functions. The autograd function I want to implement is a wrapper around torch.nn.functional.max_pool1d. Here is what I have so far:
import numpy as np
import torch
import torch.nn as…
Sean Lake
- 578
- 2
- 8
- 14
4
votes
1 answer
Improve performance of autograd jacobian
I'm wondering how the following code could be faster. At the moment, it seems unreasonably slow, and I suspect I may be using the autograd API wrong. The output I expect is each element of timeline evaluated at the jacobian of f, which I do get, but…
Cam.Davidson.Pilon
- 1,606
- 1
- 17
- 31
4
votes
1 answer
How to fix "Can't differentiate w.r.t. type " error when using autograd in python
I am getting an error "Can't differentiate w.r.t. type " when using the autograd function in python.
Basically, I am trying to write code for a generalized linear model (GLM) and I want to use autograd to get a function that describes the…
I. Stone
- 61
- 1
- 3
4
votes
1 answer
PyTorch - Do overwritten variables stay in the graph?
I am wondering if PyTorch Tensors where the Python Variables are overwritten are still being kept in the computational graph of PyTorch.
So here is a small example, where I have an RNN Model where hidden states (and some other variables) are reset…
MBT
- 21,733
- 19
- 84
- 102
4
votes
1 answer
pyTorch can backward twice without setting retain_graph=True
As indicated in pyTorch tutorial,
if you even want to do the backward on some part of the graph twice,
you need to pass in retain_graph = True during the first pass.
However, I found the following codes snippet actually worked without doing so.…
ROBOT AI
- 1,217
- 3
- 16
- 27