Autograd can automatically differentiate native Python and Numpy code and is also used by the deep learning framework PyTorch. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can also take derivatives of derivatives of derivatives. The main intended application of Autograd is gradient-based optimization.
Questions tagged [autograd]
362 questions
2
votes
2 answers
Cannot find in-place operation causing "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:"
I'm relatively new to PyTorch and am trying to reproduce an algorithm from an academic paper that approximates a term using the Hessian matrix. I've set up a toy problem so that I can compare the results of the full Hessian with the approximation. …
jalane
- 23
- 1
- 4
2
votes
1 answer
Scope of with torch.no_grad() in pytorch
with torch.no_grad():
input = Variable(input).cuda()
target = Variable(target).cuda(non_blocking=True)
y=model(input)
# many things here
Is the no_grad continue to having effect out of the "with" scope?
Luk Aron
- 1,235
- 11
- 34
2
votes
1 answer
Can my PyTorch forward function do additional operations?
Typically a forward function strings together a bunch of layers and returns the output of the last one. Can I do some additional processing after that last layer before returning? For example, some scalar multiplication and reshaping via .view?
I…
Shamoon
- 41,293
- 91
- 306
- 570
2
votes
2 answers
Understanding pytorch autograd
I am trying to understand how pytorch autograd works. If I have functions y = 2x and z = y**2, if I do normal differentiation, I get dz/dx at x = 1 as 8 (dz/dx = dz/dy * dy/dx = 2y*2 = 2(2x)*2 = 8x). Or, z = (2x)**2 = 4x^2 and dz/dx = 8x, so at x =…
max_max_mir
- 1,494
- 3
- 20
- 36
2
votes
1 answer
.grad() returns None in pytorch
I am trying to write a simple script for parameter estimation (where parameters are weights here). I am facing problem when .grad() returns None. I have gone through this and this link also and understood the concept both theoretically and…
Lot_to_learn
- 590
- 2
- 9
- 21
2
votes
1 answer
tf.function property in pytorch
I'm a beginner in pytorch, and I have some functions that are needed to implement in network.
My question is: is there any way like tf.function, or should I use "class(nn.Module)" with variable?
For example, let X be a 10x2 matrix . In…
ncalik
- 21
- 1
2
votes
1 answer
PyTorch Image transformation with gridsampler, weird behaviour while optimizing grid
Im trying to get the per-pixel transformation to fit one image (+ background) to a result.
Background Image + Input image should be transformed into desired result
to achieve this i am using the PyTorch gridsampler and autograd to optimize the…
Kevin Fischer
- 23
- 3
2
votes
1 answer
PyTorch Total CUDA time
Autograd profiler is a handy tool to measure the execution time in PyTorch as it is shown in what follows:
import torch
import torchvision.models as models
model = models.densenet121(pretrained=True)
x = torch.randn((1, 3, 224, 224),…
MTMD
- 1,162
- 2
- 11
- 23
2
votes
1 answer
accessing autograd arraybox values
I am trying to use the python package autograd and I want to use values of an autograd arraybox in an interpolation
np.interp(x, x_values, y_values)
where y_values are store in an autograd arraybox. This leads to an error
TypeError: Cannot cast…
carl
- 4,216
- 9
- 55
- 103
2
votes
0 answers
torch.optim.SGD in PyTorch results in NaN
I followed this tutorial and tried to modify it a little bit to see if I understand things correctly. However, when I tried to use torch.opim.SGD
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
device =…
casual-coder
- 33
- 2
- 5
2
votes
1 answer
How to use autograd find MIN/MAX point
Say we have a simple function y=sin(x**2), how do I use autograd find all X:s with the first order derivative value of 0?
Liu Yong
- 515
- 5
- 16
2
votes
1 answer
Using autograd.grad() as a parameter for a loss function (pytorch)
I want to compute the gradient between two tensors in a net. The input X tensor is sent through a set of convolutional layers which give me back and output Y tensor.
I’m creating a new loss and I would like to know the MSE between gradient of…
Xbel
- 735
- 1
- 10
- 30
2
votes
0 answers
Autograd fails to properly calculate Normal CDF derivative
I'm a total neophyte when it comes to Autograd, so I feel like I may be doing something very obviously wrong. However, I've spent a full day struggling on this.
I am trying to calculate the derivative of a function that includes a normal cdf. I had…
TotalFailure
- 23
- 3
2
votes
0 answers
Pytorch second derivative stuck between two errors: Buffers have been freed and variable is volatile
I have a loss function and a list of weightmatrices and I'm trying to compute second derivatives. Here's a code snippet:
loss.backward(retain_graph=True)
grad_params_w=torch.autograd.grad(loss, weight_list,create_graph=True)
for i in…
Jonasson
- 293
- 1
- 2
- 11
2
votes
0 answers
CUDNN Error in backprop for big batches
I implemented a combination of MLP, RNN, CNN. With a batch size of 420, everything seems to work fine (aka I dont get any errors). However as soon as I increase the batch to 840, I receive the following error:
Traceback (most recent call last):
…
timbmg
- 3,192
- 7
- 34
- 52