Autograd can automatically differentiate native Python and Numpy code and is also used by the deep learning framework PyTorch. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can also take derivatives of derivatives of derivatives. The main intended application of Autograd is gradient-based optimization.
Questions tagged [autograd]
362 questions
3
votes
2 answers
Recursive symbolic calculations - improve the performance
In my research I'm trying to tackle the Kolmogorov backward equation, i.e. interested in
$$Af = b(x)f'(x)+\sigma(x)f''(x)$$
With the specific b(x) and \sigma(x), I'm trying to see how fast the coefficients of the expression are growing when…
Gabrielė Mongirdaitė
- 35
- 4
3
votes
1 answer
Conditional update in JAX?
In autograd/numpy I could do:
q[q<0] = 0.0
How can I do the same thing in JAX?
I tried import numpy as onp and using that to create arrays, but that doesn't seem to work.
Andriy Drozdyuk
- 58,435
- 50
- 171
- 272
3
votes
1 answer
PyTorch gradients have different shape for CUDA and CPU
I’m dealing with a strange issue where the gradients after backward pass have different shapes depending on whether CUDA or CPU is used. The model used is relatively simple:
class Net(nn.Module):
def __init__(self):
super(Net,…
Semih Cantürk
- 93
- 1
- 7
3
votes
1 answer
Why do we need clone the grad_output and assign it to grad_input when defining a ReLU autograd function?
I'm walking through the autograd part of pytorch tutorials. I have two questions:
Why do we need clone the grad_output and assign it to grad_input other than simple assignment during backpropagation?
What's the purpose of grad_input[input < 0] = 0?…
Hoodythree
- 177
- 1
- 1
- 8
3
votes
1 answer
pytorch autograd obstructs script from terminating
Whenever I call autograds backward my script never terminates. backwardis not per se blocking, all lines after it are still executed, the script just does not terminate. It appears that there is some sort of working thread in the background which…
JonathanK
- 827
- 1
- 6
- 23
3
votes
1 answer
Assigning a parameter to the GPU sets is_leaf as false
If I create a Parameter in PyTorch, then it is automatically assigned as a leaf variable:
x = torch.nn.Parameter(torch.Tensor([0.1]))
print(x.is_leaf)
This prints out True. From what I understand, if x is a leaf variable, then it will be updated by…
Karnivaurus
- 22,823
- 57
- 147
- 247
3
votes
1 answer
Pytorch second derivative returns None
I am unable to take a second derivative of the following function. When I want the second derivative with respect to u_s it works but with x_s it doesn't work.
Does anyone know what I have done wrong here?
def cost(xt, x_goal, u, Q, R):
…
Alejandro
- 879
- 11
- 27
3
votes
0 answers
Explanation of CPU and GPU time with torch autograd profiler in CUDA mode
I am trying to profile a network with torch.autograd.profiler and I need some explanation regarding the CPU and GPU time reported. I assume that the timings are nearly equal because CPU time includes the time the kernel launch + execution. However I…
Shama
- 31
- 3
3
votes
1 answer
Pytorch: backpropagating from sum of matrix elements to leaf variable
I'm trying to understand backpropagation in pytorch a bit better. I have a code snippet that successfully does backpropagation from the output d to the leaf variable a, but then if I add in a reshape step, the backpropagation no longer gives the…
user49404
- 732
- 6
- 22
3
votes
0 answers
How to attach a tensor to a particular point in the computation graph in PyTorch?
As stated in the question, I need to attach a tensor to a particular point in the computation graph in Pytorch.
What I'm trying to do is this:
while geting outputs from all mini-batches, accumulate them in a list and when one epoch finishes,…
Aka
- 137
- 1
- 12
3
votes
1 answer
Does pytorch do eager pruning of its computational graph?
This is a very simple example:
import torch
x = torch.tensor([1., 2., 3., 4., 5.], requires_grad=True)
y = torch.tensor([2., 2., 2., 2., 2.], requires_grad=True)
z = torch.tensor([1., 1., 0., 0., 0.], requires_grad=True)
s = torch.sum(x * y *…
Julius
- 735
- 2
- 6
- 17
3
votes
2 answers
Update specific vector elements in PyTorch
I have a large vector that I would like to update. I'll update it by adding an offset to specific elements in the vector. I specify a vector of indices that I want to update (call the index vector ix), and for each index, I specify a value that I…
zillo
- 31
- 3
3
votes
2 answers
How to Use AutoGrad Packages?
I am trying to do a simple thing: use autograd to get gradients and do gradient descent:
import tangent
def model(x):
return a*x + b
def loss(x,y):
return (y-model(x))**2.0
After getting loss for an input-output pair, I want to get…
Andy Markman
- 127
- 1
- 12
3
votes
3 answers
How to find and understand the autograd source code in PyTorch
I have a good understanding of the autograd algorithm, and I think I should learn about the source code in PyTorch. However, when I see the project on GitHub, I am confused by the structure, cuz so many files include autograd. So which part is the…
StackOutOfMemo
- 33
- 5
3
votes
1 answer
Partial Derivative using Autograd
I have a function that takes in a multivariate argument x. Here x = [x1,x2,x3]. Let's say my function looks like:
f(x,T) = np.dot(x,T) + np.exp(np.dot(x,T) where T is a constant.
I am interested in finding df/dx1, df/dx2 and df/dx3 functions.
I have…
user3259937
- 567
- 7
- 16