Autograd can automatically differentiate native Python and Numpy code and is also used by the deep learning framework PyTorch. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can also take derivatives of derivatives of derivatives. The main intended application of Autograd is gradient-based optimization.
Questions tagged [autograd]
362 questions
4
votes
3 answers
Pytorch: Gradient of output w.r.t parameters
I'm interested in Finding the Gradient of Neural Network output with respect to the parameters (weights and biases).
More specifically, assume I have the following Neural Network Structure [6,4,3,1]. The input samples size is 20. What I'm interested…
Sibghat Khan
- 147
- 1
- 9
4
votes
1 answer
PyTorch Getting Custom Loss Function Running
I'm trying to use a custom loss function by extending nn.Module, but I can't get past the error
element 0 of variables does not require grad and does not have a grad_fn
Note: my labels are lists of size: num_samples, but each batch will have the…
Matthew Ciaramitaro
- 1,184
- 1
- 13
- 27
4
votes
1 answer
pytorch: how to directly find gradient w.r.t. loss
In theano, it was very easy to get the gradient of some variable w.r.t. a given loss:
loss = f(x, w)
dl_dw = tt.grad(loss, wrt=w)
I get that pytorch goes by a different paradigm, where you'd do something like:
loss = f(x, w)
loss.backwards()
dl_dw…
Peter
- 12,274
- 9
- 71
- 86
3
votes
2 answers
Purpose of stop gradient in `jax.nn.softmax`?
jax.nn.softmax is defined as:
def softmax(x: Array,
axis: Optional[Union[int, Tuple[int, ...]]] = -1,
where: Optional[Array] = None,
initial: Optional[Array] = None) -> Array:
x_max = jnp.max(x, axis,…
Jay Mody
- 3,727
- 1
- 11
- 27
3
votes
1 answer
Error: "One of the differentiated Tensors appears to not have been used in the graph"
I am trying to compute a gradient of y_hat to x (y_hat is the sum of gradients of model output to x) but it gives me the error: One of the differentiated Tensors appears to not have been used in the graph. This is the code:
class Model(nn.Module):
…
rene smith
- 83
- 1
- 9
3
votes
1 answer
I can't use my custom loss which inherited from torch.autograd.Function. RuntimeError: A view was created in no_grad mode
I'm using torch.autograd.Function to create custom loss function and I got this error when running the training part.
RuntimeError: A view was created in no_grad mode and its base or another view of its base has been modified inplace with grad…
Palm
- 31
- 3
3
votes
1 answer
Understanding gradient computation using backward() in PyTorch
I'm trying to understand the basic pytorch autograd system:
x = torch.tensor(10., requires_grad=True)
print('tensor:',x)
x.backward()
print('gradient:',x.grad)
output:
tensor: tensor(10., requires_grad=True)
gradient: tensor(1.)
since x is a…
volperossa
- 1,339
- 20
- 33
3
votes
1 answer
How does Pytorch no_grad function for a = a - b and a -= b type of operation?
import torch
def model(x, W, b):
return x@W + b
def mse(t1, t2):
diff = t1 - t2
return torch.sum(diff * diff) / diff.numel()
inputs = torch.rand(2, 3, requires_grad=True)
targets = torch.rand( 2,2, requires_grad=True)
W =…
tarmas99
- 359
- 1
- 11
3
votes
1 answer
Automatic Differentiation with respect to rank-based computations
I'm new to automatic differentiation programming, so this maybe a naive question. Below is a simplified version of what I'm trying to solve.
I have two input arrays - a vector A of size N and a matrix B of shape (N, M), as well a parameter vector…
P JMU
- 69
- 3
3
votes
1 answer
Getting pytorch backward's RuntimeError: Trying to backward through the graph a second time... when slicing a tensor
Upon running the code snippet (PyTorch 1.7.1; Python 3.8),
import numpy as np
import torch
def batch_matrix(vector_pairs, factor=2):
baselen = len(vector_pairs[0]) // factor
split_batch = []
for j in range(factor):
for i in…
TweetysOldFriend
- 39
- 1
- 3
3
votes
1 answer
Getting the autograd counter of a tensor in PyTorch
I am using PyTorch for training a network. I was going through the autograd documentation and here it is mentioned that for each tensor there is a counter that the autograd implements to track the "version" of any tensor. How can I get this counter…
learner
- 3,168
- 3
- 18
- 35
3
votes
1 answer
Derivative of BatchNorm2d in PyTorch
In my network, I want to calculate the forward pass and backward pass of my network both in the forward pass.
For this, I have to manually define all the backward pass methods of the forward pass layers.
For the activation functions, that's easy.…
sebko_iic
- 340
- 2
- 16
3
votes
1 answer
Computing the loss of a function of predictions with pytorch
I have a convolutional neural network that predicts 3 quantities: Ux, Uy, and P. These are the x velocity, y-velocity, and pressure field. They are all 2D arrays of size [100,60], and my batch size is 10.
I want to compute the loss and update the…
user3611
- 151
- 2
- 9
3
votes
0 answers
pytorch index_put_ gives RuntimeError: the derivative for 'indices' is not implemented
This is follow up question to this question. I tried using index_put_ as suggested in the answer, however I'm getting the following error
RuntimeError: the derivative for 'indices' is not implemented
I want to backpropagate the error gradients…
Nagabhushan S N
- 6,407
- 8
- 44
- 87
3
votes
2 answers
Pytorch's Autograd does not support complex matrix inversion, does anyone have a workaround?
Somewhere in my loss function, I invert a complex matrix of size 64*64. Although complex matrix inversion is supported for torch.tensor, the gradient cannot be computed in the training loop as I get this error:
RuntimeError: inverse does not support…
DeepFara
- 31
- 3