Stack Overflow
Stack Overflow

Questions tagged [autodiff]

Automatic Differentiation (AD) is a set of techniques based on the mechanical application of the chain rule to obtain derivatives of a function given as a computer program.

Automatic Differentiation (AD) is a set of techniques based on the mechanical application of the chain rule to obtain derivatives of a function given as a computer program.
AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations such as additions or elementary functions such as exp().
By applying the chain rule of derivative calculus repeatedly to these operations, derivatives of arbitrary order can be computed automatically, and accurate to working precision.

Conceptually, AD is different from symbolic differentiation and approximations by divided differences.

AD is used in the following areas:

  • Numerical Methods
  • Sensitivity Analysis
  • Design Optimization
  • Data Assimilation & Inverse Problems

Home page: http://www.autodiff.org/

94 questions
3
votes
2 answers

Defining custom gradient as a class method in Tensorflow

I need to define a method to be a custom gradient as follows: class CustGradClass: def __init__(self): pass @tf.custom_gradient def f(self,x): fx = x def grad(dy): return dy * 1 return fx, grad I am…
Mr. Nobody
  • 185
  • 11
3
votes
1 answer

Tensorflow: use different expression for forward and backward pass

I have a tensorflow expression where I want to use a different expression depending on whether I'm computing the forward or backward (gradient) pass. Specifically, I want to ignore the effects of some randomness (noise) added into the network during…
hunse
  • 3,175
  • 20
  • 25
2
votes
0 answers

Computing Gradients of Weighted State Averages in PyTorch Models

I have two trained models with the same architecture and different performances. I want to construct a new model by taking the weighted average of their states (s1, s2, s1+s2=1) and calculate the gradients of the new model's loss with respect to s1…
Amin Rezaei
  • 376
  • 2
  • 11
2
votes
1 answer

How to find the analytical gradient using tensorflow gradienttape

Suppose we have some function y=x^2 We can then use gradient tape to automatically calculate the gradient for us (when we provide some values of x to tensorflow x = tf.Variable(3.0) with tf.GradientTape() as tape: y = x**2 dy_dx =…
Ting Wai Au
  • 31
  • 3
2
votes
3 answers

Using Deriv package for derivative wrt vector

I am exploring autodiff, and I would like to use Deriv for computing a derivative of a function wrt to a vector. I write library(numDeriv) library(Deriv) h = function(x) c(1,2)%*%x grad(h,c(1,2)) #ok #[1] 1 2 dh=Deriv(h,x='x') #Error in c(1, 2) %*%…
Sam Weisenthal
  • 2,791
  • 9
  • 28
  • 66
2
votes
1 answer

Use CppADCodeGen with CMake FetchContent or ExternalProject

I am not good with CMake, and I cannot find good explanations about how to use its FetchContent functionality. Indeed, most repositories seem to require different treatment, and the rules of such treatment defy my comprehension. That said, here is…
fdev
  • 127
  • 12
2
votes
1 answer

How is the pullback defined for a function with an optional argument?

Here is a minimal example: import _Differentiation @differentiable(reverse) func g(x: Double?) -> Double { if x == nil { return 0.0 // I don't care what this value is } else { return x! * x! } } @derivative(of:…
bjschoenfeld
  • 402
  • 6
  • 14
2
votes
1 answer

Custom Gradients in Tensor Flow - Unable to understand this example

I keep thinking that I am about to understand custom gradients but then I test it out this example and I can just not figure out what is going on. I am hoping somebody can walk me through what exactly is happening below. I think this essentially is…
briry12
  • 21
  • 3
2
votes
0 answers

Is it possible to use GLM with non-fundamental types?

I have some functions and data structures using glm for vector and matrix math. Now I want to compute gradient's with respect to the input variables. Since the computations are fairly complex, I'd like to use automatic differentiation to verify my…
Adam
  • 488
  • 4
  • 17
2
votes
1 answer

tensorflow, custom loss, custom_gradient, temp tf.Variable leads to error

Consider implementing a custom loss function that requires a temporary variable instantiation. If we need to implement custom gradients, TF expects there to be an additional output of the gradient functional, when there should only be as many…
drezap
  • 83
  • 1
  • 1
  • 8
2
votes
1 answer

Using GradientTape with a SavedModel to do a gradient descent on the input

I am trying to make an adversarial attack for a model I loaded using the SavedModel API. I want to do a gradient descent of my input with respect to the loss of the model given a target. The code is a little bit long but it's the bare minimum for…
Kipr
  • 806
  • 10
  • 13
2
votes
0 answers

Pytorch: Custom Loss involving Norm of End-to-End Jacobian

Cross posting from Pytorch discussion boards I want to train a network using a modified loss function that has both a typical classification loss (e.g. nn.CrossEntropyLoss) as well as a penalty on the Frobenius norm of the end-to-end Jacobian (i.e.…
Rylan Schaeffer
  • 1,945
  • 2
  • 28
  • 50
2
votes
1 answer

Microsoft CNTK Automatic Differentiation

According to Microsoft, CNTK includes automatic differentiation. For better understanding the source (which I've successfully built) I'd like to know which C++ classes implement AD and how it is implemented in CNTK?
Rod
  • 21
  • 3
2
votes
1 answer

What operations are supported for automatic differentiation in tensorflow

I am confused with what types of operations are supported for automatic differentiation in tf. Concretely, is tensor indexing operation as follows supported? ... # feat is output from some conv layer and the shape is B*H*W*C # case one loss =…
lhao0301
  • 1,941
  • 4
  • 20
  • 26
2
votes
1 answer

How to alternate train op's in tensorflow?

I am implementing an alternating training scheme. The graph contains two training ops. The training should alternate between these. This is relevant for research like this or this Below is a small example. But it seems to update both the ops at…
Rob Romijnders
  • 833
  • 1
  • 6
  • 16
1
2
3 4 5 6 7