my nn.sigmoid() gradient different with my manual calculation

Question

so im doing a manual calculation of lstm backpropagation in excel and want to compare it to my code, but im having trouble with the gradient of sigmoid at the pytorch. :

the output here:

tensor([[0.8762]], grad_fn=<SigmoidBackward>)
tensor([-0.1238])
epoch:   0 loss: 0.13214068

so the first line is the sigmoid value and the second line is the gradient of sigmoid value. why the value of sigmoid gradient is -0.1238 while the formula of sigmoid gradient are σ(x)⋅(1−σ(x). if i calculate the sigmoid gradient manually the value is 0.10845, but in the code the sigmoid gradient is -0.1238 .is the formula for the sigmoid gradient in pytorch wrong?

Welcome to Stack Overflow! Can you cut your question and all this code down to the absolute minimal amount you need to ask your question? (As for your final question, no, it's definitely not a bug in PyTorch.) — Matt Hall, Jun 30 '22 at 00:47
@kwinkunks done, so if i calculate the sigmoid gradient manually from excel the value is 0.1084545 but in the code the gradient of sigmoid is -0.1238 — M. AASYA ALDIN ISLAMY, Jun 30 '22 at 01:00
What does `` represent? That's not Python code. Is this a function you wrote? — Matt Hall, Jun 30 '22 at 03:28

score 2 · Answer 1 · answered Jun 30 '22 at 03:29

I can't reproduce your error. For one thing, your value of 0.10845 is not correct: remember that it might be computed that way (i.e. z * (1 - z)) because you expect z to be logistic(z) in your implementation. But, in any case, the value I compute agrees with what PyTorch produces.

Here's the logistic function:

import numpy as np

def logistic(z, derivative=False):
    if not derivative:
        return 1 / (1 + np.exp(-z))
    else:
        return logistic(z) * (1 - logistic(z))
    
logistic(0.8762, derivative=True)

This produces 0.20754992931590668.

Now with PyTorch:

import torch

t = torch.Tensor([0.8762])
t.requires_grad = True
torch.sigmoid(t).backward()
t.grad

This produces tensor([0.2075]).

thank you, my bad i think i misscalculated it – M. AASYA ALDIN ISLAMY Jun 30 '22 at 07:15 — M. AASYA ALDIN ISLAMY, Jun 30 '22 at 07:15

my nn.sigmoid() gradient different with my manual calculation

1 Answers1