MXNet - multiplying scalar to an array leads to zero

Question

I am trying to implement tutorials of mxnet in this page, in computing gradient decent:

def SGD(params, lr):
for param in params:
    param[:] = param - lr * param.grad

I noticed that when lr<1, the scalar to array multiplication

lr * param.grad

becomes a zero matrix with same size of param.grad

I do not know why this happens. can any one help me to understand this?

Many thanks

If you have a debugger attached, check dtype of `param` and `lr`. Are they by any change integers? You can also print `param.dtype` and `lr.dtype`. — Sina Afrooze, Mar 16 '18 at 18:15
I found that this happens when I am trying to debug, however I do not know why that happens. — Ali, Mar 16 '18 at 19:26
type(lr) is float. type(param) is: mxnet.ndarray.ndarray.NDArray and param.dtype: Out[5]: numpy.float32 — Ali, Mar 16 '18 at 19:29
Well if `lr` is not zero, then `param.grad` must be zero. Other than the gradients being actually zero, another way for `param.grad` to be zero is if you're actually calculating the gradient by going through the forward pass under `autograd` scope and then calling `backward`. If you can't figure it out, paste a small code reproducing the issue and I'll be glad to help. — Sina Afrooze, Mar 16 '18 at 21:52
actually when I run the code it works, but when I debug it that happens. lr =.5 and param.grad is not zero- thanks @SinaAfrooze — Ali, Mar 17 '18 at 08:34

score 0 · Accepted Answer · answered Mar 18 '18 at 20:30

I ran the code from that notebook in PyCharm locally and attached a debugger to try to reproduce what you saw.

def SGD(params, lr):
    for param in params:
        t = lr * param.grad
        param[:] = param - t

When I put a breakpoint in the above code and inspected t, the debugger showed all zeroes. But when I ran the code everything ran fine. That was weird. So, I modified the code to print the sum of all gradients per pixel in every iteration.

def SGD(params, lr):
    for param in params:
        t = lr * param.grad
        print(nd.sum(param.grad, axis=-1))
        param[:] = param - t

I got this output. Output from one iteration shown below:

[  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -6.23335197e-08
   2.91620381e-08   2.32830644e-10   1.11467671e-08  -7.45058060e-09
   2.09547579e-09  -3.11993062e-08   3.44589353e-08  -7.45058060e-09
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   1.73723835e-09  -1.45343968e-08   1.51213424e-08  -4.47034836e-08
  -1.19209290e-07  -8.10250640e-08   8.66129994e-08   3.77185643e-08
   8.47503543e-08   8.66129994e-08   8.70786607e-08   5.35510480e-09
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   5.47231149e-09  -4.99075004e-10  -4.99075004e-10
   6.05359674e-09  -9.66247171e-09   1.83936208e-08  -7.50416707e-09
   1.12872200e-07   9.31513569e-08   2.45261617e-07  -1.86264515e-07
  -5.28991222e-07  -2.68220901e-07  -1.22934580e-07   0.00000000e+00
   2.08616257e-07   1.89989805e-07   4.33064997e-08   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00  -2.75546541e-09   2.39708999e-08  -2.04163371e-08
   6.70552254e-08   6.14672899e-08   4.40049917e-08   8.94069672e-08
   8.94069672e-08   2.98023224e-07   1.19209290e-07  -5.96046448e-08
  -2.38418579e-07  -4.76837158e-07  -4.76837158e-07  -2.38418579e-07
   1.78813934e-07   4.13507223e-07   2.83122063e-07   3.09199095e-07
   1.34285074e-07   6.17840215e-08   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00  -2.19943086e-09   1.58324838e-08   6.26314431e-08
   4.47034836e-08   5.96046448e-08   2.38418579e-07   4.76837158e-07
   2.38418579e-07   4.76837158e-07   5.66244125e-07   1.49011612e-07
  -4.17232513e-07  -2.38418579e-07  -3.57627869e-07   0.00000000e+00
   0.00000000e+00   1.22934580e-07   1.97440386e-07   2.73808837e-07
   4.55183908e-08   1.16733858e-08   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   7.33416528e-09   4.35829861e-08
   4.36230039e-08   3.72456270e-08   1.67638063e-08   5.96046448e-08
   1.49011612e-07   2.38418579e-07  -1.19209290e-07  -2.38418579e-07
  -1.78813934e-07   2.38418579e-07   6.25848770e-07   3.57627869e-07
   2.98023224e-08  -4.17232513e-07  -3.57627869e-07  -5.96046448e-08
  -1.04308128e-07   1.19209290e-07   1.08033419e-07  -6.14672899e-08
  -1.40164047e-07  -1.22236088e-07  -1.07334927e-07   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -2.17623892e-08   6.13508746e-08
   7.71033228e-08   4.65661287e-09   0.00000000e+00  -5.96046448e-08
   0.00000000e+00   2.38418579e-07   1.19209290e-07   3.57627869e-07
   4.76837158e-07   1.19209290e-07  -2.38418579e-07  -1.78813934e-07
  -4.76837158e-07   0.00000000e+00   1.78813934e-07   1.19209290e-07
   5.06639481e-07   1.78813934e-07  -1.19209290e-07  -1.19209290e-07
  -1.19209290e-07   0.00000000e+00  -1.34168658e-08   0.00000000e+00
   0.00000000e+00   0.00000000e+00   4.71118256e-10   2.32830644e-09
   2.53785402e-08  -2.83122063e-07   1.19209290e-07   1.19209290e-07
   2.98023224e-07   0.00000000e+00   2.38418579e-07   3.20374966e-07
   0.00000000e+00   3.57627869e-07  -2.38418579e-07   2.38418579e-07
  -9.53674316e-07  -7.15255737e-07   1.19209290e-07   3.57627869e-07
   4.17232513e-07  -7.45058060e-08  -5.96046448e-08  -1.19209290e-07
   1.49011612e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -2.59024091e-09
  -5.21540642e-08  -7.45058060e-09  -1.78813934e-07   5.96046448e-08
   1.78813934e-07   1.19209290e-07   4.76837158e-07   2.98023224e-07
  -5.96046448e-08   2.38418579e-07   3.57627869e-07   5.96046448e-08
  -4.76837158e-07   1.78813934e-07   4.17232513e-07  -1.19209290e-07
   2.98023224e-07   5.96046448e-08   2.04890966e-08  -1.30385160e-08
  -4.83123586e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   3.23634595e-08
  -5.58793545e-08  -1.34110451e-07  -1.49011612e-08  -5.96046448e-08
   1.49011612e-07   1.49011612e-08   1.78813934e-07   0.00000000e+00
   0.00000000e+00   8.94069672e-08   5.96046448e-08  -1.19209290e-07
  -3.57627869e-07   0.00000000e+00   4.76837158e-07  -2.38418579e-07
   0.00000000e+00   1.02445483e-07   1.46217644e-07  -1.32713467e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -2.79396772e-08  -1.56462193e-07  -1.93715096e-07   1.82539225e-07
   3.68803740e-07   4.32133675e-07   5.96046448e-08  -1.19209290e-07
   0.00000000e+00   2.98023224e-07  -1.19209290e-07  -5.06639481e-07
   1.19209290e-07   4.76837158e-07   8.94069672e-08   0.00000000e+00
   0.00000000e+00   5.96046448e-08   8.61473382e-08   4.39467840e-08
   4.85442797e-11   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -2.79396772e-08  -1.86264515e-07  -1.93715096e-07   2.98023224e-08
   3.57627869e-07   3.57627869e-07  -2.38418579e-07  -1.19209290e-07
  -2.38418579e-07  -1.78813934e-07  -1.78813934e-07   0.00000000e+00
  -4.02331352e-07  -2.23517418e-08   2.08616257e-07  -2.98023224e-08
   2.38418579e-07   2.08616257e-07   2.37952918e-07   7.72388375e-08
   3.80787242e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -7.45058060e-08  -2.98023224e-08   1.11758709e-07   1.78813934e-07
   4.76837158e-07   3.57627869e-07  -1.19209290e-07   0.00000000e+00
  -3.57627869e-07   2.38418579e-07  -1.19209290e-07   5.96046448e-08
   1.19209290e-07   1.49011612e-07   8.94069672e-08   1.78813934e-07
   4.17232513e-07   4.47034836e-07   4.47034836e-07   8.42846930e-08
  -3.70637281e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -1.04308128e-07   0.00000000e+00   3.35276127e-07   2.68220901e-07
   7.74860382e-07   7.15255737e-07   2.38418579e-07   1.19209290e-07
  -2.38418579e-07  -3.57627869e-07  -3.57627869e-07  -8.94069672e-08
   3.87430191e-07   3.57627869e-07   3.57627869e-07   5.96046448e-08
   2.98023224e-07   4.76837158e-07   5.36441803e-07   4.84287739e-08
   4.59044713e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -9.77270531e-09
  -1.19209290e-07   7.45058060e-09   0.00000000e+00   3.57627869e-07
   2.38418579e-07   5.96046448e-07   3.57627869e-07   9.53674316e-07
   4.76837158e-07   1.19209290e-07  -2.38418579e-07   1.19209290e-07
  -2.98023224e-07  -1.19209290e-07  -5.96046448e-08  -3.57627869e-07
   1.78813934e-07   4.47034836e-07   3.87430191e-07   1.08033419e-07
   3.81596692e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -4.18361026e-08
  -5.63450158e-08  -4.84287739e-08   5.96046448e-08   2.38418579e-07
   4.76837158e-07   3.57627869e-07   5.96046448e-07   1.78813934e-07
  -1.19209290e-07  -1.19209290e-07  -2.68220901e-07   0.00000000e+00
   0.00000000e+00   2.38418579e-07  -5.96046448e-07  -3.87430191e-07
   1.19209290e-07   3.57627869e-07   3.87430191e-07   1.24797225e-07
   6.91925379e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -4.36812968e-08
  -5.60366971e-08  -8.84756446e-08   1.19209290e-07  -1.19209290e-07
   3.57627869e-07   5.96046448e-08   0.00000000e+00   0.00000000e+00
  -5.96046448e-07   1.19209290e-07  -2.98023224e-08   1.78813934e-07
  -2.38418579e-07  -2.38418579e-07   3.57627869e-07  -3.57627869e-07
   2.38418579e-07   2.38418579e-07   3.42726707e-07   1.14087015e-07
   2.49773286e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -1.28651578e-09
  -1.01268569e-07  -1.50499545e-07  -4.47034836e-08   0.00000000e+00
   5.96046448e-08   3.57627869e-07   1.19209290e-07  -3.57627869e-07
  -3.57627869e-07   2.98023224e-07   3.12924385e-07   5.96046448e-08
   0.00000000e+00  -3.57627869e-07  -4.76837158e-07   5.96046448e-08
   2.98023224e-07   2.98023224e-07   1.78813934e-07   7.29633030e-08
   1.60424936e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -7.72104300e-08  -3.55039447e-08   2.83529516e-07   2.63098627e-08
   3.25031579e-07   7.45058060e-09   1.19209290e-07  -5.96046448e-08
  -3.27825546e-07   6.70552254e-08   3.12924385e-07   4.17232513e-07
  -3.57627869e-07  -4.76837158e-07  -1.19209290e-07   1.78813934e-07
   2.98023224e-07   1.71363354e-07   6.27478585e-08   1.92048901e-08
  -3.83870313e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -6.35419752e-08  -5.20981303e-08   2.64481059e-08   6.12053555e-08
   2.30735168e-07  -2.51457095e-08  -2.98023224e-07  -3.57627869e-07
   1.78813934e-07   6.55651093e-07   2.38418579e-07  -5.96046448e-08
  -3.57627869e-07   3.57627869e-07   1.19209290e-07   1.78813934e-07
   1.49011612e-07   8.19563866e-08   4.22005542e-08   2.06365929e-08
   2.06365929e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   2.09386428e-08   5.31318420e-08  -5.19480636e-09  -1.55114321e-07
   7.33416528e-08  -2.98023224e-08  -2.98023224e-08  -2.98023224e-07
   4.76837158e-07   2.38418579e-07   5.96046448e-08   3.57627869e-07
   1.19209290e-07   0.00000000e+00   2.38418579e-07   1.04308128e-07
   3.91155481e-08   4.49217623e-08   2.17316245e-08   2.06365929e-08
   2.23254459e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   1.64938871e-08   6.39862421e-08   5.16827185e-08   1.46355887e-07
  -1.34110451e-07  -5.96046448e-08   2.38418579e-07   7.74860382e-07
   9.53674316e-07   1.19209290e-07   0.00000000e+00   3.57627869e-07
   2.38418579e-07   0.00000000e+00   1.19209290e-07   1.44354999e-08
   2.26427801e-08   1.19798682e-09  -4.99121011e-09  -4.99121011e-09
  -2.63172950e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   3.20321369e-08   5.63813956e-08   8.19563866e-08
   0.00000000e+00  -1.19209290e-07   1.19209290e-07   1.19209290e-07
   1.19209290e-07  -1.19209290e-07   0.00000000e+00   0.00000000e+00
   5.96046448e-08  -7.45058060e-09   4.09781933e-08   1.63381628e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -7.45058060e-08  -5.96046448e-08
  -1.78813934e-07   5.96046448e-08  -2.88709998e-08  -3.35276127e-08
   6.28060661e-08   2.98023224e-08   1.19209290e-07   5.96046448e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -1.19209290e-07  -1.19209290e-07
   0.00000000e+00  -1.49011612e-08  -2.67755240e-08  -1.49157131e-09
   7.06131686e-09   6.01576176e-08  -2.69501470e-08   1.49011612e-08
   0.00000000e+00   0.00000000e+00  -7.45058060e-09   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]
<NDArray 784 @cpu(0)>

Note that the gradients for the weights from the first few tens of pixels are always zero. Since the debugger only shows the first few values on the ndarray, all you saw was only zeros.

Gradients for weights from the first few tens of pixels are zeroes very likely because those pixels do not contain any information useful to classify the digits in the image. Note that MNIST images are centered in the image. The pixels along the borders are just 0.

Hope that helps.

Thanks @indhu Bharathi. I tried some other scaler to ndarray mulyiplication, when scaler was less than one, results become zero (in debug mode only). You also may experience this. while the same multiplication in running is fine!!! — Ali, Mar 18 '18 at 21:48

MXNet - multiplying scalar to an array leads to zero

1 Answers1