Error in running backwards() function in PyTorch

Question

The code:

 import numpy as np
 predictors = np.array([[73,67,43],[91,88,64],[87,134,58],[102,43,37],[69,96,70]],dtype='float32')
 outputs = np.array([[56,70],[81,101],[119,133],[22,37],[103,119]],dtype='float32')
 

 inputs = torch.from_numpy(predictors)
 targets = torch.from_numpy(outputs)

 weights = torch.randn(2,3,requires_grad=True)
 biases = torch.randn(2,requires_grad=True)

 def loss_mse(x,y):
  d = x-y
  return torch.sum(d*d)/d.numel()

 def model(w,b,x):
  return x @ w.t() +b 
 
 def train(x,y,w,b,lr,e):
  w = torch.tensor(w,requires_grad=True)
  b = torch.tensor(b,requires_grad=True)
  for epoch in range(e):
    preds = model(w,b,x)
    loss = loss_mse(y,preds)
    if epoch%5 == 0:
      print("Loss at Epoch [{}/{}] is {}".format(epoch,e,loss))
    #loss.requires_grad=True
    loss.backward()
    with torch.no_grad():
      w = w - lr*w.grad
      b = b - lr*b.grad
      w.grad.zero_()
      b.grad.zero_()

 train(inputs,targets,weights,biases,1e-5,100)

Running this gives different errors. Once it gave the error that loss was of size 0. Then it gave the error in the update line w = w-lr*w.grad that float can't be subtracted from NoneType.

score 0 · Accepted Answer · answered Mar 05 '21 at 19:04

First, why do you wrap your weights and biases as Tensor twice?

weights = torch.randn(2,3,requires_grad=True)
biases = torch.randn(2,requires_grad=True)de here

then inside the train function you use:

w = torch.tensor(w,requires_grad=True)
b = torch.tensor(b,requires_grad=True)

Second, in the part of updating your weights change it to:

  with torch.no_grad():
   w_new = w - lr*w.grad
   b_new = b - lr*b.grad
   w.copy_(w_new)
   b.copy_(b_new)
   w.grad.zero_()
   b.grad.zero_()

you can check this discussion for a more comprehensive explanation: https://discuss.pytorch.org/t/updatation-of-parameters-without-using-optimizer-step/34244/20

Thank You! Actually while I was debugging, I wrapped the weights and biases again just to check if they were the problem and forgot to remove them before posting here. Anyways, it worked. Thank you very much. — Nirmalya Misra, Mar 06 '21 at 00:46

Error in running backwards() function in PyTorch

1 Answers1