0

So I have the following dictionary implemented for vanilla SGD:

        update_weights = dict(zip(weight_keys,
                                [grad_weight[key] -
                                 lr * convert_to_tensor(dx[key])]) for key in weight_keys)

I am trying to implement something similar with momentum, however, I am not sure how I would be able to accumulate the velocity term so that I can update it all at once in list comprehension terms:

v_(i) = mu * v_(i-1) - lr * convert_to_tensor(dx[key])
grad_weight[key] += v_i

does anyone have any idea how I might do this using list comprehension (using Tensorflow preferably)?

for key in weight_keys:
     v = mu * v - lr * convert_to_tensor(dx[key])
     update_weights[key] = grad_weight[key] + v
skidjoe
  • 493
  • 7
  • 16
  • Why do you need this to be a list comprehension? – CryptoFool Mar 13 '21 at 15:20
  • Wouldn't it be much faster? I'm asking to see if there's a way. – skidjoe Mar 13 '21 at 15:36
  • Why are you trying to implement this by hand? Just use [tf.keras.optimizers.SGD](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD); it has a momentum parameter. – o-90 Mar 13 '21 at 15:40
  • It's for a specific program where I keras optimizers do not achieve what I need. – skidjoe Mar 13 '21 at 15:42
  • It might be slightly faster, but not enough so that you should care about it. I don't understand why programmers seem so obsessed with list comprehensions and putting everything on a single line. If what you have works, then there's nothing wrong with it. If it's not obvious how to get it into a comprehension, maybe the result will be hard to read even if you find a way. Readability is important. What you have now is very readable, and there's no reason not to use it. But if you find a way to use a comprehension and have it be readable, then great! – CryptoFool Mar 13 '21 at 15:59
  • ...but wait. You aren't creating a list at all in your example. If you aren't creating a new list, then you don't want to use a list comprehension. It appears that what you'd want is a dictionary comprehension. – CryptoFool Mar 13 '21 at 16:05

0 Answers0