1

For each dense layer, after setting some of its weights to 0.0, I want them to be ignored during both inference and training. They would not affect the model's loss, and I would later be able to re-train using only the remaining (non-ignored) weights.

I am aware of Keras' Masking layer, but I can't get it to only mask individual weights (those equal to a value, say 0.0) in a layer. Apparently, it requires that all columns in a row be equal to that value (0.0 as per my preference) to mask that row, whereas I want to mask weights of 0.0 without affecting non-0.0 weights.

markos
  • 71
  • 1
  • 6
  • What do you mean by "ignoring a weight"? Do you want to freeze it (i.e. make it untrainable)? Is this for a Dense layer? – rvinas Dec 12 '18 at 09:14
  • Yes, I want a specified (by me) weight to (a) not contribute to inference and (b) be untrainable. Essentially, I want to "remove" it from the network. And yes, as I said it's a Dense layer. – markos Dec 12 '18 at 13:01
  • Is "not contributing to inference" equivalent to setting its value to zero? – rvinas Dec 12 '18 at 13:09
  • How about using Dropout? – Kurtis Streutker Dec 12 '18 at 14:28
  • Do you know beforehand *which* weights you want to be ignored? Or just a random fraction? – sdcbr Dec 12 '18 at 17:43
  • @rvinas I think it is, but that would only satisfy (a). It would still be trainable, thus non-zero after training. – markos Dec 12 '18 at 22:45
  • @KurtisStreutker I believe Dropout is not the same, as it **randomly** ignores **nodes** during training, but I want to be very **specific** about ignoring **weights** while retaining their corresponding nodes. – markos Dec 12 '18 at 22:46
  • @sdcbr Yes, I know precisely which ones I want to be ignored. That's the problem - I can't find a way in Keras to ignore multiple individual weights in a layer during both inference and training. – markos Dec 12 '18 at 22:48
  • 1
    @ChrisM. I think [this](https://stackoverflow.com/questions/50290769/specify-connections-in-nn-in-keras?answertab=active#tab-top) is precisely what you are looking for. – rvinas Dec 13 '18 at 08:17
  • @rvinas Thank you, that was very helpful. I figured I need to "stop" the gradient from being applied to weights that have been zeroed out. Apparently it's not really possible to do that with Keras, as it automates the training process. I've been trying to do it with TensorFlow, please see my [new question](https://stackoverflow.com/questions/53862678/tensorflow-how-to-compute-gradient-while-ignoring-zero-weights) if you might have some time. I'd appreciate your input. – markos Dec 20 '18 at 23:02

0 Answers0