-1

I need to freeze some of layer's parameters during training. I tried to set needs_gradient attribute by model.L1.b.needs_gradient = False but I get the following exception:

AttributeError Traceback (most recent call last)
<ipython-input-57-93ef31fae7d8> in <module>()
----> 1 model.L1.b.needs_gradient = False

/home/aj/anaconda3/envs/cntk-py27/lib/python2.7/site-packages/cntk/cntk_py.pyc in <lambda>(self, name, value)
   1263     for _s in [Variable]:
   1264         __swig_setmethods__.update(getattr(_s, '__swig_setmethods__', {}))
-> 1265     __setattr__ = lambda self, name, value: _swig_setattr(self, Parameter, name, value)
   1266     __swig_getmethods__ = {}
   1267     for _s in [Variable]:

/home/aj/anaconda3/envs/cntk-py27/lib/python2.7/site-packages/cntk/cntk_py.pyc in _swig_setattr(self, class_type, name, value)
     72 
     73 def _swig_setattr(self, class_type, name, value):
---> 74     return _swig_setattr_nondynamic(self, class_type, name, value, 0)
     75 
     76 

/home/aj/anaconda3/envs/cntk-py27/lib/python2.7/site-packages/cntk/cntk_py.pyc in _swig_setattr_nondynamic(self, class_type, name, value, static)
     64     if (not static):
     65         if _newclass:
---> 66             object.__setattr__(self, name, value)
     67         else:
     68             self.__dict__[name] = value

AttributeError: can't set attribute

Please help me to eliminate the exception or another way to freeze parameters. thanks

François Maturel
  • 5,884
  • 6
  • 45
  • 50
Mohammad Hassan Sohan Ajini
  • 150
  • 1
  • 12

2 Answers2

3

You can declare a CNTK variable as 'frozen' using needs_gradient property when you are declaring it. It will be same as declaring a constant. However, if you want to have a network trained for a while and then freeze the parameters and use it in other training, you can achieve that using this approach:

import cntk
trained_model = get_my_previously_trained_model()
frozen_model = trained_model.clone(cntk.CloneMethod.freeze)
output_from_trained_model = frozen_model(input_features)
model = cntk.layers.Dense(output_dim, activation=None)(output_from_trained_model)

Now, only the parameters in your Dense layer is train-able. Rest of the parameters in the whole network are frozen. Hope this example helps.

Zp Bappi
  • 121
  • 5
3

Another way to avoid updating parameters you don't want, is to create a learner and pass on the parameters need to be learnt to it. You can check out the GAN example that uses this to switch training between G and D network.

KeD
  • 221
  • 1
  • 3