0

During the pruning step, I zero out some filters of a Depthwise convolution. After doing this, I need to retrain the network but those weights that have been zeroed out (I have the list of indexes) should not be updated during the training, their values need to remain equal to zero. So if I have 150 filters in the Depthwise layer (I'm not counting the bias), is there any way to freeze just a subset of them?

For example, the weights of the filters are x:

x=model.layers[4].get_weights()[0]

And x is a ndarray of 150 numbers. Ideally, if I have the list of zeroed out indices pruned_filters, I would like to do something like:

x[pruned_filters].trainable = False # I know this is wrong, it's just an example

Or move them to the non_trainable_weights

Giorgio Luigi Morales Luna
  • 53
  • 7

1 Answers1

1

you can't freeze only specific filters. what you can do, if you retain it valuable, is to set them to 0. but all the others remain not trainable. here an example:

inp = Input((10,10,3))
c = Conv2D(32, kernel_size=(3, 3),
           activation='relu')
f = Flatten()
d = Dense(10, activation='softmax')

x = c(inp)
x = f(x)
out = d(x)
model = Model(inp, out)
print(model.summary())

# model.fit(.....)

pruned_filters = [1,5,9]
w,b = c.get_weights()
w[:,:,:,pruned_filters] = 0
c.set_weights([w,b])

model.layers[1].trainable = False

# model.fit(.....)

otherwise, you can apply a mask... the mask not consider the value with a specific value to compute backpropagation... in your case, this maintains the zero filters unmodified

inp = Input((10,10,3))
c = Conv2D(32, kernel_size=(3, 3),
           activation='relu')
f = Flatten()
d = Dense(10)

x = c(inp)
x = f(x)
out = d(x)
model1 = Model(inp, out)
model1.compile('adam', 'mse')
model1.fit(np.random.uniform(0,1, (5,10,10,3)), np.random.uniform(0,1, (5,10)))

pruned_filters = [1,5,9]
w,b = c.get_weights()
w[:,:,:,pruned_filters] = 0
c.set_weights([w,b])
print(w)

mask = Masking(mask_value=0)
x = c(inp)
x = mask(x)
x = f(x)
out = d(x)
model2 = Model(inp, out)
model2.compile('adam', 'mse')
model2.fit(np.random.uniform(0,1, (5,10,10,3)), np.random.uniform(0,1, (5,10)))

w,b = c.get_weights()
print(w)
Marco Cerliani
  • 21,233
  • 3
  • 49
  • 54
  • Hello. That's actually part of the pruning step that I did before. I set the weights that I want to zero (not the whole layer) and after doing that I need to retrain the network but I cannot freeze the whole layer as you did (`model.layers[1].trainable=False`) because the remaining weights of the layer need to be updated and the pruned weights should remain 0. – Giorgio Luigi Morales Luna May 15 '20 at 07:16
  • The Masking operation sounds really good! I will try that. Thanks. – Giorgio Luigi Morales Luna May 15 '20 at 07:32
  • Unfortunately, although the Masking function does sound good, I couldn't find a way to apply it to a DepthwiseConv2D, it simply does not support it. – Giorgio Luigi Morales Luna May 16 '20 at 00:41