1

Lets see the result that I got first. This is one of a convolution layer of my model, and im only showing 11 filter's weight of it (11 3x3 filter with channel=1)

Left side is original weight Right side is Pruned weight

So I was wondering how does the "TORCH.NN.UTILS.PRUNE.L1_UNSTRUCTURED" works because by the pytorch website said, it prune the lowest L1-norm unit, but as far as I know, L1-norm pruning is a filter pruning method which prune the whole filter which use this equation to fine the lowest filter value instead of pruning single weight. So I'm a bit curious about how does this function actually works?

The following is my pruning code

parameters_to_prune = (
    (model.input_layer[0], 'weight'),
    (model.hidden_layer1[0], 'weight'),
    (model.hidden_layer2[0], 'weight'),
    (model.output_layer[0], 'weight')
)

prune.global_unstructured(
    parameters_to_prune,
    pruning_method=prune.L1Unstructured,
    amount = (pruned_percentage/100),
)
SnowNiNo
  • 13
  • 3

1 Answers1

1

The nn.utils.prune.l1_unstructured utility does not prune the whole filter, it prunes individual parameter components as you observed in your sheet. That is components with the lower norm get masked.

Here is a minimal example as discussed in the comments below:

>>> m = nn.Linear(10,1,bias=False)
>>> m.weight = nn.Parameter(torch.arange(10).float())
>>> prune.l1_unstructured(m, 'weight', .3)
>>> m.weight
tensor([0., 0., 0., 3., 4., 5., 6., 7., 8., 9.], grad_fn=<MulBackward0>)
Ivan
  • 34,531
  • 8
  • 55
  • 100
  • yea I know that it does not prune the whole filter, what I mean is how does this function decided which components to prune by using L1-norm? In other words, how L1-norm decide "components with the lower norm"? – SnowNiNo Dec 14 '21 at 13:18
  • @SnowNiNo, it will mask the lowest `p`-% components, where `p` is the `amount` provided to the prune function. – Ivan Dec 14 '21 at 14:58
  • So lets assume I have weight with 1~10([1,2,3,4,5,6,7,8,9,10]), and my amount is 0.3, so I'll get [0,0,0,4,5,6,7,8,9,10] as return – SnowNiNo Dec 15 '21 at 06:42
  • @SnowNiNo Exactly, see my edit above with this example. – Ivan Dec 15 '21 at 10:56
  • Cool thanks for answering – SnowNiNo Dec 17 '21 at 03:36