6

I would like to modify the ImageNet caffe model as described bellow:

As the input channel number for temporal nets is different from that of spatial nets (20 vs. 3), we average the ImageNet model filters of first layer across the channel, and then copy the average results 20 times as the initialization of temporal nets.

My question is how can I achive the above results? How can I open the caffe model to be able to do those changes to it?

I read the net surgery tutorial but it doesn't cover the procedure needed.

Thank you for your assistance!

AMayer

AMayer
  • 415
  • 5
  • 19

1 Answers1

6

The Net Surgery tutorial should give you the basics you need to cover this. But let me explain the steps you need to do in more detail:

  1. Prepare the .prototxt network architectures: You need two files: the existing ImageNet .prototxt file, and your new temporal network architecture. You should make all layers except the first convolutional layers identical in both networks, including the names of the layers. That way, you can use the ImageNet .caffemodel file to initialize the weights automatically.

    As the first conv layer has a different size, you have to give it a different name in your .prototxt file than it has in the ImageNet file. Otherwise, Caffe will try to initialize this layer with the existing weights too, which will fail as they have different shapes. (This is what happens in the edit to your question.) Just name it e.g. conv1b and change all references to that layer accordingly.

  2. Load the ImageNet network for testing, so you can extract the parameters from the model file:

    net = caffe.Net('imagenet.prototxt', 'imagenet.caffemodel', caffe.TEST)

  3. Extract the weights from this loaded model.

    conv_1_weights = old_net.params['conv1'][0].data
conv_1_biases = old_net.params['conv1'][1].data

  4. Average the weights across the channels:

    conv_av_weights = np.mean(conv_1_weights, axis=1, keepdims=True)

  5. Load your new network together with the old .caffemodel file, as all layers except for the first layer directly use the weights from ImageNet:

    new_net = caffe.Net('new_network.prototxt', 'imagenet.caffemodel', caffe.TEST)

  6. Assign your calculated average weights to the new network

    new_net.params['conv1b'][0].data[...] = conv_av_weights
new_net.params['conv1b'][1].data[...] = conv_1_biases

  7. Save your weights to a new .caffemodel file:

    new_net.save('new_weights.caffemodel')
Community
  • 1
  • 1
hbaderts
  • 14,136
  • 4
  • 41
  • 48
  • Thank you for sharing your knowledge. I'm testing it right now. I will update you shortly about how it went. I appreciate the help. – AMayer Dec 08 '16 at 18:40
  • I've enounter an error. I have updated the main post with the code that I used based on your suggestion for better view of the problem. Later I will delete the code to not confuse the new readers. What I did wrong there? Thank you! – AMayer Dec 08 '16 at 19:26
  • I've managed to get passed that error, but enountered another one. I've updated the main post. Thank you! – AMayer Dec 08 '16 at 19:52
  • 1
    @AMayer I've edited my answer (step 1, and the code in step 6) to describe how to fix the error. Please let me know if that solves your problem. – hbaderts Dec 08 '16 at 20:49
  • It fixed the prev error, but now I get the following error: new_net.params['conv1b'][0].data[...] = conv_av_weights ValueError: could not broadcast input array from shape (96,11,11) into shape (96,20,11,11) – AMayer Dec 08 '16 at 21:01
  • Oops, seems like we lost an axis on the way ;) Please see the edited step 4. – hbaderts Dec 08 '16 at 21:25
  • It fixed it, but there's one more. Here is the error: new_net.params['conv1b'][1].data = conv_1_biases AttributeError: can't set attribute. Thank you! I truly appreciate your time and effort. – AMayer Dec 08 '16 at 21:36
  • Can you try adding `[...]` to after `data`, as shown in step 6? I forgot that before and added it in the last edit. Hope it works now :) – hbaderts Dec 08 '16 at 21:38
  • what does the [...] mean? – ng0323 Dec 06 '18 at 05:51
  • With `[...]` you modify the `data` in-place, i.e. copies the data to the place where `data` points now. Without the `[...]` it would change the reference to point to `conv_av_weights` instead of copying the data. See [this answer](https://stackoverflow.com/a/42191121/4221706) for more details. – hbaderts Dec 06 '18 at 09:19