How to add skip connection between convolutional layers in Keras

Question

I would like to add a skip connection between residual blocks in keras. This is my current implementation, which does not work because the tensors have different shapes.

The function looks like this:

def build_res_blocks(net, x_in, num_res_blocks, res_block, num_filters, res_block_expansion, kernel_size, scaling):
    net_next_in = net
    for i in range(num_res_blocks):
        net = res_block(net_next_in, num_filters, res_block_expansion, kernel_size, scaling)
    
        # net tensor shape: (None, None, 32)
        # x_in tensor shape: (None, None, 3)
        # Error here, net_next_in should be in the shape of (None, None, 32) to be fed into next layer
        net_next_in = Add()([net, x_in]) 

    return net

But I get

ValueError: Operands could not be broadcast together with shapes (None, None, 32) (None, None, 3)

How to add or merge these tensors into the correct shape (None, None, 32)? If this is not the correct approach, how could you achieve the intended result?

This is what the res_block looks like:

def res_block(x_in, num_filters, expansion, kernel_size, scaling):
    x = Conv2D(num_filters * expansion, kernel_size, padding='same')(x_in)
    x = Activation('relu')(x)
    x = Conv2D(num_filters, kernel_size, padding='same')(x)
    x = Add()([x_in, x])
return x

score 3 · Accepted Answer · answered Feb 21 '19 at 14:47

3

You cant add tensors of different shape. You could concatenate them with keras.layers.Concatenate, but this would leave you with a tensor of shape [None, None, 35].

Alternatively, have a look at the Resnet50 implementation in Keras. Their residual block features a 1x1xC convolution in the shortcut for those cases where the dimensions to be added are different.

answered Feb 21 '19 at 14:47

mrks

513
3
7

Well, something seems to still be wrong. I implemented something similar as in the Resnet50: [https://www.codepile.net/pile/KAqdVpmG] This appears during runtime: `tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [16,46,46,8] vs. [16,48,48,8]` Any ideas? – atlas Feb 21 '19 at 16:49
While the number of channel now matches, your spatial dimensions do not (46 vs 48). You could set ```padding='same'```` for the convolution(s) and/or pooling layers within your ```res_block```. – mrks Feb 21 '19 at 16:54
Yes, I have 'same' in every Conv-layer. – atlas Feb 21 '19 at 16:56
well, something __does__ strip off one pixel on each side within your ```res_block```... (Btw I dont see anything in your codepile link). – mrks Feb 21 '19 at 16:57
Yes, obviously something is doing that. I added the res_block code as well. Sorry, this link might work better: https://www.codepile.net/pile/W6K594e0 – atlas Feb 21 '19 at 17:02
My bad! I missed that I had added two convolutional layers before and after the residual blocks with padding 'valid'. I updated them to 'same' and now it works. Thank you for your help! – atlas Feb 21 '19 at 17:09

How to add skip connection between convolutional layers in Keras

1 Answers1