element-wise multiplication with broadcasting in keras custom layer

Question

I am creating a custom layer with weights that need to be multiplied by element-wise before activation. I can get it to work when the output and input is the same shape. The problem occurs when I have a first order array as input with a second order array as output. tensorflow.multiply supports broadcasting, but when I try to use it in Layer.call(x, self.kernel) to multiply x by the self.kernel Variable it complains that they are different shapes saying:

ValueError: Dimensions must be equal, but are 4 and 3 for 'my_layer_1/Mul' (op: 'Mul') with input shapes: [?,4], [4,3].

here is my code:

from keras import backend as K
from keras.engine.topology import Layer
import tensorflow as tf
from keras.models import Sequential
import numpy as np

class MyLayer(Layer):

    def __init__(self, output_dims, **kwargs):
        self.output_dims = output_dims

        super(MyLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        # Create a trainable weight variable for this layer.
        self.kernel = self.add_weight(name='kernel',
                                      shape=self.output_dims,
                                      initializer='ones',
                                      trainable=True)


        super(MyLayer, self).build(input_shape)  # Be sure to call this somewhere!

    def call(self, x):
        #multiply wont work here?
        return K.tf.multiply(x, self.kernel)

    def compute_output_shape(self, input_shape):
        return (self.output_dims)

mInput = np.array([[1,2,3,4]])
inShape = (4,)
net = Sequential()
outShape = (4,3)
l1 = MyLayer(outShape, input_shape= inShape)
net.add(l1)
net.compile(loss='mean_absolute_error', optimizer='adam', metrics=['accuracy'])
p = net.predict(x=mInput, batch_size=1)
print(p)

Edit: Given input shape (4,) and output shape (4,3) the weight matrix should be the same shape as the output and initialized with ones. So in the above code the input is [1,2,3,4], the weight matrix should be [[1,1,1,1],[1,1,1,1],[1,1,1,1]] and the output should look like [[1,2,3,4],[1,2,3,4],[1,2,3,4]]

Consider the [`tf.reshape`](https://www.tensorflow.org/api_docs/python/tf/reshape) method — Chan Kha Vu, Nov 14 '17 at 15:13
Why are you changing the output shape if you want elementwise multiplication? — Daniel Möller, Nov 14 '17 at 15:27
I think I would have to manually copy over the values for each row if I used tf.reshape. I feel like tf.multiply should work, but maybe I'm not passing it parameters in the right form. — buckithed, Nov 14 '17 at 15:38
I want the input to be broadcast across multiple rows in the output like this: input: [1,2,3] output: [[1,2,3],[1,2,3],[1,2,3]] — buckithed, Nov 14 '17 at 15:40
And how does the elementwise multiplication participate in that? — Daniel Möller, Nov 14 '17 at 15:42
When you transform (4,) into (4,3), how many weights do you want? 12? 4? 3? What do you want to multiply? — Daniel Möller, Nov 14 '17 at 15:49
if the weight matrix is [[1,1,1],[1,1,1],[1,1,1]] the input would be multiplied by it giving the output in my above comment. Sorry if I didn't make that clear in my post. — buckithed, Nov 14 '17 at 15:51
how can you passe the input and output shapes as tuples? For me, the same code, it doesn't accept but will accept strins like outputShape='4' for example, how do you do it? — SheppLogan, Feb 21 '20 at 13:48

Daniel Möller · Accepted Answer · 2017-11-14T16:03:40.200

Before multiplying, you need to repeat the elements to increase the shape. You can use K.repeat_elements for that. (import keras.backend as K)

class MyLayer(Layer):

    #there are some difficulties for different types of shapes   
    #let's use a 'repeat_count' instead, increasing only one dimension
    def __init__(self, repeat_count,**kwargs):
        self.repeat_count = repeat_count
        super(MyLayer, self).__init__(**kwargs)

    def build(self, input_shape):

        #first, let's get the output_shape
        output_shape = self.compute_output_shape(input_shape)
        weight_shape = (1,) + output_shape[1:] #replace the batch size by 1


        self.kernel = self.add_weight(name='kernel',
                                      shape=weight_shape,
                                      initializer='ones',
                                      trainable=True)


        super(MyLayer, self).build(input_shape)  # Be sure to call this somewhere!

    #here, we need to repeat the elements before multiplying
    def call(self, x):

        if self.repeat_count > 1:

             #we add the extra dimension:
             x = K.expand_dims(x, axis=1)

             #we replicate the elements
             x = K.repeat_elements(x, rep=self.repeat_count, axis=1)


        #multiply
        return x * self.kernel


    #make sure we comput the ouptut shape according to what we did in "call"
    def compute_output_shape(self, input_shape):

        if self.repeat_count > 1:
            return (input_shape[0],self.repeat_count) + input_shape[1:]
        else:
            return input_shape

This works great, thank you. I'm just curious now why tf.multiply doesn't automatically expand the dimensions and repeat the elements when used the way I have it. — buckithed, Nov 14 '17 at 16:18
Hmmm... perhaps you forgot to keep `input_shape[0]`? There is a hidden batch size dimension. When you say the input shape is `(4,)`, it's actually `(None,4)`, and the output shape is `(None, 4, 3)`. If you add all dimensions properly, the multiplication may work. — Daniel Möller, Nov 14 '17 at 16:20
For a multiplication like that to work, you must make the weights and the inputs have the same number of dimensions, like `x.shape=(batch,4,1)` and `weights.shape=(1,4,3)`. This works if only one of the dimensions is different, but I never tried with two different dimensions at once. — Daniel Möller, Nov 14 '17 at 16:23

score 1 · Answer 2 · answered Nov 17 '17 at 15:34

here is another solution that is based on the answer by Daniel Möller, but uses tf.multiply like the original code.

class MyLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim

        super(MyLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        # Create a trainable weight variable for this layer.
        output_shape = self.compute_output_shape(input_shape)
        self.kernel = self.add_weight(name='kernel',
                                      shape=(1,) + output_shape[1:],
                                      initializer='ones',
                                      trainable=True)


        super(MyLayer, self).build(input_shape)  # Be sure to call this somewhere!

    def call(self, x):

        return K.tf.multiply(x, self.kernel)

    def compute_output_shape(self, input_shape):
        return (input_shape[0],self.output_dim)+input_shape[1:]

Hello, I am trying to use this code but it seems i need to define inShape = '4' and outShape = '4' i.e. strings with 1 number inside .. because otherwise i get an error : ValueError: Error converting shape to a TensorShape: invalid literal for int() with base 10: '('. (if using '(4,)' and another error if i use a tuple (4,)... does it make sense to you? Why is that so? can you explain? — SheppLogan, Feb 21 '20 at 13:09

element-wise multiplication with broadcasting in keras custom layer

2 Answers2

Linked