trainable_variables attribute of the custom layer in Keras returns empty list

Question

I tried to build my own custom layer in tensorflow/keras that enforces the layer to be symmetric and what I ended up with is the following:

import tensorflow as tf
from tensorflow.python.framework.ops import enable_eager_execution
enable_eager_execution()

class MyDenseLayer(tf.keras.layers.Layer):
    def __init__(self, num_outputs):
        super(MyDenseLayer, self).__init__()
        self.num_outputs = num_outputs

    def build(self, input_shape):
        X = tf.random.uniform([int(input_shape[-1]),self.num_outputs],minval=0,maxval=1,dtype=tf.dtypes.float32,)
        k = tf.Variable(X, name="kernel")
        self.kernel = 0.5 * (k+tf.transpose(k))

    def call(self, input):
        return tf.matmul(input, self.kernel)

layer = MyDenseLayer(5)
print(layer(tf.ones([3, 5])))
print(layer.trainable_variables)

So far, so good. What I don't understand this: why does the last line

print(layer.trainable_variables)

give me an empty list:

[]

I thought that layer.trainable_variables would show me what my matrix looks like so that I could check whether it is symmetric or not.

Vlad · Accepted Answer · 2019-04-27T19:30:49.823

3

You need to add variables using add_weight and then call build() method to create this variable. Alternatively, instead of calling build() directly you can pass an input (as you do in your question) and it will call implicitly the build() method.

import tensorflow as tf
from tensorflow.python.framework.ops import enable_eager_execution
enable_eager_execution()

class MyDenseLayer(tf.keras.layers.Layer):
    def __init__(self, num_outputs):
        super(MyDenseLayer, self).__init__()
        self.num_outputs = num_outputs
    def build(self, input_shape):
        def initializer(*args, **kwargs):
            X = tf.random.uniform([int(input_shape[-1]),self.num_outputs],minval=0,maxval=1,dtype=tf.dtypes.float32,)
            kernel = 0.5 * (X+tf.transpose(X))
            return kernel
        self.kernel = self.add_weight(name='kernel',
                                      shape=(input_shape[-1], self.num_outputs),
                                      initializer=initializer,
                                      trainable=True)
        super(MyDenseLayer, self).build(input_shape)

    def call(self, input_):
        return tf.matmul(input_, self.kernel)

layer = MyDenseLayer(5)
layer.build((5, )) # <-- example of input shape
print(layer.trainable_variables)
# [<tf.Variable 'kernel:0' shape=(5, 5) dtype=float32, numpy=
# array([[0.04476559, 0.8396935 , 0.42732996, 0.75126845, 0.7109113 ],
#        [0.8396935 , 0.46617424, 0.71654373, 0.5770991 , 0.38461512],
#        [0.42732996, 0.71654373, 0.75249636, 0.28733748, 0.6064501 ],
#        [0.75126845, 0.5770991 , 0.28733748, 0.9417101 , 0.61572695],
#        [0.7109113 , 0.38461512, 0.6064501 , 0.61572695, 0.6960379 ]],
#       dtype=float32)>]

edited Apr 27 '19 at 19:30

answered Apr 27 '19 at 19:06

Vlad

8,225
5
33
45

Looks good. Does this layer stay symmetric after training? It seems like it is symmetric from the beginning but will change its property through backpropagation. – xabdax Apr 27 '19 at 21:32
1

Not the layer, but the weights of the layer! The layer is referred to the what your layer outputs (the return of the call method). The weights of the layer will keep the same size as you defined it in build method. It won’t change its size. The output of the layer may depend on the batch size though (the first dimension). – Vlad Apr 27 '19 at 21:38
Ok, I see. I tried to use the modified build and implement it in one of the existing keras layers (https://github.com/keras-team/keras/blob/a1397169ddf8595736c01fcea084c8e34e1a3884/keras/layers/core.py#L796) but I am having issues with the output shape as in MyDenseLayer and the given parameters of the class Dense. Do you have any suggestions about how to implement your initializer in the class Dense? Do you think that is even a good idea? – xabdax Apr 27 '19 at 21:51
To implement the initializer just replace `num_outputs` with `units`. But I think it is wrong idea since the distribution of your initializer hasn’t changed. It is still uniform[0, 1] so just use built-in uniform initializer. – Vlad Apr 27 '19 at 22:13
So back to a previous question because I am not sure if I understood correctly: do you think that the weights will stay symmetric after backpropagation? The dimensions will definitely stay the same. – xabdax Apr 27 '19 at 22:50
the weights are updated during backpropagation. So no, the values won't stay the same. – Vlad Apr 28 '19 at 09:55
Is there any way to make them have a symmetric shape despite backpropagation? – xabdax Apr 28 '19 at 13:34
It is possible, however you must specify how do you want to enforce the symmetry mathematically. – Vlad Apr 28 '19 at 13:38
Well, as I tried above: I was looking for symmetric layers in terms of matrices that fulfill transpose(A) = A, i.e. matrices whose upper triangular part is the same as the lower triangular part. The purpose of this is to take advantage of the symmetries of my system and decrease computational power. – xabdax Apr 28 '19 at 14:18

trainable_variables attribute of the custom layer in Keras returns empty list

1 Answers1