0

I have written a Keras custom layer foo (no trainable weights within it), which accepts an input x and returns x + n where n is additive white Gaussian noise with a pre-specified standard deviation. I wish to train my functional model model-trg, which includes foo, instantiated with noise level set to sigma-trg (during training phase). In the test model model-test, I duplicate all trainable layers of model-trg except foo. In model-test, I would like to create a second different instance of foo with parameter sigma-test, different from sigma-trg. The objective is to test whether the model, following training, is able to generalize its learning for different noise levels.

Surprisingly, when I print the summary of model-trg and model-test, they appear to use foo as a shared layer i.e. the standard deviation parameter for model-test appears to be set identically to the standard deviation parameter in model-trg. The end result is that the performance of my test model is no different than the performance of my training model.

Are there any suggestions on how to force Keras during compilation to avoid creating shared layers if parameters within that layer is different across the training and test model?

Sharing layers across models

if enable_trg is True:
     model_trg = Model(inputs=source_input, outputs= preds)
     # some code specifying the layers and connections
     # some code to compile and train the model
     model_trg.save_weights(model_name + '.h5') # save weights

if enable_test is True: 
    model_trg.load_weights(model_name + '.h5')    
    model_test= Model(inputs=source_input, outputs = preds)

Training model

y = foo(batch_size=trg_batch_size,noise_stdev=stdev_trg)(x)

Test model

y = foo(batch_size=test_batch_size,noise_stdev=stdev_test)(x)

Custom layer 

class Foo(Layer): 
    def __init__(self, batch_size, noise_stdev, **kwargs):
        super(Foo, self).__init__(**kwargs)        
        self.batch_size = batch_size
        self.stdev = noise_stdev                
        self.supports_masking = True

    def build(self,inputShape):
        super(Foo,self).build(inputShape)

    def call(self, inputs, training=None):                      
        y = inputs[0]
        noise = K.random_normal(shape=K.shape(y),mean=0.,stddev=self.stdev) 
        return y + noise

    def get_config(self):
        config = {
                'batch_size': self.batch_size,
                'noise_stdev': self.stdev                
                 }
        base_config = super(Foo, self).get_config()        
        return dict(list(base_config.items()) + list(config.items()))

    def compute_output_shape(self, input_shape):
        return input_shape
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
Vikram Chandrasekhar
  • 55
  • 2
  • 7
  • Ok, removed comments. As the code stands you have reassigned y.. is this an example or the final code? – Attersson May 24 '18 at 18:28
  • I don't see a problem there with 2 instances, my only guess would be how you load the test model weights. If you just use `model.save` and `model.load` it also loads the config which would be identical, but can't tell without seeing the code. – nuric May 24 '18 at 18:34
  • @nuric: thank you - you may have pointed me at the root cause. I am using `save_weights` and `load_weights` rather than `save` and `load`. Could that be tied to the issues I am facing? – Vikram Chandrasekhar May 24 '18 at 18:46
  • @Attersson: my understanding is that following model compilation, the tensorflow graph should determine y by adding to `x` noise at the noise level for that model (i.e. with std deviation `sigma-trg` for training model and `sigma-test` for test model). So, the fact that I have re-assigned `y` should not matter. – Vikram Chandrasekhar May 24 '18 at 19:05
  • Makes sense. It could have, however, kept a reference when later printing the summary or otherwise displaying. Ok though. – Attersson May 24 '18 at 19:06
  • @Attersson: Right. I plan to go through line-by-line and compare the model summaries to figure out the underlying reason - it is most likely some bug in my code. – Vikram Chandrasekhar May 24 '18 at 22:12

0 Answers0