0

I am trying to use keras tuner hyperband to select hyperparameters for my autoencoder model. Here's some pseudo code:

class AEHyperModel(kt.HyperModel):
   def __init_(self, input_shape):
      self.input_shape = input_shape

   def build(self, hp):
    
       # Input layer
       x = Input(shape=(input_shape,))
    
       # Encoder layer 1
       hp_units1 = hp.Choice('units1', values=[10, 15])       
       z = Dense(units=hp_units1, activation='relu')(x)
       
       # Encoder layer 2
       hp_units2 = hp.Choice('units2', values=[3, 4])
       z = Dense(units=hp_units2, activation='relu')(z)
       
       # Decoder
       y = Dense(units=hp_units1, activation='relu')(z)
       y = Dense(inputnodes,activation='linear')(y)

       # compile encoder
       autoencoder = Model(x,y)                                                       
       autoencoder.compile(optimizer='adam', loss='mean_squared_error')
    
       return autoencoder

   def fit(self, hp, autoencoder, *args, **kwargs):
       return autoencoder.fit(*args,
                            batch_size=hp.Choice('batch_size', [16]),
                            **kwargs)

and this is how I am performing my hyperparameter search:

   tuner= kt.Hyperband(AEHyperModel(input_shape),
                       objective='val_loss',
                       max_epochs=100,
                       factor=3,
                       directory=os.path.join(softwaredir, 'AEhypertuning'),
                       project_name='DRautoencoder2',
                       overwrite=True,
                       hyperband_iterations=5,
                       executions_per_trial=5)

   # patient early stopping and tensorboard callback
   callbacks=[EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=20, restore_best_weights=True),
              keras.callbacks.TensorBoard('C:/whatever/tmp/tb_logs')]

   # Search the best hyperparameters for the model
   tuner.search(Xz, Xz,
                       epochs=100,
                       shuffle=True,
                       validation_data=(valz, valz),
                       callbacks=[callbacks])

If I then do:

tuner.executions_per_trial
Out[12]: 5

which seems right. According to the documentation, oracle.trials[trialid] contains metrics, which contains a MetricsTracker object described as:

_Record of the values of multiple executions of all metrics.

It contains MetricHistory instances for the metrics.

Args: metrics: List of strings of the names of the metrics._

Then to know the results of each exeuction per trial, I could go into metrics.metrics, and see a val_loss MetricsHistory object, described in the docu as:

_Record of multiple executions of a single metric.

It contains a collection of MetricObservation instances.

Args: direction: String. The direction of the metric to optimize. The value should be "min" or "max"._

However,

tuner.oracle.trials['0000'].metrics.metrics['val_loss'].get_history()
Out[19]: [MetricObservation(value=[0.118283973634243], step=1)]

tuner.oracle.trials['0000'].metrics.metrics['val_loss'].get_statistics()
Out[20]: 
{'min': 0.118283973634243,
 'max': 0.118283973634243,
 'mean': 0.118283973634243,
 'median': 0.118283973634243,
 'var': 0.0,
 'std': 0.0}

suggesting only one execution per trial was performed?

Can I get any help on how to actually perform multiple executions per trial and being able to access the results of each execution, or at least the mean and std to know how stable the results are?

Thanks

YoungResearcher
  • 91
  • 1
  • 3

1 Answers1

0

I have solved it by creating a custom Tensorflow callback if it can be of use to anyone:

from keras.callbacks import Callback

class Logger(Callback):

    def on_train_begin(self, logs=None):
                               
        # Create scores holder
        global val_score_holder
        val_score_holder = []
        global train_score_holder
        train_score_holder = []
                
    def on_epoch_end(self, epoch, logs):
        
        # Access tuner and logger from the global workspace
        global val_score_holder
        global train_score_holder
        
        # Store scores
        val_score_holder.append(logs['val_loss'])
        train_score_holder.append(logs['loss'])
            
    def on_train_end(self, logs):
        
        # Access tuner and score holders from the global workspace
        global tuner
        global val_score_holder
        global train_score_holder
        
        # Get last (current) trial
        trial = list(tuner.oracle.trials.keys())[-1]
        
        # Create new attributes if not already present i.e. new trial
        if 'rep_val_loss' not in dir(tuner.oracle.trials[trial]):
            tuner.oracle.trials[trial].rep_val_loss = []
            tuner.oracle.trials[trial].rep_train_loss = []
        
        # Add min val loss and corresponding training loss    
        tuner.oracle.trials[trial].rep_val_loss.append(np.min(val_score_holder))
        tuner.oracle.trials[trial].rep_train_loss.append(train_score_holder[np.argmin(val_score_holder)])
YoungResearcher
  • 91
  • 1
  • 3