Fit_generator working single set of hyperparameter but MemoryError with multiple sets of hyperparameter using Talos

Question

I am trying to do a small assignment project where i am trying to use Siamese network to do speaker recognition. I am trying to use talos for finding best possible hyperparameters. When trying a single set of hyperparamter the code went through OK but when i am giving sets of hyperparameters i am getting MemoryError.

Background (not sure if required but for the sake of it kindly allow me to explain the same)

Splitted audio into 5 seconds chunk each
Calculated MFCC and MFCC_del and MFCC_del_del of each audio sample and conctenated them all so each audio sample is converted into(24x939) data point/matrix
Created siamese network by having two sets of matrixes where we have one set of all "similar" paired speaker audio samples and another being that of "disimilar speakers".
Total of 6000 test points and 14886 training points (not using validation now will use later)


def yield_arrays_train(array_x_train_feat1,array_x_train_feat2,array_y_train,batch_size):
    while 1:
        for i in range(14886):            
            X_feat1_train = (array_x_train_feat1[i:i+batch_size,:,:].astype(np.float32))
            X_feat2_train = (array_x_train_feat2[i:i+batch_size,:,:].astype(np.float32))
            Y_train = (array_y_train[i:i+batch_size].astype(np.float32))
            yield ([(np.array(X_feat1_train)),(np.array(X_feat2_train))],(np.array(Y_train)))

def yield_arrays_val(array_x_test_feat1,array_x_test_feat2,array_y_test,batch_size):
    while 1:
        for i in range(60):            
            X_feat1_test = (array_x_test_feat1[i:i+batch_size,:,:].astype(np.float32))
            X_feat2_test = (array_x_test_feat2[i:i+batch_size,:,:].astype(np.float32)) 
            Y_test = (array_y_test[i:i+batch_size].astype(np.float32))
            yield ([(np.array(X_feat1_test)),(np.array(X_feat2_test))],(np.array(Y_test)))

train_generator=yield_arrays_train(xtrain_np_img1,xtrain_np_img2,y_train_numpy,6)
val_generator=yield_arrays_val(xtest_np_img1,xtest_np_img2,y_test_numpy,6)

def siamese(generator,validation_data):

    W_init = tf.keras.initializers.he_normal(seed=100)
    b_init = tf.keras.initializers.he_normal(seed=50)

    input_shape = (24,939)
    left_input = Input(input_shape)
    right_input = Input(input_shape)

    encoder = Sequential()
    encoder.add(Conv1D(filters=8,kernel_size=6, padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
    encoder.add(BatchNormalization())
    encoder.add(Dropout(.1))
    encoder.add(MaxPool1D())
    encoder.add(Conv1D(filters=6,kernel_size=4, padding='same', activation='relu'))
    encoder.add(BatchNormalization())
    encoder.add(Dropout(.1))
    encoder.add(MaxPool1D())
    encoder.add(Conv1D(filters=4,kernel_size=4, padding='same', activation='relu'))
    encoder.add(BatchNormalization())
    encoder.add(Dropout(.1))
    encoder.add(MaxPool1D())
    encoder.add(Flatten())
    encoder.add(Dense(10,activation='relu'))
    encoder.add(Dropout(.1))
    encoded_l = encoder(left_input)
    encoded_r = encoder(right_input)
    distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
    adam = optimizers.Adam(lr=.1, beta_1=0.1, beta_2=0.999,decay=.1, amsgrad=False)
    earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
    callback_early_stop_reduceLROnPlateau=[earlyStopping]
    model = Model([left_input, right_input], distance)
    model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
    model.summary()
    #history = model.fit([(x_train[:,:,:,0]).astype(np.float32),(x_train[:,:,:,1]).astype(np.float32)],y_train, validation_data=([(x_val[:,:,:,0]).astype(np.float32),(x_val[:,:,:,1]).astype(np.float32)], y_val) ,batch_size=params['batch_size'],epochs=params['epochs'],callbacks=callback_early_stop_reduceLROnPlateau)
    history =model.fit_generator(generator=train_generator,validation_data=val_generator,steps_per_epoch=2481,epochs=5, validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
    return history,model

siamese(train_generator,val_generator)

OUTPUT:

Model: "model_6"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_11 (InputLayer)           (None, 24, 939)      0                                            
__________________________________________________________________________________________________
input_12 (InputLayer)           (None, 24, 939)      0                                            
__________________________________________________________________________________________________
sequential_6 (Sequential)       (None, 10)           45580       input_11[0][0]                   
                                                                 input_12[0][0]                   
__________________________________________________________________________________________________
lambda_6 (Lambda)               (None, 1)            0           sequential_6[1][0]               
                                                                 sequential_6[2][0]               
==================================================================================================
Total params: 45,580
Trainable params: 45,544
Non-trainable params: 36
__________________________________________________________________________________________________
Epoch 1/5
2481/2481 [==============================] - 29s 12ms/step - loss: 0.0056 - accuracy: 0.9986 - val_loss: 0.8333 - val_accuracy: 0.1667
Epoch 2/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 0.9993 - val_loss: nan - val_accuracy: 0.8333
Epoch 3/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 1.0000 - val_loss: nan - val_accuracy: 0.8333
Epoch 4/5
2481/2481 [==============================] - 28s 11ms/step - loss: nan - accuracy: 1.0000 - val_loss: nan - val_accuracy: 0.8333
Epoch 00004: early stopping
(<keras.callbacks.callbacks.History at 0x2969fa3fd88>,
 <keras.engine.training.Model at 0x2969f2a17c8>)

So without any Talos or it is working with some one set of random hyperparameters. Now i use Talos for first time

p = {
       'filter1':[1,2,4,6,8,12,16,24],
    'kernel_size1':[2,4,6,8,12],
    'filter3' :  [1,2,4,6,8],
    'kernel_size3' :  [1,2,4,6,8,12],
    'decay' :[.1,0.01,.001,.0001,.00001],
    'droprate1' :[.1,.2,.3],
    'filter2':[1,2,4,6,8],
    'kernel_size2':[2,4,6,8,12],
    'droprate4' :  [.1,.2,.3],
    'droprate2' :[.1,.2,.3],
    'unit1': [10,24,36,64,128,256],
    'droprate3': [.1,.2,.3],
    'lr' :[(.1,0.01,.001,.0001,.00001)],
    'batch_size' : [1,2],
    'epochs': [4,8,10]             }

def siamese(generator,validation_data,params):

    W_init = tf.keras.initializers.he_normal(seed=100)
    b_init = tf.keras.initializers.he_normal(seed=50)

    input_shape = (24,939)
    left_input = Input(input_shape)
    right_input = Input(input_shape)

    encoder = Sequential()
    encoder.add(Conv1D(filters=(params['filter1']),kernel_size=(params['kernel_size1']), padding='same', activation='relu',input_shape=input_shape,kernel_initializer=W_init, bias_initializer=b_init))
    encoder.add(BatchNormalization())
    encoder.add(Dropout((params["droprate1"])))
    encoder.add(MaxPool1D())
    encoder.add(Conv1D(filters=(params["filter2"]),kernel_size=(params['kernel_size2']), padding='same', activation='relu'))
    encoder.add(BatchNormalization())
    encoder.add(Dropout((params["droprate2"])))
    encoder.add(MaxPool1D())
    encoder.add(Conv1D(filters=(params["filter3"]),kernel_size=(params['kernel_size3']), padding='same', activation='relu'))
    encoder.add(BatchNormalization())
    encoder.add(Dropout((params['droprate3'])))
    encoder.add(MaxPool1D())
    encoder.add(Flatten())
    encoder.add(Dense((params['unit1']),activation='relu'))
    encoder.add(Dropout((params['droprate4'])))

    encoded_l = encoder(left_input)
    encoded_r = encoder(right_input)
    distance = Lambda(euclidean_distance, output_shape=eucl_dist_output_shape)([encoded_l, encoded_r])
    adam = optimizers.Adam(lr=params['lr'], beta_1=0.1, beta_2=0.999,decay=params['decay'], amsgrad=False)
    earlyStopping = EarlyStopping(monitor='loss',min_delta=0,patience=3,verbose=1,restore_best_weights=False)
    callback_early_stop_reduceLROnPlateau=[earlyStopping]
    model = Model([left_input, right_input], distance)
    model.compile(loss=contrastive_loss, optimizer=adam,metrics=[accuracy])
    model.summary()
    #history = model.fit([(x_train[:,:,:,0]).astype(np.float32),(x_train[:,:,:,1]).astype(np.float32)],y_train, validation_data=([(x_val[:,:,:,0]).astype(np.float32),(x_val[:,:,:,1]).astype(np.float32)], y_val) ,batch_size=params['batch_size'],epochs=params['epochs'],callbacks=callback_early_stop_reduceLROnPlateau)
    history=model.fit_generator(generator=train_generator,validation_data=val_generator,steps_per_epoch=2481,epochs=5,validation_steps=1000,verbose=1,callbacks=callback_early_stop_reduceLROnPlateau,use_multiprocessing=False,workers=0)
    return history,model

t=ta.Scan(x=[xtrain_np_img1,xtrain_np_img2],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')

The Error i get is:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-9-df856388a4bb> in <module>
      1 #t=ta.Scan(x=xtrain_np_img1_img2,y=y_train_numpy,x_val=xtest_np_img1_img2,y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')
      2 
----> 3 t=ta.Scan(x=[xtrain_np_img1,xtrain_np_img2],y=y_train_numpy,x_val=[xtest_np_img1,xtest_np_img2],y_val=y_test_numpy,model=siamese,params=p,experiment_name='exp_1')

~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\Scan.py in __init__(self, x, y, params, model, experiment_name, x_val, y_val, val_split, random_method, seed, performance_target, fraction_limit, round_limit, time_limit, boolean_limit, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, minimize_loss, disable_progress_bar, print_params, clear_session, save_weights)
    194         # start runtime
    195         from .scan_run import scan_run
--> 196         scan_run(self)

~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_run.py in scan_run(self)
      7 
      8     from .scan_prepare import scan_prepare
----> 9     self = scan_prepare(self)
     10 
     11     # initiate the progress bar

~\anaconda3\envs\MyEnv\lib\site-packages\talos\scan\scan_prepare.py in scan_prepare(self)
     28                                    round_limit=self.round_limit,
     29                                    time_limit=self.time_limit,
---> 30                                    boolean_limit=self.boolean_limit
     31                                    )
     32 

~\anaconda3\envs\MyEnv\lib\site-packages\talos\parameters\ParamSpace.py in __init__(self, params, param_keys, random_method, fraction_limit, round_limit, time_limit, boolean_limit)
     42 
     43         # create the parameter space
---> 44         self.param_space = self._param_space_creation()
     45 
     46         # handle the boolean limits separately

~\anaconda3\envs\MyEnv\lib\site-packages\talos\parameters\ParamSpace.py in _param_space_creation(self)
    133         if len(self.param_index) > 100000:
    134 
--> 135             final_grid = list(it.product(*self._params_temp))
    136             out = np.array(final_grid, dtype='object')
    137 

MemoryError:

My question is since i am new to python/ machine learning etc

How to properly use fit_generator (earlier i tried using just fit function and my windows would give screen of death because of memory error i switched to trying fit_generator but still same error) . I think i am not writing code properly?
Is my laptop specifications so poor to run this code? My specifications RAM : 16 GB windows 10 64 bit CPU : intel core i5 (quad core) GPU: it has inbuilt GTX1050 but i havent configured it as i tried earlier but didnt had much luck

Hi Saurabh, this is a lot of code for someone to look at and understand. You should try to reproduce the same error in as few lines of code as possible so that people can easily zero in on the specific part which is causing the problem. See [this](https://webkit.org/test-case-reduction/) explanation of test case reduction. Instead of "my code doesn't work, why not?" you should work to identify the specific conditions that cause the error, and post "Why does x condition cause y error in tensorflow, siamsese network". — FinleyGibson, Oct 15 '20 at 13:53
Thanks FinleyGibson Thanks for responding i would be removing some part of code as it might be irrelevant i wasn't sure if someone would be interested just in case. However i would like to say i have presented two sets of my code. 1st case where i am using single set of parameters which works fine 2nd case where i use hyperparameter optimization tool talos with sets of parameters where i get MemoryError so i was wondering is there problem in my code or Talos changing title Fit_generator working single set of hyperparameter but MemoryError with multiple sets of hyperparameter using Talos — Saurabh, Oct 16 '20 at 02:41
I was checking few more similar questions and few people pointed using DASK dataframe compared to fit_generator? But what i dont understand fit_generator is working fine with single set of hyperparameter and why it wont work with multiple sets of parameter? Does Talos does keeps all memory used for each iterations? — Saurabh, Oct 16 '20 at 02:45
Ok after carefully looking at error i realized it says ~\anaconda3\envs\MyEnv\lib\site-packages\talos\parameters\ParamSpace.py in _param_space_creation(self) 133 if len(self.param_index) > 100000: 134 --> 135 final_grid = list(it.product(*self._params_temp)) 136 out = np.array(final_grid, dtype='object') 137 MemoryError: So i checked how many parameters i ve given its 318,864,600,000,000 nos of total sets of parameter which is crazyyy . So i just reduced the nos of parameters and MemoryError is gone but now some other error — Saurabh, Oct 20 '20 at 01:54
So these parameters are the weights in your model, and running out of memory is not uncommon when dealing with large networks. Update your question to reflect the current error and use a simplified version of the code: as few weights, as few biases, as few network layers as possible. The most simple reduction of the code that produces the current error. — FinleyGibson, Oct 20 '20 at 10:37
Hello FinleyGibson. Thanks for responding. Yes actually i started a new thread because this MemoryError issue was resolved. This thread can be closed now. https://stackoverflow.com/questions/64437881/the-argument-must-be-a-tuple-of-1-integers-received-or-typeerror-int-argumen This is the new thread — Saurabh, Oct 20 '20 at 15:57

Fit_generator working single set of hyperparameter but MemoryError with multiple sets of hyperparameter using Talos

0 Answers0