-1

Can anyone help me to understand why this model does not give reproducible results? It changes the accuracy values for test sets and other validation sets I am using, each time I run it. I am using a defined seed. I can not understand why that is happening.

Below is part of my code:

np.random.seed(7)

# Create the model
def create_model(neurons=190, init_mode='normal', activation='relu', inputDim=8040, dropout_rate=0.8,
                 learn_rate=0.001, momentum=0.7, weight_constraint=5): 
    model = Sequential()
    model.add(Dense(neurons, input_dim=inputDim, kernel_initializer=init_mode, activation=activation, kernel_constraint=maxnorm(weight_constraint), kernel_regularizer=regularizers.l2(0.002)))
    model.add(Dense(1, activation='sigmoid'))
    optimizer = RMSprop(lr=learn_rate)

    # compile model

    model.compile(loss='binary_crossentropy', optimizer='RmSprop', metrics=['accuracy'])

model = create_model()

seed = 7
# Define k-fold cross validation test harness

kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=seed)
cvscores = []
for train, test in kfold.split(X_train, Y_train):
    print("TRAIN:", train, "VALIDATION:", test)


# Fit the model

history = model.fit(X_train, Y_train, epochs=40, batch_size=50, validation_data=(X_test, Y_test), verbose=0)

I would appreciate some comments on it.

Mauro Nogueira
  • 131
  • 2
  • 13

1 Answers1

1

Your random seed is fixing your cross-validation split but not the model weight initialization, which you specify in create_model when you set keyword "init_mode=normal".

You could try setting the RNG seed prior to calling create_model but depending on how keras is generating its random numbers, you may need to resort to using a custom initializer to get consistent results.

Seed configuration is dependent on several other factors, including which keras backend you are using (Theano vs. TensorFlow) and which python version you are using. See this github issue for additional details.

bogatron
  • 18,639
  • 6
  • 53
  • 47
  • I did not get what do you mean by setting the RNG seed. By the way `random_state` is not a function or attribute of create model, so I do not know how to set a seed there then. Using that custom initializer, does it mean I have to exclude my `init_mode='normal'` and that one proosed by the keras documentaiton you sent me the link? – Mauro Nogueira Jul 29 '17 at 15:28
  • I have tried this: `from keras import backend as K def my_init(shape, dtype=None): return K.random_normal(shape, dtype=dtype, seed=7) def create_model(neurons=190, init_mode='normal', activation='relu', inputDim=8040, dropout_rate=0.8, learn_rate=0.001, momentum=0.7, weight_constraint=5): model = Sequential() model.add(Dense(neurons, input_dim=inputDim, kernel_initializer=my_init, activation=activation, kernel_constraint=maxnorm(weight_constraint), kernel_regularizer=regularizers.l2(0.002)))` but still returns me different results. – Mauro Nogueira Jul 29 '17 at 15:44
  • My point was that there we random numbers being drawn even before you set the random seed. See the update to my answer for additional details. – bogatron Jul 30 '17 at 13:16