7

I am using Keras for computing a simple sequence classification neural network. I played with the different module and I found that there are two way to create Sequential neural network.

The first way is to use Sequential API. This is the most common way which I found in a lot of tutorial/documentation. Here is the code :

# Sequential Neural Network using Sequential()
model = Sequential()
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu', input_shape=(27 , 300,)))
model.add(MaxPooling1D(pool_size=2))
model.add(LSTM(100))
model.add(Dense(len(7, activation='softmax'))
model.summary()

The second ways is to build de sequential neural network from "scratch" with the Model API. Here is the code.

# Sequential neural network using Model()   
inputs = Input(shape=(27 , 300))
x = Conv1D(filters=32, kernel_size=3, padding='same', activation='relu')(inputs)
x = MaxPooling1D(pool_size=2)(x)
x = LSTM(100)(x)
predictions = Dense(7, activation='softmax')(x)
model = Model(inputs=inputs, outputs=predictions)
model.summary()

I trained it both with a fixed seed (np.random.seed(1337)), with the same training data and my output are different... Knowing that the only difference in the summary is the first layer of inputs with the Model API.

Is there anyone that knows why this neural network are different ? And if there are not, why did i get different results ?

Thanks

Clement Viricel
  • 256
  • 3
  • 7
  • How different are the values? And which environment are you using? – Dr. Snoopy Feb 01 '18 at 11:57
  • Small differences can happen due to random initialization of weights even if you retrain the same model multiple times. You could check by running the Sequential API model a few times. – Manngo Feb 01 '18 at 12:29
  • @MatiasValdenegro The outputs are really different, i can't show you the results because of the company i work for. I am using python3 with Keras which uses tensorflow backend. – Clement Viricel Feb 01 '18 at 13:18
  • @Manngo Ok thanks. Did I have a way to fix the seed for the weights ? – Clement Viricel Feb 01 '18 at 13:19
  • just to make sure nothing random is happening, did you try to train the same network twice and see if you get the same result? it this is not the case something is still random – WellDone2094 Feb 01 '18 at 16:21
  • @WellDone2094 I did. And it gives different output... I am going to search how to fix the seed for Keras. – Clement Viricel Feb 02 '18 at 10:54

2 Answers2

6

You setup the random seed only in numpy and not in tensorflow (in case it's the backend of keras in your case). Try to add this in your code:

from numpy.random import seed
seed(1337)
from tensorflow import set_random_seed
set_random_seed(1337)

the detailed article about this topic here

Andrey Kite Gorin
  • 1,030
  • 1
  • 9
  • 23
0
 tf.keras.backend.clear_session()
 tf.random.set_seed(seed_value)

You can use above code block and run the loaded model for some iterations and check if the error still persists . I was facing the same issue for reproducibiity,it worked for me . As mentioned by andrey, over and above these 2 seed setter, you need to setup the Python Hash Environment 

import os
os.environ['PYTHONHASHSEED']=str(seed_value)

you can still add one more block to force TensorFlow to use single thread. ( if you are using multicore) Multiple threads are a potential source of non-reproducible results.

  session_conf = tf.ConfigProto(
  intra_op_parallelism_threads=1,
  inter_op_parallelism_threads=1)
  sess = tf.Session(config=session_conf)
cc111222
  • 11
  • 3