I'm using Keras to train a netwok for classification problem on python, the model that i am using is as follows:
filter_size = (2,2)
maxpool_size = (2, 2)
dr = 0.5
inputs = Input((12,8,1), name='main_input')
main_branch = Conv2D(20, kernel_size=filter_size, padding="same", kernel_regularizer=l2(0.0001),bias_regularizer=l2(0.0001))(inputs)
main_branch = BatchNormalization(momentum=0.9)(main_branch)
main_branch = Activation("relu")(main_branch)
main_branch = MaxPooling2D(pool_size=maxpool_size,strides=(1, 1))(main_branch)
main_branch = Conv2D(40, kernel_size=filter_size, padding="same", kernel_regularizer=l2(0.0001),bias_regularizer=l2(0.0001))(main_branch)
main_branch = BatchNormalization(momentum=0.9)(main_branch)
main_branch = Activation("relu")(main_branch)
main_branch = Flatten()(main_branch)
main_branch = Dense(100,kernel_regularizer=l2(0.0001),bias_regularizer=l2(0.0001))(main_branch)
main_branch = Dense(100, kernel_regularizer=l2(0.0001),bias_regularizer=l2(0.0001))(main_branch)
SubArray_branch = Dense(496, activation='softmax', name='SubArray_output')(main_branch)
model = Model(inputs = inputs,
outputs = SubArray_branch)
opt = keras.optimizers.Adam(lr=1e-3, epsilon=1e-08, clipnorm=1.0)
model.compile(optimizer=opt,
loss={'SubArray_output': 'sparse_categorical_crossentropy'},
metrics=['accuracy'] )
history = model.fit({'main_input': Channel},
{'SubArray_output': array_indx},
validation_data=(test_Data,test_array),
epochs=100, batch_size=128,
verbose=1,
validation_split=0.2
)
when I am training this network on my training data I get high validation-loss compared to training-loss as you can see in below:
471/471 [==============================] - 5s 10ms/step - loss: 0.5723 - accuracy: 0.9010 - val_loss: 20.2040 - val_accuracy: 0.0126
Epoch 33/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5486 - accuracy: 0.9087 - val_loss: 35.2516 - val_accuracy: 0.0037
Epoch 34/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5342 - accuracy: 0.9159 - val_loss: 50.2577 - val_accuracy: 0.0043
Epoch 35/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5345 - accuracy: 0.9132 - val_loss: 26.0221 - val_accuracy: 0.0051
Epoch 36/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5333 - accuracy: 0.9140 - val_loss: 71.2754 - val_accuracy: 0.0043
Epoch 37/100
471/471 [==============================] - 5s 11ms/step - loss: 0.5149 - accuracy: 0.9231 - val_loss: 67.2646 - val_accuracy: 3.3227e-04
Epoch 38/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5269 - accuracy: 0.9162 - val_loss: 17.7448 - val_accuracy: 0.0206
Epoch 39/100
471/471 [==============================] - 5s 11ms/step - loss: 0.5198 - accuracy: 0.9201 - val_loss: 92.7240 - val_accuracy: 0.0015
Epoch 40/100
471/471 [==============================] - 5s 11ms/step - loss: 0.5157 - accuracy: 0.9247 - val_loss: 30.9589 - val_accuracy: 0.0082
Epoch 41/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4961 - accuracy: 0.9316 - val_loss: 20.0444 - val_accuracy: 0.0141
Epoch 42/100
471/471 [==============================] - 5s 11ms/step - loss: 0.5093 - accuracy: 0.9256 - val_loss: 16.7269 - val_accuracy: 0.0172
Epoch 43/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5092 - accuracy: 0.9267 - val_loss: 15.6939 - val_accuracy: 0.0320
Epoch 44/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5104 - accuracy: 0.9270 - val_loss: 103.2581 - val_accuracy: 0.0027
Epoch 45/100
471/471 [==============================] - 5s 10ms/step - loss: 0.5074 - accuracy: 0.9286 - val_loss: 28.3097 - val_accuracy: 0.0154
Epoch 46/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4977 - accuracy: 0.9303 - val_loss: 28.6676 - val_accuracy: 0.0167
Epoch 47/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4823 - accuracy: 0.9375 - val_loss: 47.4671 - val_accuracy: 0.0015
Epoch 48/100
471/471 [==============================] - 5s 11ms/step - loss: 0.5053 - accuracy: 0.9291 - val_loss: 39.3356 - val_accuracy: 0.0082
Epoch 49/100
471/471 [==============================] - 5s 11ms/step - loss: 0.5110 - accuracy: 0.9287 - val_loss: 42.8834 - val_accuracy: 0.0082
Epoch 50/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4895 - accuracy: 0.9366 - val_loss: 11.7254 - val_accuracy: 0.0700
Epoch 51/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4909 - accuracy: 0.9351 - val_loss: 14.5519 - val_accuracy: 0.0276
Epoch 52/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4846 - accuracy: 0.9380 - val_loss: 22.5101 - val_accuracy: 0.0122
Epoch 53/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4991 - accuracy: 0.9315 - val_loss: 16.1494 - val_accuracy: 0.0283
Epoch 54/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4782 - accuracy: 0.9423 - val_loss: 14.8626 - val_accuracy: 0.0551
Epoch 55/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4807 - accuracy: 0.9401 - val_loss: 100.8670 - val_accuracy: 9.9681e-04
Epoch 56/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4759 - accuracy: 0.9420 - val_loss: 34.8571 - val_accuracy: 0.0047
Epoch 57/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4802 - accuracy: 0.9406 - val_loss: 23.2134 - val_accuracy: 0.0524
Epoch 58/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4998 - accuracy: 0.9334 - val_loss: 20.9038 - val_accuracy: 0.0207
Epoch 59/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4813 - accuracy: 0.9400 - val_loss: 19.5474 - val_accuracy: 0.0393
Epoch 60/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4846 - accuracy: 0.9399 - val_loss: 15.1594 - val_accuracy: 0.0439
Epoch 61/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4718 - accuracy: 0.9436 - val_loss: 30.0164 - val_accuracy: 0.0078
Epoch 62/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4897 - accuracy: 0.9375 - val_loss: 60.0498 - val_accuracy: 0.0144
Epoch 63/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4668 - accuracy: 0.9461 - val_loss: 18.8190 - val_accuracy: 0.0298
Epoch 64/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4598 - accuracy: 0.9485 - val_loss: 26.1101 - val_accuracy: 0.0231
Epoch 65/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4672 - accuracy: 0.9442 - val_loss: 108.7207 - val_accuracy: 2.6582e-04
Epoch 66/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4910 - accuracy: 0.9378 - val_loss: 45.6070 - val_accuracy: 0.0052
Epoch 67/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4805 - accuracy: 0.9429 - val_loss: 39.3904 - val_accuracy: 0.0057
Epoch 68/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4682 - accuracy: 0.9451 - val_loss: 21.5525 - val_accuracy: 0.0328
Epoch 69/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4613 - accuracy: 0.9472 - val_loss: 46.7714 - val_accuracy: 0.0027
Epoch 70/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4786 - accuracy: 0.9417 - val_loss: 13.4834 - val_accuracy: 0.0708
Epoch 71/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4756 - accuracy: 0.9442 - val_loss: 41.8796 - val_accuracy: 0.0199
Epoch 72/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4655 - accuracy: 0.9464 - val_loss: 57.7453 - val_accuracy: 0.0017
Epoch 73/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4795 - accuracy: 0.9428 - val_loss: 16.1949 - val_accuracy: 0.0285
Epoch 74/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4755 - accuracy: 0.9440 - val_loss: 68.2349 - val_accuracy: 0.0139
Epoch 75/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4807 - accuracy: 0.9425 - val_loss: 43.4699 - val_accuracy: 0.0233
Epoch 76/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4515 - accuracy: 0.9524 - val_loss: 175.2205 - val_accuracy: 0.0019
Epoch 77/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4715 - accuracy: 0.9467 - val_loss: 92.2833 - val_accuracy: 0.0017
Epoch 78/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4736 - accuracy: 0.9447 - val_loss: 94.7209 - val_accuracy: 0.0059
Epoch 79/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4661 - accuracy: 0.9473 - val_loss: 17.8870 - val_accuracy: 0.0386
Epoch 80/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4614 - accuracy: 0.9492 - val_loss: 28.1883 - val_accuracy: 0.0042
Epoch 81/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4569 - accuracy: 0.9507 - val_loss: 49.2823 - val_accuracy: 0.0032
Epoch 82/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4623 - accuracy: 0.9485 - val_loss: 29.8972 - val_accuracy: 0.0100
Epoch 83/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4799 - accuracy: 0.9429 - val_loss: 109.5044 - val_accuracy: 0.0062
Epoch 84/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4810 - accuracy: 0.9444 - val_loss: 71.2103 - val_accuracy: 0.0051
Epoch 85/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4452 - accuracy: 0.9552 - val_loss: 30.7861 - val_accuracy: 0.0100
Epoch 86/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4805 - accuracy: 0.9423 - val_loss: 48.1887 - val_accuracy: 0.0031
Epoch 87/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4564 - accuracy: 0.9512 - val_loss: 189.6711 - val_accuracy: 1.3291e-04
Epoch 88/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4479 - accuracy: 0.9537 - val_loss: 58.6349 - val_accuracy: 0.0199
Epoch 89/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4667 - accuracy: 0.9476 - val_loss: 95.7323 - val_accuracy: 0.0041
Epoch 90/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4808 - accuracy: 0.9436 - val_loss: 28.7513 - val_accuracy: 0.0191
Epoch 91/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4583 - accuracy: 0.9511 - val_loss: 16.4281 - val_accuracy: 0.0431
Epoch 92/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4458 - accuracy: 0.9541 - val_loss: 15.3890 - val_accuracy: 0.0517
Epoch 93/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4628 - accuracy: 0.9491 - val_loss: 37.3123 - val_accuracy: 0.0024
Epoch 94/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4716 - accuracy: 0.9481 - val_loss: 24.8934 - val_accuracy: 0.0123
Epoch 95/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4646 - accuracy: 0.9469 - val_loss: 54.6682 - val_accuracy: 5.9809e-04
Epoch 96/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4665 - accuracy: 0.9492 - val_loss: 89.1835 - val_accuracy: 0.0064
Epoch 97/100
471/471 [==============================] - 5s 10ms/step - loss: 0.4533 - accuracy: 0.9527 - val_loss: 60.9850 - val_accuracy: 0.0035
Epoch 98/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4597 - accuracy: 0.9491 - val_loss: 41.6088 - val_accuracy: 0.0023
Epoch 99/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4511 - accuracy: 0.9537 - val_loss: 28.2131 - val_accuracy: 0.0025
Epoch 100/100
471/471 [==============================] - 5s 11ms/step - loss: 0.4568 - accuracy: 0.9509 - val_loss: 121.8944 - val_accuracy: 0.0041
I am well aware that the problem I am facing is due to overfitting, but when I train the same network with the same training data on Matlab, the values of training-loss and validation-loss are close to each other. The picture of Training Progress on Matlab is linked as: Training Progress
I would appreciate it, if anyone could explain to me why I can’t get the same result on python? What would you suggest to solve this problem?
