Loss not decrasing and is very high keras

Question

I'm learning deep learning in keras and I have a problem. The loss isn't decreasing and it's very high, about 650.

I'm working on MNIST dataset from tensorflow.keras.datasets.mnist There is no error, just my NN isn't learning.

There is my model:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
import tensorflow.nn as tfnn

inputdim = 28 * 28

model = Sequential()

model.add(Flatten())
model.add(Dense(inputdim, activation = tfnn.relu))
model.add(Dense(128, activation = tfnn.relu))
model.add(Dense(10, activation = tfnn.softmax))

model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.fit(X_train, Y_train, epochs = 4)

and my output:

Epoch 1/4
60000/60000 [==============================] - 32s 527us/sample - loss: 646.0926 - acc: 6.6667e-05
Epoch 2/4
60000/60000 [==============================] - 39s 652us/sample - loss: 646.1003 - acc: 0.0000e+00 - l - ETA: 0s - loss: 646.0983 - acc: 0.0000e
Epoch 3/4
60000/60000 [==============================] - 35s 590us/sample - loss: 646.1003 - acc: 0.0000e+00
Epoch 4/4
60000/60000 [==============================] - 33s 544us/sample - loss: 646.1003 - acc: 0.0000e+00
```

You can try change your optimizer and add more epochs to your train. Make some tests with that. Try using Sigmoid activation in your NN. — Alex Colombari, Jul 25 '19 at 18:56
Not sure about dataset, do you normalize the images? And loss is so big, try to set batch size. — viceriel, Jul 25 '19 at 19:49
You probably have not normalized the images. If that's the case then see [this answer](https://stackoverflow.com/a/52859705/2099607) or [this one](https://stackoverflow.com/a/53987162/2099607). — today, Jul 25 '19 at 20:10

score 1 · Accepted Answer · answered Jul 27 '19 at 17:52

Ok, I added BatchNormalization between lines and changed loss function to 'sparse_categorical_crossentropy'. That's how my NN looks like:

model = Sequential()

model.add(Flatten())
model.add(BatchNormalization(axis = 1, momentum = 0.99))
model.add(Dense(inputdim, activation = tfnn.relu))
model.add(BatchNormalization(axis = 1, momentum = 0.99))
model.add(Dense(128, activation = tfnn.relu))
model.add(BatchNormalization(axis = 1, momentum = 0.99))
model.add(Dense(10, activation = tfnn.softmax))

model.compile(loss = 'sparse_categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

and thats a results:

Epoch 1/4
60000/60000 [==============================] - 68s 1ms/sample - loss: 0.2045 - acc: 0.9374
Epoch 2/4
60000/60000 [==============================] - 55s 916us/sample - loss: 0.1007 - acc: 0.9689

Thanks for your help!

Good job in figuring out the issues; you should *accept* your own answer - please see [Can I answer my own question?](https://stackoverflow.com/help/self-answer) — desertnaut, Aug 08 '19 at 15:40

score 0 · Answer 2 · answered Jul 25 '19 at 20:05

0

You may try sparse_categorical_crossentropy loss function. Also what is your batch size? and as has already been suggested you may want to increase number of epochs.

answered Jul 25 '19 at 20:05

Amit

2,018
1
8
12

Loss not decrasing and is very high keras

2 Answers2