I am creating a lstm-seq2seq-chatbot using a translational model based off the example from here: https://github.com/ChunML/seq2seq
Instead I am using Cornell's Movie Dialogue Corpus to train the model. However when I do train it the final prediction is just zero's and during training the accuracy very quickly reaches 80%. This is my model below in which I trained for 50 epochs,3 layers with 256 units and a batch size of 64:
model = Sequential()
model.add(Embedding(x_vocab_len, 1000,input_length = x_max_len,mask_zero = True))
model.add(LSTM(units))
model.add(RepeatVector(y_max_len))
for i in range(depth):
model.add(LSTM(units,return_sequences = True))
model.add(TimeDistributed(Dense(y_vocab_len)))
model.add(Activation('softmax'))
model.compile(loss = 'categorical_crossentropy',
optimizer = 'rmsprop',metrics = ['accuracy'])
return model
I am new to this (which is probably obvious) so if you need anything else then that's fine. I just want to know if there is anything really obvious that will screw up training. Below is some snippets of part of the training in which I am using FloydHub:
2017-12-14 17:17:58,010 INFO - ----vectorising data------
2017-12-14 17:17:59,523 INFO - Training model: epoch 50th 131000/133586 samples
2017-12-14 17:17:59,523 INFO - Epoch 1/1
2017-12-14 17:18:12,840 INFO - 100/1000 [==>...........................] - ETA: 11s - loss: 1.1921e-07 - acc: 0.8198
200/1000 [=====>........................] - ETA: 10s - loss: 1.1921e-07 - acc: 0.7945
300/1000 [========>.....................] - ETA: 9s - loss: 1.1921e-07 - acc: 0.7906
400/1000 [===========>..................] - ETA: 7s - loss: 1.1921e-07 - acc: 0.7954
500/1000 [==============>...............] - ETA: 6s - loss: 1.1921e-07 - acc: 0.7967
600/1000 [=================>............] - ETA: 5s - loss: 1.1921e-07 - acc: 0.7945
700/1000 [====================>.........] - ETA: 3s - loss: 1.1921e-07 - acc: 0.7993
800/1000 [=======================>......] - ETA: 2s - loss: 1.1921e-07 - acc: 0.8039
300/586 [==============>...............] - ETA: 3s - loss: 1.1921e-07 - acc: 0.8139
400/586 [===================>..........] - ETA: 2s - loss: 1.1921e-07 - acc: 0.8127
500/586 [========================>.....] - ETA: 1s - loss: 1.1921e-07 - acc: 0.8146
586/586 [==============================] - 7s - loss: 1.1921e-07 - acc: 0.8091
2017-12-14 17:18:45,166 INFO - 18001
2017-12-14 17:18:45,587 INFO - [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2017-12-14 17:18:45,588 INFO - 0 0 0 0 0 0 0 0 0]]
2017-12-14 17:18:45,588 INFO - ['']
