My model doesn't seem to work, as accuracy and loss are 0

Question

I tried to design an LSTM network using keras but the accuracy is 0.00 while the loss value is 0.05 the code which I wrote is below.

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(1, activation = tf.nn.relu))



def percentage_difference(y_true, y_pred):
    return K.mean(abs(y_pred/y_true - 1) * 100)



model.compile(optimizer='sgd', 
             loss='mse',
             metrics = ['accuracy', percentage_difference])

model.fit(x_train, y_train.values, epochs = 10)

my input train and test data set have been imported using the pandas' library. The number of features is 5 and the number of target is 1. All endeavors will be appreciated.

@Geeocode What do you mean? you need data to work or you need more information about the job that I'm doing? — Hossein Amini, Feb 03 '20 at 13:54
Some kind of sample data, that you work with. Furthermore, why do you think, that this is a LSTM network? — Geeocode, Feb 03 '20 at 13:58
I'm working with an environmental dataset that represents a cyclic procedure and the data frame corresponds to spatially characteristics. The network which is capable of doing a similar process on the same dataset is just RNN and LSTM. That's why I used this model of the network. — Hossein Amini, Feb 03 '20 at 14:06
@Geeocode , if you saw the model details in code, by using the tf.keras.model.Sequential, I defined the LSTM layer. Am I wrong? — Hossein Amini, Feb 03 '20 at 14:33
No, Sequential is just the Keras model abstraction type: Linear stack of layers. — Geeocode, Feb 03 '20 at 14:57
@Geeocode, Yes I got it that I did not design the true LSTM architecture and I just called some libraries. [link] (https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/) — Hossein Amini, Feb 03 '20 at 15:01
In the link the author use LSTM: `model.add(LSTM(4, input_shape=(1, look_back)))` — Geeocode, Feb 03 '20 at 15:04
after writing *model.add(LSTM(4, input_shape=(1, look_back))) * the number which has been written 4 is the number of features or else? Also, for input shape the number, what does it mean? — Hossein Amini, Feb 03 '20 at 15:59

Mihai Alexandru-Ionut · Accepted Answer · 2020-02-03T14:42:55.560

1

From what I see is that you're using a neural network applied for a regression problem.

Regression is the task of predicting continuous values by learning from various independent features.

So, in the regression problem we don't have metrics like accuracy because this is for classification branch of the supervised learning.

The equivalent of accuracy for regression could be coefficient of determination or R^2 Score.

from keras import backend as K

def coeff_determination(y_true, y_pred):
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

model.compile(optimizer='sgd', 
         loss='mse',
         metrics = [coeff_determination])

edited Feb 03 '20 at 14:42

answered Feb 03 '20 at 12:09

Mihai Alexandru-Ionut

47,092
13
101
128

When I tried to work with your solution, the "coeff_determination" became [-0.0098]. I guess, regardless of the sign, we are facing with overfitting? Am I right? – Hossein Amini Feb 03 '20 at 14:02
https://stats.stackexchange.com/questions/12900/when-is-r-squared-negative – Mihai Alexandru-Ionut Feb 03 '20 at 14:05
@HosseinAmini, I think it's underfitting. – Mihai Alexandru-Ionut Feb 03 '20 at 14:08
@HosseinAmini, if you add the full code I can help you more and i will try to detect your issue. – Mihai Alexandru-Ionut Feb 03 '20 at 14:14
@HosseinAmini, just adding it to the answer text – Mihai Alexandru-Ionut Feb 03 '20 at 14:19
Meanwhile, the "coeff_determination" increased to [+0.04] just by increasing the number of epochs. – Hossein Amini Feb 03 '20 at 14:30
@HosseinAmini, try to set epochs to `100`. – Mihai Alexandru-Ionut Feb 03 '20 at 14:30
@HosseinAmini, However, until calculate the `R Squared` score, try to make some predictions and plot them in order to see how it works the algorithm. – Mihai Alexandru-Ionut Feb 03 '20 at 14:31
For doing 100 epochs it takes more than a day, is it usual? if so, I will try and let you know about the result. In the meantime, if you do a favor for me and say to me, Am I wrong with the structure of the network which I designed? Did I design the LSTM in the correct way, regardless of optimizing hyperparameters? – Hossein Amini Feb 03 '20 at 14:37
@HosseinAmini, I think you not designed correctly the LSTM network because you have no LSTM layer. See here one example: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ – Mihai Alexandru-Ionut Feb 03 '20 at 14:41
@HosseinAmini, You can try to use `20` epochs first in order to see if any improvement exists. – Mihai Alexandru-Ionut Feb 03 '20 at 14:50
Ok, I will let you know. I had spatial characteristics of 1000 points for 1000 cycles. In such, 5 features (that can be increased to 8) affect on the one target. So, how many layers can be designed? and the activation function which is gonna be used is good to be what? – Hossein Amini Feb 03 '20 at 14:59
There is no rule for how many layers can be designed. This is done based on the experiments. – Mihai Alexandru-Ionut Feb 03 '20 at 15:03
1

After 20 epochs I could reach the 0.11 accuracy coefficient which means by increasing the number of epochs I can go toward fitting. Then I will add LSTM layer to the model based on the explanation and procedure that has been written in the link you have sent to me. Thanks for you clearance and efforts – Hossein Amini Feb 03 '20 at 17:51

My model doesn't seem to work, as accuracy and loss are 0

1 Answers1

Linked