Cross-entropy validation losses comes out as a straight line

Question

I'm trying to calculate cross-entropy losses using the Iris dataset, but when I ran my model and fired up my plots, both my losses and validation losses remained a straight line at zero. I don't know what I'm doing wrong. This is my code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from tensorflow import keras
from keras import Sequential
from keras.layers import BatchNormalization, Dense, Dropout
from keras.callbacks import EarlyStopping

iris = sns.load_dataset('iris')
X = iris.iloc[:,:4]
y = iris.species.replace({'setosa': 0, 'versicolor': 1, 'virginica': 2})

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3, random_state=69)

sc = StandardScaler()
sc.fit_transform(X_train)
sc.fit_transform(X_test)

nn_model = Sequential([Dense(4, activation='relu', input_shape=[X.shape[1]]),
                     BatchNormalization(),
                     Dropout(.3),
                     Dense(4, activation='relu'),
                     BatchNormalization(),
                     Dropout(.3),
                     Dense(1, activation='sigmoid')])

nn_model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['categorical_accuracy'])

early_stopping = EarlyStopping(min_delta=1e-3, patience=10, restore_best_weights=True)

fit = nn_model.fit(X_train, y_train, validation_data=(X_test,y_test), 
                   batch_size=16, epochs=200, callbacks=[early_stopping], verbose=1)

losses = pd.DataFrame(fit.history)

And this is what the plots look like:

Any reason why it's doing this?

You have 3 classes, so use this: `Dense(3, activation='softmax')` in the last layer. And change the loss function to `loss='sparse_categorical_crossentropy'`. And, use Standard Scaler like this: `X_train = sc.fit_transform(X_train)`, your scaler isn't transforming anything. — Adarsh Wase, Jan 10 '22 at 10:42
Correct. Also change metric to only `metrics=['accuracy']`. @AdarshWase I think it's ok to post the answer with some details. — Innat, Jan 10 '22 at 10:47
@AdarshWase I didn't know that I didn't assign my splits to the scaler. Thanks for pointing that out. — javanewb1, Jan 10 '22 at 22:30

score 3 · Accepted Answer · edited Jan 13 '22 at 19:31

The fit transformation of StandardScaler() isn't in-place operation. You have to do as follows

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

Also, you have 3 outputs (check: y_train.value_counts()), so the output layer should be:

 nn_model = Sequential([ ..., 
                       Dropout(.3),
                       Dense(3, activation='softmax')])

Lastly, the loss function should be sparse_categorical_crossentropy for your integer target.

nn_model.compile(optimizer='sgd', 
                 loss='sparse_categorical_crossentropy', 
                 metrics=['accuracy'])

Thank you! It works the way it's supposed to. – javanewb1 Jan 10 '22 at 22:21 — javanewb1, Jan 10 '22 at 22:21

Cross-entropy validation losses comes out as a straight line

1 Answers1