Can someone help explain why my MLP keeps on getting a perfect classification report?

Question

I am using Sklearn.train_test_split and sklearn.MLPClassifier for human activity recognition. Below is my dataset in a pandas df:


a_x a_y a_z g_x g_y g_z activity
0   3.058150    5.524902    -7.415221   0.001280    -0.022299   -0.009420   sit
1   3.065333    5.524902    -7.422403   -0.003514   -0.023764   -0.007289   sit
2   3.065333    5.524902    -7.422403   -0.003514   -0.023764   -0.007289   sit
3   3.064734    5.534479    -7.406840   -0.016830   -0.025628   -0.003294   sit
4   3.074910    5.548246    -7.408038   -0.023488   -0.025495   -0.001963   sit
... ... ... ... ... ... ... ...
246886  8.102990    -1.226492   -4.559391   -0.511287   0.081455    0.109515    run
246887  8.120349    -1.218711   -4.595306   -0.516480   0.089179    0.110047    run
246888  8.126933    -1.209732   -4.619848   -0.521940   0.096636    0.109382    run
246889  8.140102    -1.199556   -4.622840   -0.526467   0.102761    0.108183    run
246890  8.142496    -1.199556   -4.648580   -0.530728   0.109818    0.108050    run
1469469 rows × 7 columns

I am using the 6 numerical columns (x,y,z from accelerometer and gyrosphere) to predict activity (run, sit, walk). My code looks like

mlp=MLPClassifier(hidden_layer_sizes=(10,), activation='relu', solver='adam', learning_rate='adaptive', 
                 early_stopping=True, learning_rate_init=.001)

X=HAR.drop(columns='activity').to_numpy()
y=HAR['activity'].to_numpy()

X_train, X_test, y_train, y_test=train_test_split(X,y, train_size=0.10)

mlp.fit(X_train, y_train)
predictions_train=mlp.predict(X_train)
predictions_test=mlp.predict(X_test)

print("Fitting of train data for size (10,): \n",classification_report(y_train,predictions_train))
print("Fitting of test data for size (10,): \n",classification_report(y_test,predictions_test))

Output is:

Fitting of train data for size (10,): 
               precision    recall  f1-score   support

         run       1.00      1.00      1.00     49265
         sit       1.00      1.00      1.00     49120
        walk       1.00      1.00      1.00     48561

    accuracy                           1.00    146946
   macro avg       1.00      1.00      1.00    146946
weighted avg       1.00      1.00      1.00    146946

Fitting of test data for size (10,): 
               precision    recall  f1-score   support

         run       1.00      1.00      1.00    441437
         sit       1.00      1.00      1.00    442540
        walk       1.00      1.00      1.00    438546

    accuracy                           1.00   1322523
   macro avg       1.00      1.00      1.00   1322523
weighted avg       1.00      1.00      1.00   1322523

I am relatively new to ML but I think I understand the concept of overfitting, so I imagine that is what is happening here, but I don't understand how it is being overfit when it is only being trained on 10% of the dataset? Also, presumably the classification report should always be perfect for the X_train data since that is what the model is being trained on, correct?

No matter what I do, it always produces a perfect classification_report for the X_test data no matter how little data I train it on (in this case .10 but i've done .25, .5, .33 etc.). I even removed the gyrosphere data and only trained it on the accelerometer data and it still gave a perfect 1 for each precision, recall, and F1.

When I arbitrarily slice the original dataset in half and use the resulting arrays as train and test data then the predictions for X_test are not perfect but every time I use the sklearn.train_test_split it returns a perfect classification report....So i assume I am doing something wrong with how I am using train_test_split?

score 1 · Accepted Answer · edited Jan 18 '23 at 13:40

1

It's quite hard to say without having access to the data to try out.

I wonder if within the data itself, the class separation is really clear such that the classifier has no trouble distinguishing. (It seems so just seeing the values you printed.. The distributions are very different and well separated if you plot them. So to be fair a NN is overkill, if even by visual plotting we are able to clearly distinguish different activities.)

Have you tried smaller hidden layer sizes, say only 1 or 2 nodes, or some other simpler classifier? E.g. decision tree with max_depth set, say to <4, or just a logistic regression model.

Also did you try stratifying: train_test_split(X,y, train_size=0.10, stratify=y)

My guess, I think it's just a very simple dataset, thus the classifier is doing very well because the class separations are so clear. So it's nothing to do with overfitting.

edited Jan 18 '23 at 13:40

Dharman

30,962
25
85
135

answered Nov 28 '22 at 10:40

Hanafi Haffidz

148
8

When you arbitrarily sliced the data in half, test predictions are not perfect probably because it was only trained on "sit, stand, walk" but then tested on "run, sleep". – Hanafi Haffidz Nov 28 '22 at 10:47
I figured out the main problem was that I was not normalizing my data. So I could train the model on however large a dataset I wanted, the moment it received other values it wasn't any good. Once I normalized the data, then it became clear that, yes, it was a pretty easy classification problem :) – JP1990 Nov 29 '22 at 05:05

Can someone help explain why my MLP keeps on getting a perfect classification report?

1 Answers1