How to evaluate the autoencoder used for dimensionality reduction

Question

I am using an autoencoder as a dimensionality reduction technique to use the learned representation as the low dimensional features that can be used for further analysis.

The code snippet:

# Note: implementation --> based on keras 
encoding_dim = 32

# Define input layer
X_input = Input(shape=(X_train.shape[1],))
# Define encoder:
encoded = Dense(encoding_dim, activation='relu')(X_input)
# Define decoder:
decoded = Dense(X_train.shape[1], activation='sigmoid')(encoded)
# Create the autoencoder model
AE_model = Model(X_input, decoded)
#Compile the autoencoder model
AE_model.compile(optimizer='adam', loss='mse')
#Extract learned representation
learned_feature = Model(X_input, encoded)

history = AE_model.fit(X_train, X_train, epochs=10, batch_size=32)

I was looking for a way to measure the quality of the learned representation. I found that one way is to measure the reconstruction error. I use the following code to do so:

import math
reconstr_error = AE_model.evaluate(X_train, X_train, verbose=0)
print('The reconstruction error: %.2f MSE (%.2f RMSE)' % (reconstr_error , math.sqrt(reconstr_error )))

I got 0.00 MSE (0.05 RMSE) as the result. Yet, I am not sure whether the code above is correct or not as to measure the reconstruction error ?. Also, if there is an alternative way to do so, could you please let me know.

What are the dimensions of your input? It may be that your compression is not enough. Also, what is your loss during training? — drops, Aug 25 '20 at 09:34
Thanks for your reply, the dimensions are around 113, the loss is the "mse - mean squared error". — Amhs_11, Aug 25 '20 at 10:05
I got (0.0081). I just want to know whether the way that I used to calculate the reconstruction error is correct or not. — Amhs_11, Aug 25 '20 at 11:01
what do you get when you do this `print('The reconstruction error: %.6f MSE (%.6f RMSE)' % (reconstr_error , math.sqrt(reconstr_error )))` — drops, Aug 25 '20 at 11:02
I run the code with a precision of %.6f and got 0.000194 MSE (0.013933 RMSE). — Amhs_11, Aug 25 '20 at 22:16

leo · Accepted Answer · 2020-08-28T17:25:39.363

For what purpose are you doing the compression? if you have a following classifier model in your project, you can train that model with the normal(not fed into AE) data and see the accuracy or whatever you are measuring. then train the same model but after compressing your data with AE. then if you also get comparabily good results this means you are extracting something useful with autoencoder. epecially if you don't use all of your data for training AE and see how the compression of examples not seen by AE in its training, will effect the accuracy.

In other techniques like PCA for example, the principal components are eigenvectors and corresponding eigenvalues of those eigenvectors are actually quite meaningful, they tell you how much information in the data varies in each direction, it's like variance. but in AE, especially Deep ones, such analysis is not intuitive or at least beyond my knowledge if they exist. but in 1 layer AE maybe you can still do some similar things, actually, 1 layer AE with MSE as the objective is very close to PCA. you can extract those weights in the hidden layer, also apply PCA or eigendecomposition on your data covariance matrix. Then calculate something like Cosine distance between those hidden layer weights and eigenvectors to see if it preserves something meaningful.

I don't know if anything more can be done, but maybe you be able to find some papers to address such things if this is important for you.

thank you for your insights. It makes sense and I agree with you. In terms of the hidden layers, I am planning to add more layers but I just wanted to start off with a single layer. — Amhs_11, Aug 28 '20 at 23:25
As suggested by @leo, I used a following classifier model after the autoencoder. I evaluated the classifier with precision recall and f-score. Yet, there is no improvement of the classifier result (as I tested it with and without the autoencoder). I am afraid that there might some things wrong with my autoencoder network. Any suggestion what might be the issue with the autoencoder that I used? so I can improve it. Thanks. — Amhs_11, Sep 16 '20 at 23:01
The result usually should not be improved after using the autoencoder to compress the data. you are discarding some information. but if the accuracy is similar to the case when you used the whole dataset, and the compression rate is good, like turning images into 32d vectors or something, then this is a good sign actually. it means the autoencoder reduced the input size but did not discard anything useful, so it is doing fine. — leo, Sep 17 '20 at 12:08

How to evaluate the autoencoder used for dimensionality reduction

1 Answers1