How to extract lower dimensional feature vectors from a denoising stacked autoencoder using python and tensorflow

Question

The code below imports the MNIST data set and trains a stacked denoising autoencoder to corrupt, encode, then decode the data. Basically I want to use this as a non-linear dimensional reduction technique. How can I access the lower dimensional features that the model is encoding so I can throw these into a clustering model? Ideally I would expect the lower dimensional features to be loops or straight lines (obviously this will not be the case in reality).

import numpy as np
import os
import sys
import tensorflow as tf


from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/")


def plot_image(image, shape=[28, 28]):
    plt.imshow(image.reshape(shape), cmap="Greys", interpolation="nearest")
    plt.axis("off")

def reset_graph(seed=42):
    tf.reset_default_graph()
    tf.set_random_seed(seed)
    np.random.seed(seed)


def show_reconstructed_digits(X, outputs, model_path = None, n_test_digits = 2):
    with tf.Session() as sess:
        if model_path:
            saver.restore(sess, model_path)
        X_test = mnist.test.images[:n_test_digits]
        outputs_val = outputs.eval(feed_dict={X: X_test})

    fig = plt.figure(figsize=(8, 3 * n_test_digits))
    for digit_index in range(n_test_digits):
        plt.subplot(n_test_digits, 2, digit_index * 2 + 1)
        plot_image(X_test[digit_index])
        plt.subplot(n_test_digits, 2, digit_index * 2 + 2)
        plot_image(outputs_val[digit_index])


reset_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01

noise_level = 1.0

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
X_noisy = X + noise_level * tf.random_normal(tf.shape(X))

hidden1 = tf.layers.dense(X_noisy, n_hidden1, activation=tf.nn.relu,
                          name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book
                          name="hidden2")                            # not shown
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown
                          name="hidden3")                            # not shown
outputs = tf.layers.dense(hidden3, n_outputs, name="outputs")        # not shown

reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(reconstruction_loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 10
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})
        print("\r{}".format(epoch), "Train MSE:", loss_train)
        saver.save(sess, "./my_model_stacked_denoising_gaussian.ckpt")


show_reconstructed_digits(X, outputs, "./my_model_stacked_denoising_gaussian.ckpt")

For example lower dimensional features of an 8 are two loops, or a 9 is a loop and a straight line — Ian Campbell Moore, Nov 19 '17 at 01:44

Mo- · Accepted Answer · 2017-11-18T10:48:10.413

An Autoencoder, in each layer of encoding part, learns discriminative features and then in the reconstruction phase (in the decoding part) it tries to use those features to shape the output. But, when autoencoders are used to locally extract lower dimensional features, it would be much more efficient if you use Convolutional Autoencoders(CAE).

An intuitive answer to your question could be using the feature maps which are produced in the decoding part of a CAE as lower dimensional extracted features. I mean, train an N-layer CAE on your dataset, and then ignore the output layer, and use the outputs of Convolutional layers for your clustering purposes.

Just for more clarification, each of that 5x5 feature maps (S_2) in the above image could be considered as a feature. You can find quick demonstration and implementation of a CAE here.

And finally, it'd be better to ask such a question on Data Science community.

How to extract lower dimensional feature vectors from a denoising stacked autoencoder using python and tensorflow

1 Answers1