0

I am trying to implement simple autoencoder like below.

The number of input features are 2, and I want to build sparse autoencoder for dimension reduction to feature 1. I selected the number of nodes are 2(input), 8(hidden), 1(reduced feature), 8(hidden), 2(output) to add some more complexity than using only (2, 1, 2) nodes. The number of samples N is around 10000. 'DATA' is a just a 2x10000 matrix containing integer values.

import tensorflow as tf

x = tf.placeholder(shape=[None, 2])
w1 = tf.Variable(tf.random_normal(shape=[2, 8]))
w2 = tf.Variable(tf.random_normal(shape=[8, 1]))
h1 = tf.nn.relu(tf.matmul(x, w1))
encoded = tf.matmul(h1, w2)
h2 = tf.nn.relu(encoded)
h3 = tf.nn.relu(tf.matmul(h2, tf.transpose(w2)))
y = tf.matmul(h3, tf.transpose(w1))
mse = tf.reduce_mean(tf.squared_difference(x, y))
optimizer = 
tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(mse)
sess = tf.Session()
sess.run(init)
fd = {x: DATA}
loss_value, reduced_feature = sess.run([mse, encoded], feed_dict=fd)

I have 2 questions with the implementation, as the result was quite different as I expected.

  1. Is this implementation correct? Will the variable 'reduced_feature' show the reduced feature(1d feature) from 2 feature inputs?

  2. Should I add some sparsity condition if I want to use more hidden nodes than input? If yes, can you show some sample code for this task?

z991
  • 713
  • 1
  • 9
  • 21
  • 1
    I don't know where your data comes from, but if both features are interesting, it's normal that you have trouble getting good results when going from 2 features to 1, it's a big reduction. I'm not a specialist of autoencoders, but you should probably add biases at each layer, and maybe you just need more layers (even small ones)... – gdelab Oct 17 '17 at 08:57
  • Thank you for your advice. The data I used was just two lists of random integers, which have similar tendency. I understand that losing information is inevitable, but the result was not good enough as expected. I think adding biases would be necessary in this case. But I am not sure that adding more layers can improve the performance, because the number of input features are too small. (Maybe already too complex model?) – z991 Oct 17 '17 at 16:44

0 Answers0