1

I m implementing Deep autoencoder using RBM. I understand that, for unfolding the network, we need to use the transposed weights of the encoder for the decoder. But I'm not sure which biases should we use for the decoder. I appreciate it if anyone can elaborate it for me or send me a link for pseudocode.

user1468089
  • 43
  • 1
  • 7

1 Answers1

5

I believe Geoff Hinton makes all of his source code available on his website. He is the go-to guy for the RBM version of this technique.

Basically, if you have an input matrix M1 with dimension 10000 x 100 where 10000 is the number of samples you have and 100 is the number of features and you want to transform it into 50 dimensional space you would train a restricted boltzman machine with a weight matrix of dimensionality 101 x 50 with the extra row being the bias unit that is always on. On the decoding side then you would take you 101 x 50 matrix, drop the extra row from the bias making it a 100 x 50 matrix, transpose it to 50 x 100 and then add another row for the bias unit making it 51 x 100. You can then run the entire network through backpropogation to train the weights of the overall network.

aplassard
  • 759
  • 3
  • 10
  • Thanks, you example made it clear for me. So my unfolding code should be correct correct:
    for i = 1 : no_layers
    nn.W{i} = [dbn.rbm{i}.c dbn.rbm{i}.W];
    nn.W{2 * no_layers - i + 1}=[dbn.rbm{i}.b dbn.rbm{i}.W'];
    end
    Probably there should be some other problems in my code, because I m getting very low error from pretraining, but quite high error in fine tuning. The model is underfitting! Backpropagation error is relatively high, it decreases gradually but stucks in local minima. Do you have any suggestion what could be the problem?     – user1468089 Dec 12 '13 at 06:10
  • I'm having trouble seeing what your code is doing. Could you post the entire training code with backpropogation so that I can see it better? – aplassard Dec 12 '13 at 06:12
  • BTW, c is hidden bias and b is visible bias. – user1468089 Dec 12 '13 at 06:17
  • can you give me your email address please, it s a huge code.tnx – user1468089 Dec 12 '13 at 06:26