1

I am trying to implement my own RBM, but I am not sure, how to measure it's success 100% correctly. So I started googling and have found many interpretations and I am not sure what is correct.

I am facing this problem:

I have dataset Z, so I can divide it into training set X and testing set Y. I train RBM on X and then I would like to measure success of RBM on dataset Y. More precisely, let's say, I have two RBMs and I want to compare them somehow. I am not sure if reconstructing input vector is good measurement. Or If I should compare RBMs on their energy. (and how do I calculate energy on whole set Y correctly).

I would be also interested in gaussian-visible and all-gaussian units, if possible.

NathanOliver
  • 171,901
  • 28
  • 288
  • 402
Snurka Bill
  • 973
  • 4
  • 12
  • 29

1 Answers1

1

A RBM is an unsupervised learning paradigm, and therefore is difficult to access whether one is better than another.

Nevertheless, they are usually used as a pre-training of recent and more exciting networks such as DBN. So my suggestion would be to train as much RBMs as you want to compare (unsupervised learning) and then give them to a feedforward layer for learning (supervised learning). From here you can now access how good your RBM is by measuring how good your network is to predict the class of your data.

As an example, lets have 2 RBMs (A and B):

  • you give A to a feedforward layer (train with backpropagation) and got an accuracy with the test data of 80%;

  • you give B to a feedforward layer (train with backpropagation) and got an accuracy with the test data of 90%;

As such, B is a better RBM than A, as it provided better features, leading to a better training and higher out-of-sample results. Note: as accuracy of networks vary, make sure you perform the supervised training several times and average them at the end so that your comparison is robust.

EDIT:

Regarding non-supervised evaluation the task is not as simple. As presented by Tijmen Tieleman in "Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient":

One of the evaluations is how well the learned RBM models the test data, i.e. log likelihood. This is intractable for regular size RBMs, because the time complexity of that computation is exponential in the size of the smallest layer (visible or hidden)

Yet, if you have small enough RBMs this is a possible approach. Otherwise, you can just wait...

ASantosRibeiro
  • 1,247
  • 8
  • 15
  • I've heard about DNB it is my plan to implement them too. Considered we have MNIST problem, then my RBM can be 784 visible, 100 hidden, or 784 visible and 200 hidden. Now by running backpropagation you mean adding linear classifier behind hidden layer? So for MNIST I would get network 784 - 100 - 10 resp. 784 - 200 - 10? Or I don't know how to implement it. I've shouldn't mention two datasets. I just want to separate training and testing sets, but still have unsupervised data. – Snurka Bill Nov 12 '14 at 18:14
  • simple backpropagation on top. So 784:100:10 (if 10 classes). Train the hidden with RBM, initialize the output layer with random, run simple feedforward with backpropagation on the last layer, or total network. – ASantosRibeiro Nov 12 '14 at 18:55
  • Sounds great, but what if I have unsupervised data anyway? I still play in my head with idea of comparing energy somehow. Would you mind to discuss it more? – Snurka Bill Nov 13 '14 at 11:26