0

I have written a working implementation of an RBM with binary hidden/visible units in R. I've been looking for a while but just can't figure how to change the binary units to either gaussian or ReLUs.

If I wanted my input data to be real values, would I change the visible units and the hidden units? Or just the visible units?

Lets say I wanted to change both. Currently, I'm calculating the hidden/visible probabilities using the logistic sigmoid function (1/(1+e^(-x))). The ReLU uses max(0, x + N(0,1)). As I currently understand, I would switch all occurrences of the logistic sigmoid function with the ReLU max function. However, this doesn't yield results that make a bit of sense. So I'm not sure what I'm actually supposed to be changing.

Rahul Gupta
  • 1
  • 1

1 Answers1

0

You can change the visible unit activation without changing the hidden unit activation. So you could have Bernoulli hidden units and Gaussian visible units. If pre-training for a deep network it's best to use the same activation function that you will have in your final network (commonly relus).

I've also written RBM code, a trick to get better results is to not sample the visual reconstruction or daydream (when using CD-1). I wouldn't be able to help more without looking at your code or having you clarify 'a bit of sense' for your results.

A great guide for tips on training RBMs can be found in A practical guide to training RBMs written by Hinton.

Another tip is to use Persistent Contrastive Divergence which assumes the weight updates are small enough to not change the markov chain. In code, it equates to activating the visual units from the last daydream and the daydream from the visual reconstruction. In my experience it takes much less time to train, original paper: http://www.machinelearning.org/archive/icml2008/papers/638.pdf

Again, post your code and results and I'm willing to help you look through them.

islandman93
  • 410
  • 4
  • 4