I understand we need to use different types of hidden unit other than binary, if we are dealing with rectifier, continuous input, etc. in RBM. Now suppose RBM is used as starting point for supervised deep learning, how about tanh activation? Can we fit a binary sigmoid RBM and use the trained weights as the starting point for a tanh-activation-function neural network? Essentially tanh is a re-scaled sigm that is in binary RBM, I guess it is do-able? If so, how exactly?
Thanks
