Stacked Sparse Autoencoder parameters

Question

I work on Stacked Sparse Autoencoders using MATLAB. Can anyone please suggest what values should be taken for Stacked Sparse Autoencoder parameters:
L2 Weight Regularization ( Lambda) Sparsity Regularization (Beta) Sparsity proportion (Rho).

Answer 1

It is important to realise that there are NO OBVIOUS VALUES for the hyperparameters. The optimal value will vary, depending on the data you're modeling: you'll have to try them on your data.

From sparseAutoencoder Lambda (λ) is coefficient of weight decay term which discourage weights to reach big values since it may overfit. Weight decay term (or weight regularization term) is a part of the cost function like sparsity term explained below.

Rho (ρ) is sparsity constraint which controls average number of activation on hidden layer. It is included to make autoencoder work even with relatively big number of hidden units with respect to input units. For example, if input size is 100 and hidden size is 100 or larger (even smaller but close to 100), the output can be constructed without any lost, since hidden units can learn identity function. Beta (β) is coeffecient of sparsity term which is a part of the cost function. It controls relative importance of sparsity term. Lambda and Beta specify the relative importance of their terms in cost function.

Example: You can take a loot at this example where parameter values are selected as follows.

sparsityParam = 0.1;   % desired average activation of the hidden units.
                       % (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
                       %  in the lecture notes). 
lambda = 3e-3;         % weight decay parameter       
beta = 3;              % weight of sparsity penalty term

But once again, i want to make you remember that there are NO OBVIOUS VALUES for the hyperparameters.