Stack Overflow
Stack Overflow

Questions tagged [regularized]

Regularization involves introducing additional information in order to solve an ill-posed problem or to prevent over-fitting by shrinking the parameters of the model stay close to zero

In mathematics and statistics, particularly in the fields of machine learning and inverse problems, regularization involves introducing additional information in order to solve an ill-posed problem or to prevent overfitting. This information is usually of the form of a penalty for complexity, such as restrictions for smoothness or bounds on the vector space norm.

From http://en.wikipedia.org/wiki/Regularization_%28mathematics%29

<

195 questions
6
votes
1 answer

Lack of Sparse Solution with L1 Regularization in Pytorch

I am trying to implement L1 regularization onto the first layer of a simple neural network (1 hidden layer). I looked into some other posts on StackOverflow that apply l1 regularization using Pytorch to figure out how it should be done (references:…
cyradil
  • 83
  • 1
  • 5
6
votes
1 answer

Dropout in activations or in weights

A fairly easy one, but just getting crazy with it right now. When applying dropout to regularize my neural network, where should it be applied? For the example let's imagine 2 convolutional layers followed by 1 fully connected layer. "A2" are the…
sdiabr
  • 487
  • 1
  • 8
  • 23
5
votes
2 answers

How can I break down large grid searches?

I am looking to run a very large grid search for different neural network configurations. In its fullness this would be impracticable to run using my current hardware. I am aware that there may be superior techniques to a naive grid search (e.g.…
Kievit
  • 87
  • 10
5
votes
0 answers

How to train a model in keras with multiple input-output datasets with different batch sizes

I have a supervised learning problem that I am solving with the Keras functional API. As this model is predicting the state of a physical system, I know the supervised model should follow additional constraints. I would like to add that as an…
thorbjorn444
  • 236
  • 2
  • 5
5
votes
2 answers

Choosing the regularization parameter

When applying regularized logistic regression: I split my data into training, cross-validation and test sets. I want to apply regularization and am working on choosing the regularization parameter lambda. To do so, I try different values of lambda…
bigTree
  • 2,103
  • 6
  • 29
  • 45
5
votes
1 answer

How to set intercept_scaling in scikit-learn LogisticRegression

I am using scikit-learn's LogisticRegression object for regularized binary classification. I've read the documentation on intercept_scaling but I don't understand how to choose this value intelligently. The datasets look like this: 10-20 features,…
cxrodgers
  • 4,317
  • 2
  • 23
  • 29
5
votes
1 answer

R glmnet family = binomial predict values outside of 0-1

I'm trying to find a package in R for regularized logistic regression that predicts values between 0 - 1. I haven't had much luck though, having tried the lars package and now the glmnet package. Below is code from the example in the reference…
screechOwl
  • 27,310
  • 61
  • 158
  • 267
4
votes
1 answer

Tensorflow: How to apply a regularizer on a tensor?

I am implementing a model in Tensorflow 2, and I want to apply a penalization on a tensor (multiplication from two layers' outputs) in my model. I am used to use regularization on layers (kernel, bias or activity regularization). I could build a…
Loris Pilotto
  • 247
  • 1
  • 8
4
votes
2 answers

Constrain parameters to be -1, 0 or 1 in neural network in pytorch

I want to constrain the parameters of an intermediate layer in a neural network to prefer discrete values: -1, 0, or 1. The idea is to add a custom objective function that would increase the loss if the parameters take any other value. Note that, I…
Rakib
  • 7,435
  • 7
  • 29
  • 45
4
votes
2 answers

Different coefficients for L1-L2 regularisation

I'd like to regularise the weights of a network with both L1 and L2 regularisation. However, I can't find a way to vary the strength of the regularisations independently. The Keras documentation doesn't provide any information either. So, is there a…
Felix
  • 2,548
  • 19
  • 48
4
votes
1 answer

How does multiple target Ridge Regression work in scikit learn?

I am struggling to understand the following: Scikit-learn offers a multiple output version for Ridge Regression, simply by handing over a 2D array [n_samples, n_targets], but how is it…
user2933782
  • 61
  • 4
4
votes
1 answer

TensorFlow - reproducing results when using dropout

I am training a neural network using dropout regularization. I save the weights and biases the network is initialized with, so that I can repeat the experiment when I get good results. However, the use of dropout introduces some randomness in the…
rdv
  • 682
  • 2
  • 8
  • 19
4
votes
0 answers

Does scikit-learn support general Tikhonov regularization in its Ridge modules?

Using the notation from Wikipedia, it seems that the scikit-learn Ridge modules use a multiple of the identity matrix as the Tikhonov matrix Gamma. The Tikhonov matrix is therefore specified by a single value alpha. Doing this results in all…
jonthalpy
  • 1,042
  • 8
  • 22
4
votes
1 answer

In Keras what is the difference between weight_regularizer and activity_ regularizer

I understand regularization normally adds k*w^2 to the loss to penalize large weights. But in Keras there are two regularizer parameters - weight_regularizer and activity_ regularizer. What is the difference?
simon
  • 2,561
  • 16
  • 26
4
votes
1 answer

Orthogonal matching pursuit regression - am I using it wrong?

I am trying out this method as a regularized regression, as an alternative to lasso and elastic net. I have 40k data points and 40 features. Lasso selects 5 features, and orthogonal matching pursuit selects only 1. What could be causing this? Am I…
Baron Yugovich
  • 3,843
  • 12
  • 48
  • 76
1
2
3
…
12 13