I'm currently running a ridge regression in R using the glmnet package, however, I recently ran into a new problem and was hoping for some help in interpreting my results. My data can be found here: https://www.dropbox.com/sh/hpxu3t0vqkrzfgf/AAB6F-yMYMfuI5E__gfDuW6sa?dl=0
My data consists of a 26531x428 observation matrix x and a 26531x1 response vector y. I am attempting to determine the optimal value of lambda.min, and when I run the code
> lambda=cv.glmnet(x=x,y=y,weights=weights,alpha=0,nfolds=10,standardize=FALSE)
I get
$lambda.min
[1] 2.123479
$lambda.1se
[1] 619.0054
which are results I would expect. However, I would like to add a slight tweak to this regression. I have prior knowledge of each of my 428 coefficients, and instead of shrinking each coefficient towards 0, as is the default with ridge regression, I would like to shrink each coefficient towards a specific value other than 0. After reaching out to Dr. Trevor Hastie, one of the creators of glmnet, he told me that this could be achieved by running the same code after substituting y with y2, where y2 = y - x%*%d and d is a 428x1 vector of coefficient priors. He said to then add d to my new coefficients, which would give me my prior-informed coefficients. After rerunning the code
> lambda=cv.glmnet(x=x,y=y2,weights=weights,alpha=0,nfolds=10,standardize=FALSE)
I unfortunately get
$lambda.min
[1] 220.3026
$lambda.1se
[1] 220.3026
The results of plot(lambda) look like this
Does anyone know why glmnet can't find a suitable lambda.min? Could it be because my vector of priors contains estimates that are too far off? Any help would be greatly appreciated!
