Discripencies in variable importance calculation for glmnet model in R

Question

I want to calculate variable importance for glmnet model in R. I am using glmnet package for fitting the elastic net model like

library(glmnet)
library(caret)
library(vip)

data_y <- as.vector(mtcars$mpg)
data_x <- as.matrix(mtcars[-1])

fit.glmnet <- glmnet(data_x, data_y, family="gaussian")

set.seed(123)
cvfit.glmnet = cv.glmnet(data_x, data_y, standardize=T)
cvfit.glmnet$lambda.min
coef(cvfit.glmnet, s = "lambda.min")

Then I have used vip package for variable importance as

#Using vip package
vip::vi_model(cvfit.glmnet, s = cvfit.glmnet$fit$lambda)

which returns me

># A tibble: 10 x 3
   Variable Importance Sign 
   <chr>         <dbl> <chr>
 1 cyl         -0.886  NEG  
 2 disp         0      NEG  
 3 hp          -0.0117 NEG  
 4 drat         0      NEG  
 5 wt          -2.71   NEG  
 6 qsec         0      NEG  
 7 vs           0      NEG  
 8 am           0      NEG  
 9 gear         0      NEG  
10 carb         0      NEG

The variable importance contains both positive and negative values for the variables at the same time it does not vary between 0-1 or 0-100%.

Then I have tried customised function from this answer

#Using function provided in this example
varImp <- function(object, lambda = NULL, ...) {
  
  ## skipping a few lines
  
  beta <- predict(object, s = lambda, type = "coef")
  if(is.list(beta)) {
    out <- do.call("cbind", lapply(beta, function(x) x[,1]))
    out <- as.data.frame(out)
  } else out <- data.frame(Overall = beta[,1])
  out <- abs(out[rownames(out) != "(Intercept)",,drop = FALSE])
  out
}

varImp(cvfit.glmnet, lambda = cvfit.glmnet$lambda.min)

It returns me following output

        Overall
cyl  0.88608541
disp 0.00000000
hp   0.01168438
drat 0.00000000
wt   2.70814703
qsec 0.00000000
vs   0.00000000
am   0.00000000
gear 0.00000000
carb 0.00000000

Though the output from customised function does not contain negative values, it does vary within 0-1 or 0-100%.

I know that caret package has varImpfunction which gives variable importance between 0-100%. But I want to implement the same thing for cv.glmnet object instead of caret::train object. How can I achieve the variable importance alike caret package for cv.glmnet object?

What would you say are variables more important if they have a higher coefficient at a certain lambda or if their coefficient reaches 0 at higher L1 penalty? — missuse, Sep 21 '20 at 09:46
I think the variables which are having a higher coefficient at a certain lambda, are more important. — UseR10085, Sep 21 '20 at 09:49
Why did you not call `vip::vi_model(cvfit.glmnet, s = cvfit.glmnet$lambda.min)`? as in the call `varImp(cvfit.glmnet, lambda = cvfit.glmnet$lambda.min)`. I get the same values with the difference `varImp` returns the absolute values of the coefficients. — missuse, Sep 21 '20 at 09:51
Even after using `vip::vi_model(cvfit.glmnet, s = cvfit.glmnet$lambda.min)`, I am getting different results only. — UseR10085, Sep 21 '20 at 09:58
Could you update the question with the results you are getting? — missuse, Sep 21 '20 at 10:16
These are the same results, only the `vip::vi_model` are rounded at three digits while the absolute values of coefficients is given by `varImp` — missuse, Sep 21 '20 at 10:23
Ooo yes, those are same only. But how can I achieve the variable importance alike `caret` package for `cv.glmnet` object i.e. 0-100 or 0-1? — UseR10085, Sep 21 '20 at 10:28
just rescale the `varImp` to 0-1. This can be achieved by dividing by the maximum. — missuse, Sep 21 '20 at 10:31

missuse · Accepted Answer · 2020-09-21T10:47:41.620

1

The question asks how to obtain glmnet variable importance between 0-100%.

If it is desired to assign importance based on coefficient magnitude at a certain (usually optimal) penalty. And if these coefficients are derived based on standardized variables (default in glmnet) then the coefficients can simply be scaled to the 0 - 1 range:

The slightly modified function is given:

varImp <- function(object, lambda = NULL, ...) {
  beta <- predict(object, s = lambda, type = "coef")
  if(is.list(beta)) {
    out <- do.call("cbind", lapply(beta, function(x) x[,1]))
    out <- as.data.frame(out)
  } else out <- data.frame(Overall = beta[,1])
  out <- abs(out[rownames(out) != "(Intercept)",,drop = FALSE])
  out <- out/max(out)
  out[order(out$Overall, decreasing = TRUE),,drop=FALSE]
}

Using the example in the question:

varImp(cvfit.glmnet, lambda = cvfit.glmnet$lambda.min)
#output
         Overall
wt   1.000000000
cyl  0.320796270
am   0.004840186
hp   0.004605913
disp 0.000000000
drat 0.000000000
qsec 0.000000000
vs   0.000000000
gear 0.000000000
carb 0.000000000

Another approach at assigning variable importance to glmnet models would be scoring the variables based on the penalty for inclusion - Variables are more significant if the are excluded at higher penalties. This approach will be implemented in the mlr3 package: https://github.com/mlr-org/mlr3learners/issues/28 at some point

edited Sep 21 '20 at 10:47

answered Sep 21 '20 at 10:39

missuse

19,056
3
25
47

Thank you very, can you provide the other approach you have mentioned using `mlr3` package? – UseR10085 Sep 21 '20 at 10:43
It looks I was mistaken and it is yet to be implemented: https://github.com/mlr-org/mlr3learners/issues/28 – missuse Sep 21 '20 at 10:47
Ok, the function `varImp` is basically taken from `caret` package. While we implement it using `caret` package, we get the scaled importance but when it is used as mentioned in my question, it does not provide the scaled importance. Can you tell me why this is happening? – UseR10085 Sep 21 '20 at 10:50
It appears that there is an additional processing of the output from the varImp function where it is scaled to 0 - 1 range for all learners that have importance slots in caret. Consider these examples: `caret::varImp(test_class_cv_model, lambda = 0.007)` and `caret::varImp(test_class_cv_model$finalModel, lambda = 0.007)` If you call it on the gimlet model you get unscaled importance while if you call in on the train object you get scaled importance. You can check what happens additionally in the call on the train object via `caret:::varImp.train` – missuse Sep 21 '20 at 11:05

Discripencies in variable importance calculation for glmnet model in R

1 Answers1

Linked