How to test significant improvement of LRM model

Question

Using the rms package of Frank Harrell I constructed a predictive model using the lrm function.

I want to compare if this model has a significant better predictive value on a binomial event in comparison with another (lrm-) model.

I used different functions like anova(model1, model2) or the pR2 function of the pscl library to compare the pseudo R^2, but they all don't work with the lrm based model.

What can I do best to see if my new model is significant beter than the earlier model?

Update: Here is a example (where I want to predict the chance on bone metastasis) to check if size or stage (in addition to other variabele) gives the best model:

library(rms)
getHdata(prostate)

ddd <- datadist(prostate)
options( datadist = "ddd" )

mod1 = lrm(as.factor(bm) ~ age + sz + rx, data=prostate, x=TRUE, y=TRUE)

mod2 = lrm(as.factor(bm) ~ age + stage + rx, data=prostate, x=TRUE, y=TRUE)

You don't give an example to try, but broom::glance(model1) will give the log Likelihood and you can compare log likelihoods to see if there is an improvement. — Harlan Nelson, Feb 02 '18 at 14:54
For a predictive model, I'd strongly recommend fitting to a training set and using a test set to evaluate performance. Otherwise you will probably overfit. — Gregor Thomas, Feb 05 '18 at 14:23
Thanks for the suggestion, I certainly intend to do this. The code I gave was just a minimal working example. — Klaas J, Feb 06 '18 at 08:40

score 0 · Answer 1 · answered Apr 01 '18 at 04:30

0

It seems fundamentally the question is about comparing two non-nested models.

If you fit your models using the glm function you can use the -vuong- function in -pscl- package.

answered Apr 01 '18 at 04:30

Pavel Roshanov

43
8

score 0 · Answer 2 · answered Dec 25 '20 at 00:50

0

To test the fit of 2 nested models, you can use the lrtest function from the "rms" package.

lrtest(mod1,mod2)

answered Dec 25 '20 at 00:50

Paul Tran

1
1

note that the example by the OP does not show nested models ... – Ben Bolker Dec 25 '20 at 01:38

How to test significant improvement of LRM model

2 Answers2