Stack Overflow
Stack Overflow

Questions tagged [model-comparison]

A task in statistics and machine learning for comparing models that are fit to the same data with the aim of determining which best explains the data. Questions about how to use model comparision to evaluate models are likely more appropriate for CrossValidated (https://stats.stackexchange.com)!

Model comparison is a task in model selection involving the application of a measure (criterion) to the fits of a set of candidate models to data. The resulting measurements enable a quantitative comparison of appropriateness of the models in explaining the data.

See the Model Selection Wikipedia entry.

98 questions
2
votes
3 answers

Finding model (returned from for loops) with lowest AIC in R

I am trying to find model with lowest AIC. Models are returned from two for loops that make possible combinations of columns. I am unable to make the function return model with lowest AIC. The code below demonstrates where I got stuck: rm(list =…
New2coding
  • 715
  • 11
  • 23
2
votes
0 answers

AIC and PSSE comparison

Akaike Information Criterion (AIC) and Predictive sum of squares error (PSSE) are information theoretic and predictive capability assessment measures to rank the models. I was trying to rank the models based on these two criterions. However, the…
Lucky
  • 347
  • 6
  • 15
2
votes
0 answers

Stepwise regression with a huge matrix for y

Aim is to find the best model. I use stepwise regression for it. But I've got a problem. When I try this: library("MASS") foreach (i=1:20000) %do% { Ge <- bigGeneMatrix[i,] fit <- lm(Ge ~ geneA + geneB + geneC + geneD + ... + geneX +…
Melanie Julia
  • 83
  • 10
2
votes
1 answer

Sorting ARIMA AIC in R

I have the following code which returns the model with the lowest AIC, but I want all the models with their AIC in ascending or descending order without using the built-in sort function in R sp <- rnorm(100) ## just some toy data to make code…
Waqas
  • 329
  • 3
  • 16
2
votes
0 answers

Interpreting output of analysis of deviance table from anova() model comparison

I have a large multivariate abundance data and I am interested in comparing multiple models that fit different combinations of three categorical predictor variables to my species matrix response variable. I have been using anova() to compare my…
user2096647
  • 99
  • 1
  • 9
2
votes
1 answer

MUMIn pdredge error with glmer

I'm trying to use the parallel computing version of dredge (package MUMIn) for model selection of my full glmer model: modmer.pom.full<-glmer(cbind(TEST,CONTROL)~ G+MS+l+MS*l+G*MS+G*l+I(l^2) +…
jrs-x
  • 336
  • 1
  • 2
  • 10
2
votes
1 answer

Error in eval(expr, envir, enclos): no loop for break/next, jumping to top level

I am trying to run a stepwise analysis on a negative binomial model with many…
user3391456
  • 21
  • 2
2
votes
0 answers

AIC with weighted nonlinear regression (nls)

I encounter some discrepancies when comparing the deviance of a weighted and unweigthed model with the AIC values. A general example (from ‘nls’): DNase1 <- subset(DNase, Run == 1) fm1DNase1 <- nls(density ~ SSlogis(log(conc), Asym, xmid, scal),…
A.N. Spiess
  • 21
  • 1
  • 2
1
vote
2 answers

Comparing mixed models: singular fit for random effect group but only in some models. Should I drop RE group from all models or just where singular?

I am fitting several mixed models using the lmer function in the package lme4, each with the same fixed effects and random effects, but different response variables. The purpose of these models is to determine how environmental conditions influence…
Oscar Clendon
  • 31
  • 4
1
vote
1 answer

How to statistically compare the intercept and slope of two different linear regression models in Python?

I have two series of data as below. I want to create an OLS linear regression model for df1 and another OLS linear regression model for df2. And then statistically test if the y-intercepts of these two linear regression models are statistically…
KubiK888
  • 4,377
  • 14
  • 61
  • 115
1
vote
1 answer

Is there a way in R to determine AIC from cv.glmnet?

I am using the glmnet package in R, and not(!) the caret package for my binary ElasticNet regression. I have come to the point where I would like to compare models (e.g. lambda set to lambda.1se or lambda.min, and models where k-fold is set to 5 or…
Thomas
  • 441
  • 3
  • 16
1
vote
1 answer

Best function to compare caret model objects

I have a number of caret model objects using the same data and tuning parameters. For a sanity check I want to see if each method gives me the same model object. (This is all part of a broader plan to run parallel processing and ensure my models are…
JFG123
  • 577
  • 5
  • 13
1
vote
0 answers

non nested model comparison using R

To explain my problem , i have this simulated data using R. require(splines) x=rnorm(20 ,0,1) y=rep(c(0,1),times=10) First i fitted a regular (linear effects) logistic regression model. fit1=glm(y~x ,family = "binomial") Then to check the non…
student_R123
  • 962
  • 11
  • 30
1
vote
0 answers

how to select model if there are multiple models with same or close validation errors

Questions about model selection using cross-validation Let's say a dataset was split as training and testing sets. Multiple models were compared using cross-validation on training datasets. In one scenario, some of models yielded exact same…
derec
  • 101
  • 1
  • 9
1
vote
0 answers

Warning message: models are not all fitted to the same number of observations

I am fitting several models with a different structure. Example of each structure: > m1 <- glmer(X~Y1+Y2+Y3+(1|B/C),control=glmerControl(optimizer="bobyqa", optCtrl=list(maxfun=100000)), data = dataset, family=binomial(link="logit")) > > m2 <-…
mto23
  • 303
  • 1
  • 5
  • 15
1
2
3 4 5 6 7