0

I'm fitting a custom model to binary classification data using RStan, and assessing it with the "loo" cross-validation package. I want to estimate the classification accuracy (i.e., % of out-of-sample choices predicted correctly), rather than the default ELPD estimate. I know there's descriptions of how to do this for regression models fit with BRMS, but I can't find any info on how to do this for a customized model fit with RStan. Is this possible to do with the loo package? Do I have to write a custom "_rng" function in my stan model to sample the posterior predictive distribution, or something like that?

Here's my Stan model, if it's helpful:

functions {
  real get_loglik(real inv_temp, row_vector weights, matrix option_diffs, row_vector choices) {
    row_vector[cols(option_diffs)] utilities = inv_temp * (weights * option_diffs);
    return sum(utilities .* choices - log1p_exp(utilities));
  }
}

data {
  int<lower=1> numChoices;
  int<lower=1> numAtts;    
  matrix[numAtts, numChoices] option_diffs; 
  row_vector[numChoices] choices; 
}

transformed data {
 real uniform_wts_prior = -0.6931472 * numAtts;
}

parameters {
  real<lower=0> inv_temp;
  row_vector<lower=-1, upper=1>[numAtts] weights;
}


model {
  target += gamma_lpdf(inv_temp | 4, 1) + uniform_wts_prior + get_loglik(inv_temp, weights, option_diffs, choices);
}

generated quantities {
  vector[numChoices] log_lik;
  row_vector[numChoices] utilities = inv_temp * (weights * option_diffs);
  for (n in 1:numChoices) {
    log_lik[n] = utilities[n] * choices[n] - log1p_exp(utilities[n]);
  }
}
Adam Morris
  • 293
  • 3
  • 8

0 Answers0