I'm fitting a custom model to binary classification data using RStan, and assessing it with the "loo" cross-validation package. I want to estimate the classification accuracy (i.e., % of out-of-sample choices predicted correctly), rather than the default ELPD estimate. I know there's descriptions of how to do this for regression models fit with BRMS, but I can't find any info on how to do this for a customized model fit with RStan. Is this possible to do with the loo package? Do I have to write a custom "_rng" function in my stan model to sample the posterior predictive distribution, or something like that?
Here's my Stan model, if it's helpful:
functions {
real get_loglik(real inv_temp, row_vector weights, matrix option_diffs, row_vector choices) {
row_vector[cols(option_diffs)] utilities = inv_temp * (weights * option_diffs);
return sum(utilities .* choices - log1p_exp(utilities));
}
}
data {
int<lower=1> numChoices;
int<lower=1> numAtts;
matrix[numAtts, numChoices] option_diffs;
row_vector[numChoices] choices;
}
transformed data {
real uniform_wts_prior = -0.6931472 * numAtts;
}
parameters {
real<lower=0> inv_temp;
row_vector<lower=-1, upper=1>[numAtts] weights;
}
model {
target += gamma_lpdf(inv_temp | 4, 1) + uniform_wts_prior + get_loglik(inv_temp, weights, option_diffs, choices);
}
generated quantities {
vector[numChoices] log_lik;
row_vector[numChoices] utilities = inv_temp * (weights * option_diffs);
for (n in 1:numChoices) {
log_lik[n] = utilities[n] * choices[n] - log1p_exp(utilities[n]);
}
}