1

I have two dataframes with multiple variables. Each dataframe belongs to one rater. I would like to calculate the interrater reliability (Cohen's Kappa) between the two dataframes.

For example:

Rater1 <- matrix(c(1,0,0,1,0,0,0,0,1),ncol=3,byrow=TRUE)
colnames(Rater1)<-c("V1","V2","V3")
Rater2 <- matrix(c(0,1,0,1,0,1,0,0,1),ncol=3,byrow=TRUE)
colnames(Rater2)<-c("V1","V2","V3")

It must have something to do with the 'IRR' package, but I really can't figure out how. Any help in the right direction is very much appreciated.

David Maij
  • 53
  • 8
  • In each data frame, you have 3 rows and 3 columns. Are the rows or columns the things being rated? If so, what is the other dimension? – Gladwell Apr 24 '17 at 18:09

1 Answers1

2

Using the data you provided, you can calculate the kappa for each variable with the following code:

for (dimension in 1:3) {
    v = paste0("V", dimension)
    print(irr::kappa2(cbind(Rater1[, v], Rater2[, v])))
}

You said you wanted the kappa between the two data frames however, which means we need to somehow collapse the data frames into two vectors. All you need to do is alter your definition of subjects to be a variable for anything being rated. You can essentially ignore the fact that the subjects are coming from the same source because you are interested in the the agreement between the raters (who are independent) not within the features of the things being rated (which are not independent).

irr::kappa2(cbind(matrix(Rater1), matrix(Rater2)))
Gladwell
  • 328
  • 3
  • 10
  • 1
    The first option works like a charm, thanks a lot Gladwell! The second solution does not seem to work (Error in kappa2(cbind(as.vector(D), as.vector(M))) : Number of raters exeeds 2. Try kappam.fleiss or kappam.light.), but the first solution is sufficient for my purposes. Thanks again. – David Maij Apr 25 '17 at 10:18
  • 1
    I did up-vote and select your answer Gladwell, but apparently it's not publicly available since I have less than 15 points. – David Maij Apr 25 '17 at 10:20
  • @DavidMaij try the edited version above. It's maybe useful to have one number represent the degree of agreement instead of a number per variable. – Gladwell Apr 25 '17 at 16:18