R: ICC (psych) function with multilevel organization

Question

I am trying to apply the ICC(2,k)function to assess the reliability of a new machine. Any help is appreciated!

• There was only 1 rater throughout the experiment
• 14 people were scanned monocularly
• They were scanned 3 times per session (3 traces/eye)
• They came back 3 times in the same day (9 traces/eye in a day)
• They came back 3 days in the same week (27 traces/eye in a week)

The ICC function internally uses lmer to fit a random effects model. Specifically, it fits the model lmer(y~1 + (1|samples) + (1|repeats)). When I run the ICC code for the log values of my outcome variable, I get the following output:

I interpret the number of judges as the number of repeats (3 traces per session). However, it is treating each row as an independent sample/subject (226 total), when many of them belong to the same person, as indicated by the variable ID.

I am unsure how to to make the ICC function realize there are 14 people rather than 226 and calculate the ICC accordingly. Should I reformat my data? I am not sure how to do it while conserving the hierarchy.

I will be posting to cross-validated as well. I have been using this post to guide my thought process.

This is my data:

structure(list(ID = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 6L, 6L, 6L, 6L, 
6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 7L, 7L, 
7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 8L, 
8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 
9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 12L, 12L, 
12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 
12L, 12L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 
13L, 13L, 13L, 13L, 14L, 14L, 14L, 14L, 14L, 14L, 14L, 14L, 14L, 
14L, 14L, 14L, 14L, 14L, 14L, 14L, 14L, 14L), .Label = c("S01", 
"S02", "S03", "S04", "S05", "S06", "S07", "S08", "S09", "S10", 
"S13", "S15", "S16", "S17"), class = "factor"), eye = c("L", 
"L", "L", "L", "L", "L", "L", "L", "L", "R", "R", "R", "R", "R", 
"R", "R", "R", "R", "L", "L", "L", "L", "L", "L", "L", "L", "L", 
"R", "R", "R", "R", "R", "R", "R", "R", "L", "L", "L", "L", "L", 
"L", "L", "L", "L", "R", "R", "R", "R", "R", "R", "R", "L", "L", 
"L", "L", "L", "L", "L", "L", "L", "R", "R", "R", "R", "R", "R", 
"R", "R", "L", "L", "L", "L", "L", "L", "L", "L", "R", "R", "R", 
"R", "R", "R", "R", "R", "L", "L", "L", "L", "L", "L", "L", "L", 
"L", "R", "R", "R", "R", "R", "R", "R", "R", "R", "L", "L", "L", 
"L", "L", "L", "L", "L", "L", "R", "R", "R", "R", "R", "R", "R", 
"R", "L", "L", "L", "L", "L", "L", "L", "L", "R", "R", "R", "R", 
"R", "R", "R", "R", "R", "L", "L", "L", "L", "L", "L", "L", "R", 
"R", "R", "R", "R", "R", "R", "R", "L", "L", "L", "L", "L", "L", 
"L", "L", "L", "R", "R", "R", "R", "R", "R", "R", "R", "R", "R", 
"R", "R", "R", "R", "R", "R", "L", "L", "L", "L", "L", "L", "L", 
"L", "R", "R", "R", "R", "R", "R", "R", "R", "R", "L", "L", "L", 
"L", "L", "L", "L", "L", "R", "R", "R", "R", "R", "R", "R", "L", 
"L", "L", "L", "L", "L", "L", "L", "L", "R", "R", "R", "R", "R", 
"R", "R", "R", "R"), Day = c(1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 
1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 2, 2, 2, 
3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 2, 2, 3, 3, 3, 1, 1, 
1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 1, 1, 1, 2, 2, 3, 
3, 3, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 
1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 
2, 3, 3, 1, 1, 1, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 
1, 1, 2, 2, 2, 3, 1, 1, 1, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 
3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 
2, 2, 2, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 3, 3, 
3, 1, 1, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 
2, 2, 3, 3, 3), Time = c(1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 
1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 1, 2, 3, 1, 
2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 2, 3, 1, 2, 3, 1, 2, 3, 
1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 1, 2, 3, 1, 3, 1, 2, 
3, 1, 2, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 
1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 
2, 3, 1, 2, 3, 1, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 
3, 1, 2, 3, 2, 1, 2, 3, 1, 2, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 
3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 
2, 3, 1, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 2, 3, 1, 2, 3, 
1, 3, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 
3, 1, 2, 3), Video1 = c(0.087980907, 0.138229468, 0.087265621, 
0.195986087, NA, 0.169965326, 0.087600668, NA, NA, 0.092477223, 
0.091350528, 0.032918043, 0.152168109, 0.099289976, 0.073414486, 
0.55374223, 0.145585337, 0.183976925, 0.036795848, 0.181787784, 
0.056040284, 0.118583723, 0.077361293, 0.102967699, 0.160763961, 
0.124360548, 0.149915938, 0.179175193, 0.071730328, 0.149968128, 
0.153503167, 0.095037811, 0.075409002, 0.174832967, 0.141088859, 
0.038881157, 0.061564117, 0.054163262, 0.046867349, 0.040123011, 
0.054647735, 0.060849565, 0.074817049, 0.047454334, 0.087318019, 
0.039463224, 0.079722026, 0.052817568, 0.021839778, 0.059919709, 
0.090052431, 0.180300519, 0.160604452, 0.287542322, 0.200999919, 
0.20508116, 0.083250047, 0.157708361, 0.284022821, 0.136359313, 
0.086814307, 0.16719986, 0.137407529, 0.102210657, 0.168310974, 
0.161919932, 0.130113124, 0.107775902, 0.054302189, 0.060560227, 
0.053947741, 0.062641303, 0.052143943, 0.071859386, 0.068697917, 
0.074580461, 0.062634558, 0.052877909, 0.045341566, 0.036891534, 
0.0519817, 0.068383676, 0.072422589, 0.065883961, NA, NA, NA, 
0.18704615, 0.064510084, NA, NA, NA, 0.281343398, NA, NA, NA, 
NA, NA, NA, NA, NA, 0.132005827, 0.166076829, 0.156412738, 0.124993123, 
0.153797119, 0.23523115, 0.204244576, 0.179591735, 0.241662374, 
0.206093404, 0.194222736, NA, 0.160341745, 0.228276279, NA, 0.148244217, 
0.267088919, 0.142376066, 0.120874336, 0.116038161, 0.07570081, 
0.092018565, 0.18764035, 0.097570968, 0.275982193, 0.083032991, 
0.076835039, 0.129962405, 0.039216406, 0.092339901, 0.09752681, 
0.164013828, 0.086334678, 0.107876919, 0.126481021, 0.147617556, 
0.082634062, 0.054770826, 0.028749862, 0.143956984, 0.051684409, 
0.221550722, 0.233068936, 0.151027184, 0.162539415, 0.161836479, 
0.164355007, 0.130113218, 0.094768854, 0.12844588, 0.068543029, 
0.065301438, 0.158409656, 0.043416089, 0.088338498, 0.135535504, 
0.104713826, 0.055394193, 0.067243652, 0.101152186, 0.054197084, 
0.090209342, 0.10545278, 0.074863354, 0.100904934, 0.117767968, 
0.095591052, 0.105648135, 0.031272578, 0.288074114, 0.134108567, 
0.239842391, 0.089574413, 0.112689104, 0.113394894, NA, NA, 0.154518739, 
0.324720078, 0.119985361, NA, 0.144774891, NA, NA, 0.453636633, 
0.490269309, NA, 0.082747081, 0.057837022, 0.133037941, NA, 0.165724098, 
0.229381388, 0.209086406, 0.101479891, 0.205932066, 0.234497882, 
0.079760758, 0.181086985, 0.140277599, 0.038782905, 0.127672401, 
0.220278395, 0.218946954, 0.112720006, 0.155244665, 0.171597108, 
0.086627495, 0.04037711, 0.031625864, 0.083406063, 0.044853662, 
0.070224764, 0.074856116, 0.1314991, 0.094837903, 0.062767123, 
0.04305756, 0.020832593, 0.050751347, 0.058208723, 0.114863961, 
0.07299349, 0.065228472, 0.042690128), Video2 = c(0.114125673, 
NA, 0.060309691, 0.164117324, 0.060891372, 0.41365488, 0.018451836, 
0.158369521, NA, 0.098791936, 0.035309485, 0.077134396, NA, 0.062498159, 
0.115475734, 0.08324786, NA, 0.124140619, 0.118365888, 0.117836949, 
0.047061138, 0.087292064, 0.08407374, 0.092185632, 0.090598798, 
0.200259434, 0.113220122, 0.048218976, 0.036608844, 0.118584237, 
0.125454968, 0.101731714, 0.084789024, 0.158389537, 0.210365439, 
0.062385049, 0.065149087, 0.281668702, 0.054592426, 0.046126902, 
0.087065465, 0.072164099, 0.088374659, 0.085297405, 0.048997511, 
0.100124277, 0.040702851, 0.080152996, 0.025137424, 0.078181594, 
0.057469833, 0.247824038, 0.115295019, 0.166448579, 0.137513527, 
0.186291366, 0.115566612, 0.168936072, 0.167011681, 0.197827763, 
0.184426189, 0.139032114, 0.15280477, 0.133926332, 0.165605902, 
0.162007503, 0.078905359, 0.146775916, 0.048355623, 0.049236469, 
0.074249201, 0.042370443, 0.040626626, 0.094437822, 0.046528066, 
0.05148911, 0.060980887, 0.060188385, 0.077146076, 0.053584377, 
0.063219362, 0.050217884, 0.060380718, 0.077166513, NA, 0.417317819, 
NA, NA, 0.050967868, NA, NA, 0.171600774, 0.075110846, 0.407966882, 
NA, 0.317172879, 0.14913279, 0.181668808, NA, 0.209085025, NA, 
NA, 0.181146078, 0.194934388, 0.147458219, 0.20603389, 0.159845411, 
0.167280953, 0.184498482, NA, 0.116541941, 0.203738519, 0.165296042, 
0.188392641, 0.213356813, 0.163580836, 0.16583046, 0.278539249, 
0.176512754, 0.189747006, 0.127297963, 0.097622766, 0.14965216, 
0.138453154, 0.093268444, 0.194231674, 0.109519249, 0.080222972, 
0.107647036, 0.074088679, 0.080116949, 0.087413981, 0.135143153, 
0.080069624, 0.111665621, 0.07615619, 0.123522342, 0.111357589, 
0.291819884, 0.102793506, 0.129541771, 0.077439141, 0.211007708, 
0.112699582, 0.238495917, 0.048760198, 0.10311381, 0.058783217, 
0.187679402, 0.105958326, 0.098303641, 0.080843973, 0.052070696, 
0.119010333, 0.077476701, 0.131770602, 0.088461915, 0.072832336, 
0.036611459, 0.074172134, NA, 0.03982938, 0.166489651, 0.101680555, 
0.100558381, 0.073767248, 0.082808278, 0.119402936, 0.136595719, 
0.216845566, 0.296923823, 0.228383844, 0.162111224, 0.231735943, 
0.09593012, 0.13931631, 0.092081041, 0.343116293, 0.107501464, 
NA, 0.091976987, 0.121355731, NA, NA, 0.143166393, NA, 0.206475454, 
0.183337007, NA, 0.07489074, NA, 0.151679047, 0.18253156, 0.092401174, 
0.249826082, NA, NA, 0.388233583, 0.238975608, 0.165197158, 0.03402928, 
0.107258374, 0.135504625, 0.199937322, 0.386272593, 0.282663422, 
0.079703991, 0.084003722, 0.053736181, 0.065844789, 0.035722026, 
0.070159213, 0.045913757, 0.068252653, 0.092070417, 0.050934179, 
0.053299503, 0.07119216, 0.091180257, 0.081479255, 0.061368078, 
0.079272001, 0.041578686, 0.063021913, 0.044307119, 0.052951682
), Video3 = c(0.144167749, 0.053938585, 0.074156272, 0.068384241, 
0.257212592, 0.243452094, 0.072577209, 0.063810703, 0.147408183, 
0.059354697, 0.04634975, 0.07846023, 0.196462309, 0.099933835, 
0.145683333, 0.158305327, 0.105569594, 0.060838186, 0.102196312, 
0.057709436, 0.065818121, 0.066244457, 0.105116003, 0.054189254, 
0.205065965, 0.163002871, 0.180032663, 0.051182716, 0.102318964, 
0.245352358, 0.145898239, 0.186285454, 0.095095281, 0.156748828, 
0.15570742, 0.053803903, 0.033606179, 0.07365876, 0.05104256, 
NA, 0.053655528, 0.065751871, 0.082836706, 0.080429497, 0.039112058, 
0.06394257, 0.046803938, 0.092834672, 0.034565093, 0.048813169, 
0.095532111, NA, 0.212517948, 0.227250158, 0.183296206, 0.154160041, 
0.225198708, 0.271158885, 0.039183658, 0.199951527, NA, 0.177792295, 
0.110744145, 0.199889426, 0.106507314, 0.215298426, 0.099930914, 
NA, 0.040638466, 0.055748963, 0.0884863, 0.041390095, 0.043243241, 
0.13304215, 0.065264553, 0.064561766, 0.047361205, 0.069268582, 
0.061012224, 0.079270672, 0.037756581, 0.091278191, 0.07024436, 
0.061057298, NA, NA, NA, NA, NA, 0.087830029, 0.255102736, NA, 
NA, 0.447484104, NA, NA, NA, NA, NA, NA, 0.593477276, NA, 0.18095392, 
0.192736027, 0.198110005, 0.17373631, 0.190351037, 0.183725822, 
0.201399389, 0.197716107, 0.182643112, 0.169019664, 0.184499611, 
0.15968721, 0.198214947, 0.138778453, 0.259120036, 0.254541772, 
NA, 0.138387786, 0.132525415, 0.095001906, 0.110843691, 0.157754136, 
0.099714141, 0.259360142, 0.108598563, 0.090730258, 0.08742319, 
0.076818785, NA, NA, 0.120915331, 0.112465074, 0.109784663, 0.071975707, 
0.236083514, 0.060273625, NA, NA, NA, 0.122126069, NA, 0.223049768, 
NA, 0.148602074, 0.15451306, 0.081655106, 0.08779288, 0.085110963, 
0.182273997, 0.079916129, NA, 0.07299758, 0.049128115, 0.101086955, 
NA, 0.370434594, 0.045494486, 0.05161485, 0.056470279, 0.075544203, 
0.074006203, 0.073745287, 0.066680245, 0.111868223, 0.128480808, 
0.115794281, 0.137204024, NA, NA, NA, 0.123755171, NA, NA, NA, 
NA, 0.209945869, 0.158594182, 0.192175776, NA, NA, 0.084841772, 
0.037790634, 0.110958609, 0.14481606, NA, NA, 0.115507315, 0.008844521, 
NA, 0.039626847, NA, 0.140239324, NA, 0.147622419, 0.130914464, 
0.19425935, 0.029198683, 0.123614597, 0.122745111, NA, 0.087697701, 
0.37026571, NA, 0.21681181, 0.110354032, NA, 0.059406045, 0.063982092, 
0.050064775, 0.116552375, NA, NA, 0.106103895, 0.027066922, 0.083329503, 
0.061009608, 0.072162348, NA, 0.110920872, NA, NA, NA, NA, 0.092201503
)), row.names = c(NA, -226L), class = "data.frame")

Answer 1

If you are looking at the clustering within the three "Video" columns as an outcome variable then that need to be represented in a long format. I can show you an example. Be aware that non-convergence leads to erroneous estimates. Simply stated, they cannot be used at all. Failure to converge means that the algorithm within lme4 cannot find reasonable estimates because there is not enough data on each level to do so. There is a lot of info on convergence if you google it.

df <- df %>% 
  pivot_longer(cols = 5:7)

df

# Converge ok
lmer(log(value) ~1 + (1|Day) + (1|ID),df)

# Day (random effect) explains 1.6 % of the variation
# ID (random effect) explains 38.2 % of the variation