I would say the formula should be
response ~ trt + trt:seq:rep + (trt|subj:seq)
The key difference from your specification is that (trt|subj:seq)
is fitting something like a randomized-block or random-slopes model, where we are allowing for the variation of the treatment effect across subject/sequence combinations.
There are a few issues here that I ran into/noticed when trying to fit this model to simulated data:
- if we are fitting this with "modern" mixed-model approaches, there will be some parameters aliased between the treatment effect (
trt
) and the trt:seq:rep
term. In a method-of-moments approach this doesn't matter so much (because you never explicitly estimate the parameters), but it leads to complaints about rank-deficient models (which are ignorable if you know what you're doing ...).
- it seems wrong that the random effect
delta_{ij}
is given as having a mean of {mu_R,mu_T}
; this is redundant with the fixed effect mu_k
Obviously I could have something wrong or misunderstood something about the original model specification.
I might suggest that you try follow-up questions on the r-sig-mixed-models@r-project.org
mailing list, where there is a wide readership with broad expertise (i.e., more expertise on the subject of mixed models specifically than here on StackOverflow)