I am currently working on a meta-analysis for my comprehensive exams and fit pretty much the same model you are talking about: I can have multiple effect sizes drawn from the same study. I would not fit a multilevel meta-analytic model using metafor, as it does not appropriately capture the correlated error terms. I wrote out my thinking in my paper (still working on it), so here is a rough explanation from my comps on how to address this situation:
I gathered $k = 240$ effect sizes across $m = 90$ studies. Table 2
describes the distribution across studies: Half of the studies
reported more than one effect size, with three studies reporting as
many as 15 effect sizes. Traditional meta-analytic methodologies
assume that all effect sizes are independent of one another; this
assumption is severely violated in the present analysis, as effect
sizes drawn from the same participants are dependent on one another.
One would ideally use a multivariate meta-analysis to model these
dependencies; however, this requires the meta-analyst to have access
to the full covariance matrix of all measures in all studies. This is
not realistic in many settings [@jackson2011multivariate],
particularly in the present meta-analysis of a literature where (a)
researchers hardly publish this information and (b) the research has
been published over the course of 70 years, making acquiring this
information from the authors of many of theses studies impossible.
Multilevel meta-analysis has been proposed as a way to deal with
unknown dependencies between effect sizes [@cheung2014modeling;
@konstantopoulos2011fixed; @van2013three]. While some argue that
individuals could be modeled at Level 1, effect sizes at Level 2, and
study at Level 3 [e.g., @cheung2014modeling], three-level
meta-analyses still assume that residual errors are orthogonal within
clusters [@tanner2016handling]. This assumption is violated when
multiple effect sizes are drawn from the same participants.
The current “state-of-the-art” [@polanin2017review] way to model these
dependencies and avoid underestimating standard errors is to use
robust variance estimates [RVE; @hedges2010robust;
@tanner2016handling]. I performed my meta-analysis using RVE for
correlated effects in the robumeta R package [@fisher2015robumeta].
As mentioned above, I am able to calculate the variances of effect
sizes directly from sample size, but I am not able to calculate the
covariance between effect sizes. RVE solves this problem by using the
cross products of the residuals for each study to estimate the
variance-covariance matrix of effect sizes within a study. While the
estimate of the covariance matrix in each study is not very good, the
combined variance estimate converges to the true variance as the
number of studies approaches infinity [@hedges2010robust].
Traditional meta-analyses weight effect sizes by using the inverse of
the variance. RVE weights each effect size using (a) the inverse of
the average variance across all effect sizes in a study (assuming a
constant correlation across effect sizes) (b) divided by the number of
effect sizes in the study. This ensures that a study does not get
"extra" weight simply by having more effect sizes.
This method is used primarily for the purposes of this meta-analysis:
interpreting meta-regression coefficients. The variance estimates
found in other meta-analyses (e.g., $Q, I^2, \tau^2$) are not precise
when using RVE. Given this shortcoming of RVE—and my main focus in
interpreting meta-regression coefficients—I will not focus on these
estimates.
References (from my .bib file, sorry if the format is annoying):
@article{jackson2011multivariate,
title={Multivariate meta-analysis: Potential and promise},
author={Jackson, Dan and Riley, Richard and White, Ian R},
journal={Statistics in Medicine},
volume={30},
number={20},
pages={2481--2498},
year={2011},
publisher={Wiley Online Library}
}
@article{cheung2014modeling,
title={Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach},
author={Cheung, Mike W L},
journal={Psychological Methods},
volume={19},
number={2},
pages={211--229},
year={2014}
}
@article{konstantopoulos2011fixed,
title={Fixed effects and variance components estimation in three-level meta-analysis},
author={Konstantopoulos, Spyros},
journal={Research Synthesis Methods},
volume={2},
number={1},
pages={61--76},
year={2011},
publisher={Wiley Online Library}
}
@article{van2013three,
title={Three-level meta-analysis of dependent effect sizes},
author={Van den Noortgate, Wim and L{\'o}pez-L{\'o}pez, Jos{\'e} Antonio and Mar{\'\i}n-Mart{\'\i}nez, Fulgencio and S{\'a}nchez-Meca, Julio},
journal={Behavior Research Methods},
volume={45},
number={2},
pages={576--594},
year={2013},
publisher={Springer}
}
@article{tanner2016handling,
title={Handling complex meta-analytic data structures using robust variance estimates: A tutorial in {R}},
author={Tanner-Smith, Emily E and Tipton, Elizabeth and Polanin, Joshua R},
journal={Journal of Developmental and Life-Course Criminology},
volume={2},
number={1},
pages={85--112},
year={2016},
publisher={Springer}
}
@article{polanin2017review,
title={A Review of Meta-Analysis Packages in {R}},
author={Polanin, Joshua R and Hennessy, Emily A and Tanner-Smith, Emily E},
journal={Journal of Educational and Behavioral Statistics},
volume={42},
number={2},
pages={206--242},
year={2017},
publisher={SAGE Publications Sage CA: Los Angeles, CA}
}
@article{hedges2010robust,
title={Robust variance estimation in meta-regression with dependent effect size estimates},
author={Hedges, Leon V and Tipton, Elizabeth and Johnson, Matthew C},
journal={Research synthesis methods},
volume={1},
number={1},
pages={39--65},
year={2010}
}
@article{fisher2015robumeta,
title={robumeta: An {R}-package for robust variance estimation in meta-analysis},
author={Fisher, Zachary and Tipton, Elizabeth},
journal={arXiv preprint arXiv:1503.02220},
year={2015}
}
