0

I'm currently conducting a meta-analysis study. In my study, there are several studies that reported multiple outcomes. According to the literature, I figured out that the multilevel meta-analysis could be a good way to combine these multiple outcomes.

However, when I did the analysis, some problem occurred as I briefly summarized in my title. I used the metafor package in R and I'd like to use the tutorial example to explain. You can find the link of this tutorial below (I will attach essential detail in this question page):

https://www.metafor-project.org/doku.php/tips:forest_plot_with_aggregated_values

The problem is that in this example, while study 4, 5, and 8 only have one estimate, after aggregation, their confidence intervals expand and even expand from a significant level to a non-significant level.

Moreover, in the final aggregated forest plot, several studies' confidence intervals include 0, but the summary estimate doesn't include 0. Isn't it a bit counter-intuitive?

Since the specific mathematical process of the multi-level model and aggregate() function is a black box for me, I asked chatgpt why could that occur. It answered me that because studies that reported multiple outcomes are given more precision while studies that reported only one outcomes are considered less precise.

Firstly, I'm not sure whether chatgpt got the point. Secondly, I think most applied researchers don't know the specific process behind the multi-level model, either. So, could it be problematic when this counter-intuitive aggregated forest plot is inspected?

Wolfgang
  • 2,810
  • 2
  • 15
  • 29
Yang Travis
  • 3
  • 1

1 Answers1

0

The type of aggregations discussed at that link is a specific case where one would like to create an aggregated dataset (using information from the multilevel model) such that a simple meta-analysis of the aggregated values (using an equal-effects model) produces the same result as the multilevel model. This requires calculating the sampling variances of the aggregated values in such a way that they incorporate the heterogeneity stemming from the multilevel model. As a result, CIs get wider, also for the studies that only supply a single estimate to the model.

As for the fact that most CIs around the aggregated values include 0, while the CI around the summary estimate excludes 0: That is kind of the idea behind a meta-analysis. If individual estimates are imprecise, then the pooled estimate can still be sufficiently precise to achieve statistical significance, assuming that there is a consistent pattern/signal. In fact, 16 out of the 17 aggregated values are positive. That in itself has a very low probability of occurring under the null hypothesis (in which case the sign of each estimate should either be randomly positive or negative). A simple sign test of this can be obtained with prop.test(16, 17).

Wolfgang
  • 2,810
  • 2
  • 15
  • 29
  • Dear Wolfgang, thank you so much for your answer! I also realized the logic behind the discrepancy between individual estimates and the summary estimate. However, my real problem is that shall we use this kind of forest plot to represent our results? Since most studies' CIs in the forest plot are not their real CIs (while readers may expect to get "real" CIs from the forestplot). I also read several multilevel meta-analysis article and this kind of forest plot was unexpectedly never used in these articles. So, I'm not sure whether it is rational to use this plot. – Yang Travis May 31 '23 at 12:26
  • It could be the reason that I didn't exhaust all multilevel meta-analysis articles. I will also keep reading some other articles to validate this phenomenon. Your answer will also be very valuable for me and my team :) – Yang Travis May 31 '23 at 12:41
  • The rationale for showing such a forest plot is given on the page you linked to. Of course, one should explain to the reader that the plot is not showing the individual estimates but the aggregated values (and their corresponding CIs). – Wolfgang May 31 '23 at 18:44