I am conducting a multilevel meta-analysis to evaluate vaccine effectiveness (VE) across several longitudinal studies. Each study reports VE for some time points. Multilevel meta-analyses with similar packages in R (metafor and mixmeta) show different results. By adding variables, the instability in the results increases. Now I am unsure if I can enter all my moderators into the model at the same time when the results are not the same through the same packages.
Thank you very much in advance. Kind regards Farideh
as an example of multilevel analysis without a moderator:
> library(haven)
> dat <- read_dta("3.08.01/test.dta")
> is.miss<-is.na(dat$time)
> dat<-dat[!is.miss,]
> dat1=dat[dat$variant==1,]
> dat2=dat[dat$variant==2,]
> ## model without moderator
> mixmeta1 <- mixmeta(y1~0+factor(time), S=v1, random=~time|id,
+ bscov="har1", data=dat1)
> print(summary(mixmeta1), digits=2, report="var")
Call: mixmeta(formula = y1 ~ 0 + factor(time), S = v1, data = dat1,
random = ~time | id, bscov = "har1")
Univariate extended random-effects meta-regression
Dimension: 1
Estimation method: REML
Fixed-effects coefficients
Estimate Std. z Pr(>|z|) 95%ci.lb 95%ci.ub
Error
factor(time)1 -1.31 0.22 -5.85 0.00 -1.74 -0.87 ***
factor(time)4 -1.44 0.20 -7.38 0.00 -1.83 -1.06 ***
factor(time)8 -1.23 0.16 -7.74 0.00 -1.54 -0.92 ***
factor(time)12 -1.02 0.12 -8.16 0.00 -1.26 -0.77 ***
factor(time)16 -0.77 0.09 -8.34 0.00 -0.95 -0.59 ***
factor(time)20 -0.53 0.07 -7.10 0.00 -0.67 -0.38 ***
factor(time)24 -0.51 0.07 -7.74 0.00 -0.64 -0.38 ***
factor(time)25 -0.36 0.06 -5.80 0.00 -0.48 -0.24 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Random-effects (co)variance components
Formula: ~time | id
Structure: Heterogeneous autoregressive of first order
Var Corr
(Intercept) 0.89 (Intercept)
time 0.00 -0.98
Univariate Cochran Q-test for residual heterogeneity:
Q = 25295.06 (df = 98), p-value = 0.00
I-square statistic = 99.6%
106 units, 1 outcome, 106 observations, 8 fixed and 3 random-effects parameters
(54 units removed due to missingness)
logLik AIC BIC
-4112.87 8247.73 8276.17
> newdata <- data.frame(time=c(1,4,8,12,16,20,24,25))
> predict=1-exp(predict(mixmeta1,newdata,ci=TRUE))
> predict
fit ci.lb ci.ub
1 0.7292053 0.8252245 0.5804346
2 0.7636741 0.8389221 0.6532740
3 0.7084295 0.7866307 0.6015671
4 0.6377852 0.7161588 0.5377712
5 0.5377306 0.6144175 0.4457917
6 0.4096321 0.4896154 0.3171145
7 0.4020644 0.4750902 0.3188792
8 0.3024586 0.3823891 0.2121836
> metafor1<-rma.mv(y1, v1, mods= ~as.factor(time)-1, random = ~time|id,
+ struct = "HAR", data=dat1)
> summary(metafor1)
Multivariate Meta-Analysis Model (k = 126; method: REML)
logLik Deviance AIC BIC AICc
-4600.5657 9201.1314 9235.1314 9282.2330 9241.2514
Variance Components:
outer factor: id (nlvls = 27)
inner factor: time (nlvls = 8)
estim sqrt k.lvl fixed level
tau^2.1 0.0910 0.3017 8 no 1
tau^2.2 0.0614 0.2477 19 no 4
tau^2.3 0.0494 0.2222 30 no 8
tau^2.4 0.1229 0.3506 38 no 12
tau^2.5 0.0448 0.2116 9 no 16
tau^2.6 0.0080 0.0896 6 no 20
tau^2.7 0.0759 0.2754 8 no 24
tau^2.8 0.0031 0.0553 8 no 25
rho 0.9300 no
Test for Residual Heterogeneity:
QE(df = 118) = 26670.4108, p-val < .0001
Test of Moderators (coefficients 1:8):
QM(df = 8) = 287.5490, p-val < .0001
Model Results:
estimate se zval pval ci.lb ci.ub
as.factor(time)1 -0.7962 0.1104 -7.2144 <.0001 -1.0125 -0.5799 ***
as.factor(time)4 -0.8776 0.0665 -13.1921 <.0001 -1.0080 -0.7472 ***
as.factor(time)8 -0.7173 0.0512 -14.0188 <.0001 -0.8175 -0.6170 ***
as.factor(time)12 -0.6372 0.0769 -8.2887 <.0001 -0.7879 -0.4865 ***
as.factor(time)16 -0.4714 0.0547 -8.6135 <.0001 -0.5786 -0.3641 ***
as.factor(time)20 -0.3198 0.0301 -10.6198 <.0001 -0.3788 -0.2608 ***
as.factor(time)24 -0.3176 0.1023 -3.1036 0.0019 -0.5182 -0.1170 **
as.factor(time)25 -0.1898 0.0225 -8.4504 <.0001 -0.2338 -0.1458 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> predict(metafor1, newmods=diag(8),intercept=FALSE, transf= exp, digits=3)
pred ci.lb ci.ub pi.lb pi.ub tau2.level
1 0.451 0.363 0.560 NA NA NA
2 0.416 0.365 0.474 NA NA NA
3 0.488 0.442 0.540 NA NA NA
4 0.529 0.455 0.615 NA NA NA
5 0.624 0.561 0.695 NA NA NA
6 0.726 0.685 0.770 NA NA NA
7 0.728 0.596 0.890 NA NA NA
8 0.827 0.792 0.864 NA NA NA
# dataset
id Stud-dsn variant age Prev-infect rob y1(lnor) v1(selnor) time
65 1 1 1 0 2 -0.76000 0.00920 4
41 1 2 1 1 1 -2.70000 0.13000 8
88 1 1 1 1 -0.78000 0.02300 4
88 1 2 1 1 -1.20000 0.11000 4
59 2 1 1 0 1 -0.65000 0.00005 12
95 2 1 1 0 1 -0.67000 0.00051 12
59 2 2 1 0 1 -2.10000 0.00170 12
59 2 1 1 0 1 -0.41000 0.00014 12
95 2 1 1 0 1 -0.36000 0.00016 12
11 2 1 1 0 1 -0.34000 0.00045 12
95 2 2 1 0 1 -2.00000 0.00056 12
36 2 2 1 0 1 -0.30000 0.00380 1
151 2 1 1 0 1 -0.46000 0.00190 25
151 2 1 1 0 1 -0.84000 0.00041 8
151 2 1 1 0 1 -1.20000 0.00099 4
59 2 2 1 0 1 -2.00000 0.00056 12
11 2 1 1 0 1 -0.57000 0.00190 8
95 2 1 1 0 1 -0.41000 0.00014 12
59 2 1 1 0 1 -0.75000 0.00021 12
11 2 1 1 0 1 -0.20000 0.00035 16
151 2 1 1 0 1 -0.47000 0.00003 4
59 2 1 1 0 1 -0.71000 0.00003 12
151 2 1 1 0 1 -0.72000 0.00085 8
95 2 2 1 0 1 -1.40000 0.00036 12
59 2 2 1 0 1 -1.40000 0.00036 12
151 2 1 1 0 1 -0.64000 0.00054 8
11 2 1 1 0 1 -0.17000 0.00048 20
151 2 1 1 0 1 -0.64000 0.00100 12
151 2 1 1 0 1 -0.34000 0.00120 25
11 2 1 1 0 1 -0.90000 0.00290 4
95 2 2 1 0 1 -1.70000 0.00030 12
95 2 2 1 0 1 -1.90000 0.00007 12
151 2 1 1 0 1 -0.52000 0.00071 12
36 2 2 1 0 1 -0.82000 0.00270 12
95 2 1 1 0 1 -0.71000 0.00007 12
59 2 2 1 0 1 -2.20000 0.00017 12
36 2 2 1 0 1 -0.97000 0.00860 4
151 2 1 1 0 1 -0.25000 0.00004 8
151 2 1 1 0 1 -0.71000 0.00003 4
95 2 1 1 0 1 -0.65000 0.00005 12
95 2 1 1 0 1 -0.75000 0.00021 12
59 2 1 1 0 1 -0.67000 0.00051 12
151 2 1 1 0 1 -0.62000 0.00130 4
151 2 1 1 0 1 -0.90000 0.00066 8
59 2 2 1 0 1 -1.70000 0.00030 12
151 2 1 1 0 1 -0.46000 0.00002 8
59 2 1 1 0 1 -0.36000 0.00016 12
95 2 2 1 0 1 -2.10000 0.00170 12
151 2 1 1 0 1 -0.07300 0.00030 25
151 2 1 1 0 1 -0.16000 0.00009 12
95 2 2 1 0 1 -2.20000 0.00017 12
151 2 1 1 0 1 -1.00000 0.00052 4
59 2 2 1 0 1 -1.90000 0.00007 12
123 2 1 1 0 2 -0.49000 0.01900 4
123 2 1 1 0 2 -0.78000 0.00260 8
27 2 2 1 1 1 -1.80000 0.00190 12
52 2 1 1 1 1 -0.81000 0.01300 4
52 2 2 1 1 1 -2.20000 0.00061 1
52 2 1 1 1 1 -0.61000 0.00009 16
58 2 2 1 1 1 -2.00000 0.15000 8
52 2 1 1 1 1 -1.10000 0.00013 1
52 2 1 1 1 1 -0.89000 0.00014 1
52 2 1 1 1 1 -0.98000 0.00007 4
52 2 2 1 1 1 -1.70000 0.06000 4
52 2 2 1 1 1 -2.90000 0.12000 4
36 2 2 1 1 1 -1.80000 0.01200 4
52 2 1 1 1 1 -0.80000 0.00008 12
52 2 1 1 1 1 -1.10000 0.00150 4
27 2 1 1 1 1 -1.10000 0.00090 12
52 2 2 1 1 1 -3.10000 0.00075 4
52 2 1 1 1 1 -1.00000 0.00063 12
151 2 1 1 1 1 -0.58000 0.00075 4
52 2 2 1 1 1 -1.80000 0.09200 12
164 2 1 1 1 1 -0.78000 0.04200 24
52 2 2 1 1 1 -2.50000 0.00130 1
52 2 1 1 1 1 -0.94000 0.00025 12
27 2 2 1 1 1 -1.80000 0.00250 12
20 2 1 1 1 1 -0.80000 0.17000 24
52 2 1 1 1 1 -0.86000 0.03900 1
52 2 1 1 1 1 -0.75000 0.00008 12
52 2 2 1 1 1 -2.80000 0.00590 1
52 2 2 1 1 1 -3.00000 0.00098 4
52 2 2 1 1 1 -2.30000 0.00110 16
164 2 1 1 1 1 -0.30000 0.04700 24
27 2 1 1 1 1 -1.20000 0.00790 12
52 2 2 1 1 1 -3.00000 0.02400 12
20 2 2 1 1 1 -1.50000 0.02900 16
27 2 1 1 1 1 -1.00000 0.00140 12
27 2 2 1 1 1 -2.00000 0.00630 12
52 2 1 1 1 1 -1.30000 0.00021 4
52 2 2 1 1 1 -3.10000 0.06800 1
52 2 1 1 1 1 -1.00000 0.00130 1
52 2 2 1 1 1 -3.10000 0.07000 1
52 2 2 1 1 1 -2.60000 0.00180 1
52 2 1 1 1 1 -1.30000 0.00031 1
164 2 1 1 1 1 -0.71000 0.04100 24
164 2 1 1 1 1 -0.92000 0.03800 12
52 2 2 1 1 1 -3.00000 0.01100 12
52 2 1 1 1 1 -0.50000 0.00017 16
52 2 1 1 1 1 -0.63000 0.01100 12
52 2 1 1 1 1 -1.10000 0.00010 4
164 2 1 1 1 1 -0.53000 0.03100 12
52 2 2 1 1 1 -3.30000 0.20000 4
52 2 1 1 1 1 -1.10000 0.00023 1
52 2 2 1 1 1 -3.40000 0.00670 4
52 2 2 1 1 1 -3.50000 0.00260 4
52 2 1 1 1 1 -1.00000 0.00130 4
36 2 2 1 1 1 -0.40000 0.00210 1
52 2 2 1 1 1 -1.50000 0.12000 1
52 2 2 1 1 1 -2.50000 0.00047 12
52 2 1 1 1 1 -1.10000 0.00150 1
52 2 2 1 1 1 -2.10000 0.00310 16
20 2 1 1 1 1 -1.40000 0.12000 8
36 2 2 1 1 1 -1.80000 0.00190 12
20 2 2 1 1 1 -3.20000 0.45000 16
52 2 2 1 1 1 -2.60000 0.00058 12
164 2 1 1 1 1 -0.69000 0.07500 12
20 2 1 1 1 1 -0.97000 0.02300 12
52 2 1 1 1 1 -1.20000 0.00010 4
20 2 1 1 1 1 -0.76000 0.10000 16
123 2 1 1 1 2 -0.51000 0.25000 4
123 2 1 1 1 2 -0.22000 0.01000 24
123 2 1 1 1 2 -0.58000 0.00750 12
113 2 1 1 1 2 -0.43000 0.00003 12
113 2 1 1 1 2 -0.29000 0.00003 16
123 2 1 1 1 2 -0.64000 0.03800 8
113 2 1 1 1 2 -0.17000 0.00008 25
113 2 1 1 1 2 -0.76000 0.00004 8
113 2 1 1 1 2 -0.05000 0.00031 8
36 2 2 2 0 1 -0.84000 0.00360 12
29 2 2 2 0 1 -0.13000 0.00069 1
29 2 2 2 0 1 -1.90000 0.00250 4
29 2 2 2 0 1 -1.90000 0.00250 4
29 2 2 2 0 1 -2.00000 0.00130 12
36 2 2 2 0 1 -0.78000 0.00800 4
29 2 2 2 0 1 -0.87000 0.00095 4
36 2 2 2 1 1 -0.22000 0.00170 1
36 2 2 2 1 1 -1.50000 0.00450 4
36 2 2 2 1 1 -2.10000 0.00380 12
68 3 1 1 0 1 -0.20000 0.00025 20
68 3 1 1 0 1 -0.45000 0.00039 8
68 3 1 1 0 1 -0.39000 0.00012 8
99 3 1 1 0 1 -0.64000 0.00370 8
134 3 1 1 0 1 -0.31000 0.00044 12
70 3 1 1 0 1 -1.70000 0.31000 16
68 3 1 1 0 1 -0.13000 0.00013 20
68 3 1 1 0 1 -0.13000 0.00054 25
68 3 1 1 0 1 -0.30000 0.00019 20
68 3 1 1 0 1 -0.43000 0.00025 8
73 3 1 1 0 1 -0.17000 0.06000 12
134 3 1 1 0 1 -0.12000 0.00040 12
68 3 1 1 0 1 -0.54000 0.00030 8
68 3 1 1 0 1 -0.04100 0.00034 25
75 3 1 1 0 1 -0.37000 0.00120 24
68 3 1 1 0 1 -0.07300 0.00048 8
68 3 1 1 0 1 -0.11000 0.00130 25
73 3 1 1 0 1 -0.27000 0.00950 12
99 3 1 1 0 1 -0.68000 0.00051 8
68 3 1 1 0 1 -0.16000 0.00044 25
68 3 1 1 0 1 -0.26000 0.00028 20
125 3 1 1 1 1 -0.59000 0.00036 16
34 3 2 1 1 1 -1.40000 0.00500 4
147 3 1 1 1 1 -0.74000 0.00017 8
147 3 1 1 1 1 -0.62000 0.00061 8
25 3 1 1 1 2 -1.60000 0.02800 12
19 3 2 1 1 2 -2.50000 0.02700 12
161 3 1 1 1 2 -0.94000 0.00340 12
19 3 2 1 1 2 -0.93000 0.03100 12
25 3 1 1 1 2 -2.00000 0.01900 12
120 3 1 1 1 3 -0.89000 0.14000 12
122 3 1 1 1 3 -0.13000 0.00440 24
122 3 1 1 1 3 -0.53000 0.00035 4
122 3 1 1 1 3 -0.29000 0.00093 20
122 3 1 1 1 3 -0.39000 0.00140 1
120 3 2 1 1 3 -0.29000 0.02600 12
122 3 1 1 1 3 -0.40000 0.00039 16
122 3 1 1 1 3 -0.46000 0.00030 8
75 3 1 2 0 1 -0.39000 0.00860 24
33 3 1 2 0 1 -0.65000 0.00310 8
33 3 1 2 0 1 -0.69000 0.00020 8
33 3 1 2 0 1 -0.88000 0.00140 8
33 3 1 2 0 1 -0.62000 0.00061 8
33 3 1 2 0 1 -0.81000 0.00110 8
33 3 1 2 0 1 -0.74000 0.00017 8
125 3 1 2 1 1 -0.49000 0.00047 16
90 3 2 2 1 1 -2.60000 0.10000 12
90 3 2 2 1 1 -2.40000 0.00870 12
90 3 2 2 1 1 -2.20000 0.02900 12
90 3 2 2 1 1 -3.10000 0.06200 12
38 3 2 1 3 -0.36000 0.01400 16
98 4 1 1 1 1 -0.63000 0.07000 8
98 4 2 1 1 1 -1.60000 1.20000 8
53 4 2 1 1 3 -0.64000 0.24000 1
53 4 2 1 1 3 -3.00000 0.21000 8
53 4 2 1 1 3 -2.70000 1.90000 16
48 1 1 1 1 -0.63000 0.00039 8
