Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2020 Feb 28;20(1):48.
doi: 10.1186/s12874-020-00929-9.

Comparison of exclusion, imputation and modelling of missing binary outcome data in frequentist network meta-analysis

Loukia M Spineli  1 , Chrysostomos Kalyvas  2
Affiliations
Comparative Study

Comparison of exclusion, imputation and modelling of missing binary outcome data in frequentist network meta-analysis

Loukia M Spineli et al. BMC Med Res Methodol. .

Abstract

Background: Missing participant outcome data (MOD) are ubiquitous in systematic reviews with network meta-analysis (NMA) as they invade from the inclusion of clinical trials with reported participant losses. There are available strategies to address aggregate MOD, and in particular binary MOD, while considering the missing at random (MAR) assumption as a starting point. Little is known about their performance though regarding the meta-analytic parameters of a random-effects model for aggregate binary outcome data as obtained from trial-reports (i.e. the number of events and number of MOD out of the total randomised per arm).

Methods: We used four strategies to handle binary MOD under MAR and we classified these strategies to those modelling versus excluding/imputing MOD and to those accounting for versus ignoring uncertainty about MAR. We investigated the performance of these strategies in terms of core NMA estimates by performing both an empirical and simulation study using random-effects NMA based on electrical network theory. We used Bland-Altman plots to illustrate the agreement between the compared strategies, and we considered the mean bias, coverage probability and width of the confidence interval to be the frequentist measures of performance.

Results: Modelling MOD under MAR agreed with exclusion and imputation under MAR in terms of estimated log odds ratios and inconsistency factor, whereas accountability or not of the uncertainty regarding MOD affected intervention hierarchy and precision around the NMA estimates: strategies that ignore uncertainty about MOD led to more precise NMA estimates, and increased between-trial variance. All strategies showed good performance for low MOD (<5%), consistent evidence and low between-trial variance, whereas performance was compromised for large informative MOD (> 20%), inconsistent evidence and substantial between-trial variance, especially for strategies that ignore uncertainty due to MOD.

Conclusions: The analysts should avoid applying strategies that manipulate MOD before analysis (i.e. exclusion and imputation) as they implicate the inferences negatively. Modelling MOD, on the other hand, via a pattern-mixture model to propagate the uncertainty about MAR assumption constitutes both conceptually and statistically proper strategy to address MOD in a systematic review.

Keywords: Imputation; Missing outcome data; Network meta-analysis; Pattern-mixture model; Systematic review.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1
Fig. 1
Bland-Altman plots based on the empirical analysis of 29 networks illustrate the level of agreement between ‘on average MAR’ and other strategies for MOD under MAR assumption in terms of estimated log odds ratios (of comparisons with the selected reference intervention of each network), common between-trial variance (τ2), inconsistency factors, and P-scores. Use of normal distribution on log IMORs with mean 0 and variance 1. Different colours indicate the extent and balance of MOD across 29 networks (17 networks with at least one closed loop). CCA, complete case analysis; ICAp, imputed case analysis of observed event risks; IF, inconsistency factor; OR, odds ratio
Fig. 2
Fig. 2
Bland-Altman plots based on the empirical analysis of 29 networks illustrate the level of agreement between ‘on average MAR’ and other strategies for MOD under MAR assumption in terms of the standard error of log odds ratios (of comparisons with the selected reference intervention of each network) and of inconsistency factors. Use of normal distribution on log IMORs with mean 0 and variance 1. Different colours indicate the extent and balance of MOD across 29 networks (17 networks with at least one closed loop). CCA, complete case analysis; ICAp, imputed case analysis of observed event risks; IF, inconsistency factor; OR, odds ratio
Fig. 3
Fig. 3
Mean bias for log OR (new versus old intervention) under informative MOD while accounting for the scenarios on the number of studies (typical loop, double), the extent of MOD (moderate, large), the balance of MOD (balance, imbalance), the extent of between-trial variance (τ2; 0.02 as small, 0.08 as substantial), and extent of inconsistency (absent, moderate). AVG, on average MAR; CCA, complete case analysis; ICAp, imputed case analysis of observed event risks; IF, inconsistency factor; MOD, missing outcome data; UI, uncertainty interval
Fig. 4
Fig. 4
Mean bias for between-trial variance (τ2) under informative MOD while accounting for the scenarios on the number of studies (typical loop, double), the extent of MOD (moderate, large), the balance of MOD (balance, imbalance), and extent of inconsistency (absent, moderate). AVG, on average MAR; CCA, complete case analysis; ICAp, imputed case analysis of observed event risks; IF, inconsistency factor; MOD, missing outcome data; UI, uncertainty interval
Fig. 5
Fig. 5
Coverage probability of 95% confidence interval for log OR (new versus old intervention) under informative MOD while accounting for the scenarios on the number of studies (typical loop, double), the extent of MOD (moderate, large), balance of MOD (balance, imbalance), the extent of between-trial variance (τ2; 0.02 as small, 0.08 as substantial), and extent of inconsistency (absent, moderate). AVG, on average MAR; CCA, complete case analysis; ICAp, imputed case analysis of observed event risks; IF, inconsistency factor; MOD, missing outcome data; UI, uncertainty interval
Fig. 6
Fig. 6
Mean width of 95% confidence interval for log OR (new versus old intervention) under informative MOD while accounting for the scenarios on the number of studies (typical loop, double), the extent of MOD (moderate, large), the balance of MOD (balance, imbalance), the extent of between-trial variance (τ2; 0.02 as small, 0.08 as substantial), and extent of inconsistency (absent, moderate). AVG, on average MAR; CCA, complete case analysis; ICAp, imputed case analysis of observed event risks; IF, inconsistency factor; MOD, missing outcome data; UI, uncertainty interval
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. Akl EA, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, Johnston BC, Sun X, et al. Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey. BMJ Open. 2015;5:e009368. doi: 10.1136/bmjopen-2015-009368. - DOI - PMC - PubMed
    1. Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23. doi: 10.1016/j.jclinepi.2018.02.016. - DOI - PubMed
    1. Spineli LM, Yepes-Nuñez JJ, Schünemann HJ. A systematic survey shows that reporting and handling of missing outcome data in networks of interventions is poor. BMC Med Res Methodol. 2018;18:115. doi: 10.1186/s12874-018-0576-9. - DOI - PMC - PubMed
    1. Spineli LM, Pandis N, Salanti G. Reporting and handling missing outcome data in mental health: a systematic review of Cochrane systematic reviews and meta-analyses. Res Synth Methods. 2015;6:175–187. doi: 10.1002/jrsm.1131. - DOI - PubMed
    1. Akl EA, Kahale LA, Agoritsas T, Brignardello-Petersen R, Busse JW, Carrasco-Labra A, et al. Handling trial participants with missing outcome data when conducting a meta-analysis: a systematic survey of proposed approaches. Syst Rev. 2015;4:98. doi: 10.1186/s13643-015-0083-6. - DOI - PMC - PubMed

Publication types

MeSH terms

LinkOut - more resources