Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Meta-Analysis
. 2023 Mar 29;21(1):112.
doi: 10.1186/s12916-023-02823-9.

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

Ziyu Liu  1 , Fahad M Al Amer  2 , Mengli Xiao  3 , Chang Xu  4   5 , Luis Furuya-Kanamori  6 , Hwanhee Hong  7 , Lianne Siegel  8 , Lifeng Lin  9
Affiliations
Meta-Analysis

The normality assumption on between-study random effects was questionable in a considerable number of Cochrane meta-analyses

Ziyu Liu et al. BMC Med. .

Abstract

Background: Studies included in a meta-analysis are often heterogeneous. The traditional random-effects models assume their true effects to follow a normal distribution, while it is unclear if this critical assumption is practical. Violations of this between-study normality assumption could lead to problematic meta-analytical conclusions. We aimed to empirically examine if this assumption is valid in published meta-analyses.

Methods: In this cross-sectional study, we collected meta-analyses available in the Cochrane Library with at least 10 studies and with between-study variance estimates > 0. For each extracted meta-analysis, we performed the Shapiro-Wilk (SW) test to quantitatively assess the between-study normality assumption. For binary outcomes, we assessed between-study normality for odds ratios (ORs), relative risks (RRs), and risk differences (RDs). Subgroup analyses based on sample sizes and event rates were used to rule out the potential confounders. In addition, we obtained the quantile-quantile (Q-Q) plot of study-specific standardized residuals for visually assessing between-study normality.

Results: Based on 4234 eligible meta-analyses with binary outcomes and 3433 with non-binary outcomes, the proportion of meta-analyses that had statistically significant non-normality varied from 15.1 to 26.2%. RDs and non-binary outcomes led to more frequent non-normality issues than ORs and RRs. For binary outcomes, the between-study non-normality was more frequently found in meta-analyses with larger sample sizes and event rates away from 0 and 100%. The agreements of assessing the normality between two independent researchers based on Q-Q plots were fair or moderate.

Conclusions: The between-study normality assumption is commonly violated in Cochrane meta-analyses. This assumption should be routinely assessed when performing a meta-analysis. When it may not hold, alternative meta-analysis methods that do not make this assumption should be considered.

Keywords: Cochrane Library; Effect measure; Heterogeneity; Meta-analysis; Normality assumption; Q–Q plot.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1
Fig. 1
Proportions of meta-analyses with significant non-normality in different subgroups for A meta-analyses with binary outcomes and B those with non-binary outcomes. On the horizontal axis, the first line presents restrictions on study sample sizes, and the second line of panel A presents restrictions on event rates. The corresponding numbers of meta-analyses satisfying the restrictions are in parentheses. The vertical bars represent 95% confidence intervals of the proportions
Fig. 2
Fig. 2
Approximate proportions of truly non-normal meta-analyses with A binary outcomes and B non-binary outcomes when the statistical power varies from 30 to 60%
Fig. 3
Fig. 3
Proportions of significant non-normality in different subgroups for meta-analyses when using A ORs, B RRs, and C RDs. On the horizontal axis, the first line presents restrictions on sample sizes, and the second line presents restrictions on event rates. The corresponding numbers of meta-analyses satisfying the restrictions are in parentheses. The vertical bars represent 95% confidence intervals of the proportions
Fig. 4
Fig. 4
Approximate proportions of truly non-normal meta-analyses for A ORs, B RRs, and C RDs when the statistical power varies from 30 to 60%
Fig. 5
Fig. 5
A framework of recommendations for performing a meta-analysis based on the assessments of heterogeneity and normality
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–188. doi: 10.1016/0197-2456(86)90046-2. - DOI - PubMed
    1. Brockwell SE, Gordon IR. A comparison of statistical methods for meta-analysis. Stat Med. 2001;20(6):825–840. doi: 10.1002/sim.650. - DOI - PubMed
    1. Jackson D, Riley R, White IR. Multivariate meta-analysis: potential and promise. Stat Med. 2011;30(20):2481–2498. doi: 10.1002/sim.4172. - DOI - PMC - PubMed
    1. Cheung MW-L, Ho RCM, Lim Y, Mak A. Conducting a meta-analysis: basics and good practices. Int J Rheum Dis. 2012;15(2):129–35. doi: 10.1111/j.1756-185X.2012.01712.x. - DOI - PubMed
    1. Jackson D, White IR. When should meta-analysis avoid making hidden normality assumptions? Biom J. 2018;60(6):1040–1058. doi: 10.1002/bimj.201800071. - DOI - PMC - PubMed

Publication types

LinkOut - more resources