NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bodies of Evidence
Research White Paper
Investigators: Gerald Gartlehner, MD, MPH, Andreea Dobrescu, MD, Tammeka Swinson Evans, MOP, Kylie Thaler, MD, MPH, Barbara Nussbaumer, MSc, Isolde Sommer, PhD, MPH, and Kathleen N Lohr, PhD.
Author Information and AffiliationsStructured Abstract
Objective:
The objective of our methods project was to use a diverse sample of medical interventions to assess empirically whether first trials rendered substantially different treatment effect estimates than reliable, high-quality bodies of evidence.
Study design and setting:
We employed a meta-epidemiological study design using 100 bodies of evidence from Cochrane reports that had been graded as high quality of evidence. To determine the concordance of effect estimates between first and subsequent trials, we applied both quantitative and qualitative approaches. For quantitative assessment, we used Lin's concordance correlation and calculated z-scores; to determine the magnitude of differences of treatment effects, we calculated standardized mean differences (SMDs) and ratios of relative risks. We determined qualitative concordance based on a 2-tiered approach incorporating changes in statistical significance and magnitude of effect.
Results:
First trials both over- and under-estimated the true treatment effects in no discernible pattern. Nevertheless, depending on the definition of concordance, effect estimates of first trials were concordant with pooled subsequent studies in at least 33 percent but up to 50 percent of comparisons. The pooled magnitude of change as bodies of evidence advanced from single trials to high-quality bodies of evidence was 0.16 SMD (95% confidence interval [CI], 0.12 to 0.21). In 80 percent of comparisons the difference in effect estimates was smaller than 0.5 SMDs. In first trials with large treatment effects (>0.5 SMD), however, estimates of effect substantially changed as new evidence accrued (mean change 0.68 SMD, 95% CI, .50 to 0.86)
Conclusion:
Results of first trials often change but the magnitude of change, on average, is small. Exceptions are first trials that present large treatment effects which often dissipate as new evidence accrues.
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2012-00008-I. Prepared by: RTI International–University of North Carolina Evidence-based Practice Center, Research Triangle Park, NC
Suggested citation:
Gartlehner G, Dobrescu A, Swinson Evans T, Thaler K, Nussbaumer B, Sommer I, Lohr KN. Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bodies of Evidence. Research White Paper. (Prepared by the RTI–UNC Evidence-based Practice Center under Contract No. 290-2012-00008-I.) AHRQ Publication No. 15-EHC010-EF. Rockville, MD: Agency for Healthcare Research and Quality. March 2015. www.effectivehealthcare.ahrq.gov/reports/final.cfm.
This report is based on work conducted by the Research Triangle Institute International – University of North Carolina Chapel Hill (RTI-UNC) Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2012-00008-I). The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
AHRQ or U.S. Department of Health and Human Services endorsement of any derivative products that may be developed from this report, such as clinical practice guidelines, other quality enhancement tools, or reimbursement or coverage policies may not be stated or implied.
Drs. Gartlehner and Thaler are members of the GRADE Working Group. Drs. Gartlehner and Lohr are co-authors of the AHRQ guidance for grading the strength of evidence. None of the other authors have any affiliations or financial involvement disclosures to report.
- 1
540 Gaither Road, Rockville, MD 20850; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Average effect estimates remain similar as evidence evolves from single trials to high-quality bodies of evidence: a meta-epidemiologic study.[J Clin Epidemiol. 2016]Average effect estimates remain similar as evidence evolves from single trials to high-quality bodies of evidence: a meta-epidemiologic study.Gartlehner G, Dobrescu A, Evans TS, Thaler K, Nussbaumer B, Sommer I, Lohr KN. J Clin Epidemiol. 2016 Jan; 69:16-22. Epub 2015 Feb 28.
- Review Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2014]Review Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task ForceHenderson JT, Whitlock EP, O'Conner E, Senger CA, Thompson JH, Rowland MG. 2014 Apr
- Review Screening for Colorectal Cancer: An Updated Systematic Review[ 2008]Review Screening for Colorectal Cancer: An Updated Systematic ReviewWhitlock EP, Lin J, Liles E, Beil T, Fu R, O'Connor E, Thompson RN, Cardenas T. 2008 Oct
- Review Updating Systematic Reviews[ 2007]Review Updating Systematic ReviewsShojania KG, Sampson M, Ansari MT, Ji J, Garritty C, Rader T, Moher D. 2007 Sep
- Review Screening for Cognitive Impairment in Older Adults: An Evidence Update for the U.S. Preventive Services Task Force[ 2013]Review Screening for Cognitive Impairment in Older Adults: An Evidence Update for the U.S. Preventive Services Task ForceLin JS, O'Connor E, Rossom RC, Perdue LA, Burda BU, Thompson M, Eckstrom E. 2013 Nov
- Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bod...Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bodies of Evidence
- Change Score or Followup Score?Change Score or Followup Score?
- Challenges in Conducting EPC Reviews of Behavior Change InterventionsChallenges in Conducting EPC Reviews of Behavior Change Interventions
- Assessing the Predictive Validity of Strength of Evidence Grades: A Meta-Epidemi...Assessing the Predictive Validity of Strength of Evidence Grades: A Meta-Epidemiological Study
- EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Rev...EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews
Your browsing activity is empty.
Activity recording is turned off.
See more...