A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

doi:10.1016/j.jclinepi.2019.07.005

Comparative Study

. 2019 Nov:115:77-89.

doi: 10.1016/j.jclinepi.2019.07.005. Epub 2019 Jul 11.

A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

Tianjing Li¹, Ian J Saldanha², Jens Jap³, Bryant T Smith³, Joseph Canner⁴, Susan M Hutfless⁵, Vernal Branch⁶, Simona Carini⁷, Wiley Chan⁸, Berry de Bruijn⁹, Byron C Wallace¹⁰, Sandra A Walsh¹¹, Elizabeth J Whamond¹², M Hassan Murad¹³, Ida Sim¹⁴, Jesse A Berlin¹⁵, Joseph Lau³, Kay Dickersin¹⁶, Christopher H Schmid¹⁷

Affiliations

¹ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, E6140, Baltimore, MD 21205, USA. Electronic address: tli19@jhu.edu.
² Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA; Department of Epidemiology (Secondary), Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA.
³ Department of Health Services, Policy, and Practice, Center for Evidence Synthesis in Health, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA.
⁴ Department of Surgery, Center for Outcomes Research, Johns Hopkins School of Medicine, 720 Rutland Avenue, Turner 74, Baltimore, MD 21205, USA.
⁵ Department of Surgery, Johns Hopkins School of Medicine, 600 N Wolfe St, Blalock 406, Baltimore, MD 21205, USA.
⁶ 119 Landings Drive, Suite 403, Mooresville, NC 28117, USA.
⁷ Department of Medicine, School of Medicine, University of California San Francisco, 3333 California Street, San Francisco, CA 94143, USA.
⁸ Internal Medicine, Kaiser Permanente Northwest, 500 NE Multnomah Street, Suite 100, KPB-2, Portland, OR 97232, USA.
⁹ Internal Medicine, National Research Council Canada, Digital Technologies Research Centre, 1200 Montreal Road, Ottawa, Ontario K1A 0R6, Canada.
¹⁰ Internal Medicine, College of Computer and Information Science, Northeastern University, 440 Huntington Avenue, 476 West Village H, Boston, MA 02115, USA.
¹¹ Internal Medicine, California Breast Cancer Organizations, 900 Kent Drive, Davis, CA 95616, USA.
¹² Internal Medicine, Cochrane Consumer Network, 1421 Woodstock Road, Fredericton, New Brunswick E3C 1P2, Canada.
¹³ Internal Medicine, Evidence-based Practice Center, Mayo Clinic, 200 1st Street, Rochester, MN 55905, USA.
¹⁴ Department of Medicine, University of California San Francisco School of Medicine, San Francisco, CA, USA.
¹⁵ Internal Medicine, Epidemiology, Johnson & Johnson, 1125 Trenton-Harbourton Road, K3-Zone 1, Titusville, NJ 08560, USA.
¹⁶ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, E6152, Baltimore, MD 21205, USA.
¹⁷ Department of Biostatistics, Brown University School of Public Health, 121 South Main Street, Box G-S121-7, Providence, RI 02912, USA.

PMID: 31302205
DOI: 10.1016/j.jclinepi.2019.07.005

Free article

Comparative Study

A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

Tianjing Li et al. J Clin Epidemiol. 2019 Nov.

Free article

. 2019 Nov:115:77-89.

doi: 10.1016/j.jclinepi.2019.07.005. Epub 2019 Jul 11.

Authors

Affiliations

¹ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, E6140, Baltimore, MD 21205, USA. Electronic address: tli19@jhu.edu.
² Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA; Department of Epidemiology (Secondary), Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA.
³ Department of Health Services, Policy, and Practice, Center for Evidence Synthesis in Health, Brown University School of Public Health, 121 South Main Street, Box G-S121-8, Providence, RI 02912, USA.
⁴ Department of Surgery, Center for Outcomes Research, Johns Hopkins School of Medicine, 720 Rutland Avenue, Turner 74, Baltimore, MD 21205, USA.
⁵ Department of Surgery, Johns Hopkins School of Medicine, 600 N Wolfe St, Blalock 406, Baltimore, MD 21205, USA.
⁶ 119 Landings Drive, Suite 403, Mooresville, NC 28117, USA.
⁷ Department of Medicine, School of Medicine, University of California San Francisco, 3333 California Street, San Francisco, CA 94143, USA.
⁸ Internal Medicine, Kaiser Permanente Northwest, 500 NE Multnomah Street, Suite 100, KPB-2, Portland, OR 97232, USA.
⁹ Internal Medicine, National Research Council Canada, Digital Technologies Research Centre, 1200 Montreal Road, Ottawa, Ontario K1A 0R6, Canada.
¹⁰ Internal Medicine, College of Computer and Information Science, Northeastern University, 440 Huntington Avenue, 476 West Village H, Boston, MA 02115, USA.
¹¹ Internal Medicine, California Breast Cancer Organizations, 900 Kent Drive, Davis, CA 95616, USA.
¹² Internal Medicine, Cochrane Consumer Network, 1421 Woodstock Road, Fredericton, New Brunswick E3C 1P2, Canada.
¹³ Internal Medicine, Evidence-based Practice Center, Mayo Clinic, 200 1st Street, Rochester, MN 55905, USA.
¹⁴ Department of Medicine, University of California San Francisco School of Medicine, San Francisco, CA, USA.
¹⁵ Internal Medicine, Epidemiology, Johnson & Johnson, 1125 Trenton-Harbourton Road, K3-Zone 1, Titusville, NJ 08560, USA.
¹⁶ Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, E6152, Baltimore, MD 21205, USA.
¹⁷ Department of Biostatistics, Brown University School of Public Health, 121 South Main Street, Box G-S121-7, Providence, RI 02912, USA.

PMID: 31302205
DOI: 10.1016/j.jclinepi.2019.07.005

Abstract

Objectives: Data Abstraction Assistant (DAA) is a software for linking items abstracted into a data collection form for a systematic review to their locations in a study report. We conducted a randomized cross-over trial that compared DAA-facilitated single-data abstraction plus verification ("DAA verification"), single data abstraction plus verification ("regular verification"), and independent dual data abstraction plus adjudication ("independent abstraction").

Study design and setting: This study is an online randomized cross-over trial with 26 pairs of data abstractors. Each pair abstracted data from six articles, two per approach. Outcomes were the proportion of errors and time taken.

Results: Overall proportion of errors was 17% for DAA verification, 16% for regular verification, and 15% for independent abstraction. DAA verification was associated with higher odds of errors when compared with regular verification (adjusted odds ratio [OR] = 1.08; 95% confidence interval [CI]: 0.99-1.17) or independent abstraction (adjusted OR = 1.12; 95% CI: 1.03-1.22). For each article, DAA verification took 20 minutes (95% CI: 1-40) longer than regular verification, but 46 minutes (95% CI: 26 to 66) shorter than independent abstraction.

Conclusion: Independent abstraction may only be necessary for complex data items. DAA provides an audit trail that is crucial for reproducible research.

Keywords: Accuracy; Data abstraction; Efficiency; Randomized cross-over trial; Software application; Systematic review.

PubMed Disclaimer

Cited by

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.
Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, McKenzie JE. Page MJ, et al. BMJ. 2021 Mar 29;372:n160. doi: 10.1136/bmj.n160. BMJ. 2021. PMID: 33781993 Free PMC article.
Systematic review of the method and quality of reporting of complications from studies evaluating innovative glaucoma surgical procedures.
Bonnar J, Azuara-Blanco A. Bonnar J, et al. Eye (Lond). 2023 Jun;37(9):1774-1777. doi: 10.1038/s41433-022-02268-z. Epub 2022 Oct 17. Eye (Lond). 2023. PMID: 36253459 Free PMC article. Review.
The Systematic Review Data Repository (SRDR): descriptive characteristics of publicly available data and opportunities for research.
Saldanha IJ, Smith BT, Ntzani E, Jap J, Balk EM, Lau J. Saldanha IJ, et al. Syst Rev. 2019 Dec 20;8(1):334. doi: 10.1186/s13643-019-1250-y. Syst Rev. 2019. PMID: 31862012 Free PMC article.
An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review.
Affengruber L, van der Maten MM, Spiero I, Nussbaumer-Streit B, Mahmić-Kaknjo M, Ellen ME, Goossen K, Kantorova L, Hooft L, Riva N, Poulentzas G, Lalagkas PN, Silva AG, Sassano M, Sfetcu R, Marqués ME, Friessova T, Baladia E, Pezzullo AM, Martinez P, Gartlehner G, Spijker R. Affengruber L, et al. BMC Med Res Methodol. 2024 Sep 18;24(1):210. doi: 10.1186/s12874-024-02320-4. BMC Med Res Methodol. 2024. PMID: 39294580 Free PMC article.
Optical Coherence Tomography Imaging of the Lamina Cribrosa: Structural Biomarkers in Nonglaucomatous Diseases.
Paulo A, Vaz PG, Andrade De Jesus D, Sánchez Brea L, Van Eijgen J, Cardoso J, van Walsum T, Klein S, Stalmans I, Barbosa Breda J. Paulo A, et al. J Ophthalmol. 2021 Feb 19;2021:8844614. doi: 10.1155/2021/8844614. eCollection 2021. J Ophthalmol. 2021. PMID: 33680508 Free PMC article. Review.

See all "Cited by" articles

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- ClinicalKey
- Elsevier Science

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

Affiliations

A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

Authors

Affiliations

Abstract

Similar articles

Cited by

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources