NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Structured Abstract
Background:
Living systematic reviews can more rapidly and efficiently incorporate new evidence into systematic reviews through ongoing updates. A challenge to conducting living systematic reviews is identifying new articles in a timely manner. Optimizing search strategies to identify new studies before they have undergone indexing in electronic databases and automation using machine learning classifiers may increase the efficiency of identifying relevant new studies.
Methods:
This project had three stages: develop optimized search strategies (Stage 1), test machine learning classifier on optimized searches (Stage 2), and test machine learning classifier on monthly update searches (Stage 3). Ovid® MEDLINE® search strategies were developed for three previously conducted chronic pain reviews using standard methods, combining National Library of Medicine Medical Subject Headings (MeSH) terms and text words (“standard searches”). Text word-only search strategies (“optimized searches”) were also developed based on the inclusion criteria for each review. In Stage 2, a machine learning classifier was trained and refined using citations from each of the completed pain reviews (“training set”) and tested on a subset of more recent citations (“simulated update”), to develop models that could predict the relevant of citations for each topic. In Stage 3, the machine learning models were prospectively applied to “optimized” monthly update searches conducted for the three pain reviews.
Results:
In Stage 1, the optimized searches were less precise than the standard searches (i.e., identified more citations that reviewers eventually excluded) but were highly sensitive. In Stage 2, a machine learning classifier using a support vector machine model achieved 96 to 100 percent recall for all topics, with precision of between 1 and 7 percent. Performance was similar using the training data and on the simulated updates. The machine learning classifier excluded 35 to 65 percent of studies classified as low relevance. In Stage 3, the machine classifier achieved 97 to 100 percent sensitivity and excluded (i.e., classified as very low probability) 45 to 76 percent of studies identified in prospective, actual update searches. The estimated savings in time using the machine classifier ranged from 2.0 to 13.2 hours.
Conclusions:
Text word-only searches to facilitate the conduct of living systematic reviews are associated with high sensitivity but reduced precision compared with standard searches using MeSH indexing terms. A machine learning classifier had high recall for identifying studies identified using text word searches, but had low to moderate precision, resulting in a small to moderate estimated time savings when applied to update searches.
Contents
Suggested citation:
Chou R, Dana T, Shetty KD. Testing a Machine Learning Tool for Facilitating Living Systematic Reviews of Chronic Pain Treatments. Methods Research Report. (Prepared by the Pacific Northwest Evidence-based Practice Center under Contract No. 290-2015-00009-I and the Southern California Evidence-based Practice Center-RAND Corporation under Contract No. 290-2015-00010-I.) AHRQ Publication No. 21-EHC004. Rockville, MD: Agency for Healthcare Research and Quality. November 2020. Posted final reports are located on the Effective Health Care Program search page. DOI: 10.23970/AHRQEPCMETHTESTINGMACHINELEARNING.
This report is based on research conducted by the Pacific Northwest Evidence-based Practice Center (EPC) and the Southern California EPC-RAND Corporation under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract Nos. 290-2015-00009-I and 290-2015-00010-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.
The information in this report is intended to help healthcare decision makers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of healthcare services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
This report is made available to the public under the terms of a licensing agreement between the author and the Agency for Healthcare Research and Quality. This report may be used and reprinted without permission except those copyrighted materials that are clearly noted in the report. Further reproduction of those copyrighted materials is prohibited without the express permission of copyright holders.
AHRQ or U.S. Department of Health and Human Services endorsement of any derivative products that may be developed from this report, such as clinical practice guidelines, other quality enhancement tools, or reimbursement or coverage policies may not be stated or implied.
AHRQ appreciates appropriate acknowledgment and citation of its work. Suggested language for acknowledgment: This work was based on a methods research report, Testing a Machine Learning Tool for Facilitating Living Systematic Reviews of Chronic Pain Treatments, by the Evidence-based Practice Center Program at the Agency for Healthcare Research and Quality (AHRQ).
- NLM CatalogRelated NLM Catalog Entries
- Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers.[Artif Intell Med. 2012]Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers.Bekhuis T, Demner-Fushman D. Artif Intell Med. 2012 Jul; 55(3):197-207. Epub 2012 Jun 5.
- Review A Prospective Comparison of Evidence Synthesis Search Strategies Developed With and Without Text-Mining Tools[ 2021]Review A Prospective Comparison of Evidence Synthesis Search Strategies Developed With and Without Text-Mining ToolsPaynter RA, Fiordalisi C, Stoeger E, Erinoff E, Featherstone R, Voisin C, Adam GP. 2021 Mar
- Applying machine classifiers to update searches: Analysis from two case studies.[Res Synth Methods. 2022]Applying machine classifiers to update searches: Analysis from two case studies.Stansfield C, Stokes G, Thomas J. Res Synth Methods. 2022 Jan; 13(1):121-133. Epub 2021 Nov 25.
- Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviews.[Syst Rev. 2020]Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviews.Popoff E, Besada M, Jansen JP, Cope S, Kanters S. Syst Rev. 2020 Dec 13; 9(1):293. Epub 2020 Dec 13.
- Review Updating Systematic Reviews[ 2007]Review Updating Systematic ReviewsShojania KG, Sampson M, Ansari MT, Ji J, Garritty C, Rader T, Moher D. 2007 Sep
- Testing a Machine Learning Tool for Facilitating Living Systematic Reviews of Ch...Testing a Machine Learning Tool for Facilitating Living Systematic Reviews of Chronic Pain Treatments
- Standardized Library of Depression Outcome MeasuresStandardized Library of Depression Outcome Measures
- Roadmap for Narratively Describing Effects of Interventions in Systematic Review...Roadmap for Narratively Describing Effects of Interventions in Systematic Reviews
- Registries for Evaluating Patient Outcomes: A User’s GuideRegistries for Evaluating Patient Outcomes: A User’s Guide
- Outcome Measure Harmonization and Data Infrastructure for Patient-Centered Outco...Outcome Measure Harmonization and Data Infrastructure for Patient-Centered Outcomes Research in Depression: Report on Registry Configuration
Your browsing activity is empty.
Activity recording is turned off.
See more...