Objective: Reproducibility of systemic reviews (SRs) can be hindered by the presence of citation bias. Citation bias may occur when authors of SRs conduct hand-searches of included study reference lists to identify additional studies. Such a practice may lead to exaggerated SR summary effects. The purpose of this paper is to examine the prevalence of hand-searching reference lists in otolaryngology SRs.

Methods: The authors searched for systematic reviews published in eight clinical otolaryngology journals using the Cochrane Library and PubMed, with the date parameter of January 1, 2008, to December 31, 2017. Two independent authors worked separately to extract data from each SR for the following elements: whether reference lists were hand-searched, other kinds of supplemental searching, PRISMA adherence, and funding source. Following extraction, the investigators met to review discrepancies and achieve consensus.

Results: A total of 539 systemic reviews, 502 from clinical journals and 37 from the Cochrane library, were identified. Of those SRs, 72.4% (390/539) hand-searched reference lists, including 97.3% (36/37) of Cochrane reviews. For 228 (58.5%) of the SRs that hand-searched reference lists, no other supplemental search (e.g., search of trial registries) was conducted.

Conclusions: These findings indicate that hand-searching reference lists is a common practice in otolaryngology SRs. Moreover, a majority of studies at risk of citation bias did not attempt to mitigate the bias by conducting additional supplemental searches. The implication is that summary effects in otolaryngology systematic reviews may be biased toward statistically significant findings.