Objective: To compare the sensitivity, precision, and the costs in time of searching by hand and by MEDLINE to identify randomized controlled trials (RCTs) for systematic reviews.

Design: Nine anesthetic and pain journals were searched by both MEDLINE and by hand for the years 1970, 1980, and 1990, recording number of publications and time taken to identify randomized double-blind controlled trials in pain research.

Results: Thirty-four volumes, containing 5583 full publications (24,417 pages) and 2889 abstracts and letters (1755 pages) were hand searched; 142 eligible (definite RCTs) full papers and 171 eligible abstracts and letters were identified. The MEDLINE search strategy yielded 274 reports of which 138 were eligible; 125 of these were full papers, 1 was a letter and 12 were abstracts. Two full papers which were identified by the MEDLINE search strategy were missed by hand search. The overall sensitivity of the MEDLINE search strategy for full papers was 87% ([125/144] x 100) with a precision of 52% ([125/242] x 100). This is the best combination of sensitivity and precision reported to date. Abstracts were mostly in supplement issues which were not indexed. Combining the MEDLINE search strategy with selective hand search of abstracts and letters gave a sensitivity of 94%.

Conclusions: Hand search of entire journals remains the most accurate method for identification of the eligible reports, but it is the most time-consuming. The MEDLINE search was faster, but it failed to identify at least 13% of the indexed eligible reports. Ideally, both hand search and MEDLINE should be used. The combination of MEDLINE with hand search restricted to letters and abstracts might be an acceptable alternative for reviewers with insufficient funds to support a full hand search process.