Evidence Summary

 

Google Scholar Could Be Used as a Stand-Alone Resource for Systematic Reviews

 

A Review of:

Gehanno, J. F., Rollin, L., & Darmoni, S. (2013). Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Medical Informatics and Decision Making, 13(1): 7. doi: 10.1186/1472-6947-13-7

 

Reviewed by:

Saori Wendy Herman

Education and Liaison Librarian

Hofstra North Shore-LIJ School of Medicine

Hempstead, NY, United States of America

Email: saori.w.yoshioka@gmail.com

 

Received: 27 Jan. 2015   Accepted: 27 Apr. 2015

 

cc-ca_logo_xl 2015 Herman. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

 

Abstract

 

Objective – To determine if Google Scholar (GS) is sensitive enough to be used as the sole search tool for systematic reviews.

 

Design – Citation analysis.

 

Setting – Biomedical literature. 

 

Subjects – Original studies included in 29 systematic reviews published in the Cochrane Library or JAMA.

 

Methods – The authors searched MEDLINE for any systematic reviews published in the 2008 and 2009 issues of JAMA or in the July 8, 2009 issue of the Cochrane Database of Systematic Reviews. They chose 29 systematic reviews for the study and included these reviews in a gold standard database created specifically for this project. The authors searched GS for the title of each of the original references for the 29 reviews. They computed and noted the recall of GS for each reference.

 

Main Results – The authors searched GS for 738 original studies with a 100% recall rate. They also made a side discovery of a number of major errors in the bibliographic references.

 

Conclusion – Researchers could use GS as a stand-alone database for systematic reviews or meta-analyses. With a couple improvements to the rate of positive predictive values and advanced search features, GS could become the leading medical bibliographic database.

 

Commentary

 

The number of studies examining the content coverage, accuracy, precision, and recall rate of Google Scholar (GS) compared to other medical bibliographic databases continues to increase. A majority of these studies conclude that although GS can be used as one of several bibliographic databases for literature retrieval, researchers should not use it as a stand-alone tool (Bramer, Giustini, Kramer, & Anderson, 2013; Giustini & Kamel Boulos, 2013). The authors of this study conclude otherwise, indicating that GS is sensitive enough to use as a stand-alone resource when performing systematic reviews.

 

This reviewer disagrees with the authors’ conclusion. In applying the EBL critical appraisal checklist (Glynn, 2006), several concerns arose about the study validity and applicability, focusing primarily on the methods used and the resulting conclusion.

 

The authors searched GS for the article titles of the references for 29 systematic reviews and retrieved a recall rate of 100%. Although the authors explain the methods used to select the systematic reviews and the inclusion and exclusion criteria for the references they included in their GS search, the authors do not explain in depth why the recall rate was so high. The only explanation mentioned is the ability for GS to access “the ‘invisible’ or ‘deep’ Web” through “agreements with publishers” (p. 4). The authors do not explain the method or algorithm, which can potentially affect the reproducibility of a GS search, used to access the “deep” Web.

 

Furthermore, the 100% recall rate merely indicates that GS was able to re-find articles that the authors knew already existed. This does not indicate whether GS’s search algorithm has better accuracy, precision, or recall rate compared to other medical bibliographic databases. Researchers should review all of these factors along with content coverage to make a final decision about whether a database is strong enough to use as a stand-alone resource.

 

Instead of re-finding existing references, the authors should have performed a search in GS and other medical bibliographic databases using a reproducible search string and compared the results. Doing so would have better assessed the elements (content coverage, accuracy, precision, and recall rate) needed to determine whether GS is an effective stand-alone resource.

 

One conclusion the reviewer agrees with is that GS’s coverage is more extensive than previously thought. GS has improved to the point that researchers can consider it as a possible resource to use when performing literature searches, systematic reviews, or meta-analyses. The caveat to this conclusion is to use GS in combination with other medical bibliographic databases.

 

The study provides insight into the growing usage of GS and the importance of paying close attention to the methodology of similar studies. Librarians performing literature searches, systematic reviews, or meta-analyses should be well informed about these types of research studies and utilize them to improve their own searching practices.

 

References

 

Bramer, W. M., Giustini, D., Kramer, B. M. R., & Anderson, P. F. (2013). The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Systematic Reviews, 2: 115. doi: 10.1186/2046-4053-2-115

 

Giustini, D., & Kamel Boulos, M. N. (2013). Google Scholar is not enough to be used alone for systematic reviews. Online Journal of Public Health Informatics, 5(2). doi: 10.5210/ojphi.v5i2.4623

 

Glynn, L. (2006). A critical appraisal tool for library and information research. Library Hi Tech, 24(3), 387-399. doi: 10.1108/07378830610692154