Tips from the Experts
Research Methods for Comprehensive Science Literature Reviews
Science Librarian
Mansfield Library
The University of Montana
Missoula, Montana
Barry.Brown@umontana.edu
Abstract
Finding some information on most topics is easy. There are abundant sources of information readily available. However, completing a comprehensive literature review on a particular topic is often difficult, laborious, and time intensive; the project requires organization, persistence, and an understanding of the scholarly communication and publishing process. This paper briefly outlines methods of conducting a comprehensive literature review for science topics.
Define and Refine Your Topic
Identifying potential topics of interest is typically the beginning of a literature review. A topic may be selected for you or you may need to choose one. Browsing textbooks, encyclopedias, review journals (e.g., the Annual Reviews series) or web sites that best fit the general category of your topic may help initially with selecting a topic and further refining it. You will want to begin by discovering general information about your topic and where the information fits within broader and narrower subject categories. You will want to identify major ideas, issues, controversies, and prominent researchers. The process of identifying a topic and outlining aspects of it moves ideally back and forth between finding and analyzing information and thinking critically about the topic (see Outline for Comprehensive Science Literature Reviews handout). Formulating a draft working title is a good first step for framing the topic for a particular publication or presentation with a specific audience in mind.
Before beginning to search for information resources, it is useful to write out concepts or facets of a topic on a search planner worksheet (see Search Planner -- Science Literature Databases handout). Each concept can serve as initial keywords and lead to additional related or synonymous keywords. A decision concerning the time period to cover for your topic should be made based on the importance of older research to the discipline and sometimes on the amount or type of literature found. In some disciplines, and at some institutions, this time frame will require going beyond electronic indexes into paper indexes. The interface and features of each electronic resource or database searched will structure the way search statements are entered and the ability to utilize Boolean search operators, truncation symbols, proximity operators, phrase searching, and limiters. Successful search strategies must be adapted for use in each new database. Recording precise methods of searching particular resources is as important as keeping accurate laboratory or field notes so that a particular search can later be replicated or revised.
The first results of searching an information resource are rarely the best results. It is worth reviewing the first page or two of results and examining how many of them are relevant or of interest to your topic. This assessment can be accomplished by title, by abstract, by other information in the full record display, and by obtaining the full article or source. Focusing on the full record display of one or two results that initially appear to be of most interest and closely analyzing all the words in the full records will often lead to a better reformulation of a search statement with keywords or subject headings scavenged from those records. Some databases have thesauri, which can also help with this process. Modifying and revising search statement words and operators should be an iterative process and may require several attempts in each new information resource.
When beginning a search, it is often helpful to try broadening and narrowing search statements. Use the Boolean operators to retrieve between 10 and 100 results with over half the records displayed on each page relevant to the topic. There are various suggestions for improving bibliographic database search results depending on whether you are finding no citations (i.e., are you in the right database; do you have misspelled terms; are your terms too specific?). Perhaps you have found too few citations (i.e., don't use multi-word phrases; decrease the number of concepts combined with "and"), or too many citations (i.e., decrease the number of related terms combined with "or"; increase the number of concepts combined with "and"; add some limits to your search). See Improving Bibliographic Database Search Results handout. This search challenge can help with thinking through the narrowing and broadening of the topic itself.
Search All Relevant Sources Comprehensively and Efficiently
Journal articles are the primary vehicle of communication in most science disciplines and, therefore, journal indexes and e-journal packages are critical information resources. To identify and remain knowledgeable about the main indexes to the science literature and many other specific resources for each science discipline, there are science librarians, library database lists and guides arranged by subject, and specific literature guide books to assist you. Journal and bibliographic indexes typically allow keyword searching of article titles, abstracts, and subject headings or descriptors. E-journal packages (e.g., JSTOR; ScienceDirect; SpringerLink; Wiley Interscience) often allow keyword searching of the full text of articles as well. The ability to keyword search full text enables a new level of searching. While producing many irrelevant results, full-text searches can also find information not previously discoverable by searching indexes. Multidisciplinary indexes, such as Google Scholar, Scopus, and Web of Science, which analyze article bibliographies and track citations (i.e., who is citing who) are valuable tools for identifying the most cited and presumably important papers on particular topics. Once a relevant article is found, by examining its bibliography additional older results can often be identified for your topic. Conversely, once a relevant article is found, by examining who has cited it additional newer results can often be identified for your topic. A caveat is that more recent articles will not have had the time to build up the numbers of citing articles that older articles have.
After searching several indexes or e-journal packages and identifying the best articles for your topic, it may become apparent that the best articles are consistently found in a handful of journals. Once you have identified the best journals for your topic, then you should directly browse the table of contents of those journals for the last few years, or longer depending on the scope of your topic. No search strategy is perfect and many indexes have selective indexing of journal contents or have lag times before recent citations appear. Only by directly browsing the most important journals can you be assured of finding all the current, relevant articles on your topic from them. Many journals allow setting up e-mail alerts or RSS feeds with automatic notification of table of contents information from new issues. Similarly, many databases allow setting up automatic searches on a weekly or monthly basis. And many web sites have RSS feeds for notification of new content added.
Before a research project leads to a published or discoverable presentation or publication, you may find out about it by searching grant databases. Major research granting agencies for the United States, such as the National Science Foundation, and the National Institutes of Health have searchable grants databases that can identify ongoing research projects. These records often have abstracts and describe the project and identify the individuals and institutions involved. Other current awareness resources are newspaper indexes, blogs, and internet discussion groups. Identifying a leading researcher or expert on a topic and then contacting them for a brief discussion or to answer a focused question can be another excellent way to find out about new developments. And web search engines can also uncover all sorts of current, miscellaneous information. To find information on web sites there are a variety of general web search engines, meta-search engines, and specialized search engines. Some specialized search engines for science topics currently include: Science.gov, Scirus, Scitopia.org, and WorldWideScience.org.
Many bibliographic indexes include records for books, government documents, theses and dissertations, media materials, and other gray literature. However, some indexes only include journal articles and even those which do include other types of information sources usually are very selective about what they include. The most comprehensive information resource for books, government documents, theses and dissertations, media materials, and gray literature is WorldCat, the global union catalog of over 70,000 libraries. Additionally, most individual libraries have their catalogs publicly available for searching online. The key to a comprehensive search of library catalogs is utilizing subject headings as part of a search strategy. A new level of searching for books beyond catalog records is now available from Google Books, which allows keyword searching of the full text of an unknown number (but currently believed to be over 7 million) of books.
There is no single, all encompassing, comprehensive database for the sciences or for most individual science disciplines. There is overlap between many bibliographic indexes, but to search the literature comprehensively for a specific topic typically requires the use of multiple information resources (see Main Indexes to the Science Literature handout). A good strategy for many science topics is to search the following: Scopus or Web of Science; Google Scholar; one or two specialized databases for a particular science discipline; Google Books; and WorldCat. That combination plus scouring bibliographies, browsing the table of contents of the most relevant journals, searching web sites using general and science web search engines, and talking to experts is likely to comprehensively uncover all types of relevant information sources. Note that many libraries offer federated, simultaneous searching of many databases at once. Those searches can be efficient ways to initially test the relevance of particular databases for specific topics; however, once the best databases are identified, it is optimal to individually search them, using the advanced search option, to take advantage of their unique, powerful search features.
Find and Evaluate and Manage the Information
As you search databases and analyze search results, you should continually review your search statement and modify it to provide better results. There is a trade off between comprehensiveness (retrieving all the relevant results mixed with lots of other irrelevant results) and precision (retrieving only relevant results, but missing lots of other relevant results). You will want to craft a search statement for each database that strikes a balance between comprehensiveness and precision. This strategy is often accomplished by utilizing various limiters and subject headings. Depending on your topic, you may wish to limit your results to scholarly sources and peer-reviewed or refereed articles. Scholarly sources are not necessarily better than popular sources; the appropriateness of the source depends on the topic and the audience.
Evaluating the results of your searches can be done based on many different criteria including these important concepts: relevance to your topic, objectivity, accuracy and credibility of the information source, significance of the research, authority of the author, date published, the type of information source, etc. A skeptical researcher is a better researcher. On the other hand, there is some evidence that individuals performing literature searches often only see those studies confirming their pre-existing beliefs or biases. So the ability to really look objectively at all results and weigh the evidence on a topic is important. There are specific methods, typically from medical disciplines, for overcoming researcher bias and performing systematic literature reviews.
Retrieving the information source from the citation via the open web or through a library site or requesting document delivery can be challenging some times. Librarians are always eager to assist with that process. There are books and courses devoted to techniques for critically reading and reviewing research articles and then organizing and synthesizing many information sources into a new review.
Nevertheless, understanding technical, research articles can prove quite challenging for those lacking a PhD in the discipline. There may only be parts of some primary articles and sources that are useful. Secondary and tertiary information sources may have to be relied on sometimes.
It is critical to carefully store citations and papers and annotations of them. With a large literature review, it is easy to lose the connection between notes and sources. There are a number of commercial and open source bibliographic management tools and social bookmarking sites available to assist with this process including: 2collab, CiteULike, Connotea, EndNote, EndNote Web, Mendeley, Refworks, and Zotero. Keeping track of irrelevant citations that initially look relevant but aren't, in addition to relevant citations, can avoid wasted time spent repeatedly evaluating some of the same results from multiple database searches.
Synthesize the Literature and Integrate it Into Your Writing or Presenting
Ideally, there is continual movement back and forth between searching for information on a topic, evaluating results, adapting search strategies, narrowing and broadening topics, reviewing and synthesizing the literature and integrating it into a publication or presentation which then triggers new questions leading to further searches for additional information. Knowing when to stop searching is sometimes as hard or harder than knowing where to begin searching, particularly when information is not found on a specific question or subtopic. Finding the best sources repeatedly in different databases, with a variety of search strategies, often leads to confidence that the topic has been well searched.
Choosing the appropriate type and venue of publication or presentation is important for the impact of a literature review and typically dictates the bibliographic style required. The Council of Science Editors (CSE) style is a good choice if there is no discipline or publication specific requirement. There are many books and guides to assist with style questions. Having a good model to emulate and tailoring your writing or presentation to your audience is one of the keys to its effectiveness.
Appendix
Handouts:
Appendix 1: Outline for Comprehensive Science Literature Reviews
Appendix 2: Search Planner - Science Literature Databases
Appendix 3: Improving Bibliographic Database Search Results
Appendix 4: Main Indexes to the Science Literature
