Using Bibliometrics to Demonstrate the Value of Library Journal Collections 410 Using Bibliometrics to Demonstrate the Value of Library Journal Collections Christopher W. Belter and Neal K. Kaske Christopher W. Belter is Informationist, National Institutes of Health Library, Bethesda, Maryland; e- mail: Christopher.Belter@nih.gov. Neal K. Kaske is Retired Director, National Oceanic and Atmospheric Administration Central and Regional Libraries, Silver Spring, Maryland; e-mail: Neal.Kaske@noaa.gov. This is a work of the U.S. Government and is not subject to copyright protection in the United States. Although cited reference studies are common in the library and information science literature, they are rarely performed in nonacademic institutions or in the atmospheric and oceanic sciences. In this paper, we analyze more than 400,000 cited references made by authors affiliated with the National Oceanic and Atmospheric Administration between 2009 and 2013. Our results suggest that these methods can be applied to research libraries in a variety of institutions, that the results of analyses performed at one institution may not be applicable to other institutions, and that cited reference analyses should be periodically updated to reflect changes in authors’ referencing behavior. s scientific journal subscription prices rise at rates above the Consumer Price Index and the Higher Education Price Index, and as library subscription budgets mostly remain flat or increase more slowly than journal prices, academic and research libraries face increasing pressure from their insti- tutions to justify their journal subscription budgets.1 At the same time, the rise of big data and analytics has resulted in renewed interest in evidence-based decision making on the part of institutional leaders. This interest means that anecdotal evidence of the value of library journal collections no longer carries the weight it once did, prompting libraries to provide quantitative evidence of value to support their claims. Many academic libraries have responded to these trends by analyzing the use of their collections through electronic usage statistics or through the cited references of their insti- tution’s faculty and students.2 Although government research libraries perform a similar function as academic libraries in that they support the information needs of institutional researchers, published collection analyses performed in government library settings are extremely rare. In this paper, we analyze the cited references made by intramural research- ers at the National Oceanic and Atmospheric Administration (NOAA) to demonstrate the value of the NOAA libraries system to NOAA scientists and to demonstrate that various techniques developed in the academic library setting can be applied at other types of research libraries and vice versa. We also compare our results to the few citation analysis studies that have been performed in the atmospheric and oceanic sciences. doi:10.5860/crl.77.4.410 crl16-758 Using Bibliometrics to Demonstrate the Value of Library Journal Collections 411 The NOAA libraries system consists of 28 partially to completely autonomous librar- ies located across the United States. These libraries differ greatly in terms of size, report- ing structures, and funding sources, ranging from unstaffed reading rooms containing a few hundred items for in-person use, to one- and two-person libraries of hundreds to thousands of items serving NOAA scientists at specific laboratories and science centers, to the NOAA Central Library, with 14 full-time staff and a collection of more than 400,000 items. Whereas most of these libraries only provide services to the specific NOAA laboratory or science center that provides their funding, the NOAA Central and Regional Libraries—located in Silver Spring, Miami, and Seattle—provide services to NOAA staff across the agency. The services provided by these libraries are similar to those provided by academic libraries to their faculties: access to collections, refer- ence and literature searching, custom information support, and so forth. The NOAA libraries are working with the newly established NOAA Library Advisory Committee to identify areas in which cooperation and coordination between the libraries would enhance access to, and development of, library collections and services. Demonstrat- ing the value of library journal collections to the entire agency is one of those areas. Literature Review Our method of demonstrating that value through the analysis of intramural research citations is well established in the library and information science literature. Local referencing patterns have been used to inform library collection development since at least the early 1970s.3 Although much of this literature focuses on the references made by undergraduate and graduate students, and therefore is not directly relevant to a government agency like NOAA, a number of studies have been performed that analyze faculty publications in various scientific areas. McCain and Bobick and Davis analyzed references by faculty and students in biology, Tsay and Rethlefsen studied references in biomedicine and public health, Choinski analyzed references in pharmacy, Ortega analyzed references in chemistry, LaBonte analyzed references in nanoscience, and Stephens et al. analyzed references in aerospace engineering.4 Other authors analyzed cited references across multiple subject areas or across their entire institutions.5 Hoffmann and Doucette reviewed 34 cited reference analysis studies published from 2005 to 2010 to assess the transparency and rigor of their methodologies.6 They found that most of these studies do not provide sufficient detail for their results to be replicated by others. They also found that many of the decisions made by the authors of these studies were not adequately explained by a clear and defined rationale. Hoff- man and Doucette recommend that future cited reference studies not only provide sufficient detail and explanation about their methods, but also present their results in ways that facilitate comparison across studies. Despite the frequency with which citation analysis methods have been used, Reth- lefsen is the only citation analysis we are aware of that was performed in a government library setting.7 Citation analyses of faculty publications in the marine, fisheries, and atmospheric sciences are similarly rare. Walcott analyzed more than 5,000 cited refer- ences by faculty at the Marine and Sciences Research Center at the State University of New York at Stony Brook to identify the most frequently referenced journals for inclusion in the information center’s collection.8 Ibeun identified core fisheries journals for acquisition by Nigerian fisheries libraries through a targeted survey of Nigerian fisheries scientists and found that the distribution of recommended titles conformed to Bradford’s law of scattering.9 More recently, Kimball et al. analyzed more than 5,000 cited references of faculty publications from the Department of Atmospheric Sciences at Texas A&M University to identify the sources, formats, and ages of the publications referenced.10 412 College & Research Libraries July 2016 Finally, a few studies suggest that there are positive and significant correlations between local cited reference patterns and other measures of journal use. Blecic com- pared in-house use, circulation, and citation by faculty at the University of Illinois at Chicago’s Library of the Health Sciences and found positive and significant correlations between the measures, although the strength of the correlations, ranging from 0.591 to 0.776, was not particularly high.11 McDonald analyzed journals owned by the Cali- fornia Institute of Technology to compare local print use, online use, local publication counts, and local references and found that online use was a significant variable in predicting local cited reference patterns.12 Knowlton et al. found that local citation and usage statistics are highly associated with faculty valuations for titles that faculty use frequently and value highly but are not highly associated with titles that faculty use infrequently or rate as lower value.13 These studies suggest that local cited reference statistics can be used to identify titles of high value to local faculty. Methods References cited by NOAA-authored articles were retrieved from the Web of Science, Science Citation Index Expanded database (WoS) during March 2014. We used the following search string to identify articles by NOAA-affiliated authors published each year: AD = (NOAA OR “nat* ocean* atmos* adm*” OR “nat* mar* fish* serv*” OR NMFS) AND PY = 2009 We specifically included the National Marine Fisheries Service (NMFS) in our search string because we know from prior experience of identifying publications by NOAA authors that authors affiliated with NMFS laboratories and offices are less likely to include NOAA in their stated affiliations than NOAA authors affiliated with other branches of the agency. Although our search string does not retrieve publications by NOAA authors who listed their NOAA laboratory or division but did not include any reference to NOAA or NMFS in their stated affiliation, we feel that, given the large number of articles and references the string did retrieve, it is unlikely that excluding such publications has significantly affected our results. In addition, the more restrictive search string has the advantage of increasing the precision of the articles retrieved, meaning that our analyses are unlikely to be affected by false matches. We then added the results of each search to our “Marked List,” selected the Authors, ISSN, Title, Cited References, Times Cited, Accession Number, and Source fields for export and exported all of the records to “Other File Formats.” We then copied the resulting files into a single text file for each year and then saved it as an .isi file for later import into the Science of Science (Sci2) Tool.14 To analyze the cited references by year and by journal, we converted the text file for a single year into a new .csv file and opened the .csv file with Excel, which parsed the cited references into columns for further analysis. We discarded all of the columns except the year and journal, sorted the references by publication year, and discarded the extraneous rows as well as a small number of incorrectly formatted references. We then performed some rudimentary data cleaning on the titles of referenced journals to increase the accuracy of our final counts. We standardized variant forms of the journal title Eos, Transactions of the American Geophysical Union, since this standard- ization is not performed automatically by WoS. We also modified the cited references to two journals that had changed titles, so that all references to these journals refer to their current titles. That is, we changed references to the Journal of Applied Meteorol- ogy into references to the journal’s current title, the Journal of Applied Meteorology and Using Bibliometrics to Demonstrate the Value of Library Journal Collections 413 Climatology, and changed references to Estuaries to become references to Estuaries and Coasts. References to single journals that have subsequently split into multiple parts, such as the Journal of Geophysical Research and Deep-Sea Research, were allowed to stand as referenced, since it was impossible to assign these citations to a specific part of the current journal. Finally, we used various functions in Excel to calculate the number and percent- age of references per year and per journal. We then repeated this process for each additional year. Next, we created journal co-citation networks for each of the five years in our analysis. Journal co-citation networks are an elaboration of document co-citation networks in which nodes represent journals, and edges, or connections between nodes, represent journal co-citations.15 In such networks, journals are connected if at least one current article referenced previous articles published in both journals in the same current article. Journal co-citation networks have been frequently used to map the intellectual structure of scientific research in general or within certain disciplines and used to identify journals for collection development.16 To create a journal co-citation network for a single year, we loaded the original .isi file for that year into Sci2, which converted it into a .csv file, opened the new .csv file, and copied the “Cited References” column into a new Excel file. We then parsed the cited references using the “Text to Columns” feature in Excel, removed all of the cita- tion information except the journal name, and saved the file as a .csv. We then opened the .csv file in Notepad++, replaced all of the commas (,) with pipes (|), removed the excess pipes, and then saved the file as a .csv file. We then loaded the new .csv file into Science of Science Tool and created a co-occur- rence network on the modified “Cited References” column, using a custom function file to count the number of articles that cited each journal. Once the network had been created, we removed edges, or journal co-citations, with a weight of less than 500 to focus the network on the strongest co-citation relations between journals, extracted only the largest connected component of the network, and visualized the resulting network using Gephi.17 We then repeated this process for the remaining years. Results We identified 8,737 articles published by NOAA-affiliated authors from 2009 to 2013. These articles contained a total of 402,126 cited references, which formed the basis of our analysis. Figure 1 shows the total number of articles, cited references, and aver- age number of cited references per article per year in our document set. Both the total number of articles published per year and the number of cited references per year increased substantially during this time period, with the number of articles increas- ing from 1,532 in 2009 to 1,981 in 2013 (a 29% increase) and the number of references increasing from 66,753 in 2009 to 94,244 in 2013 (a 41% increase). The average number of references per paper, however, grew at a more modest pace, increasing from 43.57 in 2009 to 47.53 in 2013 (a 9% increase). Next, we analyzed the age of publications referenced by NOAA authors. Figure 2 plots the percentage of cited references in each year by the age of the publication referenced at the time the reference was made. That is, approximately 7 percent of the cited references made in 2013 were to articles that were one year old. The plotted curves are remarkably consistent over the five years analyzed, with refer- ences to publications peaking at 2 to 3 years after their original publication date, decaying at a relatively linear rate between years 4 and 15, and gradually flattening out after year 20. Figure 3 shows the cumulative distribution of all cited refer- ences by approximate quartile. The distribution shows that the citation half-life 414 College & Research Libraries July 2016 of NOAA cited references is approximately 8 years, with 50 percent of the cited references being to publications 0 to 7 years old, 25 percent to publications 8 to 15 years old, and so on. This half-life is normal for publications in the NOAA- related disciplines, since the 2012 edition of Journal Citation Reports gives citation half-lives of between 7.7 years for Meteorology and Atmospheric Sciences and 9.4 years for Oceanography. FIGURE 1 Total Numbers of Articles, References per Article, and Cited References per Year for NOAA-Authored Articles from 2008 to 2013 FIGURE 2 Percentage of Cited References in Each Year per Year of Age of the Publication Referenced Using Bibliometrics to Demonstrate the Value of Library Journal Collections 415 Next, we analyzed the journals referenced by NOAA authors. Figure 4 plots the cumulative percentage of cited references per journal for the most frequently referenced journals in each year. Even more than with the distributions of references per year, the distribution curves per journal are remarkably consistent across years. It is interesting to note that 15 percent of the more than 400,000 cited references in our analysis were to articles published in just 5 journals, while 25 percent of the references were to 10 journals, and 50 percent of the references were to 60 journals. These distributions are consistent with those predicted by Bradford’s law of scattering, which suggests that the vast majority of the references in any publication set are to a relatively small number of “core” journals, and the remaining references are more widely dis- persed among a larger number of peripheral journals. This suggests that the NOAA Central Library can provide for the majority of the references made by NOAA authors with a relatively small collection of core journals. A collection of just 60 journals would provide for 50 percent of the cited references made during 2009–2013, and a collection of 200 journals would provide for 66 percent of all cited references during this time period. We next analyzed the number of references per journal per year to determine if the individual journals that make up this core in one year remained the same in sub- sequent years. That is, we attempted to determine if the journals that make up this FIGURE 3 Age Distribution, in Years, of All the Cited References in Our Analysis FIGURE 4 Cumulative Percentage of Cited References per Journal for the 75 Most Frequently Referenced Journals per Year 416 College & Research Libraries July 2016 core are fixed or if they change from year to year. Figure 5 charts the number of cited references per journal per year for the 50 most frequently referenced journals in 2013. We focused on journals frequently referenced in 2013 both to see if journals that are currently important to NOAA authors were also important in previous years and to identify journals that may have gained importance over this time period. Figure 5 suggests that, for the most part, those journals most frequently referenced in 2013 were also frequently referenced in all of the previous years in our analysis. Although some journals changed position from year to year, the 12 most frequently FIGURE 5 Number of Cited References per Journal per Year for the 50 Most Frequently Referenced Journals in 2013 (Note the Logarithmic Scale of the Y-Axis) TABLE 1 The Most Frequently Referenced Journals by NOAA Authors from 2009 to 2013 Journal References % of Total Journal of Geophysical Research—Atmospheres 19,128 4.76 Geophysical Research Letters 13,383 3.33 Journal of Climate 12,339 3.07 Monthly Weather Review 9,679 2.41 Journal of the Atmospheric Sciences 8,121 2.02 Science 7,988 1.99 Atmospheric Chemistry and Physics 7,738 1.92 Marine Ecology Progress Series 6,968 1.73 Canadian Journal of Fisheries and Aquatic Sciences 5,981 1.49 Bulletin of the American Meteorological Society 5,732 1.43 Nature 5,732 1.43 Journal of Geophysical Research—Oceans 5,547 1.38 Using Bibliometrics to Demonstrate the Value of Library Journal Collections 417 cited journals in 2013 were also the 12 most frequently cited journals in each of the previous years. We list these journals in table 1. The remainder of the journals shown in figure 5 remained highly referenced from year to year, although the yearly variability in reference counts increased as the total number of references decreased, suggesting that this second tier of journals is more susceptible to fluctuations in research focus by NOAA authors. Finally, we created journal co-citation networks for each of the five years in our analysis to determine whether these core journals were referenced by all NOAA au- thors, or if there were differences in what journals were considered to be core by dif- ferent NOAA authors. A visualization of the network derived from articles published in 2013 is given in figure 6. As with the distributions of references per year and journal, the overall structure of the network remained fairly constant over the five years, with journals related to weather, climate, and the atmospheric sciences clustered on the left and journals related to marine biology and ecology clustered on the right. These two clusters of journals are connected by interdisciplinary and oceanography journals, although the interdisciplinary journals tend to be more frequently referenced in combination with the atmospheric science journals than with the marine biology journals. Within the atmospheric sciences cluster, there also seems to be a loose divide between basic journals covering the physical composition and processes of the atmosphere and ap- plied journals dealing with weather and climate. A similar divide seems to occur in the marine biology cluster between basic journals on marine biology and ecology and applied journals on fisheries and fishery policy. Finally, there seem to be relatively few FIGURE 6 Journal Co-Citation Network Derived from the Cited References of NOAA-Authored Articles Published in 2013 Font size indicates the number of NOAA articles that referenced articles in each journal. Line size and darkness indicates higher numbers of co-citations between the connected journals. To focus the visualization on the strongest journal relationships, connections of less than 500 co-citations were removed and only the largest connected component is shown 418 College & Research Libraries July 2016 connections between these two clusters of journals—that is, few NOAA articles refer- ence both marine biology and atmospheric science journals—suggesting that NOAA authors tend to work along disciplinary lines. Discussion We find that access to substantial numbers of scientific publications is necessary for NOAA authors to produce new articles. Our analysis suggests that access to between 40 and 50 publications is necessary for an author to produce a single new article. Our results also suggest that NOAA authors required access to more than 90,000 publications in 2013 alone to produce nearly 2,000 original articles. Of course, it is likely that some of these references are duplicates, since some NOAA authors undoubtedly published multiple articles during this time period that cite the same prior publications. Despite this, however, our results do indicate that NOAA authors require access to tens of thousands of publications every year to produce their original research. Since NOAA requires its employees to publish as proof of their scientific activity, our results suggest that NOAA thereby requires access to scientific information to support the production of these articles. Furthermore, since our results indicate the number of publications, references per publication, and cited references have all increased during the past five years, it is logical to conclude that this need has grown over time, and it seems likely that it will continue to grow in the near future. The emphasis on recent publications in NOAA authors’ referencing patterns—refer- ences peak after two years, and nearly a quarter of all references are to publications less than four years old—is not surprising. Nearly 50 years ago, de Solla Price found that the majority of references from current papers are to other relatively recent papers, so this aspect of NOAA authors’ referencing behavior is to be expected.18 Perhaps more surprising is the relative importance of older publications in addition to the emphasis on current publications. Nearly a quarter of all the cited references in our analysis, or more than 90,000 cited references during these five years, are to publications 16 or more years old. Taken together, these findings suggest that NOAA authors not only require access to the latest scientific publications; they also need continuing access to those publications for decades. A similar trend in the ages of cited references was reported by Kimball et al. for atmospheric science faculty publications at Texas A&M.19 They too found that, although the cited references at their institution emphasized recent publications, there were also numerous references to publications more than 16 years old. Walcott did not analyze the ages of the references analyzed, so our results could not be compared with hers.20 The general pattern identified in our analysis and in that of Kimball et al. has also been observed by numerous cited reference analyses in other disciplines, although the age of the peak and the rate of decay vary between disciplines.21 Our analysis of the journals referenced by NOAA authors suggested that, although the identity of the journals most highly referenced by NOAA authors remained fairly constant over time, the number of references to two journals in particular increased by more than 500 percent during this time period. These journals are Nature Geoscience and Plos One. References to Nature Geoscience increased from 45 references in 2009 to 312 references in 2013, a 593 percent increase. Since the first issue of Nature Geoscience was published in January 2008, it is perhaps unsurprising that references to the jour- nal would increase as increasing numbers of articles were published in the journal, but it is interesting to note the speed at which this new journal established itself as a major source of information for NOAA authors. References to Plos One, on the other hand, increased 1,729 percent during this time period, from 24 references in 2009 to 439 references in 2013. The reasons for this remarkable increase are not clear. Certainly Using Bibliometrics to Demonstrate the Value of Library Journal Collections 419 the total number of articles published in Plos One increased dramatically during this time period—from more than 4,000 articles in 2009 to more than 31,000 articles in 2013—so, as with Nature Geoscience, the increasing availability of articles to reference may have driven some of this change. But mere increases in the number of articles published in both journals cannot entirely explain the increases in NOAA references, because articles published in the journals had to be brought to the attention of NOAA authors, be sufficiently relevant to their work, and be of sufficient quality for them to be referenced. Additional research seems necessary to fully understand the reasons for these increases and to determine if such changes are reflected in referencing pat- terns at other institutions. Smaller increases in the number of references to other journals may signal shifts in the research focus of NOAA authors. Increases in the number of references to journals such as Climatic Change (a 214% increase), Global Change Biology (a 113% increase), and Climate Dynamics (a 150% increase) during this time period may signal a shift toward research on climate change and its effects on terrestrial and marine ecosystems. Other notable increases were to journals such as Atmospheric Chemistry and Physics (a 115% increase), IEEE Transactions on Geoscience and Remote Sensing (a 112% increase), Fisheries Research (a 101% increase), and the Journal of Hydrometeorology (a 106% increase). Our results suggest that analyzing the rate of references to various journals over time could be used to identify shifts in the research focus of authors at academic institutions as well. Taken together, our analyses suggest that there is a set of 12 core journals for NOAA authors that remained constant throughout this time period and were consistently referenced more often than any other journals in our analysis. This identification of a core set of journals in local cited references was also reported by Walcott, Ibeun, Kimball, et al., and by studies in a range of other disciplines.22 Beneath this core set of journals were a large number of more peripheral journals whose identity and number of references were more fluid, although the magnitude of references to each journal remained relatively constant during the five years. Nevertheless, increases in references to certain journals suggest that the identity of the journals in this core is not firmly fixed and that journals that were peripheral in the past may become core in the future. This suggests the need to repeat cited reference analyses over time to identify such changes not only at NOAA but at other institutions as well. Our analysis also suggests that there are different sets of core journals for different communities of NOAA authors. The central journals for atmospheric science authors include four American Meteorology Society journals (Journal of Climate, Monthly Weather Review, Bulletin of the American Meteorological Society, and Journal of the Atmospheric Sci- ences) and three American Geophysical Union journals (Geophysical Research Letters, Journal of Geophysical Research—Atmospheres, and Journal of Geophysical Research—Oceans). The central journals for marine biology authors, on the other hand, are Canadian Journal of Fisheries and Aquatic Sciences and Marine Ecology Progress Series. Only a few journals like Nature and Science are referenced by both communities of authors. Many of the core journals that we identified were also listed in previous studies. The core journals for NOAA’s atmospheric science authors are similar to those reported by Kimball et al.23 The high usage of the American Meteorology Society journals and the American Geophysical Union journals, as well as other journals like Atmospheric Environment, Quarterly Journal of the Royal Meteorological Society, and Atmospheric Chem- istry and Physics, by NOAA authors is also reflected in their study. There are, however, differences in emphasis between their results and ours, with NOAA authors using more journals related to weather forecasting and Texas A&M authors using more journals related to optics. Our list of core journals for fisheries science is also similar to, but not exactly the same as, the list of fisheries journals identified by Ibeun, with 420 College & Research Libraries July 2016 his list emphasizing aquaculture journals and our list emphasizing marine ecology.24 Our and Ibeun’s lists are also similar to, but slightly different from, that of Walcott, which has more titles related to geochemistry and geology.25 These results suggest that, although there may be broad agreement between institutions on the most highly used journals in a particular discipline, the actual journals most often used may differ between institutions depending on local interests and priorities. In addition, all three lists differ, in some cases substantially, from those generated by the 2012 edition of Journal Citation Reports for the Meteorology & Atmospheric Sciences and Fisheries subject categories according to total cites, Journal Impact Factor, and 5-Year Journal Impact Factor. This suggests that the journals most often referenced by an entire discipline do not necessarily align with those most often referenced at a particular institution. This calls into question the appropriateness of using aggregate referencing statistics like the Journal Impact Factor to make collection development decisions at a particular institution, although more data are needed to examine potential differences at other institutions and in other disciplines. Finally, although this analysis was performed in a government library, the similarities between our results and those performed at academic libraries suggest that this general method can be applied at a range of research libraries at academic, governmental, and corporate institutions. The journal co-citation method in particular could be applied in an academic library to identify the degree of overlap between the resources used by different faculty departments within the institution. It could also be used to identify potential differences between the resources used by faculty and those used by gradu- ate students in their theses and dissertations. Our method of tracking references per journal per year could also be used in academic libraries to identify shifts in research focus at their institutions. Finally, the amounts of total references and references by age could be used to justify academic library journal collections, set thresholds for journal subscription vs. interlibrary loan, and identify less-used journals for potential cancellation. Conclusion Our analysis of the references cited by NOAA-affiliated authors in their articles pub- lished between 2009 and 2013 produced a number of potentially useful results for use within our agency. We found that NOAA authors require access to tens of thousands of scientific publications every year to produce their original research. Although the majority of these publications were published during the past 8 years, NOAA authors required access to substantial numbers of older publications as well. A small number of core journals published the majority of the publications referenced by NOAA authors, and, for the most part, these core journals remained highly referenced throughout the years in our analysis. However, we found substantial differences in the journals ref- erenced by authors from different disciplines, with relatively few journals referenced by authors across the agency. These findings largely support and confirm the results of cited reference studies performed in similar disciplines at academic institutions. Our analysis also produced several results of broader interest to the library com- munity. Our results demonstrate that the technique of analyzing cited references by institutional scientists to gain insights for collection development and justification purposes can be successfully applied to research-oriented libraries both inside and outside the academic environment. Our analysis also suggests that there may be differ- ences in the journal referencing patterns of authors in the same discipline at different institutions, suggesting that the findings from an analysis performed at one institution may not be applicable to authors at other institutions. This also calls into question the use of general citation indicators like the Journal Impact Factor for collection develop- Using Bibliometrics to Demonstrate the Value of Library Journal Collections 421 ment decisions, since the priorities and referencing patterns of an entire discipline may not be reflected in those of a particular institution. Finally, our analysis suggests that authors’ referencing patterns can change somewhat over time, suggesting that cited reference analyses performed at a particular institution should be updated periodically to account for such changes. Acknowledgements Thanks to Cindy Clark of the NIH Library for editing a previous version of this paper. Thanks also to two anonymous reviewers for comments that improved the clarity and presentation of the paper. Notes 1. Charles B. Lowry, “ARL Library Budgets after the Great Recession, 2011–2013,” Research Library Issues 282 (2013): 2–12. 2. Michael J. Kurtz and Johan Bollen, “Usage Bibliometrics,” Annual Review of Information Science and Technology 44 (2010): 3–64. 3. George R. Chambers and James S. Healey, “Journal Citations in Master’s Theses: One Measurement of a Journal Collection,” Journal of the American Society for Information Science 24, no. 5 (1973): 397–401. 4. Katherine W. McCain and James E. Bobick, “Patterns of Journal Use in a Departmental Library: A Citation Analysis,” Journal of the American Society for Information Science 32, no. 4 (1981): 257–67; Philip M. Davis, “Where to Spend Our E-Journal Money? Defining a University Library’s Core Collection through Citation Analysis,” portal: Libraries & the Academy 2, no. 1 (2002): 155–66; Ming-yueh Tsay, “The Relationship between Journal Use in a Medical Library and Citation Use,” Bulletin of the Medical Library Association 86 (1998): 31–39; Melissa L. Rethlefsen, “Citation Analysis of Minnesota Department of Health Official Publications and Journal Articles: A Needs Assess- ment for the RN Barr Library,” Journal of the Medical Library Association 95, no. 3 (2007): 260–66; Elizabeth M. Choinski, “Journal Use in Pharmacy: A Citation Analysis of Faculty Publications at a School of Pharmacy,” Science & Technology Libraries 27, no. 3 (2007): 53–64; Lina Ortega, “Age of References in Chemistry Articles: A Study of Local Authors’ Publications from Selected Years, 1975–2005,” Science & Technology Libraries 28, no. 3 (2008): 209–46; Kristen B. LaBonte, “Citation Analysis: A Method for Collection Development for a Rapidly Developing Field,” Issues in Sci- ence & Technology Librarianship 43 (2005); Jane Stephens, David E. Hubbard, Carmelita Pickett, and Rusty Kimball, “Citation Behavior of Aerospace Engineering Faculty,” Journal of Academic Librarianship 39, no. 6 (2013): 451–57. 5. Lutishoor Salisbury and Jeremy S. Smith, “The Use of Web of Knowledge to Study Pub- lishing and Citation Use for Local Researchers at the Campus Level,” Collection Management 35, no. 2 (2010): 69–82; Lea Currie and Amalia Monroe-Gulick, “What Do Our Faculty Use? An In- terdisciplinary Citation Analysis Study,” Journal of Academic Librarianship 39, no. 6 (2013): 471–80; Concepción S. Wilson and Carol Tenopir, “Local Citation Analysis, Publishing and Reading Patterns: Using Multiple Methods to Evaluate Faculty Use of an Academic Library’s Research Collection,” Journal of the American Society for Information Science & Technology 59, no. 9 (2008): 1393–408. 6. Kristin Hoffmann and Lise Doucette, “A Review of Citation Analysis Methodologies for Collection Management,” College & Research Libraries 73, no. 4 (2012): 321–35. 7. Rethlefsen, “Citation Analysis of Minnesota Department of Health,” 260–66. 8. Rosalind Walcott, “Serials Cited by Marine Sciences Research Center Faculty, University at Stony Brook, 1986–1991,” Science & Technology Libraries 14 (1994): 15–33. 9. M.O. Ibeun, “Applied Bibliometrics and Cooperative Acquisition, as Tools for Selecting Journals and Sharing in Nigerian Fisheries Libraries,” African Journal of Library, Archives & Infor- mation Science 11 (2001): 39–47. 10. Rusty Kimball, Jane Stephens, David Hubbard, and Carmelita Pickett, “Citation Analysis of Atmospheric Science Publications by Faculty at Texas A&M University,” College & Research Libraries 74, no. 4 (2013): 356–67. 11. Deborah D. Blecic, “Measurements of Journal Use: An Analysis of the Correlations between Three Methods,” Bulletin of the Medical Library Association 87, no. 1 (1999): 20–25. 12. John D. McDonald, “Understanding Journal Usage: A Statistical Analysis of Citation and Use,” Journal of the American Society for Information Science and Technology 58, no. 1 (2007): 39–50. 422 College & Research Libraries July 2016 13. Steven A. Knowlton, Adam C. Sales, and Kevin W. Merriman, “A Comparison of Faculty and Bibliometric Valuation of Serials Subscriptions at an Academic Research Library,” Serials Review 40, no. 1 (2014): 28–39. 14. Science of Science (Sci2) Tool. Indiana University and SciTech Strategies. 15. Katherine W. McCain, “Mapping Economics through the Journal Literature: An Experi- ment in Journal Cocitation Analysis,” Journal of the American Society for Information Science 42, no. 4 (1991): 290–96; Henry Small, “Co-Citation in the Scientific Literature: A New Measure of the Relationship between Two Documents,” Journal of the American Society for Information Science 24, no. 4 (1973): 265–69. 16. Katherine W. McCain, “Core Journal Networks and Co-Citation Maps, New Bibliometric Tools for Serials Research and Management,” Library Quarterly 61 (1991): 311–36; Thomas E. Nisonger, “Journals in the Core Collection,” Serials Librarian 51, no. 3/4 (2007):51-73. doi:10.1300/ J123v51n03_05. 17. Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy, “Gephi: An Open Source Software for Exploring and Manipulating Networks,” paper presented at the International AAAI Conference on Weblogs and Social Media, San Jose, Calif., 2009. 18. Derek J. de Solla Price, “Networks of Scientific Papers.” Science 149, no. 3683 (1965): 510–15. 19. Kimball et al., “Citation Analysis of Atmospheric Science Publications,” 356–67. 20. Walcott, “Serials Cited by Marine Sciences Research Center Faculty,” 15–33. 21. Kimball et al., “Citation Analysis of Atmospheric Science Publications,” 356–67. 22. Walcott, “Serials Cited by Marine Sciences Research Center Faculty,” 15–33; Ibeun, “Applied Bibliometrics and Cooperative Acquisition,” 39-47; Kimball, “Citation Analysis of Atmospheric Science Publications,” 356-67. 23. Kimball et al., “Citation Analysis of Atmospheric Science Publications,” 356-67. 24. Ibeun, “Applied Bibliometrics and Cooperative Acquisition,” 39–47. 25. Walcott, “Serials Cited by Marine Sciences Research Center Faculty,” 15–33.