460 A Citation Analysis of College & Research Libraries Comparing Yahoo, Google, Google Scholar, and ISI Web of Knowledge with Implications for Promotion and Tenure Charles Martell Charles Martell is Emeritus Dean & University Librarian at California State University, Sacramento; e-mail: charles_martell@msn.com. Two hundred and seventeen articles in College & Research Libraries from 2000 to 2006 were searched by title on Yahoo, Google, Google Scholar, and ISI Web of Knowledge to determine the frequency with which articles in the journal are cited, the effectiveness of the four search services, and the relevance and applicability of findings to promotion and tenure. Yahoo, Google, and ISI Web of Knowledge averaged between 2.8 and 3.5 cita- tions per title for the period covered and Google Scholar averaged 6.4. The value of citations counts in the promotion and tenure process and the importance of publications in the evaluation of librarians are discussed. ollege & Research Libraries (C&RL) was ranked num- ber one by Association of Research Libraries (ARL) Library Directors in “Journal Prestige in Terms of Value for Tenure and Promo- tion” in a 1985 study by David Kohl and Charles Davis and in a 2005 replication study by Thomas Nisonger and Charles Davis.1 Given the significance of C&RL in promotion and tenure and its importance to members of the Association of College & Research Libraries generally, it seems both timely and instructive to determine the following: (1) the frequency with which articles published in C&RL are cited; (2) the citation retrieval strengths of Yahoo, Google, Google Scholar, and ISI Web of Knowledge (WOK) in retrieving citation information; and (3) the relevance and applicability of findings to promotion and tenure. To accomplish these objec- tives, every refereed article in C&RL from 2000 to 2006 was searched on each of the four retrieval services. Literature Review College & Research Libraries In “Publication Patterns of U.S. Academic Librarians from 1998 to 2002,” Stephen Wiberley et al. noted that it was “an im- portant task for any discipline to monitor the patterns of its publications.”2 Citation analysis qualifies as a pattern. The authors also cited C&RL as the “type of publica- tion most favored by academic librar- A Citation Analysis of College & Research Libraries 461 ians.”3 This article was a follow-up to their “Publication Patterns of U.S Academic Librarians from 1993 to 1997.”4 George Crawford compared College & Research Libraries and the Journal of Academic Librarianship, which he referred to as the primary journals in academic librarianship.5 He concluded that C&RL published significantly more research and was “the primary publisher of research-based articles in academic li- brarianship.”6 In “Investing Wisely: Citation Rank- ings as a Measure of Quality in Library and Information Science Journals,” Bar- bara Via and Deborah Schmidle studied the frequency of citations in 116 library and information science journals from 2002 through 2005.7 C&RL had the fourth highest frequency of citations and was first among library science journals.8 Except for six journals provided free of cost, C&RL had the best price per citation value of any journal at $0.09.9 By way of contrast, Library Management had a price per citation of $614.16.10 Impact factors are sometimes used as a measure of a published work’s signifi- cance. The importance of the journal in which the article appears is one factor. Eigenfactor ranks scholarly journals by field (for instance, history, medicine, economics, and information science).11 The Information Science field contains 23 journals. College & Research Libraries ranks 4th in this field behind the Journal of the American Society for Information Science and Technology, Information Processing & Management, and Journal of Documenta- tion.12 C&RL was ranked similarly against these journals in the Via and Schmidle study cited above. Eignefactor ranks Article Influence (AI): “a measure of a journal’s prestige based on per article citations and comparable to Impact Factor” and Eigenfactor (EI): “a measure of the overall value provided by all of the articles published in a given journal in a year.”13 These measures can be used to compare C&RL with other journals tracked by Eigenfactor. One of Eigenfactor’s more interesting aspects is its adjustment for citation dif- ferences across disciplines. This is deemed necessary because the average article in one discipline “may receive as many as 10–30 citations within two years of publica- tion” while the average article in another discipline “may do well to receive two.”14 Defining a Citation What is counted as a citation differed significantly among authors writing on the subject. ISI’s Web of Science is the source of publication data for John Budd’s “Faculty Publishing Productivity: Comparisons over Time.”15 Only articles were counted. Book reviews and editori- als were excluded. Thomas Nisonger tracked his own publications in “Citation Autobiography: An Investigation of ISI Database Cover- age in Determining Author Citedness.”16 Nisonger searched for citations to his published work by his name or variants thereof. Manuscripts, course syllabi or reading lists, Web bibliographies, con- ference presentations, and electronic journals were included. These categories met his criterion of a Web citation “as a decision to reference the author’s work in a document or page mounted on the Web.”17 Table of contents listings, current awareness services, accession lists, forms of advertising, and use of the author’s name without citing a specific work did not qualify for inclusion. Self-citations were also excluded. Nisonger found 29.8 percent of his total citations (with self citations excluded) in the ISI databases.18 Blaise Cronin suggested a citation count that included eleven components: abstract, article, conference, current awareness, external homepage, listserv, personal homepage, resource guide, book review, syllabus, and table of contents.”19 In a C&RL News article (May 2007), Carl Bergstrom, a founder of Eigenfactor, defined a citation as “the assessment of an individual scholar regarding which pa- pers are interesting and relevant to his or her work.”20 Like most definitions, there is 462 College & Research Libraries September 2009 room for interpretation. For example, do master’s theses qualify as the work of in- dividual scholars? Are self-citations, class syllabi, and assigned readings acceptable? Yahoo, Google, Google Scholar, and ISI Web of Knowledge Yahoo and Google are search engines, not citation indexes. None of the articles found for this Literature Review used either Yahoo or Google as a means for gathering citations. This was undoubtedly a practical matter, as ease of use for cita- tion searching is not one of their strengths. Alireza Noruzi referred to Google Scholar as one of “The New Generation of Citation Indexes” and compared it to the Web of Science.21 The author listed the Most-cited Authors in the field of Webometrics and provided two columns: Times Cited on Google Scholar and Times Cited on WoS.22 While there were notable similarities, the range of differences was instructive. Google Scholar had advan- tages over Web of Science in international coverage, ability to recognize variant forms of a citation, and inclusion of material not covered by Web of Science (such as theses, dissertations, conference proceedings, and technical reports).23 Publish or Perish’s Anne-Wil Harzing has studied citation indexes extensively. In “Reflections on Google Scholar,” she reviewed research comparing the Google index with ISI’s Web of Science and Elsevier’s Scopus, which was introduced in 2004.24 Her list of the disadvantages of each of these indexes is informative. Publish or Perish, which is “designed to help individual academics to pres- ent their case for research impact to its best advantage,” uses Google Scholar “to obtain raw citation data,” because “it generally provides a higher citation count than ISI,” particularly in the social sciences, arts, and humanities.25 Publish or Perish uses citation metrics indexes such as Hirsch’s h-index, Egghe’s g-index, and Age-Weighted citation rate and AW-index to estimate the impact of an individual researcher ’s work. They are described more fully under Citation Metrics, with references to the original source, in the Publish or Perish User’s Manual.26 Daniel Pauly and Konstantinos Ster- gious compared ISI’s citation index and Google Scholar in eleven disciplines.27 They concluded that Google Scholar “can substitute for ISI” and “may gradu- ally outperform” it.28 In addition to the advantage of free access, Google Scholar allowed “more transparency in tenure re- views” because “citation counts…[can] be performed and duplicated by anyone.”29 Elsevier’s Scopus, which was not included in the study, was deemed very expensive. Peter Jasco examined citation indexing, analysis, and searching.30 He expressed general dissatisfaction with the hype and inflated counts dispensed by Google Scholar, which he considered “partly due to the inclusion of non-scholarly sourc- es.”31 Author name recognition in Google Scholar was mentioned as particularly problematic. While applauding Google Scholar for making citations counts easy, the author cautioned researchers about the potential negative effects arising from this attribute.32 Roger Clarke studied ISI and Google Scholar to determine their impact on the field of information systems in Australia.33 He concluded by recommending Google Scholar and advising The Association of Information Systems “to take steps to avoid ISI-based citation analysis causing serious harm to the discipline’s access to research funding and senior academic positions.”34 Clarke noted that citation counts based on an author search tended to favor those with a long publishing record, whereas a title search favored “researchers with few publications but one or two ‘big hits.’”35 Implications of Citation Counts in Promotion and Tenure Thomas Nisonger discussed the role of citations in promotion and tenure in his C&RL (March 2004) article.36 While he stated that citations may support positively the contributions a promotion A Citation Analysis of College & Research Libraries 463 and tenure candidate’s work has made to scholarly communication, he also remarked that “citation data are contro- versial” and “citations can be negative.”37 In addition, citation counts can be influ- enced by the number of authors in the field. Whether a candidate published in “a high- or low-impact-factor journal” was an important consideration to Nisonger, who served for five years as the chair of the Promotion and Tenure Committee at the Indiana University School of Library and Information Science.38 Pamela Bradigan and Carol Mularski surveyed academic library directors at re- search and other doctoral-granting insti- tutions to rank ”various criterias’ relative importance in evaluating journal articles and monographs for tenure and initial promotion.”39 The four most important criteria were as follows: (1) appearance in a refereed journal; (2) the nature of the article (for example, research-based rather than practice-based); (3) perceived value of the journal to the profession; and (4) solicited assessments by outside experts.40 Librarians at those libraries with faculty status “were more likely to be reviewed on the basis of their publications.”41 There was no indication that any of these librar- ies had established criteria for the number or type of publications expected.42 The au- thors suggested that librarians “take care to inquire closely about the criteria used to judge publications in their particular institutions.”43 This can be important when perspectives among the parties are at variance. Library directors expressed a preference for research-based over practice-based publications.44 However, most librarians interviewed at Pennsyl- vania State University, which was ranked as the 2nd Most Productive Library in Wiberley’s study, described their research as practice-based. One librarian stated: “Everything I have done is based firmly on practice.”45 Joan Leysen and William Black sur- veyed peer review in Carnegie-Research Libraries.46 Forty-eight of the eighty-one libraries that responded to the survey indicated that the majority of their librar- ians had faculty status.47 Publication was required at 29 of these libraries. Publish- ing in a refereed journal was essential. In her Publish or Perish User’s Manual, Anne-Wil Harzing stated: “If an academic shows good citation metrics, it is very likely that he or she has made a significant impact on the field. However, the reverse is not necessarily true.”48 Working in a small field may be one cause for weak citation metrics. In “Publication Quality Indicators for Tenure or Promotion Decisions: What Can the Librarian Ethically Report?” Barbara A. Rice and Tony Stankus offered guid- ance to librarians who were providing and interpreting the results of a citation data search for teaching and research fac- ulty and other interested parties.49 Making comparisons of citation counts across disciplines was considered improper.50 The prestige of the journal in which the author’s article appeared was rated im- portant, as was the prestige of the editor and the referees (editorial board).51 The roles and obligations enumerated for the varying participants may have applicabil- ity within the library, although the article is a quarter century old. Methodology This section explains the methodology used (1) to meet the study’s objectives, (2) to identify a citation, and (3) to classify the data collected. Objectives The primary objective of this study was to determine the frequency with which ar- ticles in C&RL were cited. The secondary objective was to examine the citation-re- trieval strengths of Yahoo, Google, Google Scholar, and ISI WOK. WOK encompasses the Web of Science, Current Contents, and Journal Citation Reports. The third objective was to consider the relevance and applicability of findings to promo- tion and tenure. The fourth objective was to classify and quantify entries that were retrieved but did not qualify as citations. 464 College & Research Libraries September 2009 Accordingly, every refereed article in C&RL from 2000 to 2006 was searched by title on Yahoo, Google, Google Scholar, and ISI Web of Knowledge. Two hundred and seventeen articles met this criterion. Articles entitled “Selected References” were not included. Usually the full title was searched: some short titles were so unique that the full title was unnecessary. Quotes were placed around titles as a retrieval aid when it was determined that they would be helpful: they were used heavily in searching Google. The study was not designed to uncover citation variants or to demonstrate the degree of overlap between the sources searched. The data were collected in the summer and fall of 2007. It would have been pos- sible to include C&RL articles from 2007 in the citation analysis study; however, the results would not have been meaningful because the articles were only recently published. A search by author was con- ducted for the year 2000. This method proved unwieldy, unhelpful, and unneces- sary. It was not used in subsequent years. Identifying a Citation For the purposes of this study, a citation was defined as any reference to an article written in C&RL between 2000 and 2006 by an author (1) whose purpose related to the intellectual content of the article being cited and (2) whose work appeared in an article, chapter, book, conference, proceeding, technical report, thesis, dis- sertation, or scholarly bibliography. This definition excluded student class papers at any level. It excluded promo- tional book reviews but not scholarly book reviews. Several of Nisonger’s ex- clusions were adopted under this defini- tion: table of contents listings, current awareness services, accession lists, forms of advertising.52 In this study, however, unlike Nisonger’s, self-citations were in- cluded; but course syllabi or reading lists, and some bibliographies, were excluded. University, school, or department homep- ages or profiles and curriculum vita were also excluded. Duplications were eliminated when they were discovered. For example, if a citation to a C&RL article occurred in an author ’s manuscript, preprint, and article, then the citation count would be held at one. Entries derived from blogs were handled on a case-by-case basis. The author considered the length and substance of the material before making a decision. This was not a difficult process, but subjectivity was involved. Entries from Yahoo and Google had to be scanned carefully to distinguish between citations and noncitation list- ings. This was not necessary for ISI WOK, because its citation algorithm did not produce citations inconsistent with the definition established in this study. Some Google Scholar ’s listings did not meet the necessary criteria and were set aside. They were easy to identify and were few in number. As a rule, the author attempted to open each entry that appeared to be a citation. Proprietary and language barri- ers occasionally hindered this effort. By opening an entry, it was possible to review bibliographies, blogs, and other items in arriving at a decision. The noncitation listings for Yahoo and Google were counted and categorized. The data may assist other researchers. This information satisfied the fourth ob- jective of the study. Classifying the Data Each referral to an article in C&RL was placed into one of two categories: citations or noncitation listings. Subdivisions for citations included: theses, dissertations, and citations from China because they were so numerous. There were relatively few citations from other non–English- speaking nations. The noncitation listings category was subdivided into: ALA, ERIC, Haworth Press, EPrints, Garfield (Site of Eugene Garfield, founder and chairman emeritus of the Institute for Scientific Informa- tion—now Thomson Scientific. HistCite- index: College & Research Libraries at www. A Citation Analysis of College & Research Libraries 465 garfield.library.upenn.edu), country of origin (Non-English Speaking), and other (Probable machine-generated bib- liographies or those lacking intellectual content were placed in this subdivision; and Oddities or infrequent listings, such as Ask Eric, bNet, Judy Jeng, big6.com, faculty profiles, curriculum vita, and university announcements). Results Citation Counts Yahoo and Google title searches were ardu- ous to perform as it was necessary to screen out the noncitation list- ings, comprising 84 percent and 79 percent of the total entries pro- cessed for each. Entering the short title in quotation marks was the preferred search method. For the title, “Usability of the Academic Library Web Site,” 302 hits were received in Yahoo using quotes and 1,600,000 with quotes omit- ted. Google’s results were 274 with quotes and 886,000 without them. Some titles were quite general; and, with these, it was necessary to enter the complete title rather than merely the short title. Table 1 depicts the results for Yahoo. The average number of citations per C&RL article for the seven-year period was 2.8, with a high of 5.3 in 2001. Google’s re- sults in table 2 mirrored Yahoo’s for 2005 and 2006; but its average for the entire period was 3.5, with a high of 7.1 in 2001. Both Google Scholar (table 3) and ISI WOK (table 4) had falloffs similar to Yahoo and Google in the two most recent years. ISI WOK’s average citation count (3.1) was close to Google’s (3.5) but less than half of Google Scholar ’s (6.4). Likewise, Google Scholar’s high mark average of 11.6 in 2001 handily surpasses ISI WOK’s 6.5 high average, also in 2001. The results from Google Scholar included 60 theses, of which 54 were from the University of North Caro- lina School of Information and Library Science, 30 dissertations, and 120 Chinese. An article in November 2001 received 2 ci- tations, both of which were self-citations. This was an anomaly. Table 5 presents Google Scholar cita- tions to C&RL articles in ranges extend- ing from 0–4 (126 articles) to 30–100 (5 articles). Fifty-eight percent received 0 to 4 citations. Ten percent received fifteen or more citations. No article received more Table 1 Non-Citation listings, Citations, and average Citations per article Yahoo Title Search: College & Research Libraries Year Non- Citation listings Citations articles average Citations per article 2000 252 91 35 2.6 2001 576 179 34 5.3 2002 616 125 36 3.5 2003 368 85 26 3.3 2004 430 85 26 3.3 2005 422 26 28 0.9 2006 437 16 32 0.5 Totals 3,101 607 217 2.8 Table 2 Non-Citation listings, Citations, and average Citations per article Google Title Search: College & Research Libraries Year Non- Citation listings Citations articles average Citations per article 2000 341 144 35 4.3 2001 613 240 34 7.1 2002 618 149 36 4.1 2003 315 90 26 3.5 2004 332 72 26 2.8 2005 315 26 28 0.9 2006 365 26 32 0.8 Totals 2,899 752 217 3.5 466 College & Research Libraries September 2009 than nine citations in 2005 and 2006. The five titles with thirty or more citations ac- counted for 2 percent of the titles and 18 percent of the citations. Forty-eight titles received no citations. Table 6 displays the ten most cited articles in C&RL from 2000 to 2006. The highest citation count in Google Scholar was 100 for Kristin Antelman’s “Do Open- Access Articles Have a Greater Research Impact,” which appeared in the Septem- ber 2004 issue. ISI WOK had 22 citations for the same article. Six of the ten most cited articles appeared in 2001, four of these in the January issue. Interestingly, citations from China had a major influence on two of the ten articles: Charles Townley’s “Knowledge Management and Academic Libraries” (fourteen citations from China) and Susan Davis Herring’s “Use of Electronic Re- sources in Scholarly Electronic Journals” Table 5 Google Scholar Number and Percent of articles in each Citation Range College & Research Libraries Citation Range Number of articles Total articles Percent 2000 2001 2002 2003 2004 2005 2006 0 – 4 12 10 20 13 14 27 30 126 58% 5 – 9 9 9 10 5 7 1 2 43 20% 10 – 14 10 7 0 6 3 0 0 26 12% 15 – 19 2 0 3 1 0 0 0 6 3% 20 – 24 2 3 0 1 1 0 0 7 3% 25 – 29 0 3 1 0 0 0 0 4 2% 30 – 100 0 2 2 0 1 0 0 5 2% Totals 35 34 36 26 26 28 32 217 100% Table 3 Citations and average Citations per article Google Scholar Title Search College & Research Libraries Year Citations articles average Citations per article 2000 285 35 8.1 2001 396 34 11.6 2002 246 36 6.8 2003 171 26 6.6 2004 233 26 9.0 2005 41 28 1.5 2006 22 32 0.7 Totals 1,394 217 6.4 Table 4 Citations and average Citations per article ISI Web of Knowledge Title Search College & Research Libraries Year Citations articles average Citations per article 2000 143 35 4.1 2001 222 34 6.5 2002 123 36 3.4 2003 70 26 2.7 2004 82 26 3.2 2005 26 28 0.9 2006 14 32 0.4 Totals 680 217 3.1 A Citation Analysis of College & Research Libraries 467 (twelve citations from China). Citations from China also had an impact on Google Scholar’s total citation count. Of the 1,394 total citations, 120 were from China, or 8.6 percent. Of the ten most cited articles, seven were written by academic librarians, two by associate directors of information technology (who may be librarians), and one by a library school professor and a researcher-in-residence. In his study, Wiberley reported that 84 percent of all referred articles in C&RL were written by one or more academic librarians.53 Noncitation Listings The majority of Yahoo (table 1) and Google (table 2) entries fell into the non- citation listings category. Google Scholar and ISI WOK had none. The listings were varied, as is evident in table 7. ALA’s listings were similar in Yahoo (431) and Google (469). These were mainly, if not exclusively, C&RL’s Table of Contents for each issue. Among the list- ings were curriculum vita, homepages, course listings, student papers, university faculty biographies, brief bibliographic notices, and current awareness services. Non–English-speaking nations accounted for roughly 25 percent of the total listings in both search engines. Occasionally an article would be linked to a business. A humorous tie-in was ABC Body Building’s link to “Building Preservation” (May 2003), whose subtitle Table 6 Ten Most Cited articles in C&Rl 2000-2006 Citations author Title 100 (22) K. Antelman, Do Open-Access Articles Have a Greater Research Impact? (September 2004) 50 (26) D. Grimes Worries with the Web: A Student Look at Student Use ofWeb Resources (January 2001) 42 (14) P. Davis The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update (January 2002) 30 (16) B. Lynch The Changing Nature of Library Work (September 2001) 30 (12) S. Herring Use of Electronic Resources in Scholarly Electronic Journals: A Citation Analysis (July 2002) 29* (5) C. Townley Knowledge Management and Academic Libraries (January 2001) 28 (9) P. Maughan Assessing Information Literacy Among Undergraduates: A Discussion of the Literature and the University of California- Berkeley Assessment Experience (January 2001) 28 (17) L. McGillis Usability of the Academic Library Web Site: Implications for Design (July 2001) 27 (12) M. Foley Instant Messaging Reference in an Academic Library: A Case Study (January 2002) 26 (13) S. Rogers Electronic Journal Usage at Ohio State University (January 2001) *This article had 14 entries from China. The form and validity of the entries as citations were not verified by the author due to the language. Note: Google Scholar citation counts are listed first and ISI Web of Knowledge second in parentheses, with the exception of the article by S. Rogers. 468 College & Research Libraries September 2009 was “The University of Illinois at Urbana- Champaign’s Stacks Assessment.” Rich- ard Meyer ’s “A Tool to Access Journal Price Discrimination” (May 2001) was a Blackwell Prize Winner. Over a dozen listings in 2001 and 2002 were created by Judy Jeng, an ASIS Garfield Travel Award Winner in 2003. There were five duplicates for a single 4 General Works listing, two listings from zoominfo.com, and as many as a dozen in 2002 from 3.5 and 3.6 Electronic Serials. Articles in C&RL were frequently listed by current awareness and other services in non–English-speaking countries (table 8). Almost one quarter of the listings in Yahoo and Google were from these countries, with China and France in the number one and two positions. Discussion Yahoo, Google, Google Scholar, and ISI WOK Yahoo, Google, and Google Scholar pro- vide citation information free of charge. ISI’s Web of Knowledge (WOK) is a fee- based proprietary resource. The former are readily accessible on the Internet, while the latter is normally available to authorized academic users through their library’s portal. Cost is only one factor. Ease of access to citation information, the comprehensiveness of the resource, and the accuracy of the information cited are important considerations. The Yahoo and Google search engines are not citation indexes. As a result, the process of obtaining cita- tion information from them is laborious. Google Scholar and ISI WOK provide citation indexes that significantly ease the search process. The average number of citations per C&RL article is relatively close between Yahoo (2.8), Google (3.5), and ISI WOK (3.1). The ISI WOK advantage is ease of use and the quality of its citations. It adheres to founder Eugene Garfield’s definition of cita- tions as “the formal, explicit linkages be- tween papers that have particular points in common.”54 ISI WOK’s focus on journal articles and the cost of its product are among its disadvantages. Google Scholar’s coverage is broader. It is free and retrieved an average of 6.4 cita- Table 7 Yahoo and Google Non-Citation listings College & Research Libraries listing Subdivision Yahoo Google ALA (Table of Contents) 431 469 ERIC 17 143 Haworth Press 6 85 EPrints 66 43 Garfield’s Hist-Cite Index 241 162 Non-English Speaking Nations 700 778 Other 1,640 1,219 Totals 3,101 2,899 Table 8 Yahoo and Google listings by Country College & Research Libraries Country Yahoo Google China 137 203 France 128 248 Japan 90 28 Estonia 85 68 Spain 82 55 Denmark 7 9 Turkey 5 5 Italy 4 10 Belgium 3 9 Iran 1 5 Netherlands 1 - Korea - 2 English Speaking 2,323 2,199 Other 157 136 Totals 3,101 2,899 A Citation Analysis of College & Research Libraries 469 tions per C&RL article in comparison to ISI WOK’s 3.1. Its drawbacks are less obvious, but they are noteworthy. For example, almost half the citations of two of the top ten articles were from China. Library committees that use citation counts in the promotion and tenure process would probably be uneasy about taking them at full value without further evaluation. The inclusion of citations from non– English-speaking nations has been viewed as one of Google Scholar’s ad- vantages. If citation counts are a tool to judge how a colleague’s work is regarded by those peers who are themselves re- spected in the profession, then barriers of language, culture, and familiarity may diminish their usefulness. Self-citations serve a legitimate schol- arly purpose and were included within the definition of a citation for the purpose of this study. In the promotion and tenure process, it is one’s colleague’s opinions that are sought. Self-citations are, there- fore, most likely set aside. Applicability of Citation Counts to Promotion and Tenure There is no established benchmark for the number of citations a librarian should receive to be evaluated as poor, fair, good, or excellent on this criterion. The average number of citations for Google Scholar was 6.4. This average would not be useful to a librarian who had published within the past three years when the C&RL count average was in the 1.0 or less range. Rice posed the question of norms and noted that “the average annual citation rate for a scientific paper is only 1.7 and many papers are not cited at all.”55 In a study of psychology journals, Nor- man Endler and his coauthors reported that 40 percent of the articles had no ci- tations and 3 out of 5 had two or fewer.56 Endler found: “A disproportionately large amount of citations and publications is accounted for by a disproportionately small number of individuals.”57 A “How to Sheets: Impact Factors” was displayed on HealthLinks, a Science Librar- ies at the University of Washington Web site.58 Among the several controversial aspects of using impact factors was: “It is not clear whether the number of times a paper is cited measures its actual quality.”59 Indeed, the article cited may have received negative commentary. Those in the pro- motion and tenure process—candidates, committee members, and decision mak- ers—are advised to prepare and require a full accounting of citations counted. This will help to dispel uncertainty and improve the effectiveness of the process. ARL Library Directors reported that publishing in C&RL was important in promotion and tenure.60 According to the research of Bradigan and Mularski, how- ever, publishing was a requirement in only a handful of ARL libraries.61 Therefore, the word important in the survey may not con- vey the practical relevance of publishing for librarians in ARL libraries. Additional research is needed to clarify this issue. There is evidence that librarians at large academic libraries, especially re- search libraries, are the main contributors to the professional literature.62 Wiberley stated that “the twenty most productive libraries published more than 10 percent of all refereed articles in the thirty-two journals studied and nearly one third of the articles by academic librarians.”63 Citation counts appear to have the most applicability at libraries where librar- ians have faculty status and where the publishing record of librarians makes an observable and documented difference as to whether they are promoted or receive tenure. But how many librarians in these li- braries actually publish in the professional literature; and what about the remaining libraries where publication in refereed journals is not a significant criterion for promotion and tenure and where librar- ians publish infrequently in these journals? Wiberley’s article and NCES Academic Library Statistics provide a few clues. Li- brarians in the Wiberley study were from 379 institutions of higher education.64 The National Center for Education Statistics reported that there were 3,600 academic 470 College & Research Libraries September 2009 The dramatic falloff in the average number of citations per article in the years 2005 and 2006 was a surprise. It is unlikely 2007 would have fared any better. What is the explanation? Most likely, this is the normal lag time (two to three years) for articles published in the journals of academic librarianship; but other factors may be operative. Conclusion C&RL articles are a significant resource for those who are seeking to understand or research the issues facing academic libraries and the profession, more specifi- cally. Google Scholar returned an average of 6.4 citations per article. Although this was more than double the return for ISI WOK, the differences in their respective coverage should be considered before one or the other is selected for citation counting. The Yahoo and Google search engines are not recommended for gather- ing citations because they are not easy to use for this purpose. Google Scholar will serve as a convenient substitute. This is an exciting time to be a librarian. The common sense understandings of the past are being challenged. Great changes are occurring. Our profession’s journals can serve as guideposts. The articles we read in them can inspire and motivate us. Assuming that citation counts are one indicator of a journal’s utility, the findings in this study may lead one to ask why we are not making more use of this resource. Citation counting may be a dull busi- ness, but the authors who are writing in our journals are engaged in a serious en- terprise. They are turning the lessons they have learned into material from which we can obtain guidance. Apparently we need more librarians translating the valiant efforts of these authors into articles that will motivate others to take up similar challenges and, in the process, give their citation counts a big and well-deserved boost. The librarian at Pennsylvania State University who said, “Being cited gave me great satisfaction,” would certainly agree. libraries in 2006.65 Thus, 1 in 9.5 libraries employed librarians who published in the journals examined between 1998 and 2002. Citation counts may be useful to librar- ians who are attempting to demonstrate that they have a national or international reputation of merit. It is not known how many citations would be necessary to validate such a position. Members of promotion and tenure committees must occasionally struggle with this issue. In Wiberley’s study, 41 percent of the articles published by academic librarians were coauthored.66 Parsing out credit must be a challenge. On the other hand, librarians at Pennsylvania State University are advised to “strive for single-authored articles in peer-reviewed journals.”67 Citation counts have probable utility to library school faculty who are required to publish research as a condition of employ- ment. Collectively, the number of library school faculty who are likely to publish in C&RL is small, and the relative benefits to the profession of citation counting for promotion and tenure is not greatly ex- tended by their inclusion. Furthermore, the deans of ALA-accredited education programs ranked C&RL 11th behind six information science journals and four library science journals.68 Bradigan and Mularski concluded that “librarians who must publish are a minor- ity of all academic librarians. However, this still indicates that hundreds are on the tenure track and must understand how their publications will be judged.”69 Further Research The average number of citations for articles in C&RL exceeds those of articles in other library journals.70 Nonetheless, two areas warrant attention: (1) articles with no cita- tions and (2) the falloff in the average num- ber of citations in the years 2005 and 2006. An examination of those articles that received no citations (48), or a mere one or two, would appear beneficial. Determin- ing why they appear less useful, based on citation counts, may yield valuable information that could prove helpful. A Citation Analysis of College & Research Libraries 471 Notes 1. David F. Kohl and Charles H. Davis, “Ratings of Journals by ARL Library Directors and Deans of Library and Information Science Schools,” College & Research Libraries 46 (Jan. 1985): 40–47; Thomas E. Nisonger and Charles H. Davis, “The Perception of Library and Information Science Journals by LIS Education Deans and ARL Library Directors: A Replication of the Kohl–Davis Study,” College & Research Libraries 66 (July 2005): 346, available online at www.ala.org/ala/acrl/ acrlpubs/crljournal /backissues2005a/crljuly05/nisonger2.pdf [Accessed 22 August 2008]. 2. Stephen E. Wiberley Jr., Julie M. Hurd, and Ann C.Weller, “Publication Patterns of U.S. Academic Librarians from 1998 to 2002,” College & Research Libraries 67 (May 2006): 205. Avail- able online at www.ala.org/ala/acrl/acrlpubs/crljournal /backissues2006a/maycrl/Wiberley06.pdf. [Accessed 22 August 2008]. 3. Ibid. 4. Ann C. Weller, Julie M. Hurd, and Stephen E. Wiberley Jr., “Publication Patterns of U.S. Academic Librarians from 1993 to 1997,” College & Research Libraries 60 (July 1999): 352–62. 5. Gregory A. Crawford, “The Research Literature of Academic Librarianship: A Comparison of College & Research Libraries and Journal of Academic Librarianship,” College & Research Libraries 60 (May 1999): 225. Available online at www.ala.org/ala/acrl/acrlpubs/crljournal/backissues1999b/ may99/crawford.pdf. [Accessed 22 August 2008]. 6. Ibid., 229. 7. Barbara Via and Deborah Schmidle, “Investing Wisely: Citation Rankings as a Measure of Quality in Library and Information Science Journals,” portal: Libraries and the Academy 7 (July 2007): 333–73. 8. Ibid., 341. 9. Ibid., 357. 10. Ibid., 347. 11. Eigenfactor information available online at http://eigenfactor.org. [Accessed 25 August 2008]. 12. Eigenfactor field of Information Science is available online at www.eigenfactor.org/results. php?fulljournalname1=&issnnumber=&pub=&finecat=NU&rosvcat=%25&year=2006&nam=nam es&grping=%25&resultsperpage=100&ordering=perarticle&Submit=Search. [Accessed 22 August 2008]. 13. Eigenfactor ranking of College & Research Libraries is available online at www.eigenfactor.org/results.php?fulljournalname1=COLLEGE+%26+RESEARCH+LIBRAR IES&rosvcat=%25&year=2006&resultsperpage=100&issnnumber=&ordering=perarticle&grping =%25&nam=names&Submit=Search. [Accessed 22 August 2008]. 14. Eigenfactor information available online at www.eigenfactor.org/whyeigenfactor.htm. [Accessed 22 August 2008]. 15. John M. Budd, “Faculty Publishing Productivity: Comparisons over Time,” College & Re- search Libraries 67 (May 2006): 231. Available online at www.ala.org/ala/acrl/acrlpubs/crljournal/ backissues2006a/maycrl/Budd06.pdf. [Accessed 22 August 2008]. 16. Thomas E. Nisonger, “Citation Autobiography: An Investigation of ISI Database Coverage in Determining Author Citedness,” College & Research Libraries 65 (March 2004): 152–63. Available online at www.ala.org/ala/acrl/acrlpubs/crljournal/crl2004/crlmarch04/nisonger.pdf. [Accessed 22 August 2008]. 17. Ibid., 156. 18. Ibid., 157. 19. Blaise Cronin et al., “Invoked on the Web,” Journal of the American Society for Information Science 49 (Dec. 1998): 1319–28. 20. Carl Bergstrom, “Eigenfactor: Measuring the Value and Prestige of Scholarly Journals,” C&RL News 68 (May 2007). Available online at www.ala.org/ala/acrl/acrlpubs/crlnews/backis- sues2007/may07/eigenfactor.cfm. [Accessed 22 August 2008]. 21. Alireza Noruzi, “Google Scholar: The New Generation of Citation Indexes,” Libri 55 (2005): 170. Available online at www.librijournal.org/pdf/2005-4pp170-180.pdf. [Accessed 22 August 2008]. 22. Ibid., 172–73. 23. Ibid., 174. 24. Anne-Wil Harzing, “Reflections on Google Scholar,” Publish or Perish User’s Manual (Feb. 2007): 1–3. Available online at www.harzing.com/pophelp/2020-gs.htm. [Accessed 22 August 2008]. 25. Ibid., 1. 26. Anne-Wil Harzing, Publish or Perish’s User’s Manual. Available online at www.harzing. 472 College & Research Libraries September 2009 com/pophelp/metrics.htm. [Accessed November 30, 2007]. 27. Daniel Pauly and Konstantinos I. Stergious, “Equivalence of Results from Two Citation Analyses: Thomson ISI’s Citation Index and Google’s Scholar Service,” Ethics in Science and Envi- ronmental Politics (Dec. 22, 2005): 33–35. 28. Ibid., 34. 29. Ibid., 35. 30. Peter Jacso, “Deflated, Inflated and Phantom Citation Counts,” Online Information Review 30, no. 3 (2006): 297–309. Available online at www.jacso.info/PDFs/jacso-deflated-inflated.pdf. [Accessed 22 August 2008]. 31. Ibid., 300. 32. Ibid., 301. 33. Roger Clarke, “An Exploratory Study of Information Systems Research Impact,” Com- munications of the Association of Information Systems 22 (Jan. 2008): 1–32. Available online at www.anu.edu.au/people/Roger.Clarke/SOS/Cit-CAIS.pdf. [Accessed 22 August 2008]. 34. Ibid., 2. 35. Ibid., 3. 36. Nisonger, “Citation Autobiography,” 153–54. 37. Ibid., 154. 38. Ibid., 153–54. 39. Pamela S. Bradigan and Carol A. Mularski, “Evaluation of Academic Librarians’ Publica- tions for Tenure and Initial Promotion,” Journal of Academic Librarianship 22 (Sept. 1996): 360. 40. Ibid., 363. 41. Ibid., 360. 42. Ibid. 43. Ibid., 364. 44. Ibid., 361. 45. Joseph Fennewald, “Research Productivity among Librarians: Factors Leading to Publica- tions at Penn State,” College & Research Libraries 69, no. 2 (Mar. 2008): 108 46. Joan M. Leysen and William K. Black, “Peer Review in Carnegie-Research Libraries,” Col- lege & Research Libraries (Nov. 1998): 511–21. 47. Ibid., 512. 48. Harzing, “Reflections on Google Scholar,” 1. 49. Barbara A. Rice and Tony Stankus, “Publication Quality Indicators for Tenure or Promotion Decisions: What Can the Librarian Ethically Report?” College & Research Libraries 44 (Mar. 1983): 173. 50. Ibid., 174. 51. Ibid., 176. 52. Nisonger, “Citation Autobiography,” 156. 53. Wiberley et al., “Publication Patterns,” 207. 54. Eugene Garfield, Citation Indexing: Its Theory and Applications in Science, Technology and the Humanities. (New York: Wiley Interscience, 1979): 1. 55. Rice and Stankus, “Publication Quality Indicators,” 174. 56. Norman S. Endler et al., “Productivity and Scholarly Impact (citations) of British, Canadian, and U.S. Departments of Psychology (1975).” American Psychologist 33 (July 1978): 1064–82. 57. Ibid. 58. “How to Sheets: Impact Factors,” HealthLinks, University of Washington, Science Libraries (Apr. 2008): 1–4. Available online at http://healthlinks.washington.edu/ howto/impactfactors.html. [Accessed 22 August 2008]. 59. Ibid., 1. 60. Nisonger and Davis, “The Perception of Library and Information Science Journals,” 346. 61. Bradigan and Mularski, “Evaluation of Academic Librarians’ Publications,” 364. 62. Mickey Zemon and Alice Harrison Bahr, “An Analysis of Articles by College Librarians,” College & Research Libraries 59 (Sept. 1998): 422–23, 426. 63. Wiberley et al., “Publication Patterns,” 205. 64. Ibid., 212. 65. “Academic Libraries” (Washington, D.C.: National Center for Education Statistics, 2006): 1. Available online at http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid =2008337. [Accessed 22 August 2008]. 66. Wiberley et al., “Publication Patterns,” 209. 67. Fennewald, “Research Productivity among Librarians,” 105. 68. Nisonger and Davis, “The Perception of Library and Information Science Journals,” 346. 69. Bradigan and Mularski, “Evaluation of Academic Librarians’ Publications,” 364. 70. Via and Schmidle, “Investing Wisely,” 340–46. Online Seminars multi-week, delivered through moodle instructional design for online teaching and learning (october 5-31, 2009) Four week hands-on course focused on using good instructional design and Web page design principles. Participants will be introduced to Web-based Teaching techniques and materials using standard Web pages and the Moodle LMS. Requires a significant amount of hands-on learning-activities, project work, and readings. Will result in a completed instructional design plan. Contact Jon Stahler at jstahler@ala.org for more information about e-learning. Live Webcasts one to two hours, delivered through online classroom Podcasting for libraries (october 1, 2009) Podcasting is like a blog with audio. Use the power of RSS syndication to automatically deliver new episodes to listeners. Any library can produce a podcast using free software and inexpensive hardware. How can your library use podcasting as a tool for teaching, promotion, outreach and programming? Contact Jon Stahler at jstahler@ala.org for more information about Webcasts. Go Mobile acrl workshoPs @ midwinter (January 15, 2010) anytime, anywhere, any device: develoPing a mobile website for your library Get an overview of the current state of mobile technologies both within and outside of libraries, and learn how to launch a mobile library website. text messaging, twitter, and libraries Learn how libraries are leveraging text messaging (also known as SMS) and Twitter. Discover basic and advanced applications of these tools for extending and enhancing academic library services and collections. Contact Margot Conahan at mconahan@ala.org for more information about workshops. 50 East Huron Street | Chicago, IL 60611 (800) 545-2433 ext. 2529 www.acrl.org | acrl@ala.org New Media & aCRL