College and Research Libraries Research Notes 173 Publication Quality Indicators for Tenure or Promotion Decisions: What Can the Librarian Ethically Report? Barbara A. Rice and Tony Stankus Academic faculties are becoming in- creasingly aware of the many capabilities of Science Citation Index, 1 Social Sciences Ci- tation Index, 2 and Arts & Humanities Cita- tion Index. 3 These indexes are used for sub- ject searching, for obtaining lists of publications by a given author, for deter- mining centers of certain types of re- search, for comparative evaluations of ac- ademic departments, and for evaluation of peers in tenure and promotion consid- erations. Although considerable contro- versy surrounds the use of citation counts for the latter practice, they are being used in such evaluations. It is not the purpose of this paper to reexamine or enter into the controversy surrounding the use of cita- tion data in promotion or tenure consider- ations. The literature on this subject is vo- luminous. We wish to discuss the impact of online availability of citation data and provide information relating to provision and interpretation of search results by li- brarians. In addition, we recommend other types of information which the li- brarian should suggest to patrons in order to supplement citation data, or to be used when citation data is nonexistent or ap- pears incomplete. ONLINE CITATION SEARCHING Until quite recently a patron wishing to determine whether or not a given author's works were cited could do a manual search of the appropriate citation index and come up with a tally, or more often, a tally and listing of who was citing the au- thor being evaluated. In order to do an ef- fective search it was necessary to under- stand how the index worked, and it is likely, although not guaranteed, that the searcher would read the introductory ma- terial to learn how to do the search and be- come aware of some of the difficulties in- herent in compiling a listing which are due to the nature of the index. For example, in doing a manual search it is likely that the patron would become aware of the fact that only the first author is listed in the ci- tation index and that, by tracing the cita- tion to the source index, would realize that there might be more than one author of the same name. However, with the ad- vent of online availability of these indexes, this is no longer the case. A patron can bring in a list of references, leave them with a searcher, and return to pick up a list of citations. Although Caldwell and Livingston 4 state that ''citation indexes provide a way to determine how highly valued and widely read the research pro- duced by your faculty is,'' we caution that the process is not as simple as running an online search, as these authors imply. The librarian performing a citation search has an ethical responsibility to in- form the patron of the nature of the cita- tion index being searched, with the inher- ent limitations this places on the search. The patron should also be provided with information on known citation practices in the evaluation field and references to the literature, which give caveats relating to the application of the references obtained in evaluation procedures. Before attempting the search, the searcher should discuss the list of refer- ences with the patron to ensure that it is complete, and that the patron under- stands the search process. It is frequently Barbara A. Rice is principal librarian for reference seroices, New York State Library, and Tony Stankus is sci- ence librarian, Science Library, College of the Holy Cross, Worcester, Massachusetts. 174 College & Research Libraries the case that the person gathering the in- formation sends a graduate student to de- liver the list and, if required by the library, participate in the search interview or search process. The librarian should not per- mit this. The limitations of the index and search capabilities are difficult to explain and the searcher should not assume that the student will convey all information to the requester. During the interview the searcher should examine the submitted bibliogra- phy carefully to see that references are complete and that actual first authors are given. Cited items are listed only by the first author in the citation indexes. There- fore, simply doing a search to find items citing Jane A. Doe will find only those pa- pers which list her as sole author or first author, but will not retrieve those papers where she is second or third author unless the first author is provided . Whether or not this is important is dis- cussed by Garfield. 5 It is decidedly impor- tant that the requester know whether or not his bibliography is accurate in this re- spect and, if not, what qualifications this imposes on the search results. The patron must also be made aware of the scope of the particular citation index being searched and be shown the list of journals covered. Each citation index cov- ers the most heavily cited journals in each field covered, but there are certain aca- demic specialties for which the source journals might not be a very comprehen- sive list of works citing the author being evaluated. A quick way to check is to see whether the author's own papers are listed in the journals as possible citations. If a majority are not, the searcher should advise the patron of the alternative strate- gies discussed later in this paper. Another difficulty sometimes encoun- tered is that the index cannot distinguish between two people having the same last name and first initial, who may be pub- lishing during the same time period. The patron should understand the necessity for examining the search results carefully for entries that seem to be in a field differ- ent from the author being evaluated. March 1983 COMPARATIVE CITATION PRACTICES IN DIFFERENT FIELDS As Garfield points out, "It certainly is improper to make comparisons between citation counts generated in different fields. " 6 Not only are there differences with respect to the average number of cita- tions in a given field, but there are differ- ences with respect to the type of material cited, e.g., mathematicians cite books more often than chemists. Science Citation Index and Social Sciences Citation Index pri- marily examine journal sources for cita- tions. What then can the searcher give to the patron relating to citation practices in dif- ferent fields? It is recommended that the user be given a reprint of Garfield's chap- ter entitled "Perspective on Citation Anal- ysis of Scientists" 7 for a summary of known norms in the sciences. In actuality, limited information is available, but it should encourage the user to consider what norms should be used and what sig- nificance any raw number of citations, whether 0 or 5 or 500, means in the field where the information supplied is to be used. The average annual citation rate for a scientific paper is only 1.7 and many pa- pers published are not cited at all. Endler, 8 in studying Canadian psychologists using Social Sciences Citation Index, reports that during 1975, 40 percent of those studied had zero citations and three out of every five had two citations or less. "A dispro- portionately large amount of citations and publications is accounted for by a dispro- portionately small number of individ- uals," Endler stated. Helmreich 9 also points out that there are large sex differ- ences in citation rates among psycholo- gists and analyzes possible causal factors . If no norms are known for the candi- date's field, it is conceivably possible to construct them using the following tech- nique: the patron supplies a list of depart- ments considered to be of comparable nat- ure (e.g., size, national reputation, student body, etc .). Using the corporate source index of the appropriate citation in- dex, the number of papers from each of these institutions is tallied. Then an aver- age number of papers per faculty member can be obtained by dividing the total num- ber of papers by the number of faculty. This could be done for one year, or, prefer- ably, for several years. Such techniques have been used for comparing institu- tions. 10'11 '12 Alternatively, the records of only those faculty from these peer institu- tions whose specia~ization matches that of the candidates could be found. As noted earlier, Science Citation Index and Social Sciences Citation Index examine predominantly journal sources for cita- tions. Citations appearing in most books are not included, although books cited in the journals scanned are. Citation counts may or may not include citations to book reviews, and what would seem apparent here is that it is not as important to know whether or not there was a review, as it is to know what the review stated as to the quality and importance of the book being reviewed . A librarian should know the major review sources in any field and can assist the patron compiling the informa- tion for the promotion or tenure review to ferret out reviews should the candidate's vita include books as well as journals. The searcher should inform the patron that a library staff member, knowledge- able in the use of citation indexes, is avail- able to discuss the search results and inter- pret the results and other techniques which should be considered in order to obtain as complete information as possible on citations to the work of the individual being evaluated. The use of the citation data in promotion and tenure processes has been urged as a quantitative, objective standard of evalua- tion. Given the little that is known about norms, it does not seem that citation data is yet a wholly effective tool in this respect. What if no norms are known for the candi- date's field? What if the candidate's publi- cations are very recent and have not yet been cited, or cited comparatively little? What then? The librarian, in addition to supplying information on known norms, should suggest other information relating to the journals in which the candidate has published which can be of use in the eval- uation process . Information about the quality of the Research Notes 175 journals within which the candidate being evaluated has published can easily be sup- plied by librarians. An evaluator may wish to consider some or all of the following fac- tors in addition to or in lieu of citation in- formation. • What is the purpose of the journal? • Who publishes the journal? • What types of articles does it carry? • Who are the editors and referees? • What role do the editors and referees play in the acceptance or publication process? • What institutions are represented among contributors? • Who are the individual contributors? JOURNAL PURPOSE, PUBLISHER, AND TYPES OF PAPERS Journals cited in the vita may be charac- terized by purpose, publisher background and specialization, and the nature of the papers being published. There are many for-profit firms, university press, and pro- fessional societies which are known to be quality publishers. Most librarians can list publishers of known repute in various ac- ademic disciplines. The librarian, how- ever, should not report unsubstantiated hearsay, but should instead gather factual information for analysis. This may be types of journals published, e.g., schol- arly versus popular, general fields in which the publisher specializes, e.g., sci- entific, legal, etc., and the number of jour- nal titles published . If the publisher is a so- ciety, then information on the society's membership and purpose should be re- ported. The type(s) of paper(s) carried is also a consideration in evaluating the nature of the candidate's contribution. For exam- ple, is it a review journal or does it consist of original research contributions? Is it an informal news bulletin-an alumni jour- nal, for example? Is the contribution of the candidate a full-length article? Book re- view? Brief communication? Letter to the editor? Are there special requirements for publication, e.g., membership or alumnus status? While some of this information may be known to the individual request- ing the information, it may not all be 176 College & Research Libraries known, nor may it be known uniformly through the department or by other per- sons or review bodies which are often in- volved in the evaluation process, such as a campus-wide promotion and tenure com- mittee or the vice-president or president of the college or university. Samples of specific information which a librarian might compile follow. 1. The J ou mal of'' A'' Studies, founded in 1948, publishes about 200 eight-to-fifteen- page contributed, original research papers each year in the general field indicated by the title. It also publishes monthly society personnel news, an editorial, five to ten book reviews, and job placement notices. It is sponsored by the Society for "A" Re- search and Treatment, located in New York, which has a membership of 12,000. Membership is open to those with an M.D. or Ph.D. in an appropriate discipline who demonstrate an interest in the field and can provide references from two es- tablished members. Membership is not necessary for acceptance of an article, al- though members do not have to pay page charges. The candidate is a member and the publications 1 and 5 in his vita repre- sent a research article of standard length for this journal and a standard length, 200- word book review, respectively. 2. The "B" Review has been published by the Midwestern University press since 1975. It is the only journal published by them. Each year it publishes approxi- mately 30 twenty-to-forty-five-page sur- veys of recent developments in law and politics. Articles are written largely by its own students, faculty, and alumni. It ap- pears on a trimester basis with the spring issue listing biographies of honorary- degree recipients, the text of commence- ment speeches, and a directory of recent graduates. The candidate is an alumnus of this institution. His contribution, number 8 on the vita, is a letter to the editor criticiz- ing the choice of an honorary degree recip- ient and rebutting that recipient's article in a prior issue. 3. Acta "C" Internationalis has been published since 1960 by Alexandrian Press, a for-profit scientific, technical, and medical publishing house which currently publishes about 300 journals in New York, March 1983 Paris, London, and Vienna. Acta "C" In- ternationalis is a bimonthly organic chemis- try journal which carries an annual total of approximately 70 twenty-to-twenty-five- page overviews of recent research, ap- proximately 400 eight-to-twelve-page con- tributed articles, approximately 250 four-to-six-page brief communications, . and approximately 300 paragraph ab- stracts of presentations at professional symposia in the field. Publications 1, 2, and 3 on this candidates vita are abstracts from the symposia. Item 4 is a brief com- munication of the results announced in the first abstract, and item 5 is an article of standard length elaborating on the find- ings of the second and third abstracts. JOURNAL RANKINGS AND REJECTION RATES Another possible way of describing the importance of a given journal is to use Journal Citation Reports, an annual compi- lation which is part of Science Citation Index and Social Sciences Citation Index, which rank journals in several ranking packages, e.g., total citations and impact factor. The experiences of author Stankus working with use of impact factors to provide infor- mation on science journals in promotion and tenure decisions has been that stronger candidates have published in journals with relatively high impact fac- tors . Rejection rates are a piece of information that is not always readily available, but in some cases can be found by digging through the front matter of journals, or in the annual reports of publishing houses or societies. The information is usually avail- able on request from the editor or pub- lisher. EDITORS, REFEREES, AND CONTRIBUTORS Another important consideration in journal quality is the prestige of its editors and referees. While there is no guarantee that all members of an editorial board are actively involved in the journal, or that stated refereeing policies are followed, in- formation about the reputations of the edi- tors and referees can be gathered from bio- graphical directories. For example, American Men and Women of Science can supply an outline of factual information. A quick literature search can establish the extent of a given editor's or referee's own publication history. Compilations of aca- demic institutional evaluations, e.g., An- derson & Roose's Rating of Graduate Pro- grams, can give an idea of the degree of distinction of the program with which the editor or referee is affiliated. Some sample information that might be supplied fol- lows. 1. The editors of Journal "X" are Jones, Harvard University; Smith, University of Chicago; and Johnson, University of Cali- fornia, Berkeley. Jones, Smith, and John- son are all full professors in programs rated by Anderson & Roose as distin- guished. Jones and Johnson are members of the National Academy of Sciences. Manuscripts are read by one of the editors plus two outside referees of his choosing. 2. ''Y'' Journal has more than sixty listed editors. A sampling shows that slightly more than half are affiliated with clinical programs in drug and alcohol abuse, while the remainder are equally divided between highly rated schools of social work or psychology. Manuscripts are sent to the editor in chief who designates three of the editors as referees. 3. The Yearbook of the "Z" State Univer- sity Teaching Effectiveness Workshops is edited by a committee drawn from the chairpersons of the departments of educa- tion at each of the university's branches. Workshop faculty and all working teach- ers who plan to attend furnish an outline of their presentation three months before the meeting for review by all members of the board. Approximately 10 percent are selected for general presentation at the meeting and subsequent printing in full in Research Notes 177 the yearbook. The remaining are accepted for poster presentations with subsequent printing as one-page summaries. In addition to evaluating editors, all of whom may have been selected for high re- pute in their fields, it is also possible to sample contributors. Virtually every au- thor makes some commitment to a journal by having submitted a paper to it. An au- thor's choice of journals lives on in per- sonal biographies which the author (if aca- demic) is aware will be reviewed by tenure or promotion committees, for grant pro- posals, etc. In the aggregate, academic de- partments gain a reputation by publishing in the best journals, and the best journals are characterized by repeatedly publish- ing the work of the best departments. Consequently, librarians can use online techniques to sample departments to see which have contributed to a journal in which the candidate has published. CONCLUSION Academic librarians who are asked to supply information for tenure or promo- tion decisions have an ethical responsibil- ity not only to supply the requested infor- mation but to educate the requester in the limitations of any search techniques used (manual or online) and the types of factual information that can be supplied. Librari- ans should require from requesters that they participate in the search process ( es- pecially in the case of online searches) and provide a clear statement of what is being asked for. The authors believe that the is- sues raised in this article merit serious dis- cussion by academic librarians, which would result in statements of library pol- icy with respect to library participation in provision of information to faculties for promotion or tenure consideration. REFERENCES 1. Science Citation Index (Philadelphia: Institute for Scientific Information, 1961- ). 2. Social Sciences Citation Index (Philadelphia: Institute for Scientific Information, 1973- ). 3. Arts and Humanities Citation Index (Philadelphia: Institute for Scientific Information, 1980- ). 4. JaneL. Caldwell and Celia S. Livingston, "On-Line Bibliometrics," BRS Brief Paper Series, no.4:2 Oan. 1980). 5. Eugene Garfield, "Perspective on Citation Analysis of Scientists" in his Citation Indexing-Its The- ory and Application in Science, Technology and Humanities (New York: Wiley, 1979). 178 College & Research Libraries March 1983 6. Ibid., p.248. 7. Ibid. 8. NormanS. Endler, "Beyond Citation Counts: Developing Research Profiles," Canadian Psycholog- ical Review 19:152-57 (1978). 9. Robert L. Hemreich and others, ''Making It in Academic Psychology: Demographic and Personal- ity Correlates of Attainment," Journal of Personality and Social Psychology 39:896-908 (Nov. 1980). 10. David Lewis Smith, Thomas Roche, and William Snizek, "Quality of Graduate Productivity in Sociology as Measured by the Citation Index: A Longitudinal Overview," Research in Higher Edu- cation 13:343-52 (1980) . 11. NormanS . Endler, J. Philippe Rushton, and Henry L. Roediger ill, "Productivity and Scholarly Impact (Citations) of British, Canadian and U.S. Departments of Psychology (1975)," American Psychologist 33:1064-82 (Dec. 1978). 12. Richard C. Anderson, Francis Narin, and Paul McAllister, "Publication Ratings versus Peer Rat- ings of Universities,'' Journal of the American Society for Information Science 29:91-103 (March 1978). ''Non-Depository'' or Not: An Examination of the Designation of Non-Depository Titles in the Monthly Catalog of United States Government Publications Steven D. Zink The Monthly Catalog of United States Gov- ernment Publications includes citations to various government publications not dis- tributed to federal depository libraries by the Superintendent of Documents. For the most part these elusive publications represent titles which are produced in re- gional government printing facilities or by private contractors for a government agency. Although this distinction alone does not preclude distribution to deposi- tory libraries, the sheer lack of available copies has frequently resulted in only an entry in the Monthly Catalog. 1 The sole dis- tinction in the Monthly Catalog entries be- tween these non-depository titles and the publications distributed by the Superin- tendent of Documents is that the deposi- tory publications are distinguishable by an item selection number and a black dot (•). The black dot simply indicates that the ti- tle was available for distribution to deposi- tory libraries and that, at the very least, re- gional depositories, as required by law, have a copy of the publication. The item selection number is even more crucial. The nearly 5,500 current item numbers represent all of the possible categories of publications available which selective de- pository libraries may elect to receive. For example, item number 17 represents the Yearbook of Agriculture. If a depository would select that item number, it would annually receive the Yearbook of Agricul- ture. Therefore a depository is capable of establishing a selective profile to suit us- ers' needs much like a library would with an approval plan. Absence of these two el- ements indicates that the publication was not available for distribution to depository libraries. In an effort to have as complete a collection of government publications as possible, many depositories have sought to acquire some or all of these non- Steven D. Zink is head, Government Publications Department, University of Nevada Library, Reno. The au- thor wishes to thank library intern Jeanne Smith for her assistance in compiling the data used in this article ..