Untitled-3 562 College & Research Libraries November 2002 562 Evaluating Academic Journals without Impact Factors for Collection Management Decisions Juris Dilevko and Esther Atkinson Juris Dilevko is on the Faculty of Information Studies at the University of Toronto; e-mail: dilevko@fis.utoronto.ca. Esther Atkinson is Assistant Executive Director of the Health Science Informa- tion Consortium of Toronto, Gerstein Science Information Centre, at the University of Toronto; e-mail: esther.atkinson@utoronto.ca. Evaluation of academic journals for collection management decisions is made all the more difficult when some journals do not have impact fac- tors as assigned by the Institute for Scientific Information and its Journal Citation Reports. Focusing on science, technology, and medicine jour- nals, this study presents a method of evaluating such nonranked jour- nals. The method is based on finding a comparator journal to the nonranked journal, distinguishing between original research articles and other article types, tracing citations to these two target journals in citing journals, comparing the quality of the citing journals that cite both target journals, and describing the contextual typology of the citations to the target journals. A case study of two medical science journals, the nonranked Annals of the Royal College of Physicians and Surgeons of Canada and the comparator ranked Canadian Family Physician, illus- trates the method. This method can help in determining the value of a nonranked journal in relation to a ranked journal. here has been a long-standing debate about the kinds of evi- dence that should be used to make informed collection management decisions about journals. Should actual use statistics be relied on, or should citation data and analysis be used? In a 1998 textbook on serials man- agement, Thomas E. Nisonger summa- rized the opposing viewpoints, noting the findings of such scholars as Pauline Scales, who observed relatively low cor- relations between journal citation rankings and actual use of science jour- nals at the National Lending Library in the United Kingdom, and Tony Stankus and Barbara Rice, who concluded that ci- tation data and use data “correlate well” for biochemistry, cell biology, ecology, geosciences, and mathematics journals that are used “at least 25 times per year” at a major university library in the state of New York.1 One reason for these dif- ferent results is “different library con- texts,” and so many scholars recommend that “citation data should be used in com- bination with other evaluative mea- sures.”2 As is well known, citation analysis is based primarily on data produced by the Evaluating Academic Journals without Impact Factors 563 Institute for Scientific Information (ISI), which offers a number of products and databases of great help to librarians and other information professionals. Among these databases are Science Citation In- dex (SCI), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI), each of which “provides access to current and retrospective biblio- graphic information, author abstracts, and cited references” in its respective area.3 In addition, ISI produces Journal Citation Reports (JCR), which is an evalu- ation and analysis tool consisting of “quantifiable statistical data that provides a systematic, objective way to determine the relative importance of journals within their subject categories.”4 JCR contains four ISI measures for evaluating journals: impact factor; immediacy index; cited half-life; and citing half-life. Arguably, impact factor is the most important and most widely used of these measures. It is calculated annually by “dividing the number of current citations a journal re- ceives to articles published in the two previous years by the number of articles published in those same years.”5 Theoreti- cally, the fact that a journal is cited fre- quently indicates that scholars in the field deem it to be influential. Because JCR citation data “may be viewed as reflecting journal usage by re- searchers at the national and international level,” a second debate centers on the question of whether such international and national citation data “correspond to the local needs” of individual libraries.6 Among the initiators of this debate were Robert N. Broadus, Elizabeth Pan, and Maurice B. Line, with Broadus and Pan suggesting that JCR is a valid tool to use when evaluating local collections and Line arguing the opposite.7–10 Stephen E. Wiberley Jr. found that journal rankings based on local faculty citation analysis better predicted local journal use (and subsequent value of that journal to a par- ticular institution) than did citation analy- sis of the kind performed by national da- tabases.11 Keith Swigger and Adeline Wilkes thus recommended that libraries compile local citation indexes to help in serials management decisions, something that has become relatively easy to do through the type of institution-specific rank commands available in electronic databases.12,13 Despite these debates, it is safe to say that “an increasing number of academic and research libraries have begun to re- alize the potential use of citation analysis as a tool for serials collection manage- ment.”14 Yet, when libraries do use JCR to evaluate journals in their collections, it is often the case that “the journals one wishes to evaluate may not be contained in the JCR.”15 For example, of the approxi- mately 126,000 scientific journals pub- lished worldwide, the JCR Science Edi- tion covers only about 5,700 of them.16,17 Faced with the fact that many of the jour- nals in their collections, or journals they may wish to purchase for their collections, are not part of JCR and therefore cannot easily be compared with ranked journals in the same field, collection management librarians may not be properly evaluat- ing such journals for selection and de-se- lection purposes. Starting from the premise that citation analysis is an impor- tant tool for collection management librar- ians, the present study provides a meth- odological framework for evaluating a nonranked journal in comparison with a ranked journal (collectively referred to as target journals) using available citation data and then applies this framework to a case study in the field of medical sci- ence. Method The method for comparing a nonranked journal with a ranked journal involves a number of steps. These steps, together with justifications for each step, are out- lined below according to the order of pro- cedure. Theoretically, the fact that a journal is cited frequently indicates that scholars in the field deem it to be influential. 564 College & Research Libraries November 2002 Finding a Comparator Journal Finding a comparator journal involves examining a significant number of rel- evant journals in the same field that ful- fill a similar function for the same gen- eral readership. Ideally, both the ranked journal and the nonranked journal should have the same number of issues per year, or as close to the same number of issues per year as possible, with approximately the same number of source items (defined below) and articles. Why is this impor- tant? Roger Taylor was among the first to suggest that the number of and length of articles published has a great deal of bear- ing on a journal’s impact factor. A journal publishing a small number of longer re- view papers will have an advantage, in terms of the calculation of impact factor, over one that publishes a greater number of shorter articles, notes, and brief com- munications.18 The reason is that when doing literature reviews for their own articles, scholars gravitate to one perti- nent inclusive review article in their field because it provides a convenient sum- mary and relevant overview of recent re- search rather than using far-flung and possibly difficult-to-find shorter original research articles.19 Some journals, in fact, have moved toward increasing their im- pact factor through the publication of spe- cific types of articles.20 Thus, to get a good sense of the quality of the source items (including articles) in a nonranked jour- nal, it should be compared with a journal that is similar in terms of the number and length of source items and articles, as well as in scope and content.21 Recognizing That All Published Articles Are Not the Same and Thus Comparing Only the Same Types of Articles Eugene Garfield, founder of ISI, ex- plained that ISI processes eleven differ- ent article types and that a journal’s im- pact factor depends on the extent to which these article types are included in JCR source item counts (the denominator for calculating impact factors).22 For example, journals may publish many letters that are frequently cited in the next few years, but if letters are included as source items only when they meet a certain “point” level that qualifies them as “substantive re- search,” that journal’s impact factor may decline because this new and tighter cri- terion not only eliminates many letters as source items (denominators), but also the numerous citations to those letters (nu- merators in the impact factor calculation). Brian D. Scanlan argued that original research articles are most typically “rep- resentative of advances in science.”23 In addition, they are the most frequent type of article published, thus providing a criti- cal mass of data on which to base conclu- sions. Thus, the method described here looks only at original research articles. Nevertheless, if an academic library wishes to evaluate a nonranked journal that typically publishes many review ar- ticles, it would be appropriate to compare review articles in the nonranked journal with review articles in the comparator ranked journal. The main point here is that when comparing two or more jour- nals, similar articles should be compared with similar articles; more precisely, the citations received by one type of article should only be compared with those re- ceived by the same type of article. Fur- thermore, as noted by M. Amin and M. Mabe, when “citations to only selected article types are divided by the number of those selected article types, consider- able differences can emerge from the pub- lished impact factors.”24 In this regard, the best approach is, first, to scan the table of contents (TOC) of each issue of the journal under exami- nation. The TOC typically classifies ar- ticles according to type, indicating whether the editor considers an indi- vidual article to be original research, a brief communication, a viewpoint, an editorial, or a review. However, a second step should be to examine each article to see whether it is, in fact, what it is listed as being. Occasionally, there are surprises. In the sciences, for example, overviews about the future direction of a field or subfield sometimes are classified as “original research articles” when they are Evaluating Academic Journals without Impact Factors 565 really more akin to informed opinion and discussion papers. For those interested in a greater degree of precision, Garfield discussed a “method for identifying jour- nal items that contain substantive re- search” wherein an “algorithm is used that weighs the characteristics of an item,” characteristics that include page counts, number of references, number of authors, and so on.25 To be sure, any such proce- dure runs the risk of being criticized as subjective, but the very attempt to differ- entiate articles by type will be beneficial for accurately evaluating journals. Selecting a Range of Years to Study Selecting a range of years to study con- sists of choosing a manageable number of years so that all research articles of in- terest published during the selected years in both the nonranked journal and the comparator ranked journal can be ana- lyzed in terms of the number of citations each article of interest received, the jour- nals where each article of interest was cited, and the type of article in which each article of interest was cited (e.g., original research article, review article, editorial, letter to the editor). Working backward from the current year, a five-year period then is selected where the last year is at least three years removed from the cur- rent year. By way of example, for the case study discussed below, research proce- dures were conducted in the first two months of 2001. Thus, the five-period fit- ting the above criteria was 1993 to 1997. This three-year gap is necessary to ensure that a sufficient amount of time has elapsed to allow an article to be cited else- where. Overall, the criterion of a relatively recent five-year period allows the genera- tion of current data about the nonranked journal. Then, working within the se- lected five-year period, two or three years should be picked arbitrarily. For the case study discussed below, the years 1994, 1995, and 1997 were selected. Although the authors chose a five-year period and then selected three years within that pe- riod, these choices were arbitrary. That is, if and when others replicate the proce- dures described here, those individuals might choose an initial ten-year period, for example, and then four years within that period. Obviously, if a longer initial time period is selected and a large num- ber of years within that time period are picked, the procedures described here would entail a greater time commitment. Any choices of this kind are necessarily based on such issues as time constraints, staff availability to carry out the proce- dures, and the degree to which the insti- tution is committed to ensuring that nonranked journals are given the same consideration as ranked journals. Recording Information about the Articles of Interest All articles of interest published in the nonranked journal and the comparator ranked journal in the selected years were searched on Web of Science to find a list of articles that had cited them. The fol- lowing data were recorded: the number of citations, if any, that each article of in- terest received; the names of the citing journals; the number of times a citing jour- nal cumulatively cited articles of interest from either the nonranked journal or the comparator ranked journal; and the types of articles in the citing journals in which each article of interest was cited. Calculating a Series of Adjusted Impact Factors for the Nonranked Journal and the Comparator Ranked Journal Based on the traditional ISI impact factor formula, where the total number of cita- tions is divided by the number of citable items, the authors developed an adjusted impact factor for original research articles only by counting the total number of re- search articles of interest in the arbitrarily selected years. This number, rather than the total number of citable source items (which may include features, columns, editorials, and reviews), is used as the denominator. Rather than following Finally, the magnitude of a journal’s impact factor is often influenced by journal self-citation… 566 College & Research Libraries November 2002 Amin and Mabe’s policy, the numerator used in the present study includes all ar- ticle types, including reviews, letters, and editorial material. Next, after gathering the names of the journals that cited the articles of interest from each of the nonranked and compara- tor ranked journals, the cumulative im- pact factors of the citing journals for each of the target journals were calculated us- ing the JCR Science Edition and the JCR Social Sciences Edition. The phrase “cu- mulative impact factor of each citing jour- nal” is defined here as the current-year (2001) JCR impact factor for each citing journal multiplied by the number of times the citing journal cited each of the nonranked and comparator ranked jour- nals, respectively. All the individual cu- mulative impact factors then are added for a “grand total cumulative impact fac- tor.” This grand total for each of the tar- get journals then is divided by the total number of citations received by each tar- get journal. This gives the “impact value of citing journals per citation.” For example, nonranked journal ABC was cited by journal XYZ five times, by journal QRS four times, and by journal LMN three times. Journal XYZ has an impact factor of 1.8, journal QRS an im- pact factor of 2.7, and journal LMN an impact factor of 3.1. Therefore, the grand total cumulative impact factor for the nonranked journal ABC is 29.1 ([5 x 1.8] + [4 x 2.7] + [3.1 x 3]). When this total is divided by the total number of citations received by journal ABC (in this case, 12 [5 + 4 +3]), the journal’s “impact value of citing journals per citation” is 2.425. In addition, journal ABC’s “impact value of citing journals per journal” can be calcu- lated by dividing the total of the unique value of the impact factors of the citing journals by the number of citing journals. In the example above, the total of the unique values of the impact factors of the citing journals is 7.6 (1.8 + 2.7 + 3.1). Be- cause there are three citing journals, 7.6 should be divided by 3 to get the “impact value of citing journals per journal” for journal ABC, which, in this case, is 2.533. These two resulting figures provide a good indication of the range and quality of the journals that have cited the nonranked journal and the comparator ranked journal. Differences in the range and quality of the journals that cite the nonranked journal and the comparator ranked journal may indicate differences in the perception of the quality of the nonranked and comparator ranked jour- nals by others. In those situations where nonranked journal ABC is cited by an- other nonranked journal, a value of zero (0) should be assigned to the nonranked journal that cites nonranked journal ABC, and proceed as above. Finally, the magnitude of a journal’s impact factor is often influenced by jour- nal self-citation; that is, articles published in a specific journal are subsequently cited by other articles published in that same journal presumably because journal edi- tors may prefer to accept papers “with prolific citations of the publishing jour- nal.”26 Whether or not this is true, jour- nals that have high self-citation rates nec- essarily increase their impact factors, as shown in separate studies examining self- citation practices of anesthesia and urol- ogy journals.27,28 A high self-citation rate also may indicate that the journal is not, relatively speaking, highly regarded by scholars outside the immediate and regu- lar circle of readers of that journal, that it is, in short, of limited scope. In light of these considerations, it is appropriate to calculate, as outlined above, the adjusted impact factor with self-citations removed. Comparing the Quality of Citing Journals That Cite the Nonranked Journal and the Comparator Ranked Journal In this step, citing journals are ranked ac- cording to their JCR impact factors to more precisely determine the reputations of the nonranked and comparator ranked journals. In addition, it is useful to list the citing journals that cite the target journals most frequently. For example, if one of the target journals is cited almost exclusively by citing journals with impact factors of less than one and the other target journal Evaluating Academic Journals without Impact Factors 567 is cited to a great extent by citing jour- nals with high impact factors, an impor- tant conclusion may be drawn about the relative quality of the target journals. In this example, the second-mentioned tar- get journal is used more often by authors who publish in highly regarded publica- tions than the first-mentioned target jour- nal is. This difference in type of reader- ship may suggest that authors who publish in highly regarded journals may not consider the first-mentioned target journal as sufficiently worthy to cite. In addition, ranking the frequency with which citing journals cite a target journal gives a clear picture of the degree to which a target journal cites itself. Moreover, it is interesting to calculate the “impact value of citing journals per citation with self-citations removed” of the nonranked journal and the comparator ranked journal. In the present study, de- pending on whether the impact factor of a target journal is relatively low or high, the “impact value of citing journals per cita- tion with self-citations removed” will rise or fall, respectively. As an example, the ranked journal DEF has been cited a hun- dred times, including thirty times by jour- nal DEF itself, and has a “grand total cu- mulative impact factor” of 500. Its “impact value of citing journals per citation” is therefore five (500 divided by 100). Say that journal DEF has an impact factor of .325. In this case, only 9.75 (30 times .325) of the “grand total cumulative impact factor” of 500 is accounted for by journal DEF itself. Thus, when the “impact value of citing journals per citation with self-citations re- moved” is calculated, the 9.75 value is re- moved, as are thirty self-citations. The new calculation gives a value of 7.004 [(500 – 9.75) divided by 70]. However, if journal DEF has an impact factor of six, 180 (30 times six) of the “grand total cumulative impact factor” of 500 is accounted for by journal DEF itself. Thus, when the “impact value of citing journals per citation with self-citations removed” is calculated, the 180 value is removed, as are the thirty self- citations. The new calculation gives a value of 4.571 [(500 – 180) divided by 70]. In the case of a nonranked journal with no published impact factor, the same prin- ciples as above apply. Consider unranked journal STW, which has been cited a hun- dred times, including thirty self-citations. It has a “grand total cumulative impact factor” of 500. Therefore, its “impact value of citing journals per citation” is five (500 divided by 100). Because journal STW is unranked, it is assigned a zero impact fac- tor value and thus does not account for any of the value of the “grand total cu- mulative impact factor” of 500. When the “impact value of citing journals per cita- tion with self-citations removed” is cal- culated, the zero value is removed, as are the thirty self-citations. The new calcula- tion gives a value of 7.143 [500 divided by 70]. Admittedly arcane, these calcula- tions nevertheless give a clearer indica- tion of the relative value of the journals that cite the target journals under study. Performing an Analysis of Citation Context As discussed by Garfield, contextual analy- sis of citations is used “to find out why” a relevant work has been cited. It may very well be that an author is highly cited, but that most of the citations are negative (i.e., they refute that author’s work).29 To gar- ner more information about the reasons authors cited other authors, Daryl E. Chubin and Soumyo D. Moitra developed six mutually exclusive categories: affirma- tive essential basic, affirmative essential subsidiary, affirmative supplementary additional, affirmative supplementary per- functory, negational partial, and negational total. An affirmative essential basic citation is defined as a “referenced paper [that] is declared central to the reported research [or it is] a reference on which its findings depend.” An affirmative essential subsid- iary citation is one in which “a specific method, tool, or mathematical result is not directly connected to the subject of the paper, but it is still essential to the reported research.” An affirmative supplementary additional citation occurs when “the ref- erenced paper contains an independent supportive observation (idea or finding) 568 College & Research Libraries November 2002 with which the citer agrees.” An affirma- tive supplementary perfunctory citation relates “to the reported research without additional comment.” A negational partial citation occurs when “a citer suggests that the [cited] paper is erroneous in part and then offers a correction.” Finally, a nega- tional total citation is one in which “a citer refers to the [cited] paper as being com- pletely wrong and offers an independent interpretation or solution.”30 Of course, Chubin and Moitra are not the only ones to have created citation typologies, but their system is well recognized and refer- enced positively by Garfield.31 Practically speaking, this step involves physically gathering all the articles that cited the articles of interest from the An- nals of the Royal College of Physicians and Surgeons of Canada (Annals) and Canadian Family Physician (CFP) and then reading them to make a judgment about citation types. Often many of the citing articles may only be available through interli- brary loans, and this circumstance adds to the time needed to accomplish this step of the procedure. Although contextual analysis is used most often to investigate the quality of an individual author’s citations, it is used in the present study as a measure of the qual- ity of particular journals. In effect, the to- tality of the authors publishing in a target journal may be viewed metaphorically as representing the journal itself, and so a contextual analysis of individual authorial citations, when added together, is one gauge of journal quality insofar as that journal has chosen to publish those indi- vidual authors. This approach is valuable because it allows, for instance, a journal that very rarely receives affirmative essen- tial basic citations or affirmative essential subsidiary citations to be differentiated from one that receives a significant num- ber of these types of citations. Still, classi- fication of citation context into categories can be a subjective undertaking, and col- lection development specialists may want to consider a procedure that takes into ac- count intercoder reliability formulae, one of which was provided by Ole Holsti.32 Case Study The above procedures were applied to a case study involving the Annals, a broadly based medical and health science journal that appears eight times annually and has been published in Canada since 1968. Its mandate is to publish articles dealing with “clinical or laboratory topics, medi- cal education, bioethics, history of medi- cine, or the social, economic and political aspects of patient care [and that are] writ- ten for all specialists, or for a range of specialists.”33 It is not included in either JCR Science Edition or JCR Social Sciences Edition. To choose a suitable comparator jour- nal for the Annals, the domain of Canadian medical science publishing was examined. This geographic restriction was imposed because little attention is typically paid to Canadian medical publishing, which tends to be overshadowed by medical publish- ing emanating from the United States and the United Kingdom. Twenty-one other medical science journals whose country of origin is Canada are listed in SCI JCR: Ca- nadian Association of Radiologists Journal; Canadian Family Physician; Canadian Jour- nal of Anaesthesia; Canadian Journal of Ap- plied Physiology; Canadian Journal of Cardi- ology; Canadian Journal of Dietetic Practice and Research; Canadian Journal of Gastroen- terology; Canadian Journal of Neurological Sciences; Canadian Journal of Ophthalmology; Canadian Journal of Physiology and Pharma- cology; Canadian Journal of Psychiatry; Ca- nadian Journal of Surgery; Canadian Medical Association Journal; Clinical Biochemistry; European Journal of Gynaecological Oncology; Journal of Otolaryngology; Journal of Pallia- tive Care; Journal of Pharmacy and Pharma- ceutical Sciences; Journal of Psychiatry & Neu- roscience; Journal of Rheumatology; and Obe- sity Surgery. Of these, nineteen are meant for various specialists and subspecialists. Only CFP and the Canadian Medical Asso- ciation Journal (CMAJ) are general-interest medical journals and thus appear under the JCR subject category of “Medicine, General & Internal.” Like the Annals, both CFP and CMAJ publish a wide range of articles, including original clinical re- Evaluating Academic Journals without Impact Factors 569 search, reviews, case histo- ries, critical appraisals, fea- tures, and editorials. Both would be good comparators for the Annals from the per- spective of scope, content, and editorial purpose. However, CFP appears twelve times per year whereas CMAJ publishes twenty-four issues annu- ally. Accordingly, CFP was selected as the comparator journal for the Annals because it is closer to the Annals in the number and length of articles published. Although CFP relies on advertising revenue to a greater degree than the Annals does and focuses exclu- sively on family medicine, a large number of the articles and features in the Annals also would be useful to family medicine practitioners. The impact factor for CFP has increased from .053 in 1993 to .348 in 2001, moving “its relative ranking from 98th to 66th place in the world medical literature.”34 Given this ranking, should the Annals be seen as being below, equal to, or above CFP in terms of quality? The Annals received 112 citations of all kinds to the 99 research original research articles it published in 1994, 1995, and 1997.35 CFP received 264 citations to the 102 original research articles it published in the same years. As shown in table 1, research articles from CFP were cited by other research articles in citing journals at a slightly greater rate (79.9%) than were research articles from the Annals (75.9%). The same is true for review articles. Al- though research articles from CFP were cited twenty-one times (7.9%) in review articles in citing journals, research articles from the Annals were cited in review ar- ticles in citing journals only five times (4.5%). On the other hand, research ar- ticles from the Annals were cited at a sig- nificantly greater rate (14.3%) in editori- als in citing journals than research articles from CFP (4.2%) were cited in editorials. In general, research articles in CFP seem to be slightly more highly regarded by scholars working in the field whereas re- search articles in the Annals generate more comment from editors of journals. This may show a qualitative difference in the type of research articles published by the two journals: Articles in the Annals may raise controversial issues that demand immediate comment in editorials but are unlikely to have much bearing on other original research. As indicated in table 2, of the 99 re- search articles published in the Annals TABLE 2 Frequency of Citations to Research Articles Appearing in the Annals and CFP Annals CFP Articles with 10 or more citations 2 (2%) 2 (2%) Articles with 6�9 citations 3 (3%) 10 (9.8%) Articles with 4�5 citations 3 (3%) 16 (15.7%) Articles with 2�3 citations 13 (13.1%) 29 (28.4%) Articles with 1 citation 15 (15.2%) 26 (25.5%) Articles with no citations 63 (63.6%) 19 (18.6%) Total articles 99 (100%) 102 (100%) TABLE 1 Comparison of Publication Types that Cited Research Articles from the Annals and CFP Annals CFP Cited in articles 85 (75.9%) 211 (79.9%) Cited in reviews 5 (4.5%) 21 (7.9%) Cited in editorials 16 (14.3%) 11 (4.2%) Cited in letters 6 (5.4%) 20 (7.6%) Cited in corrections 0 (0%) 1 (0.4%) Total 112 (100%) 264 (100%) 570 College & Research Libraries November 2002 and 102 published in CFP, only two ar- ticles from each journal were cited ten or more times by citing journals. However, twenty-eight research articles from CFP were cited four or more times (27.5%) whereas only eight research articles from the Annals were cited four or more times (8%). In addition, 63.6 percent of research articles from the Annals were never cited, compared to only 18.6 percent of articles from CFP. More specifically, one research article from the Annals was cited twenty- four times, constituting 21.4 percent (24 out of 112) of all citations in citing jour- nals to research articles published in the Annals in the years in question. In fact, the top-two-cited research articles from the Annals (35 citations) make up 31.3 percent (35 out of 112) of its total citations whereas the top-two-cited research ar- ticles from CFP (25 citations) comprise 9.5 percent (25 out of 264) of its total citations. Based on these figures, the quality of the articles in the Annals is substantially more mixed than those in CFP. Many more re- search articles in CFP receive at least one or two citations than do articles in the Annals, suggesting that the overall qual- ity of the research articles published by CFP may be higher than the overall qual- ity of the research articles published in the Annals. As shown in table 3, when an adjusted impact factor is calculated for these two journals based on the formula described in the method described earlier, the ad- justed impact factor is 1.131 for the An- nals and 2.588 for CFP. Yet, when the quality of the journals that cite each of the target journals is taken into account and calculated in the manner outlined earlier, the “impact value of citing jour- nals per citation” of the Annals is 2.615, but only 1.696 for CFP. In addition, the “impact value of citing journals per jour- nal” of the Annals is 2.469, as opposed to 1.904 for CFP. This difference in favor of the Annals holds true even when self-ci- tations are removed from the calcula- tions. The “impact value of citing jour- nals per citation with self-citations re- moved” remains 2.615 for the Annals (be- cause it has no self-citations) whereas the “impact value of citing journals per cita- tion with self-citations removed” of CFP increases to 2.336. TABLE 3 Comparison of Impact Factors and �Impact Values� Variously Calculated Adjusted impact factor as calculated per procedures described in the present study1 Adjusted impact factor as calculated per formula described in the present study, but with self-citations removed1 Grand total cumulative impact factor of citing journals2 Impact value of citing journals per citation2 Impact value of citing journals per journal2 Impact value of citing journals per citation with self- citations removed (and also own cumulative impact factor removed)2 Annals 1.131 1.131 292.834 2.615 2.469 2.615 CFP 2.588 1.755 447.77 1.696 1.904 2.336 1 This is not the ISI impact factor, but follows in broad terms the traditional definition of impact factor 2 Based on ISI data Evaluating Academic Journals without Impact Factors 571 In addition, as shown in table 4, the Annals is cited by fifteen citing journals that have impact factors of three and above, which is 23.1 percent of its total citations. Although CFP is cited thirteen times by citing journals that have impact factors of three and above, this only rep- resents 13 percent of its total citations. Fifteen journals with ISI impact factors of three and above cited the Annals nineteen times. (See table 5.) The cumulative im- pact factor of these journals was 159.117, and after dividing the cumulative impact factor by 19, the “impact value of high- impact factor citing journals per citation” of the Annals is 8.375, as shown in table 5. On the other hand, thirteen journals with ISI impact factors of three and above cited CFP twenty-four times, as shown in table 6. The cumulative impact factor of these TABLE 4 Comparison of the Quality of Citing Journals that Cited the Annals and CFP Number of Times Cited by This Type of Journal: Annals CFP Journals with ISI impact factors above five 7 (10.8%) 6 (6%) Journals with ISI impact factors between three and five 8 (12.3%) 7 (7%) Journals with ISI impact factors of two 9 (13.8%) 19 (19%) Journals with ISI impact factors of one 18 (27.7%) 29 (29%) Journals with ISI impact factors of less than one 20 (30.8%) 35 (35%) Journals with no ISI impact factor 3 (4.6%) 4 (4%) Total of different citing journals 65 (100%) 100(100%) TABLE 5 Journals with Impact Factors of Three (3) and Above That Cited the Annals Journal ISI Impact Frequency Cumulative Factor for 2001 Impact Factor JAMA�Journal of the American Medical Association 17.569 4 70.276 Journal of the National Cancer Institute 14.24 1 14.24 Lancet 13.251 1 13.251 British Medical Journal 6.629 2 13.258 Journal of the American Society of Nephrology 6.337 1 6.337 American Journal of Medicine 6.106 1 6.106 Neurology 5.212 1 5.212 Kidney International 4.815 1 4.815 Psychological Inquiry 4.214 1 4.214 Cancer 3.909 1 3.909 American Journal of Kidney Diseases 3.614 1 3.614 Seminars in Oncology 3.6 1 3.6 Critical Care Medicine 3.486 1 3.486 European Journal of Cancer 3.46 1 3.46 Cancer Treatment Reviews 3.339 1 3.339 Total 19 159.117 Impact value of high-impact factor citing journals per citation (based on ISI data) 8.375 572 College & Research Libraries November 2002 journals was 177.389, and after dividing the cumulative impact factor by 24, the “impact value of high-impact factor cit- ing journals per citation” of CFP is 7.391. Based on data presented in tables 3 through 6, it can be argued that, on aver- age, when the Annals is cited by a jour- nal, the quality of that citing journal, as measured by ISI impact factors and the calculated impact values, is better than the quality of the journal citing CFP. As shown in tables 7 and 8, research articles in the Annals do not make a prac- tice of citing other research articles pub- lished in the Annals. Quite the opposite occurs in the case of CFP, which cites it- self eighty-five times (32.2%). This cir- cumstance has the effect of increasing its impact factor. If all self-citations were eliminated when calculating the impact factor for CFP, it would decrease from 2.588 to 1.755 (179 citations from other TABLE 6 Journals with Impact Factors of Three (3) and Above That Cited CFP Journal ISI Impact Frequency Cumulative Factor for 2001 Impact Factor JAMA�Journal of the American Medical Association 17.569 4 70.276 Archives of General Psychiatry 11.981 1 11.981 Journal of Clinical Oncology 8.53 1 8.53 Archives of Internal Medicine 6.661 3 19.983 British Medical Journal 6.629 4 26.516 American Journal of Medicine 6.106 1 6.106 British Journal of Cancer 3.942 2 7.884 Pediatrics 3.708 2 7.416 Sexually Transmitted Diseases 3.212 1 3.212 Annals of Oncology 3.153 1 3.153 Psychopharmacology 3.145 1 3.145 Psychological Medicine 3.119 1 3.119 American Journal of Public Health 3.034 2 6.068 Total 24 177.389 Impact value of high-impact factor citing journals per citation (based on ISI data) 7.391 TABLE 7 Journals that Cited the Annals Most Frequently Times Cited 2001 Impact Journal the Annals (n = 112) Factor Canadian Medical Association Journal 16 (14.3%) 2.762 Psycho-Oncology 6 (5.4%) 1.388 Canadian Family Physician 5 (4.5%) 0.348 Canadian Journal of Psychiatry 5 (4.5%) 1.624 JAMA�Journal of the American Medical Association 4 (3.6%) 17.569 Clinical and Investigative Medicine 3 (2.7%) 0.883 Academic Psychiatry 3 (2.7%) 0.275 Evaluating Academic Journals without Impact Factors 573 journals to CFP research articles divided by 102 published research articles). (See the second row of figures in table 3.) Al- though journal self-citation is not neces- sarily a negative factor, it nevertheless serves as a warning that a journal may not, for a variety of reasons, be engaging in as wide an intellectual dialogue with the readership of other journals than it might. Based on Chubin and Moitra’s citation typology (shown in table 9), the Annals received fifteen affirmative essential ba- sic or affirmative essential subsidiary ci- tations (13.4%), compared with CFP’s three (1.1%). (See tables 10 and 11.) More- over, the Annals had seventeen affirma- tive supplementary additional citations (15.2%), compared with CFP’s twenty- seven (10.2%). Viewed from another angle, 224 (84.8%) of the citations to CFP were affirmative supplemental perfunc- tory, compared with 74 (66.1%) for the Annals. In absolute terms, though not in percentage terms, CFP also had slightly more negational partial citations (10) than the Annals (6). In terms of Chubin and Moitra’s typology, research articles pub- lished in the Annals were made use of by citing papers in a slightly more “core” manner than were research articles pub- lished in CFP. Overall, when compared with CFP, the Annals exhibits some positive and nega- tive qualities. On the negative side, 63.6 percent of the research articles published by the Annals are never cited elsewhere (table 2). Indeed, two articles make up a significant percentage (31.3%) of the to- tal citations received by the Annals as a whole. Thus, the Annals has an adjusted impact factor of only 1.131, compared with a revised impact factor of 2.588 for CFP (table 3). Less troubling, although noteworthy, is the fact that 14.3 percent of the citations to the Annals in citing jour- nals are in editorials, compared with only 4.2 percent for CFP (table 1). On the posi- tive side, the Annals does not engage in self-citation whereas about one-third of all citations to CFP (85) are self-citations (table 8). Thus, when self-citations are removed, the adjusted impact factor of CFP falls to 1.755, which is still higher than the adjusted impact factor of the Annals (1.131), but nevertheless closer to it (table 3). Moreover, the journals that cite the Annals have, on average, a higher “im- pact value” than those that cite CFP, and its research articles are cited as Chubin TABLE 8 Journals that Cited CFP Most Frequently Times Cited 2001 Impact Journal the Annals (n = 264) Factor Canadian Family Physician 85 (32.2%) 0.348 Canadian Medical Association Journal 28 (10.6%) 2.762 Academic Medicine 6 (2.3%) 1.401 Journal of Palliative Care 6 (2.3%) 0.622 Journal of General Internal Medicine 4 (1.5%) 2.404 JAMA�Journal of the American Medical Association 4 (1.5%) 17.569 Alcohol and Alcoholism 4 (1.5%) 1.753 International Journal of Geriatric Psychiatry 4 (1.5%) 1.778 British Medical Journal 4 (1.5%) 6.629 British Journal of General Practice 4 (1.5%) 1.418 Thus, the choice of comparator journal is crucial. 574 College & Research Libraries November 2002 and Moitra’s top three citational types (af- firmative essential basic, affirmative es- sential subsidiary, and affirmative supple- mentary additional) to a greater extent than research articles published by CFP, especially in terms of percentages. For collection management librarians special- izing in medical science journals, any de- cision to stop, or not to begin, a subscrip- tion to the Annals should take into con- sideration the factors mentioned above. The evidence presented here suggests that the Annals and CFP may be considered within the same general qualitative level for the purposes of journal selection or de-selection decisions. TABLE 9 Chubin and Moitra�s Citation Typology Definition* The referenced paper is declared central to the reported research [or it is] a reference on which its findings depend. A specific method, tool, or mathematical result is not directly connected to the subject of the paper, but is still essential to the reported research. The referenced paper contains an independent supportive observation (idea or finding) with which the citer agrees. Related to the reported research without additional comment. A citer suggests that the [cited] paper is erroneous in part and then offers a correction. A citer refers to the [cited] paper as being completely wrong and offers an independent interpretation or solution. Category Affirmative essential basic Affirmative essential subsidiary Affirmative supplementary additional Affirmative supplementary perfunctory Negational partial Negational total * Definitions are taken from Daryl E. Chubin and Soumyo D. Moitra, �Content Analysis of References: Adjunct or Alternative to Citation Counting,� Social Studies of Science 5 (1975): 423�441. TABLE 10 Types of Citations Received by the Annals Category Type 1994 1995 1997 All Three Years Affirmative essential basic 1 1 0 2 (1.8%) Affirmative essential subsidiary 4 7 2 13 (11.6%) Affirmative supplementary additional 8 6 3 17 (15.2%) Affirmative supplementary perfunctory 41 27 6 74 (66.1%) Negational partial 5 1 0 6 (5.4%) Negational total 0 0 0 0 (0%) Total 59 42 11 112 Evaluating Academic Journals without Impact Factors 575 Of course, had a different comparator journal for the Annals been picked, the Annals may not have fared so well. Thus, the choice of comparator journal is cru- cial. In addition, the present case study should in no way be interpreted as call- ing into question the high quality of CFP. CFP has a well-earned reputation as an important medical journal. Rather, the purpose of this case study is to provide a model for the analysis of the quality of a nonranked journal using a ranked jour- nal as a basis of comparison. Conclusion Based on the present case study, a jour- nal that has not been assigned an ISI im- pact factor, and is thus not readily sus- ceptible to ranking, should not be viewed automatically as less worthy than a jour- nal that is ranked by ISI. It should not be rejected in favor of a ranked journal with- out extensive study. A process such as the one described here allows collection man- agement specialists to make informed decisions about a nonranked journal by relying on a wide range of both quantita- tive and qualitative citation criteria. Oth- erwise, as in the procedure outlined by Rikie Deurenberg with regard to a method of ranking journals for de-selec- tion at a medical library in the Nether- lands, journals without an impact factor are placed at a disadvantage.36 Eliminat- ing journals with low or nonexistent im- pact factors often means eliminating ex- cellent papers because, as a study of a random sample of 120 articles from three pharmacology journals showed, a journal’s impact factor does not represent the impact factor of an individual article, nor the reputation of the researcher or researchers authoring a specific article.37,38 To be sure, the method described here concentrates on a relatively current im- pact factor based on an arbitrary selec- tion of three years, from 1993 to 1997. However, as Garfield demonstrated, the calculation of long-term cumulative im- pact factors (based on fifteen years of data) significantly affected, in both an upward and downward fashion, the rankings of numerous top-ranked sci- ence journals.39 For instance, Lancet dropped to 28th place in the fifteen-year ranking carried out in 1998 from its third- place rank in 1983. On the other hand, the Journal of General Physiology was ranked 110th in 1983 but soared to 20th place in the cumulative 1998 ranking.40 Hepatology rose from 199th place in 1983 to 29th place in 1998. Two lessons may be drawn from the fact that impact fac- tors are fluid over time. First, the proce- dures delineated in the present study should be repeated on a regular basis to gauge the long-term stability and qual- ity of an unranked journal. Second, it would be foolhardy to suggest that, de- spite their low rankings, the Journal of General Physiology and Hepatology were not important journals in 1983. In much the same way, it would be foolhardy to suggest that a journal such as the Annals of the Royal College of Physicians and Sur- geons of Canada is not an important jour- TABLE 11 Types of Citations Received by CFP Category Type 1994 1995 1997 All Three Years Affirmative essential basic 0 0 2 2 (0.8%) Affirmative essential subsidiary 0 1 0 1 (0.4%) Affirmative supplementary additional 5 6 16 27 (10.2%) Affirmative supplementary perfunctory 101 60 63 224 (84.8%) Negational partial 1 1 8 10 (3.8%) Negational total 0 0 0 0 (0%) Total 107 68 89 264 576 College & Research Libraries November 2002 nal because of its unranked state in JCR. As the method and case study described above show, it compares well with the ranked Canadian Family Physician. Finally, the procedures described above are time-consuming and labor-in- tensive. Not every collection develop- ment department will want to devote the extensive person-hours to an exercise of this kind. Yet, the procedures reported here show that it is theoretically possible to compare unranked and ranked journals using available citation data. Certainly, there may be disagreements about many of the assumptions, procedures, and mea- sures outlined above. Some may find them overly abstruse and, in the final analysis, inconsequential. Nonetheless, the present study is an attempt to show that unranked journals can be ranked using citation data. Large academic librar- ies with significant science, technology, and medicine (STM) journal collections, as well as hospital libraries, may find the principles discussed in the present study sufficiently interesting to develop a sys- tem of their own to compare ranked and unranked STM journals—a system that does not rely on usage data. As well, other large academic libraries with significant holdings in arts & humanities and social sciences journals may want to explore using some aspects of the procedural model described here as a starting point in thinking about ways to improve their collection management decision making with regard to unranked journals in these broad areas. Notes 1. Thomas E. Nisonger, Management of Serials in Libraries (Englewood, Colo.: 1998), 147. 2. Ibid., 148. 3. Institute of Scientific Information, “Science Citation Index Expanded,” available online from http://www.isinet.com/isi/products/citation/scie/index.html. Accessed 10 September 2002. 4. ———, “Journal Citation Reports,” available online from http://www.isinet.com/isi/prod- ucts/citation/jcr/index.html. Accessed 10 September 2002. 5. M. Amin and M. Mabe, “Impact Factors: Use and Abuse,” Perspectives in Publishing 1 (Oct. 2000): 2. Available online from http://www.elsevier.com/homepage/about/ita/editors/ perspectives1.pdf. Accessed 10 September 2002. 6. Nisonger, Management of Serials in Libraries, 123. 7. Robert N. Broadus, “A Proposed Method for Eliminating Titles from Periodical Subscrip- tion Lists,” College & Research Libraries 46 (Jan. 1985): 30–35. 8. Elizabeth Pan, “Journal Citation as a Predictor of Journal Usage in Libraries,” Collection Management 2 (spring 1978): 29–38. See especially her statement that there was a “statistically significant correction … between the ranking of journals by their citation and use counts” (p. 31). 9. Maurice B. Line, “Rank Lists Based on Citations and Library Uses as Indicators of Journal Usage in Individual Libraries,” Collection Management 2 (winter 1978): 313–17. See his statement that “no measure of journal use other than one derived from a local-use study is of any signifi- cant practical value to librarians” (p. 313). 10. A good overview of these debates is provided by Klaus G. Altmann and G. E. Gorman, “Usage Citation Analysis and Costs as Indicators for Journal Deselection and Cancellation: A Selective Literature Review,” Australian Library Review 13 (Nov. 1996): 379–92. 11. Stephen E. Wiberley Jr., “Journal Rankings from Citation Studies: A Comparison of Na- tional and Local Data from Social Work,” Library Quarterly 52 (Oct. 1982): 348–59. 12. Keith Swigger and Adeline Wilkes, “The Use of Citation Data to Evaluate Serials Sub- scriptions in an Academic Library,” Series Review 17, no. 2 (1991): 41–46, 52. 13. Stephanie C. Haas and Vernon N. Kisling Jr., “The Use of Electronic Ranking to Analyze Scientific Literature Used in Research at the University of Florida,” Collection Management 18, no. 3/4 (1994): 49–62. 14. Nisonger, Management of Serials in Libraries, 123. 15. Ibid., 132. 16. G. H. Whitehouse, “Citation Rates and Impact Factors: Should They Matter?” Journal of Radiology 74 (2001): 1–3. 17. Institute of Scientific Information, “Journal Citation Reports.” 18. Roger Taylor, “Is the Impact Factor a Meaningful Index for the Ranking of Scientific Re- Evaluating Academic Journals without Impact Factors 577 search Journals?” Canadian Field Naturalist 95 (July–Sept. 1981): 236–40. 19. F. Hecht, B. K. Hecht, and A. A. Sandberg, “The Journal ‘Impact Factor ’: A Misnamed, Misleading, Misused Treasure,” Cancer Genetics & Cytogenetics 104 (1998): 77–81. 20. J. Gowrishankar and P. Divakar, “Sprucing Up One’s Impact Factor,” Nature 401 (1999): 321–22. 21. Tony Stankus and Barbara A. Rice, “Handle with Care: Use and Citation Data for Science Journal Management,” Collection Management 4 (1982): 95–110. 22. Eugene Garfield, “Which Medical Journals Have the Greatest Impact?” Annals of Internal Medicine 105 (1986): 313–20. Available online from http://www.garfield.library.upenn.edu/es- says/v10p007y1987.pdf. Accessed 8 July 2002. 23. Brian D. Scanlan, “Coverage by Current Contents and the Validity of Impact Factors: ISI from a Journal Publisher’s Perspective,” Serials Librarian 13, no. 2/3 (1987): 57–66. 24. Amin and Mabe, “Impact Factors,” 6. 25. Garfield, “Which Medical Journals Have the Greatest Impact?” 316. 26. N. C. Janke, “Journal Evaluation,” Science 182 (1973): 1996–97. 27. A. Fassoulaki, A. Paraskeva, K. Papilas, and G. Karabinis, “Self-citations in Six Anaesthe- sia Journals and Their Significance in Determining The Impact Factor,” British Journal of Anaes- thesia 84 (2000): 266–69. 28. Tiiu Ojasoo and JeanChristophe Dore, “Citation Bias in Medical Journals,” Scientometrics 45 (1999): 81–94. 29. Eugene Garfield, “How to Use Citation Analysis for Faculty Evaluations, and When Is It Relevant? Part 2,” Current Contents 45 (1983): 5–14. Available online from http:// www.garfield.library.upenn.edu/essays/v6p363y1983.pdf. Accessed 9 July 2002. 30. Daryl E. Chubin and Soumyo D. Moitra, “Content Analysis of References: Adjunct or Alternative to Citation Counting,” Social Studies of Science 5 (1975): 423–41. 31. Another citation typology is contained in M. J. Moravcsik and P. Murugesan, “Some Re- sults on the Function and Quality of Citations,” Social Studies of Science 5 (1975): 86–92. 32. Ole R. Holsti, Content Analysis for the Social Sciences and Humanities (Reading, Mass.: Addison-Wesley, 1969). 33. Annals of the Royal College and Surgeons of Canada, available online from http:// rcpsc.medical,org/english/annals/contributors_e.php3. Accessed 12 July 2002. 34. Canadian Family Physician, available online from http://www.cfpc.ca/communications/ cfp/cfpeabvision.asp. Accessed 11 July 2002. 35. Great difficulty was experienced in categorizing twelve items in Annals 28, no. 8, for 1995. This volume contained twelve short pieces with titles typically beginning with “Future Trends.” Although the Annals calls these pieces research articles, the authors did not feel that they were, in fact, research articles, and so these items (and any citations to them) were not counted in their calculations and tables. Nevertheless, the authors report this quandary here. If these twelve items were counted, the Annals would have 114 citations to 111 articles. In table 3, its adjusted impact factor would be 1.027; its “grand total cumulative impact factor” would be 293.316; its “impact value of citing journals per citation” would be 2.573; its “impact value of citing journals per journal” would be 2.435; and its “impact value of citing journals with self-citations removed” would be 2.435. These are not large differences and do not affect the reported results. 36. Rikie Deurenberg, “Journal Deselection in a Medical University Library by Ranking Peri- odicals Based on Multiple Factors,” Bulletin of the Medical Library Association 81 (July 1993): 316– 19. 37. J. Boldt, G. Haisch, and W. H. Maleck, “Changes in the Impact Factor of Anesthesia/ Critical Care Journals within the Past 10 Years,” Acta Anaesthesiologica Scandinavica 44 (2000): 842–49. 38. A. Rostami-Hodjegan and G. T. Tucker, “Journal Impact Factors: A ‘Bioequivalence’ Is- sue?” British Journal of Clinical Pharmacology 51 (2001): 111–17. 39. Eugene Garfield, “Long-term vs. Short-term Journal Impact: Does It Matter?” Scientist 12 (Feb. 2, 1998): 1-7. Available online from http://www.garfield.library.upenn.edu/commentar- ies/tsv12(03)p10y19980202.pdf. Accessed 8 July 2002. 40. The data in Garfield’s article show that the Journal of the National Cancer Institute also was ranked 110th in 1983.