antelman.indd Do Open-Access Articles Have a Greater Research Impact? Kristin Antelman Although many authors believe that their work has a greater research impact if it is freely available, studies to demonstrate that impact are few. This study looks at articles in four disciplines at varying stages of adoption of open access—philosophy, political science, electrical and electronic engineering and mathematics—to see whether they have a greater im- pact as measured by citations in the ISI Web of Science database when their authors make them freely available on the Internet. The finding is that, across all four disciplines, freely available articles do have a greater research impact. Shedding light on this category of open access reveals that scholars in diverse disciplines are adopting open-access practices and being rewarded for it. here is currently an explosion of interest in the academic and publishing communities about the promise—and pos- sible perils—of open-access scholarship and publishing. Although the term “open access” is somewhat fluid, included under its banner are the so-called “two roads to open access”: open-access journals and “e-print” (i.e., preprints or postprints) repositories, both of which make the full text of scholarly articles freely available to everyone on the open Internet.1 Al- though debate swirls around questions of copyright, peer review, and publishing costs, individual authors are taking action in this arena by posting their articles to personal or institutional Web pages and to disciplinary repositories. Since Steve Lawrence circulated his study of the impact of free online avail- ability of computer science conference documents under the catchy title “Online or Invisible,” the notion that freely avail- able papers have a greater research im- pact has taken hold. (Lawrence’s preprint itself was evidence of this phenomenon: the version published in the journal Na- ture [under a different title] was almost an a erthought; more than seven hundred documents in Google reference “Online or Invisible.”2) Despite the limited do- main of the Lawrence study, making it difficult to make assumptions about other disciplines based on his findings, it is now common to see the assertion that research impact is increased by open access. For example, Eugene Garfield casually noted in a recent post to the American Scientist Open Access Forum listserv that “it has been demonstrated that on line access improves both readership and citation Kristin Antelman is Associate Director for Information Technology at North Carolina State University Libraries; e-mail: kristin_antelman@ncsu.edu. 372 mailto:kristin_antelman@ncsu.edu Do Open-Access Articles Have a Greater Research Impact? 373 impact.”3 Although this statement may be intuitively believable, the evidence to document it is still being collected. In addition to the Lawrence study, small studies of the research impact of e-prints have been done for physics, chemistry, and a subdiscipline of computer sci- ence.4–6 Preliminary results from a larger study, which will look at articles from seven thousand journals from the ISI Web of Science database, indicate a significant increased research impact for open-access articles in physics.7 In related research, a study performed using the Citebase data found that the more o en a paper is downloaded, the more likely it is to be cited (with the strongest correlation for high-citation papers and authors).8 In April 2004, ISI released a study titled “The Impact of Open Access Journals,” in which it compared impact factor and the number of citations of open-access journals in the natural sciences with non-open-ac- cess journals. ISI found that “the OA jour- nals have a broadly similar citation pa ern to other journals, but may have a slight tendency to earlier citations.”9 However, it qualifies the findings by noting that many of the journals in the study only recently shi ed to open access, that high-profile ti- tles (such as PLoS Biology) were too new to be included, and that their relatively small sample included many regional titles that would not be expected to be high-impact journals. It also should be noted that this study was of journals, not articles; it is open to question whether these two groups of journals are sufficiently comparable with one another in measures other than their access model to produce a meaningful gauge of the impact of open access. Given that we are at an early stage in the evolu- tion of open-access journals, comparing the research impact of articles taken from the same issues of the same journals is a more solid methodology for measuring the impact of open access. A common author perspective is re- flected in a reported conversation with an electrophysiologist in the journal Science: “Free online papers are likely to reach more readers, he figures, and therefore attract more citations.”10 Backing this up is a survey of authors (primarily in the life and medical sciences) sponsored by the Joint Information Systems Com- mi ee (JISC), which found that the two principal reasons for publishing in an open-access journal was belief in “the principle of free access for all readers,” followed by “I perceive OA journals to have faster publication times.” The next two most important reasons were related to perception of research impact: “I per- ceive the readership to be larger” and “I think my article will be more frequently cited.”11 One quantifiable measure of au- thors’ belief that free access is valuable to them is their adoption of an open-access option initiated by the Entomological Society of America and now offered by a number of publishers. A er offering open access to individual articles for a (low) fee, they found that by the second year more than half of the authors elected to purchase it.12 There also is more indirect evidence of a link between free online availability and impact. Studies have shown that authors, as consumers of research information, rely heavily on browsing online journals and articles.13 Brody and colleagues have used data from the CiteSeer repository to demonstrate that the “peak of citations occurs higher and sooner for papers deposited in each succeeding year.”14 Moreover, research has been done on the tangential question of whether online journals receive greater use than journals available only in print. That research, un- dertaken at a point when only a portion of journals were available in so-called “combination” online and print editions, did not make the distinction between ac- http:articles.13 374 College & Research Libraries cess to a licensed online version versus a freely available online version.15 University libraries contemplating providing support for open-access initia- tives, such as institutional repositories or open-access journals, face several key challenges. Librarians must be able to draw on a sophisticated understanding of the scholarly communication practices of individual disciplines even as they are rapidly evolving, including scholars’ use of prepublication research material not traditionally part of the domain of librar- ies in a print environment. If we choose to implement institutional repositories, we also must be able to persuade faculty, many of whom are for a variety of reasons quite reluctant, to contribute their prime research output. Data showing that freely available articles in their discipline are more likely to be cited is powerful evi- dence of the value of repositories as well as other open-access channels. The Research Question This study’s hypothesis is that scholarly articles from disciplines with varying rates of open-access adoption have a greater research impact if the articles are freely available online than if they are not. To determine whether a difference in impact exists, the mean citation rates, as recorded in the ISI Web of Science da- tabase, of freely available articles (1) were compared with those that are not (0) for a sample population of journal articles in four disciplines. The null hypothesis is that there is no difference between the mean citation rates: H0: d1 = d0; H1: H0 is false. These data provide a snapshot in time of open-access adoption in a few disparate disciplines. Within Christine L. Borgman’s framework describing the intersection of scholarly communication and biblio- metrics, this study examines the impact and potential of networked information September 2004 on artifacts of scholarly communication (published articles). Two of Bormgan’s four bibliometric research question types are addressed, namely, characterizing larger pa erns of behavior in scholarly communities and evaluating measures of influence of scholarly contributions.16 This study does not attempt to ad- dress the variable of publisher policies regarding posting of pre- or postprints of articles from their journals, or the ques- tion of why authors choose to post or not to post e-prints. Methodology The four disciplines—mathematics, electrical and electronic engineering, political science, and philosophy—were chosen with the expectation that they would represent different points on the continuum of open-access adoption. They also were selected as disciplines whose scholars have a tradition of active use of preprints. Ten leading journals in each discipline, as defined by ISI’s Journal Citation Reports (JCR) for 2002 were se- lected (except philosophy, where there is no JCR).17 High-impact journals were se- lected as indicators of leading researcher behavior while making no assumptions about journal quality. Open-access articles published in lower-impact journals may, in fact, have a greater relative research impact because they are not so widely available to authors through personal and institutional subscriptions. ISI Web of Science citation data were used as a measure of research impact. Although citations cannot in themselves be said to measure research impact, nev- ertheless, citedness as measured by ISI is a measure that is commonly relied on as a surrogate for such impact.18,19 Use of this inexact proxy for research impact is less of a concern in this study because citedness in this literature is itself viewed by scholars as an objective, at least in the http:contributions.16 http:version.15 Do Open-Access Articles Have a Greater Research Impact? 375 sciences and social sciences, and an ac- tual assessment of research impact is not of interest but, rather, the effect of open access on one traditional and frequently used measure of research impact. Articles from 2001 and 2002 were selected as the population from which to draw the sample. Philosophy was the exception, where 1999 and 2000 were selected because of the lower level of citation of philosophy articles. Sample A systematic presample, which included a minimum of fi y open-access and fi y restricted articles, was taken for each of the disciplines to estimate the expected frequency and variance of open access in each population in order to calculate the necessary sample sizes. (See table 1.) Data collection for political science and electri- cal and electronic engineering began at the midpoint of the population (the first 2002 issue) and expanded equally into the second half of 2001 and first half of 2002 until the sample target for open-access articles was reached. For mathematics and philosophy, where the frequency of the restricted and open-access articles, respectively, was low, data were taken for the entire population. A potential bias introduced, namely, that some journals have more articles per issue than others, is not a concern because it is unlikely that TABLE 1 Sample (Number of Articles)* and Frequency of Open Access Discipline ss (Total) ss (1) ss (0) % of Total Open Access Philosophy 602 101 501 17 Political science 299 87 212 29 Electrical and electronic engineering 506 188 318 37 Mathematics 610 426 184 69 * (1) = open; (0) = not open an author who publishes a given article in a higher-page-count title is either more or less likely to post that article online than an author who publishes in a lower-page- count title. Data Collection Article titles and the number of citations to each article, as recorded in the ISI Web of Science database, were collected from the sample population. Self-citations, citations from articles within the same journal issue, and citations from 2004 were excluded. A er collecting these data, the article title was searched as a phrase in Google. If any freely available full-text version (including dra s, preprints, and postprints) was available, the article was considered to be open access. Google is a particularly powerful search engine as it now indexes not only the full text of PDF files, but also some PostScript files. Zipped files and dvi files (output from the TeX typese ing program commonly used in mathematics), on the other hand, were only discoverable through Google from Web page links. Some full-text articles in mathematics, therefore, may have been missed; however, the majority of open-access articles in mathematics are contained in repositories, which are well indexed. (See table 2.) Searching on the full title, which in most cases was a phrase unique to an article, resulted in any full- text copy of that article appearing at the top of the result list ahead of references to it (because the title i s t y p i c a l l y e n- coded in the title tag). Parentheti- cal additions to the title made by some philosophy journals were re- 376 College & Research Libraries September 2004 TABLE 2 Where Open Access Articles Are Found (n = 50) Discipline Author’s Site Discipline Repository Other Repository Dept’l / Company Site Conf. / Assoc. / Project Site Working Paper Series Another Person’s Site Course Archive Philosophy 36 7 0 1 2 2 0 2 Political science 23 3 2 5 6 8 0 3 Electrical and electronic engineering 25 9 0 9 2 0 4 1 Mathematics 15 30 0 0 3 0 1 1 moved before searching, as were nontext or encoded characters. Three phenomena indicate that the total number of articles coded as open access may be underestimated. The first is that a number of articles were “ephem- eral,” meaning that there was evidence that they were available at one time but not when the sample was taken (e.g., the link was dead). Another set of ephemeral articles is reflected in the practice of au- thors who post a preprint but remove it when the article is published or replace it with a link to the restricted publisher’s copy. Whether they do that knowing that that copy is restricted is not always clear. (Of course, it also is possible that articles available at the time of data collection were posted only recently.) There also is the likelihood that some article titles changed significantly enough to not be discoverable between the preprint and the final publication. Finally, because only Google was searched, it is very likely that articles not indexed in Google would be discoverable through other search engines. Results and Discussion Because bibliographic distributions are highly skewed, the nonparametric equivalent of the t-test, the two-sided Wilcoxon signed rank test, was run for each discipline. (See table 3 for summary results.) The data show a significant dif- ference in the mean citation rates of open-access articles and those that are not freely available online in all four dis- ciplines. The relative increase in citations for open-access articles ranged from a low of 45 percent in philosophy to 51 percent in electrical and electronic engineering, 86 percent in political science, and 91 percent in mathematics. The disciplines selected did indeed represent a spectrum of adoption of open-access practices TABLE 3 Comparison of Mean Citation Rates Between Freely Available Articles and Those That Are Not Freely Available Discipline Mean (1) Mean Standard Error (1) Mean (0) Mean Standard Error (0) Difference in Means Percent Difference in Means Wilcoxon Two-tailed p Value SD (1) SD (0) Philosophy 1.60 0.491 1.10 0.230 .500 45% .0012 2.51 2.62 Political science 2.20 0.477 1.18 0.353 1.016 86% <.0001 2.27 1.73 Electrical and electronic engineering 2.35 0.449 1.56 0.275 .798 51% .0006 3.14 2.50 Mathematics 1.60 0.270 0.84 0.230 .762 91% <.0001 2.84 1.60 Do Open-Access Articles Have a Greater Research Impact? 377 FIGURE 1 Comparison of Citation Rates Across Disciplines for Open and Not Open Articles Mathematics—not open (n = 184) Mathematics—open (n = 424) Electrical and electronic engineering —not open (n = 317) Electrical and electronic engineering —open (n = 186) Political Science—not open (n = 212) Political Science—open (n = 87) Philosophy—not open (n = 500) Philosophy—open (n = 102) 0 2 4 6 8 10 12 14 Number of Citations (Source: ISI Web of Science) (outliers > 15 excluded) 16 among scholars. Seventeen percent of articles in philosophy were open access; 29 percent in political science; 37 percent in electrical and electronic engineering; and 69 percent in mathematics (table 1). It is interesting to note that the discipline with the highest rate of adoption of open access (mathematics) is not the discipline with the greatest impact of open access on citation rates (political science). On the other hand, the discipline with the lowest open-access rate (philosophy) also exhibited the most tenuous link between citation and open access (the 95% confi- dence interval includes both means). Although it has been shown that scien- tists prefer to access their research mate- rial online, and this study indicates that that may be the case for social scientists as well, it may well not yet be true for humanists.20 It is likely that a critical mass of open articles is needed before authors will become accustomed to regularly looking for needed articles online but, when they do, the move away from print is irreversible. Research supporting this interpretation comprises studies done in medical libraries (whose journals were the first to move online), which indicate that even journals only available in print—or back issues of online journals—started to see a dramatic decline in usage when a critical mass of journals went online.21 As more research is available online, readers lower the threshold of effort they are will- ing to expend to retrieve documents that present any barriers to access. This indi- cates both a “push” away from print and a “pull” toward open access, which may strengthen the association between open access and research impact. As Lawrence hypothesized, it is likely that the greater citation rate for open-source articles indicates that authors are finding them more easily, reading them more o en, and therefore citing them disproportionately in their own work. It is conceivable that the converse is true, that high impact articles are for some reason more likely to be posted online (maybe their authors get more requests for reprints and want to facilitate that process). However, that seems unlikely because the author behav- ior observed during the gathering of these http:online.21 http:humanists.20 378 College & Research Libraries data indicates that the typical practice of each individual is to post either all or none of his or her articles. Because comparing means of highly skewed distributions can be misleading, the citation distributions of each popula- tion were examined to see if there was a difference. The box plots in figure 1 show that across all disciplines the distribu- tion of number of citations indicates that articles in the open-access sample have higher citation counts. These distributions also suggest that the greatest impact of open access is with the most-cited articles, as found by Steve Hitchcock et al. and Elana Broch had found.22 This also is re- flected in the higher standard deviations for the open sample for three of the four disciplines studied (philosophy being the exception). Author and Reader Behaviors To be er understand the individual choice that posting a research article online rep- resents, subsamples of fi y open-access articles were sampled randomly from each discipline. For each article, its loca- tion was noted (table 2).23 Links to articles most o en were found on author home pages (50% overall) or in disciplinary repositories (25%), with clear differences across disciplines. Also noted was whether September 2004 the article was a preprint or a postprint, which also varies by discipline. (See figure 2.) Perhaps the fact that it is more typical to post a postprint in electrical and electronic engineering reflects a somewhat weaker preprint culture in that discipline. What is not reflected in these numbers is that some authors will post a preprint from a given journal and others will post a postprint. Thus, author choice is clearly a factor in the decision to post a pre- or postprint (as well as any restrictions a journal may place on posting pre- or postprints). One conclusion that could be drawn from the increased citation rates of ar- ticles whose authors made them available as preprints is that preprints are being used as “near substitutes” for the less accessible published versions for many readers (although those readers still cite the published version).24 In the increasing use and distribution of the preprint, there is a blurring of the boundary between “research documentation” and “scholarly literature,” to use terms from an IFLA statement on open access.25 A likely driver of the observed author behavior is that the more accessible online preprint, for disciplines that practice the exchange of preprints, is an innovation that facilitates long-established peer communication behavior. FIGURE 2 Preprint/Postprint Distribution in Subsample (n = 50) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Philosophy Political Science Engineering (electrical and electronic) Mathematics postprint preprint http:access.25 http:version).24 http:found.22 Do Open-Access Articles Have a Greater Research Impact? 379 One may speculate that when articles are only a mouse click away, “bad” author behaviors that have been described in the citation analysis literature will be less common. One example is citation bias, where authors reference only journals they can access.26 Another author behav- ior that may become less prevalent thanks to open access is hollow citing, where the author does not read the article he or she cites, evidenced by the carrying forward of prior citation errors.27 Publisher Behaviors Some interesting publisher practices also were uncovered during the course of this study. Publishers such as Blackwell, IEEE, ACM, and Kluwer are exposing article- level metadata, sometimes through for- mal agreements with Google, linking back to a restricted or pay-per-view copy.28 The result is that the Google searcher will see as many as five or six different links to the restricted copy of the article. Google searching currently works in favor of the seeker of an open-access copy, however. If an open copy of the article is available, and the article itself has been indexed by Google, that copy will appear ahead of the restricted copies in the search results dis- play. Where the open-access article itself is not indexed but is available through a link, the page with the link will follow the links to the restricted copies. Publisher use of the Digital Object Identifier (DOI) to expose article-level metadata deserves singling out as a metadata exposure technique. Many publishers note the DOI on the electronic copy of the article and promote its use as a unique identifier for an article, one that some authors use on their own Web pages.29 Publishers are currently leverag- ing the DOI through partnerships, such the one between CrossRef and Google in which CrossRef will provide Google with publisher metadata, including DOIs.30 But because publishers register DOIs with the International DOI Foundation (IDF), which also provides resolution services, DOI links will always resolve to the publisher ’s copy of the article.31 This practice points to the need for the DOI to be directly actionable on the Web, perhaps through the info URI, so that authors, libraries, or others could provide services that resolve DOI-based links to open-access (or locally licensed) copies of the articles.32 Conclusions This study indicates that, across a variety of disciplines, open-access articles have a greater research impact than articles that are not freely available. Although this finding is only a part of the complex picture of ongoing changes in scholarly communication in a networked informa- tion environment, it can help to inform librarians’ strategies in working on initia- tives such as building institutional reposi- tories, pursuing open-access publishing alternatives, or working with faculty on negotiating rights with publishers. Un- derstanding disciplinary differences in authors’ posting of e-prints has implica- tions for existing library services as well. Some approaches to changing the existing scholarly communication model, such as disciplinary or institutional repositories and open-access journals, are longer term and top-down. Author self-posting, as seen in the ad hoc manner in which it is occurring, is a present-day grassroots response to clearly perceived benefits in sharing scholarly output. Libraries as institutions must respond in both the long and short terms by participating in major new initiatives and at the same time be er encompassing the new open-access literature in their collections. For instance, the contents of many repositories can be accessed using the Open Access Initiative Metadata Harvesting Protocol. In many http:articles.32 http:article.31 http:pages.29 http:errors.27 http:access.26 380 College & Research Libraries September 2004 cases, simply passing a title through to Google would be a valuable addition to a reference linking service. It is well known that despite the fact that such journal-level impact factors are routinely used to evaluate authors of in- dividual articles, “journal impact factors correlate poorly with actual citations of individual articles.”33 The high standard deviations of these samples bear this out and point to the value of new citation measures, such as CiteSeer or ParaCite, which assess the impact of individual articles.3 4 Open-access articles make these new, more meaningful measures of research impact possible. Evidence of the rapid evolution of bibliometrics toward “webometrics, “cybermetrics,” and “influmetrics,” as Blaise Cronin has characterized them, is the partnership between ISI and CiteSeer to create a new citation measurement tool.35,36 Studies such as this one can help to shed light on the “dark ma er” of open access. Outside a few disciplines, the majority of freely available articles will not be found in a repository or in an open-access journal but, rather, on personal home pages. Those articles are not amenable to collecting hit- count measures of use, yet they are clearly used. Shedding light on this category of open access reveals that scholars in di- verse disciplines are adopting open-access practices at a surprisingly high rate and are being rewarded for it, as reflected in a traditional measure of research impact. Notes 1. Bethesda Statement on Open Access Publishing, Available online from h p://www.earlham. edu/~peters/fos/bethesda.htm. [Cited 13 March 2004]. Budapest Open Access Initiative. Available online from h p://www.soros.org/openaccess/. [Cited 13 March 2004]. 2. Steve Lawrence, “Online or Invisible,” preprint. Available online from h p://www.neci.nec. com/~lawrence/papers/online-nature01/ [Cited 18 March 2004]; ———, “Free Online Availability Substantially Increases a Paper’s Impact,” Nature 411 (May 2001): 521. 3. Eugene Garfield, post to American Scientist Open Access Forum listserv, 18 January 2004. Available online from h p://listserver.sigmaxi.org/sc/wa.exe?A2=ind04&L=american-scientist- open-access-forum&F=l&S=&P=9766. [Cited 13 march 2004]. 4. Gregory K. Youngen, “Citation Pa erns to Traditional and Electronic Preprints in the Published Literature,” College & Research Libraries 59 (Sept. 1998): 448–56. 5. Cecelia Brown, “The Role of Electronic Preprints in Chemical Communication: Analysis of Citation, Usage, and Acceptance in the Journal Literature,” Journal of the American Society for Information Science and Technology 54 (2003): 362–71. 6. Elana Broch, “Cite Me, Cite My References? (Scholarly Use of the ACM SIGIR Proceedings Based on Two Citation Indexes,” Proceedings of the 24th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, La., September 9–13, 2001 (New York: Association for Computing Machinery, 2001), 446–47. 7. Stevan Harnad and Tim Brody, “Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals,” D-Lib Magazine 10 (June 2004). Available online at h p://www. dlib.org/dlib/june04/harnad/06harnad.html. [Cited 21 July 2004]. 8. Ibid. Correlation Generator available online at h p://citebase.eprints.org/analysis/correla- tion.php. [Cited 13 March 2004]. 9. “The Impact of Open Access Journals: A Citation Study from Thomson ISI,” April 15, 2004. Available online from h p://www.isinet.com/media/presentrep/acropdf/impact-oa-journals. pdf. 10. David Malakoff, “Opening the Books on Open Access,” Science 302 (Oct. 24, 2003): 552. 11. “JISC and Publishers Work Together to Open Up Access to Journals,” press release. Avail- able online from h p://www.jisc.ac.uk/index.cfm?name=news_openaccess_0304. [Cited 18 March 2004]. 12. Thomas J. Walker, “Two Societies Show How to Profit by Providing Free Access,” Learned Publishing 15 (2002): 279–84. Publishers offering this option to authors and rates of adoption for 2003 include: Entomological Society of American for four journals (62%); Oxford University Press for Nucleic Acids Research annual issue, American Society of Limnology and Oceanography for Limnology and Oceanography (46%), American Physiological Society for Physiological Genomics, Company of Bi- http:articles.34 Do Open-Access Articles Have a Greater Research Impact? 381 ologists for three journals, Infotrieve for The Scientific World. Thomas Walker, post to American Sci- entist Open Access Forum listserv, March 10, 2004. Available online from h p://listserver.sigmaxi. org/sc/wa.exe?A2=ind04&L=american-scientist-open-access-forum&F=l&S=&P=34319; Peter Suber, post to American Scientist Open Access Forum listserv, March 10. Available from h p://listserver. sigmaxi.org/sc/wa.exe?A2=ind04&L=american-scientist-open-access-forum&F=l&S=&P=34446. [Cited 18 March 2004]. 13. Bo-Christer Björk, “Open Access to Scientific Publications: An Analysis of the Barriers to Change?” Information Research 9 (Jan. 2004). Available online from h p://informationr.net/ir/9- 2/paper170.html. [Cited 13 March 2004]; Carol Tenopir and Donald W. King, “Reading Behavior and Electronic Journals,” Learned Publishing 15 (2002): 259–65. 14. Steve Hitchcock, Donna Bergmark, Tim Brody, et al., “Open Citation Linking,” D-Lib Maga- zine 8 (Oct. 2002). Available online at h p://www.dlib.org/dlib/october02/hitchcock/10hitchcock. html. [Cited 13 March 2004]. 15. “Collection Management Strategies in a Digital Environment,” final report to the Andrew W. Mellon Foundation, January 2004. Available online from h p://www.ucop.edu/cmi/finalre- port/index.html. [Cited 13 March 2004]; Eun-Ja Shin, “Do Impact Factors Change with a Change of Medium? A Comparison of Impact Factors When Publication Is by Paper and through Parallel Publishing,” Journal of Information Science 29 (2003): 527–33; Andrew Odlyzko, “The Rapid Evolu- tion of Scholarly Communication,” Learned Publishing 15 (Jan. 2002): 7–19. 16. The others are evolution of scholarly communities and diffusion studies. Christine L. Borgman, “Bibliometrics and Scholarly Communication,” Communication Research 16 (Oct. 1989): 589–90. 17. Philosophy journals were selected by using a combination of methods. Ulrich’s International Periodicals Directory, Magazines in Libraries, online college or university library pathfinders, and individual scholars were consulted to develop a set of leading philosophy titles. Finally, to be included in the sample, titles had to be indexed by Philosopher’s Index and have five or more citations to the journal in ISI Web of Science in the year 2000. The journals included in the sample follow. For philosophy, American Philosophical Quarterly, The British Journal for the Philosophy of Science, British Journal of Aesthetics, Ethics, Journal of Philosophical Logic, Journal of Philosophy, Mind, Noûs, The Philosophical Quarterly, and Philosophical Review; for political science, American Journal of Political Science, American Political Science Review, Annual Review of Political Science, Brit- ish Journal of Political Science, Comparative Political Studies, Comparative Politics, Journal of Conflict Resolution, Journal of Peace Research, Political Geography, and Public Opinion Quarterly; for electrical and electronic engineering, IEEE Communications Magazine, IEEE Electron Device Le ers, IEEE Journal of Selected Topics in Quantum Electronics, IEEE Network, IEEE Signal Processing Magazine, IEEE Transactions on Image Processing, IEEE Transactions on Medical Imaging, IEEE Transactions on Pa ern Analysis and Machine Intelligence, Journal of Microelectromechanical Systems, and Proceedings of the IEEE; for mathematics, Acta Mathematica, Annales Scientifiques de L’Ecole Normale Superieure, Annals of Mathematics, Bulletin of the American Mathematical Society, Communications on Pure and Applied Mathematics, Foundations of Computational Mathematics, Inventiones Mathematicae, Journal of the American Mathematical Society, Random Structures & Algorithms, and Computational Geom- etry—Theory and Applications. 18. Locke J. Morrisey, “Bibliometric and Bibliographic Analysis in an Era of Electronic Scholarly Communication,” Science and Technology Libraries 22 (2002): 149–60. 19. Jonathan R. Cole, “A Short History of the Use of Citations as a Measure of the Impact of Scientific and Scholarly Work,” in The Web of Knowledge: A Festschri in Honor of Eugene Garfield, ed. Blaise Cronin and Helen Barsky Atkins (Medford, N.J.: Information Today, Inc., 2000), 281–300. 20. Bo-Christer Björk and Ziga Turk, “How Scientists Retrieve Publications: An Empirical Study of How the Internet Is Overtaking Paper Media,” Journal of Electronic Publishing 6 (Dec. 2000). Available online at h p://www.press.umich.edu/jep/06-02/bjork.html. [Cited 13 March 2004]. 21. Oliver Obst, “Pa erns and Costs of Printed and Online Journal Usage,” Health Information and Libraries Journal 20 (2003): 22–32; Thomas C. Rindfleisch, “W(h)ither Health Science Libraries, Preliminary Study of the Dynamics and Effects of Digital Materials Use on the Future Roles of Health Science Libraries,” Sept. 17, 2001. Available online at h p://smi-web.stanford.edu/people/ tcr/tcr-hsl-futures.html. [Cited 13 March 2004]. 22. Steve Hitchcock, Tim Brody, Christopher Gu eridge, Les Carr and Stevan Harnad, “The Impact of OAI-based Search on Access to Research Journal Papers,” Serials 16 (November 2003). Available online at h p://opcit.eprints.org/serials-short/serials11.html. [Cited 21 July 2004]. Broch, “Cite Me, Cite My References?” 23. The location was coded as a department site if it was linked from a departmental page, even when it was stored in an individual’s file space; it was coded as being on the author’s site if the link was from an individual’s curriculum vitae or located in a personal directory. If there was a repository copy, the article was coded repository even when a copy was also on the author’s site. 382 College & Research Libraries September 2004 24. Odlyzko reports, “Readers will se le for inferior forms of papers if those are the ones that can be reached easily.” Odlyzko, “The Rapid Evolution of Scholarly Communication,” preprint, 3. Available online from h p://www.dtc.umn.edu/~odlyzko/doc/complete.html. [Cited13 March 2004]. 25. International Federation of Library Associations and Institutions, “IFLA Statement on Open Access to Scholarly Literature and Research Documentation,” adopted 5 December 2003. Available online from h p://www.ifla.org/V/cdoc/open-access04.html. [Cited 13 March 2004]. 26. Pritpal S. Tamber, Fiona Godlee, and Peter Newmark, “Open Access to Peer-reviewed Research: Making It Happen,” Lancet 362 (Nov. 8, 2003): 1576. 27. “CrossRefTM Launches Pilot Program of CrossRef Search, Powered by Google,” press re- lease, April 28, 2004. Available online from h p://www.crossref.org/02company/pr/press20040428. html. [Cited 14 May 2004]; “Google Users Flock to IEEE Xplore,” What’s New @ IEEE for Stu- dents 6 (Mar. 2004). Available online from h p://www.ieee.org/portal/index.jsp?pageID=corp_ level1&path=products/whats-new/wnstudents&file=wnstudents0304.xml&xsl=generic.xsl. [Cited 14 May 2004]. 28. H. L. Hoerman and C. E. Nowicke, “Secondary and Tertiary Citing: A Study of Referencing Behaviour in the Literature of Citation Analyses Deriving from the Ortega Hypothesis of Cole and Cole,” Library Quarterly 65 (1995): 415; a recent empirical study was conducted by M. V. Simkin and V. P. Roychowdhury, “Stochastic Modeling of Citation Slips,” Jan. 2004. Available online at h p://arxiv.org/pdf/cond-mat/0401529. [Cited 13 March 2004]. 29. Some authors are using the DOI in their lists of publications. For example, Lars Engebret- sen’s personal Web page at h p://www.nada.kth.se/~enge/enge-bib. 30. “CrossRefTM Launches Pilot Program.” 31. Users are directed to the appropriate copy if they connect through an institution’s reference linking service that uses CrossRef data to resolve DOIs to the local resolution server. 32. Herbert Van de Sompel, Tony Hammond, Eamonn Neylon, and Stuart L. Weibel, “The ‘Info’ URI Scheme for Information Assets with Identifiers in Public Namespaces” (Internet Engineering Task Force Internet-Dra , Dec. 2003). Available online at h p://www.ietf.org/internet-dra s/dra - vandesompel-info-uri-01.txt. [Cited 13 March, 2004]. 33. Per O. Seglen, “Why the Impact Factor of Journals Should Not Be Used for Evaluating Research,” British Medical Journal 314 (Feb. 1997): 498–502. 34. In March 2004, CiteSeer relocated from NEC to Pennsylvania State University (and changed its name to CiteSeer.IST). Available online at h p://citeseer.ist.psu.edu. [Cited 18 March 2004]. ParaCite is available at h p://paracite.eprints.org/. [Cited 18 March 2004]. 35. Blaise Cronin, “Bibliometrics and Beyond: Some Thoughts on Web-based Citation Analy- sis,” Journal of Information Science 27 (2001): 1–7. 36. “Thompson ISI and NEC Team Up to Index Web-based Scholarship,” press release, Feb. 25, 2004. Available online from h p://www.isinet.com/press/2004/8217120. [Cited 13 March 2004]; Vincent Kiernan, “New Database to Track Citations of Online Scholarship,” Chronicle of Higher Education (Mar. 12, 2004). http:h�p://paracite.eprints.org http:h�p://citeseer.ist.psu.edu