herring.p65 334 College & Research Libraries July 2002 334 Use of Electronic Resources in Scholarly Electronic Journals: A Citation Analysis Susan Davis Herring Susan Davis Herring is the Engineering Reference Librarian and Associate Professor of Bibliography in the M. Louis Salmon Library at the University of Alabama in Huntsville; e-mail: herrings@email.uah.edu. Although information gathering and use patterns in the traditional print envi- ronment have been studied for many years, the electronic environment pre- sents a new and relatively unexplored area for such study. This article de- scribes a citation analysis of research articles from scholarly electronic jour- nals published in 1999–2000. The analysis focused on the extent to which scholars are using electronic resources and the types and subject areas of online resources that are being referenced. Results indicate a growing reli- ance on electronic resources by scholars, a high occurrence of nontraditional types of resources, and a relatively high use of interdisciplinary references. e are facing a radical change in the way people find and use in- formation resources, a change that all information profession- als need to understand. Much research has been done on the information-gathering behavior of various groups (for example, David Ellis1; Gloria J. Leckie, Karen E. Pettigrew, and Christian Sylvain2; and Wil- liam Sugar3), so we have a reasonably good idea of how and why scholars use tradi- tional print-on-paper and personal re- sources. Now, that work needs to be ex- panded to understand how the electronic revolution is changing information selec- tion and use patterns. How has the prolif- eration of electronic resources changed re- searchers’ information use? What kinds of online resources are researchers using? Does the diversity of the online environ- ment and its search tools encourage re- searchers to go beyond the traditional re- sources of their disciplines? The exploratory study described here, a citation analysis of references from a sample of research articles published in electronic journals during the 1999–2000 publication year, is an attempt to answer these questions. It provides a new look at a changing information environment and its impact on scholarly research activity. Review of the Literature Some early effects of the increasing avail- ability of electronic resources are already apparent. As Jeffrey MacKie-Mason, Maria S. Bohn, Juan F. Riveros, and Wendy P. Lougee wrote, “Electronic access to schol- arly journals has become an important and commonly accepted tool for researchers.”4 According to Lawrence Rudner, digital li- braries, when used efficiently, can be a boon to both researchers and practitioners, providing rapid access to both research findings and practical information.5 Major studies investigating the impact of e-journals on scholarly communication have been reported by Stephen P. Harter and by Harter and Hak Joon Kim.6–8 Using ISI’s Journal Citation Reports to analyze the Use of Electronic Resources in Scholarly Electronic Journals 335 citation patterns and impact factors of a sample of e-journals published between 1993 and 1995, these researchers attempted to determine the extent to which scholars were aware of e-journals and built their own work on research published in e-jour- nals. Harter’s analysis of thirty-nine e-jour- nals found very little impact. Of the top five journals studied, three appeared in both print and electronic form, making it impossible to determine the true impact of the e-journals. Overall, Harter concluded that “almost none of the scholarly, peer- reviewed electronic journals in the sample have had a significant impact on formal communication in their respective fields.”9 Harter and Kim analyzed a sample of citations from e-journals to determine the extent to which researchers publishing in e-journals relied on other electronic re- sources for their own research. Less than 2 percent of the total references in their study were to online sources, and only 0.2 percent were to e-journals. They con- cluded that “e-journals presently play al- most no role in scholarly communication, as measured by references cited,” but also noted that this might change over time.10 Two years later, in a study of the impact of Internet-based resources on scholarly communication in library and information science, Yin Zhang (1998) not only showed a higher impact rate, but also an increase in use over time. Almost 7.5 percent of the articles in Zhang’s sample included refer- ences to online resources. In addition, ar- ticles in e-journals were more likely to cite electronic resources than were articles in print journals. Zhang noted that a differ- ent sample might show different results.11 One factor to consider in analyzing ref- erences to online sources is the type of re- source being cited. Examining how indi- vidual scholars are invoked on the Web, Blaise Cronin, Herbert W. Snyder, Howard Rosenbaun, Anna Martinson, and Ewa Callahan pointed out that “The defining feature of the Web, hypertexuality, affords the possibility of multiple modes of men- tioning or linking to a named individual and/or related cognitive resources.”12 Their research identified eleven categories of Web-based documents linked to schol- arly research, including electronic articles, conference papers, home pages, syllabi, and book reviews. Harter and Kim listed fourteen different types of sources found in their citation analysis. These included Web pages, personal papers, e-journal ar- ticles, e-mail, listservs, and local files, plus a large category of “not determined.”13 The variation in types of resources found in electronic research may reflect the more porous disciplinary boundaries of Internet- based searching. As Cronin and Hert have pointed out, the Web supports browsing and discovery by scholars, well beyond the typical, traditional bibliographic tools from well-defined bodies of literature.14 Because most commonly used Web search engines are not subject specific, sites and pages found using these search engines will not be limited by the disciplinary confines and expectations of the searcher. Methodology The research described here focused on the following three questions: 1. To what extent are scholarly re- searchers using electronic resources in their research? 2. What types of electronic resources are being used? 3. Do the electronic resources that are being used reflect the interdisciplinary potential of the Web? As mentioned above, some initial re- search has examined the impact of elec- tronic resources on scholarly communi- cation by analyzing citations in electronic journals. The current study follows both Harter and Kim, and Zhang in assuming that researchers who publish in e-journals are familiar with at least some online re- sources and have accepted such resources in their own research. The data for this study were drawn from a selective sample of scholarly, peer- reviewed e-journals available through the Articles in e-journals were more likely to cite electronic resources than were articles in print journals. 336 College & Research Libraries July 2002 TABLE 1 Distribution of Electronic Resources Journal Subject Total Articles With Electronic Citations Percent Computer-Mediated Comm. 21 19 90.5% Ecology 16 8 50.0% Education Technology 22 9 40.9% Gender Studies 25 5 20.0% Law & Technology 23 19 76.0% Library/Info Science 43 35 82.6% Psychology 9 1 11.0% Religion & Society 16 1 6.3% Total 175 97 55.4% Web without subscription or registration. A total of twelve journals were selected representing areas of active interdiscipli- nary research, namely: • library and information science (LIS/IS) • gender studies • ecology • psychology • religion and society • computer-mediated communica- tion (CMC) • educational technology • law and technology All articles published in these twelve journals during a one-year period cover- ing approximately summer 1999 through spring 2000 were examined (the time pe- riod varied because of different publica- tion schedules), and all peer-reviewed research articles that included reference citations were selected. A total of 175 ar- ticles were identified. The references listed for each article were examined, and duplicate references in each individual list were removed. Data concerning each article, including total number of unique citations, number of electronic citations, and subject disci- pline affiliations of the authors, were en- tered into a spreadsheet. Attempts were made to view each of the electronic re- sources. Information about each elec- tronic citation, including type of resource and general discipline category, was en- tered into the spreadsheet for analysis. Research Results The 175 articles examined had a total of 4,289 unique references. Slightly over 55 percent of the articles in the sample (97 out of 175) cited electronic resources; 685 citations, or 16 percent of the total, were to electronic resources. All of the journal subject areas included articles with references to elec- tronic resources, although the distribution varied by discipline as is shown in table 1. The ninety-seven articles that referenced electronic resources had a total of 2,584 unique citations, 26.5 percent of which were to electronic resources. A total of 42.5 percent of the references were to articles in online periodicals, and almost 25 percent were to the same journal in which the ar- ticle being analyzed appeared. (For the pur- pose of this study, articles were defined as publications in scholarly, peer-reviewed e- journals and in online newspapers and magazines. Scholarly articles comprised 73 percent of the total number of electronic articles cited.) The remainder of the refer- ences represented a broad array of types of resources, as shown in table 2. Several categories within this list are worth examining in more detail. “Reports,” research studies produced for specific pur- poses and organizations, represented just over 20 percent of all electronic resources cited. “Home pages,” over 15 percent of the total, included personal Web sites, home pages of organizations or institutions, sites promoting or providing information about products and services, and project Web sites. Use of Electronic Resources in Scholarly Electronic Journals 337 TABLE 2 Types of Electronic Resources Cited Type of Resource Number (n = 685) Percent Article from same journal 169 24.7% Reports 138 20.1% Article from different journal 122 17.8% Home pages 105 15.3% Conference papers 32 4.7% Papers 32 4.7% Standard/specification 19 2.8% Other 17 2.5% Books 15 2.2% Press releases 10 1.5% Seminar papers 8 1.2% Electronic databases 7 1.0% Bibliographies 4 0.6% Speech/address 3 0.4% Unknown 2 0.3% Note: Total comes to less than 100 percent due to rounding. “Papers” consisted of textual documents not categorized elsewhere and not formally published. The “Other” category included class syllabi, slide presentations, tables with- out accompanying text, and similar items. The two items categorized as “Unknown” could neither be accessed nor identified from the citations. For comparison, the print resources cited were analyzed by type as well. Books comprised 45.2 percent of the print cita- tions and articles made up another 43.4 percent, accounting for nearly 90 percent of the total references. Of the remaining citations, 6 percent were conference pa- pers, 3 percent were reports, and 1.1 per- cent were unpublished documents, prima- rily theses and dissertations. Personal com- munications, media (film, television, and recordings), laws and court cases, stan- dards, speeches, and press releases each made up less than 1 percent of the refer- ences. A large number of the Web- based resources (nearly 18% of the total) could not be accessed. The majority of those (seventy-eight) re- turned HTML 404 messages (“Page not found”), indicating either a dead link or a bad URL. In several cases, a connection was successful after obvious typographical errors in URLs had been corrected. When a connection attempt returned a “Page not available” message, fur- ther attempts were made during a period of one week. If the connec- tion still failed after that time, the resource was coded “not available/ not accessible.” URLs that returned “Forbidden” messages or asked for a user ID and password also were coded as “not available/not acces- sible.” The authors in the sample rep- resented a wide variety of subject disciplines. (When an article had authors from more than one subject discipline, only the first two disciplinary areas were con- sidered.) However, as might be expected, authors’ works tended to cluster within publications focused in their disciplinary affiliation. Table 3 shows the affiliations of the majority of authors in each journal cat- egory. Three of the eight categories—Li- brary and Information Science, Gender Studies, and Computer-Mediated Commu- nication (CMC)—evidenced strong interdisciplinarity, with authors from more than one subject area sharing the majority. The electronic citations in the articles studied also tended to cluster within disci- plinary areas representative of the journal subjects. However, some categories showed much more interdisciplinary use of elec- tronic resources than did others. Gender Studies and Ecology were tightly focused; Library and Information Science, although exhibiting a greater spread of topic re- sources, still exhibited a tighter focus than did CMC, Educational Technology, or Law and Technology. The leading subject areas for each of these journal topics are shown in table 4. The categories of Religion and Society and Psychology of Language, each The development of the Internet has vastly enhanced the ability of research- ers to find and use sources that previ- ously would have been unavailable. 338 College & Research Libraries July 2002 TA BL E 3 Pr im ary Au tho r A ffil iat ion s, b y J ou rna l T op ic LIS /IS Ge nde r Ed uc CM C Re l & So c Ec olo gy Psy ch La w & Sci enc e Tec h (n = 5 2) (n = 2 5) (n = 2 2) (n = 2 5) (n = 1 6) (n = 1 6) (n = 9 )( n = 28 ) Lib rar y & In fo. Sc i. 26 Co mp ute r S cie nce 20 Lit era tur e 5 Ge nde r S tud ies 4 Ed uca tio n 13 Co mm uni cat ion 6 Ma nag em ent 5 Re lig ion /Ph ilo sop hy 12 Ec olo gy/ En v. S ci. 11 Psy cho log y 5 Law 19 with only one article citing electronic re- sources, are excluded in tables 4 and 5. Assuming that authors tend to select citations from the subject area with which they are most familiar (i.e., their own dis- ciplines) and that the subjects with the highest number of citations represent the primary discipline areas for each journal, it is possible to define citations from other subject areas as representing interdiscipli- nary resources. Table 5 shows the percent- ages of same-discipline and cross-disci- pline electronic resource citations for the journals studied. Overall, 27 percent of the total electronic citations can be catego- rized as interdisciplinary. Findings As mentioned previously, earlier research into the effect of electronic resources on scholarly research has shown relatively low impact. In their analysis of articles pub- lished in e-journals in 1995, Harter and Kim found that 1.9 percent of the total references were to electronic resources.15 Zhang, studying LIS journals published from 1994 through 1996, found that 7.49 percent of the articles included electronic references, but only 1.13 percent of the total references were to electronic resources.16 Herring, ex- amining articles focusing on Internet search engine design in 1996, found that 2.1 percent of the total references were to electronic resources.17 The current study, focusing exclusively on e- journals, shows that more than half of the articles studied included elec- tronic references and that 16 percent of the total references were to elec- tronic resources. It is generally assumed that most scholars typically select journal ar- ticles, monographs, and conference papers as their references, with the occasional personal communica- tion, unpublished paper, or manu- script reference added. Indeed, this was the pattern found in the printed resources cited in the articles stud- ied in this citation analysis. How- ever, the pattern of documents found among the electronic re- sources in this study was quite dif- ferent and reflected a much greater range of types of publications. The development of the Internet has vastly enhanced the ability of researchers to find and use sources that previously would have been unavailable. Almost any competent Internet searcher can now access working papers, unpublished re- ports and studies, government pa- pers, and other “gray literature” that previously would have been unknown and inaccessible. Data from Herring’s 1999 study of ar- ticles on the design of Web search engines indicates that only 6 percent Use of Electronic Resources in Scholarly Electronic Journals 339 TABLE 4 Leading Subjects of Electronic Resources Cited, by Journal Topic LIS/IS Gender Education Computer-Mediated Ecology Law & Studies Technology Communication Technology Library & Information Activist Sites Education Computer-Mediated Ecology Law & Science Communication Technology Digital Libraries Gender Studies Psychology Internet Technology Education Computer Business & Science Management Law of the total citations represented gray lit- erature.18 In the current study, 345 of the electronic resources were to such unpub- lished or ephemeral resources. Although this accounts for only 8 percent of the total number of unique citations, it represents over 50 percent of the electronic resources cited and more than 13 percent of the total citations in articles referencing electronic resources. The body of research literature available has expanded exponentially for scholars using the Internet. Over one-quarter (27%) of the electronic resources cited in the current study were categorized as interdisciplinary or outside the primary discipline areas of the journals or the authors’ affiliations. However, the direct effect of online resources on interdisciplinarity is unclear. Although it is tempting to assume that the same Internet search capabilities that lead researchers to a wide variety of types of documents also lead to more interdisciplinary resources, this can- not be supported without further research. All one can conclude is that the use of elec- tronic resources appears to encourage some interdisciplinary resource use. Both the use of electronic resources and the extent of interdisciplinarity in the re- sources used varied by discipline and au- thor. Obviously, the acceptance and use of Web resources depend upon the user. As Zhang has noted, this is strongly influenced by the recognition of online sources as le- gitimate by the particular scholarly commu- nity in which the user operates.19 Use of electronic resources outside the researcher’s specific discipline may be even more strongly influenced by such acceptance. Further research is needed on this topic. Finally, it is worth noting once again that almost 18 percent of the electronic resources cited in the articles analyzed either were not found or could not be accessed. This is an unfortunate, but common, situation in the online environment because of the instabil- ity of many electronic documents, inad- equate or inaccurate citations, and less-than- robust connections and equipment. Al- though inaccurate citations have always been found in reference lists, the transitory nature of many Internet documents only ag- gravates this situation. Conclusion The research described in this article con- firms that online resources are increas- ingly important to today’s scholars and researchers. This was an exploratory study using a nonrandom sample of ar- ticles and is inherently not generalizable. By examining a sample of articles pub- lished in e-journals, we may be looking at researchers who are at the leading edge of use and acceptance of electronic sources. However, the findings reflect the Almost 18 percent of the electronic resources cited in the articles analyzed either were not found or could not be accessed. 340 College & Research Libraries July 2002 fact that a radical change in information- seeking behavior and information re- source use is taking place as scholars and researchers become more comfortable and familiar with the resources available through the Web. An awareness of this change can only help us in preparing for the future. Online resources offer an ex- citing alternative to, and an expansion of, traditional research sources and tools. By understanding how scholars use these re- sources, librarians and other information professionals will be able to provide en- hanced service to their users. TABLE 5 Percentage of Disciplinary and Interdisciplinary Electronic Citations, by Journal Topic LIS/IS Gender Educ. CMC Ecology Law& Overall Science Tech. Tech. Same Discipline 91% 82% 46% 55% 85% 71% 73% Cross-Discipline 9% 18% 54% 45% 15% 29% 27% Notes 1. David Ellis, “Modeling the Information-seeking Patterns of Academic Researchers: A Grounded Theory Approach,” Library Quarterly 63 (Oct. 1993): 469–86. 2. Gloria J. Leckie, Karen E. Pettigrew, and Christian Sylvain, “Modeling the Information Seeking of Professionals: A General Model Derived from Research on Engineers, Health Care Professionals, and Lawyers,” Library Quarterly 66 (Apr. 1996): 161–93. 3. William Sugar, “User-centered Perspective of Information Retrieval Research and Analy- sis Methods,” Annual Review of Information Science and Technology 30 (1995): 77–109. 4. Jeffrey K. MacKie-Mason, Maria S. Bohn, Juan F. Riveros, and Wendy P. Lougee, “A Re- port on the PEAK Experiment,” D-Lib Magazine 5, no. 7/8 (July/Aug. 1999). Available online from http://www.dlib.org/dlib/july99/mackie-mason/07mackie-mason.html 5. Lawrence Rudner, “Who Is Going to Mine Digital Library Resources? And How?” D-Lib Maga- zine 6, no. 5 (May 2000). Available online from http://www.dlib.org/dlib/may00/rudner/05rudner.html. 6. Stephen P. Harter, “The Impact of Electronic Journals on Scholarly Communication: A Citation Analysis,” Public-Access Computer Systems Review 7, no. 5 (1996). Available online from http://info.lib.uh.edu/pr/v7/n5/hart7n5.html. 7. ———, “Scholarly Communication and Electronic Journals: An Impact Study,” Journal of the American Society for Information Science 49, no. 6 (May 1998): 507–16. 8. Stephen P. Harter and Hak Joon Kim, “Electronic Journals and Scholarly Communication: A Citation and Reference Study.” Paper presented at the Midyear Meeting of the American Soci- ety for Information Sciences, May 20–22, 1996. Available online from http:// ezinfo.ucs.indiana.edu/~harter/harter-asis96midyear.html. 9. Harter, “Scholarly Communication and Electronic Journals,” 515. 10. Harter and Kim, “Electronic Journals and Scholarly Communication,” np. 11. Yin Zhang, “The Impact of Internet-based Electronic Resources on Formal Scholarly Com- munication in the Area of Library and Information Science: A Citation Analysis,” Journal of Infor- mation Science 24, no. 4 (July/Aug. 1998): 241–54. 12. Blaise Cronin, Herbert W. Snyder, Howard Rosenbaun, Anna Martinson, and Ewa Callahan, “In- voked on the Web,” Journal of the American Society for Information Science 49, no.14 (Dec. 1998): 1319–28. 13. Harter and Kim, “Electronic Journals and Scholarly Communication,” np. 14. Blaise Cronin and C. A. Hert, “Scholarly Foraging and Network Discovery Tools,” Journal of Documentation 51, no. 4 (Dec. 1995): 388–403. 15. Harter and Kim, “Electronic Journals and Scholarly Communication,” np. 16. Zhang, “The Impact of Internet-based Electronic Resources,” 245–50. 17. Susan Davis Herring, “The Value of Interdisciplinarity: A Study Based on the Design of Internet Search Engines,” Journal of the American Society for Information Science 50, no. 4 (Apr. 1999): 358–65. 18. Herring, “The Value of Interdisciplinarity,” 361. 19. Zhang, “The Impact of Internet-based Electronic Resources,” 252–53.