Leiding.indd Using Citation Checking of Undergraduate Honors Thesis Bibliographies to Evaluate Library Collections Reba Leiding This study utilizes citation checking of advanced undergraduate research papers as a method for evaluating library collections at an institution with growing undergraduate and graduate research demands. A random sam- ple of 101 honors thesis bibliographies from the period 1993–2002 was examined for format, discipline, and local availability rates.The proportion of journal citations in relation to books increased slightly over the period, but no other clear trends emerged.The incidence of Web citations began during the period but did not steadily increase. The study highlighted specific use patterns and collection weaknesses. Results serve as a baseline for further study of the library’s undergraduate user population; further citation studies are encouraged to assess continued use of online resources as the Internet and electronic technologies evolve. maller college libraries and undergraduate libraries at larger institutions by defini- tion focus their collection development activities on materials that support the undergraduate curriculum. First-year writing-intensive class as- signments and term papers for classes in curriculum majors require collections that cover a broad range of research topics but typically are not as comprehensive as collections at research institutions. When undergraduate students at larger institutions undertake more advanced research, they direct their research needs to the main library’s stacks or its subject libraries, where their usage is assimilated into that of the overall user population. But when two-year institutions evolve into four-year institutions, or when smaller colleges and universities change their curriculum missions to emphasize undergraduate research or add advanced- degree programs, those institutions’ libraries must find ways, to say nothing of the means, of making the transition to support more advanced research. Because in larger institutions the evolving needs of advanced undergraduate research are not readily observable, few studies exist as a guide to libraries in transition on how to address this new mission. How can a library assess how adequately it is responding to such research demands? An answer may be found by analyzing bibliographic citations within existing Reba Leiding is the Assistant to the Dean at James Madison University Libraries; e-mail: leidinrm@jmu.edu. 417 mailto:leidinrm@jmu.edu 418 College & Research Libraries September 2005 examples of advanced undergraduate research. Literature Review The citation is a bibliographical entry in a footnote, reference list, or bibliography of a document that contains enough in- formation (e.g., author, title, publisher, or journal title) to verify the original item. As Linda C. Smith’s early review of citation analysis pointed out, the relationship be- tween the citing document and the cited references rests on a number of assump- tions, two of the most important being (1) actual use of the cited document by the citing author and (2) citation of “the best possible works… of all possible docu- ments that could be cited.”1 Given these assumptions of value and actual use, a citation can be viewed as a relevant arti- fact of use data. Further, citation checking of research documents and comparison of those citations with the availability of materials in a local collection offers an unobtrusive and cost-effective method of evaluating that collection’s ability to support research.2,3 As Smith’s and Robert N. Broadus’s literature reviews noted, citation check- ing is a time-honored and well-studied method for determining the usefulness of collections.4,5 It is part of a broad category known as citation studies, defined by Paul H. Mosher as “any specific methodologies that use source citations or references drawn from the scholarly apparatus of articles and books as the basis for manipu- lation, research, and study.”6 Applications for this research in this broad area have branched in several directions. Mosher noted the distinction between studies of bibliographical citations and the more “bibliometric” methodologies that ex- amine the linking of documents through jointly cited references.7 One branch of citation-checking studies flourished from the 1970s onward as libraries responded to budgetary crises and rising journal prices. Citations in research literature within a discipline were checked in order to find a rationale for canceling subscriptions or allocating funds. Harry M. Kriz examined the ratio of books to monographs cited in master ’s theses for engineering and found a justification for protecting book budgets in that field.8 Checking citations in master’s theses was one of the methods Christina E. Bolgiano and Mary Kathryn King used to evaluate journal collections.9 Thomas E. Nisonger checked citations in selected political science journals to evalu- ate the subject collection.10 Robin B. Devin and Martha Kellogg recommended using citation checking and analysis to develop a formula for a serial/monograph ratio.11 More recent studies have used citation checking of local faculty or graduate student research or of scholarly journals to evaluate subject collections in biology, psychology, polymer science, education, and interdisciplinary studies such as tour- ism.12–16 A study by Erin T. Smith used a sample of graduate theses and disserta- tion bibliographies from 1991 and 2001 to form “snapshots” of graduate research demands on the library collection. Smith’s study is similar to the present one in that it analyzes citations by material type, dis- cipline, and local availability; she used the snapshots to gauge the impact of budget constraints on collection usefulness for the library’s graduate user population.17 Although numerous studies have examined research journals or bibliog- raphies of more advanced researchers, a smaller body of studies has used cita- tion checking to gain information about undergraduate users. Rose Mary Magrill and Gloriana St. Clair looked at differ- ences in citation behavior by course level and in different disciplines.18 Margaret Sylvia and Marcella Lesher analyzed journal citations of undergraduate, as well as graduate, research papers to evaluate an academic library’s journal collection in psychology.19 A series of studies by Philip M. Davis and Davis and Suzanne A. Co- hen analyzed undergraduate term paper bibliographies to determine the effect of the Internet on citation behavior.20–22 One outcome of Davis and others was the for- mation of recommendations for research http:psychology.19 http:disciplines.18 http:population.17 http:ratio.11 http:collection.10 Using Citation Checking of Undergraduate Honors Thesis Bibliographies 419 paper assignments. Karen Hovde used citation checking to evaluate the impact of library instruction on student research skills.23 Even fewer studies have analyzed undergraduate papers’ bibliographies to evaluate local collection use. Kathleen E. Joswick compared local library holdings of materials cited in freshman composi- tion papers in order to form recommenda- tions for building collections that address undergraduate research needs.24 Joswick and Jeane Koekkoek Stierman analyzed journal citations from freshman compo- sition papers, as well those from faculty research papers and international citation pa erns, and found very li le overlap; their study recommended local use stud- ies of multiple-user types.25 Background James Madison University (JMU) is state- supported university focused primarily on undergraduate education. The institu- tion was founded in 1908 as a women’s normal school but by the 1960s had become a coeducational institution with a broader curriculum. Since the 1970s, when the university took its present name, the number of students has nearly tripled and the university currently has an enrollment of more than 15,500 students. Although undergraduate education re- mains the university’s primary mission, graduate enrollment is increasing along with the number of graduate programs, which now include a small number at the doctoral level. At the same time, the university is placing greater emphasis on undergraduate research and encouraging faculty to include undergraduates in their research projects. In addition, members of the teaching faculty have registered demands for collections to support faculty research through numerous user satisfac- tion surveys. This growing emphasis on advanced research presents a dilemma to a university library that has long focused its mission on support of an under- graduate curriculum. How can librarians evaluate how adequately current library collections support research or determine what specific needs exist? The present study examines a par- ticular niche of undergraduate research: honors thesis bibliographies. Although the number of students producing an honors thesis makes up a small percent- age of the undergraduate population at JMU (about two percent in 2002), it can be argued that they are some of the most intensive users of library resources. Hon- ors theses are the culmination of a serious research undertaking. For this reason, their bibliographies are more likely than those of shorter undergraduate composi- tion papers to reliably fulfill those two assumptions underlying citation analysis methodology: that the cited material was actually used by the document’s author, and that the sources used were the best possible for the purposes of his or her research. Because of the long-term nature of their projects, authors of honors theses also are less likely to limit their bibliogra- phies to resources available only within the university’s library, as is o en the case with freshman composition papers. Further, as Louise S. Zipp noted in her study of graduate thesis citations, student research can serve as an indicator of re- search interests of their faculty advisors.26 Thus, a study of theses bibliographies also offers an efficient way to assess faculty research needs. Typically, as a ma er of policy, many academic libraries collect and house copies of student theses. Even though students are a fluid population, these collections constitute a stable and accessible source of data about evolving research needs of both faculty and ad- vanced undergraduate students. At James Madison University, two copies of all undergraduate honors theses have been collected and housed in the library since the inception of the honors program in the 1960s. Because theses are cataloged in-house, their records show remarkable consistency in detail across years. All undergraduate theses have a unique call number, yet all share the same root call number, which facilitates http:advisors.26 http:types.25 http:needs.24 http:skills.23 420 College & Research Libraries September 2005 the ability to define the entire population of records within the online catalog and move the list into a database for sample selection. Methodology The present study, undertaken in 2003, examined a sample of ten years’ worth of thesis bibliographies from 1993 to 2002 to determine the adequacy of the collection for undergraduate research. The study also sought to detect any trends in the undergraduate research environment, such as the increased use of journals or the impact of the Internet on citation behavior. It was hypothesized that the growing availability of online indexes and TABLE 1 Number of Academic Departments Represented in Study, 1993–2002 Psychology 15 Biology 9 Political Science 9 Integrated Science & Technology 8 History 7 Anthropology and Sociology 6 Business, Economics 6 English 6 Health Sciences 6 Theatre and Dance 6 Philosophy and Religion 5 Art 2 Chemistry 2 Communications Sciences, Disorders 2 Math 2 Media Arts and Design 2 Social Work 2 Speech Communication 2 Computer Science 1 General Studies 1 Kinesiology 1 Music 1 Total 101 abstracts would lead to an increase in the percentage of journals cited as compared to monographs. Access to electronic jour- nals also was increasing throughout the study’s duration, but because citations during this early period of electronic use may not indicate whether a journal article was accessed in print or via online, no hypothesis about e-journal use was pro- posed. In addition, it was hypothesized that Web citations (that is, citations refer- ring to Internet pages) would make up an increasing percentage of citations in the la er half of the 1990s. Out of a total of 1,244 honors theses in the JMU library as of January 2002, 674 theses containing bibliographies were added to the collection between 1993 and 2002. Call numbers for this popu- lation were imported into Excel and sorted by year, and a stratified sample was chosen using Excel’s random num- ber generator. A total of 101 thesis bib- liographies comprised the sample, or approximately 15 percent of the theses added in each year. Bibliographies were checked to determine the total number of citations and what types of materi- als were cited, such as books, journals, newspapers, primary sources (in this study defined as interviews, original correspondence, surveys, etc.), Web citations, and ”other” miscellaneous sources, including government docu- ments. Data from the bibliographies were entered in an Access database. The total number of citations per bibli- ography was tracked, but for purposes of analysis duplicate citations within a bibliography (that is, subsequent cita- tions referring to a previously cited source) were counted only once. This eliminated skewing in availability rates that could occur when, for example, an anthology was cited repeatedly and reconciled differences in citation styles that used endnotes rather than a list of works cited. Citations were checked against the online catalog to determine local availability at the time. In the case of books, ownership was Using Citation Checking of Undergraduate Honors Thesis Bibliographies 421 determined by the cata- log date in the record; because weeding activ- ity was minimal during the study years, current catalog records were assumed to accurately reflect holdings for the period. For journals, the volume holdings in the catalog record were checked to determine availability of particular issues when the thesis was wri en. Availabil- ity of full-text articles through aggregator da- tabases was not checked for this study, with the exception of LexisNexis Academic, which the li- brary acquired in 1999.27 Those citations not veri- fied through the online catalog were searched in other sources such as WorldCat or Google to determine citation veracity. Data were ana- lyzed by material type by year and by disci- pline to determine if any significant trends in composition of bibliog- raphies emerged. Rates of local availability were calculated and analyzed to evaluate how well lo- cal collections serve this research population. Results The total number of citations in the random sample was 3,564. The number of unique cita- tions was 3,407 when duplicate citations were eliminated. The shortest bibliography contained two citations and the TA B L E 2 C om po si ti on o f B ib lio gr ap hi es , b y Y ea r Y ea r To ta l U ni qu e C it at io ns B oo ks C it ed % B oo ks Jo ur na ls C it ed % Jo ur na ls N ew sp ap er s C it ed % N ew sp ap er s P ri m ar y So ur ce s C it ed % P ri m ar y So ur ce s O th er So ur ce s C it ed % O th er So ur ce s W eb Si te s % W eb Si te s 19 93 28 7 10 6 36 .9 % 93 32 .4 % 32 11 .1 % 3 1. 0% 53 18 .5 % — — 19 94 25 8 79 30 .6 % 14 5 56 .2 % — — 26 10 .1 % 8 3. 1% — — 19 95 33 9 16 4 48 .4 % 10 7 31 .6 % 23 6. 8% 3 0. 9% 42 12 .4 % — — 19 96 29 6 16 2 54 .7 % 99 33 .4 % — — 5 1. 7% 30 10 .1 % — — 19 97 26 4 11 8 44 .7 % 11 1 42 .0 % 9 3. 4% 1 0. 4% 20 7. 6% 5 1. 9% 19 98 36 9 14 3 38 .8 % 11 4 30 .9 % 16 4. 3% 20 5. 4% 48 13 .0 % 28 7. 6% 19 99 45 6 12 5 27 .4 % 19 3 42 .3 % 15 3. 3% 1 0. 2% 49 10 .7 % 73 16 .0 % 20 00 47 6 18 9 39 .7 % 17 6 37 .0 % 16 3. 4% 19 4. 0% 27 5. 7% 49 10 .3 % 20 01 40 1 10 3 25 .7 % 20 8 51 .9 % 16 4. 0% — — 28 7. 0% 46 11 .5 % 20 02 26 1 49 18 .8 % 16 4 62 .8 % 2 0. 8% 10 3. 8% 16 6. 1% 20 7. 7% To ta ls / A vg . % 3, 40 7 1, 23 8 36 .3 % 1, 41 0 41 .4 % 12 9 3. 8% 88 2. 6% 32 1 9. 4% 22 1 6. 5% 422 College & Research Libraries September 2005 TABLE 3 Materials in “Other” Category Material Type No. of Times Cited Percent Owned by JMU Government documents, including those from federal, state, municipal, foreign countries, etc. 88 50.0% Law texts, court cases, bills, etc. 66 96.9% Reports, including annual, financial, planning, and technical reports; discussion, policy, and working papers 56 1.7% Conference papers and proceedings 24 0% E-Resources, library 16 100.0% Brochures, pamphlets 11 0% Theses 13 76.9% Videos 8 50.0% Dissertation, Ph.D. 6 0% Miscellaneous 33 21.2% Total 321 longest contained 135, with the median being 31. Twenty-two academic depart- ments were represented in the sample. (See table 1.) Psychology, the largest department in the university, had the greatest number with a total of 15; the list as a whole showed a mix of the physical sciences, social sciences, and humanities. Table 2 shows the composition of bib- liographies by material format by year. Looking at the average percentage for all years combined, it appears that the overall reliance on books and journals is fairly evenly divided (36.3% for books as com- pared to 41.4% for journals). Newspapers made up a minor part of all citations, at 3.8 percent overall and showed consistent use near that level even in later years of the study when local access to newspaper sources increased with the availability of Lexis-Nexis. Primary sources likewise were a minor source overall at 2.6 percent. The study separated out primary sources as a material type to highlight any use of the library’s special collections of local historical sources; however, no theses using local primary sources were part of the sample. The proportion of “other ” sources ranged between 3 and 18 percent in the years examined and overall comprised 9.2 percent of all citations. In table 3, which also shows local availability of miscellaneous sources, government documents made up the largest compo- nent of the “other” category, accounting for 27.2 percent of all miscellaneous sources, with law texts making up an additional 20.4 percent. The library had access to 50 percent of government documents cited and 96.9 percent of law texts. The remainder of this category was dominated by grey literature, such as unpublished technical and financial reports, conference papers, or brochures, or with nonprint material. Web citations accounted for 6.2 per- cent of citations overall, but no such citations appeared at all until 1997. In 1999, Web citations amounted to 16 per- cent of total citations (when the Internet as a new phenomenon was in fact the subject of a few theses). Interestingly, the percentage dropped to around 10 percent in the next two years and continued to decrease in the last year of the study to 7.7 percent. Using Citation Checking of Undergraduate Honors Thesis Bibliographies 423 Effect of Web Citations Because the inception of the Internet marked a watershed in the types of sources available for research, it is informative to look at the composition of bibliographies before and a er Web citations began to appear in the sample bibliographies. (See table 4.) In the years of the study before Web citations, books accounted for 45.3 percent of total citations and journal cita- tions amounted to 36.1 percent. In the years 1997–2002, Web citations accounted for nearly 10 percent of all citations, books dropped to 34.2 percent, and journals rose to 42.1 percent. If Web citations are TA B L E 4 C om po si ti on o f B ib lio gr ap hi es : P re - a nd P os t I nt er ne t C it at io ns in B ib B oo ks Jo ur na ls N ew sp ap er s P ri m ar y So ur ce s “O th er ” R es ou rc es W eb Si te s To ta l 1 99 3– 19 96 1, 26 6 57 3 45 7 57 38 14 1 — Pe rc en t 1 99 3– 19 96 10 0. 0% 45 .3 % 36 .1 % 4. 5% 3. 0% 11 .1 % — To ta l 1 99 7– 20 02 2, 29 8 78 7 96 8 82 52 18 8 22 1 Pe rc en t 1 99 7– 20 02 10 0. 0% 34 .2 % 42 .1 % 3. 6% 2. 3% 8. 2% 9. 6% To ta l w ith ou t W eb Si te s 19 97 –2 00 2 2, 07 7 78 7 96 8 82 52 18 8 — Pe rc en t w ith ou t W eb S ite s 10 0. 0% 37 .9 % 46 .6 % 3. 9% 2. 5% 9. 1% — removed from the equation, book citations increase (38%) but do not reach the level found before 1997, whereas the percentage of journal citations increases to 47 per- cent. Although the data are not sufficient to argue that Web citations negatively impact the use of books, the generally ris- ing percentage of journal citations when looking at pre- and post-Internet periods demonstrates that thesis authors are ap- parently relying more heavily on journals for their research. Local Availability Table 5 shows local availability of cited references by material type by year. The percentage of books held by the library ranged from 53 to 85 percent throughout the years studied, with the mean at over 65 percent. Whether this percentage can be termed adequate is a ma er of opinion, but it is worth noting that the library’s book budget was static for most of the period covered by the study. Availability of journals was even lower overall at 58.2 percent; however, this low percentage is a bit misleading, and an accurate analysis needs to factor in the high number of individual titles cited and the range of frequency by title. The 1,410 total journal citations referenced 729 unique journal titles. The most frequently cited journal was referenced 22 times; a total of 16 journal titles were cited ten or more times. (See table 6.) Of these frequently cited titles, the library had access to over 83 percent. (One frequent citation was a local newspaper cited heavily in a single bibli- ography. Without this title, the availability rate was over 86%.) Over two-thirds of the journal titles in the list (494) were cited only once; of these, the library pro- vided access to 41.5 percent on average. Newspapers, though not frequently cited (as to be expected in scholarly research), displayed the highest rates of local avail- ability. This can be explained by the fact that even prior to access to LexisNexis, the library had extensive back files to the most frequently cited newspapers, the New York Times and the Washington Post. 424 College & Research Libraries September 2005 Primary sources showed the lowest rates of availability at 3.4 percent over- all. Most primary sources cited in bib- l i o g r a p h i e s , s u c h as personally con- ducted interviews, surveys, and so on stemmed from the student author’s own research, rather than use of library ma- terials. The overall percentage of mate- rials in the “other ” category that were available locally was at 45.4%, however, the availability rate for government doc- uments was higher at 50% and law texts was 97%, likely due primarily to Lexis- Nexis. Results by Discipline The data also can be examined from the standpoint of discipline for col- lection development p u r p o s e s . T h e s e s we r e a s s i g n e d t o one of four broad disciplines based on the student’s aca- demic department, as found in the thesis catalog record: busi- n e s s , h u m a n i t i e s , sciences, and social sciences. (See table 7.) Composition of bibliographies by discipline was gen- erally as expected: theses in the humani- ties relied most heav- TA B L E 5 A va ila bi lit y of C it ed M at er ia ls , b y Y ea r Y ea r B oo ks C it ed N um be r H el d (% ) Jo ur na ls C it ed N um be r H el d (% ) N ew sp ap er s C it ed N um be r H el d P ri m ar y So ur ce s C it ed N um be r H el d O th er S ou rc es C it ed N um be r H el d 19 93 10 6 65 (6 1. 3) 93 69 (7 4. 2) 32 3 2 (1 00 .0 ) 3 0 (0 .0 ) 53 47 (8 8. 7) 19 94 79 50 (6 3. 3) 14 5 63 (4 3. 4) — — 26 0 (0 .0 ) 8 1 (1 2. 5) 19 95 16 4 96 (5 8. 5) 10 7 79 (7 3. 8) 23 20 (8 7. 0) 3 0 (0 .0 ) 42 10 (2 3. 8) 19 96 16 2 10 8 (6 6. 7) 99 52 (5 2. 5) — — 5 1 (2 0. 0) 30 10 (3 3. 3) 19 97 11 8 84 (7 1. 2) 11 1 77 (6 9. 4) 9 9 (1 00 .0 ) 1 1 (1 00 .0 ) 20 16 (8 0. 0) 19 98 14 3 78 (5 4. 5) 11 4 74 (6 4. 9) 16 9 (5 6. 3) 20 0 (0 .0 ) 48 6 (1 2. 5) 19 99 12 5 68 (5 4. 4) 19 3 12 2 (6 3. 2) 15 14 (9 3. 3) 1 1 (1 00 .0 ) 49 30 (6 1. 2) 20 00 18 9 16 2 (8 5. 7) 17 6 96 (5 4. 5) 16 15 (9 3. 8) 19 0 (0 .0 ) 27 8 (2 9. 6) 20 01 10 3 73 (7 0. 9) 20 8 12 5 (6 0. 1) 16 13 (8 1. 3) — — 28 14 (5 0. 0) 20 02 49 26 (5 3. 1) 16 4 64 (3 9. 0) 2 2 (1 00 .0 ) 10 0 (0 .0 ) 16 4 (2 5. 0) To ta ls /A ve ra ge Pe rc en t 1, 23 8 81 0 (6 5. 4) 1, 41 0 82 1 (5 8. 2) 12 9 11 4 (8 8. 9) 88 3. 4 (2 4. 4) 32 1 14 6 (4 5. 5) y re fe r t o ei th er a p hy si ca l i te m is in th e lib ra ry c ol le ct io n or e le ct ro ni c ac ce ss o f f ul l- te xt e -j ou rn al s ub sc ri pt io ns . C ita tio ns a ls o w er e ch ec ke d ag ai ns t m ic fo r a va ila bi lit y (1 99 9– ); c on te nt fr om o th er a gg re ga to rs w as n ot c on si de re d fo r t hi s st ud y. “H el d” m a L ex is N ex is A ca de N ot e: Using Citation Checking of Undergraduate Honors Thesis Bibliographies 425 TABLE 6 Most Frequently Cited Journals Journal Title Citations Held Locally % Held Locally Journal of Personality and Social Psychology 22 22 100.0% Science 20 20 100.0% Business Week 18 18 100.0% Psychological Reports 16 16 100.0% Casselman Chronicles 15 0 0% International Journal of Eating Disorders 15 15 100.0% Nature 15 15 100.0% Physical Therapy 14 14 100.0% Economist 13 13 100.0% Journal of Sex Research 12 11 91.7% Euromoney 11 11 100.0% JAMA 11 11 100.0% Gerontologist 10 10 100.0% Journal of Quality Technology 10 0 0% New England Journal of Medicine 10 10 100.0% Radiology 10 0 0% Total/Avg. 222 186 83.8% ily on books (69.3%), whereas theses in the sciences and social sciences utilized books to a lesser extent (20.2% and 29.5%, respectively). Theses in the area of busi- ness were relatively dependent on books as well, at 48.4 percent. Predictably, the highest percentage of journal citations was in the sciences (58.8%) and second highest in the social sciences (46.2%), whereas journal citation use in the hu- manities was low at 12.6 percent. Use of Web resources was the highest for busi- ness at 13.2 percent. Some particular pa erns emerge when looking at local availability of materials in terms of discipline. (See table 8.) The percentage of books held locally ranged from 68 to 72.2 percent for all disciplines except the sciences, where the local avail- ability was lower at 40.8 percent. Likewise for journals, the local availability rate was lowest for the sciences at 44.4 percent, whereas for the other disciplines the range was from 64.9 to 70.5 percent. These figures highlight the historical weak- nesses in both monograph and journal collections for the pure sciences, as would be expected at a smaller institution with a liberal arts orientation. The high cost of science materials, particularly journals, is no doubt a factor, as well. For “other” resources, the social sciences had the highest rates of local availability at 65.7 percent, reflecting the use of the govern- ment documents and local law collections by such departments as history and politi- cal science. Observations What implications can be seen from an analysis of thesis citation data? First of all, books remain tremendously important for researchers in the humanities and, to some extent, for business. Although the former in particular may seem an obvious point, we should recall that the library ex- 426 College & Research Libraries September 2005 perienced steady state budgets for much the percentage of books cited relative to of the study period and during some years journals by year in the overall composi- the size of book budgets eroded in the face tion of bibliographies, but grouping the of journal inflation. Thus, evidence for the data into pre- and post-Internet periods importance of book collections serves as does provide some evidence that use of justification for increased funding. No journals in undergraduate research is in- overall trend emerges when looking at creasing. Access to journals is particularly important for undergraduate researchers TA B L E 7 C om po si ti on o f B ib lio gr ap hi es b y D is ci pl in e To ta l U ni qu e C ita tio ns B oo ks Jo ur na ls N ew sp ap er s P ri m ar y So ur ce s O th er So ur ce s W eb S ou rc es D is ci pl in e To ta l % To ta l % To ta l % To ta l % To ta l % To ta l % B us in es s 34 9 16 9 48 .4 % 97 27 .8 % 8 2. 3% 5 1. 4% 24 6. 9% 46 13 .2 % H um an iti es 61 9 42 9 69 .3 % 78 12 .6 % 57 9. 2% 6 1. 0% 27 4. 4% 22 3. 6% Sc ie nc es 86 2 17 4 20 .2 % 50 7 58 .8 % 4 0. 5% 27 3. 1% 10 4 12 .1 % 46 5. 3% So ci al Sc ie nc es 1, 57 7 46 6 29 .5 % 72 8 46 .2 % 60 3. 8% 50 3. 2% 16 6 10 .5 % 10 7 6. 8% To ta l 3, 40 7 1, 23 8 1, 41 0 12 9 88 32 1 22 1 in the sciences and social sciences, as would be expected. Use of Web citations does not appear to be a growing trend for advanced under- graduate research. This would appear to be a somewhat surprising result given the impact of the Internet, but it parallels the findings of Davis and Cohen, and Davis’s later findings.28–30 This tempering of Web citation use may reflect the involvement of faculty advisors in encouraging students to use more traditional and authoritative resources. The present study’s findings can serve as a baseline in future studies to determine whether Web citations will increase as the Internet matures and what impact Internet use will have on the com- position of future bibliographies. One clear finding is the importance of government documents materials, par- ticularly for social science research. As noted in table 3, the overall availability rate for government documents cited at this library, which is a government document depository, was 50 percent. The growing number of government docu- ments available online and the federal government’s proposed scaling back of the document depository program will reduce the significance of libraries’ physi- cal depository collections, however. In the future, access to government documents will likely depend more on the skills of researchers—and reference librarians—in searching and navigating government Web sites. A significant proportion of materials was unavailable locally, primarily in sources categorized as miscellaneous and primary, but perhaps most notably in the large number of journal titles cited only a few times across the years of the study. This fact points to the importance of inter- Using Citation Checking of Undergraduate Honors Thesis Bibliographies 427 TA B L E 8 A va ila bi lit y of C it ed M at er ia ls b y D is ci pl in e D is ci pl in e B oo ks Jo ur na ls N ew sp ap er s P ri m ar y So ur ce s O th er S ou rc es To ta l C it at io ns N um be r H el d % H el d To ta l C it at io ns N um be r H el d % H el d To ta l C it at io ns N um be r H el d % H el d To ta l C it at io ns N um be r H el d % H el d To ta l C it at io ns N um be r H el d % H el d B us in es s 16 9 12 2 72 .2 % 97 63 64 .9 % 8 6 75 .0 % 5 0 0. 0% 24 2 8. 3% H um an iti es 42 9 30 0 69 .9 % 78 55 70 .5 % 57 52 91 .2 % 6 1 16 .7 % 27 14 51 .9 % Sc ie nc es 17 4 71 40 .8 % 50 7 22 5 44 .4 % 4 3 75 .0 % 27 0 0. 0% 10 4 21 20 .2 % So ci al Sc ie nc es 46 6 31 7 68 .0 % 72 8 47 8 65 .7 % 60 53 88 .3 % 50 2 4. 0% 16 6 10 9 65 .7 % To ta l 1, 23 8 81 0 1, 41 0 82 1 12 9 11 4 88 3 32 1 14 6 library loan (ILL) services for libraries in transition from a curricular to a research collection. ILL proves more advantageous for the undergraduate researcher working on a long-term project than for students working on shorter writing assignments, which tend to be researched at the last minute. Although ILL requests can pro- vide important information on emerging needs for journal subscriptions, this study shows that in most cases it would not be cost efficient to add the occasionally cited title to the library collection. Conclusion This study found an increasing propor- tion of journal citations in undergraduate thesis bibliographies over time, especially for science and social science disciplines. The data did not support the hypothesis that use of Web citations in thesis bibli- ographies was increasing. The absence of this trend may be the result of faculty advisor intervention; faculty and stu- dents may not have viewed the Web as a scholarly resource in its early years. Also, the study did not make clear that use of Web citations had any singular impact on the composition of thesis bibliographies over time. Although the proportion of journal citations did increase relative to monographic citations in the la er por- tion of the period, the data do not point to specific causes for this shi . It may be due to a gradually rising number of journal subscriptions during the study period or may be the result of the eroding strength of book collections in a sustained period of steady state funding, in which case the student researchers are exhibiting at least a tendency to rely on accessibility rather than the best sources for their purposes. Or the shi may simply be a result of the particular characteristics of the sample. At the same time, relatively low local availability rates occurred for some dis- ciplines and material types throughout the study period. Low availability rates highlight weaknesses in the collection (a locally useful outcome of the study) but also provide a positive validation that the 428 College & Research Libraries September 2005 citations under study, in fact, fulfill those basic assumptions of actual use and qual- ity of the documents cited. That is, rather than confining their research to what was locally available, the study provides evi- dence that the authors made an effort to obtain and use the best possible resources for their projects. That no other strong trends or pat- terns emerged from the data over time demonstrates the complexity of support- ing academic research needs. Even when liaison librarians or bibliographers know the research interests and pa erns of in- dividual faculty, students, or even depart- ments, these interests and emphases can shi in response to numerous interacting factors. Traditionally, large institutions have responded to the complexity of academic research demands by using the knowledge of subject specialists and main- taining large “just-in-case” collections. A smaller academic library with limited resources must be nimble in assessing and responding to changing needs. Locally, results of this study provide a baseline to track future trends in the use of Internet citations and online journals. Repeating the study across disciplines in future years, or focusing on theses from particular disciplines or departments, will provide further assessment of how well this institution’s library resources are meeting research needs in a transition- ing academic environment. Libraries at other higher education institutions whose academic mission is shi ing—whether they are two-year colleges that are mov- ing toward four-year institutions or col- leges and smaller universities that are emphasizing undergraduate research, adding academic disciplines, or graduate programs—could employ methodologies such as those used in this study to evaluate how well their collections are responding to changing research demands. Librarians at these institutions are probably already aware of existing undergraduate research populations that serve as bellwethers of emerging research needs. Several factors make this a propitious time to undertake further citation analy- sis studies of undergraduate research behavior. The time frame covered in this study occurs relatively early in the life of the World Wide Web and e-journal access. Style manuals by now have in- corporated formats for citing online ac- cess, and librarians, faculty, and students have had time to assimilate them. This makes the task of tracking use of online materials in bibliographies more reliable. The Web continues to mature, offering an increasing number of authoritative resources. Citation studies also may provide information on how effectively student researchers use online indexes and emerging link resolver technologies. Findings from such studies will not only provide a baseline for future trends stud- ies as technologies continue to evolve but also will contribute to what has to date been a relatively small body of research on undergraduate library users. Notes 1. Linda C. Smith, “Citation Analysis,” Library Trends 30 (summer 1981): 83–106. 2. Margaret J. Sylvia, “Citation Analysis as an Unobtrusive Method for Journal Collection Evaluation Using Psychology Student Research Bibliographies,” Collection Building 17, no. 1 (1998): 20–28. 3. Thomas E. Nisonger, “A Test of Two Citation Checking Techniques for Evaluating Political Science Collections in University Libraries,” Library Resources & Technical Services 27 (Apr./June 1983): 163–76. 4. Smith, “Citation Analysis.” 5. Robert N. Broadus, “The Applications of Citation Analysis to Library Collection Building,” in Advances in Librarianship, ed. Melvin J. Voight and Michael H. Harris, pp. 299–335 (New York: Academic Pr., 1977). 6. Paul H. Mosher, “Quality and Library Collections: New Directions in Research and Practice in Collection Evaluation,” in Advances in Librarianship, ed. Wesley Simonton, pp. 211–38 (New Using Citation Checking of Undergraduate Honors Thesis Bibliographies 429 York: Academic Pr., 1984). 7. Ibid., 222. 8. Harry M. Kriz, “Citation Counting and the Future of Engineering Libraries,” Engineering Education 67 (Apr. 1977): 707–10. 9. Christina E. Bolgiano and Mary Kathryn King, “Profiling a Periodicals Collection,” College & Research Libraries 39 (Mar. 1978): 99–104. 10. Nisonger, “A Test of Two Citation Checking Techniques.” 11. Robin B. Devin and Martha Kellogg, “The Serial Monograph Ratio in Research Libraries: Budgeting in Light of Citation Studies,” College & Research Libraries 51 (Jan. 1990): 46–54. 12. Rosalind Walco , “Local Citation Studies: A Shortcut to Local Knowledge,” Science & Technology Libraries 14, no. 3 (1994): 1–14. 13. Margaret Sylvia and Marcella Lesher, “What Journals Do Psychology Graduate Students Need? A Citation Analysis of Thesis References,” College & Research Libraries 56 (July 1995): 313–18. 14. Sherri Edwards, “Citation Analysis as a Collection Development Tool: A Bibliometric Study of Polymer Science Theses and Dissertations,” Serials Review 25, no. 1 (1999): 11–20. 15. Laurel A. Haycock, “Citation Analysis of Education Dissertations for Collection Develop- ment,” Library Resources & Technical Services 48 (Apr. 2004): 102–6. 16. Juris Dilevko and Keren Dali, “Improving Collection Development and Reference Services for Interdisciplinary Services through Analysis of Citation Pa erns: An Example Using Tourism Studies,” College & Research Libraries 65 (May 2004): 216–41. 17. Erin T. Smith, “Assessing Collection Usefulness: An Investigation of Library Ownership of the Resources Graduate Students Use,” College & Research Libraries 64 (Sept. 2003): 344–55. For a comment on Smith’s methodology and response, see “Le ers to the Editor,” College & Research Libraries 65 (Jan. 2004): 6–7. 18. Rose Mary Magrill and Gloriana St. Clair, “Undergraduate Term Paper Citation Pa erns by Disciplines and Level of Course,” Collection Management 12, no. 3/4 (1990): 25–56. 19. Sylvia, “Citation Analysis as an Unobtrusive Method.” 20. Philip M. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update,” College & Research Libraries 63 (Jan. 2002): 53–60. 21. ———, “Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholar- ship in a Networked Age,” Portal: Libraries and the Academy 3 (Jan. 2003): 41–51. Available online at h p://muse.jhu.edu/journals/portal_libraries_and_the _academy/voo3/3.1davis.pdf. 22. ——— and Suzanne A. Cohen, “The Effect of the Web on Undergraduate Citation Behavior 1996–1999,” Journal of the American Society for Information Science and Technology 52, no. 4 (2001): 309–14. 23. Karen Hovde, “Check the Citation: Library Instruction and Student Paper Bibliographies,” Research Strategies 17 (2000): 3–9. 24. Kathleen E. Joswick, “Library Materials Use by College Freshmen: A Citation Analysis of Composition Papers,” College and Undergraduate Libraries 1, no. 1 (1994): 43–65. 25. ——— and Jeane Koekkoek Stierman, “The Core List Mirage: A Comparison of the Jour- nals Frequently Consulted by Faculty and Students,” College & Research Libraries 58 (Jan. 1997): 48–55. 26. Louise S. Zipp, “Thesis and Dissertation Citations as Indicators of Faculty Research Use of University Library Journal Collections,” Library Resources and Technical Services 40 (Oct. 1996): 335–42. 27. The library had access to two databases containing some full-text articles (Gale Expanded Academic and General Business File) beginning in 1996, but their full-text content was too un- stable to be verified reliably as a source of availability. Citations were checked against Lexis-Nexis Academic as a source for newspapers and law journal articles cited in theses a er 1999. Rarely, students’ citations indicated that the cited reference was accessed through a database, in which case the citation was counted as locally held, but in most cases citations did not include any information about electronic access. 28. Davis and Cohen, “The Effect of the Web on Undergraduate Citation Behavior 1996– 1999.” 29. Davis, “The Effect of the Web on Undergraduate Citation Behavior: A 2000 Update.” 30. ———, “Effect of the Web on Undergraduate Citation Behavior: Guiding Student Scholar- ship in a Networked Age.” CHOICE - C&RLFull Size Ad CHOICE.C&RL(Full) 12/23/04 10:59 AM Page 1 1 2 3 4 4 DIFFERENT WAYS TO PURCHASE MATERIALS FOR YOUR COLLEGE LIBRARY Read CHOICE Magazine for hundreds of brief, critical, expert reviews of new books each month. Every issue features essays, special features and lists of forthcoming titles. Use CHOICE Reviews on Cards. You’ll get a complete set of reviews and nothing but the reviews! Each pack is easily divided for quick reference and portability. Visit ChoiceReviews.online FREE for 60 days! Simply click on www.ChoiceReviews.org to gain 24/7 access to over 100,000 reviews for up to 20 authorized users. Begin your Free Trial today! Try our ChoiceReviews.online Site License Edition FREE for 60 days! Now available, our newest CHOICE option eliminates passwords for quick and easy campus-wide Internet access to our entire database of reviews. The Go ld Standa rd Among Academ ic Library Review Journa ls. 60 DAYS ONLINE FOR FREE! To try ChoiceReviews.online Password or ChoiceReviews.online Site License Edition FREE for 60 days, click on www.ChoiceReviews.org. ­ To subscribe to CHOICE Magazine or Reviews On Cards, contact us at www.ala.org/acrl/choice, or call (240) 646-7027. ­ CHOICE Meeting the Book Reviewing Needs of College Libraries and Their Faculty for Over 35 Years! ­ CHOICE is a publication of the Association of College & Research Libraries, a Division of the American Library Association. www.ala.org/acrl/choice http:www.ChoiceReviews.org