Fuchs.indd


  

                 
             

           
                  
                

                 
            
               

 
 

 

 

 

 

   
 

    
      

  

    

      
        

      
   

    

     
 

 
      
     

    

    

    

    

Behavioral Citation Analysis: Toward 
Collection Enhancement for Users 

Beth E. Fuchs, Cristina M. Thomsen, Randolph G. 
Bias, and Donald G. Davis Jr. 

A pilot study was developed to determine use of the University of Texas 
at Austin General Libraries’ research collections in the fields of civil en-
gineering and educational psychology and to investigate the research 
behavior of graduate students. First, the authors sampled bibliographic 
citations from dissertations completed during the years 1997 and 2002 
in the above-named fields. Then, a survey was sent to the dissertation 
writers to gain insight into use and opinions of library services for their 
graduate research. Analysis of information provided by both collection- 
and user-centered data-gathering techniques serves to underscore the 
value of the merged evaluation methods. 

ncreasing journal and mono-
graph costs juxtaposed with 
flattening appropriations 
have placed academic library 

collection development staffs in the 
midst of a dilemma: How can libraries 
provide enhanced services from dimin-
ishing resources and, at the same time, 
protect the quality of research materials 
provided to users? Librarians address 
this question in several ways, one being 
to gather information on usage and utility 
of the library’s collection as a way to focus 
selection and perhaps digitization of ad-
ditional materials to be er meet the needs 
of constituents. Collection evaluation 
techniques, whether they provide data 
on the collection or on the user, generate 

information that can be integrated into a 
larger evaluation effort to help reshape 
the institution’s vision.1-4 

The researchers set out to gather and 
then integrate qualitative and quantita-
tive data on collection use by merging 
the results of two evaluation methods 
characterized as a “behavioral citation 
analysis,” an enhanced citation analysis 
combining a traditional citation analysis 
with parallel behavioral (survey) data. 
Within the larger framework of informa-
tion needs assessment, citation analysis 
offers a well-documented method for 
gathering data on the utility of a collec-
tion. Early use of bibliographic citations 
as a means of evaluating library holdings 
against other libraries’ collections extends 

Beth E. Fuchs is a Reference and Electronic Resources Librarian in Reeves Library at Moravian College & 
Moravian Theological Seminary; e-mail: fuchsb@moravian.edu. Cristina M. Thomsen is the Library Director in 
Chan Shun Centennial Library at Southwestern Adventist University; e-mail: thomsen@swau.edu. Randolph 
G. Bias is an Associate Professor in the School of Information and the Director of the Information Experience 
Lab at the University of Texas at Austin; e-mail: rbias@ischool.utexas.edu. Donald G. Davis Jr. is Professor 
Emeritus of Library History in the School of Information and Department of History at the University of 
Texas at Austin; e-mail: dgdavis@ischool.utexas.edu. The research was conducted May–August 2004 when 
Fuchs and Thomsen were students in the School of Information at the University of Texas, Austin. 

304 

mailto:dgdavis@ischool.utexas.edu
mailto:rbias@ischool.utexas.edu
mailto:thomsen@swau.edu
mailto:fuchsb@moravian.edu


       
 

 
    

    
   
   

     
    

    
     

     
    

 
       

    

     

    

     

    

  

 

     

     
     

    

    

 
     

     
 

   
     

      
 

      
    

     

     
      

   

    
    

    

 
    

 
 

     
    

      

      

    
      

    
     

     

Behavioral Citation Analysis 305 

back to the work of Charles Coffin Jewe , 
in 1848.5 More recently, citation analysis 
has been used to identify significant au-
thors in a field, core lists of journals and 
monographs for a subject, and publish-
ers of influence.6-8 For academic research 
libraries, dissertations offer a particularly 
fertile resource for bibliographic citation 
analysis, primarily because they represent 
in-depth investigations into particular 
research questions. Dissertation bibli-
ographies are of considerable value to 
subject specialists in libraries charged 
with collection maintenance and indicate 
research trends of which academic librar-
ies should be aware. 

This project began as a pilot study to 
assist the University of Texas at Austin 
Libraries’ Collection and Information Re-
sources Department (CIRD) in assessing 
its research collections’ utility. The time 
period chosen for this study, 1997 to 2002, 
coincides with two significant changes in 
service provided by UT Libraries: the ar-
rival of electronically accessible full-text 
resources, and budget decreases that 
necessitated a reduction in paper-based 
serials and monograph acquisitions. The 
question of utility and perceived util-
ity of the research collection is accorded 
particular significance when libraries are 
inundated with changing expectations by 
users, triggered by technological options 
that did not exist a decade ago for access-
ing resources. 

The researchers recognized, however, 
that a traditional citation analysis re-
quires much inference on the part of the 
researchers and those in charge of library 
collections. By deciding to supplement 
the citation analysis with survey data on 
information-seeking behavior collected 
from the dissertation authors, the study 
aimed to probe deeper into the authors’ 
experiences, needs, and wishes. 

Review of Literature 
According to Anne L. Buchanan and 
Jean-Peirre V. M. Herubel, bibliographic 
citation analysis of dissertations offers an 
unparalleled means of learning the ex-

tent and completeness of an institution’s 
library research collection. These bibliog-
raphies contain a wealth of information 
resources vital for creating discipline-
specific subject guides that ensure com-
prehensive coverage.9 In 1978, Elizabeth 
Pan concluded that citation studies are 
reliable indicators of resource use in 
libraries.10 More recently, Margaret Stieg 
Dalton and Laurie Charnigo describe the 
study of “information-seeking behavior” 
as populated by thousands of reports, 
primarily created in the 1990s and for-
ward.11 The importance of examining the 
information needs of graduate students 
as a distinct user population is frequently 
addressed in the professional literature.12-
13 Couching citation analysis not only 
in traditional terms of resource use but 
also in information needs and behavior 
seemed a fruitful path to pursue. 

William L. Emerson published one of 
the earliest uses of student bibliographies 
to assess collection value in 1957. He 
examined the bibliographies of twenty-
three engineering dissertations wri en 
by students at Columbia University and 
then checked them against Columbia’s 
collection to determine its adequacy 
in supporting academic research. The 
study’s basic assumption, one currently 
taken for granted in citation studies, was 
that items listed in the dissertations’ bib-
liographies revealed the materials used 
by students.14 

Citation analysis also has been used 
to recommend materials for selection and 
deselection. Joy Thomas observed psy-
chology graduate students at California 
State University, Long Beach, from 1981 
to 1990, and created a profile of library 
resource use generated from the results 
of a thesis citation project. This work, 
undertaken as part of a collection evalu-
ation effort to identify journals that could 
be canceled with limited repercussions, 
demonstrated that the students depended 
heavily on the local collection for their 
citations (91.9%), with the remainder 
coming through interlibrary loan or some 
other means of access.15 Graduate stu-

http:access.15
http:students.14
http:literature.12
http:libraries.10


 

   
     
      

     

      
      

   
     

     

      
    

 

     

   

   
     

     

     
   

    

      

    

   
 

      
    

    

 

      

     
 

   

      

     

      
   

 
    

    

   
 

    
     

      

     

 

      

     
     

    
    

     
 

306 College & Research Libraries July 2006 

dent citation analysis thus provided the 
library staff not only with materials used 
but also with substantive information to 
bolster recommendations for continuance 
or discontinuance of specific titles, on 
the basis of a comprehensive survey of 
student research. 

Citation analyses of theses and dis-
sertations are conducted most frequently 
within a single institution as a means 
of assessing the collection utility of the 
library system at that particular college 
or university. Lois Kuyper-Rushing 
described a different form of citation 
analysis, in which the graduate papers 
of multiple universities are analyzed and 
then shaped into core lists for specific 
disciplines, thus transcending the focused 
interests of individual institutions.16 The 
advantage of this type of citation study 
is that individual institutions can then 
compare their resources and usage with 
national data for a more thorough evalu-
ation of their program. 

Although citation analysis provides 
one means of evaluating the utility of a 
library’s research collection, additional 
techniques are increasingly seen as a 
necessary part of a comprehensive evalu-
ation. In a recent publication, Laurel A. 
Haycock offered several caveats in using 
citation analysis to determine a library 
research collection’s adequacy. Among 
them, limited student research skills 
might reflect an incomplete understand-
ing of the extent of available library 
resources, and graduate students might 
choose readily available local materials 
over more important material not locally 
held, threatening self-perpetuating me-
diocrity.17 Margaret Sylvia and Marcella 
Lesher described the results of a thesis 
reference analysis extracted from psy-
chology and counseling theses during 
a six-year period. By adding a shelving 
count that tracked psychology journals’ 
in-library use as well as a cost-effective-
ness measure that compared use in theses 
with subscription costs, the authors cre-
ated a data-driven proposal for journal 
selection.18 

In studying the information-seeking 
behavior of humanities scholars in 1975, 
Lois Bebout, Donald G. Davis Jr., and 
Donald Oehlerts recognized that citation 
analysis had been carried out for a few 
academic disciplines and proposed that 
large-scale surveys also should be con-
ducted, consisting of questionnaires and 
interviews to be er focus library collec-
tion development in the humanities. The 
authors noted that a large-scale English 
program of evaluating resource use in the 
social sciences, INFROSS (Information 
Requirements of Social Scientists), mod-
eled such a survey.19 This effort consisted 
of surveying, interviewing, and observing 
researchers regarding their preferred re-
search venues, types of information used 
(e.g., historical, statistical), physical forms 
and other channels of information (e.g., 
periodicals, monographs, theses), and 
methods used to locate information.20 

Recent interest has turned toward 
studying the impact of full-text electronic 
resources on the needs and research be-
haviors of library users. A study con-
ducted to identify the use and impact of 
electronically accessible journals demon-
strated that the availability of online ma-
terials had changed the research pa erns 
of Yale University scholars between the 
years 1991 and 2001. Mean and median 
age of bibliographic citations dropped in 
the more recent articles, prompting the 
researchers to speculate that scholars are 
becoming less inclined to use (and there-
fore cite) classic monographs or research 
sources.21 

The UT Libraries were well aware of 
the trend toward use of electronic re-
sources and were interested in studying 
how resource use has changed since the 
broad availability of the Internet and the 
online full-text library.22 In se ing out 
to conduct this analysis, the research-
ers agreed with Haycock that a citation 
analysis alone might miss important 
information about motivations for use 
of certain citations. Rather than employ 
a large-scale, broadcast survey as sug-
gested by Bebout et al.,23 the researchers 

http:library.22
http:sources.21
http:information.20
http:survey.19
http:selection.18
http:diocrity.17
http:institutions.16


    

     
    

     
    

    

      
  

 
      

     

      
    

      

   
   

      

      

      

      
     

       
     

      
     

     
     

      
    

      
    

    
     
      
        
         

     
       

      
      

       

 
     

    
     

    

     
    

     

      

     

    
     
    

     

    
   

    
     

 

Behavioral Citation Analysis 307 

employed a focused survey of the author 
cohorts whose dissertations provided the 
citations for analysis. 

Method 
The present study of the research collec-
tion at the University of Texas at Austin 
Libraries consisted of two evaluative com-
ponents: one was collection-centered and 
the other, user-centered. The first part, 
a classic bibliographic citation analysis, 
consisted of sampling dissertations from 
four academic cohorts: civil engineering 
(CE) and educational psychology (EP), 
each for 1997 and 2002. These disciplines 
offered a contrast in resource use by 
graduate students in the applied sciences 
and in the behavioral sciences, according 
to information-seeking behavior studies 
of the groups.24 The programs also pro-
duced sufficient graduates in each year to 
provide a meaningful study population. 
The year 2002 was the most current year 
for which UT dissertations were available 
to the researchers, and the earlier cohort, 
1997, reflects a time prior to extensive 
access to electronic full-text resources, 
spanning a period of substantial change 
in the UT Austin Libraries’ budget and 
acquisitions. 

Dissertation Bibliographic Citation Analysis 
Twenty-six dissertations were selected 
from the total population for each cohort, 
which ranged in number from 26 to 34. 
The smallest number of dissertations per 
chosen discipline per year was 26, for 
educational psychology for 1997. There-
fore, for the purposes of this study, the 
researchers used all these and randomly 
selected 26 of the up-to-34 dissertations 
available from the other cohorts, for a 
total of 104 dissertations from a popula-
tion total of 126. The populations included 
spring, summer, and fall graduates for 
each year, as provided by the university’s 
Office of Records. 

To keep the size of the study manage-
able, the researchers investigated the 
first 30 citations for each dissertation. 
Bibliographic citations of the same 

resource, whether because the student 
created duplicate entries or because 
the student cited multiple papers from 
conference proceedings, were included 
only once per dissertation. References to 
newspapers, popular magazines, class 
notes, and personal communications 
were excluded. Unpublished works on 
their way to publication (identified as 
being in press) also were excluded. In the 
event that one of the first 30 citations was 
excluded, we then incorporated citation 
number 31 to maintain a full complement 
of 30 from each dissertation. No disserta-
tion contained fewer than 30 qualifying 
citations. 

In order to track the results of this 
citation analysis, the researchers created 
an electronic spreadsheet that recorded 
an identification number for each dis-
sertation, its date of completion and the 
department from which it emanated. For 
each bibliographic citation, the work’s 
format was identified (and categorized as 
books and monographs, journals, govern-
ment documents, conference papers and 
proceedings, Web pages, or other) and 
additional details were recorded, such 
as date of publication, title, publisher, 
and whether or not the resource is held 
currently at the University of Texas at 
Austin, and if so, in which medium (pa-
per, electronic, or both). 

User Survey 
The second component of this project, 
the user survey, involved contacting all 
doctorate recipients in the identified 
programs for the years specified and 
providing them with a seven-question 
survey that asked about their experi-
ence in using the UT Libraries for their 
doctoral research (See appendix.) The 
survey gathered no personal identifica-
tion information other than departmental 
affiliation and year of graduation. The 
survey results provide basic information 
on the perceived value of UT Libraries’ 
research collections for these disciplines 
and insight into the changes in perceived 
value over a five-year period. 

http:groups.24


 

    

       

     
       

      

 
     

      

     
   

    
      

     
    

       

     
       

      

         
     

    

      

     
    

 

      
 

    
      
 

  
    

 
      

       

     

     

     

     
 

      

        

      

308 College & Research Libraries July 2006 

Results of Bibliographic Citation 
Analysis 
Citation analysis of the four dissertation 
cohorts consisted of investigating 780 
citations for each cohort (i.e., 30 accept-
able citations for each of 26 dissertations 
from each of two years in two depart-
ments, for a total of 3,120 citations). Each 
citation yielded information on format of 
the resource, availability or ownership of 
the resource at the UT Libraries, title of 
the publication and its citation frequency, 
the media in which UT holds the resource, 
year of publication, the span of years 
between the date of publication and the 
dissertation’s year of completion, and the 
publisher for the resource. 

Format 
The researchers grouped the format of 
cited materials into six categories as seen 
in table 1, ranging in frequency from 
journals (representing 51.0% of the cita-
tions across all four cohorts) down to a 
miniscule 0.7 percent for Web pages. 
Government documents included the 
publications of various cooperative agen-
cies that mingle university, state, and fed-
eral entities. Other documents included 
pamphlets, audio or visual materials, and 
reports not placed within government 
documents or some other category. Table 
2 breaks the results down further for each 
of the four cohorts. 

Journals served as the primary source 
of bibliographic citations for each group 
of dissertations studied, ranging from 45.3 
percent (CE 97) to 62.4 percent (EP 02) 
of the total. Books and monographs also 
retained the second position for format of 
choice, ranging from 17.9 percent (CE 97) 
to 41.4 percent (EP 97) of the total for each 
cohort. CE students relied more heav-
ily on government documents (12.2%) 
and conference papers and proceedings 
(9.6%) than did EP students (0.5% and 
1.9%, respectively). Differences between 
the disciplines are notable within the 
theses/dissertations category as well, with 
0.4 percent of EP (02) citations fi ing this 
category and 5.3 percent of CE (02) refer-

encing dissertations or theses. Web page 
citations were nonexistent in 1997 CE dis-
sertations, appeared in just one 1997 EP 
dissertation, but increased significantly to 
16 for 2002 CE dissertations, whereas 2002 
EP dissertations recorded a more modest 
increase of 5 citations. 

Within each discipline, format use re-
mained consistent for the years studied, 
with the exception of EP students’ use of 
journals and books/monographs for their 
dissertations. In 1997, EP students used 
journals for 50.4 percent of their citations, 
but in 2002, that number increased to 62.4 
percent with their dependence on books 
and monographs dropping from 41.4 to 
33.7 percent. 

Availability 
Availability of the resources cited in dis-
sertation bibliographies from each of the 
four cohorts was investigated through use 
of the UT Libraries home page, found at 
h p://www.lib.utexas.edu. In particular, 
the existence of the resource, and more 
specifically the actual volume or issue 
noted in the bibliography, as recorded in 
the libraries’ NetCat, or online catalog, 
was considered indication that the mate-
rial could have been accessed through 
the UT Libraries system. A clear limita-
tion of this method, however, is that the 
researchers had access only to the current 
catalog, not to a retrospective catalog as 
it existed in 1997, or even 2002. A total 

TABLE 1 
Percentage of Citations by Category 

Type of Citation     Percentage of 
Citations 

Journals 51.0 
Books and monographs 27.9 
Government documents 6.4 
Conference papers and 
proceedings 

5.8 

Other documents 5.4 
Theses/dissertations 2.9 
Web pages 0.7 

http:h�p://www.lib.utexas.edu


      

 

      

     

    
 

      
     
      

       
 

      

 

 

 

 

 

 

 

 

 

 

 

 
 

 
 

Behavioral Citation Analysis 309 

TABLE 2 
Format of Citation Titles 

Format Year Civ. 
Eng. 

Ed. 
Psych. 

Grand Total 

Journals 1997 353 393 746 
2002 358 487 845 

Journals total 711 880 1,591 
Books and monographs 1997 140 323 463 

2002 145 263 408 
Books and monographs total 285 586 871 
Government documents 1997 105 7 112 

2002 86 1 87 
Government documents total 191 8 199 
Conference papers and proceedings 1997 79 10 89 

2002 71 20 91 
Conference papers and 
proceedings total 

150 30 180 

Other 1997 68 35 103 
2002 63 1 64 

Other total 131 36 167 
Theses/dissertations 1997 35 11 46 

2002 41 3 44 
Theses/dissertations total 76 14 90 
Web pages 1997 1 1 

2002 16 5 21 
Web pages total 16 6 22 
Grand total 1,560 1,560 3,120 

of 22.0 percent of resources cited do not 
appear in NetCat, implying that at least 
one in five resources used in the disserta-
tion bibliographic citations investigated 
was not supported by the UT research 
collection. 

Frequency of Title Citation 
A total of 2,033 titles were represented by 
the 3,120 citations in this study, of which 
25 titles were cited ten or more times, as 
can be seen in table 3. This group of 25 
titles represents 483 individual citations 
(187 from CE and 296 from EP), or 15.5 
percent of the total citations. At the op-
posite end of the frequency scale, more 

than half the total citations (54.6%, or 
1,704) come from titles that appear just 
once in this study. These numbers vary 
substantially from a study published in 
1969, which concluded that 80 percent 
of citations came from 20 percent of the 
journals, according to Richard Trueswell’s 
80/20 rule.25 The numbers are consistent, 
however, with a 1993 study of psychology 
theses at California State University, Long 
Beach, which found that 80 percent of 
7,797 citations in their study came from 80 
percent of the 1,050 journals cited.26 

Electronic versus paper resources 
Citation of resources in 1997 and 2002 

http:cited.26


 

       

      
   

      
      

     
 

    

     

   
    

    
 

   
    

    
   

   

     

    
   

   

   

     
  

    

  

    

  
   

 

   
     

 

 

310 College & Research Libraries July 2006 

that appear in the 2004 UT NetCat catalog 
only in paper or only in electronic media, 
or those that appear in both, identifies 
further distinctions between cohorts. 
(See figure 1.) Of the total citations, 59.4 
percent are held today by the UT Librar-
ies in only a paper medium, whereas 
16.5 percent appear in both paper and 
electronic media and 2.1 percent are only 

TABLE 3 
Most Frequently Cited Titles by Department 

Civil Engineering 
Transportation Research Record 35 
Journal of Construction Engineering and 
Management 

30 

Environmental Science & Technology 23 
Operations Research 22 
Water Resources Research 22 
Transportation Research, Part B: Methodological 19 
Journal American Water Works Association 14 
Management Science 11 
Transportation Science 11 

Educational Psychology 
Journal of Personality and Social Psychology 68 
American Psychologist 30 
Journal of Consulting and Clinical Psychology 24 
Journal of Learning Disabilities 21 
Journal of Abnormal Psychology 16 
Psychological Bulletin 15 
Psychology of Women Quarterly 14 
Journal of Educational Psychology 14 
Sex Roles 14 
Psychological Review 13 
Diagnostic and statistical manual of mental 
disorders: DSM-IV (4th ed.) 

13 

Archives of General Psychiatry 13 
American Journal of Psychiatry 11 
Personality and Social Psychology Bulletin 10 
Developmental Psychology 10 
Child Development 10 

electronic. Figure 1 represents these find-
ings, indicating that substantially more of 
the resources used by EP students were 
accessible in paper-based resources, (67.2 
percent compared with 51.6 percent for 
CE students). Among the four cohorts, 
resources cited that are available in both 
formats (today) ranged from 15.8 to 18.2 
percent, demonstrating modest increases 

from 1997 to 2002 (2.2% 
increase for CE and 0.2% 
for EP). A somewhat larger 
increase was evident in the 
use of electronic-only media 
in this five-year span, 4.4 
percent for CE and 2.7 per-
cent for EP. These numbers 
do not indicate a substan-
tial shi  toward digitizing 
paper resources, as might 
have been expected during 
this period. 

Age Span 
The average span of time be-
tween dates of publication 
for resources cited in disser-
tations and the dissertations 
themselves was fairly stable 
for each of the four cohorts, 
ranging from 11.3 years 
(EP97) to 13.7 years (EP02), 
as shown in figure 2. When 
the researchers plo ed the 
data, it became clear that the 
concentration of citations 
is weighted heavily within 
the span of twenty years 
preceding the dissertation’s 
completion. These find-
ings contrast with those of 
Bluma C. Peritz and Dina 
Sor who, in 1990, observed 
that “psychological research 
makes fairly extensive use 
of material that is at least 20 
years old.”27 

Publishers 
Citation analysis data re-
vealed a wide distribution of 



 
      
      
       

      
       

     
    

     
        
     

     
      

         
     

      

 

  

Behavioral Citation Analysis 311 

FIGURE 1 
Citation Titles and UT Availability 

Number of Titles 
600 

500 

400 

300 

200 

100 

125 

192 

142 

257 

344 

123 
105 

125 

549 

24 

131 

500 

3 

37 

461 

2 
0 

1997 2002 2002 1997 
Civ. Eng. Ed. Psych. 

Both Print & Electronic 
Electronic 
Not Available 
Print 

publishers in each discipline. There were 
742 total publishers: 363 in civil engineer-
ing and 379 in educational psychology. 
The 363 publishers of CE reference sources 
were associated with 1,475 citations, and 
the 379 publishers of EP resources were 
associated with 1,533 citations. Incomplete 
citations precluded identifying publisher 

information for the remaining citations 
from each cohort. As shown in table 4, 
there was one overwhelming publisher 
in each academic discipline. The Ameri-
can Society of Civil Engineers accounted 
for 124 (8.4%) of the total number of CE 
citations, and the American Psychological 
Association accounted for 248 (16.2%) of 

FIGURE 2 
Average (mean) Time Span between Citation Publication Year 

and Dissertation Year 

N
u
m
b
e
r 
o
f 
Y
e
a
rs

 

16.0 

14.0 

12.0 

10.0 

8.0 

6.0 

4.0 

2.0 

0.0 
1997 

13.39 

11.34 

13.70 

12.68 

2002 

Civ. Eng. 
Ed. Psych 



 

     
     

      
       

       
 

       
    

     
     
      

      
     

      
      

     
      

      
     

   
      

    
 

      
      

     
      

    
      

      
     

     
      

        
        

      
     

      
       

      
       

    
       

       
    

    

 

    

     

    
     

     

  
 

    

 

312 College & Research Libraries July 2006 

the total EP citations. Further analy-
sis demonstrated that 37.0 percent 
(546) of the CE citations were pub-
lished by the top ten publishers in 
the field, as identified in this study, 
and 41.0 percent (628) of EP cita-
tions were published by the top ten 
publishers in that field. 

Results of User Survey 
Had the researchers decided to con-
duct only a traditional citation anal-
ysis, the project would have ended 
at this point. They would have 
gathered some data on availability 
of resources but would be required 
to make serious leaps of inference 
regarding a reasonable course of ac-
tion for the UT Libraries. However, 
the researchers believed that a user 
survey to explore awareness of in-
formation-seeking behavior would 
give more substance to the numbers 
collected from the bibliographic 
citation analysis. 

Out of a population of 126 possi-
ble doctorate recipients in the fields 
of educational psychology and civil 
engineering in 1997 and 2002, the 
researchers received current mailing 
addresses for 116; thus this became 
the study population for the user 
survey. Six requests were returned 
as undeliverable mail; 44 surveys 
were completed for a 40.0 percent 
return rate: 4 from 1997 CE graduates, 13 
from 2002 CE graduates, 8 from 1997 EP 
graduates, and 17 from 2002 EP gradu-
ates. (Two surveys were unidentifiable 
by department and year.) Because only 
12 surveys were returned from those who 
graduated in 1997, the researchers chose 
to analyze these survey data according to 
departmental affiliation only. (The surpris-
ingly small change in use of electronic re-
sources from 1997 to 2002 further justifies 
this decision.). (See table 5.) 

Utility and Accessibility of Collection 
Of the 43 participants who answered the 
question about the collection’s utility, 

TABLE 4 
Top-Ten-Cited Publishers for Civil 

Engineering and Educational Psychology 
Civil Engineering 
American Society of Civil Engineers 124 
University of Texas, Austin 61 
Transportation Research Board 56 
Pergamon Press 56 
Construction Industry Institute 50 
John Wiley and Sons 48 
American Chemical Society 41 
American Concrete Institute 40 
Elsevier 36 
American Society for Testing Materials 34 

Educational Psychology 
American Psychological Association 248 
Sage Publications 62 
Plenum Press 57 
Lawrence Erlbaum Associates 52 
Academic Press 50 
Pergamon Press 41 
American Psychiatric Association 35 
Cambridge University Press 34 
Guilford Press 26 
John Wiley and Sons 23 

nearly all (90.7%) found the UT Libraries’ 
collection moderately to very useful for 
researching and writing their disserta-
tions, not surprising for a top-ten-ranked 
research library with more than 8.4 mil-
lion volumes. Similarly, of 43 participants 
who chose to answer the question about 
the collection’s accessibility, nearly all 
(90.7%) found the UT Libraries’ collec-
tion moderately to very accessible for re-
searching and writing their dissertations. 
Even more interesting, however, is what 
the graduates chose to reveal about their 
information-seeking behavior through 
optional wri en comments. One 2002 EP 
graduate wrote, “Almost all books/pe-
riodicals required were owned by UT.” 



        

Behavioral Citation Analysis 313 

TABLE 5 
User Survey Data 

Q1 The UT General Libraries’ collection of resources in my subject area was 
useful for researching and writing my dissertation. 

Legend 1=Not useful; 7=Very useful 
1 2 3 4 5 6 7 Median 

CE 0 0 1 1 6 6 3 6 
EP 1 0 0 1 6 11 6 6 
No ID 1 
Total 1 0 1 2 12 17 10 6 

Q2 The UT General Libraries’ collection of resources in my subject area was 
accessible for researching and writing my dissertation. 

Legend 1=Not accessible; 7=Very accessible 
1 2 3 4 5 6 7 Median 

CE 0 0 1 1 2 6 7 6 
EP 1 0 0 1 5 13 5 6 
Total 1 0 1 2 7 19 12 6 

Q3 My understanding of the availability of library resources in my subject area 
at the UT General Libraries had a ____________ impact on the selection of 
my dissertation topic. 

Legend 1=Low; 7=High 
1 2 3 4 5 6 7 Median 

CE 10 4 0 1 1 0 1 1 
EP 9 9 4 2 0 1 0 2 
No ID 1 
Total 19 13 4 3 1 1 2 2 

Q4 I accessed approximately ___________ percent of my resources 
electronically. 

Legend 1=0; 2=1-19; 3=20-39; 4=40-59; 5=60-79; 6=80-99; 7=100 
1 2 3 4 5 6 7 Median 

CE 1 4 1 5 4 2 0 4 
EP 2 11 4 2 4 2 0 2 
No ID 1 
Total 3 15 6 7 8 4 0 3 



 314 College & Research Libraries July 2006 

TABLE 5 
User Survey Data 

Q5 I used interlibrary loan services from the UT General Libraries while doing 
research for my dissertation. 

Legend 1=Yes; 2=No 
1 2 Median 

CE 13 4 1 
EP 20 5 1 
No ID 1 
Total 34 9 1 

Q6 Interlibrary loan was useful for conducting my research. 
Legend 1=Not useful; 2=Very useful 

1 2 3 4 5 6 7 Median 
CE 0 0 1 3 1 2 7 6.5 
EP 0 1 0 4 2 3 11 7 
No ID 1 
Total 0 1 1 7 3 5 19 7 

Q7 I used the following resources to carry out my dissertation research, ranked 
in order from most important (1) to least important/valuable (5). 

1 2 3 4 5 Median 
UT General Libraries - CE 5 3 9 0 0 3 
UT General Libraries - EP 15 7 1 1 1 1 
UT General Libraries - No ID 1 
UT General Libraries - Total 20 11 10 1 1 2 

Interlibrary Loan - CE 2 3 2 6 4 4 
Interlibrary Loan - EP 2 8 7 4 3 2 
Interlibrary Loan - No ID 1 
Interlibrary Loan - Total 4 11 10 10 7 3 

Colleagues - CE 4 7 1 4 1 2 
Colleagues - EP 3 5 10 4 2 3 
Colleagues - No ID 1 
Colleagues - Total 8 12 11 8 3 3 

Departmental libraries - CE 4 4 2 3 3 2.5 
Departmental libraries - EP 1 2 4 7 9 4 
Departmental libraries - No ID 1 
Departmental libraries - Total 5 6 6 10 13 4 



 
     

     
      

     

    

     

    
     

 

     
     

    

      

       

 
      

   
     

       
      

     
    

      

 

  
    

     
       

     

     

     
     

      
     

    
     

  

       

    

  

      
     

 

   
      

Behavioral Citation Analysis 315 

TABLE 5 
User Survey Data (continued) 

Other - CE 
Other - EP 
Other - No ID 
Other - Total 

2 
4 

6 

0 2 2 10 5 
1 2 4 10 4 

1 
1 4 7 20 5 

A 2002 CE graduate wrote, “Most of my 
references are from conferences which UT 
General Libraries usually don’t have.” Al-
though an initial reading of this comment 
might appear to express disappointment 
in the collection, couching it, instead, 
within the information-seeking behavior 
of engineering doctoral students reveals 
something different. This anecdote dif-
fers from the research of Gloria J. Leckie, 
Karen E. Pe igrew, and Christian Sylvain 
who found that when engineers consult 
written sources, they prefer technical 
reports and trade journals to scholarly 
books, journals, and conference papers.28 
The changes in information-seeking be-
havior from doctoral student to practicing 
professional in the field of engineering 
are notable. 

Several students commented on the 
convenience of the collection when they 
had access to online databases and online 
journals from home. One 2002 CE gradu-
ate wrote, “I could find the material either 
in hard copy or digital format … most of 
the time.” An EP graduate wrote, “In the 
uncommon event UT did not have the 
required resource immediately available, 
they worked w[ith] me to find/acquire 
the materials.” In contrast, another 2002 
EP graduate stated that he or she had 
to visit three libraries on campus to 
conduct research, indicating a degree of 
discomfort with the experience of visit-
ing multiple repositories. Based on the 
citation analysis alone, the researchers 
concluded that from 1997 to 2002, the 
UT Libraries did a good job of providing 
resources for the EP and CE programs. 
The results implied that the ramp up of 
the use of digital resources was not steep. 
Supplemented with these survey data, 

however, the researchers were able to 
advise the libraries about a need for more 
education on available resources in all 
formats to enable all students (including 
doctoral students) to become more effec-
tive library users. Here is an example of a 
finding that a traditional citation analysis 
alone would not have revealed. 

Selection of Dissertation Topic 
Of the 43 participants who answered 
this question, most (83.7%) chose their 
dissertation topic with no to li le regard 
for the availability of library resources at 
UT. Just 9.3 percent of participants chose 
their dissertation topic with moderate to 
high regard for the availability of UT’s 
library resources, and even fewer respon-
dents (7.0%) were neutral. These results 
are replicated within each department. 
Several 2002 CE graduates wrote that 
their dissertation topic was determined 
by research project needs, their advisor, 
or the availability of a research grant and, 
according to one participant, “had noth-
ing to do with the library.” One 2002 CE 
graduate expressed the situation quite 
succinctly: “Money equals a topic in en-
gineering.” A 2002 EP graduate selected 
a dissertation topic via personal interest 
“and did not consider library resources in 
that decision.” Another 2002 EP graduate 
expressed regret that he or she was not 
“ever given a ‘proper’ introduction to the 
resources available.” 

Electronic Resources 
Of the 43 participants who chose to an-
swer this question, half (55.8%) indicated 
that they accessed one-third or fewer of 
their research materials electronically. 
Over a quarter of the respondents (27.9%) 

http:papers.28


 

   
     
       

    
     
     

     
    

    
     

    
     

   
    

    
    

    
       

 
    

   

      
    

     

    

 
     

        
     

      
     

 
    

 

       
     

    

     
    

        

 

     
    

     
    

    

    
      

      
      

   
     

    
     

       

     

316 College & Research Libraries July 2006 

accessed approximately two-thirds or 
more of their research materials electroni-
cally, and 7 (16.3%) indicated that they ac-
cessed between one-third and two-thirds 
of their resources electronically. No one 
reported accessing all resources electroni-
cally, but three participants indicated that 
they accessed none of their resources 
electronically. 

When the responses to this question 
are analyzed according to departmen-
tal affiliation, some differences appear. 
Among CE graduates, there was a 
nearly equal distribution of responses 
into thirds: one-third of the graduates 
accessed approximately one-third or 
fewer of their resources electronically, 
one-third accessed between one-third and 
two-thirds of their resources electroni-
cally, and one-third of the CE graduates 
reported to have accessed two-thirds or 
more of their resources electronically. 
However, this is not the case with re-
sponses from EP participants. Seventeen 
EP respondents (68.0%) indicated that 
they accessed approximately one-third 
or fewer of their resources electronically. 
In general, two-thirds of the total number 
of respondents accessed two-thirds or 
fewer of their resources electronically. The 
difference between the two departments, 
however, is striking. 

Respondents also provided brief vi-
gne es of the experience of users trying 
to acquire information in today’s hybrid 
(print/electronic) libraries. Although one 
2002 EP graduate emphasized that elec-
tronic access was a “terrific resource,” 
another expressed a common sentiment: 
“I don’t like to read stuff from a computer 
screen.” Another 2002 CE graduate wrote 
about her or his changing approach to 
academic research: “Early on, I used 
mainly hard copies. Last 1.5 years it was 
online exclusively.” 

Interlibrary Loan Services 
Of the 43 participants who answered this 
question, most (79.1%) used interlibrary 
loan (ILL) services while doing research 
for their dissertation. Of 36 respondents 

who chose to gauge the utility of ILL ser-
vices, three-quarters found the services 
to be moderately to very useful. Nearly 
identical results are found according to 
departmental affiliation. 

Three 2002 EP graduates and one 2002 
CE graduate described their perceptions 
of the value of ILL services for their 
research needs in quite positive terms: 
“Impossible without ILL’s help. The 
staff was extremely helpful;” “Crucial. I 
needed materials from journals not pub-
lished in Europe or North America and 
they were able to deliver”; “Extremely 
useful, efficient, dependable. Got many 
obscure resources. Not sure I could have 
completed research without it”; “I used 
the ILL service only a few times, mainly to 
supplement the material I could already 
find at UT.” This is the sort of detail that 
could not be revealed by a citation analy-
sis. For most researchers, ILL is a crucial 
service. 

Ranked Resources 
Participants were asked to rank five 
resources (UT General Libraries, ILL, 
colleagues, departmental libraries, other 
institutions) in order of importance and/ 
or value for conducting their dissertation 
research. Although each respondent did 
not necessarily rank each resource, the 
results reveal interesting details about 
the information-seeking behavior of each 
group of graduates. 

When analyzed only according to 
number of votes for most valuable re-
source, the UT General Libraries comes 
in first with 46.5 percent of the first-place 
votes, colleagues is next with 18.6 percent 
of the votes, followed by other institutions 
(13.9%), departmental libraries (11.6%), 
and ILL (9.3%). When analyzed according 
to average (mean) median score as dis-
played in table 5, the results are not quite 
so overwhelming. (Although calculating 
means implies, incorrectly, that these 
rankings represent an interval scale in-
stead of just an ordinal scale, the resultant 
means, if considered only by their rank 
order rather than making inferences about 



      

       

      

    
      

    

     
       

    
     

   

     
       

      
      

  

      
    

     
      

 
     

    
    

     
   

     

     
      

    

    

   
   

       
    

    
     

    
      

    

     
 

      
    

     

     
     

      

 
     

    
      

 

Behavioral Citation Analysis 317 

the size of the differences, are revealing.) 
UT General Libraries did earn the best 
average median score of 1.9. Colleagues 
is still next with an average median score 
of 2.7. ILL (which came in last among 
first-place votes) had an average median 
of 3.1. Departmental libraries had an aver-
age median of 3.5, and other institutions, 
which came in as the third-best among 
first-place votes, had an average median 
of 3.9. These differences can be explained 
when departmental association is taken 
into account. 

EP graduates ranked the following 
resources according to most votes for first-
place rank: UT General Libraries (60.0% 
of the first-place votes), other institutions 
(16.0% of the votes), colleagues (12.0%), 
ILL (8.0%), and departmental libraries 
(4.0%). By an overwhelming number of 
votes, the UT General Libraries ranks 
as the most important resource for EP 
graduates. CE graduates ranked the 
first-place resources this way: UT General 
Libraries (29.4% of the first-place votes), 
colleagues and departmental libraries 
(tied with 23.5% of the votes each), and 
ILL and other institutions (tied with 
11.8% of the votes each). There is no 
overwhelming winner as far as CE gradu-
ates are concerned. In fact, UT General 
Libraries, colleagues, and departmental 
libraries are almost in a three-way tie for 
most important resource. In addition, far 
fewer CE graduates than EP graduates are 
represented, rendering the collective first-
place vote winner more representative of 
EP respondent views. 

When analyzed according to median 
score by department, the results change 
yet again. Rankings of average (mean) 
median scores for the resources as as-
signed by EP graduates are: UT General 
Libraries (1.6), colleagues (2.88), ILL (2.92) 
(additional significant digit added for 
emphasis), other institutions (3.7), and 
departmental libraries (3.9). Rankings of 
average median scores as assigned by CE 
graduates are: UT General Libraries (2.2), 
colleagues (2.5), departmental libraries 
(2.8), ILL (3.4), and other institutions 

(4.1). These results from CE concur with 
those of Leckie, Pe igrew, and Sylvain, 
who state that engineers rely more on 
colleagues and internal sources of infor-
mation than on repositories outside their 
own organization.29 The other institu-
tions recorded varied widely from other 
university libraries in the United States 
(e.g., University of Pi sburgh, University 
of Michigan at Ann Arbor, University of 
Denver) to government-sponsored offices 
(e.g., Federal Aviation Administration, 
Texas Department of Health, and Texas 
Department of Transportation). 

Comparisons 
In comparing the results of the user study, 
which provided qualitative and behav-
ioral data on library usage and perceived 
utility, with the results of the dissertation 
citation analysis, which provided observ-
able data on collection usage, several 
interesting trends emerge that support 
the value of conducting a citation analysis 
enhanced by adding a behavior-based 
user survey. 

Perceived and Actual Value of UT 
Libraries’ Resources 
In contrasting the results of these stud-
ies, the researchers noted with interest 
that 60.0 percent of the EP respondents 
ranked the libraries as the most significant 
resource for their dissertation research, 
which corresponds well with the data 
that 84.9 percent of their citation sources 
are held by UT Libraries. In comparison, 
29.4 percent of CE respondents ranked 
the libraries as their primary research 
resource, yet 71.2 percent of their cita-
tion sources are held by the UT Libraries. 
When questioned about the value of the 
libraries, 71.4 percent of CE cohorts and 
76.2 percent of EP cohorts identified the 
UT Libraries’ collection in their respective 
subject areas as useful to very useful for 
researching and writing their disserta-
tions. Similarly, 88.2 percent of CE cohorts 
and 92.0 percent of EP cohorts indicated 
that the UT collections were accessible to 
very accessible. Nevertheless, the survey 

http:organization.29


 

     
  

    
       
    

      
      

       
     

      
    

     

   

     
     
      

     

     
     
       

     

    
     

    

     

    

     
    

       
     

      

    

    

 
      
     

    
    

   
     

 
     

    

     
   

     

    

      
    

      
     

      

318 College & Research Libraries July 2006 

indicated a strong preference by CE stu-
dents for acquiring their research materi-
als from nonlibrary sources. Although the 
information-seeking behavior literature 
corroborates these findings, an important 
service choice for the UT Libraries in sup-
porting engineering student research may 
be to arrange access options for students 
to ensure that materials not owned or 
acquired by UT are available from nearby 
agencies when needed by UT students. 

Perceived and Actual Use of Electronic 
Materials 
The researchers found that 18.6 percent 
(581) of the total bibliographic citations 
investigated for this study are available 
to users at the UT Austin Libraries in 
either electronic or electronic and print 
media. More precisely, 19.6 percent (306) 
of CE citations and 17.6 percent (275) of 
EP citations are available in the above-
mentioned media. When the observed 
data are compared to the behavioral data 
for electronic resources, the survey results 
offer a different picture. As noted above, 
approximately half the respondents 
(55.8%) indicated that they accessed one-
third or fewer of their research materials 
electronically and the other half indicated 
greater usage of electronic resources. This 
results in an apparent, but intriguing, dis-
crepancy between actual use of electronic 
materials, which is less than one in five, 
and perceived use of electronic resources, 
which implies higher usage. The Haw-
thorne effect in research states that the fact 
and knowledge of being studied leads to 
altered performance and could provide an 
explanation for this gap if self-reporting 
of electronic resource use is higher than 
citation analysis indicates simply because 
of the approbation accorded to digital 
and electronic resources today. Another 
explanation for the discrepancy could 
be that students use electronic resources 
extensively for initial research and review 
of their subjects and hence report heavy 
use, yet still rely most heavily on print-
format resources for their final write-up 
and analysis. The variation between 

usage figures for electronic resources in 
self-reporting versus actual performance 
warrants additional study, as shi s in 
usage affect collection development deci-
sions with long-lasting consequences for 
researchers. 

Perceived and Actual Use of Interlibrary 
Loan, Other Institutions, and Colleagues 
Just over one-fi h, or 22.0 percent (685), of 
the total number of bibliographic citations 
investigated in the four research cohorts 
were unavailable in the UT Libraries. The 
results of the user survey imply how the 
dissertation authors located their non-
UT research materials, whether because 
the resource was unavailable through 
the UT Libraries or because the authors 
simply preferred to access the materials 
elsewhere. The survey demonstrates that 
79.1 percent (34) of the respondents used 
ILL services, with 9.5 percent of respon-
dents stating that ILL was the primary 
source for their research materials. Ac-
cording to respondents, colleagues and 
other institutions provided needed access 
to research resources as well, with 19.0 
percent and 15.8 percent ranking these 
as the primary source for their research. 
Given that ILL ranked behind colleagues 
and other institutions as the primary 
source for research materials suggests 
the very necessary supporting role such 
a service plays at a large research library. 
The finding also serves to underscore 
the importance of information-seeking 
behaviors that o en turn first to internal, 
“known” sources of information, prefer-
ences that would have gone undetected 
without the survey data. 

Conclusions and Recommendations 
The authors gathered data on research 
collection utility through citation analy-
sis and supplemented and contrasted 
the results with data gathered via a user 
survey on the perceived utility of the 
collection, thereby using two research 
avenues to drive toward a richness of 
analysis. As described in the examples 
above, the results of one approach can 



     
        

      
    

       
      
      

       
       
      

       
      

       
      
      

     
      

     
      

     
      
         

       
       

  

     
    

    
     

 
     

    
    

    

     
     

     
    

      

     

      
    
   

     
   

    
     

   

 

      

     

    

   
      

     

   
      

     
     

      

      

     
      

Behavioral Citation Analysis 319 

complement and illuminate those of the 
other, demonstrating the need to engage 
both measures in order to create a more 
complete understanding of research col-
lection utility. 

If just one investigative approach 
had been used to question the place or 
importance of ILL services in doctoral 
dissertation research, the conclusions 
might have been quite different. Had the 
authors conducted only a classic citation 
analysis, the data would have prompted 
the conclusion that one in five resources 
cited by doctoral students in our study 
likely came from ILL, simply because 
these resources were not available at UT. 
But the survey data indicated that col-
leagues and other libraries play a greater 
role in providing resources than had 
been anticipated. This finding fits neatly 
into the mentor-apprenticeship model of 
doctoral work, leading to the more rea-
sonable conclusion that doctoral students 
might turn to colleagues and professors 
for assistance in gathering research ma-
terials not otherwise available on campus 
rather than to ILL. Though the role of ILL 
as a service provider may remain stable, 
ILL does have an opportunity to expand 
its service. 

As noted earlier, the use of electronic 
resources appears to be greater from self-
reporting than the actual figures from 
citation analysis indicate. Using only 
bibliographic citation data, we might 
conclude that approximately 20 percent of 
the resources used by CE and EP gradu-
ates were in electronic form. However, 
self-reporting by authors indicates that 
half the participants accessed one-third 
or more of their materials electronically. 
An integrated analysis, using a combina-
tion of data-gathering techniques, can 
help diminish the likelihood of drawing 
conclusions falsely. 

Study Qualifications and 
Suggestions for Further Research 
From the perspective of hindsight, the 
two research approaches could have been 
constructed to allow closer correlation of 

results, which would have increased the 
reliability of the data. Cross-over ques-
tions between the two data-gathering 
efforts would have made possible quan-
titative tests on the results and allowed 
for correlation of the data, which then 
would have accorded greater reliability 
to the information extracted as individual 
survey responses could be matched with 
behavioral data for citation use. Had iden-
tified individuals been surveyed rather 
than anonymous authors, behavioral, 
qualitative data could be correlated with 
the quantitative citation analysis for an 
intriguing perspective on information-
seeking behavior. 

Greater a ention to definitions used 
in both approaches may have diminished 
misunderstandings by respondents on 
several survey questions as well as pro-
vided greater consistency in classifying 
the citations. Lack of clarity on the ques-
tion of use of electronic resources may 
have drawn in and lumped together use 
of the library’s electronic catalog, data-
bases, and other tools for research with 
accessing journals, books, and disserta-
tions electronically. Additionally, lack of 
definitional distinction ranking the most 
useful sources of research material, par-
ticularly with regard to the UT Libraries 
and departmental libraries categories, 
may have muddied the results for this 
question. 

The researchers discovered some lack 
of conformity in study definitions that 
translated into variations in the database 
when combining citation spreadsheets 
prepared by two individuals. For the sake 
of data reliability, these definitions should 
be stronger. Titles and publishers were 
difficult to standardize as well. 

With regard to ownership of the re-
source being cited, the researchers noted 
only whether the UT Libraries NetCat of 
2004 indicated ownership of the specific 
volume or edition that was cited and the 
media in which that volume was held, 
if present. Clearly, the use of the current 
catalog to determine the presence and 
media of resources two to seven years 



 

    

    
     

       
      

      

      
     

    
     

    

    
 

     

    

     

      

       
        

       
       
       

      

     
       

      
      

     
       

     
 

     
         

        
     

     
      

    
     

      
     

       
       

      

      

      
       

     
     

     

     
    

     
     

      

320 College & Research Libraries July 2006 

ago is problematic. UT personnel stated 
that little retrospective collecting had 
occurred during this period and that the 
current catalog thus offered a reasonable 
approximation of resources present in 
previous years. Surely, however, an ex-
ception to this could be the appearance of 
resources in dual media, which may have 
been available only in paper at an earlier 
date, thus leading to an overestimation of 
the use of electronic media. 

The curious results generated by aver-
aging citation age for each cohort (results 
ranged from 11 to 13 years) warrant fur-
ther investigation. The existing research 
design did not include any means of 
weighting the value or significance of 
each bibliographic citation in comparison 
with that of others. One-time citations 
ranked equally with those receiving 
heavy use throughout the dissertation as 
they were chosen simply by appearance 
within the first 30 applicable titles in the 
bibliography. Adding a layer of analysis 
that combines textual citation with bib-
liographic citation, ranking the value of 
each, could affect the mean age of cita-
tions used in the dissertations. The 2002 
cohorts did not demonstrate use of sub-
stantially younger materials, as expected, 
based on the findings of Sullenger.30 Part 
of the explanation for this could be the 
phenomenon of courtesy citations. 

Because the only contact information 
received from the University of Texas’s 
Texas Exes alumni association was postal 
addresses, the researchers were unable 
to contact survey population via e-mail. 
Had survey recipients been given the 
opportunity to respond on paper or been 
directed to an online survey response site, 
the researchers believe that the response 
rate likely would have been greater, thus 
increasing the reliability of the data that 
was retrieved. 

A valuable addition to this study would 
have been the comparison of titles not held 
by the University of Texas Libraries with 
those identified as the top-cited (and thus, 
core) journals in the Journal Citation Reports, 
of the Institute for Scientific Information, 

Science Edition and Psychology Edition, 
particularly for the years 1997 and 2002. 
Just one title, Journal of Contemporary Psy-
chotherapy, appeared in both years’ “not 
present” lists, minimizing the concern 
that the UT Libraries are not supporting 
research adequately in the studied disci-
plines. Yet from the libraries’ perspective, 
the data (retrieved through citation analy-
sis that covers a span of years) could be 
reason enough to consider adding a title of 
enduring interest to the collection. 

Conclusion 
The results of this behavioral citation 
analysis study suggest that the UT Li-
braries might market existing research 
resources more effectively, as well as 
extend collections to meet the demands of 
users. Clearly, the research collection does 
match resources to the research interests 
of dissertation authors. Given that more 
than 50 percent of citations out of more 
than 3,000 come from titles cited just once 
among the 104 authors, UT has built a 
broad collection. Four out of five citations 
are accessible through UT. However, the 
perceived value of the UT Libraries as 
a research resource by our respondents 
from civil engineering, as noted above, in-
dicated that they ranked the library as less 
valuable to their research than would be 
expected from these figures. This research 
encourages the UT Libraries to engage 
in more outreach to graduate students, 
perhaps through orientation training or 
increased presence of pathfinders or other 
educational resources on the graduate 
school’s Web site for doctoral students. 

Collection enhancement is an ongoing 
assignment for libraries, made challeng-
ing by the disparity between increasing 
costs of materials, expanded offering of 
materials, and limited funds for acqui-
sitions. As demonstrated in this study, 
librarians can engage in collection evalu-
ation using a number of data-gathering 
techniques to profile the efficacy of a 
collection as well as its perceived utility, 
and thus determine how best to respond 
to user needs. 

http:Sullenger.30


      
    

 

Behavioral Citation Analysis 321 

Appendix: The User Survey 
Survey: Library Resource Utilization at UT-Austin General Libraries by Doctoral 
Students in Educational Psychology and Civil Engineering, 1997 & 2002 

Date: ____________________ 

Departmental Affiliation Date of Dissertation
 Educational Psychology  1997
 Civil Engineering  2002 

1. The UT General Libraries’ collection of resources in my subject area was useful 
for researching and writing my dissertation.

 1 2 3 4 5 6 7 
(Not useful) (Neutral) (Very useful) 

Comments: 

2. The UT General Libraries’ collection of resources in my subject area was acces-
sible for researching and writing my dissertation. 

1 2 3 4 5 6 7 
(Not accessible) (Neutral)  (Very accessible) 

Comments: 

3. My understanding of the availability of library resources in my subject area at 
the UT General Libraries had a impact on the selection of my dis-
sertation topic.

____________ 

1 
(Low) 

2 3 4 
(Moderate) 

5 6 7 
(High) 

Comments: 

4. I accessed approximately _____________ percent of my resources electronically. 

0 1–19 20–39 40–59 60–79 80–99 100 

Comments: 



 

            

322 College & Research Libraries July 2006 

5. I used Interlibrary Loan services from the UT General Libraries while doing 
research for my dissertation. 

 Yes (If yes, proceed to Question #6.) 

 No (If no, proceed to Question #7.) 

6. Interlibrary loan was useful for conducting my research.

 1 2 3 4 5 6 7 
(Not useful) (Neutral) (Very useful) 

Comments: 

7. I used the following resources to carry out my dissertation research, ranked in order 
from most important/valuable (1) to least important/valuable (5). 

________ UT General Libraries 

________ Interlibrary loan 

________ Colleagues (i.e., fellow students, faculty members) 

________ Departmental libraries 

________ Other institutions (please specify below) 

Comments: 

Thank you for your participation!
1
Please return survey by July 15, 2004 in the enclosed stamped envelope to:
1

ATTN: B. Fuchs and C. Thomsen
1
The University of Texas at Austin
1
School of Information
1
1 University Station D7000
1
Austin, TX 78712-0390
1



 
 

 

 

 
 

           
          

 
            

  

 

 

 

 

          

            

            

  

 

 
 

  

 

 

 

 

 
 

 

 
             

           

Behavioral Citation Analysis 323 

Notes 

1. G. Bonn, “Evaluation of the Collection,” Library Trends 22 (1974): 265–304. 
2. Martin Faigel, “Methods and Issues in Collection Evaluation Today,” Library
1

Acquisitions, Practice and Theory 9 (1985): 21–35.
1
3. T. E. Nisonger, Collection Evaluation in Academic Libraries: A Literature Guide and Annotated 

Bibliography (Englewood, Colo.: Libraries Unlimited, 1992). 
4. F. W. Lancaster, If You Want to Evaluate Your Library, 2nd ed. (Champaign: Univ. of Illinois, 

1993). 
5. Joanna Tan Yeok Ching and K. R. Chennupati, “Collection Evaluation through Citation 

Analysis Techniques: A Case Study of the Ministry of Education, Singapore,” Library Review 51, 
no.8 (2002): 398–405. 

6. Margaret Sylvia, “Citation Analysis as an Unobtrusive Method for Journal Collection 
Evaluation Using Psychology Student Research Bibliographies,” Collection Building 17, no.1 (1998): 
20–28. 

7. Philip M. Davis, “Where to Spend Our E-journal Money? Defining a University Library’s 
Core Collection through Citation Analysis,” Portal: Libraries and the Academy 2, no.1 (2002): 
155–66. 

8. Rose B. Okiy, “A Citation Analysis of Education Dissertations at the Delta State University, 
Abraka, Nigeria,” Collection Building 22, no.4 (2003): 158–61. 

9. Anne L. Buchanan and Jean-Pierre V. M. Herubel, “Profiling Ph.D. Dissertation Bibliog-
raphies: Serials and Collection Development in Political Science,” Behavioral and Social Sciences 
Librarian 13, no.1 (1994): 2. 

10. Elizabeth Pan, “Journal Citation as a Predictor of Journal Usage in Libraries,” Collection 
Management 2 (spring 1978): 29–38. 

11. Margaret Stieg Dalton and Laurie Charnigo, “Historians and Their Information Sources,” 
College & Research Libraries 65, no.5 (Sept. 2004): 402. 

12. Marion L. Buzzard and Doris E. New, “Research Notes: An Investigation of Collection 
Support for Doctoral Research,” College & Research Libraries 44, no.6 Nov. 1983): 469–75. 

13. Carle e Washington-Hoagland and Leo Clougherty, “Identifying the Resource and Service 
Needs of Graduate and Professional Students: The University of Iowa User Needs of Graduate 
Professional Series.” Portal: Libraries and the Academy 2, no.1 (2002): 125–43. 

14. William L. Emerson, “Adequacy of Engineering Resources for Doctoral Research in a 
University Library,” College & Research Libraries 18, no.6 (1957): 455–460, 504. 

15. Joy Thomas, “Graduate Student Use of Journals: A Bibliometric Study of Psychology 
Theses,” Behavioral & Social Sciences Librarian 12, no.1 (1993): 1–7. 

16. Lois Kuyper-Rushing, “Identifying Uniform Core Journal Titles for Music Libraries: A 
Dissertation Citation Study,” College & Research Libraries 60, no.2 (Mar. 1999): 153–63. 

17. Laurel A. Haycock, “Citation Analysis of Education Dissertations for Collection Develop-
ment,” Library Resources & Technical Services 48, no.2 (April 2004): 102–6. 

18. Margaret Sylvia and Marcella Lesher, “What Journals Do Psychology Graduate Students 
Need? A Citation Analysis of Thesis References,” College & Research Libraries 56, no.4 (July 1995): 
313–18. 

19. Lois Bebout, Donald Davis Jr., and Donald Oehlerts, “User Studies in the Humanities: A 
Survey and a Proposal,” RQ 15, no.1 (fall 1975): 40–44. 

20. M. B. Line, “The Information Uses and Needs of Social Scientists: An Overview of IN-
FROSS,” Aslib Proceedings 23 (Aug. 1971): 412–34. 

21. Paula Sullenger, “E-journals and Citation Pa erns: Is It All Worth it?” Serials Librarian 44, 
no.3/4 (2003): 109–213. 

22. For a recent study examining the impact of electronic resources on graduate student re-
search behavior, see Erin T. Smith, “Assessing Collection Usefulness: An Investigation of Library 
Ownership of the Resources Graduate Students Use,” College & Research Libraries 64, no.5 (Sept. 
2003): 344–55. 

23. Bebout, Davis, and Oehlerts, “User Studies in the Humanities,” 42. 
24. See David Ellis, “Modeling the Information-seeking Pa erns of Academic Researchers: 

A Grounded Theory Approach,” Library Quarterly 63, no.4 (1993): 469–86, for a classic study. 
For information-seeking behavior of social scientists, see Lokman I. Meho and Helen R. Tibbo, 
“Modeling the Information-seeking Behavior of Social Scientists: Ellis’s Study Revisited,” Journal 
of the American Society for Information Science and Technology 54, no.6 (2003): 570–87. For informa-
tion-seeking behavior of engineers, see Gloria J. Leckie, Karen E. Pe igrew, and Christian Sylvain, 
“Modeling the Information-seeking of Professionals: A General Model Derived from Research on 
Engineers, Health Care Professionals, and Lawyers,” Library Quarterly 66, no.2 (1996): 161–93. 



 

 

 
 

         

 

 
   

     
    

      

      
  

   
    

  
    

      

   
    

     

    
    

    
   

     
  

   

   
    

    
    

 

324 College & Research Libraries July 2006 

25. Richard Trueswell, “Some Behavioral Pa erns of Library Users: The 80/20 Rule,” Wilson 
Library Bulletin 43 (1969): 458–61. 

26. Thomas, “Graduate Student Use of Journals,” 4. 
27. Bluma C. Peritz and Dina Sor, “The Use of Libraries by Graduate Students in Psychology 

as Indicated by Citations,” Collection Management 12, no.3–4 (1990): 11–23. 
28. Leckie, Pe igrew, and Sylvain, “Modeling the Information-seeking of Professionals,” 

165. 
29. Ibid. 
30. Sullenger, “E-journals and Citation Pa erns.” 

Applications/Nominations Invited 
for C&RL Editor 

The deadline for receipt of applications is December 4, 2006. 
Finalists will be interviewed at the Midwinter Meeting in January 2007. 

Applications and nominations are 
invited for the position of editor of 
College & Research Libraries (C&RL), 
the bimonthly, scholarly research 
journal of the Association of College 
and Research Libraries (ACRL). The 
editor is appointed for a three-year 
term, which may be renewed for an 
additional three years. Applicants 
must be a member of ALA and 
ACRL . Q ua l i f ic at ion s i nc lude 
professional experience in academic 
libraries, a record of scholarly 
publication, editing experience, 
a n abi l it y to meet publ icat ion 
deadlines, an understanding of the 
scholarly communication process, 
and a broad knowledge of the issues 
confronting academic libraries. 

S o m e f u n d i n g f o r e d i t o r i a l 
assistance and travel to relevant 
conferences is available, and there 
is a sma l l honora r iu m for t he 
editor. 

Appoi nt ment will be made by 
t he AC RL Boa rd of D i r e c tor s 
at the 2007 Annual Conference 
up on t he r e c om me ndat ion of 
the search commi ee and of the 
ACRL Publications Coordinating 
Commi ee. The incoming editor 
will serve for at least one year as 
editor-designate, working with the 
current editor, before assuming 
full responsibility for C&RL in 
July 2008. 

Nominations or resumes and le ers 
of application, including the names 
of three references, should be sent 
to: 

C&RL Search Commi ee 
c/o Dawn Mueller 
ACRL Production Editor 
50 East Huron Street 
Chicago, IL 60611 
e-mail: dmueller@ala.org