jacoby.indd Assessing the Impact of Reference Services Provided to Undergraduate Students JoAnn Jacoby and Nancy P. O’Brien This article describes a study assessing the impact of reference services onundergraduatestudents.Thestudytargeted undergraduates receiving nondirectional reference assistance, yielding sixty-nine survey responses and five follow-up interviews. Three outcomes were examined: (1) Do undergraduate students perceive the reference staff as being friendly and approachable? (2) Do they learn something during the course of the reference interaction? and (3) Do they feel more confident about their ability to find information after the reference interaction than they did before? Our findings suggest that reference services can play a signifi- cant role in helping students become confident, independent information seekers. Correlations between variables and a multiple regression model further indicate that friendliness of the reference staff was one of the best predictors of students’ confidence in their ability to find information on their own. These outcomes are particularly salient in a college and university environment where building skills for independent information exploration is a primary goal. inding effective and mean- ingful methods for demon- strating the impact of library services has become increas- ingly important as almost every library faces reduced budgets that affect services. Systematic evaluation and assessment of reference services can help demon- strate how library services contribute to broader educational goals and provide an opportunity to examine how well lo- cally defined service goals are being met. The current study assesses the impact of nondirectional reference interactions on undergraduate students by focusing on three specific outcomes: perception of JoAnn Jacoby is an Assistant Professor and Anthropology & Sociology Subject Specialist and Nancy P. O’Brien is a Professor and Head, both in the Education and Social Science Library at the University of Illinois at Urbana-Champaign; e-mail: jacoby@uiuc.edu and npobrien@uiuc.edu, respectively. The authors acknowledge the Research and Publication Commi ee of the University of Illinois at Urbana-Champaign Library, which provided support for the completion of this research, as well as our colleagues Lynne Rudasill and Allison Su on and the Outcomes-based Evaluation Team (Daniel Gelaw Alemneh, Leanne Kallemeyn, Betsy Kelly, Eileen McElrath, Christine E. McNew, Muzghan Nazarova, and Beverly Obert) from the summer 2003 Institute of Museum and Library Services Evaluation Institute held at the UIUC campus, who helped develop the assessment plan. Thanks also to Jennifer Anderson, Sandra Diaz, and Nicole Mailhiot for their capable administration of the surveys and interviews and to Steven J. Taylor and David Ward for their comments on earlier versions of this manuscript. 324 mailto:npobrien@uiuc.edu mailto:jacoby@uiuc.edu The Impact of Reference Services Provided to Undergraduate Students 325 staff approachability, awareness of library resources, and confidence in the ability to find information independently. The results contribute to the body of literature on reference services and information literacy, provide a model of assessment, and serve as a benchmark for services to students. The methods used in this study also provide a model for conducting an evaluation of services that avoids some of the pitfalls associated with standard survey techniques. The inclusion of interviews and the administration of surveys immediately following a refer- ence interaction (a variation of the critical incident technique1) offer an opportunity to gain deeper insight into student needs and perceptions. By limiting our scope to just a few, clearly defined outcomes and focusing on a specific reference interac- tion, we can be confident that the student responses pertain to the situation we are interested in examining. Triangulating between the data gathered via surveys and the rich descriptions from the inter- view transcripts allows for a multifaceted analysis of the impact of reference on undergraduates.2 Literature Review Although there is already a rich and ex- tensive body of literature examining the reference transaction from a variety of perspectives, this particular study focuses on dimensions of the reference interaction that have not been examined system- atically in the research literature. Jennifer Mendelsohn’s 1997 article, “Perspectives on Quality of Reference Service in an Academic Library: A Qualitative Study,” for example, assessed time and morale in relation to quality of service.3 Many other studies have focused on accuracy of reference responses,4 user satisfaction,5 or how effectively the librarian identified the user’s need.6 In their book analyzing the reference interaction, Ma hew Saxton and John V. Richardson identified the “three desirable outcomes of the refer- ence process: utility, user satisfaction, and accuracy.” 7 These outcomes, it should be noted, are particularly well suited to the public library context that was Saxton and Richardson’s primary focus. The present study takes a slightly different tack, focus- ing on user perceptions of the approach- ability of the reference staff; awareness of library resources; and confidence in using information resources independently as the “desirable outcomes.” In a college and university environ- ment, working with students to build skills for independent information dis- covery is paramount, and awareness and confidence are as important as accuracy, utility, and satisfaction. This facet of refer- ence services is just beginning to receive systematic attention in the reference literature. In his recent article analyzing online chat reference transactions, Da- vid Ward introduced a “completeness” measure that uses specific criteria to ex- amine whether chat sessions include “an adequate amount of library instruction” or simply provided “spoon-fed complete citations/answers.” Explaining the impor- tance of this aspect of reference service quality, Ward noted that: Especially in academic environ- ments, there is a learning impera- tive inherent in reference, where librarians seek not just to provide facts or answers to users, but to take advantage of the teaching moment that these encounters provide.8 Librarians’ behavior, including non- verbal cues, is essential to effective refer- ence services. The 2004 RUSA Guidelines for Behavioral Performance of Reference and Information Service Providers address ap- proachability as the first topic, highlight- ing friendliness as one of the foremost be- haviors in reference situations. 9 As Lynda M. Baker and Judith J. Field found in their 1999 study, the demeanor of reference personnel is a critical factor in the inter- view process, students’ perception of the library, and the success of the reference interaction.10 Comparing their findings to two earlier studies by Joan C. Durrance http:interaction.10 326 College & Research Libraries July 2005 in 198911 and by Patricia Dewdney and Catherine Sheldrick Ross in 1994,12 Baker and Field found similar results despite the ten-year interval: the behavior of ref- erence staff can have a direct impact on the success of the reference interaction. Virginia Massey-Burzio’s focus group study also confirmed the importance of friendliness and approachability.13 Marie L. Radford explored the role approachability plays in successful ref- erence interactions in her 1998 article on nonverbal cues from reference staff and their impact on a library user’s decision to initiate a reference interaction.14 Specifi- cally, she found that the appearance and nonverbal behavior of reference staff and any prior experience with librarians had a major effect on that decision. Radford’s research provides some insight into why interpersonal behaviors are important. Drawing on communication theory, her 1996 article suggests that all reference interactions have both a “content dimen- sion” (the information being conveyed by the message) and a “relational dimen- sion” (how the message is expressed).15 The relational dimension, according to Radford, establishes the dynamics of the relationship between the librarian and the person making the inquiry, thus affecting the quality of communication and, ulti- mately, the relative success of the interac- tion.Of particular relevance to the current study, Radford’s book-length study of the reference encounter in academic libraries found that students place a high degree of importance on the interpersonal dimen- sion of the interaction and many value the interpersonal aspects of the encounter more than the information received.16 Background/Setting The Education and Social Science Library (ESSL) is one of nearly fi y departmental libraries at the University of Illinois at Urbana-Champaign. It serves a cluster of social science disciplines encompassing anthropology, education, political sci- ence, psychology, social work, sociology, and speech communication. In addition, it houses several special collections af- filiated with these disciplines such as the children’s literature collection, the arms control, disarmament and international security collection, K–12 curricular and textbook materials, an occult sciences collection, and test instruments. Because of the diversity of its collections, this unit a racts not only undergraduate and grad- uate students, faculty, and staff from the institution, but also community members with interests in these areas. Between July 1, 2003, and June 30, 2004, the ESSL had a total of 264,414 visitors and answered 14,042 inquiries at the reference desk, averaging more than 5,000 visitors and 270 reference questions per week.17 Given these high levels of use and the increasing demand for accountability in undergraduate education, it was a natural decision to survey the subset of users who are undergraduate students. Following is a description of the methods employed and the findings. Methods Librarians at the ESSL collaborated with the Outcomes-based Evaluation Team from the Institute of Museum and Library Services (IMLS) summer 2003 Evaluation Institute to develop an assessment plan for evaluating the impact of reference ser- vices provided to undergraduate students. A er receiving “human subject” clear- ance, the authors trained reference staff (graduate assistants and librarians) in the administration of surveys to undergradu- ate students. The authors developed a sur- vey calendar that designated twenty-one survey days throughout the spring 2004 semester. Survey days were scheduled on three nonconsecutive Sundays, three nonconsecutive Mondays, and so on, such that each day of the week appeared on the survey schedule during different phases of the semester. On survey days, every undergraduate student receiving nondirectional reference assistance was invited to complete a survey.18 The IMLS team conducted a brief pilot of the survey on July 14, 2003, and the sur- http:survey.18 http:received.16 http:expressed).15 http:interaction.14 http:approachability.13 The Impact of Reference Services Provided to Undergraduate Students 327 vey was subsequently revised. ESSL staff made additional minor modifications prior to distribution. (The revised survey instrument can be seen online at h p:// www.library.uiuc.edu/edx/ReferenceSur- vey/reference_survey.pdf.) Respondents had the choice of completing a print or an online version, which differed only in mode of presentation. Surveys required less than ten minutes to complete, and the print version fit on one double-sided page. No personally identifying informa- tion was collected, and a consent form was provided along with both versions. Surveys were administered immedi- ately following a reference interaction, when the specifics of the encounter were still fresh in the respondents’ minds. This method enabled the authors to ask a few tightly focused questions and be certain that the answers pertained to the particular interaction in which we were interested, in this case, a reference interaction at the ESSL. A survey tied to a specific “critical incident” can provide more precise and nuanced data than large-scale surveys that are not connected to a particular event or context. With surveys that are distributed to a broad population, there is o en the risk that the respondent is conflating more than one incident or referring to something other than what the survey authors intended. A critical incident survey with just a small number of carefully targeted questions is far less susceptible to such pitfalls.19 On the twenty-one days that surveys were distributed, every fi h undergradu- ate student was invited to participate in an interview. Those who agreed were given a consent form and a scheduling card to be completed and dropped in the survey collection box. All participants in the interviews were given a $15.00 incentive in the form of a value card that could be used to purchase photocopies at various campus locations or libraries, or snacks and supplies at a store in the stu- dent union. Interviews were conducted by phone or in person as close as possible to the date of the reference interaction. Interviews were taped with permission, or notes were made by the interviewer and/or observer. The authors hired and trained non- library-affiliated undergraduate students to handle scheduling, interviewing, and transcribing. In order to focus recruitment on students enrolled in social science research methods classes, the authors asked professors teaching research meth- ods classes in anthropology, psychology, and sociology to forward the position announcement to their students. In ad- dition, the opening was posted on the mailing list of the UIUC chapter of the Psi Chi Psychology Honor Society. This targeted recruitment provided a large pool of highly qualified candidates. All students hired had previous experience conducting, transcribing, and/or coding interview responses and were given ad- ditional training to ensure confidentiality, accuracy, and reliability. During the first eight sample days, the response rate was lower than expected due, in large part, to the difficulty of administering the survey while also pro- viding responsive service in a very busy se ing. Teaching and instruction are the fulcrum of the reference philosophy at the ESSL. Because we strive to show people how to be independent information seek- ers, reference interactions o en involve showing a student how to use a tool or suggesting an approach or strategy, then following up with further suggestions as the student continues the process on his or her own. When they had set a student going in a productive direction, reference staff were o en loath to interrupt in order to ask the student to complete a survey. Moreover, the reference desk is a very busy service point, which can make it difficult to follow up on asking a student to complete the survey, especially during peak times when a single staff member might be working with a number of different people simultaneously. In addi- tion, some staff members were reluctant to approach an individual they had just helped and ask for something in return, http:pitfalls.19 www.library.uiuc.edu/edx/ReferenceSur 328 College & Research Libraries July 2005 as they perceived it. A er discussing these challenges with the rest of the reference staff, the authors decided to schedule the students who had been hired for data entry and interviewing to serve as survey administrators. The student survey ad- ministrators stood or sat by the reference desk and watched for the least intrusive moment to approach people after the reference transaction. They could wait until students had finished their search but catch them before they le the library. Not surprisingly, the rate of response improved when the undergraduate stu- dents began administering the survey. The number of respondents rose from an average of 1.25 per sample day to over 4.5 per sample day.20 The assessment was designed to address the following three basic ques- tions: 1. Do undergraduate students per- ceive the reference staff as being friendly and approachable? 2. Do they learn something during the course of the reference interaction? 3. Do they feel more confident about their ability to independently find the information they are seeking following the reference interaction? Various dimensions of these three outcomes were measured using a combination of survey questions (both open-ended and Likert scale) and follow-up inter- views. An iterative content analy- sis technique was used to analyze the five-interview transcripts.21 SAS System for Windows V8 was used to analyze the survey data. In addition to basic descriptive statistics, the strength of the re- lationship between the variables associated with the Likert scale survey questions was examined using the Pearson correlation coefficient. The authors also ex- plored the explanatory power of a multivariable model of the factors predicting undergraduate student confidence following the reference transaction using a multiple regression model. For all statistical tests, p-values less than 0.05 were considered significant. Results and Discussion Survey Sixty-nine undergraduates responded to the survey (49 print and 20 online). The preference for print surveys was unex- pected but may be explained by the fact that the respondents could clearly see that the survey was quite brief with the print version. On one occasion, in fact, a student who had declined to complete the survey online changed her mind when shown the print version, commenting, “Well, if that’s all there is to it, I can do it.” Profile of the Undergraduate User The majority of the undergraduates re- sponding to the survey (62.3%, n = 43) ma- jored in a social science discipline that fell within the officially designated purview of the ESSL (figure 1), and most came from the largest departments served: education (29.0%, n = 20) and psychology (21.7%, n = 15). Another 34.8 percent (n = 24) of stu- dents came from a wide variety of majors, including Afro-American studies, biology, FIGURE 1 Major Disciplines of Undergraduate Survey Respondents Psychology Other / Undeclared Education Anthropology Speech Comm Political Science http:transcripts.21 The Impact of Reference Services Provided to Undergraduate Students 329 FIGURE 2 Year in School of Survey Respondents 0 5 10 15 20 25 30 35 P er ce n t Freshman Sophomore Junior Senior business administration, engineering, and premed. The remaining 2.9 percent (n = 2) were undeclared. This confirms the au- thors’ long-standing assumption that al- though most of the undergraduates using the ESSL major in education or one of the social science disciplines, the library also serves a significant number of students from across campus. The majority of respondents were juniors (34.8%, n = 24) and seniors (33.3%, n = 23), as shown in figure 2. In contrast, fresh- men (24.6%, n = 17) and sophomores (7.2%, n = 5) are probably more likely to use the undergraduate library as their primary library. The fairly large contingent of freshmen and non–social scientists might be partially explained by the universi- ty’s rhetoric requirement, frequently completed dur- ing the freshman year, for which many students seem to choose to write on topics related to social issues. Most of the respondents were regular users of the E S S L , w i t h a l m o s t h a l f (46.4%, n = 32) reporting that they had visited the library on a weekly basis during the semester of the study. (See figure 3.) Another 14.4 percent (n = 10)) reported that they visited several days per week or daily. However, 17.4 percent (n = 12) re- ported that the reference transaction took place during their first visit to this library. The responses to the question “Why did you come to the Education and Social Science Li- brary today?” are listed in table 1. Respondents could select more than one option. Most people indicated that they had come to look for an article (50.7%, n = 35) and/or a book (47.8%, n = 33). “Browsing journals” (29.0%, n = 20) was the third most frequent response. The two people (2.9%) answering “other” indicated that they were there to return or renew a book. Similarly, the top responses to “Why did you ask the person at the reference desk for assistance?” were “help finding articles” (44.9%, n = 31) or “help FIGURE 3 Frequency of Visits during the Semester of the Study 0 10 20 30 40 First Visit Occasionally Weekly Several Days Per Week Everyday Percent 50 330 College & Research Libraries July 2005 finding books” (39.1%, n = 27) and “help using databases or e-journals” (20.3%, n = 14). (See table 2.) Three of the six people answering “other ” (8.7%, n = 6) indicated that they wanted help renewing or requesting books in the online catalog. Other respondents indicated that they wanted to “know how to tell the dif- ference between a magazine and a journal while searching online since they’re grouped together ” or needed “help with specific issues within research topic.” Undergraduate Perceptions of Reference Staff The survey included three separate questions pertain- ing to undergraduate student perceptions of the friendliness and approachability of the ref- erence staff. The first question asked, “On a scale of 1–6, how approach- able was the person at the reference desk?” with 6 being “extremely approachable” and 1 being “not at all approachable.” The mean of the responses was 5.78. (See figure 4A for the distribution.) The next question in this grouping was unambiguous: “Did the person at the reference desk smile when you first talked with him/her?” One person (1.4%) indicated that the person at the reference desk was not smiling when first spoken to. Another eleven (15.9%) did not notice one way or another. The remaining fi y-seven respondents (82.6%) indicated that the person at the reference desk smiled when they first talked to him or her. The final question asked, “On a scale of 1–6, how friendly was the person you talked to at the reference desk?” The mean of the responses was 5.71, with 6 being extremely friendly and 1 being “not at all friendly.” (See figure 4B.) Perhaps instead of asking whether the person at the reference desk was friendly, it would have been be er to ask whether he or she TABLE 1 Responses to “Why did you come to the Education and Social Science Library today?” Number of Respondents Percent Look for article 35 50.7% Look for book 33 47.8% Browse journals 20 29.0% Photocopy 13 18.8% Consult library staff 11 15.9% Use computers 10 14.5% Do work or study 10 14.5% Look for other materials (CD, microfilm, etc.) 4 5.8% Do group work/projects 4 5.8% Browse shelves for books 2 2.9% Other (please specify) 2 2.9% Review newly arrived items 0 0.0% Note: Respondents could choose more than one response, so total exceeds 100 percent. was helpful because the responses to the open-ended survey questions suggest that “helpful” may be the more salient term. The word “helpful” or “help” was used in thirteen of the sixteen open-ended comments; only four people specifically mentioned that the staff were friendly. Confidence Before and After Another series of questions measured the confidence that the students had in their ability to find information indepen- dently both before and a er the reference interaction. Undergraduates felt much more confident in their ability to find the answer to their question a er ge ing help from the ESSL reference staff. (See figure 5.) The respondents’ mean confidence level rose from 3.38 to 5.03 (on a scale from 1 to 6, with 6 being “extremely confident” and 1 “not at all confident”) a er they received assistance. A paired T-test shows that the difference between the students’ confidence level before (N = 69, M = 1.65) and a er the reference interaction is sig- nificant, t(68) = 8.26, p < .0001) . The Impact of Reference Services Provided to Undergraduate Students 331 TABLE 2 Responses to “Why did you ask the person at the reference desk for assistance?” Number of Respondents Percent Help finding articles 31 44.9% Help finding books 27 39.1% Help using databases or e-journals 14 20.3% Wanted to save time 12 17.4% Item wasn’t on shelf 11 15.9% Person at desk asked if I needed help 7 10.1% Other (please specify) 6 8.7% Learning my way around 5 7.2% Referred by professor or TA 4 5.8% Referred from another library 1 1.4% Note: Respondents could choose more than one response, so total exceeds 100 percent. The open-ended comments from sur- vey respondents also suggest that refer- ence assistance helps build the confidence students need to become independent information seekers. As one student noted, “They’re very patient and always take time to show me how to find what I’m looking for and show me new things to make it easier.” Another student who came in with a high degree of confidence nonetheless acknowledges the value of expert assistance: “I can almost always find stuff on my own; however, it’s always faster asking and they help point other, overlooked things out.” Awareness of New Resources and Learning Most of the survey respondents (63.8%, n = 44) indicated that they learned about new resources during the reference encounter. The open-ended responses to the prompt to “list the resources you learned about” included a mix of particular resources, library facilities and services, as well as more general skills and approaches to information seeking. Specific resources mentioned included Educa- tion Full-Text, OVID and In- foTrac, and UMI Proquest Digital Dissertations. Library facilities and services included “the stacks upstairs,” “remote storage,” and “requesting from other libraries.” In addi- tion, a number of respondents indicated that they learned new approaches to finding the information they sought. Responses in this category included: “how to search for online journals and the full- text articles in them,” “com- puter catalogue headings,” and “how to search for peer- reviewed journals.” O en the students indicated that they learned how to use a known tool more effectively (e.g., “how to use Psych info more efficiently,” “learned a new technique on Psycinfo—how to search by author,” and “how to do multiple search [sic] on OVID”). Self-assessment of Information Retrieval Skills Immediately following the reference encounter, the undergraduates surveyed were confident about their ability to find information independently, as noted above. Figure 6 shows the respondents’ self-evaluation of their ability to complete specific information retrieval tasks, such as finding a book or searching the online catalog. Students rated their ability to per- form these tasks on their own by selecting options on a scale of 1 to 5, with 1 being “never” and 5 being “always.” A final op- tion, “don’t use” (#6), was selected only once, in response to “I can find journal articles without assistance.” In general, the students were fairly confident in their abilities to complete these tasks. They expressed the least confidence in their ability to retrieve print sources such as books, journals, encyclopedias (M = 3.71, SD = .925, N = 69), even though they were 332 College & Research Libraries July 2005 mostly confident in their ability to search the online catalog (M = 4.01, SD = 0.962, N = 69). This may be due to local conditions characterized by a decentralized departmental library structure that can make physical retrieval of an item challenging because libraries are scattered across campus and o en have different hours. Or it may be due to a more general difficulty undergradu- ates have in transitioning from full-text databases (one-stop shopping) to navigating a hybrid and distributed information en- vironment. Relationships among the Variables Examination of the correlations among all variables measured on a Likert scale, as well as fre- quency of visits (using Pearson Correlation Coefficients in SAS V8), shows numerous significant ones, but most were fairly weak, accounting for only a small portion of the variation in the variables. (See table 3.) For in- stance, student perception of the friendliness of the reference staff showed a statistically significant correlation with confidence fol- lowing the reference interaction (p = 0.0178, N = 69), but percep- tion of friendliness accounted for only 8.10 percent (r2 = 0.081) of the variation in confidence a er the interac- tion. This suggests that friendly service may help bolster students’ confidence in their ability to find the answers to their questions on their own, but the effect is rather slight. Frequency of visits to the library during the semester of the study was correlated with students’ confidence in their ability to answer their questions independently before the reference interaction (r2 = 0.142, p = 0.0014, N = 69), suggesting that more frequent users start out somewhat more FIGURE 4 Perception of Reference Staff (A) Approachability (M = 5.78, SD = 0.510, N = 69) and (B) Friendliness (M = 5.71, SD = 0.88, N = 69) 0 10 20 30 40 50 60 Extremely Friendly Not at All Friendly Number of Respondents 0 10 20 30 40 50 60 Extremely Approachable Not at All Approachable Number of Respondents A. Friendliness of Reference Staff B. Approachability of Reference Staff confident in their ability to find answers to their questions than less frequent library users. (See table 3.) Frequency of visits also was correlated with all self-as- sessment of ability indicators, except the ability to find journal articles without assistance. Perhaps frequent library users learn that finding articles can be more challenging than they initially thought! In addition, frequency of visits was nega- tively correlated with the perception of the approachability of the reference staff (r2 = -0.168, p = 0.0005, N = 69), but not significantly correlated with the percep- The Impact of Reference Services Provided to Undergraduate Students 333 FIGURE 5 Confidence in Ability to Find Information Independently (A) before (M = 3.38, SD = 1.58, N = 69) and (B) after the Reference Interaction (M = 5.03, SD = 1.01, N = 69) 0 5 10 15 20 25 30 35 Extremely Confident Not Confident at All Number of Respondents A. Before Reference Interaction 0 5 10 15 20 25 30 35 Extremely Confident Not Confident at All Number of Respondents B. After Reference Interaction tion of friendliness. That is, 16.8 percent of the variation in undergraduate perception of the approachability of the reference staff was explained by how frequently they had visited the library during the semester of the study, with the percep- tion of approachability declining as the frequency of visits increased. Three pos- sible explanations of this follow. It may be a case of familiarity breeding contempt. Or perhaps approachability is simply not an issue with more frequent users who may take it for granted that they can ask the reference staff for assistance. It also may be a ma er of regular users being unwilling to admit they still need assistance. That is, users more conversant with the library through prior interactions may feel reluctant to reveal that they still are unaware of all the re- sources and therefore identify the staff as less approachable due to their self-assessment. Drawing on information from the interviews, it appears that most users are comfortable asking for assistance whether or not they have used the library previously. Four of the five students interviewed were prior users of the library and indicated that the staff was friendly and they had no problem approaching the reference desk for assistance. The sole interviewed student who had not previously used the library recognized a need for assistance and immediately approached the reference desk. At least one of the students interviewed, how- ever, indicated that despite feeling quite competent, she would have no problems asking for help. Multiple Regression Model As discussed above, examination of the correlations among vari- ables measured on a Likert scale as well as the variable “frequency of visits” indicates that there are multiple positive correlations among the metrics. (See table 3.) There- fore, the authors chose to examine the explanatory power of a multiple-vari- able model in predicting “confidence after ” the reference interaction. The eight predictor variables can be thought of as belonging to one of four groups that seem to explain different aspects of the same or closely related phenomena: Frequency of Visits, Confidence Before Refer- ence Interaction, Perception of Reference Staff (“approachability,” “friendliness”), and Self-Assessment of Abilities (“can use online catalog,” “can use online databases,” “can 334 College & Research Libraries July 2005 FIGURE 6 Self-assessment of Information Retrieval Skills (N = 69) (A) Can Find Books, Journals and Reference Materials Such as Encyclopedias without Assistance (M = 3.71, SD = .925); (B) Can Find Journal Articles without Assistance (M = 3.82, SD = 1.02); (C) Can Use Online Databases without Assistance (M = 3.97, SD = 0.923); and (D) Can Use Online Catalog without Assistance (M = 4.01, SD = 0.962) Note: Respondents could also check “not used,” which was selected once in response to “Can Find journal articles.” 0 5 10 15 20 25 30 Always Never Don't Use Number of Respondents 0 5 10 15 20 25 30 Always Never Don't Use Number of Respondents 0 5 10 15 20 25 30 Always Never Don't Use Number of Respondents 0 5 10 15 20 25 30 Always Never Don't Use Number of Respondents A. Can Find Books B. Can Find Articles C. Can Use Online Databases D. Can Use Online Catalog find articles,” “can find books”). Indeed, there is high correlation among the vari- ables within these groups, as shown in table 3. Although “confidence before” reference interaction seems obviously related to “confidence a er ” reference interaction (r2 = 0.0579, p = 0.0465), the correlation is barely significant and only a small portion of the variation (5.79%) in “confidence a er” reference interaction is explained by “confidence before” refer- ence interaction. Confidence Before Refer- ence Interaction was therefore retained as a separate predictor variable for evaluation in the final model. Frequency of Visits is the only other “group” containing just one variable (accounting for 0.90% of the variation in “confidence a er” reference interaction). Because correlations among variables within Self-Assessment of Abilities and Perception of Reference Staff are gener- ally high (p < 0.0011, table 3), the authors chose to first identify the best predictor of “confidence a er” reference interaction within each of these two groups before The Impact of Reference Services Provided to Undergraduate Students 335 TA B L E 3 C or re la ti on a m on g th e V ar ia bl es (N = 6 9, e xc ep t “ C an F in d A rt ic le s, ” w he re N = 6 8 be ca us e on e re sp on de nt s el ec te d “D on ’t U se ”) . C on fid en ce B ef or e C on fid en ce A ft er A pp ro ac ha bi lit y F ri en dl in es s C an U se C at al og C an U se D at ab as es C an F in d A rt ic le s C an F in d B oo ks F re qu en cy o f V is it s r2 0. 14 2 0. 00 9 0. 16 8 0. 02 6 0. 09 1 0. 07 1 0. 04 2 0. 10 0 p 0. 00 1 0. 43 7 0. 00 1 0. 18 4 0. 01 2 0. 02 7 0. 09 2 0. 00 8 C on fid en ce B ef or e r2 na 0. 05 8 0. 02 3 0. 00 2 0. 17 8 0. 11 6 0. 12 5 0. 05 6 p 0. 04 7 0. 21 3 0. 70 2 0. 00 0 0. 00 4 0. 00 3 0. 05 0 C on fid en ce A ft er r2 0. 05 8 na 0. 01 0 0. 08 1 0. 01 1 0. 00 4 0. 00 8 0. 04 5 p 0. 04 7 0. 42 5 0. 01 8 0. 39 0 0. 60 3 0. 46 1 0. 07 9 A pp ro ac ha bi lit y r2 0. 02 3 0. 01 0 na 0. 32 4 0. 00 7 0. 01 1 0. 03 6 0. 06 7 p 0. 21 2 0. 42 5 <. 00 1 0. 49 7 0. 38 1 0. 12 3 0. 03 1 F ri en dl in es s r2 0. 00 2 0. 08 1 0. 32 4 na 0. 00 0 0. 00 0 0. 02 7 0. 00 8 p 0. 70 2 0. 01 8 <. 00 1 0. 85 6 0. 87 7 0. 17 9 0. 45 7 C an U se C at al og r2 0. 17 8 0. 01 1 0. 00 7 0. 00 0 na 0. 74 2 0. 33 3 0. 14 8 p 0. 00 0 0. 39 0 0. 49 7 0. 85 6 <. 00 1 <. 00 1 0. 00 1 C an U se D at ab as es r2 0. 11 6 0. 00 4 0. 01 1 0. 00 0 0. 74 2 na 0. 35 3 0. 22 3 p 0. 00 4 0. 60 3 0. 38 1 0. 87 7 <. 00 1 <. 00 1 <. 00 1 C an F in d A rt ic le s r2 0. 12 5 0. 00 8 0. 03 6 0. 02 7 0. 33 3 0. 35 3 na 0. 42 3 p 0. 00 3 0. 46 1 0. 12 3 0. 17 9 <. 00 1 <. 00 1 <. 00 1 C an F in d B oo ks r2 0. 05 6 0. 04 5 0. 06 7 0. 00 8 0. 14 8 0. 22 3 0. 42 3 na p 0. 05 0 0. 07 9 0. 03 1 0. 45 7 0. 00 1 <. 00 1 <. 00 1 336 College & Research Libraries July 2005 a empting to identify the best regression model. To determine whether more than one variable within each of these groups should be included in the final model, the authors first conducted a stepwise regression within each group. For the Self-Assessment of Abilities group, “can find books” had the greatest explanatory value, accounting for 4.54 percent of the variation in “confidence a er” reference interac- tion. Adding the other variable, “can find articles,” explained only 0.40 percent more of the variation in “confidence a er” ref- erence interaction. Because of its lack of ability to explain a meaningful proportion of the remaining variation in “confidence a er” reference interaction, and because of its high degree of correlation with “can find books” (r2 = 0.4235, p < 0.0001), “can find articles” was excluded from the final model. Other variables in Self-Assessment of Abilities (“can use online catalog,” “can use online databases”) explained even less of the remaining variance in “confidence a er” reference interaction and thus were excluded from the final model. For Percep- tion of Reference Staff, “friendliness” of ref- erence staff had the greatest explanatory value, accounting for 8.10 percent of the variation in “confidence a er” reference interaction. Adding “approachability” of reference staff as a second variable in the Perception of Reference Staff group ex- plained only an additional 0.61 percent of the unexplained variance in “confidence a er” reference interaction. This lack of unique explanatory value, combined with the strong correlation of “approachability” of reference staff and friendliness of refer- ence staff (r2 = 0.3241, p < 0.0001), resulted in the exclusion of “approachability” from the final model. As a result of the above analyses, the authors were left with the single-best Likert scale variable from each group of related metrics (“confidence before,” “frequency of visits,” “can find books,” “friendliness”) and then proceeded with a stepwise regression to determine a fi- nal model that best predicts “confidence a er” reference interaction. The regres- sion coefficients of the first two variables, “confidence before” reference interaction and “friendliness” of reference staff, added to the model were both significant (p = 0.0289 and p = 0.0114, respectively), explaining a total of 14.56 percent of the variation in “confidence a er” reference interaction. In the best three-variable model, the regression coefficient of the third variable, “can find books,” was not significant (p = 0.1034) and only explained an additional 3.44 percent of the variation in “confidence a er” reference interac- tion; therefore, the authors dropped “can find books” and chose the two-variable model: “Confidence a er” reference interac- tion = 0.96048 + (“friendliness” of refer- ence staff * 0.61604) + (“confidence before” reference interaction * 0.16312) as the best model, explaining 14.56 percent of the variation in “confidence a er” reference interaction (F2,66 = 5.62, p = 0.0056). Thus, the best predictors of students’ “confidence a er” reference interaction are their confidence before the interaction and the perceived friendliness of the ref- erence staff. These two variables account for more variation than their self-assess- ment of abilities or frequency of visits to the library. This suggests that librarians’ service behaviors may play an important role in encouraging students to become confident in their own abilities to find the information they seek. It also lends sup- port to previous studies that found that nonverbal cues and the user’s perception of a librarian’s demeanor contribute to the success of the reference interaction.22 Interviews Twelve people agreed to be interviewed, but because of a number of no-shows, only five interviews were conducted. Although it was disappointing that fewer than half of those willing to be interviewed actually participated, it does indicate the level of busyness of students and the reality of following through. The interviewers made multiple efforts by e- mail and telephone to contact, schedule, http:interaction.22 The Impact of Reference Services Provided to Undergraduate Students 337 and reschedule students who agreed to be interviewed. The interviews were transcribed, cod- ed, and analyzed using content analysis to identify passages relevant to the focus of this study. Consistent themes appeared based on the coding of responses that largely corroborated the survey data and provided context-rich qualitative data that captured nuances missed by the statistical analyses. In terms of friendliness and approach- ability, all students interviewed used phrases such as “nice, friendly, and helpful.” Four of the five used the word “nice” eight times to describe staff, one used the word “friendly” three times, and three mentioned “helpful” five times. Working to create a climate of ap- proachability helps ensure that students feel comfortable enough to approach staff. Demonstrating behaviors such as friendliness and helpfulness creates this climate and, in fact, these behaviors are mentioned repeatedly in training sessions for new staff and modeled by continuing staff. In their 1994 article, Dewdney and Ross found that high ratings of friendli- ness “were significantly associated with both overall satisfaction and willingness to return.”23 Students reported different levels of awareness of new resources and strategies for finding information a er their interac- tions with the reference staff. Two of the five students interviewed indicated that although they were unable to get exactly the items they were searching for (in one case a journal not held in the library and in another an unpublished test), they did learn new strategies to search for resources, including electronic sources and databases, and the physical location of print materials. Other general strategies they learned included improved skills in searching article databases and the online catalog, greater awareness of the resourc- es available, and the process for locating and retrieving materials, notably from a remote storage facility. Two students specifically mentioned using knowledge they gained from their reference interac- tion in subsequent library visits. The level of confidence displayed by students before and a er their reference interactions supports the findings from the survey. Four of the five students interviewed had previous experience in this departmental library; one was totally unfamiliar with the library and immediately asked for help. All five in- dicated an increased level of confidence a er receiving assistance and believe they could navigate library resources on their own, although at different levels. Being provided with assistance seemed to instill a sense of comfort and independence a er each reference interaction. Three of the five indicated that they would be able to use databases on their own in future; one indicated an increased awareness of more complex services involving retrieval from off-site facilities; and one expressed some initial anxiety about using the library but, based on interactions, felt able to be more independent in future. In each of these instances, the student expressed a sense of being able to locate resources without assistance. Each also indicated that he or she would be quite comfortable asking for assistance when he or she did not know how to approach a research project or assignment. Based on the interviews, even students who are quite confident when they approach the library believe that they gain from the reference interac- tions. Those who believed they needed help initially found that the assistance they were given taught them sufficient skills to begin their next assignments on their own. Two of the students interviewed ex- pressed some anxiety about their skills prior to asking for assistance. One clearly expressed feeling “kind of shaky, that’s why I asked for help.” These two were clear that the library’s numerous resources were confusing and asking for assistance was an early decision in each case. The other three students expressed li le anxi- ety about using resources because they either had used the library before and 338 College & Research Libraries July 2005 felt comfortable asking staff for assistance or had previous positive experiences at other libraries that reduced their anxiety about asking for assistance. In fact, all five students described previous experiences that developed expectations for positive interactions. One of these experiences was at a public library; the other four were at campus libraries or reading rooms. All of the experiences on campus were positive, with friendly, approachable staff. The public library experience involved several librarians assisting a youth group and was an equally positive experience. Because of previous interactions, the anxiety levels for approaching reference staff were low and the expectations for friendly, helpful service were high. This lends support to Durrance’s suggestion, based on her un- obtrusive study of 266 reference interac- tions in 142 public, academic, and special libraries, that “willingness to return” is an important measure of success in reference interactions.24 In fact, Durrance found that users were more likely to excuse inaccu- rate reference service if the reference staff were friendly and pleasant than if they were uninterested or unhelpful. Although every library should strive for accurate service, it also is important to provide friendly and courteous interactions so that the library experience is positive even when an answer is not readily available. Conclusion The majority of survey respondents indi- cated that they learned something new in the course of the reference interaction. Of- ten they learned about a specific resource or library service. In addition to becom- ing aware of particular tools, facilities, or services, many students learned new approaches to finding information. This suggests that the reference interaction in college and university libraries can be an effective means of teaching students not only about specific library resources, but also about the process of finding, evaluat- ing, and using information. The data from the study also indicate that students found the reference staff at the ESSL to be friendly and approachable. When undergraduates perceive the staff to be approachable, they have crossed the first barrier to ge ing whatever as- sistance they might need. A friendly and approachable staff helps ensure that students will ask for help when they need it; therefore, the ESSL reference staff are encouraged to watch for users who “look like they need help” and initiate the reference interaction. Indeed, more than 10 percent of the survey respondents indicated that the reason they asked “the person at the reference desk for assis- tance” was, in fact, because the “person at desk asked if I needed help.” Friendliness of the reference staff also may play a role in how confident students feel about their own abilities to find the information they seek, as suggested by the correlation between these two variables (table 3) and by the multiple regression model. Reference staff friendliness was one of the best predictors of student confidence in their ability to find infor- mation independently. Demonstrating service behavior that is friendly, helpful, and encouraging not only contributes to students’ “willingness to return,” it also may help students to become more con- fident in their ability to find information on their own.25 Students were clearly more confident in their ability to find information following the reference interaction than they were before they spoke to the reference staff (figure 5). However, their self-assessment of their ability to find books, journals, and reference materials such as encyclopedias without assistance remained relatively low (figure 6). At a discussion session on explaining the catalog hosted by the RUSA Catalog Use Commi ee at the 2004 ALA Annual Meeting in Orlando, librarians from a variety of college and university libraries noted that, for many of the stu- dents they encountered during reference interactions and in the classroom, figur- ing out how to move from a citation to a library catalog and then from a holding record to a physical item somewhere in http:interactions.24 The Impact of Reference Services Provided to Undergraduate Students 339 the library system was a major challenge. This issue also is discussed at some length by Julie M. Chapman, Charlcie K. Pe way, and Steven A. Scheuler, who offer a selec- tion of interactive classroom exercises designed to teach students how to inter- pret citation elements, OPAC records, and online database components.26 These difficulties speak to the need to find ways of helping students navigate between these o en disjunct “information silos.”27 Clearly, the library community should pay close a ention to the challenges students face in this area whenever we work with them, whether in the classroom or at the reference desk. Moreover, we need to be mindful of the challenges students face when designing interfaces and selecting products. Perhaps in the quest for seam- less, confederated searching, we need to find a way to incorporate more cues to the structure of the literature that would allow users to move easily between citation and sources, regardless of format, so that the jump from citation (or catalog record) to the physical item is less of a blind leap. Reference services clearly can play a significant role in helping students become confident, independent informa- tion seekers. Considerable a ention has been given to this aspect of library ser- vices in the vast literature on information literacy, but the present study findings suggest that the reference interaction is also an important locus for produc- ing lifelong learners able to navigate a complex information environment.28 Ad- ditional studies are needed that explore the teaching dimension of reference, especially in academic libraries where reference services can play a vital role in the overall educational process. Notes 1. Grace Fivars, Critical Incident Technique: A Bibliography (Palo Alto, Calif.: American Institutes for Research in the Behavioral Sciences, 1980). 2. Lynn Westbrook, “Evaluating Reference: An Introductory Overview of Qualitative Meth- ods,” Reference Services Review 18 (spring 1990): 73–78. Westbrook pointed out that multiple methods of evaluation and assessment are best for determining whether a user’s needs have been met. By combining methods such as surveys and interviews, it is likely that more rigorous data will result. 3. Jennifer Mendelsohn, “Perspectives on Quality of Reference Service in an Academic Library: A Qualitative Study,” RQ 36 (summer 1997): 544–57. 4. Goldhor’s landmark study on the accuracy of reference responses (Herbert Goldhor, A Plan for the Development of Public Library Service in the Minneapolis–Saint Paul Metropolitan Area (Minneapolis: Department of Education, Library Division, 1967), introduced the 55 percent rule that holds that just over half of all reference questions are answered accurately. Additional accu- racy studies are reviewed in Ma hew Saxton and John V. Richardson Jr., Understanding Reference Transactions: Transforming an Art into a Science (San Diego, Calif.: Academic Pr., 2002), 35—40; and Peter Hernon and Charles McClure, “Unobtrusive Reference Testing: The 55% Rule,” Library Journal 111 (Apr. 15, 1986): 37–41. The 55 percent rule has recently been challenged by John V. Richardson (“Reference Is Be er Than We Thought,” Library Journal 127 (Apr. 15, 2002): 41–42) in a study that found that librarians recommended an accurate source or strategy in 90 percent of cases reviewed by a panel of experts. 5. User satisfaction studies are reviewed in Saxton and Richardson, Understanding Reference Transactions, 40–44. In “Digital Reference: Reference Libraries’ Experiences and A itudes,” Journal of the American Society for Information Science and Technology 53 (May 2002): 549–66, Joseph Janes noted a dearth of studies examining user a itudes toward digital reference services, though it appears that such studies have begun to be published. See, for instance, Margie Ruppel and Jody Condit Fagan, “Instant Messaging Reference: Users’ Evaluation of Library Chat,” Reference Services Review 30 (2002): 183–97. 6. The literature on question negotiation and how effectively the librarian identified the user’s need is reviewed in Richard E. Bopp, “The Reference Interview,” in Reference and Information Services: An Introduction, 3rd ed., ed. Richard E. Bopp and Linda C. Smith (Englewood, Calif.: Libraries Unlimited, 2001) and in Saxton and Richardson, Understanding Reference Transactions, 47–49. 7. Saxton and Richardson, Understanding Reference Transactions, 2. http:environment.28 http:components.26 340 College & Research Libraries July 2005 8. David Ward, “Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study,” Reference and User Services Quarterly 44 (fall 2004): 53. 9. Guidelines for Behavioral Performance of Reference and Information Service Providers (Chicago: Reference and User Services Association, ALA, 2004). Available online at h p://www.ala.org/ala/ rusa/rusaprotools/referenceguide/guidelinesbehavioral.htm [accessed 12 October 2004]. 10. Lynda M. Baker and Judith J. Field, “Reference Success: What Has Changed over the Past Ten Years?” Public Libraries 39 (Jan./Feb. 2000): 23–30. 11. Joan C. Durrance, “Reference Success: Does the 55% Rule Tell the Whole Story?” Library Journal 114 (Apr. 15, 1989): 31–36. 12. Patricia Dewdney and Catherine Sheldrick Ross, “Flying a Light Aircra : Reference Service Evaluation from a User’s Viewpoint,” RQ 34 (winter 1994): 217–30. 13. Virginia Massey-Burzio, “From the Other Side of the Reference Desk: A Focus Group Study,” Journal of Academic Librarianship 24 (1998): 208–15. 14. Marie L. Radford, “Approach or Avoidance? The Role of Nonverbal Communication in the Academic Library User’s Decision to Initiate a Reference Encounter,” Library Trends 46 (spring 1998): 699–717. 15. ———, “Communication Theory Applied to Reference Encounter: An Analysis of Critical Incidents,” Library Quarterly 66 (Apr. 1996): 123–37 16. ———, The Reference Encounter: Interpersonal Communication in the Academic Library (Chicago: ALA, 1999). 17. These numbers are based on an actual tally, not sample periods. The reference total includes both directional and nondirectional reference questions answered by librarians and graduate as- sistants at the reference desk, via e-mail, or by phone. Questions answered by other staff at other service points are not included. 18. Directional questions (referring to the location of books, journals, rest rooms, and so on or having to do with the operation of copiers, computers, printers, or other machinery) were not considered reference interactions for the purposes of this study. 19. John C. Flanagan, “The Critical Incident Technique,” Psychological Bulletin 51 (1954): 327–59; Grace Fivars, Critical Incident Technique: A Bibliography (Palo Alto, Calif.: American Institutes for Research in the Behavioral Sciences, 1980). For a discussion of the limitations of surveys, see Doris J. Schlichter and J. Michael Pemberton, “The Emperor’s New Clothes? Problems of the User Survey as a Planning Tool in Academic Libraries,” College and Research Libraries 53 (May 1992): 257–65. 20. On average, the Education and Social Science Library answered just over thirty nondi- rectional questions per day during the 2004 fiscal year (July 1, 2003–June 30, 2004), but it is not known what proportion of these questions was asked by undergraduates. 21. Earl Babbie, “Content Analysis,” in The Practice of Social Research (Belmont, Calif.: Wad- sworth, 2001): 304–15; William L. Neuman, “Content Analysis,” in Social Research Methods: Qualita- tive and Quantitative Approaches, 5th ed. (Boston: Allyn and Bacon, 2003), 310–19. 22. Durrance, “The Influence of Reference Practices on the Client–Librarian Relationship,” College and Research Libraries 47 (Jan. 1986): 57–67; Roma M. Harris and B. Gillian Mitchell, “The Social Context of Reference Work: Assessing the Effects of Gender and Communication Skills on Observers’ Judgement of Competence,” Library and Information Science Research 8 (Jan.–Mar. 1986): 94–99; Jo Bell Whitlach, The Role of the Academic Reference Librarian (New York: Greenwood Pr., 1990); Dewdney and Ross “Flying a Light Aircra .” 23. Dewdney and Ross, “Flying a Light Aircra .” 24. Durrance, “Reference Success.” 25. Ibid. 26. Julie M. Chapman, Charlcie K. Pe way, and Steven A. Scheuler, “Teaching Journal and Serials Information to Undergraduates: Challenges, Problems, and Recommended Instructional Approaches,” Reference Librarian 79/80 (2002/2003): 363–82. 27. Albert Gore, “Infrastructure for the Global Village,” Scientific American 265 (Sept. 1991): 151. In this essay, Gore deplored the barriers that impeded people’s access to information: “To realize the full benefit of the information age, high speed networks that tie together millions of computers must be built…. Our current national information policy resembles the worst aspects of our old agricultural policy, which le grain ro ing in thousands of storage silos while people were starving. We have warehouses of unused information ro ing.” 28. Hannelore Rader, “Information Literacy 1973–2002: A Selected Literature Review,” Library Trends 51 (fall 2002): 242–59.