College and Research Libraries Maybe the 55 Percent Rule Doesn't Tell the Whole Story: A User-Satisfaction Survey Carolyn W. Jardine For many years the evaluation of reference service has focused on the accuracy with which patrons' questions are answered. It has been sug- gested that an area that needs further study is the behavioral aspect of reference service, i.e., the interaction between reference librarian and patron. Patrons will judge the service they receive not only on whether or not they get what they came in for, but also on the reference librarian's attitude, behavior, interest, and enthusiasm. This paper discusses a sur- vey done at the University Library of the University at Albany as a study of reference success based solely on users' satisfaction with librarians' behaviors and, most important, users' willingness to return to the same librarian for help another time. Results of this survey indicate that users' satisfaction with reference service does depend on more than the accu- racy with which their questions are answered. II recently went into a local drug- store for some cough syrup. I wasn't sure what would be the best kind and had to rely on the pharmacist, whose job it is to know about such things, for help. I eventually did get what I needed, but I had to wait for the pharmacist to notice I was there. He didn't really seem to know much about what was available, and he was less than enthusiastic if not downright surly. Was my need-the right cough medi- cine-satisfied? Yes. Would I return to that particular pharmacist for help again if I had a choice? Probably not. Was the en- counter successful? Well, it depends on how you define success. Strictly speak- ing, it could be considered successful be- cause I got what I wanted. From a behav- ioral point of view, however, I was less than satisfied with the service I received and will think twice before returning to that drugstore in the future-something the manager should keep in mind when evaluating that particular pharmacist's job performance. And so it is-or should be-with li- braries. Probably since the beginning of libraries, people have questioned the ef- fectiveness of the services they provide and how to make them better. This is es- pecially true of reference services, since the Reference or Information Desk is, in a way, the liaison between the library and the clientele it is there to serve; reference librarians can provide (or fail to provide) Carolyn W. Jardine is a reference librarian at the Newburgh Free Library, Newburgh, New York. This article is adapted from the author's M .L.S research project while at the Rockefeller School of Information Science and Policy at the University at Albany, Albany, New York . Special thanks to David Tyckoson for his support and encouragement with this project. 477 478 College & Research Libraries patrons with access to the information they expect to find . There has been much research into the accuracy with which ref- erence librarians answer questions, and this is an ongoing concern of the library community. But there is a consensus among experts that there needs to be more emphasis on reference service as a whole, There is a consensus among experts that there needs to be more empha- sis on reference service as a whole, not just how accurately patrons' questions are answered. not just how accurately patrons' questions are answered. Or, in the words of one Reference Department head, "We must evaluate not only the answer, but the pro- cess as well." 1 Research suggests that one area of the reference process which needs further study is the interaction between reference librarian and patron. The question of suc- cess rests not just upon "Did the patron get the correct information?" but also upon "Would the patron return to that li- brarian for help?" Did the librarian's at- titude, behavior, interest, enthusiasm, etc., leave the patron with a good impression of the library's service, and of that librar- ian in particular, so that the patron would feel confident about approaching that li- brarian for help another time? Particu- larly in academic libraries, where often reference questions are not of a strictly factual nature, users' satisfaction with the service may depend as much or more on librarians' attitudes and behavior than on whether or not the librarian answered the question successfully. This paper dis- cusses the design, administration, and results of a survey done at the University Library at the University of Albany as a study of "reference success" based solely on users' satisfaction with librarians' be- haviors and, more importantly, users' willingness to return to the same librar- ian for help another time. November 1995 Literature Review Research into the literature on evaluating reference service has revealed that (1) there is an enormous amount of it, and (2) very little of it relates specifically to evaluating service based primarily on users' satisfaction with librarians' atti- tudes and behavior. Many studies that have evaluated reference service based on the percentage of questions answered correctly have been done. So many, in fact that the "55 percent rule," which states that barely more than 55 percent of ques- tions asked at a reference desk are an- swered correctly, has become somewhat of a cliche in library literature. 2 As previ- ously stated, experts seem to agree that more qualitative research into reference effectiveness needs to be done; quantita- tive evaluation, which tends to focus on numbers of reference questions asked and answered (successfully or unsuccess- fully), does not reflect the whole picture of reference service, particularly with re- gard to academic libraries. The nee~ for a more qualitative approach to evaluation of reference services is well documented in the literature: "the correct answer fill rate appears to be a useful, but [an] ex- tremely limited, measure of reference per- formance"; "One needs to study the in- teraction between user and librarian"; "To provide a complete picture of the effec- tiveness of the reference librarian, any accountability measures need to eval- uate ... behavioral factors along with the accuracy of the final response"; and, "The most promising methodologies for evalu- ating reference service librarians are those that focus on what reference librarians do and how they do it, the evaluation of behav- iors" (emphasis added). 3-6 The need for qualitative research in the area of reference effectiveness appears well established and accepted. However, a search of recent literature turns up very little in the way of studies which focus on user satisfaction based primarily on "the evaluation of behaviors" by the us- ers themselves, independent of whether their questions were successfully or un- successfully answered. In a 1984 article reviewing research in reference effective- ness, Ronald Powell found that, "Few studies that have focused on the patron as the primary source of data on reference effectiveness have been reported in the literature."7 In the nearly ten years since then, little seems to have changed. A few studies incorporated user evaluation of behaviors and attitudes of reference li- brarians as part of a larger overall analy- sis of reference effectiveness. These in- clude: a comprehensive study of user sat- isfaction with reference services at the University of South Africa; a study in which videotapes of reference transac- tions were watched and evaluated by public library users; Linda Olson's study of academic library reference services; and the work of Marjorie Murfin and Gary Gugelchuk on the development and testing of a "reference transaction assess- ment instrument" in fifteen academic li- braries (especially worth mentioning in this context because their results showed a relatively high degree of user satisfac- tion). 8-11 However, while these studies can all provide helpful insight into the evalu- ation of user satisfaction, as well as some concrete hints and models for testing methods, for the most part their hypoth- eses and methods are not directly relevant to this study because they include vari- ables (such as number of questions an- swered correctly, or gender of librarian and observer) which this study did not. Recently, Patricia Hults cited a "study done in Maryland [which] concluded that the highest predictive factor of success is the individual librarian's behavior such as reference question negotiation skill, li- brarian interest and comfort with the question and perhaps most importantly, follow up"; but a study reported in 1989 by Joan Durrance is probably the most relevant to the survey discussed in this paper both in method and in emphasis on behavioral factors. 12 Although it was done unobtrusively and on a much larger The 55 Percent Rule 479 scale (data were gathered by M.L.S. graduate students who posed as reference patrons in a variety of libraries and then filled out a questionnaire based on the reference transaction), Durrance's study is particularly worth noting because "the measure chosen [to measure reference success] is the willingness of the inquirer to return to the same staff member at a later time," and because she also reported a high degree of "patron" satisfaction: be- tween 60 and 64 percent said that they would be likely to return to the same li- brarian for help another time. 13 Method of Investigation Survey Design In an attempt to determine the level of user satisfaction with reference services at the University at Albany University Library, based strictly on behavioral fac- tors, the researcher surveyed patrons dur- ing the fall1993 semester. Several consid- erations went into the design of this sur- vey. First, the questions had to relate solely to the user's appraisal of the librarian's attitudes and behaviors dur- ing the reference interaction, with ulti- mate success or failure measured as the user's willingness to return to that librar- ian in the future. The survey placed em- phasis on behavioral characteristics of the librarians, such as interest, confidence, friendliness, and enthusiasm, as well as The survey placed emphasis on behavioral characteristics such as interest, confidence, friendliness, and enthusiasm, as well as on the patrons' degree of ... overall satisfaction with the librarians. on the patrons' degree of comfort and overall satisfaction with the librarians. A ratings scale of 1-5 was used, with one being high and five low. The variable "Was your question satisfactorily an- swered?" was included in the survey, not as the primary issue but as one of several 480 College & Research Libraries factors to be considered when analyzing responses to the main question, "If you had a choice, would you return to this li- brarian for help another time?" Environ- mental considerations, such as number of librarians on duty, number present at the desk, and amount of time it took to be helped, were included in the survey as factors which certainly influence users', as well as librarians', attitudes. Less criti- cal but still important, the physical de- sign of the survey kept it as short as pos- sible, to encourage maximum participa- tion and completion, while still incorpo- rating all questions considered necessary for the patron to give an accurate, as well as fair, evaluation of the librarian's ser- vice. In addition, the design of the sur- vey purposely avoided any items that would encourage identification or make it possible to identify individual librar- ians. The intent of the survey was to evaluate behavioral aspects of the refer- ence service as a whole rather than of any particular staff member. The design of the survey and some of the questions in- cluded were modeled in part on the pre- viously cited work by Olson, Durrance, and Murfin and Gugelchuk. Data Collection A graduate student who was stationed behind the reference desk handed out the surveys and approached patrons upon completion of a reference transaction, ask- ing for voluntary participation in the sur- vey. Patrons who asked questions such as "Where is the restroom?" were not sur- veyed, nor were those who left the build- ing or went to another floor of the library before they could be approached. Other than that, the student attempted to ask as many patrons as possible to participate in the survey, making no distinctions be- tween short and long questions, which librarian was asked, or patrons who "seemed" satisfied and those who didn't. Originally, the researcher intended, based on the size of the student popula- tion at the University at Albany, to collect November 1995 approximately 200 completed surveys for this research. Ultimately, because of time constraints on the project, it was not pos- sible to distribute that many surveys. At the end of the two-week time period al- lotted for data collection, 111 surveys had been distributed. Six were not returned and five were returned incomplete, invali- dating them, which resulted in a total of 100 completed surveys. This is admittedly a statistically suspect number but, while it did simplify analysis of the results, it was purely coincidental. The surveys were distributed at times scheduled to represent all operating hours of the refer- ence desk (i.e., both busy and quiet) and all staff as equally as possible. People were surprisingly cooperative; less than ten percent of those asked to fill out a survey refused and those who did almost all cited lack of time as the reason. Data Analysis Initial returns showed a high degree of user satisfaction as measured by this sur- vey. Results continued to be high and, by the completion of the administration of the survey, fully 99 percent of the respon- dents had said they would return to the same librarian for help another time- impressive results, especially as that par- ticular question was considered the most important of the survey. (A sample sur- vey, with a breakdown of the raw data, is available from the author.) Although 99 out of 100 respondents said they would return to the same librar- ian and all other results were very favor- able, it is worth looking at how patrons' satisfaction and comfort ratings compare with their ratings of the behavioral traits. That is, how did a patron who was only marginally satisfied overall (a 3 or 4 rat- ing) rate the librarian as far as knowledge, friendliness, etc.? Conversely, how did those very satisfied and comfortable pa- trons rate the behavioral traits? Tables 1 and 2 show, for each of the ratings (1-5, with 1 being high and 5 being low) on the Degree of Comfort and Overall Satis- The 55 Percent Rule 481 TABLEt Degree of Comfort (Number of patrons responding in each category [ 1 =High, 5 =Low] on Question 6) 1 2 3 4 5 (69) (24) (6) (1) (0) Knowledgeable 1.2 1.4 2.2 4 N/A Self-confident 1.3 1.6 2.3 4 N/A Helpful 1.1 1.3 1.5 4 N/A Friendly 1.2 1.7 1.5 4 N/A Patient 1.1 1.6 1.7 4 N/A Interested 1.3 1.8 1.8 4 N/A Enthusiastic 1.7 2.0 2.0 4 N/A Numbers in the· Table represent average behavioral trait ratings given by category respondents TABLE2 Degree of Satisfaction (Number of patrons responding in each category [ 1 =High, 5 =Low] on Question 7) 1 2 3 4 5 (73) (22) (3) (2) (0) 1 Knowledgeable 1.2 1.7 3.0 2.5 N/A Self-confident 1.3 1.8 3.0 2.5 N/A Helpful 1.1 1.5 1.6 2.5 N/A Friendly 1.2 1.6 1.6 3.5 N/A Patient 1.2 1.6 2.0 3.0 N/A Interested 1.3 1.9 2.0 3.5 N/A Enthusiastic 1.5 2.0 2.7 4.0 N/A I Numbers in the Table represent average behavioral trait ratings given by category respondents 482 College & Research Libraries November 1995 TABLE3 librarians who helped Overall Average for Behavioral Traits them. As for the four people who responded that their questions had . not been answered satis- factorily, three of them said that they were "very satisfied" and "very comfortable" with the li- brarian who helped them, and indicated in the comments section that the fault was not with the librarian but rather because the infor- mation they sought was unavailable. Conversely, (1 =High, 5=Low) Knowledgeable 1.4 Self-confident 1.5 Helpful 1.2 Friendly 1. 4 Patient 1.3 Interested Enthusiastic faction questions (#6 and #7 respectively), the average ratings given by those pa- trons to the behavioral traits. As would be expected, lower degrees of comfort and overall satisfaction are reflected in lower ratings for most behav- ioral characteristics. Obviously, a patron's perception of a librarian's knowledge, interest, and enthusiasm will affect how satisfied a patron feels with that librar- ian's service. On the whole, however, overall ratings for the behavioral traits (from question #5 on the survey) were very high, as indicated in table 3. It is also possible, based on this table, to rank these traits from highest to lowest, to see what the patrons considered the strengths and (relative) weaknesses of the reference li- brarians. Further analysis of data gathered by this survey reveals some other informa- tion. Interestingly (although not terribly relevant to this study), by far the largest number of questions asked were for re- search guidance (see Question #9). Also, although nearly one-third of the people surveyed were not encouraged by the li- brarian to ask for further assistance if it was necessary, this seems not to have been very important to those patrons since 90 percent of them indicated that they were satisfied or very satisfied overall with the 1.5 1.8 94 patrons did have their questions satisfactorily answered but only 72 patrons said they were "very sat- isfied" overall. The Results Obviously, we could not have asked for better results, with a 99 percent success rate and very high ratings for the behav- ioral traits and overall satisfaction. This does, of course, bring up the question of the Hawthorne phenomenon. Did these librarians, who were 'certainly aware of the survey being done, change their nor- mal behaviors so that the study would reflect only positive opinions about them? The researcher addressed this concern during the planning stages of this project, when the merits of a study of this type were weighed against the possibility of unrealistic results. The department head, knowing the professionals on the staff, felt that although they might initially be aware of the survey being done, they would not consciously alter their behav- ior, and believed that this survey would give an accurate picture of the reference service provided by the staff. It is thought that this was in fact what happened-that the librarians, serving patrons at a busy reference desk and not able to be sure which patrons would participate in the survey, were too absorbed in their work to be artificially helpful, enthusiastic, or friendly. However, whether or not any of these librarians did treat patrons differ- ently because of the survey being done, the effect on the patrons remains the same. The central question of this study focused on patrons' satisfaction with li- brarians' behaviors, regardless of the motivation behind those behaviors. The averages for the behavioral char- acteristics-which all fell between the highest and next highest possible rating- show that patrons gave the librarians the highest marks for helpfulness and pa- tience, and the lowest for self-confidence and enthusiasm. In fact, one patron noted, "I don't believe I've ever met an enthusi- astic librarian," and the ratings seem to show that her fellow patrons shared her view. Enthusiasm received by far more middle-to-low ratings than any of the other behavioral characteristics on the survey. However, as the overall average rating for enthusiasm was still better than two, on a scale of 1-5, this is hardly cause for serious alarm-just something to keep in mind. Overall, it seems safe to say that users of the University Library reference service are remarkably satisfied with the service. It also is The 55 Percent Rule 483 Conversely, of the six patrons who did not indicate that their questions were an- swered satisfactorily, five of them still were "Very Satisfied" (the highest rating) with the librarian who helped them. Ob- viously, things like patience, friendliness, and enthusiasm do influence a patron's overall impression of a librarian. From table 4 it can be seen that of the ninety- four patrons whose questions were satis- factorily answered, those who were most satisfied with the service they received gave higher overall ratings to the behav- ioral traits than those who indicated that they were less than completely satisfied. Since all of these patrons said they did receive the information they were look- ing for, it can be inferred that the differ- ences in their overall satisfaction were because of their perception of the librarian's knowledge, interest, friendli- ness, etc. It will be noted that there is no discus- sion of the results with regard to the time of day the surveys were filled out, num- ber of librarians on duty, how long a pa- tron had to wait, etc. This is primarily because these factors had little influence on patrons' overall satisfaction. Re- TABLE4 safe to say, based on this survey, that patron satis- faction does indeed rest Comparison of Behavioral Trait Ratings* on more than simply an- swering questions cor- rectly. Fewer than 75 per- cent of the patrons who indicated that their ques- tions were answered cor- rectly said they were "very satisfied" overall with the librarians who helped them, so slightly more than 25 percent of the patrons who had their questions correctly answered were still not completely satisfied with the service they received. Less than "Very "Very Satisfied" Satisfied" Knowledgeable 1.2 1.8 Self-confident 1.3 1.9 Helpful 1.1 1.6 Friendly 1.2 1.8 Patient 1.2 1.7 Interested 1.4 2.0 Enthusiastic 1.6 2.2 *Of all patrons who said their questions ~ satisfactorily answered 484 College & Research Libraries sponses did not differ statistically for dif- ferent times of day; patrons seemed equally satisfied whether the desk was quiet or busy, and whether they had to wait a short time or not at all (no patrons said they had to wait a "long time"). Conclusion It should be remembered that this was a very small study, with a very narrow pur- pose. We wanted to find out how satis- fied patrons are, based on behaviors, with the librarians who staff the reference desk at the University Library, so we asked them. And what they told us is, "We are very satisfied (but you could be a little more enthusiastic)." It is thought that ... the librarians, serving patrons at a busy reference desk, ... were too absorbed in their work to be artificially helpful, enthusiastic, or friendly. It is hard to say whether the impres- sive results of this survey would be du- plicated elsewhere or even whether the results would be the same were the iden- tical survey to be done again in this li- brary. For our purposes, it can be consid- ered successful in that (1) it gave us a good indication of how patrons feel about the reference librarians, and (2) showed that patrons do, indeed, take attitudes and November 1995 behavior into account when assessing their overall satisfaction with those librar- ians. For others interested in evaluation of reference service and personnel, a sur- vey of this type can provide valuable in- formation from the patron's point of view. If it is true that "library users are ... al- most invariably satisfied with reference service" (and what is wrong with that?), it still is worth looking at why.14 There are probably those who would be tempted to discount the results of a survey that shows a 99 percent satisfac- tion rate with reference librarians, espe- cially a survey such as this which is based on evaluation of the librarians by the pa- trons themselves. It has been suggested by more than one writer in the area of ref- erence evaluation that patrons are notre- liable judges of the services libraries pro- vide and that "library users seldom pos- sess the expertise to evaluate the quality of reference service." 15 There are others, however, who would agree with George D'Elia and Sandra Walsh, who wrote in their article "User Satisfaction With Li- brary Service" that "the user, as the ulti- mate consumer of these services, is most qualified to evaluate the performance of these services." 16 Patrons will judge, and obviously do judge, the services librari- ans provide, and the library profession as a whole needs to acknowledge its clientele's judgment and be responsive to it. Notes 1. David A. Tyckoson, "Wrong Questions, Wrong Answers: Behavioral vs. Factual Evalua- tion of Reference Services," Reference Librarian 17 (1992): 151-73. 2. Peter Hernon and Charles McClure, "Unobtrusive Reference Testing: The 55 Percent Rule," Library Journal111 (Apr. 15, 1986): 37-41. 3. Jo Bell Whitlach, "Unobtrusive Studies and the Quality of Academic Library Reference Services," College & Research Libraries 50 (Mar. 1989): 181-94. 4. Louise Koller Green, "Accessing [sic] the Effectiveness of Reference Services: A Difficult But Necessary Process," Catholic Library World 50 (Jan./Feb. 1988): 168-71. 5. Tyckoson, "Wrong Questions," 159. 6. William F. Young, "Methods for Evaluating Reference Desk Performance," RQ 25 (fall1985): 69-75. 7. Ronald R. Powell, "Reference Effectiveness: A Review of Research," Library and Information Science Research 6 (Jan. 1984): 3-19. 8. Gwenda M. E. Dalton, "Quantitative Approach to User Satisfaction in Reference Service The 55 Percent Rule 485 Evaluation (at the University of South Africa)," South African Journal of Library and Information Science 60 (June 1992): 89-103. 9. Roma M. Harris and B. Gillian Michell, "The Social Context of Reference Work: Assessing the Effects of Gender and Communication Skill on Observers' Judgments of Competence," Li- brary and Information Science Research 8 (Jan. 1986): 85-101. 10. Linda M. Olson, "Reference Service Evaluation in Medium-Sized Academic Libraries: A Model," Journal of Academic Librarianship 10 (Jan. 1984): 322-29. 11. Marjorie E. Murfin and Gary M. Gugelchuk, "Development and Testing of a Reference Transaction Assessment Instrument," College & Research Libraries 48 (July 1987): 313-38. 12. Ralph Gers and Lillie Seward, "Improving Reference Performance: Results of a Statewide Study," Library Journal10 (Nov. 1, 1985): 32-35; cited by Patricia Hults in "Reference Evaluation: An Overview," Reference Librarian 17 (1992): 141-49. 13. Joan C. Durrance, "Reference Success: Does the 55 Percent Rule Tell the Whole Story?" Library Journal114 (Apr. 15, 1989): 31-36. 14. Young, "Methods for Evaluating Reference Desk Performance," 74. 15. William F. Young, "Evaluating the Reference Librarian," Reference Librarian 9 (fall/winter 1984): 123-29. 16. George D'Elia and Sandra Walsh, "User Satisfaction with Library Service-A Measure of Public Library Performance?" Library Quarterly 53 (Apr. 1983): 109-33. Elegant Solutions for PreseiVation Call for a complete catalog Protective Enclosures Pamphlet Binders Music Binders Bound Four Flap Enclosures Tan Archival Board Grey/White Archival Board Drop Spine Archival Boxes Academy Folder Manuscript Folder 2134E Grand Avenue Des Moines, Iowa 50305 PH. 800.526.5640 FAX 515.262.6013 e-mnil /NET: 70670,2635@Compuserve.com "Cite While You Write"™ Now You Can Cite Bibliographic References From Within Your Word Processor Program Did you ever wish you could cite references as you typed on your keyboard? Did you ever wish you could create perfectly formatted bibliographies directly from within your word processor? With ProCite 3.0 for Window's new "Cite While ' You Write" you can easily cite references and produce bibliographies from within your word processing program. As you create your docu- ment in your Windows word processor, simply identify the selected text in your document with one click of your mouse and ProCite 3.0 will automatically link that text to the appropriate reference in your database. When you're finished, ProCite 3.0 will automatically generate a fully formatted bibliography in the output style you select. No more wasted and unproductive time searching for references. No more mistaken attribution or citations. ProCite 3.0 for Windows ensures that in-text references and appended bibliographies reflect the citations you want. Every time. Automatically! ProCite offers a convenient way to manage, distribute, and print bibliographic information. ProCite handles the difficult layout and formatting of records, letting you concentrate on the task at hand. Experience the power and versatility of ProCite, the leading bibliographic management software program. The only bibliographic reference management software solution with "Cite While You Write." Dt.••f .... Hr.adea::F ebrutry27,19&9 Vet-W.adlcade•:44 IN•W.adlklldla:D4 •ace<•):l46 lSSN: Nea: ~::!;7or~':=d6-o• WI onbnt 14rrict Wllo ProC1lt w:~n~ AMtrwt: A bncfdetCI!fltiorl ofthtrelalaft•mltofunnsdmaactllnciiON to Callf"fu~Mr. .c.,...JW:Bihllo-ltnkii/ChuYul [!] •