College and Research Libraries Application of a Methodology Analyzing User Frustration Gene K. Rinkel and Patricia McCandless A user satisfaction study was undertaken at the University of Illinois at Urbana-Champaign in an attempt to demonstrate and analyze an application of the Kantor branching analysis of user frustration in another environment. The impact of an automated circulation system with public access to use availability of materials was monitored as was direct user access and par- tially controlled access to materials. The paper also discusses satisfaction rates and causes of user frustration in relation to academic status and prior use experience. The results support the concept that a hierarchy of categories provides a useful methodology for analysis of user frus- tration, but the scope and definition of those categories must be environment specific. ecent library literature in- cludes a number of studies on library effectiveness consid- ered in relation to a given set of users and book availability that employ the techniques of operations research. 1 Various suggestions have been made re- garding ap1,ropriate types of performance measures. However, as yet there are no universally accepted standards or meth- ods for the study of library effectiveness. Capability indexes and satisfaction indexes of library effectiveness are among those measures gaining increasing acceptance for evaluating library performance. Some form of capability index appears to be the appropriate tool for collection evaluation and is especially valuable for selection or analysis of research collections. Satisfac- tion indexes tend to address the delivery of services to fulfill expressed demand. These measures are complementary in that improvement of immediate or ex- pressed demand for titles should also im- prove the long-range research values of a collection. In light of current budget re- strictions, further refinement of a satisfac- tion index could provide the basis for more efficient use of existing resourses. User satisfaction is a loosely defined con- cept in contemporary librarianship. It has frequently been qualified by terms such as immediate (time related) or by source (i.e., local versus interlibrary or network lend- ing implying length of loan, delivery time, etc.) or by complexity of the access process ranging from hands-on browsing, which allows almost instantaneous satisfaction of a newly discovered need, to reference assistance in providing either biblio- graphic information or specific facts by use of online retrieval services. Generally the term satisfaction index has been re- stricted to specific document requests as a measure of the percentage of biblio- graphic items immediately obtained or de- livered. Widely varying satisfaction rates have been documented in various size col- lections and environments. 4 Further ap- plication of this and other indexes in a wide variety of environments is needed to develop preciseness. Clearly, develop- Gene K. Rinkel is with the General Services Division, and Patricia McCandless is assistant to the director of public services, University of Illinois Library at Urbana-Champaign . At the time of the study the authors were circulation librarian and LCS coordinator for public services, respectively, University Libraries, University of Illinois at Urbana-Champaign . · 29 .----------------------------------------------------------------------- - ---- 30 College & Research Libraries January 1983 ment and applications of these measures are desirable if they are also to provide standards for comparability in evaluation of library services. Refinement of such measures would also benefit the evalua- tion of the delivery of library services through developing networks. RESEARCH DESIGN A promising and practical method for analyzing library effectiveness uses such a satisfaction index but extends it by allow- ing comparison of sources of user frustra- tion within and among a variety of library environments. The hierarchy proposed by P. B. Kantor and used at Case Western5 identified "parameters characterizing the causal factors which inhibit fatrons from finding the items they seek.'' Application of such a measure across a variety of li- braries and in the context of specific envi- ronments should not only contribute to- ward establishing standards for satisfaction levels but also identify further factors that if addressed would increase li- brary effectiveness. The present study seeks to demonstrate and analyze an application of that method in another environment. A modified ver- sion of the Kantor categories was applied to the general library stacks of the Univer- sity of Illinois at Urbana-Champaign (UIUC) in November 1980. The University of Illinois at Urbana-Champaign is a state- supported institution with an enrollment of 34,376 and a faculty of2, 176. The library system consists of thirty-five departmen- tal units, the largest being the general bookstacks with approximately four mil- lion volumes. An additional two million volumes are housed in other units on cam- pus, half of which are located in the main library building that houses the book- stacks. The bookstacks' collection repre- sents older materials transferred from other units and collection areas where no departmental unit exists. The bookstacks are closed except to faculty and graduate students, who are encouraged to enter and retrieve their own materials . Some undergraduates also have stacks access while working on special projects. As- signed study carrels in the stacks for fac- ulty and graduate students contribute to a sense of "open" stacks for those patrons. A desk paging service is provided for un- dergraduates and the general public. With the exception of several thousand items kept in a semi-rare area, all items in the bookstacks circulate: periodicals, serials, and microforms as well as books. The UIUC Library uses an online circu- lation system, Library Computer System, which, in addition to handling circulation functions, permits known-item searching7 with copy and circulation status available on any one of ninety-six LCS terminals lo- cated in departmental units around cam- pus. At the time of the study, five termi- nals in the card catalog area were for public use and patrons are encouraged to use LCS to check for location, circulation status, and availability. In this application, the addition of a vari- able Pbib to the four reported in the Case Western study8 was considered desirable. The researchers felt Pbib was warranted to identify the components of user failure while maintaining analysis of the logical sequence of a user's search for a given item. Later applications at Case Western have also included a similar variable in the hierarchy. 9 Pbib is a user variable with fail- ure to locate a copy attributed to not find- ing a call number for the item sought, fail- ure to report the call number correctly and/or completely, and the inability to de- termine the location of copies, i.e., failure to use the card catalog and/ or LCS compe- tently. Satisfaction in this application was de- fined as obtaining material within one hour for known-item requests whether by paging or direct access. Several major dif- ferences from the Case Western study should be noted: the larger collection at UIUC with some duplication in other li- brary units; the presence of an automated circulation system with public access for known-item searching to determine own- ership and copy and circulation status; the inclusion of all circulating materials, all se- rials and microforms as well as mono- graphs; and limited or controlled access for various users as opposed to an open collection. Furthermore, this study also examines satisfaction rates for specific user groups: faculty, graduate and under- graduate students, and courtesy borrow- ers. METHOD On fifteen randomly selected days in November 1980, representing a busy sea- son of the regular fall semester, a ques- tionnaire (see appendix A) was distrib- uted to users seeking specific library materials from the general bookstacks. The format of the questionnaire provided space for author and title of one to three specific items sought on the current visit of the user to the stacks. It also included factual questions providing information on whether the users looked, paged, re- quested a search, and obtained the item sought. Other questions provided infor- mation about the users and their approach and about the requested material. Al- though the unusual problems accompa- nying data collection by questionnaire were encountered, an attempt was made to minimize subjective user judgments and reduce problems of recall by asking only short factual questions regarding the present visit to the stacks. An excep- tion was a question requiring self- characterization of the user's borrowing experience during the current semester. The sample reflects two broad user groups coming to the general stacks for known items. One target population was primarily faculty, staff, and graduate stu- dents (many with study carrels within the stacks). A second environment sampled involved a distinctly undergraduate pop- ulation and courtesy borrowers who re- quest materials from the stacks through a paging service at the circulation desk. The samples were taken during a time of class term paper demand and included some queueing of those using the desk service. Both of these groups interact: some with stacks access also request paging service and searching. Pre-sample test trials resulted in design changes to the questionnaire and data col- lection procedures for the subsamples. Using past circulation figures, the re- searchers concluded that approximately two-thirds of the circulation was from us- ers with stacks access. A random sample was stratified proportionately between Application of a Methodology 31 users with stacks access and those using the desk paging service. Although the same questionnaire was distributed to both sample groups, it was systematically distributed to each tenth user entering the stacks on the sample days and to a user at the circulation desk when a randomly set beeper sounded. Three hundred thirty- nine or 63 percent of those with direct ac- cess accepted forms and of those 74.9 per- cent returned them. Sixty-two or 18.3 percent of those returning forms indicated they were using their study carrels or browsing but not seeking specific items. One hundred forty forms were distributed to those using the paging service with a 73 percent return rate of 102 forms. These samples were analyzed separately and in combination with use of the Statistical Package for the Social Sciences10 to provide a clearer delineation of the user population and the effects of paging versus stacks ac- cess. Frequencies and crosstabs of se- lected variables in relation to satisfaction were examined. The questionnaires were coded to allow analysis of each as a whole and individual requests regardless of how many were from one user. A combined sample of 509 cases was reported on 379 forms. All user groups were represented; library staff and repeat individuals were not excluded. Follow-up on each case was completed daily by the researchers, with clerical assistance, to determine the causes of fail- ure. Coding was done by. the authors and cross-checked to ensure reliability. FINDINGS Table 1 presents the results of the survey of satisfied and unsatisfied requests for li- brary materials at UIUC in November 1980. The notation is as follows: W = total number of requests. S = number of immediately satisfied re- quests. P = number of unsatisfied requests. These requests were classified into five categories: Pacq = number of unsatisfied requests attributable to acquisition failure (not ac- quired, not yet in database or card catalog, all copies withdrawn, specific edition not 32 College & Research Libraries January 1983 TABLE 1 RESULTS OF THE UIUC GENERAL BOOKSTACKS USER STUDY -NOVEMBER 1980 Paged I. Total number of re~uests 193 II. Number immediate y sat- isfied by a stacks copy 138 III . Distribution of unsahs- fied requests: A. Number of tiles not 55 acquired 4 B. NuJTiber of bibio- ~aphic errors 17 C. umber of items 15 D. Number of library malfunctions 11 E. Number of user errors 8 owned, unpublished). Pbib = number of unsatisfied requests attributable to bibliographic errors (call number not found, incorrect or incom- plete call number, location not stacks or copies available only in other units). Pcir = number of unsatisfied requests attributable to the item being in circulation (on loan to individual user, on reserve, on loan to interlibrary loan, on loan to an- other unit). Plib = number of unsatisfied requests attributable to library malfunction (error · in card catalog or LCS, housed in special location within stacks, paging failure, re- shelving delays, misshelved, missing af- ter search). Puser = number of unsatisfied requests attributable to user errors (orientation to stacks arrangement, special markings/lo- cation). In the paging sample, Puserrepre- sents library staff performance. Sample sizes were too small to analyze subcategories of reasons for failure. Figure 1 and table 2 present the satisfac- tion rates attributable to the potential sources of frustration with their standard error using the Kantor branching analysis. A test was made of the difference between the proportions satisfied in the subsam- ples paging and direct access, using formula 1 to weigh for sample size. Formula 2 was applied to provide normalized Z scores. The authors concluded that overall satis- faction rates represent a common popula- tion. Direct Access Combined 316 509 228 366 88 143 6 10 31 48 14 29 14 25 23 31 Pest N1p1 + N2p2 N p1-p2 Z = p(1-p)N N1 + N2 Symbol w s p Pacq Pbib Pcir Plib Puser [11 [21 Table 3 examines satisfaction rates by status as stratified by size of each status in the sample. The results show that users of the general bookstacks at UIUC are satis- fied at approximately the same level for all categories of borrowers. The relatively high satisfaction rates re- ported here may be attributed to several factors that merit further study. The size of the collection and its organizatidn into departmental units undoubtedly im- proved user satisfaction by providing a higher pr_oportion of multiple copies for frequently used materials. LCS as an ele- ment' in the context of Pbib may have sys- tematically screened Pcir requests from the sample while increasing overall satis- faction by helping to locate another avail- able copy. Since LCS shows missing cop- ies, copies that have been borrowed, and copies on reserve, in binding, etc., pa- trons using LCS learn the potential avail- ability of a sought item much sooner than they would have otherwise. Some patrons may not have requested or reported items they knew were in circulation or were housed in another unit, although at UIUC holds are frequently placed on copies in circulation. The questionnaire, as an in- strument, while less obtrusive than an in- Application of a Methodology 33 Paging Direct Access Puser 31 Puser 8 FIGURE 1 Satisfaction Rates TABLE2 T = S+ Puser U = T+ Plib V = U+ Pcir X= V+ Pbib W = X+ Pacq \ Pacq 10 MEASURES OF PERFORMANCE FOR THE GENERAL BOOKSTACKS FROM THE OUTCOME OF REQUESTS FOR MATERIALS-NOVEMBER 1980 Standard Error Direct Direct Paged Access Combined Paged Access Percent Percent Percent Percent Percent AcJuisitions Pac~ = XIW 97.93 98.10 98.04 1.02 .77 Bib iographic Pbi = VIX 91.00 90.00 90.38 2.08 1.70 Circulation Pcir = U/V 91.28 94.98 93.57 2.15 1.31 Library Plib = TIU 92.99 94.72 94.08 2.04 1.37 Users Puser = SIT 94.52 90.84 92.19 1.88 1.82 Satisfaction level 5/W 7.150 72.15 71.91 3.25 2.52 Combined Percent .61 1.32 1.15 1.15 1.35 1.99 r----------------------------------------------------------------------------- 34 College & Research Libraries January 1983 TABLE 3 TOTAL SATISFIED BY STATUS Undergrads Graduates Faculty Courtesy Borrowers Percent of Requests by Status 74.8 74.7 66.7 77.3 Percent of Status in Sample 40.4 39.1 10.1 10.5 Percent of Total Satisfied 40.9 39.3 8.7 11.1 terview, may have skewed the results: pa- trons unsure of their needs and/ or library skills may have declined to respond or did so only with identifiable "successes." A replication of this study using the inter- view technique might test the degree of selective response to the questionnaire. Consistent with other studies is the high performance of acquisitions, a finding that may suggest self-fulfillment rather than merely sound acquisition policies. Since this index is based on expressed demand for specific titles, it is difficult to isolate or control the effects of the collection itself as a bibliographic tool in creating demand. Future research should test the hypothe- sis that expressed demand for specific ti- tles is directly correlated with biblio- graphic access provided to items, regardless of collection size. The study also tested whether prior suc- cessful use and/ or patron status would have a positive effect on success as shown in tables 4 and 5. Assuming that more ex- perience in using the library would im- prove library skills, the authors hypothe- sized that extensive use would correspond with success rate. It was also hypothesized that undergraduates would be less successful than graduates and/or faculty because of limited bibliographic skills and knowledge of library proce- dures. Presumably, a faculty member who is a heavy library user should be more suc- cessful than the freshman or other user who had used the library only a few times. The results at UIUC, as exhibited in table 4, show significance at the .05 or better level where prior use improved skills in using the desk paging service. In table 4 the variable Puser, significant in the pag- ing sample at .0015, represents the inter- action of the user and the desk staff receiv- ing the request. Use by Pbib, significant at .0003, also reflects user and staff interac- tion. Table 5 shows crosstabs of status and sources of user frustration and demon- strates that while overall satisfaction rates were approximately the same for under- graduates, graduates, and faculty, the sources of frustration were not the same. Pacq, crosstabbed with status, was signifi- cant and the results show that graduates and faculty are more likely to be frustrated due to seeking a title not acquired. Under- graduates presumably find substitute ma- terials more often to meet their needs. Pbib was also significant in the paging sample and reflects undergraduate errors primar- ily in serial citations when requesting ma- terials. Pcir was significant at the .0115 level for the paging sample, again reflect- ing undergraduates' requests for the same items. Status by format of requested material was noteworthy. Undergraduates, repre- senting 40.4 percent of the sample, re- quested 48.4 percent of the periodicals and only 24.3 percent of serials, whereas grad- uate students (39 .1 percent of the sample) used more serials. Fifty-eight and a half percent of serial use was graduate and only 34.9 percent periodicals. When weighted by their percentage of the sam- ple, status showed approximately equal distribution in use of monographs. How- ever, the satisfaction rate was not mark- edly different between monographs and periodicals or serials in either subsample. The questionnaire technique did not permit analysis of bibliographic failure. Other studies have and should continue to explore the patron's use of the card cat- alog, and similar studies of LCS are needed to provide insight for those en- gaged in bibliographic instruction. The completeness and accuracy of the user's citation, his/her approach and skill in us- ing the card catalog, whether LCS com- mands were structured and typed accu- rately, and the user's interpretation of either the card or machine record are ques- tions that should be examined in detail. CONCLUSION The application of Kantor's branching analysis in this environment shows that Application of a Methodology 35 TABLE 4 CROSSTABS OF USE WITH SOURCES OF USER FRUSTRATION Use by Pacq Significance of X2 Cramer's V Use by Pbib Significance of X2 Cramer's V Use by Pcir Significance of X2 Cramer's V Use by Plib Significance of X2 Cramer's V Use by Puser Significance of X2 Cramer's V Paging N = 195 .5254 .12804 N = 190 .0003 .33153 N = 170 .8449 .09063 N = 154 .8961 .08407 N = 72 .0015 .49402 Direct Access N = 307 .6753 .08730 N = 296 .0653 .17280 N = 262 .9965 .02561 N = 245 .1505 .16581 N = 248 .1171 .17250 Combined N = 502 .6491 .70722 N = 486 .0020 .18632 N = 432 .7594 .06582 N = 399 .4998 .09173 N = 320 .0009 .24210 TABLE 5 CROSSTABS OF STATUS WITH SOURCES OF USER FRUSTRATION Paging Status ~ Pacq N = 195 Sign. icance of X2 .0000 Cramer's V .59652 Status ~ Pbib N = 190 Sign· icance of X2 .0204 Cramer's V .29535 Status bh Pcir N = 170 Signi icance of X2 .0115 Cramer's V .32632 Status ~ Plib N = 154 Sign· icance of X2 .8943 Cramer's V .12115 Status ~ Puser N= 72 Sign. icance of X2 .1443 Cramer's V .36443 the concept of a hierarchy of categories does provide a useful methodology for analysis of user frustration. However, the researchers have concluded that the con- tent and definition of these categories must be environment specific. In this en- vironment, for instance, the introduction of LCS, which provides circulation infor- mation on availability of copies, may have reduced the measured frustration rate or caused it to occur earlier in the search pro- cess. The content of subcategories, al- though logically related, cannot be fixed for every environment. The creation of categories is a conceptualization that can be modified to isolate specific factors for study. If the categories are narrowly de- Direct Access Combined N = 310 N = 505 .8805 .0263 .09917 .17728 N = 299 N = 489 .1969 .5720 .18156 .10822 N = 265 N = 435 .5608 .1837 .14821 .15228 N = 247 N = 401 .7007 .7995 .13744 .09769 N = 251 N = 323 .3649 .1577 .17453 .18104 fined operationally, they may be applied to each step in a search process. The limi- tations are primarily those of practical ap- plication. Costs of sample size and man- agement's uses of the results should provide guidelines for logical develop- ment of additional categories to be stud- ied. An overall performance measure in the form of a satisfaction index would clearly be useful in comparisons among similar li- braries or branches of a library. A measure of user satisfaction that identifies the sources of user frustration such as that used here allows administrators to target ~pecific factors or programs, which, if im- proved, would increase overall perform- ~--------------------~-------------------------------------------------~-~~-- ~ 36 College & Research Libraries January 1983 ance with a minimum investment. It is es- pecially meaningful when employed as a pre/post technique following administra- tive, organizational, or policy changes. 11 A potential advantage of the Kantor con- cept, while running counter to direct com- parability, is that it does allow further elaboration in identifying sources of frus- tration in specific environments. Al- though the addition of categories alters the distribution of individual indexes, which are then not directly comparable with other libraries, the overall satisfac- tion rate for a library would remain un- chang~d but redistributed to reflect rela- tive magnitudes in local sources of frustration. Following wider replication to determine appropriate subdivision of these variables, an agreed-upon set of cat- egories and/or subcategories could be-. come the basis for standardization. Practi- cal application of the technique in terms of sample sizes required, may dictate a lim- ited set of variables for standardized com- parisons, while individual libraries could further subdivide those areas showing high user frustration for in-depth analy- sis. REFERENCES 1. P.M. Morse, Library Effectiveness: A Systems Approach (Cambridge, Mass.: MIT Press, 1968). 2. F. W. Lancaster and Deanne McCutcheon, "Some Achievements and Limitations of Quantitative Procedures Applied to the Evaluation of Library Services," in Quantitative Measurement and Dy- namic Library Service (Phoenix : Oryx Pr., 1978), p.12-30. See especially p.22-26 on Document De- livery Studies. 3. Ibid . 4. D. Gore, "Let Them Eat Cake while Reading Catalog Cards : An Essay on the Availability Prob- lem," Library Journal 100:93-98 (15 Jan. 1975); Marjorie E. Murfin, "The Myth of Accessibility: Frustration and Failure in Retrieving Periodicals," Journal of Academic Librarianship 6:16-19 (March 1980). 5. T. Saracevic, W . M . Shaw, Jr., and P. B. Kantor, "Causes and Dynamics of User Frustration in an Academic Library," College & Research Libraries 38:8 (Jan. 1977). 6. Paul B. Kantor, "Demand-Adjusted Shelf Availability Parameters," Journal of Academic Librarian- ship 7:78 (May 1981). 7. James F. Corey and Lynne M. Blair, ''The Library Computer System at the University of Illinois,'' Illinois Libraries 60, no.4: 365-74 (April1978). 8. Saracevic, Shaw, and Kantor, "Causes and Dynamics," p.7. 9. W. M. Shaw, Jr., ' ' Longitudinal Studies of Book Availability, ' ' in Library Effectiveness : A State of the Art, Papers from a 1980 ALA Preconference (Chicago: American Library Assn., 1980), p.338-49. 10. Norman H. Nie and others, Statistical Package for the Social Sciences (2d ed.; New York : McGraw Hill, 1975). 11. Shaw, "Longitudinal Studies," p.338-49 . APPENDIX A UIUC GENERAL LIBRARY STACKS 1980/81 USER GROUP Please list up to three (3) books you are seeking on this trip to the library and check (x) the appropriate columns. Please list the author and title for each book you were seeking today even if you did not find the book in the system. 1. Author: Title: 2. Author: Title: 3. Author : Title: CALL NUMBER Location How did you find the call number? LCS CARD CAT OTHER How did you find the location? LCS CARD CAT OTHER Did you get help with? LCS CARD CAT Did you ask staff to? PAGE BOOK SEARCH Did you look for book yourself today? IN STACKS Did you get book? YES NO Application of a Methodology 37 1 2 3 1. What is your current status at the University of Illinois at Urbana-Champaign? Freshman __ Sophomore __ Junior __ Senior __ Grad Student __ Faculty __ Visitor __ Other __ 2. What is your Department? (Faculty and Graduate Students) ------------- OR Major? (Undergraduates)-------------------------- 3. Approximately how many books have you borrowed from the library (all departments) this semester? Fewer than 5 __ 5-10 __ 11-20 __21-30 __ More than 30 __ 4. As you leave the stacks today, please drop this form into box marked "USER STUDY" at the Circulation Desk or Exit to the Stacks. Your cooperation in completing this form will enable the library to improve service to users of the stacks. Thank you.