College and Research Libraries Book Availability: Academic Library Assessment Eugene S. Mitchell, Marie L. Radford, and Judith L. Hegg In the climate of increasing calls for academic assessment, the authors undertook a study to ascertain book availability in an academic library. The study de- scribed here uses the methodology pioneered by Tefko Saracevic, W. M. Shaw, Jr., and Paul Kantor and is a follow-up of earlier research reported in College & Research Libraries in 1987. The authors designed the study to identify any improvements in availability after the implementation of recommendations following the first study. The study provided a quantitative measure of library performance based upon the outcomes of card catalog searches. The research serves as a model for ongoing assessment in the library. • he 1980s could be character- ized as the decade of assess- ment for those involved with education in the United States. The new era probably began in April1983 when the National Commis- sion on Excellence in Education pub- lished Nation at Risk: The Imperative for Educational Reform.1 This document chronicling the myriad failures of the educational system served as the impe- tus for new initiatives in hundreds of political arenas and in all levels of edu- cation. The resulting call for accountabil- ity led to assessment efforts in colleges and universities from coast to coast. In a 1987 study, 25 percent of the institutions surveyed reported their state agencies were mandating assessment procedures while 70 percent without such require- ments were expecting to introduce some form of assessment soon. 2 Originally the term assessment focused on the measurement of student outcomes for the purpose of student development and institutional accountability.3 Many writers expanded its scope to include such activities as evaluation, program review, and accreditation.4 Using this wider understanding, the entire aca- demic institution in all its many inter- faces with the student becomes a possible focus for assessment proceed- ings. "Teaching, after all, is only one of the things university faculty do, only one of the activities into which institu- tions invest energy and resources.''5 This new academic introspection carries within it the seeds for a blossoming of institu- tional awareness. Assessment can be de- signed to serve a variety of purposes and evaluate a multitude of programs. The greatest service it can perform in each of these functions is to "identify aspects of performance where improvement is desirable."6 Assessment can transcend mere evaluation of present performance. It can become a vital agent for change, Eugene S. Mitchell is Director of Library Services at the Dr. Frank A. Franco Library Learning Center, Alvernia College, Reading, Pennsylvania 19607, Marie L. Radford is Head of Curriculum Materials at the Sarah Byrd Askew Library, William Paterson College, Wayne, New Jersey 07470, and Judith L. Hegg is former Head of Collection Development at the Sarah Byrd Askew Library. 47 48 College & Research Libraries providing an impetus for structuring a meaningful direction for growth. Within this context, the college library, as a significant component of the aca- demic community and its curricular goals, would appear to be an appro- priate agency for assessment.Yet except for an article by Thomas Shaughnessy determining library quality, no substan- tive work tying the current concerns and controversies in academic assessment to the campus library exists.7 The reason for this is unclear, but may rest solely on the fact that "there is no relatively straight- forward mechanism by which a library can demonstrate effectiveness."8 Using a book availability methodology pioneered by Paul Kantor, Tefko Saracevic, and others, and later modified by Anne Ciliberti, a group of librarians at a me- dium-sized public college library in New Jersey prepared a study to determine how that library could improve its service.9•10•11 The resulting study, designed around the notion that one of an academic library's major goals was to provide books for its patrons' curricular needs, reported in 1987 that 54 percent of the patrons surveyed found the materials they were seeking.12 Although these results mirrored those of other college libraries, the researchers sought improvement. The study design provided an opportunity to ascertain the stage in the process at which patrons had difficulty in locating or retrieving the desired books. The librarians made thir- teen recommendations to remedy library malfunctions and patron misunderstand- ings. The staff implemented these rec- ommendations during 1987 and 1988. The authors then undertook a follow-up study during the 1988 fall semester to assess whether any improvement in availability occurred. THE MODEL The model used in this study provides a quantitative measure of library per- formance based upon the outcomes of known-item and subject card catalog searches. A known-item search is one in which the patron is looking for a specific book and knows the author's name or the book's title or both. The measure of library January 1994 Start t-------Bibliographic error t-------Selection error 1------Catalog use error 1------Circulation error 1------Library malfunction 1------Retrieval error Success Start FIGUREl Branching Analysis of Known-Item Searches 1------Matched query error t------- Catalog use error t------ Circulation error t------ Library malfunction t------- Retrieval error t------- Appropriate titleenor Success FIGURE2 Branching Analysis of Subject Searches performance, expressed as a probability of success, is calculated both as an over- all measure of library performance and as a series of discrete, sequential steps which all successful patrons must complete. These calculations, therefore, provide im- portant diagnostic information about the relative strengths and weaknesses of li- brary subsystems such as selection, circu- lation, cataloging, and so forth. Figures 1 and 2 represent the sequential steps in- volved in known-item and subject searches, respectively. For each of these branches an independent performance measure is calculated. Definitions of Error Categories Selection and bibliographic errors occur only in known-item searches. Selection errors occur when the library has not purchased the desired material (earlier studies referred to these as acqui- sition errors). Selection errors also occur when the library does not fully represent the material in the card catalog when it is on order, in process, etc. Bibliographic er- rors occur when patrons do not find the desired materials because their biblio- graphic citations are incorrect. Two types of errors are present in sub- ject searches only. Appropriate title errors occur when patrons either fail to select titles found in the catalog or fail to borrow or use them in the library. Patrons may choose not to consult items found on their topics because they have already read the material, it is written in the wrong lan- guage, outdated, at an inappropriate read- ing level, or is in another way unsuitable to the information need. Matching errors involve matched query terms. A matched query term is one that either fully or partially agrees with the subject heading used in the cat- alog. When patrons fail to discover a subject heading that matches their query terms, a matched query error occurs. Matching errors may be of two kinds. They oceur when no match can be made from the initial query to a standard sub- ject heading because none exists in the alphabetical range of the query term. They also occur when the subject head- ing is not listed in the catalog. The final four types of errors occur in both known-item and subject searches. Catalog use errors occur when patrons cannot properly identify the call num- ber. Circulation errors occur when the desired material is on loan or on a "hold" shelf waiting to be charged out. Library malfunction errors are due to shortcom- ings in the policies or routines of the library or its staff. For example, desired items may be missing, misshelved, wait- ing to be reshelved, etc. The final type of Book Availability 49 error, retrieval, occurs when patrons can- not find the desired material although they identify the correct and complete call number and the book is in its proper shelf location. METHODOLOGY Data collection took place during the fall 1988 semester from card catalog users. The first patron approaching the card catalog at randomly selected times received a self-selecting data collection form. Of 137 forms distributed, 111 were returned for a rate of 81 percent. Each day, the authors collected the forms returned during the previous twenty-four hours and randomly selected one title from each form for an~lysis. If the patron had not found or used this title, the type of error was determined by systematically verifying the call number, subject heading, or bibliographical refer- ence; checking the card catalog; search- ing the book stacks and sorting shelves; and examining the circulation records and reserve book shelves. RESULTS A total of61 (55 percent) ofthe patrons conducted known-item searches while 50 (45 percent) conducted subject searches. These results were consistent with the 1986 study. Apparently, some patrons had difficulty understanding the concept of subject searching. For ex- ample, one patron was searching for The Autobiography of Cyrus McCormick, a known item, in the subject card catalog. Another patron was searching the sub- ject apartheid in the title catalog. This problem may have been exacerbated by the divided card catalog in the library under study. If patrons are not sure whether they are looking for a title, name, or subject, they may choose the wrong section of the card catalog. They also may lack the persistence or prob- lem-solving skills to look somewhere else or to ask for help when they find few books on a topic. Finally, patrons may have low expectations; when they do not find the books they want, they may not question it because they never expected to be successful. 50 College & Research Libraries TABLEt SUCCESS IN KNOWN-ITEM AND SUBJECT SEARCHES BY STUDY Success 1986 1989 Study Study (N=401) (N = 111) All searches 215 (54%) 71 (64%) Known-item searches 107 (50%) 40 (56%) Subject searches 108 (50%) 31 (44%) Analysis of the Success Rates The overall success rate for the 111 usable searches was 64 percent (see table 1). Although this represented a 10 per- cent improvement over the 1986 success rate of 54 percent, chi-square tests on the failure of known-item and subject searches by study and on the compari- son of success and failure by study both indicated that the improvement was not statistically significant at the .05level of confidence. Analysis of Search Failures As previously noted, the problems en- countered by patrons resulting in their failure to locate the books they seek can be divided into six categories for both subject and known-item searches. The data shown in tables 2 and 3 represent these error categories listed in the order in which patrons encounter them and illustrate the success rate at each step of the search process. Known-Item Search Failures. The data in table 2 show that two of the 61 patrons conducting a known-item search had erroneous bibliographic citations. Of the 59 persons who had correct biblio- graphic information, 4 were searching for titles the library had not purchased for the collection. Of the 55 who had accurate cita- tions and were looking for books the li- brary owned, one was unable to use the card catalog correctly, that is, to locate the appropriate card and identify information necessary to find the book. Another patron at the next step failed to find the book because it was in circulation. At this point 53 people were looking for titles that osten- January 1994 sibly should have been on the shelves. Seven of these were unsuccessful be- cause of some library malfunction-that is, the books were not in their expected location. Another 6 were unable to retrieve a volume shelved in the correct location. The total error rate was 34 percent. Placing these errors in order of relative negative impact on the search process, it is possible to assess the greatest needs for future library planning. The success rates of 87 percent at both the library malfunction and retrieval error stages of a patron's search were of primary con- cern. They were followed by selection, TABLE2 KNOWN-ITEM SEARCH PERFORMANCE BY TYPE OF ERROR AND SUCCESS RATIO No. Total of Patrons Type of Error Errors Searching Bibliographic 2 61 Selection 4 59 Catalog use 1 55 Circulation 54 Library malfunction 7 53 Retrieval 6 46 Total errors: 21. Total known-item searches: 61. % errors: 34%. Start Success Ratio(%) 97 93 98 98 87 87 1------- Bibliographic error 97% 1------ Selection error 93% 1------- Catalog use error 98% 1------- Circulation error 98% 1------ Library malfunction 87% 1------ Retrieval error 87% Success bibliographic, catalog use, and circula- tion concerns. Subject Search Failures. Table 3 also lists subject search errors in the order the patron encountered them. One patron of the 50 conducting subject searches was either seeking a subject for which the library had purchased no titles or was unable to select subject terms that matched his or her need. Of the 49 per- sons remaining, 2 had difficulty in using the card catalog. Either they could not correctly identify the call number or they left out the location symbol such as "Ref." Forty-seven patrons successfully reached this point, but 3 of these were looking for titles that were in circulation and therefore not accessible. Three pa- TABLE3 SUBJECT SEARCH PERFORMANCE BY TYPE OF ERROR AND SUCCESS RATIO No. of Tyee of Error Errors Matching and selection Catalog use 2 Circulation 3 Library malfunction 3 Retrieval 8 Appropriateness 2 Total errors: 19. Total subject searches: 50. % errors: 38%. Start Total Patrons Success Searching Ratio(%) 50 98 49 96 47 94 44 93 41 81 33 94 1-------- Matched query 98% error 1-------- Catalog use error 96% 1-------- Circulation error 94% 1------- Library 93% malfunction 1------- Retrieval error 81% 1------- Appropriate title 94% Success error Book Availability 51 trons of the 44 remaining were unable to locate their books because of a shortcom- ing in the policies or procedures of the library that caused the book to be un- available. Examples of library malfunc- tions include missing books, volumes waiting to be shelved, or books awaiting cataloging or repair. Forty-one patrons successfully negotiated these problem categories, but 8 more errors occurred because patrons could not find books correctly shelved in the stacks. Two patrons encountered the sixth type of error, appropriateness. These pa- trons found books on the shelf but decided that they were inappropriate for their needs. Thus, only 31 patrons performing subject searches located books appropriate to their needs. Nineteen were unsuccess- ful, resulting in a failure rate of 38 percent. Again, it is possible to place the patron errors in order of their negative impact on the search process. This order is retrieval error followed by library malfunction, appropriateness and circulation, catalog use, and matching and selection errors. Library Errors. The data in this study were further examined to evaluate the origin of the failures. Forty-seven per- cent of the search failures could be TABLE4 ORIGINS OF LIBRARY AND PATRON ERRORS Errors Orig!n of Error No. % Library errors Malfunction 10 (56) Circulation 4 (22) Selection 4 (22) Total 18 (100). Patron errors Retrieval 14 (70) Matching and catalog use 4 (20) Bibliographic 2 (10) Total 20 (100)+ ,. These 18library errors represent 47% of all search errors excluding those classified as approriateness errors. t These 20 patron errors represent 53% of all search errors excluding those classified as appropriateness errors. 52 College & Research Libraries considered library errors, or shortcom- ings in library routines. As table 4 indi- cates, 10 patrons (56 percent) failed because they could not locate the titles sought on the shelves or in the circula- tion records. An additional4 (22 percent) sought titles that were already on loan. The remaining 4 (22 percent) desired titles not owned by the library. Library Malfunction Errors. These er- rors constituted 56 percent of all library errors. In more than half of these cases patrons consulted the card catalog and found titles they determined to be useful but were unavailable. A closer look at these ten errors indica ted one was the result of a book being located on a sorting shelf, two were declared lost, and the remaining seven were unable to be located by library staff and were con- sidered lost. These seven may have been unavailable for a variety of reasons. For example, they may have been stolen, misshelved, or in staff offices but not checked out. Circulation Errors. Four (22 percent) of the failures resulted from the fact that titles were already on loan when the pa- tron searched for them. Selection Errors. Four (22 percent) of the library errors were selection errors. Patrons were searching for specific titles or books by a specific author that the library did not own. An analysis of these titles determined the extent to which they were compatible with the collection development goals of the library. One was clearly inappropriate for the collection, the second was a textbook considered outside the usual collection criteria, and the third had been purchased at the time of its pub- lication but could not be replaced when it was lost or stolen. The fourth title prob- ably should have been in the collection at the time of this study. Patron Errors. Over half the search failures were errors committed by the patrons. Of those thus identified, 14 (70 percent) occurred because patrons could not locate a title on the shelf when it was there. Another 4 (20 percent) failed either to use the card catalog correctly or inter- pret its contents accurately. The final 2 (10 percent) of the errors resulted from January 1994 erroneous bibliographic information bro1;1ght to the catalog by the patrons. Retrieval Errors. A disturbingly high percentage of the patron errors were re- trieval errors. A total of 14 (70 percent) of the 20 patron errors represented patrons who, while they had correct biblio- graphic and card catalog information, could not find books on the shelf even though the books were in their correct locations. Matching and Catalog Use Errors. A total of 4 (20 percent) patrons made matching and catalog use errors. These people were ineffective users of the card catalog. They experienced difficulty in gaining subject access to the catalog, in understanding the use of the call num- ber, and in differentiating between the various sections of the divided catalog. Bibliographic Errors. Judging from the small percentage (10 percent) of pa- trons who committed bibliographic errors, most patrons brought adequate biblio- graphic information to the card catalog. Other Sources of Error. In addition to library and patron errors, a third source of failure existed in subject searches, termed appropriateness errors. Whereas errors in the initial two categories typi- cally represented titles not available at the time of need, appropriateness errors occurred when patrons either failed to select titles found in the card catalog or, after examining selected titles at the shelf, decided not to use the books found. In contrast to the large numbers of library and patron errors surveyed, only 5 percent of the errors were appro- priateness errors. It was not possible from the available data to document the reasons why patrons did not select or use these books. If patrons had a better un- derstanding of the information con- tained on the catalog card, perhaps they could have distinguished inappropriate titles earlier. COMPARISON OF THE TWO STUDIES The purpose of this study was to fol- low up on the efforts of the 1986 study to determine if the implementation of the recommendations made any difference in the patrons' success at finding library books. Although it cannot be said with certainty that these changes helped in specific instances, the situation did im- prove with respect to most of the sources of error examined. A comparison of the performance of the library patrons between the 1986 and 1989 studies indicates that there was a large increase in the overall success rate from 54 percent to 64 percent. This in- crease is significant at the .06 level of confidence. Table 5 shows the differences in success rates between the two studies. With respect to known-item searches, success rates improved in four out of six error categories: selection (from 90 per- cent in 1986 to 93 percent in 1989), cata- log use (from 92 percent to 98 percent), circulation (from 91 percent to 98 per- cent), and library malfunction (from 74 percent to 87 percent). Success rates dropped in the bibliographic (98 percent to 97 percent) and retrieval (93 percent to 87 percent) error categories. In subject searches, the success rates improved in five out of six error categories: matching TABLES SUCCESS RATES BY TYPE OF ERRORFOREACHSTUDY Success Rates(%) 1986 1989 Type of Error Study Study Known-item searches Bibliographic 98 97 Selection 90 93 Catalog use 92 98 Circulation 91 98 Library malfunction 74 87 Retrieval 93 87 Subject searches Matching and selection 94 98 Catalog use 94 96 Circulation 93 94 Library malfunction 87 93 Retrieval 91 81 Appropriateness 89 94 Book Availability 53 and selection (from 94 percent to 98 per- cent), catalog use (from 94 percent to 96 percent), circulation (from 93 percent to 94 percent), library malfunction (from 87 percent to 93 percent), and appropriate- ness (from 89 percent to 94 percent). Again, retrieval error success rates dropped (from 91 percent to 81 percent). TABLE6 ORIGIN OF LIBRARY AND PATRON ERRORSFOREACHSTUDY Errors(%) 1986 1989 Type of Error Study · Study Library errors Malfunction 56 56 Circulation 25 22 Selection 21 22 Total 100 100 Patron errors Retrieval 32 70 Matching and catalog use 60 20 Bibliographic 8 10 Total 100 100 Table 6 displays a comparison of li- brary errors versus patron errors by study. The library error category shows approximately the same breakdown of errors between the two studies. The pa- tron errors, however, show retrieval failures to be a much greater problem in the 1989 study. Conversely, matching and catalog use success improved. The follow-up study revealed that there was an increase in overall success rate from 54 percent to 64 percent since 1986. With respect to known-item searches, success rates improved in the selection, catalog use, circulation, and library mal- function error categories. Success rates dropped in the retrieval and biblio- graphic error categories. With respect to subject searches, success rates improved in the matching and selection, catalog use, circulation, library malfunction, and appropriateness error categories. Success rates dropped in the retrieval 54 College & Research Libraries error category. Overall, retrieval errors were the greatest source of patron errors. RECOMMENDATIONS Besides its usefulness as a tool for assessment purposes, the book availabil- ity study can be used as a guide to im- provement. For example, the first study indicated that the overall success rate of card catalog users was 54 percent. An extensive list of recommendations was developed, giving priority to the largest causes of patron failure. The greatest causes of patron failure were library malfunctions, followed by appropriate- ness, retrieval, circulation, matching and selection, and catalog use errors. Among the recommendations for im- proving library malfunctions were the initiation of an inventory and regulariza- tion of shelf-reading programs. Recom- mendations for remedying circulation, patron, and selection errors included improving signs, purchasing duplicate copies of high demand items, and incor- porating discussions of patron retrieval and card catalog use problems into bibli- ographic instruction classes. In the second study, the major causes of patron errors were library malfunc- tion and retrieval problems, as seen in tables 2 and 3. With respect to retrieval errors, a continuing need was recog- nized to focus on the difficulty patrons appeared to have in locating a desired title when the book was on the shelf in its proper location. Despite the installa- tion of new signs after the 1986 study, it was recommended that improved and more creative signs be investigated. Other recommendations included the regular assignment of a staff member or student assistant at an information or help desk during the busiest hours of the semester and preprinted forms at the catalog with appropriate spaces for call number, title, and author to aid those students who do not have all the neces- sary information from the catalog card when they go to the shelves. The reverse side of the form would have a simplified floor plan of the library inqicating regu- lar shelves, sorting shelves, and special collection areas. January 1994 Recommendations to improve library instruction classes included enhanced discussions on the use and interpreta- tion of the card catalog, the arrangement of the books on the shelves, and the loca- tion of special collection areas. Discus- sions would also emphasize that patrons should persist in asking for help when they cannot locate books. Finally, the ref- erence librarians were urged to be alert to patrons who, in asking questions, re- veal a lack of understanding of the card catalog. With respect to library malfunc- tion errors, the conduct of an annual par- tial inventory was recommended. The effect of these new recommenda- tions remains to be seen. Shaughnessy stated that assessment "presents library managers with an opportunity to forus staff attention on service quality and library effectiveness." 13 The availability study de- scribed here provides both an opportunity and a mechanism to determine whether any benefits have accrued and the extent to which they may have had an impact. CONCLUSIONS The study as described has major impli- cations for assessment. This methodology provides a means for evaluation of one component part of a complex system. This work has focused on an important aspect of the academic library, that of providing access to library materials. The design of this study does not pro- vide a complete assessment of the my- riad variables involved in the evaluation of library services. Many areas were not addressed here (reference, staffing, fund- ing, etc.). This research tool could be em- ployed as one of several components if a broader assessment is desired. Another vital implication for this work is that the results of this and similar studies provide benchmarks for future investigations. One purpose of assess- ment is to evaluate present performance to determine the impact of change. Im- provements in performance and areas that continue to be problematic can be identified and monitored. Academic li- braries exist as dynamic entities, their form and function shifting in an increas- ingly computerized environment. The li- brary in this study is soon to convert its existing card catalog to an online cata- log. A possible future direction would be to modify the technique described above to assess the impact of the online catalog on the success this academic library has in providing the materials its patrons want and need. This analysis has provided valuable information and insights into the work- Book Availability 55 ings of the complex system of the aca- demic library. In undertaking and per- forming this evaluation, many benefits have resulted, including an increased awareness of patrons' major problems in interacting with the library organization and access points of materials. In order to promote productive use of the library, aca- demic institutions should include librar- ies in their assessment plans. REFERENCES 1. U.S. National Commission on Excellence in Education, A Nation at Risk: The Imperative for Educational Reform; A Report to the Nation and the Secretary of Education (Washington, D.C.: Government Printing Office, 1983). 2. J. Rossman and E. El-Khawas, Thinking about Assessment: Perspectives for Presidents and Chief Academic Officers (Washington, D.C.: American Council on Education and the American Association for Higher Education, 1987). 3. P. Gray, Achieving Assessment Goals Using Evaluation Techniques (San Francisco: Jossey- Bass, 1989). 4. B. Davis, "Demystifying Assessment: Learning from the Field of Evaluation," in P. Gray, Achieving Assessment Goals Using Evaluation Techniques (San Francisco: Jossey- Bass, 1989), 5-20. 5. _ S. Spangehl, "The Push to Assess," Change, 19 (Jan./Feb. 1987): 35-39. 6. G.R. Sell, "Making Assessment Work: A Synthesis and Future Directions," in P. Gray, Achieving Assessment Goals Using Evaluation Techniques (San Francisco: Jossey-Bass, 1989), 109-19. 7. T. Shaughnessy, "Assessing Library Effectiveness," Journal of Library Administration, 12 (1990): 1-8. 8. N. Brown, "Assessment Perspectives: How to Make the Case for Better Library Fund- ing," Library Administration and Management, 3 (Spring 1989): 80-83. 9. P. Kantor, "Availability Analysis," Journal of the American Society for Information Science, 27 (Sept./Oct. 1976): 311-19. 10. T. Saracevic, W. Shaw, and P. Kantor, "Causes and Dynamics of User Frustration in an Academic Library," College & Research Libraries, 38 (Jan. 1977): 7-18. 11. A. Ciliberti, The Development and Methodological Study of an Instrument for Measuring Material Availability in Libraries (Ann Arbor, Mich.: University Microfilms, 1985). 12. A. Ciliberti, M. Casserly, J. Hegg, and E. Mitchell, "Material Availability: A Study of Academic Library Performance," College & Research Libraries, 48 (1987): 513-27. 13. Shaughnessy, "Assessing Library Effectiveness," 2. You get one chance with authority control, so it's important to get it done right. LTI guarantees .that its affordable, machine-only authority control will link 95% or more of your library's controlled headings to an LC or LTI authority record. No exceptions! No excuses! When manual review is requested, only professional librarians are used as editors and link rates approach 100%. LTI maintains the complete LC MARC authority ftles (updated weekly), supplemented with over 410,000 LTI authority records and 350,000 proprietary "cross links." Contact LTI for more information on authority record link results. ~~uthority Control for the 21st Century,, • LIBRARY TECHNOLOGIES, INC. 1142E Bradfield Road Abington, PA 19001 (215) 576-6983 Fax: (215) 576-0137 (BOO) 795-9504