College and Research Libraries CAROLE WEISS MOORE User Reactions to Online Catalogs: An Exploratory Study Designers of online library catalogs can benefit from the experience of the first libraries to test public access systems. In this study, use of four such online catalogs was observed. Success-failure rates were compared and user opinions analyzed. Results were consistent in all systems: user reaction was overwhelm- ingly favorable compared to manual catalogs, and improved subject access was considered the greatest need. Several common problems emerged in the display and access systems. BACKGROUND Libraries planning to move to an online catalog in the 1980s hope to use the opportu- nity to improve upon existing public catalogs. The basic needs of the library's clientele are now being reexamined: what bibliographic in- formation is required, and how information can best be presented and accessed. U nfortu- nately, much of the information available at this stage to designers of new systems amounts to little more than speculation. User-oriented interface has been a common goal of bibliographic retrieval systems design for more than a decade. The scope of the problem, however, is only beginning to be understood. 1 Among college and university libraries only a few are close to the goal of replacing their manual catalogs with online systems that allow the public direct access. In this study, an exploratory survey was conducted at four institutions to observe user reaction to their fledging online catalogs, and to determine de- ~irable characteristics for design of such catalogs. Carole Weiss Moore is head, Reference Depart- ment, and acting coordinator, Bibliographic Pro- cessing Department, University of Toronto Library, . Ontario, Canada. Research for this article was sup- ported by a Social Sciences and Humanities Re- search Council (Canada) grant. METHODOLOGY The libraries surveyed included those at Ohio State University, University ofToronto, Guelph University, and Ryerson Polytechni- cal Institute.* Athough the systems and state of development at each library differed, it was thought that valuable comparative data could be gathered by using consistent question- naires and interviewing techniques at all in- stitutions. The interviewer/observer recorded the purpose of the catalog search, access terms used, time spent, and success/failure. Following the search the user was asked to fill out a brief questionnaire, rating the online catalog compared to other types of catalogs previously used, commenting on a list of de- sirable qualities for catalogs, and giving per- sonal information about his or her field and level of study. It was hoped that this approach would pro- duce two results. First, enough statistical data would be gathered to indicate the general ap- proach of users to online catalogs and their success or failure. The latter could then be compared to previous use studies of other types of catalogs. Second, the user evaluation section could . provide valuable information *Many other libraries were contacted and pro- vided extremely useful information regarding their thinking and planning on the subject, but these four kindly agreed to let a sample of their online catalog users be studied for comparative purposes. I 295 296 I College & Research Libraries • july 1981 TABLE 1 USES OF PUBLIC ONLINE CATALOGS : LIBRARY A AND B Library A B Known-Item 30 (57%) 36 (53%) *Searches conducted using title file . Subject 16 (30%) 14* (20%) about feelings and opinions of users while on- line systems are still in the design stage. The exploratory nature of this study must be emphasized. A relatively small sample at each library was used to determine if use of online catalogs related to use of older systems as shown in previous studies. This pilot study could also isolate key areas for more extensive research. It must also be noted that none of the in- stitutions studied claims to offer its users an online catalog. In all four cases the systems were designed for circulation and other pur- poses, and are only offered to users as additional sources to manual public catalogs. Two have had regular public access to the online file for more than two years, but two allowed direct public access only for purposes of this study. All have most of their collections included in the online database, but none were completely .listed at the time of study. Not one advocates that users go to the online source as a substitute for the manual catalog. On the contrary, the limitations are well ad- vertised and known by many online users; however, as shown by the study, a large majority of users found the online catalogs, even in their present, rather crude forms, more convenient than the complete manual catalog. RESULTS In the two cases that provide regular public access, primary inter~st was what the public online catalogs were used for. Both showed similar patterns, i.e., the majority of use was for known-item bibliographic searches (table 1). The greater use of circulation functions in library B may be attributed to the wide range of functions available to the public user in this system. However, substantial use of tradi- tional catalog functions was made in both cases. Of particular interest were the number of subject searches, since in library B, no sub- ject access points were available and in library Browsing! Learning System 4 (8%) Circulation Information 4 ( 8%) 18 (26%) Total 53 68 A, subject access to the collection was ex- tremely limited. In the other two libraries circulation fea- tures were not yet available for public use. Since these samples were drawn on a random basis from users of the manual catalogs, the type of online search corresponded to what users intended to look for in a more tradi- tional catalog. The results corresponded to a previous user study of the manual catalog done at library C (table 2). The differing use of known-item and subject searches reflects the needs of quite different user populations. Library C D TABLE 2 UsE OF PuBLIC ONLINE CATALOG : LIBRARY C AND D Known-Item 41 (82%) 12 (34%) Subject 9 (18%) 23 (66%) Total 50 35 Access points chosen by users were similar in the first two libraries having regular public access to online catalog information (table 3). Note that the 20 percent subject searches in library B (table 1) were included in the title access column, since no real subject access was provided. In libraries C and D, where public use of online catalog information was available only for purposes of this study, users demonstrated different choices for access. Most of the known-item searches were at- tempted first by author, while subject searches were frequently carried out under the title index even when subject term access was available (table 4). Although new users tended to try author access more than experi- enced users, several of both user groups commented that title access appeared to be more efficient for searching these online sys- tems than author access, especially for com- mon names. The learning process for adapt- ing search strategy to the system appeared to be very quick. The success-failure rate for searches done Reactions to Online Catalogs I 297 TABLE 3 ACCESS POINTS USED : LIBRARY A AND B Library A B Author 13 (23%) 16 (22%) *Not possible in this system. Title 19 (33%) 37 (51%) Author/ Title 8 (14%) * Subject 13 (23%) * Call Number 4 (7%) 10 (14%) Bar Code * 9 (13%) Total 57 72 TABLE 4 ACCESS POINTS USED : LIBRARY C AND D Library Author Title c 30 (51%) 28 (47%) D 13 (29%) 10 (22%) •Not possible in this system. using all four online systems was comparable tn the 60-85 percent range found in previous manual-catalog studies . 2 Considering the many variables among libraries, the success rate was remarkably similar for three libraries (table 5) . Library B's high overall rate may be partly attributed to the inclusion of circula- tion inquiries , nearly all of which were suc- cessful. The figures broken down by type of search show that known-item searches had a higher success rate than subject searches (ta- ble 6). This result has not been typical of manual-catalog studies. 3 Success in these figures was determined by the user after searching as much as he considered worth- while . No attempt was made to check the database for the user to determine if items had been missed. The high success rate for both types of searches is somewhat surprising considering the incompleteness of online bases in relation to manual sources. The sub- ject results are especially provocative since the success rate does not appear to relate to the provision of subject file. TABLE 5 SUCCESS-F AlLURE RATE FOR SEARC HES Percentage of Percentage of Successful U nsU<:cessful Library Searches Searches A 75 25 B 86 14 c 73 27 D 73 27 Only 7 percent of all users indicated they would look further by checking manual catalogs, while 4 percent said they would try further by browsing on the shelves. Call Subj ect Numbe r Total * 1 (2%) 59 22 (49%) 0 (0%) 45 Time spent per search was difficult to com- pare since two of the systems were used en- tirely by first-time users, but the pattern was similar to that of manual catalog searches: most searches were quick checks and a few , mainly subject searches , were quite lengthy (table 7). What was notable was the speed with which many experienced users carried out a known-item check (figure 1). The per-item average for both library A and B appeared to be less than one minute. The median times may appear slower in some cases than in some manual-catalog studies. 4 It must be remem- bered, however, that all access points may be searched from one online terminal , and fre- quently a number of items were checked. In most manual catalogs one must physically move from one area to another to search dif- ferent items; thus, the search time reported was based on mainly single-item searches. Users' ratings and comments provided con- siderable useful data. In rating ease of use, a large majority of users of the established sys- tems A and B rated them easy to use. The majority of first-time users testing systems C and D rated them as fairly easy , but many considered them easy even for the first use . Only eight users of all systems combined con- sidered them to be less than " easy or fairly easy." In all systems the online catalog was the preferred choice of users over other forms of catalog they had used . The second and third choices varied considerably and appeared to depend on the relative merits of the local ver- sion users had encountered (table 8). Users were asked to make subjective com- 298 I College & Research Libraries • july 1981 TABLE 6 SUCCESS -FAIL URE RATE BY TYPE OF SEARCH Known -Item Subject Library Pe rce nt Successful A B c D 74 87 75 78 *Searched in title fil e since no subj ect fil e was available . ments in response to a list of ten qualities. · This list of suggested qualities desirable for catalogs was based on previous research by Sigfried Treu. 5 1. Simplicity (clarity) 2. Order (file arrangement) 3. Completeness (comprehensiveness) 4. Association (connectedness) 5. Accessibility (convenient access) 6 . Responsiveness (prompt reaction) 7. Control (manageability) 8 . Versatility (variety in modes of access) 9. Reliability (confidence) 10. Support (assistance on demand) Comments were analyzed for content and for positive, neutral, or negative reaction. A majority of users rated all four systems posi- tively in all ten areas . The relatively few critical comments were helpful, however, and were frequently repeated with regard to all of the systems. Library A B c D TABLE 7 MINUTES SPE NT PER SEARC H Median Low 5 Tess than 1 2 less than 1 5 1 8 2 High 30 23 20 30 Simplicity was rated very highly, although all systems' users mentioned the need to be shown how to use the system the first time or to have better instruction sheets. Some Pe rce nt Unsuccessful 26 13 25 22 Pe rcent Successful 71 70* 75* 70 Perce nt Unsu ccess ful 29 30 25 30 users in all systems had problems extricat- ing themselves from mistakes and backtrack- ing in desired directions. Although most users felt satisfied with the order of the on- line systems, there were some notable ex- ceptions. The words of one user summarize the general state of file order : " Clear mostly , but bizarre at times. " Incompleteness, i.e. , some library hold- ings not included in the online file, was commented upon by approximately one- third of the experienced users in libraries A and B. On the other hand, nearly all users judged the test systems C and D to be complete based on their one experience . Experienced users at libraries A and B had a surprisingly good understanding of what was included in the online file, based on library publicity and instruction pro- grams. Without such information it appears to be difficult for users to judge the com- pleteness of a file. Again, experienced users commented more with regard to association (connected- ness) of the file. Most criticism related to problems in subject files or in trying to use title as a subject file. Only one user men- tioned a problem of disorientation at not having titles come up in alphabetical order in an author search. Under accessibility, lack of sufficient ter- minals and lineups during busy times was mentioned frequently ; however, convenient access was rated highly in other respects . Responsiveness was the most highly rated TABLE 8 PREFERRED FORM OF CATALOG (1 BEST) (A VERAGED RANKINGS) Library Card Microfich e Microfilm Book Online A 2.4 4.1 3.9 3.2 1.3 B 2. 1 2.6 4.1. 4.0 1.5 c 3.2 2.8 ·1.9 4.1 1.5 D 3.3 1.8 3.4 4.4 1.3 ~ 0 F s E A R c H E s 0 F s E A R c H E s 30 Library A ~ Library B MINUTES Reactions to Online Catalogs I 299 25 Library C MINUTES Library D ~ Fig. 1 Time per Search 300 I College & Research Libraries • july 1981 quality for all systems. A few users com- plained about computer downtime, but most frequent were comments such as "unsurpassed" and "instantaneous." Problems with control were mentioned more frequently in the test systems, where all were new users, than in libraries A and B. Nearly all systems elicited requests for good instructional information. Except for frequent requests for better sub- ject access, all four systems were judged ver- satile by nearly all users. Reliability was also considered generally very good, although ex- perienced users found some errors, particu- larly in foreign-language materials . All sys- tems experienced some downtime during the survey, but very few users mentioned that factor in their comments. Nearly all users were satisfied with the level of support, machine and human. The survey sample, selected on a random basis from actual catalog users, included a representative group of undergraduates, graduates, faculty, library staff, others, and a cross section of the various disciplines within the ·humanities, social sciences, and sciences. The analysis did not reveal sig- nificant differences in use or reaction by any particular group, with two exceptions. Fac- ulty and graduate students appeared to be using the subject approach less than others, and users in science fields appeared to use title access more than other disciplines. Al- though the sizes of these subgroups were too small to be statistically reliable , these results would seem to confirm similar evidence from studies directed toward this question. The general comments of the users were surprisingly consistent. Most were favorable ones: "convenient," "great," "a timesaver," "great because you don't have to run all over the place," "works perfectly." "Add subject access" or "improve subject access" were by far the most repeated suggestions . Many requested a keyword subject approach. One user summarized: "It doesn't allow a specific subject search- would be a world-beater if it did." Next most frequently mentioned were suggestions for making the online file com- plete (adding retrospective material and/or journal information) so that double-checking manual files could be eliminated. Many users suggested more or improved instructional programs in how to use the system efficiently. Several wanted more terminals and access from other locations. Some suggested adding periodical articles and abstracts. Others thought commands could be simplified to one key or letter and that Boolean search capabilities should be added. Several users of the test systems in librar- . ies C and D felt that inability to type would limit use of the online catalog. k few also feared machine breakdown and expressed concern about expense of the new system. It was interesting to note that none of these three concerns was expressed in libraries A and B, where users were already accus- tomed to public online access. Requests for more complete bibliographic information about individual items were conspicuous by their absence, although only one of the systems provided as much biblio- graphic information as is normally provided on a Library of Congress catalog card. Other systems provided abbreviated author, title , imprint, and location information only. CONCLUSIONS Based on user reactions observed in this study, it appears that libraries are on the right track in provision of online public catalogs. The online catalogs examined did not seem overly complex for the infrequent user, as most commercially offered databases have proven. Neither did these catalogs lead to the frustrations of the limited-access systems cur- rently available in videotext services. Author and title · access in these online catalog systems, although not perfect, did not present problems for users as significant as those caused by lack of specific subject access and lack of retrospective data. The question of what is needed for good subject access is par- ticularly intriguing and one that requires fur- ther study. More specific subject headings were most frequently requested, but some users expressed a need for broader headings than those in the Library of Congress Subject Headings list. Boolean search capaqilities of- fered on keywords from the title, corporate headings, and Library of Congress headings have been suggested and seem an attractive possibility. Such a system needs testing to de- termine if users would find it more successful. The need for greater subject access varies considerably among the different libraries' user populations . The total of subject searches for all libraries in this study was 30 percent. A similar study conducted at the Li- brary of Congress revealed that approxi- mately 70 percent of its users wished to ap- proach their catalog by subject. 6 More academic users might search by subject if they had a more useful subject access system. However , the cost of providing such im- proved access is substantial, and although de- sirable, will be justified in the present eco- nomic climate only if needed by a large pro- portion of library users. The need for inclusion of retrospective data or other information not included in the database (e.g. , reserve items, journals) was heard wherever such material was lacking. This is another expression of the desire of the user to have one place to search for library holdings. Some librarians have attributed this desire to laziness of modern users or a decline in scholarly methods . A user's expectation of increased conven- ience is not surprising, however, given the present technological environment. In any case, it is evident that what is not found in the first place a user looks is often not found or used at all. 7 Thus , parts of a collection not included in a new online catalog will be ig- nored by most users and will be an aggrava- tion to the conscientious ones who remember to search further. Some common problems were observed in the current display and access systems . The major difficulty centered around the question of what to display on the first screen the user sees after inputting his search request. Somehow a balance must be reached between- the number of items shown and the amount of bibliographic and circulation information displayed per item. The number of items displayed in the sys- tems examined varied from one item to fourteen. Either extreme meant looking at several additional screens for most searches before the desired information was found. A compromise somewhere between these two · meant that many more searches could be ended with fewer steps. No doubt reason- able compromises will be found as libraries Reactions to Online Catalogs I 301 gain more experience with such systems as used by their particular clientele.* Even in the most efficient systems observed, there was a tendency for users to skip the added step needed to find exact shelf information, and instead to find a call number and chance going directly to the shelves. Although typing ability appeared to have little effect on use of the online catalog, punctuation and spacing, if significant for in- formation retrieval, was a serious problem for new users . Inexperienced users tended not to input commas and spaces until ad- vised to do so. On the other hand, the same users consistently used initial articles when searching by title . Although all systems were designed to disregard initial articles, three showed frequent retrieval problems in this regard. Inexperienced users also fre- quently tried to correct errors or end their searches by pressing "clear," "rub out, " or "erase" keys , sometimes causing undesirable results such as throwing the user off the sys- tem completely. Detailed planning for on- line catalogs should involve a study of the keyboard to be used and elimination of un- necessary and confusing keys. Filing difficulties were apparent in all sys- tems for voluminous personal authors and corporate authors and common serial titles. These entries have always created filing problems, but may appear worse when sev- eral are displayed on a screen together , rather than viewed one by one in a card catalog . Sophisticated Boolean search capabilities might alleviate the problems eventually, but no doubt these will be messy areas in catalogs for some time. Perhaps the most important conclusion to be drawn from users' written responses as well as observed reactions was that all four online systems , rough , incomplete, and im- perfect as they were, were welcomed over- whelmingly by most library users . They *Libraries that have designed user-interface systems since the four examined in this study have benefited from previous experience. The li- braries of Northwestern University, Dartmouth College , Lister Hill, University of Chicago, Uni- . versity of Waterloo, and the National Library of Canada offer good examples of recently designed online systems that were not possible to include . in this user survey. 302 I College & Research Libraries • July 1981 were able to use online catalogs with mini- mal instruction, with as much success or more in finding items as when using other forms of catalogs. Moreover, users felt the online catalog was a tremendous improve- ment in convenience. In the words of one user, the online system is "the best way I kriow to find books easily." REFEREN CES l. Charles M. Goldstein and William H. Ford, "The User-Cordial Interface," Online Review 2:269 (1978). 2. F. W. Lancaster, " Studies of Catalog Use ," in his Measurement and Evaluation of Library Services (Washington , D .C. : Information Re- sources Press, 1977), p.19-72. 3. Ibid. 4. Katherine Packer , " Microcatalogue User Study, " University of Toronto, Faculty of Li- I' , r' ,. brary Science (unpublished data, 1979). 5. Sigfried Treu , "A Conceptual Framework for the Searcher-System Interface, " in Donald E . Walker, ed., Interactive Bibliographic Search: The User/Computer Interface (Montvale , N .J.: AFIPS Press , 1971), p.53-66. 6. Sarah Pritchard , " Library of Congress SCORPIO User Survey" (unpublished data, 1979). 7. Lancaster, " Studies of Catalog Use ," p . 70.