Novotny.indd I Don’t Think I Click: A Protocol Analysis Study of Use of a Library Online Catalog in the Internet Age Eric Novotny A protocol analysis study was undertaken to explore how a Web-savvy generation of library users searches the online catalog. Eighteen users, including experienced and novice searchers, were recruited. Partici- pants agreed to be recorded and to express their thoughts aloud while searching. Analysis of these data has revealed several distinct trends. Most notable among these trends has been the impact of Internet search engines on user expectations. Given the influence of the Web, these as- sumptions are likely common throughout higher education.This research reveals where bottlenecks occur and provides insights into how libraries can design systems that help users around trouble spots. nline public access catalogs (OPACs) were the subject of intense study in the 1980s and 1990s, but interest in these basic library tools tapered off as libraries added other online resources.1 Meanwhile, a new generation of Web- savvy users has arrived on campus, po- tentially rendering obsolete the findings of earlier research. A protocol analysis study was undertaken to explore how current library users think about and search the online catalog. The goal of the study was to address questions such as: What expectations or assumptions do our users have? What aspects of li- brary catalogs do users find confusing, surprising, or frustrating? How does the performance of novice users compare to that of experienced users? Analysis of the study results has revealed several distinct trends. Most notable among these trends has been the impact of Internet search engines on user strategies and expectations. Given the widespread influence of the Web, these assumptions and tendencies are likely common throughout higher education. This research reveals where bo lenecks occur and provides insights into how libraries can design systems that help users around trouble spots. Related re- search findings are discussed throughout this article to integrate our study into the broader context of library and informa- tion science research. Eric Novotny is the Humanities Librarian at the Pennsylvania State University Libraries; e-mail: ecn1@psu.edu. The author wishes to thank the members of the Protocol Analysis Group, Pamela Czapla, Henry Piscio a, and Linda Struble, and the members of the Assessment Task Force, especially co-chair, Laurie Probst 525 mailto:ecn1@psu.edu 526 College & Research Libraries November 2004 Methodology In 2001, the Pennsylvania State University Libraries migrated to a new commercial online catalog offered by SIRSI and lo- cally dubbed The CAT. Penn State made substantial modifications to the default interface offered by SIRSI. Based on an earlier in-house study of user prefer- ences, it was decided to create a relatively complex initial search screen. Users of Penn State’s catalog are presented with multiple search boxes and limiter options. (See figure 1.) To study the effectiveness of the design choices made by Penn State, an assessment group was created with the following goals: • to solicit feedback from users of The CAT; • to determine if the interface design supports basic searching; • to identify search obstacles and causes; and • to gain insight into the mental models of users. Three groups were formed to explore these issues. One group surveyed users of The CAT to determine satisfaction with the new system. A second group explored using focus groups. The findings of the third group are the focus of this paper. This group was charged with looking at how our users actually searched The CAT. The main goal of the group was to determine whether the library’s users were able to effectively conduct typical searches in the online catalog. To study this, the research team developed five tasks for participants. The tasks were selected to represent a range of activities commonly performed by library users, including finding a book by title, finding a journal by title, and searching for mate- rials on a subject. The five tasks required different search skills in order to minimize the learning effect on later tasks. The first FIGURE 1 The CAT Initial Search Screen I Don’t Think I Click 527 tasks were deemed to be relatively easy. The complete task list is provided in the appendix to this article. Eighteen participants agreed to be ob- served while completing the tasks. They also agreed to talk aloud while working. The protocol analysis, or “think aloud,” method was chosen because the research- ers wanted to explore not only how users searched the catalog, but also what they were thinking and feeling as they did so. This approach complements studies done using transaction log analysis that tell how users search naturally but do not reveal motivations or a itudes. Participants were expected to continue talking throughout the session. This running commentary provided a richer set of information for analysis than simply recording key- strokes. Students were asked to complete the task as naturally as possible and were instructed to give up when they normally would in a real-world situation. Despite the research team’s efforts, the nature of being observed probably led some partici- pants to persist in their queries longer than they might otherwise have done.2 In determining the number of ses- sions, the group was influenced by Jakob Nielsen, who argues that a sample size as small as five users can identify up to 85 percent of the problems in a system.3 Although the research team found this ar- gument persuasive, it eventually opted for a more conservative approach, using nine volunteers for each group. At the comple- tion of the project, the team was convinced that this number of participants was suf- ficient to identify obstacles and trends. The pa erns became clear a er only a few sessions. Each subsequent session largely reinforced earlier findings, adding only the occasional bit of new information. To compare experienced and first-time users, two distinct groups were recruited. Nine novice volunteers were recruited from LEAP, a summer program for first- year students. The students were all newly admi ed to the university and had no prior experience with the library’s online cata- log. These sessions were conducted in a consultation lab during the month of June 2002. A second group, which comprised experienced searchers, was recruited separately and observed in September 2002. The second group included three upper-level undergraduates, four gradu- ate/doctoral students, one faculty member, and one part-time, nondegree student. All had prior experience using The CAT. Seven of the nine had used the library catalog at least ten times in the year preceding the study. The tasks were the same for both novice and experienced users. Data Gathered Data from the sessions were collected using audio recordings, screen capture so ware, and investigator notes. Camtasia so ware was used to record the audio and screen activity (such as mouse clicks). One or more investigators were present to encourage the students to think aloud and to take notes. The presence of human note- takers proved useful as technical failures prevented the capture of complete data in some cases. A er completion of the tasks, participants were interviewed to explore ambiguous actions and to solicit additional feedback about the online catalog. The team did find it helpful to gather some statistical information. The time to complete a task was recorded. The comple- tion time establishes a benchmark against which the team can measure progress in future studies. (See figure 2.) As changes are made to the interface, the average time on task should be reduced. As participants worked through the task, the observers assigned a “successful,” “partially suc- cessful,” “fail,” or “search aborted” rating. The aggregate success ratings for each task provided a quick summary of the difficulty of the task. http:reduced.As 528 College & Research Libraries FIGURE 2 Average Time on Task All times rounded to the nearest minute Novice Users (LEAP Students) Task 1 Approx. 3 minutes (range 1–5 minutes) Task 2 Approx. 2 minutes (range 1–3 minutes) Task 3 Approx. 5 minutes (range 4–6 minutes) Task 4 Approx. 10 minutes (range 4–16 minutes) Task 5 Approx. 8 minutes (range 5–15 minutes) Average total time: Approximately 28 minutes Experienced CAT Users Task 1 Approx. 3 minutes (range 1–6 minutes) Task 2 Approx. 2 minutes (range 1–7 minutes)** Task 3 Approx. 4 minutes (range 1–10 minutes) Task 4 Approx. 5 minutes (range 2–12 minutes) Task 5 Approx. 10 minutes (range 3–15 minutes) Average total time: Approximately 24 minutes ** Most were done in 1–2 minutes; higher times were caused by authentication difficulties Basic demographic data were gathered, including university status and level of prior experience with The CAT. Regre a- bly, the team did not formally collect user satisfaction ratings. It would have been instructive to discover if any correlation existed between time, or success rates, and satisfaction with the catalog. The observers were occasionally surprised at how posi- tively some participants rated the catalog. Novice users, some of whom struggled to successfully complete three of the five tasks, typically remarked that the system was “easy to use.” This may be the result of low expectations, a desire to please the investigators, or some other factor. The results discussed in this paper are based on a detailed analysis of all of the sources of information. The audiotapes, screen capture files, and investigator notes were reviewed multiple times for broad pa erns. These were disseminated among the investigators who discussed the findings and reached a consensus November 2004 on the conclusions. Behaviors that were unique to one or two participants or specific to Penn State’s environ- ment have been omit- ted from this report. General Findings and Observations This section presents findings relevant to both experienced and novice users. Despite many differences, the t wo g r o u p s s h a r e d some tendencies. Us- ers in both categories were relatively un- sophisticated subject searchers. Task 5 was designed to measure this. Users were asked to find four books on “Efforts to combat teen smoking.” As anticipated, most of the participants (14 of 18) used “teen smoking” as their initial search terms. It is interesting that none of the users en- tered the phrase “Efforts to combat teen smoking” or a empted to enter the search as a question. Participants in this study demonstrated an ability to convert topic phrases into keyword search statements. Unfortunately, this approach resulted in only two records. The participants then faced the task of refining their search to produce the required four books. In their quest to find two additional books, participants made minimal use of Boolean operators. Penn State’s library catalog is designed to encourage sophis- ticated searches. There are multiple search boxes and drop-boxes where users can select “and,” “or,” or “not” to combine fields. Despite this, the majority of us- ers preferred to stick with phrases (teen smoking). They entered multiple search I Don’t Think I Click 529 terms on a single line without any con- necting Boolean operator. Previous library catalog and Internet studies report similar findings regarding the use of Boolean.4 A review of Excite search engine use found that only eight percent used Boolean.5 A er observing the majority of catalog users fail in subject queries, Charles R. Hildreth stated that “it is time to put end- user Boolean retrieval systems, including OPACs, behind us …. Boolean retrieval has fundamental shortcomings and is not a design model that represents users’ information-seeking behavior.”6 Without judging the merits of Boolean-oriented search engines, it is clear from this and other studies that library users do not in- corporate such operators in their searches. The team did not formally discuss this issue with participants, so it remains un- clear whether they were aware of Boolean operators and chose not to use them or whether the absence of such search strate- gies was due to a lack of knowledge. In addition to shunning Boolean, participants demonstrated an inability to repair searches using synonyms. Task 5 was designed to foster the use of related terms. Teenagers, adolescents, or youth could have been substituted for teen, and smokers, cigare es, or tobacco could have been used in place of smoking. Only the sole faculty participant tried related terms. The more common approach a er retrieving insufficient results was to select another search type, add another word, or eliminate a term (e.g., dropping teen and just searching smoking). These results are similar to the findings of a transaction log study reported at the 8th Annual Reference Research Forum. The analysis at New Mexico State University revealed that users who retrieve small result sets most o en respond by switching search types. Instead of rethinking their search terms, they changed to another type of search. The next most common tactic was to retype the search.7 The study participants exhib- ited similar behaviors. The first inclination was o en to change the search type. If this did not work, many participants were in- clined to broaden the search by subtracting terms. Participants appeared to intuitively understand the concept of broadening or narrowing a search. They did not typically think to substitute other, potentially more effective search terms. Finally, both experienced and novice users demonstrated a somewhat hazy conception of what is in a library catalog. Many users expected to find individual articles in the catalog. Some performed journal title searches thinking they were looking for articles by title words. More problematic is that users showed minimal curiosity about the inner workings of the catalog. Although they were occasionally puzzled or frustrated, they rarely took the time to explore why the results were unsatisfactory. Instead, they were far more likely to try another search than to examine the reasons why the initial search failed. Again, the problem does not appear to be new or unique to Penn State users. In 1996, Christine L. Borgman observed that online catalogs were designed for skilled searchers who had developed a set of con- ceptual tools to use the system effectively. She found that most catalog users were not inclined to explore the system and lacked understanding of what was going on behind the scenes. This combination of factors le many users unable to select the most appropriate search options or effectively utilize the catalog.8 Studies have demonstrated consistently that users are not inclined to become expert searchers. As Patricia Wallace noted, users want to accomplish their task in the library catalog as quickly as possible.9 Even before the widespread adoption of the Web, Bar- bara Valentine noted that students tend to look for the quickest and easiest way to complete an assignment.10 The Web http:assignment.10 530 College & Research Libraries likely reinforces this tendency. A new dis- incentive to learning is the fact that many searchers believe they are already profi- cient information seekers. As experienced Web searchers, many new students enter college confident in their own ability to locate information.11 Although education will always play a role, it seems clear that a significant portion of library users does not know—or care about—the intricacies of library catalogs. We can either abandon this population or design systems that do not require expert knowledge to be used effectively. Novice Users: Observations It is difficult to convey in print the im- patience and occasionally near-random search habits exhibited by novice users. As a rule, the first-year students were not care- ful about perusing all the elements of the screen. The typical novice searcher scrolled quickly, noticing one or two things at the bo om and top of the screen. He or she did not ordinarily take the time to review all the options but, instead, selected the first remotely appropriate link that caught the eye. This a itude is exemplified by one participant who was asked in the follow- up interview why he chose one option over another. His response aptly summarized the actions of many of his cohorts, “I don’t think I click.” By this, he meant that he did not pause to consider all his options but, instead, selected the first link he noticed that appeared relevant. Novice users displayed similar tenden- cies toward hasty decision making when evaluating the results of their searches. A single screen of matches was all most users would peruse. A study of Internet search engine users revealed similar tendencies and documented a downward trend in the percentage of searchers willing to view more than one page. In 1997, 28.6 percent of Excite users studied confined themselves to one page of results. This November 2004 number increased to 42 percent of search- ers in only two years.12 It was very rare for a novice user to browse past the initial set of matches. In some cases, this was because the results were clearly irrelevant; in others, a match might have been found with more persistence. The tendency was to examine only the first few results to determine whether a search was effective. This inclination was fueled, in part, by the common assumption that the results of a search would be relevancy ranked, as they are on many Internet search engines. The novice users, all fresh out of high school, demonstrated an admirable will- ingness to try new things. These users were perfectly willing to explore The CAT and to click on various search options. They were not concerned about “crashing the system” as an earlier generation of computer users might be. This fearlessness, or recklessness, led novice users to travel down many false trails. Their searches were conducted rap- idly and were just as easily discarded when the results were deemed unsatisfactory. In general, the novice users spent less time thinking about constructing the perfect search and more time trying different things. As an example, several novice us- ers tried an option labeled Quick Search. When interviewed later, they admi ed having no idea what this search did; they were simply trying something different Experienced Users: Observations As a group, experienced users were more effective and efficient searchers than their novice cohorts. On average, they spent less time per task and were more likely to complete tasks successfully. Somewhat surprisingly, this relative success cannot be a ributed to knowledge of advanced search strategies. Keyword searching remained the favorite option, even for known items. There was minimal use of Boolean and no demonstrated under- standing of LC subject headings. http:years.12 http:users.As http:information.11 I Don’t Think I Click 531 The factors contributing to success in this study were relatively pedestrian and have implications for instruction and screen design. Although experienced users utilized keyword searches as o en as novice users did, they were somewhat more specific. They changed the default options, conducting slightly more focused searches (e.g., keyword title instead of keyword anywhere). This subtle change led to manageable search results with fi y matches, instead of thousands. Another distinguishing factor was persistence. Ex- perienced users were much more likely to stick with a search to its logical conclusion. They carefully reviewed the search results, and if a match was present, they found it. Novice users frequently changed searches if the match was not apparent. Occasion- ally, this could cause them to discard a search set even when it contained the item they were looking for, albeit “bur- ied” beneath other seemingly irrelevant results. The persistence of experienced users should not be exaggerated. A review of library catalog studies conducted in 2001 found that the vast majority of OPAC users viewed fewer than fi y matches.13 In this study, two screens (40 records) ap- peared to be the maximum. A study of Internet use suggests how experience impacts searching styles. Researchers found that searchers with limited experience struggled with the In- ternet regardless of age. A lack of Internet experience affected user persistence and o en led to quick abandonment of the In- ternet as an information source.14 A similar dynamic may be at work in the library cat- alog. The novice users in this study were decidedly more willing to abandon their searches, occasionally to their detriment. The age differences of the two groups also may be a factor, although one of the few studies to explicitly study the subject con- cludes that older users encounter similar difficulties using the catalog as younger adults, including conceptual difficulties, constructing effective keyword searches, and Boolean operators.15 The prior knowledge of experienced users proved invaluable. This group was already familiar with the campus and the structure of the libraries. This was helpful when encountering things such as abbreviated locations. These designa- tions made no sense to novice users and thus caused confusion. Experienced users also benefited from their knowledge of library terms and jargon. They typically knew what a call number was and knew about library services such as “I Want It.” They recognized information terms such as periodicals and generally knew what a journal was and how it was represented in a library catalog. This knowledge helped them avoid obstacles common to novice users, such as adding the publication date to a journal search. As a result, they spent less time pursuing unproductive options and were less overwhelmed than the nov- ice users, for whom everything was new. What Users Expect: The Impact of the Internet Student use of the Web has been the subject of much research. Not surprisingly, these studies have documented the pervasive influence of the Internet. A Pew Founda- tion study notes that all of the students surveyed had used computers by the time they were eighteen years old, with almost half using the Internet before they enrolled in college.16 This generation of students finds the Web easy to use. An OCLC survey reported that three out of four students agree that they are successful in finding school-related information on the Web.17 Given this background, it is not surpris- ing that many users in this study expected the library catalog to function as an Inter- net search engine. They typed in broad keyword searches and expected that the ”computer” would interpret their search http:college.16 http:operators.15 http:source.14 http:matches.13 532 College & Research Libraries and process the results. This may explain the lack of persistence in reviewing screens. The expectation was that a match should appear in the first few records. Many expressed confusion when seemingly ir- relevant records appeared ahead of the item they were looking for. Few appeared to notice that records were sorted chrono- logically. These findings clearly support giving library catalog users the option to relevancy-rank results of keyword search- es, in addition to more traditional means of sorting results by author, title, or date. Faith in the computer is a possible explanation for the strategy of adding keywords to searches when the goal was to find *more* records. A significant minority of users added a term to their initial search when trying to find additional books on “teen smoking.” The mental model may have been expressed by one participant. A er locating two matches for “teen smok- ing,” she typed a third term in, wondering aloud, “Does the computer need more information to make a match?” Although others did not express this as explicitly, the underlying thought process seemed to be the same: the more information you give the computer, the be er results you will get. This may work when searching billions of Internet sites, but it is an ineffective strategy in library catalogs operating on strict key- word matching and Boolean logic. Probably the most obvious effect of the Internet was what the Penn State team dubbed the “lure of the links.” This described the a raction that hyperlinks seemed to hold, especially for the younger, mostly novice users. These users almost invariably focused their a ention on the fields in the catalog that were hyperlinked. In some cases, this approach had beneficial results, clicking on LC subject headings to get related records, for example. In other contexts the links proved a distrac- tion. The focus on links caused them to overlook other important elements of the November 2004 screen, including bibliographic details, li- brary holdings, and shelf locations. Screen designers will need to contemplate how to format information in the catalog record so that important details are noticed by a generation that is drawn to hyperlinks. How Our Users Would Improve Library Catalogs Participants were not shy about offering verbal and wri en suggestions. Many of the services described below are already offered by libraries and library vendors, but it is worth documenting their impor- tance to users. Summaries and Tables of Contents When present, summaries were highly valued. Users quickly keyed on these along with chapter headings as key pieces of information. Generally, these were read aloud and used to assess the relevance of a title. Author, publication date, and other bibliographic details were not as o en used in evaluating a record. These findings are consistent with earlier studies that found users examined only a few fields when con- sidering the relevance of a match. A 2000 study by David H. Thomas found that over 75 percent of study participants concentrat- ed their efforts on two fields: the title and subject fields. In the same study, summaries were used by participants over 70 percent of the time they were available.18 Relevancy Rankings and Exact Title Matching Users were puzzled by the order of search results. A common comment was “Why didn’t the exact match of what I typed in come up first?” Users clearly wanted the computer to sort the results by relevancy as most Internet search engines do. Sev- eral opined that an “exact title” feature would be helpful, presumably unaware that this already was available by using the Browse option in The CAT. http:available.18 I Don’t Think I Click 533 “Find Related” Several participants commented that a “Find related” or “Find more like this” bu on would be helpful. Amazon.com was mentioned as a model Web site that enables users to locate similar items. Giv- en the problem with synonyms discussed earlier, such a feature would ideally go beyond the listing of LC headings but also would suggest related search terms. The Research Libraries Group is working on a prototype for a system that links users to related content.19 Context-sensitive and Interactive Help In “10 Usability Heuristics,” Jakob Neilsen includes the commandment “Help users, recognize, diagnose, and recover from errors.”20 On the other hand, library researchers have long noted that many patrons do not use online help options.21 This may be because the type of assistance being offered does not match the user’s needs. Although few users in this study employed the static FAQ or Help pages, several expressed a desire afterwards for more context-sensitive help that was visually interesting. The team’s research indicates that users struggle with repair- ing searches that result in very large or very small search sets. These moments represent opportunities for instruction. There is reason to believe that timely in- terventions can improve search results. A transaction log study that analyzed library help screens found that “implementing changes… resulted in changed searching behaviors and improved user success.”22 Conclusion and Directions for Further Research The results of this study indicate that li- brary catalog users are heavily influenced by trends in Web searching. This is particu- larly evident in the newest users—novice undergraduates just entering academia. This group demonstrated a tendency for rapid-fire searching that included exten- sive use of hyperlinks. Both experienced and novice users adopted search strate- gies more appropriate to Google than to a library catalog. In general, the study par- ticipants took an unsophisticated approach to computer searching. They expected the library catalog to function as a search engine, where one types words into a box, and gets results. The standard strategy, even when looking for a known item, was to use the default keyword option. Partici- pants demonstrated minimal knowledge of how information is structured in a library catalog and how that underlying structure affects their searches. More important, they showed no curiosity about how the catalog worked nor did they feel they should. Although evident, the impact of the Internet should not be overly emphasized as an obstacle to using the online catalog. Myriad earlier studies demonstrate that library users have always valued speed and ease of use in an online catalog. Re- searchers do not always have the time or inclination to learn every aspect of the information tools at their disposal. Most utilize the Principle of Least Effort, which states that “most researchers (even serious scholars) will tend to choose easily avail- able information sources, even when they are objectively of low quality and, further, will tend to be satisfied with whatever can be found easily in preference to pursuing higher-quality sources whose use would require a greater expenditure of effort.”23 Of greater concern are the fundamental conceptual difficulties. Participants clearly lacked an accurate understanding of the capabilities of a library online catalog. Knowledge of the underlying structure of the catalog (fields, authority files, etc.) was not apparent in this group of users. Ad- vanced features such as Boolean operators or authority files were li le utilized. The conceptual difficulties with syn- onyms observed in this study merit http:options.21 http:content.19 http:Amazon.com 534 College & Research Libraries further investigation. The ability to sub- stitute related concepts is critical in mak- ing effective use of most online library catalogs, yet it does not appear to be a skill that even experienced users possess. The profession will need to determine why library users lack this skill and the appropriate responses. The answer may lie in education. As a result of this study, the author has placed greater emphasis in his instruction sessions on how to incorporate synonyms into a search and, just as important, why one would want to do such a thing. Alternatively, it may be that we need to design future systems that do not assume this ability. Such systems could automatically offer a thesaurus of related terms, especially in cases where users retrieve relatively low numbers of matches. Although the study focused on inter- face design issues, some findings emerged that have implications for public service providers. A few specific examples are provided based on the author’s experi- ences as a reference and instructional librarian. It is likely that readers with a public service background will already have formed some conclusions based on their own service environment and users. The finding that most broadly im- pacted the author’s instruction and refer- ence was learning that even experienced users lack a full understanding of what they are doing when they search a library catalog. Although experienced users, on average, performed be er than novices in this study, they were not expert searchers of the Penn State library catalog. They employed inefficient strategies and occa- sionally expressed misconceptions about the nature of the catalog (e.g., that it con- tained journal articles or that the most rel- evant items would be placed first). A er reviewing the study results, it no longer seems safe to assume that an experienced November 2004 user is also an advanced user. Where the author may have assumed that minimal instruction was needed, he now offers more advice. As noted earlier, library catalog users do not typically think to question their re- sults or examine the inner workings of the system. In instructional and reference set- tings, the author has a empted to convey to students why they might occasionally want to stop and smell the bibliographic roses. During an instructional session, this may involve asking students questions that require them to focus on specific parts of the screen (e.g., date of publication). A discussion would then follow on why those fields are included. The aim is to help users understand the function of what seems to them to be an unnecessar- ily complicated display. This also requires them to pause and examine the screens they are presented with. Another approach the author is ex- perimenting with is that of asking the students in an instructional session to per- form selected tasks on their own before any lecture takes place. The tasks serve as starting points for a general discussion of The CAT. This approach to instruction moves away from teaching users which bu on to push and toward enabling users to understand the structure and purpose of a library catalog. Our major instruc- tional efforts must focus on improving the user’s understanding of what a library catalog is and what it is not This article is far from the last word on user mental models. Many issues re- quire further exploration, including the potential impact of library instruction efforts on use of the library catalog. The research team did not ask about prior BI experience in this study. It would be valu- able to learn if instruction can change user mental models and allow users to make more efficient use of library catalogs as they are presently designed. This study I Don’t Think I Click 535 only examined in a limited way other factors potentially affecting success, such as age and academic experience. All of the novice users were under twenty years old and new to the campus, whereas the experienced users were older and had been on campus a longer period of time. It is possible that the differences between groups may be explained by these influ- ences as much as by experience with The CAT. A study that looks at academic users with similar catalog experiences across age groups could tease out the influence of these factors on searching. It is important that library catalog ven- dors be included in the conversation. Li- brarians can continue to report study results, but unless this information reaches the companies that design the systems, changes will not be forthcoming. Toward that end, the principal investiga- tors recently discussed their study at the 2004 SIRSI SuperConference. A ended by hundreds of SIRSI users, as well as many company representatives, the conference presented a rare opportunity for librar- ians and vendors to speak directly to one another about areas of mutual concern. It is hoped that other researchers will pursue similar venues to reinforce the importance of their findings with key decision makers. The lengthy literature on library catalogs suggests that the problem is a long-standing one that defies simple explanations. An effective solution re- quires collaboration among all involved. Conversations with vendors and other stakeholders will be enhanced if librarians are willing to concede that they are at least partly responsible for the difficulties users encounter with library catalogs. Vendors design systems based on what they believe their customers value. Many already offer features advocated in this article, such as relevancy ranking and enhanced tables of contents. The more difficult struggle is ge ing library staff to agree to change. As information professionals, librarians pos- sess conceptual tools and searching skills that are far beyond those of the majority of their patrons. For experienced staff, it can be difficult to truly remember what it is like to be a novice user. We are used to catalog systems the way they are. Librarians are sophisticated users who take full advantage of the features the catalog offers and sometimes resist anything seen as “dumbing down” the library catalog. If a library catalog vendor developed a search interface as simplistic as Google or Amazon, librarians would immediately demand enhancements: format and date limiters, ISBN/ISSN searching, the ability to combine multiple search fields, and so on. There are valid reasons for wanting a catalog with these searching features, but the addition of more and more options can lead to library catalogs that many users find intimidat- ing and confusing. Librarians will need to decide if they are willing to make the trade-offs required for a simpler “Google- like” search system. Only if we are will- ing to answer yes, can we then demand that vendors develop such a system. Librarians and information scientists have been researching online catalogs for more than three decades, yet a lament u ered in 1996 still rings true: “Despite the advances in technology, research continues to show that these systems are ineffective and hard to use.”24 This will likely remain the case until librarians know more about how their users search. More important, librarians will have to accept the notion of building systems and interfaces that truly match how end users approach the task of searching the library catalog. If we continue to change the ap- pearance without touching the functional- ity, it is likely that another researcher will be conducting a study in ten or twenty years to determine why online catalogs remain difficult to use. 536 College & Research Libraries November 2004 Notes 1. For a review of research, see Andrew Large and Jamshid Beheshti, “OPACs: A Research Review,” Library and Information Science Research 19 no. 2 (1997): 111–33. 2. For studies using protocol analysis in user interface design, see Susan Wiedenbeck, Robin Lampert, and Jean Scholtz, “Using Protocol Analysis to Study the User Interface,” Bulletin of the American Society for Information Science 15, no.5 (June/July 1989): 25–26; Heather G. Morrison, “Online Catalogue Research and the Verbal Protocol Method,” Library Hi Tech 17, no.2 (1999): 197–206; Carol A. Hert, “User Goals on an Online Public Access Catalog,” Journal of the American Society for Information Science 47, no.7 (1996): 504–18. 3. Jakob Nielsen, “Why You Only Need to Test With 5 Users,” Alertbox (Mar. 19, 2000). Avail- able online at h p://www.useit.com/alertbox/20000319.html. [Cited 9 February 2004]. 4. Thomas A. Peters, “When Smart People Fail—An Analysis of the Transaction Log of an Online Library Catalog,” Journal of Academic Librarianship 15 (1989): 573–92; Yee, Martha M., “System Design and Cataloging Meet the User: User interfaces to Online Public Access Catalogs,” Journal of the American Society for Information Science 42 no.2 (1991): 78–98. 5. Amanda Spink, Dietmar Wolfram, Bernard J. Jansen, and Te o Saracevic, “Searching the Web: The Public and Their Queries,” Journal of the American Society for Information Science 53 no.2 (2001): 226–34. 6. Charles R. Hildreth, “The Use and Understanding of Keyword Searching in a University Online Catalog,” Information Technology and Libraries (June 1997): 61. 7. Nancy B. Turner and Susan E. Beck, “Search and Rescue: Repair Strategies of Remote Us- ers Searching the Online Catalog,” Eighth Annual Reference Research Forum, American Library Association Conference, Atlanta, June 16, 2002. 8. Christine L. Borgman, “Why are Online Catalogs Still Hard to Use?” Journal of the American Society for Information Science 47, no.7 (1996): 493–503. 9. Patricia Wallace, “How Do Patrons Search the Online Catalog When No One’s Looking?” RQ 33, no.2 (winter 1993): 239–52. 10. Barbara Valentine, “Undergraduate Research Behavior: Using Focus Groups to Generate Theory,” Journal of Academic Librarianship 19, no.5 (1993): 300-304. 11. John Lubans Jr., “When Students Hit the Surf”, School Library Journal 45, no. 9 (Sept. 1999): 146– 12. Dietmar Wolfram, Amanda Spink, Bernard J. Jansen, and Te o Saracevic, “Vox Populi: The Public Searching of the Web,” Journal of the American Society for Information Science and Technology 52, no. 12 (2001): 1073–74. 13. Bernard J. Jansen and Udo Pooch, “A Review of Web Searching Studies and a Framework for Future Research,” Journal of the American Society for Information Science and Technology 52, no.3 (2001): 235–46. 14. Debra J. Slone, “Internet Search Approaches: The Influence of Age, Search Goals, and Experience,” Library & Information Science Research 25, no.4 (2003): 403–18. 15. Richard A. Sit, “Online Library Catalog Search Performance by Older Adult Users,” Library and Information Science Research 20, no.2 (1998): 115–31. 16. Steve Jones, The Internet Goes to College (Pew Internet and American Life Project, 2002. Available online at h p://www.pewinternet.org/reports/pdfs/PIP_College_Report.pdf. [Cited 9 February 2004]. 17. OCLC, White Paper on the Information Habits of College Students: How Academic Librarians Can Influence Student’s Web-Based Information Choices (2002). Available online at h p://www5.oclc. org/downloads/community/informationhabits.pdf. [Cited 9 February 2004]. 18. David H. Thomas, “The Effect of Interface Design on Item Selection in an Online Catalog,” Library Resources and Technical Services 45, no. 1 (Jan.2001): 20–46. 19. Research Libraries Group, “RedLightGreen,”2004. Available online at h p://www.red- lightgreen.com. [Cited 9 February 2004]. 20. Nielsen, “10 Usability Heuristics.” Available online at h p://www.useit.com/papers/heu- ristic/heuristic_list.html. [Cited 9 February 2004]. 21. Jennifer Schwartz, “Internet Access and End-user Needs: Computer Use in an Academic Library,” Reference and User Services Quarterly 41, no. 3 (2002): 253–62. 22. Deborah D. Blecic et al., “Using Transaction Log Analysis to Improve OPAC Retrieval Results,” College and Research Libraries 59, no.1 (Jan. 1998): 39–49. 23. Thomas Mann, Library Research Models: A Guide to Classification (New York: Oxford Univ. Pr., 1993): 91–101. As cited in Schwartz, “Internet Access and End-user Needs,” 254. 24. Borgman, “Why Are Online Catalogs Still Hard to Use?” 501. http:lightgreen.com I Don’t Think I Click 537 APPENDIX Protocol Analysis Tasks The participants in the study worked on the tasks below. They were asked to continue until they had answered the question to their own satisfaction or were out of ideas. The rationales were for internal use only and did not appear on the sheets given to participants. A Protocol Analysis Study of the Library’s Online Catalog Protocol Analysis Tasks 1. A friend recommends that you read a book called History by Hollywood. Does the Penn State Libraries have this book? Question Rationale: To examine how users search for known items and whether they could locate a book when they knew the title. The title was chosen so that the broadest keyword search resulted in fewer than fi y matches. 2. Please use the library catalog to ask for a copy of the book History by Hollywood. Question Rationale: This task was designed to see if users could determine how to use the “I Want It” bu on in The CAT to request materials. Analysis of e-mail queries revealed that many users were unable to determine how to request materials. 3. The article you need is in the journal Kansas Law Review, volume 49, issue 5, June 2001. Is this issue of the journal in the Penn State libraries? If so, where is it? Question Rationale: A unique title was chosen so that users retrieved only one match regardless of which search type was selected (keyword or browse). The research team was interested in determining how users interpreted a serials record. 4. Another article you need is in a journal called Civil Engineering. Specifically, you need volume 70, December 2000. Is a copy for this year available on the University Park campus? If so, which library can it be found in? Question Rationale: To see how users navigated a potentially complex search. The default keyword search options results in over 2,000 matches. The record display includes multiple holdings, with the UP copy at the bo om of a very long record. 5. You have been assigned to write a paper on the topic “Efforts to combat teen smoking.” For this paper, you need to find four books at the Penn State Libraries. Please note the call numbers and locations of four books on this topic. Question Rationale: To see how users search for materials on a subject. The topic was constructed so as to make it likely that multiple strategies would be required. Teen smoking as a keyword only retrieves two of the necessary four matches. The LC sub- ject heading is Teenagers-United States-Tobacco-Use-Prevention, but other keywords would also work: teens, adolescents, smokers, and so on.