chisman.p65 552 College & Research Libraries November 1999 Usability Testing: A Case Study Janet Chisman, Karen Diller, and Sharon Walbridge Usability testing is a technique for identifying difficulty that individuals may have using a product. Usability tests of the Washington State Uni­ versity (WSU) Libraries’ online public access catalog (OPAC), Article Indexes, Full Text, More, and Other Library Catalogs sections revealed problem areas. A task force used test findings to recommend solutions that led to the participation of the working group involved in designing search screens, the libraries’ User Education Department, and Innova­ tive Interfaces Inc., the OPAC vendor. Solutions are currently being imple­ mented. ashington State University (WSU) is a multicampus insti­ tution with an Innovative In­ terfaces Inc. (III) integrated system that serves both WSU and East­ ern Washington University (EWU). This shared catalog creates considerable am­ biguity for its users. Even if the catalog were not shared with another institution, there is enough complexity within the WSU environment to create user confu­ sion, with six libraries on the Pullman campus and four extended campus librar­ ies located around the state. In January 1998, WSU implemented the Web-based version of the III OPAC. This OPAC, named Griffin, is used on all WSU campuses and at EWU. In the five months spent designing the Web OPAC (WebPac), it became evident that there were many opinions, but few data, on how it should look and work to make it easy to use. The III WebPac has enough flexibility that decisions regarding screen design, button text, order of information, and other usability questions were under debate. Some at WSU were concerned that decisions were being made based on “gut feelings,” anecdotal experiences, scat­ tered observations, and so on. What did the WSU team really know about user perceptions and understanding? During the design stage, the Septem­ ber/October 1997 issue of the OCLC Newsletter ran a series of short articles on usability testing at OCLC.1–3 This was es­ pecially timely as the institution struggled with customizing the WebPac. The idea of usability testing could go a long way toward providing information that could be used to make informed choices and decisions on WebPac design. Improved screen displays and help features, easier movement among screens, and clearer instructions could result in a more user- friendly catalog, empowering the user and freeing the librarian from dealing with the more routine questions about the catalog. Problems not within the librar­ ians’ direct control would be communi­ cated to III directly, to the Innovative Us­ ers Group, and to the libraries’ User Edu- Janet Chisman is the Systems Librarian at Washington State University Libraries-Pullman; e-mail: chisman@wsu.edu. Karen Diller is the Assistant Campus Librarian at Washington State University Li­ braries-Vancouver; e-mail: diller@vancouver.wsu.edu. Sharon Walbridge is the Assistant Director for Technical Services at Washington State University Libraries-Pullman; e-mail: walbridg@wsu.edu. 552 mailto:walbridg@wsu.edu mailto:diller@vancouver.wsu.edu mailto:chisman@wsu.edu Usability Testing: A Case Study 553 cation Department so that solutions could be worked on from a variety of fronts. Three interested librarians, two from the Pullman campus and one from the Vancouver campus, began serious discus­ sions of WebPac usability testing in Feb­ ruary 1998. The timing was fortuitous because one of the team members was planning to travel to OCLC for a confer­ ence. Arrangements were made to visit with OCLC Usability Lab Director Mike Prasse, who was very generous with his time. He offered a tour of the OCLC us­ ability lab, shared history of the OCLC effort and a description of the process, offered practical advice, and even allowed the visiting librarian to observe an actual test. Prior to the OCLC visit, the WSU team had conducted a literature search for ar­ ticles specifically on usability testing of library Web-based catalogs. Although the team did not find any articles on the topic, it did discover The Handbook of Usability Testing by Jeffrey Rubin, which became the team’s bible.4 Rubin’s book offered practical advice and helped lay out a road map to follow. By adapting the structure and concepts discussed by Rubin, it was possible to develop a purpose statement, a test plan, a task list, and the beginnings of a test. Drafts of those went to OCLC with the visiting librarian. Here again, Prasse was generous with his comments and suggestions, and noted that OCLC uses Joseph S. Dumas and Janice Redish’s Practical Guide to Usability Testing as a re­ source.5 Some of the important points that Prasse made were: • Limit the test to an hour. • Emphasize and reemphasize that it is the system that is being tested, not the person. • Keep the test limited and focused; don’t try to cover too much. It should be noted that the OCLC us­ ability lab is high tech. The WSU librar­ ies did not have access to such a facility nor the funds to develop one. The team looked for a low-tech alternative. Prasse had noted that videotaping the partici­ pant and the computer screen would be nice, but that a simple tape recording of the participant’s responses would be an acceptable bare-bones approach. The WSU team decided to have two persons observe each test participant and to tape- record his or her comments during the session. Purpose and Preparation The purpose of the usability testing was simple: to test how easily users could navigate the Web-based Griffin catalog and whether they understood what they were seeing. At Pullman, the team also wanted to test whether library patrons could find the electronic indexes and links to non-WSU library catalogs from the WSU Libraries Gateway shown in figure 1. One of the first steps was to develop a test set of questions for the Griffin test, which would be the first test. A pretest on the WebPac questions done by the Vancouver team member affirmed one of the most important things that Prasse had shared, namely, to reemphasize repeat­ edly that it is the system that is being tested and not the person. OCLC has this motto posted in large letters in its lab, and the team posted the same words in its test room. The pretest also helped the team refine some of the questions. Both literature and Prasse indicated that eight participants would identify 80 percent of the problems users might have with the system. The team devised a screening questionnaire to help identify a range of users based on gender, age, li­ brary, and computer experience. It de­ cided to do the initial Griffin test with eight participants on the Pullman cam­ pus, four novice computer/library users and four expert computer/library users, and four participants on the Vancouver campus. At Pullman, an advertisement for vol­ unteers was run in the student newspa­ per. The team felt that paying test partici­ pants $10 for their time would add legiti­ macy to the effort. The library adminis­ tration agreed to provide up to $200. The advertisement yielded twenty-nine ex­ 554 College & Research Libraries November 1999 FIGURE 1 WSU Libraries Gateway Used at WSU-Pullman Campus GRIFFIN Catalog I Article IndexeslFull Text & More I Other Libraries I What's New I Contact Us pressions of interest. The team decided to draw participants for the electronic in­ dexes/non-WSU catalogs test from this pool as well. At Vancouver, volunteers responded to flyers posted around cam­ pus and only the Griffin usability test was conducted. Each volunteer was asked to complete the screening questionnaire. Three ques­ tions formed the nucleus for decisions relative to whether the person was a nov­ ice or an expert computer/library user and with the level of experience he or she had with computers, with the WSU Li­ braries, and with the Internet. Other ques­ tions on age, gender, and university sta­ tus helped create demographic diversity among the potential test participants. WSU requires a formal review of pro­ cedures whenever human subjects are used in research. Because the team planned to ask students to complete a series of tasks, the usability testing fell into this category of research. Thus, the team filed a detailed human subject form for review by the Institutional Review Board, complete with consent forms and copies of the tests and procedures being used. The Institutional Review Board ap­ proved the testing. In preparing for the actual test sessions, several checklists were compiled as rec­ ommended in the Rubin book (see figure 2). Equipment was located, tapes bought, and a data collection form designed. The data collection form listed the participant tasks, with room below each to record success, false starts, beginning time, and ending time. The Tests At Pullman, the team met each volunteer at the entrance of the library because the office where the tests were to be held was not easy to find. Each session was opened by reading a standard greeting and set­ ting the scene. Team members explained that the same opening statement was be­ ing read to each participant to ensure that everyone heard the same set of directions. The Griffin WebPac test featured forty- five questions. Was this too many? Per­ haps, because the test seemed grueling for both test participants and observers. Even though the test was completed within the hour maximum suggested by Prasse, the team came away feeling that a smaller number of questions would be more ap­ propriate. Initially, the observers played the script very straight, trying not to com­ Usability Testing: A Case Study 555 FIGURE 2 Usability Checklists* Checklist 1: Six weeks before the test-May 1, 1998 Decide what to tell volunteers. Decide what data to collect. Performance data Preference data Develop data collection method and manual data collection form p. 165. Develop an orientation script. Develop debriefing outline (pp. 211, 245). Checklist 2: Three weeks before the test-May 26, 1998 Take the test. Check all equipment and the testing environment. Get copies of background questionnaire ready. Advertise for participants. Administer background questionnaire. Choose participants based on questionnaire responses. Schedule test times, leaving adequate time between tests to debrief. Checklist 3: One day before the test-June 12, 1998 Check that the product is working. Assemble all written test materials and data collection forms. Double-check test environment and equipment. Contact participants to remind them of test date, time and place. Checklist 4: Day of the test-June 15-19, 1998 Scan the checklist. Prepare mentally. Greet the participant. Have the participant complete and sign any preliminary documents. Read the orientation scripts and set the stage. Move to the testing area and prepare to test. Establish protocol for observers in the room. Distribute written task scenario(s) to participant. Have the participant complete the posttest questionnaires. Debrief the participant (pp. 211, 245). Thank the participant, provide remuneration, and show the participant out. Organize data collection and observation sheets. Monitors debrief. Prepare for the next participant. Checklist 5 following the tests-July 1998 Transform data into findings and recommendations. Make needed changes to the system. *Page numbers refer to pages in Rubin. 556 College & Research Libraries ment at all. However, by a sort of unspo­ ken agreement, the observers eventually “loosened up” and made comments aimed at reassuring the test participants. The mood-lightening comments were found to be helpful in keeping the participants re­ laxed and the test moving along. Each observer recorded search strategy, comments made by the test participant, observations about the participant’s re­ sponses, success, and the time to complete the task. After ten tests were completed (seven at WSU-Pullman and three at WSU-Vancouver), the data were compiled to form a picture of the ease or difficulty of each task. One of the suggestions made by both Rubin and Prasse was that observers de­ brief participants soon after the test ses­ sion and then compare notes on what had occurred. During the first series of tests, the team chose not to debrief the partici­ pants but later regretted its decision. Dis­ cussing the sessions soon after each was completed would be especially helpful in defining what constituted a success. For example, if the user found something, but not in the most efficient way, was the ef­ fort considered successful? The second test dealing with electronic indexes and non-WSU library catalogs was administered at WSU-Pullman only. By this time, the observers with seven Griffin tests to their credit were more re­ laxed and willing to offer suggestions when participants ran into difficulty. This, along with fewer tasks (fourteen instead of forty-five), made the second test less intense and intimidating. The final ques­ tions led to valuable discussions between observers and participants. Results and Discussion: What Was Learned about the WSU WebPac The results of the testing showed that the WSU team’s categories of novice and ex­ pert computer/library user did not cor­ relate to a participant’s ability to use the WebPac. The profiles for WSU-Pullman campus participants are given in table 1, and the results of the WebPac test are pre­ sented in table 2. November 1999 The most important thing the team learned is that many students know how to move from screen to screen using WebPac buttons, browser features, and hot-linked, clickable text. With a Web- based interface, experienced Internet us­ ers can get around quite easily. But this is not always beneficial. For example, users may feel comfortable with the use of scroll-down menus for changing search options. But when using a shortcut such as this, they do not see examples of the search format to be used in each search option (i.e., using last name first for an author search). Typing in a search in the wrong format led some of the participants to conclude that the item they were look­ ing for was not in the WSU libraries’ cata­ log. Typing in a search in the wrong format led some of the participants to conclude that the item they were looking for was not in the WSU libraries’ catalog. Many users are so supremely confident in their searching that when they do not find something, they immediately con­ clude that it is not owned by the WSU li­ braries. It does not occur to them that they may have searched incorrectly and that they could/should try another search. Users often do not understand con­ cepts and are unable to interpret content. Participants did not recognize the differ­ ence between what is available in the WebPac and what is available in the Ar­ ticle Indexes, Full Text, and More sections. How to get this idea of structure, of what is where, across effectively is still an open issue. The idea of a title search meaning the title of the periodical and not the title of the article was a problem. Participants were provided a reference to a periodical article and asked whether the library owned it. They proceeded to search ar­ ticle author, article title, and finally peri­ odical title. Obviously, the only hit would come from the search on the periodical title. Unfortunately, this was often discov­ ered by trial and error. But for eight of Usability Testing: A Case Study 557 TABLE 1 WSU-Pullman Campus Participant Profiles Results of Screening Questionnaires Questions 1. Experience with a computer -None -One-two years -More than two years 2. Status at WSU -Undergraduate student -Graduate student -Postgraduate student -Faculty member -Staff member -Other 3. Age category -18-22 -23-3o -over 3o 4. Gender -Female -Male No. Questions No. 5. Preferred learning style -Trial and error 6 1 -Consult with others o 2 -Read documentation 2 4 7. Employed by the libraries o 3 8. WSU library experience 3 -First-time user 1 -Occasional use (once a month) 2 -Frequent use (once a week) 4 -Had a library instruction class o 1 9. Internet access/use -World Wide Web 6 3 -Telnet 5 2 -E-mail 7 2 1o. Internet Use -Never or rarely 1 4 -Occasional use (once a week) 1 3 -Frequent use (once a day) 5 the ten participants, even trial and error did not lead to the information because they did not recognize the answer when they had it. This result led to another key finding. Users of the WebPac do not understand serials, cannot identify them in a browse display containing both books and seri­ als, and cannot navigate through complex serial information. How to find current check-in records, how to locate different formats such as microfilm, how to deter­ mine which libraries have what volumes is a mystery to most students. Clearly, li­ braries need to do a better job of explain­ ing this, and the OPACs must be more helpful in aiding users to find this impor­ tant information. Understanding the concept that Grif­ fin includes the library holdings of two institutions is not difficult. However, us­ ers did not always understand how to search for items on a specific WSU cam­ pus. With six libraries on the WSU cam­ pus at Pullman and four campus librar­ ies around the state, participants found it difficult to determine what library they were currently searching and how to search only one library at a time. Other discoveries included: 1. The limiting feature, where users can narrow by date, type of material, and so on was not readily understood. Most users were able to find the limiting func­ tion but were unable to use it effectively. Sometimes they found the exact thing they needed but did not recognize it. Es­ pecially troublesome was limiting by se­ rial format. Participants overlooked this serial option in a drop-down list even when looking at it. The use of MARC for­ mat terminology in drop-down lists is extremely “user-unfriendly” in limiting searches. How would any student know TABLE 2 Griffin WebPac Test N = 10 Tasks uestions � �roblems 1. Book by title at particular location Does Washington State University-Vancouver campus library have a copy �� of Gone with the Wind? 2. Book in 1 at EWU Does Eastern Washington University library have a copy of Gone with the Wind� �� 3. Book by title. libraries holding Which of the WSU libraries have Hamlet by Shakespeare? �� 4. Omit 5. Periodical from a given citation Does WSU libraries own the following: Albertson. John. �� Supermarkets in the Northwest. Advertising Age 68: 12-25 (1997). 6. Material by corporate author Find material in the libraries by IBM. 6� 7. Book by author and display complete record Find books by Laurie Garrett at any WSU location. Look at the detailed information for one of these books. 2� 8. Books about a person Find books written about Toni Morrison. �� 9. Book by author/title Do WSU Pullman libraries have a book called Guns, germs, and steel by Diamond� �0 10. Periodical by title Do WSU Pullman libraries have a copy of the periodical Audubon? 30 11. Particular library owns what years of periodical located in 10. What years does Owen library have of this periodical? �0 12. Current issue of periodical located in 10 Did the Owen library receive the Sept/Oct 1997 issue? �0 13. Particular volume of periodical located on shelf Is volume 94 (1992) of Audubon on the shelf at the Owen library? �� 14. Government document by title Do the WSU libraries own the government document, Domestic Price Director�? 20 15. Call number browse from detailed display Locate the titles of other items shelved near this government document. 40 16. ISSN search Does the library subscribe to a ournal with the ISSN (International Standard Serial Number) of 0002-9114? 10 17. Keyword search books at particular Locate books at WSU-Pullman on the use of steroids by athletes. Look at a location. display detailed record detailed record for one of these books. �0 558 C ollege & R esearch L ib raries N ovem b er 1999 TABLE 2 Griffin WebPac Test (cont.) Tasks uestions � Problems 18. Subject heading for more items Using only the information on the screen and a mouse click, find more items on the use of steroids by athletes. �0 19. Call number search given gov't doc number Check the Griffin Catalog to see if there is an item available with the call number HE 1.2:Ad 7/2. �0 20. Course reserve material What material has Lynn Levy put in the library for her English 402 students to use? 30 21. Pseudonym search Look up the author Evans, Marian. Read the screen and find a listing of books by this author. �0 22. Navigate from title to author search What do you do if you have chosen a title search and then discover you really want to do an author search? 0 23. Navigate from title to call number search# What do you do if you have chosen a title search and then discover you really want to do a call number search? 0 24. Limit result list to periodical Choose the first item in the list. Limit this resulting list to periodicals only. 66 25. Title in microform Is this title available in microform? 22 26. Title in paper copy Is this title available in paper copy? 56 27. Go to online Go to the online version. 56 28. Main menu Return to the main Griffin catalog search page. 11 29. Image database* Do a subject heading search for juarez, benito. Find a portrait and display it. 56 30. LC call number search Locate this book in the catalog DF 275.W92. 50 31. Status of book Is this book checked out, or will you find it in the library? 50 32. Periodical search Locate a copy of Nature (this is a scientific ournal [periodical]) �5 33. Keyword search Do a keyword search on steroids. �5 34. Sort by date Put the resulting list in order with the newest displaying first. �� 35. Scope to particular location Return to your original search on steroids. Are any of these items at WSU-Vancou�er� � 36. Limit by language! Return to your original search on steroids. Are any of these items in German� 37 37. Limit to periodicals Return to your original search on steroids. Are any of these items ournals (periodicals�� 7� U sab ility T estin g : A C ase S tu d y 559 = o= . 560 College & Research Libraries November 1999 that the abbreviation proj medium (projected medium) is where to locate a video? 2. Users do not understand cross-references. For example, one question dealt with a pseud­ onym. The III system message appears to be straightforward. It indicates that the heading is not used and that users should search under the correct form. Although it even offers the op­ tion to “click here” to do the proper search, half the partici­ pants did not understand. 3. Users do not understand the use of multiple call number schemes in the library. Most of the books in the WSU libraries are classified in the Library of Congress scheme. Unfortunately, WSU libraries also have materi­ als in Dewey, SuDoc, and a local government document scheme called Jackson. In Griffin, there are separate search buttons for Dewey, LC, SuDoc, ISSN, and lo­ cal call numbers, as well as a search button for miscellaneous number searches. Under each type of call number search, ex­ amples of the call number pattern help users distinguish the types of call numbers. The test revealed that to the user, many of these patterns looked the same, which rendered the examples meaning­ less. Often users did not even take note of these examples, which are available only after choosing a call number type. Some users guessed that local meant any call number used lo­ cally. Some guessed correctly that when you are clearly searching a U.S. document, you should use the US Doc search button. One user wisely noted that he never had to search by call number be­ cause he never had that informa­ tion when he came to find some­ thing in the library. Usability Testing: A Case Study 561 Results and Discussion: What Was Learned about Article Indexes/Other Library Catalogs Sections The second test, conducted only at WSU- Pullman, focused on electronic indexes and access to non-WSU library catalogs. The team scheduled eight participants, but only six of them kept their appoint­ ments. The results are shown in table 3. The team already knew from a question in the WebPac test that students had dif­ ficulty understanding the difference be­ tween the bibliographic online catalog and where to find online article citations and full text. Two of the test participants did not choose the Article Indexes, Full Text, and More sections of Griffin from the gateway page. For them to answer the remaining questions, they had to be told how to get to this page. Five of the six participants could not locate a nursing index that was listed in the medicine category. They chose the correct category but then scanned too quickly to notice the word nursing in the subheading under CINAHL (Cumulative Index to Nursing and Allied Health Literature). However, there is a lot of text on the page and it became obvious that this needed to be changed. Participants did better locating the index covering literature and linguis­ tics. This may have been the result of get­ ting used to the layout of the subject cat­ egory table. Participants did notice data­ base coverage in terms of dates included, although three later commented that they were unsure what the phrase “dates of coverage” actually meant. One partici­ pant did not locate the indexes asked about. This may have happened because he seemed to skip the first four or five questions to find one he wanted to try. Only half the participants found the alphabetical list of databases. One partici­ pant later noted that she thought it was an alphabetical list of the titles covered in a particular database. This indicates that better labeling is needed. Participants did not notice the full-text designation in the alphabetic database list. When asked whether a database contained full text, they clicked on the database and read the description from the database Web site. Users were able to find the Encyclopaedia Britannica and to search and navigate it fairly well. It is located in a category la­ beled “Encyclopedias, Dictionaries and Almanacs,” so the choice was obvious. There was a problem locating the link to the Center for Research Libraries cata­ log. This is in a section of the WSU Li­ braries Gateway labeled “Other Library Catalogs.” Because there are many librar­ ies at WSU, it was unclear what the label actually meant. For many library patrons, the library they use most often is the “only” library and anything else would be “other.” Only half the participants could locate the “Restrictions on use” in­ formation. Obviously, these pages must be modified to make that information more readily available. Suggestions that arose from questions 12–14, which dealt with the amount of information in the subject category list­ ings, importance of providing log-on in­ structions, and the arrangement of the alphabetical list, included: • A separate list of full-text databases would help. • Make the full-text icon more obvi­ ous. • How important are coverage dates? In some databases, they could be critical and in others insignificant. • Descriptions are too wordy. One participant suggested they be put in a bulleted list format. • Repeating information in the data­ base description and then in the actual database is unnecessary. • Screens are not too busy; they have clean background and have no “glitzy” graphics. Results and Discussion: What Was Learned about the Process When the test participants actually ar­ rived and performed the tasks, the team found that its determination of novice and expert computer/library users was not always accurate. In fact, to make an accu­ rate determination of novice or expert 562 College & Research Libraries November 1999 TABLE 3 Article Indexes/Other Library Catalogs Test N = 6 Question % Problem 1. From the WSU Libraries Gateway page, go to the page where you expect to find information on topics that have been published in periodicals (also known as serials or journals). 33 2. Using the subject category table of Article IndexeslFull Text & More items: 0 2a. Locate an index to the nursing literature. 83 2b. What other topics does this nursing index cover? 67 2c. Locate an index covering literature and linguistics. 17 2d. What dates does it cover? 17 3. Go to the alphabetical list of Article IndexeslFull Text & More. 50 3a. Is there any full-text information in the Agricola database? 50 3b. Is there any full-text information in IDEAL? 33 4. Return to the page with the subject categories for Article IndexeslFull Text & More. 33 4a. Find an encyclopedia article on drums and the American Civil War. 0 4b. Return to the Griffin catalog from the encyclopedia. 17 5. Return to the WSU Libraries Gateway page. 0 6. Go to the Center for Research Libraries catalog and log-in as if you were going to use the catalog. 33 7. Return to the WSU Libraries Gateway page. 0 8. Go the page with information on Article IndexeslFull Text & More. 0 9. Click on the Restrictions on use link located below the table. Assume you have been denied access to ERIC, an education database. Go to the page where you would send a message to find out how to get a password andlor authorization to access ERIC. 50 10. Return to the page with the Restrictions on use link. 0 11. What is WorldCat? 33 11a. How often is it updated? 17 11b. Who produces it? 17 Possible Question Responses No. 12. Return to the page with the subject categories for Article IndexeslFull Text & More. 12a. Choose GenerallMultidisciplinary. Yes 0 Read some of the database descriptions. No 5 Is there too much information here? No response 1 12b. If you think there is too much information, Description 0 what would you leave out? Please put an Dates of coverage 0 X by what you would omit. Full text or index 1 13. Do you think the LOGON INSTRUCTIONS Yes 3 column is useful? Please mark one. No 1 No response 2 14. Go to the alphabetical list of Article Indexesl Alphabetical by column 2 Full Text & More. Please mark which Alphabetical by row 3 arrangement you prefer. No response 1 computer/library users in the future, it would be necessary to design a more in­ tensive screening process. After the tests, it became obvious that there is value in having observers in a separate room as occurs in the OCLC us­ ability test lab. Observers can feel free to discuss what the test participant is expe­ riencing and decide whether they want or need to help the user. Some test par­ ticipants may have been uncomfortable with two observers watching them, al­ though no one really complained or seemed unduly bothered. This aspect of testing was probably ameliorated as the observers themselves loosened up. Test participants who are not native English speakers had difficulty with the questions. Several of the test participants were young students, and observers noted that some of them were impatient. Several did not read instructions completely. This is especially problematic when questions build on previous ones. Thus, it is pref­ erable not to have questions that build on one another. Test participants who are not native English speakers had diffi­ culty with the questions. Because their participation was especially useful, the team recommends straightforward ques­ tions free of jargon or popular idiomatic expressions. For example, one of the questions in the second series of tests asked the user to find a database that covers nursing literature. The first test participant for whom English was not the native language, keyed in on the word literature in the question and went to the Modern Language Association (MLA) database where literature is in­ cluded in the description. It is advisable to check questions right up to test time when testing a database that is dynamic. For example, one ques­ tion tested the user’s knowledge of check­ ing other locations for a book title that was not held on the Pullman campus. By the time the test was administered, the title had been cataloged for Pullman. Usability Testing: A Case Study 563 Finally, it is inevitable that some people will fail to show up for the test. Testers should plan for a 15 percent no-show rate. In addition, scheduling is one of the most time-consuming parts of testing and is easiest to do with participants who had e-mail. Implementation The results of the usability tests were turned over to a group responsible for overseeing development of the Web- based OPAC. This group reviewed all the areas where participants encountered dif­ ficulty in using the online catalog. For the Griffin WebPac test, they developed a table with task, problem, name of the in­ dividual responsible, and action to be taken (see table 4). The team is in the pro­ cess of working on solutions that are within its local control; it will be submit­ ting a series of enhancement requests to III. WSU’s User Education Department has received a list of problem areas that will need to be included in instructional sessions. WebPac developers are working on making the recommended changes un­ der local control. As a result of the electronic indexes/ non-WSU library catalogs test at Pullman, the alphabetical list was changed to a drop-down menu from a table format. Descriptions in the subject category sec­ tions have been shortened, and informa­ tion is presented in a bulleted format. The “Log-on Instructions” and “Alternative Connections” columns have been elimi­ nated and a column on restrictions on use added. The label for “Other Library Cata­ logs” remains unchanged for the mo­ ment. Will WSU Do More Usability Testing? Needless to say, the team participants in the usability testing are true believers in its value. However, the process is inten­ sive and takes time to prepare and ex­ ecute. As more people become involved, the workload can be shared. Ideally, us­ ability testing will occur naturally at any point when substantive changes are be­ ing made to the catalog. TABLE 4 Griffin Online Catalog-Results to Actions Task Probems Identified Who and Action 1-3. Books by titles at particular locations 4. Omit-not valid at test time 5. Periodical from a given citation 6. Material by corporate author 7. Book by author and display complete record 8. Books about a person 9. Book by author/title What are the Pullman Libraries; difference between libraries in Griffin and our Other Libraries Button. Inability to decipher a reference to a periodical article. Word or subject searches performed; difference between searches for information "by" or "about" not clear. Searches done from the drop-down box do not show examples of how to enter author name. No example of a person as a subject on the subject or keyword search screens. No one used the author/title option. UE: Library User Education Department III: Innovative Interfaces. Inc WWG: Web Working Group SWG: Serials Working Group UE: Give overview of WSU Libraries Gateway and what each item is. UE: Give overview of WSU campuses. UE: Give overview of libraries in the WSU system. UE: Explain "scoping." UE: Teach how to interpret a citation to a periodical article. UE: Teach access point in Griffin UE: Explain that corporate entities can be authors. UE: Explain difference between works "by" an author and works "about" an author. WWG: Put context-sensitive help button on each search screen. WWG: Add examples to Subject and �eyword search screens. WWG Add more complex subject heading example to subject search screen. UE: Teach or point out use of author/title as an efficiency. 564 C ollege & R esearch L ib raries N ovem b er 1999 TABLE 4 (cont.) Griffin Online Catalog - Results to Actions Task Probems Identified Who and Action 10. Periodical by title 11. Particular library owns what years of periodical in 10 12. Current issue of periodical located in 10 13. Particular volume of periodical located in 10 14. Government document by title 15. Call number browse from detailed display 16. ISSN search No one could recognize a periodical from the browse display list resulting from the title search. Complex location listings and inability to decipher item records!summary statement. Cannot locate check-in record. Cannot decipher item records, don't know what they are. None. Did not notice that call number is a hotlink. None. WWG: Provide access to a periodical title search option. UE: Teach terminology of serials�periodicals, journals, magazines, newspapers. UE: Develop instruction approaches to explain information on serials screen for questions 11�13. SWG: Review summary holdings statement format to see if it can be simplified. III: Make "Latest Received" hotlink more evident and! or re-label. UE: See question 11. UE: See question 11. None. UE: Note in classes what the call number hotlink does. None. U sab ility T estin g : A C ase S tu d y 565 TABLE 4 (cont.) Griffin Online Catalog - Results to Actions Task Probems Identified Who and Action 17. Search books at particular location on a topic. display detailed record 18. Subject heading for more items 19. Call number search given gov't doc number 20. Course reserve material 21. Pseudonym search 22. Navigate from title to author search 23. Navigate from title to call number search Entering search statement; problems with word vs. subject searching. Did not locate official. hotlinked subject heading in the record. Don't know which type of call number to search; don't recognize pattern from examples. No course reserve buttons when scoped to entire catalog. See reference message is not understood. None. None. WWG: Put mouse overlays on each button main Griffin search screen. WWG: Change Subject Headings button to ready LC Subjects Only or ...? WWG: Provide mouse overlays on subjects: "Use ONLY for known Library of Congress subject headings." WWG: Re-label subject radio button to LC Subject. WWG: Put help buttons on each search screen. WWG: Use more complex subject examples. WWG: Add corporate example. UE: Teach search strategy of keyword search then click on subject headings in "best" items found. WWG: Change order of call # buttons: LC. Dewey. U.S. Docs in first row then ISSN. Local. �isc. in second. WWG: Put mouse overlays to define each call number button. WWG: Find a new label for Local. WWG: Add course reserve buttons to entire catalog and "WSU all locations" scope. III: Reword the message to something like: "For items on XXX use XXX." None. None. 566 C ollege & R esearch L ib raries N ovem b er 1999 TABLE 4 (cont.) Griffin Online Catalog - Results to Actions Task Probems Identified Who and Action 24. Limit result list to periodical 25. Title in microform 26. Title in paper copy Limit/Sort not noticed. Words in drop-down box are jargon. Did not notice microform after title in browse displays. Paper is the default format and so not indicated. WWG: Provide a journal title search option. III: Use understandable terminology as labels within drop-down box of the Limit/Sort. WWG: Talk to Bib Control about inconsistency of microform displaying after titles in browse list. None. 27. Go to online 28. Main menu 29. Image database 30. LC call number search 31. Status of book 32. Periodical search 33. Keyword search 34. Sort by date 35. Scope to particular location Did not notice hotlinks; messages very confusing; multiple links and only one works. None. Hotlink to portrait not found; correct record not chosen from subject browse listing; click on call number and get into browse; don't follow instructions. Unable to recognize call number type so cannot choose correct search. None. Cannot pick out the serial in a browse display. Not following instructions. Don't see limit/sort button. Don't see limit/sort button. WWG: Work with EWU and Bib Control to simplify hotlinks. WWG: Work with Bib Control to simplify wording displaying as call number for online resources. None. WWG: Purchase III image database. WWG: See actions in question 19. Possibly develop intermediate page from one call number button. This page would give explanation and examples of every call number type. None. WWG: See actions in question 19. None. For questions 34-39. UE: Alert users to limit/sort option. UE: See actions in question 33. U sab ility T estin g : A C ase S tu d y 567 TABLE 4 (cont.) Griffin Online Catalog - Results to Actions Task Probems Identified Who and Action 36. Limit by language. 37. Limit to periodicals 38. Limit by date 39. Limit to videos 40. Extended display 41. E-mail three items 42. Print three items 43. Periodical articles 44. Griffin catalog button 45. Online suggestion Don't see limit/sort button. Terminology not understandable on sort labels and within the drop-down box. None. Terminology not understandable on sort labels and within the drop-down box. Not known what extended button does. Concept of marking and saving records to create export list not understood. Save marked records button only seen if scroll down; not clear that user needs to choose display option as well as e-mail; display options not clear. Concept of marking and saving records to create an export list not understood. Save marked records button only seen if scroll down. Concept of what is and is not in Griffin is not clear. None. None. UE: See actions in question 33. See actions in questions 24 and 33. None. See actions in question 24. UE Teach use of extended display for viewing date and location. UE: Explain concept of "export.� III: Improve export screen. See actions in question 41. UE: Explain difference between what is in Griffin vs. article indexes. WWG: Put mouse overlays on WSU Libraries Gateway. WWG: Review size and placement of buttons on WSU Libraries Gateway None. None. 568 C ollege & R esearch L ib raries N ovem b er 1999 Usability Testing: A Case Study 569 To integrate usability testing naturally into library procedures, budget support is necessary. A token payment to test par­ ticipants helps with recruiting and legiti­ mizing the process. Payment for all equip­ ment and materials also should be part of the budget. It is unnecessary to com­ mit great sums of money here; establish­ ing a modest budget for usability testing has a tremendous return on investment. The MSW team offers the following simple words of advice to anyone who wants to do usability testing: • Begin now! • Test early and test often. • Keep the process simple and straightforward. • Keep the tests narrow in focus. • Do not try to learn too much in one test. • Pay the test participants in some way. • Debrief promptly after each test ses­ sion. • Communicate findings to system developers, user education instructors, public service librarians, and the system vendor. WSU-Vancouver has already com­ pleted another round of testing of its li­ brary Web site. The team there used sev­ eral different methods to attain data on how the site should be organized and what should be on it. Another project is under way at WSU-Pullman to test the usability of its library Web site. This test­ ing will include the usability test format described here as well as card sort, cat­ egory membership, and an online survey. The card sort and category membership tests will be similar to those already com­ pleted at the pioneering WSU-Vancouver campus library. Yes, WSU is in this for the long haul. Notes 1. Bob Murphy, “Usability Testing,” OCLC Newsletter 229 (Sept/Oct. 1997): 21–23. 2. “Usability Testing Started in OCLC Office of Research,” OCLC Newsletter 229 (Sept/Oct. 1997): 24–25. 3. Lois Yoakam, “Writers Test Documentation to Determine Users’ Needs,” OCLC Newsletter 229 (Sept/Oct. 1997): 27. 4. Jeffrey Rubin, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (New York: Wiley, 1994). 5. Joseph S. Dumas and Janice Redish, A Practical Guide to Usability Testing (Norwood, N.J.: Ablex, 1993).