College and Research Libraries CD-ROM Software Evaluation: Comparison of Searching ERIC on IBM and Macintosh Edwin B. Spragg In late 1989, the Albert R. Mann Library at Cornell University conducted a comparative -study of user competence, behavior, and preference in the applica- tion of text and graphics search-and-retrieval software. Silver Platter's IBM PC and Apple Macintosh programs (PC SPIRS and MacSPIRS respectively) were evaluated. Librarians observed 40 students who searched the ERIC database. The reactions of these student subjects were collected in written questionnaires. This study suggests the limitations of the current state of the art in the application of graphical user interface principles to bibliographic search-and- retrieval software. It also provides insight as to some components and features necessary in an excellent workstation for use in accessing an electronic library. The computer revolution will be judged not by the complexity or power of technology but by the service to human needs. By focusing on the user, re- searchers and designers will generate powerful yet simple systems that permit users to accomplish their tasks . . . . Putting the user's needs first will lead to more appropriate choices of system fea- tures, a greater sense of mastery and con- trol, and the satisfaction of achievement.1 The Albert R. Mann Library at Cornell University is developing an electronic li- brary that provides access at scholars' workstations to many types of informa- tion in a variety of formats from multiple locations. Resoun::es available at the work- station include an online catalog with li- brary holdings and circulation information, an electronic reference service, and locally mounted bibliographic and numeric databases. In the near future, fulltexts of journal articles will also be available. The system interface design that best meets the needs of users is critical to the success of this electronic library. As a way to identify some of the characteris- tics of good interfaces, we chose to study an existing system for database access and to compare the two different kinds of hardware and software used by this system. We chose SilverPlatter's PC SPIRS software (version 1.6) for the IBM interface and their MacSPIRS software for the Macintosh interface to ERIC, a database of education literature. SilverPlatter's IBM and Macintosh soft- ware for the ERIC database have major differences. The product that runs on IBM PCs and compatibles, PC SPIRS, is text-based. The user types search terms; the return key initiates action. Control keys are used to move the cursor. Func- tion keys allow the user to access cer- tain functions without having to type text. A menu is provided at the bottom Edwin B. Spragg is Reference Service Coordinator at Albert R. Mann Library, Cornell University, Ithaca, New York 14853-4301. 342 of each screen. There is no graphic rep- resentation. SilverPlatter's Macintosh software (MacSPIRS) is based on a combination of text and graphics and represents a sub- stantial effort to employ graphical user interface technology within a bibliographic search-and-retrieval application. As with the IBM interface, the user types search terms. The return key or mouse inputs commands. The mouse controls cursor movement. Textual icons (buttons) and graphic icons on the screen are used to perform actions. Drop-down menus are accessed by clicking on the menu bar across the top of the screen. Windows enable the screen to be divided into smaller subscreens. They may be left on the screen or removed; their size may be adjusted. INTERFACE EVALUATION Until recently IBM PC and compatible microcomputers running search andre- trieval software in a DOS environment have been the standard for CD-ROM systems for bibliographic databases. With the appearance of search-and-re- trieval systems for the Apple Macintosh, librarians are confronted with deciding whether to provide users with access to that software as well as to the IBM soft- ware. As part of a hardware grant re- ceived from Apple Computer, Inc., Mann Library conducted, with the as- sistance of SilverPlatter Information Inc., a comparative evaluation of user response · to IBM PC-based and Macin- tosh-based CD-ROM systems. Silver- Platter provided a three-month loan of its MacSPIRS search software to enable the library to conduct a comparison of MacSPIRS and PC SPIRS. These two packages have similar functional capa- bilities but divergent approaches to the management of the user interface. Purpose The purpose of this evaluation was to compare user competence under, and user preference for, two varieties of search- and-retrieval software for bibliographic databases: (1) software based on a textual interface; and (2) software employing a graphical user interface. User competence CD-ROM Software Evaluation 343 was defined in terms of user ability to perform a set of predetermined tasks successfully. Through this study, the author sought to gain insight into the current state-of-the-art in bibliographic retrieval software and the hardware and software features necessary in an excel- lent workstation for the provision of access to an electronic library. Objectives • To identify the major differences be- tween text-based and graphics-based software interfaces to bibliographic databases on CD-ROM by evaluating SilverPlatter's PC SPIRS and Mac- SPIRS software; • To identify the elements of the hard- ware systems and software biblio- graphic database interfaces that appeal to users in the access of infor- mation stored on CD-ROM; • To study user reaction to IBM hard- ware and SilverPlatter's PC SPIRS software in order to determine those features users regard as easy and effi- cient and those they consider difficult; • To study user reaction to Macintosh hardware and SilverPlatter's Mac- SPIRS software to determine those fea- tures users regard as easy and efficient and those they consider difficult; • To determine user preference for SilverPlatter's text-based, cursor-con- trolled interface or SilverPlatter's graphics-based, icon-controlled inter- face to information in the ERIC database on CD-ROM. EXPERIMENTAL DESIGN The author chose students for the study because CD-ROM systems are popular and heavily used. Students searched the ERIC database during a three-week period in November-December 1989. Each stu- dent was given a series of tasks to perform on both the Macintosh and IBM versions of Silver Platter's search software for CD- ROM, operating against the ERIC database. A librarian observed the stu- dents' searches and made notes of their comments and observations. Each stu- dent filled out a questionnaire after both searches were completed. 344 College & Research Libraries The assignment and questionnaire were pretested by seven library staff members who had varying degrees of microcom- puter and compact disc searching ex- perience. Forty undergraduate and graduate students participated in the eval- uation. Each student completed a regis- tration form that collected the following demographic information: name, col- lege or school, class and major (under- graduates), degree and field (graduates). Two workstations, a Macintosh SE and an IBM PC-AT, were set up adjacent to each other. Before beginning a search, each student was given a brief summary of the purpose of the experiment and a description of how the session would be conducted. Each student performed the predetermined subject search on both ver- sions of ERIC in a quiet atmosphere free from distractions. The same search was performed on both machines, twenty minutes being allotted for each search. The machine to be searched first was deter- mined in random fashion by a coin toss. The speed of the hardware in retrieval of records was not part of the experiment. Virtually all of the users relied heavily on online and printed help. The librarian observing student search- ing recorded observations but did not an- swer questions or make comments, so as to allow the student to search in a natural manner relying on system features. Silver- Platter's Quick Reference Guide was posted at both machines, thereby providing the student with printed as well as online help. The student was encouraged to comment aloud so that his or her thought process could be noted. After completing the search on both programs, and before any com- mentaryordiscussion with the librarian, the student answered the questionnaire. Information pertaining to compact disc searching experience and microcom- puter experience was collected in the first part of the questionnaire. From this infor- mation each student was placed in one of the three categories indicating levels of CD and microcomputer experience: naive July 1992 user (never use), minimal user (use less than once a week), sophisticated user (use once a week or more). The observation of searching and the follow-up interview had a twofold pur- pose: first, to allow the librarian to ob- serve the user search in a natural manner, free of interference by the evaluator; and second, to gather data on users' reactions to specific hardware and software features by means of a questionnaire administered by the librarian in a follow-up interview. The qualitative analysis of user prefer- ences and opinions consisted of review- ing the comments made by the students in the questionnaires and studying the notes and observations made by the librarian who observed the searches. FINDINGS The data provided a correlation of ex- perience levels and machine preference with task success. They also provided a quantitative picture of our test population's success in performing the ERIC searches. Most (78%) of the students were un- dergraduates, and three-fourths were enrolled in the colleges and divisions of Cornell University served by Mann Li- brary. Although 24 of the students had never performed a search on a CD database, all were at least minimal-level users of IBM PC, PC compatible, or Mac- intosh personal computers (see appendix A). The study involved no naive micro- computer users. This is an accurate reflec- tion of Cornell's student population. Table 1 compares success by the par- ticipants in performing the tasks neces- sary for completing the ERIC search using PC SPIRS on the IBM and Mac- SPIRS on the Macintosh. Table 1 does not consider level of experience and original preference for a machine. Some students did not complete the assignment and therefore did not attempt all the tasks. Overall, the success rate on both pro- grams was similar. More students were able to print successfully a specified number of records using MacSPIRS. Task success correlated with CD ex- perience and microcomputer experience showed a direct correlation between ex- perience and success. Students with CD-ROM Software Evaluation 345 TABLEt TASK SUCCESS PCSPIRS Success PC SPIRS PC SPIRS MacSPIRS MacSPIRS MacSPIRS Failure No Attempt Success Failure No Attempt Task No. % No. % No. % No. % No. % No. % Enter terms Display results Display titles only Combine two concepts Print results Print abstracts Print specified number of records 40 100.0 0 0.0 38 95.0 2 5.0 21 52.5 19 47.5 20 54.1 17 45.9 29 82.9 6 17.1 20 58.8 14 41.2 14 41.2 20 58.8 Limit by date 10 32.3 21 67.7 Use index 13 44.8 16 55.2 Complete 0 0.0 0 0.0 0 0.0 3 7.5 5 12.5 6 15.0 6 15.0 9 22.5 11 27.5 40 100.0 0 0.0 38 95.0 2 5.0 22 55.0 18 45.0 23 57.5 15 37.5 26 74.3 9 25.7 20 58.8 14 41.2 20 58.8 14 41.2 11 37.9 18 62.1 13 54.2 11 45.8 0 0.0 0 0.0 0 0.0 2 5.0 5 12.5 6 15.0 6 15.0 1 2.5 16 40.0 assignment 4 10.0 36 90.0 NA 0.0 2.5 39 97.5 NA 0.0 minimal or sophisticated levels of CD experience performed, overall, some- what better on both machines than the naive users. This finding was particu- larly obvious in the case of the more difficult tasks toward the end of the as- signment. Also, the naive CD users had more difficulty completing the assign- ment. The sophisticated microcomputer users performed better than those with minimal-level experience. This was more apparent when they searched on the Macintosh with MacSPIRS. Overall, the students performed equally well on both machines, regard- less of their initial preferences. However, those who initially preferred Macintosh had a difficult time printing a specified number of records and using the index on IBM. Conversely, those who pre- ferred IBM had an easier time perform- ing these tasks on the Mac. More students indicated an initial preference for Macintosh than IBM (table 2). Twenty-four expressed preference for Macintosh. Thirteen preferred IBM. Three were neutral. The fact that there were so many Mac users means it is important to offer Macintosh-based CD services. The students did not change their preference for IBM or Macintosh. On the whole they stayed with their initial pref- erences (table 3). Of the 13 who initially preferred IBM, just 1 switched to Macin- tosh with MacSPIRS. Only 3 who pre- ferred Macintosh changed to IBM with PC SPIRS. Two of the 3 students who were neutral changed preference, 1 to IBM and 1 to Macintosh. The 4 students who changed to IBM commented that IBM was easier to use and understand and better designed for searching. The Mac was easier to use for the students who switched to it. TASK SUCCESS ANALYSIS During the twenty-minute search of ERIC by the students on both machines, we had ample opportunity to note search behavior. A few performed the assign- 346 College & Research Libraries TABLE2 INITIAL MICROCOMPUTER PREFERENCE Initial IBM preference 13 32.5% Initial Macintosh preference 24 60.0% Initial neutral preference 3 7.5% Total 40 100.0% TABLE3 CHANGE OF MICROCOMPUTER PREFERENCE Change No Change System No. % No. % IBM 1 7.7 12 92.3 Macintosh 3 12.5 21 87.5 Neutral 2 66.7 1 33.3 mentwith both programs with a minimum of difficulty. Most experienced frustration, struggling to complete the tasks in spite of online help and the printed Quick Reference Guide. Comments such as "I'm so con- fused" and "I must tell you, I hate this" were not unusual. The students tried many different things to overcome diffi- culties. For example: • Immediately reading online help screens and the printed Quick Refer- ence Guide rather than trying to follow the programs' interface; • Restarting the search session rather than clearing the search or returning to the Find prompt (IBM) or the Search History window (Mac); • Reading the tutorial in the middle of a search; • Retyping search terms as opposed to returning to search strategy by using F2 (IBM) or clicking on the Search His- tory window (Mac); • Highlighting (IBM) or dragging (Mac) an entire abstract rather than specific words or terms when selecting search terms from retrieved records (lateral searching); • Typing subject terms instead of the proper two-letter field labels for author, title, source, language, etc. at the Show Fields prompt (IBM). Virtually all the users relied heavily on online and printed help. After taking their July 1992 first look at the screen, they frequently sought immediate help. For some, at- tempting to find help consumed much of the twenty-minute session. Those most confused read through numerous online screens, referred to the Quick Reference Guide, and occasionally resorted to the tu- torial. The less experienced micro users seemed to adapt to PC SPIRS F1 key more readily than to the MacSPIRS help menus and context sensitive help. For instance, many clicked on the help icon but failed to drag the question mark pointer to the element in need of explanation. Most participants entered search terms quickly and were able to display results (95% on IBM, 95% on Mac), pri- marily by paging through retrieved rec- ords. Most had great difficulty, however, when they attempted to display only the titles of articles. Only 53% were able to display successfully titles using PC SPIRS on the IBM and just 55% with MacSPIRS on the Macintosh. Here are some of the difficulties encountered: • Inability to comprehend the meaning of the term fields; • Lengthy effort to find online and printed help for field descriptions; • Tendency to display all fields rather than type the two-letter field label (IBM) or click on "ti" (Mac); • Paging through records displayed in full or citation format to select titles rather than displaying only titles. Terminology was frustrating for the stu- dents. One commented, "I never did find out what a field is." Another said, ''What in the world is CITN?" With PC SPIRS, great difficulty occurred at the Show Fields prompt accessed by pressing the F4 key. That the field label"ti" was necessary to display only titles was not apparent. Some users entered a search statement number, such as 3 or #3, while others typed "title" or "titles" rather than "ti." With MacSPIRS, some gave up because of the inability to click properly on "ti," many clicked on the word "title." A few stu- dents typed the two-letter field labels at the Find prompt (IBM) rather than a t Show Fields. Fifty-four percent of the students were able to combine successfully two con- cepts while searching on PC SPIRS, and 57% could do so on Mac SPIRS. Proper use of the and operator to narrow a search proved to be difficult. For example: • Students tended to page through rec- ords rather than combine terms. • A number of students immediately at- tempted to search laterally rather than use the and opera tor to combine terms. • Many students spent much time seek- ing help to perform this task. • Only two students used the and button on Mac SPIRS. The seemingly simple tasks of print- ing records, printing selected records, or printing the abstract of a record were anything but simple. On the contrary, they were challenging for most of the participants in this experiment. Forty- one percent were unable to print an ab- stract in either program. Only 41% could print a specified number of records using PC SPIRS, only 59% using Mac SPIRS. Here are some of the problems: • As in the task of displaying records, fields and their two-letter labels proved to be a stumbling block. • Students would begin to print all rec- ords in full or citation format and break to start over. • Students were unable to use effec- tively the Print Records dialogue box in Mac SPIRS; they stumbled through trying all sorts of pathways. • Students clicked on Selected Records in the Print Records dialogue box even though they had not created a subset of records for subsequent printing; many students were frustrated by the lack of a response. • Puzzlement occurred over the term searches in the Print Menu (PC SPIRS); that the term refers to search strategy was not clear. For the students who progressed far enough in the assignment, the task of limiting by date was challenging. Sixty- eight percent were unable to perform this task successfully using PC SPIRS, and 62% could not perform it in Mac SPIRS (table 1 ). The command was simply too confusing. Of those who succeeded, few were able to do it correctly on the first attempt. Most failed to use the operator CD-ROM Software Evaluation 347 and between the search statement number and the limit command. Instead, many tried "with" or "since" or simply omitted the operator. In PC SPIRS more than one student typed "py>1987" after the Show Fields prompt. Several students expressed a desire for the system to respond with examples rather than an error message or error noise. Only 45% of the test population who tried to use the index in PC SPIRS were successful, and just 54% could use it properly in Mac SPIRS (table 1). The stu- dents did not readily use the F5 function to access the index while searching in PC SPIRS. For many, the index was difficult to locate in Mac SPIRS. Once found, it was common for the author's name to be entered incorrectly. SilverPlatter's re- lease in February 1991 of its 2.0 version of SPIRS software for the IBM PC and compatible is expected to address some of the problems described here. SOFTWARE ANALYSIS The individuals in this test population provided much commentary concerning both software programs and the conven- tions upon which the programs rest. Subjects made their comments during the search sessions and on the question- naires. The following summarizes stu- dent commentary on the software features of both programs. Help The students made extensive use of help in both programs. Most found PC SPIRS' help easily accessible, clear and easy to follow, and sufficiently informa- tive. They used the F1 function immedi- ately. A few said help in PC SPIRS was confusing and unclear. Many students described help in MacSPIRS as more ac- cessible, easy and straightforward, more detailed, more visible, and quicker than PC SPIRS in providing exact help. For a few it was confusing. Many were unable to use the Help icon properly. Menus Some viewed PC SPIRS' menus as easy to find and understand, more or- ganized, and less confusing. There was 348 College & Research Libraries more favorable commentary on the Mac- SPIRS' menus, however (62% preferred MacSPIRS' menus, 18% preferred PC SPIRS'). The MacSPIRS menus were seen as faster and simpler, self-explanatory, clearer, easier, and more available at any given time. However, the non-Mac users struggled with the help icon and did not always use the Help menu effectively. Find Records PC SPIRS' Find (F2) function for searching the database for words or phrases was described as easy, fast, very direct, and straightforward. MacSPIRS' Find Records dialogue box was also easy to use for most of the students. Show Records Students characterized displaying rec- ords in PC SPIRS as faster, easier, com- prehensible, quicker, automatic, and organized. One described it as terrible, another was confused at first, and three could not determine the options for show- ing records. Those who preferred Mac- SPIRS said it was more flexible, easier "to do and undo," and simpler. Many found MacSPIRS easier for selecting fields than PCSPIRS. Printing Printing in PC SPIRS was easier for a majority (40% versus 23%) and "less of a hassle," as one put it. Some preferred Mac- SPIRS, but most found it too confusing and complicated and requiring too many steps. Those who preferred MacSPIRS found it easier for selecting options. Lateral Searching Most of the students did not attempt to search laterally (i.e., select words or terms from a displayed record). Those who did preferred MacSPIRS. To them, performing this function with MacSPIRS was easier than with PC SPIRS. (Eleven students preferred MacSPIRS; only two favored PC SPIRS.) Icons Many students commented on Mac- SPIRS' icons. Several students said they were easy to understand, helped to clarify July1992 procedures, simplified various functions, and were friendly and clear. "Great!" wrote one student. However, others strongly criticized the icons. The follow- ing are some of the students' comments about MacSPIRS' icons as recorded in their answers to the questionnaires: Not obvious how to use Self-explanatory Sometimes confusing Somewhat ambiguous Easy to get from place to place Frustrating Not familiar with how they work Very friendly Prefer explanations rather than pictures Uncomfortable with Clearer Hard to find buttons at times Changed suddenly, interrupting thought processes Not obvious what "?" meant "?"was great Scroll Bars and Windows Those who liked MacSPIRS' scroll bars-which are a standard element of the Mac's graphical user interface-said that they were effective, easy to use, simple, fast, and "wonderful." Those who were less familiar with them said that they weren't aware of them or that they took a while to learn to use. One said, "I didn't have the slightest idea how to make them work." Some failed to click on a window's close box. HARDWARE ANALYSIS Students slightly preferred the ar- rangement of the MacSPIRS' screen. They described it as more compact, or- ganized, and easier to figure out. Those who disliked it said it was cluttered and had too many menus or too much "junk" across the top of the screen. Those partial to the arrangement of PC SPIRS' screen described it as visually clearer, less con- fusing, easier to follow, better organized, and having a larger area to read. With regard to hardware, of course, a variety of display sizes and resolutions are available for both machines. The non-IBM users were uncom- fortable with PC SPIRS' cursor, had a difficult time moving it, found it slower than the mouse, and were confused by its purpose. Some had difficulty using the arrow keys. Others said it was easy to use, straightforward, and took less time. The Macintosh users thought it was easier to move about the screen in Mac- SPIRS with the mouse than with the cur- sor. They found it easier and faster than using arrow keys and tabbing. Those less experienced with Macintosh regarded the mouse as a cumbersome hassle to click precisely, hard to keep on the pad, and requiring experience for efficient use. One student summed up the frustra- tion observed ·among many of the other students by saying, "I hate the mouse." STUDENT PREFERENCE FOR SEARCHING ERIC The test group expressed strong origi- nal preference for the Macintosh (24 to 13, 3 neutral). There was a slight prefer- ence for searching ERIC with PC SPIRS, however (20 to 17, 3 neutral). CONCLUSIONS This evaluation compared two differ- ent systems for database access. Its pur- pose was to study user competence, behavior, and preference in the applica- tion of text and graphics search-and-re- trieval software. · The text-based PC SPIRS product has been in existence since 1987, while the MacSPIRS search software, which ern- ploys many elements of a popular graphical user interface model, became available in 1990. The author was im- pressed that MacSPIRS performed as well as it did, given its relative newness. However, this study provides no evi- dence that, at this time, in the realm of bibliographic database searching, a graphical user interface significantly en- hances user performance over more tradi- tional textual interfaces. Overall, the success rate on our structured tasks was similar with both programs. Users did not obviously prefer one SilverPlatter pro- gram and style of interface. Users liked and disliked features of both programs. The most alarming aspect of this ex- periment was the inability of the users to CD-ROM Software Evaluation 349 follow the intended flow of both pro- grams, despite extensive use of online and printed help. Since 60% of the test group had never searched a CD database, some difficulty was expected, but most of the users had great difficulty following the program design on both machines. The software program designers may have perceived a natural flow, but it was not obvious to the naive user. Donald A. Norman points out that the responsi- bility of the designer is "to make the System Image explicit, intelligible, con- sistent."2 In other words, everything with which the user interacts, including screens, printed help, error messages, etc., should be designed to communicate a natural flow to all users. The students were frustrated; only one was able to complete the assignment on both PC and Mac. The findings indicate that end users are not successful at searching bibliographic databases on compact disc. User microcomputer skill means little to success in this kind of searching. This lack of success has seri- ous implications for the electronic li- brary, whose users will not be within the four walls of the library and near to human assistance. Librarians will need to take seriously the responsibility for educating the user in information tech- nology skills if the user is to be successful in using the electronic library. In analyzing the performance of the graphically oriented MacSPIRS software in our testing, perhaps two hypotheses would explain our results. The first is that, in searching textual bibliographic databases, the use of a graphical user interface does not present advantages over a traditional textual interface. A sec- ond possibility is that software developers, who have only begun to apply graphical user interfaces to search-and-retrieval programs, are not yet exploiting the full potential of the technology. This area is rich for further exploration. THE IDEAL WORKSTATION Bibliographic information systems are an important component in the develop- ing electronic library. The findings from this evaluation have implications for the 350 College & Research Libraries design of search-and-retrieval interfaces . to electronic scholarly information sys- tems and the need for librarians to in- fluence it. Based on the findings, the author makes the following recommen- dations and observations to help inter- face designers develop an electronic workstation that emphasizes users' needs. • Users want examples of correct com- mands when they get error messages or error noises while searching. They prefer not to page through help screens or tutorials for examples. Responses such as "cannot interpret highlighted character'' annoy most users. • Adults do much learning by trial and error or active involvement. 3 A system with a user population including inex- perienced users should have an inter- face that supports trial-and-error learning. When an error occurs, the system should provide informative feedback. When a command is used incorrectly, the system should re- spond by indicating this to the user and providing an example of correct usage or by giving the user an option to find one. Examples presented deep within the help system or tutorial are not effective. • Terminology that is obvious in mean- ing to users should be selected. Fields is a term few users comprehend. • Labels and abbreviations should be used with care. They are difficult for first-time users, causing them to seek help. As one student put it, "I don't know what the two-letter things on July 1992 the side are." "CITN" (citation) is baf- fling to almost all first-time users. Where it is important for the user to learn abbreviations, labels, and specific terminology, the interface should be de- signed to facilitate that learning. Rele- vant information should be readily available to the user at any point in the searching process. David Owen sug- gests the "Answers First, Then Ques- tions" format of information presentation. Information useful for decision making is presented to the user in an unsolicited manner.4 A screen or win- dow of definitions of terms, etc., would be a better way to make this information available than burying it in help screens or a tutorial. • Users prefer a large screen, perceived by them as being free of clutter and well organized. It should focus the user's attention on the task at hand and clearly display the minimal but necessary information needed to ac- complish the task. • Every effort should be taken to design a screen that is soothing to the eye. • The system should be designed so that the sequence of functions necessary for searching is obvious to the user. The interface should facilitate easy performance of tasks and provide nec- essary information in an appropriate format to support decision making by users. Its design should enable users to concentrate on their purpose for using the system, rather than having to struggle to apply its features. REFERENCES 1. Ben Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction (Reading, Mass.: Addison-Wesley, 1987), p.434. 2. Donald A. Norman, "Cognitive Engineering," in User Centered Design: New Perspectives on Human-Computer Interaction, eds. Donald A. Norman and Stephen W. Draper (Hillsdale, N.J.: Lawrence Erlbaum Associates, 1986), p.47. 3. Jenny Rogers, Adults Learning, 3d ed. (Philadelphia: Open Univ. Pr., 1989), p.40. 4. Norman and Draper, User Centered Design, p.361-75. Test Population Experience levels CD use Microcomputer use CD-ROM Software Evaluation 351 APPENDIX A DEMOGRAPHIC SUMMARY Undergraduate Students and Class Graduate Students 1990 9 1991 9 Naive Never use 24 60.0% 0 0.0% 1992 10 1993 3 Minimal 9 Sophisticated Use less than once a week Use once a week or more 15 37.5% 10 25.0% 2.5% 30 75.0% OCLC/AMIGOS Collection Analysis Systems Make a wise investment Choose from three options to analyze your library's data: Collection Analysis CD compares quantitative data BCL3 Tape Match measures against a standard Tape Analysis fits individual specifications Available exclusively from AMIGOS Bibliographic Council, Inc. 12200 Park Central Drive , Suite 500 Dallas, Texas 75251 214/851-8000 or 800/843-8482