Untitled-8 Online Catalogs from the Users� Perspective 403 403 Online Catalogs from the Users� Perspective: The Use of Focus Group Interviews Lynn Silipigni Connaway, Debra Wilcox Johnson, and Susan E. Searing In an attempt to elicit information from the users of the University of Wisconsin-Madison’s online catalog, Network Library System (NLS), the General Library System (GLS), conducted focus group interviews with undergraduate students, graduate students, and faculty. Under- graduate students tend to utilize subject searching capabilities. Gradu- ate students and faculty utilize subject searching only as a last resort; they typically search by known author or title. A significant number of the participants reported experience with library online catalogs other than NLS, although the majority of faculty reported very little experi- ence with other online catalogs. All the focus group participants, but particularly the undergraduate students, evidenced confusion between keyword searching and searching using controlled vocabulary. Inclu- sion of circulation status in the bibliographic record was identified as an important feature of the catalog. he University of Wisconsin- Madison conducted a partial evaluation of its online library catalog by holding focus groups of catalog users. The focus group interview technique was selected by the General Library System (GLS) to achieve an in-depth view of the use of online cata- logs by UW-Madison faculty and students and to elicit data not readily obtained by other methods, such as questionnaires or transaction log analysis. The objectives of the GLS study were to identify the strengths and weaknesses of the online catalog system from the users� perspective and to determine us- ers� priorities for improving the system. Users� expressed needs would later be weighed against technical feasibility, cost, and other factors to inform decisions about changes to the online catalog. The GLS used a combination of three meth- ods for gathering data. Focus group in- terviews were the primary method, supplemented by a question on a user survey and controlled �hands-on� com- parisons between catalog systems. This article concentrates on the user focus Lynn Silipigni Connaway is Director of the Library and Information Services Department, University College, at the University of Denver; e-mail: lconnawa@du.edu. Debra Wilcox Johnson is President of Johnson and Johnson Consulting in Waunakee, Wisconsin. Susan E. Searing is Visiting Professor of Li- brary Administration and Acting Library and Information Science Librarian, University of Illinois at Urbana/Cham- paign; e-mail searing@duit.wisc.edu All correspondence should be addressed to Lynn Silipigni Connaway. 404 College & Research Libraries September 1997 groups and describes the focus group process, discusses the study�s findings, and reports changes to the online cata- log since the study was completed. This interest in the study of the needs and behaviors of catalog users is reflected in the library and information science lit- erature. From 1930 through 1981, there were more than fifty studies of the use and users of traditional card catalogs.1 A num- ber of online catalog use studies have ap- peared in the library and information sci- ence literature since the 1980s. The origin of catalog use studies is credited to a 1930 paper written by William M. Randall. Randall contended that catalogs cannot be improved by studying catalog rules or the catalogs themselves but, rather, by study- ing catalog users� needs, backgrounds, and mental capabilities.2 Since Randall�s time, more than two hundred catalog use studies, incorporat- ing different methodologies, have been reported in the library literature. Re- searchers have used questionnaires, transaction log analysis, user observa- tions, individual and focus group inter- views, examinations of interlibrary loan (ILL) records, and combinations of these methods to investigate the behaviors, expectations, satisfaction, and needs of catalog users. The majority of the card catalog use studies employ the survey method, uti- lizing questionnaires completed by inter- viewers or respondents. Online catalog use studies also have been based on sur- veys. As the present study demonstrates, the focus group interview is a method conducive to the study of the needs, sat- isfaction, and expectations of online cata- log users, although it is not as frequently reported as the questionnaire. One of the most extensive online cata- log use studies did employ the focus group methodology. The study began in 1980 and was sponsored by the Council on Library Resources, Inc. (CLR).3 A to- tal of twenty-nine research, academic, community college, public, state, and federal libraries participated in the study, representing sixteen online catalogs. 4 �Much of our knowledge of online cata- log searching comes from these studies.�5 A questionnaire was used to examine pa- tron and staff experiences with online public-access catalogs (OPACs). In addi- tion to the questionnaire, OCLC con- ducted focus group interviews and ana- lyzed transaction logs. The focus group interviews conducted by OCLC reported that users want dif- ferent search modes that correspond to �different levels of patron experience with the system.�6 The focus group par- ticipants also suggested computer-as- sisted instructions and machine prompts for step-by-step instructions when using OPACs. A computer system that identi- fied the errors made by the searcher (i.e., why errors occurred and how to correct them) was desired. Because it is difficult to select the appropriate subject headings, OPAC users wanted online subject head- ing displays, as well as subject and name cross-references.7 The CLR study indicates that users of OPACs wanted more online prompts and dialogue with the system. The investiga- tors also report the users� desire to execute their searches in terms of the system. The results of the study indicate that users would like the system to give instructions to limit searches if too much information is retrieved and to give them other terms to search if no information is retrieved.8 The findings of the CLR study affected the design of online catalogs. Not all of the desired features have been perfected, and increasing sophistication of OPAC users raises expectations of the system. Despite the time difference, the CLR stud- ies and the GLS findings are parallel. The CLR study indicates that users of OPACs wanted more online prompts and dialogue with the system. Online Catalogs from the Users� Perspective 405 Focus Group Interview Method The focus group interview method is well documented as an evaluation and re- search technique in the literature across disciplines. By the late 1930s, social sci- entists recognized the limitations of pre- determined, closed-ended questions. They began to adopt a nondirective in- terview technique that used open-ended questions, allowing the interviewee an opportunity to comment, explain, and share experiences and attitudes. The fo- cus group method initially was devel- oped to meet problems growing out of communications research and propa- ganda analysis.9 Focus group interview- ing was used extensively during World War II as a means of increasing morale in the military. The technique has been used widely in marketing research since the 1950s.10 However, despite its initial introduction in sociological research and approximately forty years of successful application in marketing, the focus group interview technique is under utilized in the social sciences.11 The focus group interview is an in- depth, face-to-face interview of a purpo- sive sampling of a target population. The group consists of eight to twelve people chosen for homogeneity in background, though not homogeneity in attitude to- ward the topics discussed in the inter- view. A moderator asks three to six ques- tions, guiding the discussion but remain- ing neutral. The questions are open- ended, which gives the moderator the flexibility to probe and explore topics as they emerge from the interactions. The focus group interview permits assess- ment of nonverbal responses and reveals group interaction patterns. The data are captured by a tape recorder or a note taker.12 Focus groups can provide in-depth feedback on issues and ideas, and can give a detailed qualitative analysis of the needs used in libraries to evaluate pub- lic services and collections. Focus group interviews have been used in academic, newspaper, hospital, public, and state li- braries to gather information on users� perceptions of library services and col- lections.13 A public library system utilized the methodology to gather information on the lifestyles of senior citizens in or- der to identify barriers to, and methods for, increasing library use by the elderly.14 The New York State Education Depart- ment used the focus group interview methodology in conjunction with other methodologies to evaluate advisory and information referral services.15 Mellon incorporated focus group interviews in the evaluation of bibliographic instruc- tion by undergraduate students.16 The methodology can be and has been used to gather information about the work and beliefs of practicing librarians. For instance, reference and technical ser- vice librarians participated in focus group interviews to identify their percep- tions of the need for authority control in online systems.17 The focus group inter- view technique also has been used in li- braries to evaluate online searching by end users, as well as in the research and development of OPACs.18 In Wisconsin, academic and public library technical ser- vices managers and catalogers partici- pated in focus group interviews to assist the researchers in developing hypotheses for an extensive study of the decisions made and tasks performed by technical services managers and catalogers.19 The technical services managers and catalog- ers were asked to describe the typical day on the job, the decisions made during a workday, and how the job has changed in the previous two years. The focus group interview methodology was used as an exploratory approach in this in- stance to develop a list of tasks performed and decisions made by technical services managers and catalogers. The Focus Group Method Applied The Madison campus is the largest in the University of Wisconsin System, home to 117 academic departments, 40,005 stu- 406 College & Research Libraries September 1997 dents, and 2,284 faculty. The campuswide online catalog includes records for thirty- seven libraries, many, but not all, of which are administered by the GLS. The locally developed software for the cata- log is named Network Library System (NLS). It has been the UW-Madison�s online system since the early 1980s. 20 Until recently, NLS also was used as the name for the catalog. Because other da- tabases now have been loaded under the NLS software, the catalog has been given a nickname�MadCat. The NLS software is characterized by a natural language approach to search- ing and greater sophistication than many online catalogs. Its flexibility is evident in four key areas of functionality: 1. Search methods: NLS is searchable in keyword and linear modes. One may search by keyword using the Boolean operators (and, or, not), by key phrase using a string of words (i.e., implied ad- jacency), or strictly left to right, first word first, using the = sign. Furthermore, sev- eral search techniques can be combined into a single search statement such as pe=journal of and education* and (his- tory or philosophy). These techniques can be used in a global keyword search or in combination with one or more search commands. 2. Search commands: NLS offers a wide range of search commands, including the usual (author, title, subject) and the less common (series, periodical title, contents notes, conference name, publisher, and others). These commands may be linked using the Boolean operators in a single search statement. At the time of the re- ported study, a search statement in which no qualifier was specified would search for the specified words within any single indexed field of the record. Users had the option, by explicitly issuing a com- mand, to search across ALL fields. Limit features in NLS enable the retrieval of items by language, date, form, or library location. At the time of the study, limits could be invoked only after a search had been done and a group of entries re- trieved. 3. Display options and maneuverability: NLS preserves the results of a user �s searches throughout the session and al- lows the user to move easily both forward and backward among the results. Users may display selected items, a range of items, or all items retrieved in any of the formats available for display. Having done this, users always may return to a previ- ous display. Sets can be retrieved at any time for redisplaying, limiting, or com- bining with other sets or terms. 4. Special features: NLS has a number of special features, including the �cite� command, which permits a branching search from the added entries and/or subject tracings in a record; and the �pro- file� command, which retrieves data on individual libraries on campus. University library staff were excluded from the focus group study. Librarians are sophisticated catalog users, many of whom contributed to the design of the NLS software. They are not typical users in the same sense as the nonlibrarians in the study. As library staff expressed in- tense interest in the focus group process, however, it became clear that they too needed a forum for expressing their needs and opinions, as well as their pro- fessional assessments of users� needs. Instructions for accessing online catalogs at other universities via the Internet were distributed, and ten staff members re- sponded with thoughtful comparisons that amplified and complemented the findings of the user focus group study. The focus group process consists of several steps: recruitment of inter- viewees, the interview process, data analysis, and reporting of the findings. Each of these steps is discussed below, along with pros and cons of the process as a tool for evaluating online catalogs. Recruitment of Interviewees The first step in recruiting participants was to determine how many groups were Online Catalogs from the Users� Perspective 407 needed, their ideal size, and the desired characteristics of the group members. The focus group methodology is not based on random sampling but, rather, on a careful selection of participants to represent certain dimensions of the �mar- ket� for a product or service. A well- rounded study required input from a broad range of catalog users who spe- cialize in different disciplines and use various libraries as their primary gate- ways to campus library resources. Separate groups of faculty, graduate students, and undergraduate students were organized, reflecting assumptions that their patterns of catalog use and their visions of the perfect catalog would dif- fer, and perhaps more important, that mixing statuses within focus groups might affect the conversational dynam- ics adversely. One of the advantages of the focus group method is the synergy of the group, and the literature strongly suggests that the more homogenous the group members, the more freely the dis- cussion will flow. An effort was made, however, to rep- resent a spread of disciplines in each fo- cus group. Participants were drawn from thirty-six departments. It should be stressed that disciplinary differences in catalog use were not under investigation. Rather, the in- clusion of many disciplines enabled gener- alization about the use of, and satisfaction with, the NLS catalog regardless of the user �s disciplinary training. With these criteria in mind, lists of potential participants were compiled. The supervisor of the Catalog Informa- tion Desk, reference librarians, subject bibliographers, and branch librarians rec- ommended people, largely faculty, whom they knew to be frequent users of the NLS catalog. Circulation staff, from personal memory, identified graduate students who were heavy borrowers of library materials. Faculty and graduate students who were invited to participate were asked to recommend others. Word of mouth worked well to solicit faculty and graduate student participants. The chairs of the University Library Commit- tee (a faculty/student advisory commit- tee) and the University Committee (the executive committee of the faculty) also provided leads to potential participants. Recruiting undergraduate students, however, proved more difficult. Al- though librarians interacted frequently with undergraduate students, names were not exchanged. Faculty were more likely to recommend graduate students than undergraduate students. Lists of undergraduate students in honors pro- grams and students who serve as advi- sors in the campus Students Orienting Students program turned out to be fruit- ful sources. Contacts through the Wiscon- sin Student Association and the Dean of Students office yielded a few more par- ticipants. Less successful recruitment strategies included sign-up flyers left near NLS stations in three of the larger libraries and an announcement in the �news� file of NLS. A few willing volunteers were ex- cluded from the study because they had insufficient familiarity with NLS. In gen- eral, a person was considered to be fa- miliar with the system if he or she had used it within the past two to three months. This was gleaned through con- versational screening questions. Because the researchers sought typical users, not necessarily experts, some people who questioned their own knowledge of NLS turned out, upon probing, to be accept- able candidates for the focus groups. When people declined, it was prima- rily because of scheduling conflicts. So many undergraduate students found the One of the advantages of the focus group method is the synergy of the group, and the literature strongly suggests that the more homogenous the group members, the more freely the discussion will flow. 408 College & Research Libraries September 1997 prescheduled daytime sessions inconve- nient due to school or job priorities that the undergraduate groups were resched- uled in the evening. As recruitment pro- ceeded, it became clear that faculty and graduate students were willing to par- ticipate simply in order to be of use to the university, but undergraduate stu- dents (particularly busy honors students) needed an added incentive. Therefore, arrangements were made to pay students ten dollars for the one-and-a-half-hour group meeting. Once payment was promised, undergraduate students sug- gested the names of friends and class- mates, and the sessions quickly filled. Approximately 180 phone calls yielded fifty-nine people committed to participate, in groups ranging from seven to eleven members. (With greater lead time, the researchers would have aimed for six groups of twelve, or a total of sev- enty-two, on their rosters.) Letters were sent to all participants a few days before their focus group session, reminding them of the time and place. The letters also announced opportunities to com- pare NLS to other online catalogs at scheduled times in a library microcom- puter lab. A few group members took advantage of this. One or two requested instructions for accessing other catalogs themselves through the Internet. This optional preparation for the focus group sessions was offered in lieu of time-con- suming demonstrations during the ses- sions themselves. Also enclosed with the reminder was a one-page fact sheet about the project and the focus group method. Last-minute dropouts and no-shows brought the final number of participants to forty-nine. The group was composed of fifteen faculty, fourteen graduate stu- dents, and nineteen undergraduate stu- dents. Roughly speaking, four contacts were required to produce one participant. Interview Process Essential to the interview process is an interview guide. A guide helps organize the discussion as it moves from general questions to more specific ones. This �funnel� pattern aids participants to move from very general comments to more specific examples and answers. Three to five questions are used to focus the discussion, with numerous probes used to encourage a broad range of responses and address spe- cific concerns of the library. One mistake that occurs in focus group interviewing is the use of too many specific questions, rather than a limited number of guiding ques- tions and in-depth probes. The interview is designed first to build rapport among the group members and with the interviewer. As each person an- swers the initial, warm-up question, the interviewer begins to evaluate each participant�s nature. Although the writ- ten interview questions provide the framework, the process requires a very natural conversational approach, with shifts in the discussion appearing to oc- cur as a logical result of the participants� comments. Because each group has a dif- ferent dynamic, the interviewer needs to remain flexible to accommodate the natu- ral flow of the discussion while cover- ing all the topics of interest. It is the interviewer�s responsibility to ensure that everyone in the group re- sponds in each of the question areas. He or she does not provide information or correct misunderstandings. Individuals within the group sometimes begin to query each other in the areas of interest, which assists the interview process. The interviewer can use more talkative group members to draw out less vocal ones. No one can accurately predict the nature of a group�s interaction prior to the inter- view, but use of an interview guide sets the necessary parameters for gathering comments from all participants and en- suring that the needed information is gathered during the process. The following questions were used to guide the GLS interviews. The probes are not included here. The interview was designed in consultation with GLS rep- Online Catalogs from the Users� Perspective 409 resentatives and pretested with a small number of students. 1. Please briefly describe what library catalogs you have used and how often you use them. [warm-up question] 2. For what purposes do you use the library catalog? 3. What features of NLS do you find useful? 4. Which features of UW-Madison online catalog, NLS, cause you difficulty or are lacking altogether? 5. What characteristics do you think are the most important in an online catalog? Six interviews were conducted over six consecutive days in January and Feb- ruary. Two groups each were used for undergraduate students, graduate stu- dents, and faculty. A faculty member from the UW-Madison School of Library and Information Studies conducted the interviews, and a doctoral student in the school�s program participated as note taker. Following each interview, the in- terviewer and note taker reviewed the notes for completeness. A tape recorder was not used for these sessions. Each in- terview lasted an hour and a half, and each group covered all the questions, al- though the order of the main questions varied depending on the natural flow of the discussion. Following the interviews, notes from all six sessions were read through at one time, and the themes that emerged from the data were listed. Using these themes, the participants� responses were labeled. This step helps to ensure that there is evidence to support these themes. With some refinement, the themes were used to organize the final report. Responses under each section were further subdi- vided by type of participant�under- graduate, graduate student, and fac- ulty�and findings across groups were summarized. Data Analysis Generally, the information acquired from focus group interviews is used to help researchers understand perceptions and attitudes of the target population. The results of focus group interviews cannot be used to generalize to an entire popu- lation. Instead, the results provide an op- portunity to consider a range of re- sponses. The focus group interview permits assessment of nonverbal re- sponses and reveals group interaction patterns. The researcher must analyze the data acquired from the moderator �s re- ports and taped interview sessions. The analysis begins with getting an overview or global picture of the entire process, and involves consideration of words, tone, context, nonverbal communica- tions, responses, and i deas of the interviewees. There are two basic approaches to ana- lyzing focus group data. The ethno- graphic summary is a qualitative ap- proach that relies on direct quotations of the group discussions. The content analy- sis approach produces numerical de- scriptions of the data. Content analysis is the tallying of mentions of very spe- cific factors. Mentions can be brief or very extensive and can be weighted. The eth- nographic summary and the content analysis approach are not conflicting means of analysis. The combination of the two approaches brings an additional strength to the analysis.21 Ethnography involves establishing rapport, selecting research participants, transcribing observations and conversa- tions, and keeping diaries. Although Clifford Geertz agrees that none of these techniques or procedures adequately de- fines the venture, he believes ethnogra- phy is defined by the kind of intellectual effort it is: �an elaborate venture in �thick description�.�22 The experiences or events are described in such detail that the The ethnographic summary and the content analysis approach are not conflicting means of analysis. 410 College & Research Libraries September 1997 reader can experience the total event as if he or she actually had been involved in or observed it. The data collected from focus group interviews also can be defined by the in- tellectual effort. The data are �thick de- scriptions� and must be interpreted in a manner that retains this richness. The written observations and descriptions should include exact phrases used by the participants, and should depict their intonations and the group and individual interactions and reactions. This thick description is one of the ad- vantages to the focus group interview methodology. One of the recurring questions that arises with the discussion of focus group interview methodology is that of its va- lidity and reliability as a research meth- odology. Face validity is the degree to which the procedure really measures what it proposes to measure. It is per- haps the weakest form of validity be- cause it rests on a single variable.23 If the researcher deviates from the established procedures outlined above, and if the research questions do not lend them- selves to focus group interview method- ology, the focus group interviews are in- valid. The focus group interview meth- odology is like other social science meth- odologies where validity depends not only on the procedures used, but also on the context within which the procedures are used.24 The validity of the analysis of the data collected during the focus group inter- views is another concern. If content analysis is used as a method of data analysis, �the validity between the classification schemes, or variables derived from it, and the validity of the interpretation relating content vari- ables to their causes or consequences� is crucial.25 This means that in the dis- semination and interpretation of the results, the researcher must ascertain that the findings are not generalizable beyond, nor dependent upon, specific methods, data, or measurements outside the specified study.26 Reliability is divided into three types with regard to content analysis: stability, accuracy, and reproducibility. Stability can be determined when the same data are coded more than once by the same coder with the same results.27 Accuracy can be determined by the extent to which the classification of text corresponds to a standard. Standard codings for text are infrequently available, so accuracy in re- liability seldom is determined by re- searchers.28 Reproducibility, or intercoder reliability, can be determined when the same data are coded with the same re- sults by more than one coder. This is im- portant in content analysis because it measures the consistency of understand- ings or meaning held by two or more coders.29 Intercoder reliability should be determined when analyzing focus group interview data. If one person examines one set of transcripts and another con- centrates on a different set of transcripts, two perceptions and perspectives of dis- covery also can be included in the report- ing of the results.30 Both researcher and recorder analyzed the data using both the ethnographic summary and content analysis ap- proaches. There was a high level of reli- ability in the interpretation of the results of this study in relation to stability and intercoder reliability. The researchers also acknowledge that the findings of this study are not generalizable to a larger population. Reporting of the Findings The study centered on the NLS online catalog, which is installed only at the Madison and Milwaukee campuses. Many of the findings, however, may re- flect general opinions about online cata- logs and their interfaces. These will be highlighted in this article. Responses con- cerning the informational content of the catalog and local service policies are, for the most part, omitted from this report. Online Catalogs from the Users� Perspective 411 Experience with Using NLS and Other Catalogs A significant number of the participants reported experience with library online catalogs other than NLS. Among under- graduate students, 84 percent had used other OPACs; among graduate students, 71 percent. The majority of faculty, how- ever, had experience only with NLS; only 27 percent had used other online catalogs. The majority of faculty and graduate students reported accessing NLS re- motely from their offices or homes via modem, whereas only two undergradu- ate students (10%) did so. Approaches to Searching Undergraduate students differed from graduate students and faculty in the way they approached NLS. Undergraduates tend strongly toward subject searching. After retrieving records for materials of interest, they proceed to the shelves and browse for additional materials in the same classification. Some students exam- ine subject terms in the catalog record (displayed only when a �full� display format is specified) to refine their searches. Graduate students and faculty, on the other hand, report utilizing subject searching only as a last resort. Graduate students typically have an author or title in mind; after searching NLS to deter- mine a known item�s call number, they browse the stacks for related items. Like- wise, faculty usually search by known author or title, resorting to subject searches only when working outside their own fields. Faculty reported that the standard LC subject headings are too broad to pinpoint their specialized re- search interests. All the focus groups, but undergradu- ate students particularly, evidenced con- fusion between keyword searching and searching using controlled vocabulary. Many users reported employing the �su� qualifier, which limits searching to the subject tracings that conform to LCSH or MESH, when a keyword search may have been more appropriate. Positive Feedback on the Catalog The library recently had linked a display of circulation status to the bibliographic records in NLS; this was singled out as one of the system�s best features. Users also praised the availability of worksta- tions at many campus locations and dis- tributed points within the larger librar- ies, as well as the dial-in capability. Printed documentation, although out- dated at the time of the study, was per- ceived as the best guide to using NLS. The helpfulness of library staff also was praised by faculty and graduate students. Barriers to Use of the Catalog NLS is a sophisticated program with powerful search features found in few online catalogs at the time this study was conducted. However, the focus group study intentionally spotlighted difficul- ties users had in interacting with the sys- tem. Students and faculty alike criticized the catalog as �illogical,� �counter-intui- tive,� and �intimidating.� The command- driven system was seen as confusing and cumbersome. At the time of the study, the system was accessed within the li- braries from dedicated �dumb� termi- nals. Most users found the system annoy- ingly slow. The libraries had begun to support end-user searching of other bib- liographic resources on CD-ROM, and NLS suffered in comparison to commer- cially produced systems, which users found easier to manipulate because they are simpler systems with fewer features and more online prompts. Although the information content of the catalog was not evaluated in this Graduate students and faculty, on the other hand, report utilizing subject searching only as a last resort. 412 College & Research Libraries September 1997 study, faculty in particular stressed their unmet need for access points to chapters within edited books and proceedings. Be- cause retrospective conversion of the card catalog is not yet completed, users are uncertain about the online catalog�s cov- erage of older materials. Characteristics of the Ideal Catalog Although the faculty, graduate students, and undergraduate students often ech- oed each other �s comments, their ap- proaches to the catalog and the impor- tance they attached to elements of online catalog performance varied. Undergraduate Students. Quick sys- tem response time topped the list of characteristics undergraduate students considered important in an online catalog. As noted above, undergraduate interviewees reported good experiences with CD-ROM bibliographic databases and wished that NLS included some of the features common in CD-ROM prod- ucts, such as an online thesaurus of sub- ject terms, menu-driven search protocols, natural language queries, and user-de- fined display and print options. They also desired to access CD-ROM library re- sources and the catalog of the Madison Public Library from NLS workstations. Undergraduate students recommended improving NLS�s online help features. In particular, they wanted context-specific help rather than general messages sug- gesting they consult a librarian. They also suggested split screens that would dis- play the original search and the help message side by side. Undergraduates suggested enhancing the system with maps indicating the lo- cation of materials. They also expressed the desire to renew materials and recall items in circulation via NLS. Finally, they called for more printers at library work- stations. Graduate Students. Graduate students echoed many of the undergraduate stu- dents� comments, including: the need for more detailed, prescriptive help screens; access to journal indexes and abstracts; the inclusion of maps to aid in finding materials within libraries; and greater availability of printers. They also desired the capability of reserving, renewing, and recalling items online. Heading the graduate students� list of the most important characteristics of an online catalog were better approaches to searching by subject, including call num- ber searching. Graduate students also desired the display of author and title cross-references and subject hierarchies, and the automatic matching of incorrect headings to LCSH terms. Graduate students recommended dif- ferent search options for naive and so- phisticated users, suggesting a menu- driven system utilizing icons and a mouse or touch screen for new users. Experienced users are comfortable with a command-driven system but wished for simpler commands and greater ease in limiting searches (e.g., by date or lan- guage). The ability to download searches and a feedback mechanism for reporting cataloging errors or problems with the system also characterized the ideal on- line catalog in the minds of graduate stu- dents. Faculty. Like graduate students, fac- ulty reported searching NLS primarily by author and title, and likewise wished for cross-references for names and titles to be displayed. Despite their reliance on known item searching, however, they echoed the graduate students in calling for the addition of a call number search feature, an online hierarchy of subject headings, and less cumbersome ways to limit retrieval (e.g., by language or date). Faculty participants in the focus groups shared student perceptions that the ideal catalog would incorporate pre- scriptive help screens and different �lev- els� of searching for naive and experi- enced users. Like students, they praised the software of CD-ROM products and felt that the online catalog interface should emulate such systems and, fur- Online Catalogs from the Users� Perspective 413 ther, that CD-ROM products should be accessible from NLS terminals. Faculty suggested a few features that were not mentioned by the graduate or undergraduate students, including the display of diacritics in foreign-language entries. They also envisioned a catalog that would �translate� abbreviations of journal titles. Like other groups, faculty faulted the printing capability of NLS (at that time limited to screen prints) and called for better downloading capabilities, includ- ing the ability to send search results to e- mail accounts. They shared the students� wish to reserve library materials and to recall circulating items when consulting the catalog. They also wanted access to the catalogs of other research libraries. Summary of Key Findings All the participants found the incorpo- ration of circulation status and the due date of materials on NLS very useful. In addition, there was overall agreement that NLS is better than using the card catalog, largely because NLS can be ac- cessed from remote sites via modem and is perceived as faster. Users are satisfied with author and title search capabilities. Library staff were described as �very helpful� in assisting patrons in retriev- ing information on NLS. The faculty and graduate students do not hesitate to ask library staff for assistance, but under- graduate students reported that they rarely, if ever, ask for assistance with NLS. Nearly two-thirds of the participants had used online catalogs other than NLS, and most students had used other online databases. Overall, the undergraduate students had accessed more online cata- logs than either graduate students or fac- ulty. However, certain difficulties with NLS were expressed consistently. Focus group participants described NLS as �intimidat- ing,� �antiquated,� and �counter-intui- tive.� For example, users cannot limit their query by language, date, or format until a search has been executed and a result set is retrieved. Users are not told why the search retrieved 0 or 500 items, nor are they prompted on how to broaden or narrow the search. Specific errors are not identified and accompa- nied by instructions to correct the errors or refine the search. All the focus group participants complained of having diffi- culty retrieving materials when subject searches were used. The focus group participants recom- mended a variety of enhancements. Among the most strongly recommended were: l more prescriptive and explicit help screens or online prompts; l subject cross-references and an on- line list of terms or thesaurus to narrow or modify searches; l option to combine multiple in- dexes or to be able to limit before the ini- tial search; l more intuitive search commands; l acceptance of variant forms of commands; l display items in brief format im- mediately after the items are retrieved (default command); l ability to scroll; l option to browse by call number; l diacritics in the records; l new functions and changes publi- cized on initial screen; l improved quality control of the records; l two levels of search capabilities� a novice user approach that is menu driven, and a sophisticated, command- driven approach. One of the uses of focus group inter- views is to elicit a broad range of re- The faculty and graduate students do not hesitate to ask library staff for assistance, but undergraduate students reported that they rarely, if ever, ask for assistance with NLS. 414 College & Research Libraries September 1997 sponses on a given topic. Consensus usu- ally is not an intended product of the process. What was most surprising from these interviews was the consistency of the responses�so much so that consen- sus was virtually achieved. Similar prob- lems and suggested enhancements were found across all groups. This consistency lends more weight to the findings. The Focus Group Method: Pros and Cons How effective are focus groups as a method for evaluating an online catalog? At the UW-Madison, focus group inter- views gleaned information that might have been difficult to obtain otherwise; still, they did not tell the whole story. To assess the benefits and costs, the research- ers examined the focus group experience along three dimensions: What was mea- sured? What resources were invested? And how credible were the results? What Was Measured? The focus groups measured user satis- faction with the NLS online catalog. The group leader asked participants to report their experiences searching NLS; thus, the researchers gained some background on user behavior as well. Finally, by prob- ing to determine characteristics users value in the NLS catalog and features they wished NLS offered, the research- ers learned about user expectations. It is possible to measure these vari- ables using other methods. In fact, a sur- vey undertaken in three of the larger li- braries a few weeks after the focus group interviews also measured user satisfac- tion with the NLS catalog. The survey was a snapshot of user opinion about a wide range of library resources and ser- vices, intended to identify areas needing further investigation. The respondents were selected randomly from users en- tering the three libraries at predetermined days and times. NLS was included in a list of library resources that respondents were asked to rate on a scale of 1 (low satisfaction) to 5 (high satisfaction). NLS earned an over- all rank of 3.9, somewhat lower than the average rating for all items. More telling, perhaps, NLS was the third most fre- quent topic for written comments on the survey�behind library hours and pho- tocopiers, but ahead of building ameni- ties, collections, and circulation policies. Nearly all written comments were nega- tive in tone. Although the survey corroborated the staff�s impression that students were not highly satisfied with NLS, it did not tell why. The volunteered written comments did not provide enough detail about user needs and expectations to suggest spe- cific changes. By contrast, in the focus group interviews, the leader probed for the reasons behind the participants� atti- tudes. This is a decided advantage of fo- cus groups over questionnaires, which can only provide answers to the ques- tions asked. A possible disadvantage of the focus groups is that they did not incorporate any hands-on components or demonstra- tions. The idea of conducting searches of NLS or other OPACs during the inter- view sessions was rejected for several reasons. Interactions with the machine could disrupt the flow of discussion, and complications could occur if the system went down (as has happened with dial- in access during library instruction classes). Moreover, by relying on their past experiences with NLS, users natu- rally would recall the features and func- tions that mattered most to them. Finally, the time needed to do both the demon- stration and the interview would double the time commitment of users and po- tentially reduce participation. As it turned out, participants, especially the students, brought experiences with other online catalogs into the discussions. A hands-on methodology was part of a small concurrent study of faculty opin- ions of NLS, undertaken by the UW- Madison Health Sciences Libraries. Ref- Online Catalogs from the Users� Perspective 415 erence librarians were teamed with eight faculty members from several health-re- lated disciplines. Side-by-side worksta- tions were set up to search the NLS cata- log and Michigan Research Library Net- work (MIRLYN), the NOTIS-based cata- log at the University of Michigan. Michi- gan was selected because it has com- pleted retrospective conversion, is simi- lar in size, and has a medical school. During the first half hour, the faculty conducted their own �typical� catalog searches on each system. During the sec- ond half hour, the librarians demon- strated specific functions they consid- ered important (e.g., author search, au- thor/title search, subject search, periodi- cal search). These examples were selected in advance and pretested. The faculty then completed a brief form to record what they did and did not like about each system, as well as their general com- ments. The Health Sciences Library study deliberately focused on the basic features of each OPAC in order to draw out fac- ulty reactions to what librarians per- ceived as each system�s strengths and weaknesses. In general, faculty did not voice great enthusiasm for either system. They found MIRLYN somewhat friend- lier and praised its keyword search (with implicit Boolean and), automatic right- hand truncation, and various elements of screen design. Although they stated that NLS required users to remember too many different commands, and their searches revealed confusion about name qualifiers in particular, they appreciated the �pe� search that makes it easy to lo- cate periodical titles in NLS. Several par- ticipants wished for a feature that neither system offered�a display of �closest match� for misspelled words. The Health Sciences Library study was designed, in part, to provide a reality check by comparing NLS�s performance not to abstract notions of a �good� cata- log but, rather, to an actual system in place at a peer institution. Actually, com- parisons were implicit in the focus group study as well. Nearly two-thirds of the participants in the focus groups, students included, had used other online catalog systems in academic and public librar- ies. They evaluated NLS in light of their familiarity with other catalogs and with other electronic bibliographic databases, including CD-ROMs. None of the methods employed at UW-Madison�focus groups, written surveys, or hands-on comparisons� overtly aimed to assess users� skill in searching, the impact of formal user training, the utility of point-of-use hand- outs, the effectiveness of online help screens, or any number of relevant fac- tors that might be measured with experi- mental or observational methods. Be- cause these and other factors obviously influence online catalog users� expecta- tions, behaviors, and satisfaction levels, however, some were raised by partici- pants in the focus groups. By zeroing in on user expectations and satisfaction, the focus groups helped to identify specific concerns for research and action. Other research methodolo- gies corroborated and complemented the focus group interviews, but the groups yielded information that might not have been gleaned otherwise. Ultimately, the usefulness of the findings for decision making helped to confirm the value of the focus group approach. What Resources Were Invested? Although the GLS did not attempt a rig- orous cost-benefit analysis of the project, an understanding of what is entailed, in outlay of both time and money, to un- dertake focus group interviews was gained. The up front costs were low, in large part because the group leader, Debra Wilcox Johnson, donated her time. Her familiarity with the UW-Madison online catalog meant that initial consultations could be relatively brief. Had outside consultants been employed, a longer lead 416 College & Research Libraries September 1997 time probably would have been needed to plan the interviews, along with signifi- cant fees. The budgeted items in the project were limited to hourly wages for the research assistant, payment ($10) to undergradu- ate participants, and refreshments. Untracked incidental expenses included photocopying, postage for reminder no- tices and post interview thank-you notes, and clerical assistance with scheduling rooms. Because this was the GLS�s first expe- rience with the focus group methodology and its first systematic attempt to gather user feedback on NLS, the deputy direc- tor elected to shoulder the work of recruit- ment herself. Given the significant amount of time invested in recruitment, the benefit of having a high-level admin- istrator directly involved must be ques- tioned. Most undergraduate and gradu- ate students probably would have re- sponded equally well to an invitation from someone else. However, faculty seemed to appreciate hearing from a li- brary administrator. Several faculty mem- bers who could not participate in the groups nevertheless shared their opinions of NLS and other library services with the deputy director. How Credible Were the Results? Although the validity of the focus group method has been proved in other fields, particularly marketing, it is still viewed with skepticism by some members of the library profession. In advance of the group interviews, some library staff were worried that the groups would elicit anomalous results because the partici- pants were not a random sample. Some staff articulated concerns about the vagar- ies of group dynamics. There was specu- lation that the focus groups would merely serve as �gripe sessions� in which old complaints about the library would be rehashed by perennial malcontents. Still others believed that, even if the group interviews yielded insights, the lack of �hard� (i.e., quantitative) data would preclude basing management decisions on the findings. To assuage these con- cerns, a one-page fact sheet about the focus group project was distributed to staff. Furthermore, the library adminis- tration stressed that one of the project�s objectives was to evaluate the focus group method itself as a tool for deci- sion-making. When Johnson and Connaway pre- sented the findings to an assembly of li- brary staff at the conclusion of the project, the audience did not challenge the validity of their conclusions. If any- thing, the findings appeared self-evident to many and confirmed their own ob- servations. Librarians� professional judg- ments of user satisfaction and expecta- tions matched closely what users them- selves reported, and the improvements and new features desired by users par- alleled the library staff�s wish list of cata- log enhancements. Reinforcing the credibility of the re- sults was the unusual consensus that emerged across the faculty, graduate stu- dent, and undergraduate student groups. As plans for the future of the online cata- log are made, it is useful and reassuring to have evidence of the congruence of users� and librarians� viewpoints. Some library staff were troubled by the study�s emphasis on NLS�s weak- nesses. An evaluative study aimed at improving a system will inevitably stress its deficiencies. Also distressing was the evidence that users often misunderstood or under utilized the system�s advanced features. However, this information was useful to librarians who conduct classes and workshops on library resources. Besides the findings about user opin- ions of NLS, the focus group study achieved other benefits for the library. First, it had a positive impact on the library�s public relations. The mere ex- istence of the NLS user study signaled the library�s interest in listening to user needs. That the faculty were singled out Online Catalogs from the Users� Perspective 417 as a distinct user category was a political plus. On the negative side, the focus groups may have fostered unrealistic ex- pectations on the part of some partici- pants about the nature and speed of sys- tem enhancements, although no major problems arose from this situation. Second, the study pinpointed other services that should be improved. For example, the number of users who de- scribed accessing NLS from homes and offices spotlighted the urgency of ad- dressing education and training for re- mote users of library resources. Third, it reaffirmed the value faculty and graduate students place on knowl- edgeable library staff, whom they char- acterized as �very helpful� in making full use of the catalog�s capabilities. Evalua- tive research and a push for quality im- provement can generate anxiety among staff and a concern for their future role in the organization. As the library under- takes to assess its services and resources, with inevitable implications for work flow and staff assignments, positive feed- back on staff-user interactions benefits everyone. Poststudy Changes to the Online Catalog Many changes have been made to NLS since the focus group interviews were conducted, although not all the user con- cerns have been addressed. At the time of the interviews, work was under way to migrate the system from outmoded Series I IBM minicom- puters to an RS6000 UNIX platform. This technical work consumed the program- mers for many months, while work on the user interface was deferred. Several im- portant improvements resulted from mi- grating to client/server technology. First, and most immediately commented upon by users, was a huge leap in the speed at which the system responds to commands. Whereas the focus group interviewees complained that NLS was �too slow,� us- ers today remark on its speed. Searches on common terms, such as �history,� no longer terminate with a message that the search will take too long. Second, the migration included the re- placing of �dumb� catalog terminals with multipurpose PC-type workstations. Ini- tially, users were greeted with a main menu that listed the library catalog, re- named MadCat, first. Through the menu structure, users connected to library cata- logs at other institutions, journal and in- formation databases on local area net- works (LANs), remote resources such as CARL and Nexis, the campuswide infor- mation system (WiscInfo), and the Inter- net. This suite of information resources was labeled the �Electronic Library.� Third, the library began running the Wilson databases under the NLS soft- ware. This was done for several reasons: to alleviate the load on the CD-ROM LAN, where heavy use was negatively affecting system performance; to provide unlimited access on campus to some of the most popular electronic indexes; and to offer a better alternative to the Wilsondisc software, which many librar- ians and users found cumbersome. Fourth, the commands were simpli- fied in conformity with the common command language of NISO Z39.58. The �find� and �display� commands are now implied; users need no longer type �f� or �d� for most searches. Confusing field labels were replaced; for example, one now enters �au� for an author search, instead of �n;a� for name-as-author. Fifth, direct searching of limits�lan- guage, date, form, and library location� was made available. This improvement enables the user to search directly for sound recordings in French in the under- graduate library, for example. Previously, the user would have been required to conduct an author, subject, or title search to retrieve a group of entries before in- voking limits. Sixth, user requests for printers were answered by the installation of laser print- ers at selected workstations. The printers 418 College & Research Libraries September 1997 are operated with the same debit cards that operate the library�s photocopiers. Seventh, the capability of download- ing search results to a diskette or send- ing them to an e-mail account was added to the system. Because all entering stu- dents now acquire e-mail accounts, this feature is very popular. Eighth, the desire for different inter- faces for novice and accomplished users was satisfied by the development of a graphic user interface that incorporated pull-down menus and other conventions familiar to Macintosh and Windows us- ers. Finally, although context-sensitive help screens remain an unmet need, con- siderable progress was made in making documentation available through the Electronic Library and the library�s go- pher and World Wide Web server, includ- ing the text of a comprehensive NLS us- ers� manual.31 In the fall of 1994, a working group was formed to recommend further pri- orities for system enhancements. Based on the focus group study and subsequent consultations with library staff and pro- grammers, the working group proposed several new features that have since been implemented, including: commands for searching by classification numbers and other standard numbers (such as ISBN, ISSN, OCLC, and government document numbers); a call number scan that mim- ics a shelf list search; expanded indexing of note fields; and internal and left-hand truncation. The default search was changed so that a keyword search now searches across all fields, rather than searching for occurrences of the specified words within the same indexed field. This is consistent with how most focus group participants seemed to believe the keyword search worked. The addition of subject authority files to create cross-ref- erence displays was recommended but has not yet been implemented. In 1995, the Electronic Library menu was replaced by a Web page (http:// www.library.wisc.edu). This approach guides users not only to MadCat and other bibliographic databases, but also to locally prepared home pages for indi- vidual libraries and to pages that orga- nize resources by discipline. A new graphical interface for MadCat, based on OCLC�s WebZ software, will be released in summer 1997. Improved help features will be incorporated into the web inter- face, thus responding to another need identified in the focus groups. Most recently, the findings of the NLS focus group study were consulted in the process of drafting a request for propos- als for a new integrated library system for the UW System. Although additional means have been used to gather up-to- date feedback from users, the focus group study continues to yield useful insights. In short, the GLS has made a number of improvements in the online catalog since undertaking a study of its users and anticipates further improvements in the future. The focus group methodology proved fruitful for gathering rich, reliable data for decision-making. The authors would like to thank Nolan Pope and Dennis Hill at the University of Wis- consin-Madison for their thoughtful com- ments and additions to this article, and Roch- elle Logan, Research Assistant in the Library and Information Services Department at the University of Denver, for her assistance in preparing the manuscript for publication. Notes 1. Karen Markey, Subject Searching in Library Catalogs: Before and After the Introduction of Online Catalogs (Dublin, Ohio: OCLC Online Computer Library Center, 1984), 18�20. 2. William M. Randall, �The Uses of Library Catalogs: A Research Project,� in Catalogers� and Classifiers� Yearbook Number Two (Chicago: ALA, 1930), 24�32. Online Catalogs from the Users� Perspective 419 3. Neal K. Kaske, �Studies of Online Catalogs,� in Online Catalogs/Online Reference, eds. Brian Aveney and Brett Butler (Chicago: ALA, 1984), 20�30. 4. Joseph R. Matthews, Gary S. Lawrence, and Douglas K. Ferguson, eds., Using Online Cata- logs: A Nationwide Survey, A Report of a Study Sponsored by The Council on Library Resources (New York: Neal-Schuman, 1983). 5. Christine L. Borgman, �Why Are Online Catalogs Hard to Use? Lessons Learned from In- formation-Retrieval Studies,� Journal of the American Society for Information Science 37 ( Nov. 1986): 388. 6. Markey, Subject Searching in Library Catalogs, 147. 7. Ibid., 147�48. 8. Kaske, �Studies of Online Catalogs,� 22�23; Markey, Subject Searching in Library Catalogs, 147�48; and Matthews, Lawrence, and Ferguson, Using Online Catalogs, 122�43. 9. Robert K. Merton, Marjorie Fiske, and Patricia L. Kendall, The Focused Interview: A Manual of Problems and Procedures (Glencoe, Ill.: Free Pr., 1956). 10. Thomas L. Greenbaum, The Practical Guide to Focus Group Research. (Lexington, Mass.: Heath, 1988); D. A. Hill, User�s Guide to the Library Computer Catalog: Network Library System (NLS) Catalog Users� Guide, 4th ed. (Madison, Wisc.: General Library System, University of Wisconsin-Madison, 1993); Richard A. Krueger, Focus Groups: A Practical Guide for Applied Research (Beverly Hills, Calif.: Sage, 1988); David L. Morgan, Focus Groups as Qualitative Research (Newbury Park, Calif.: Sage, 1988). 11. Morgan, Focus Groups as Qualitative Research, 11�12. 12. R. I. Goodman, �Focus Group Interviews in Media Product Testing,� Educational Technology 24, no. 8 (Aug. 1984): 39�44; Debra W. Johnson, �Keeping Things in Focus: Information for Deci- sion Making,� in Keeping the Book$: Public Library Financial Practices, eds. Jane B. Robbins and Dou- glas L. Zweizig (Fort Atkinson, Wisc.: Highsmith Pr., 1992), 405�19; and Debra W. Johnson, �Focus groups,� in TELL IT! Evaluation Sourcebook and Training Manual, eds. Douglas L. Zweizig, Debra W. Johnson, and Jane B. Robbins (Madison,Wisc.: School of Library and Information Studies, Univer- sity of Wisconsin-Madison, 1994), 161�70. 13. Thomas K. Fry, �Undergraduate Perceptions of Library Service: Use of Focus Groups and Surveys in Strategic Planning,� in Continuity & Transformation: The Promise of Confluence, proceed- ings of the Seventh National Conference of the Association of College and Research Libraries, Pittsburgh, Mar. 29�Apr. 1, 1995, ed. Richard AmRhein (Chicago: ACRL, 1995), 41�52; Debra W. Johnson and Lynn S. Connaway, Arrowhead Library System, the Older Adult in Rock County: Implica- tions for Library Service, unpublished consultants� report, Mar. 1991; Kathryn Robbins and Ruth Holst, �Hospital Library Evaluation Using Focus Group Interviews,� in Bulletin of the Medical Library Association 78, no. 3 (July 1990): 311�13; Barbara Robinson, A Study of Reference Referral and Super Reference in California (Sacramento, Calif.: California State Library, 1986); M. K. Scharf and Jean Ward, �A Library Research Application of Focus Group Interviews,� in Energies for Transition, proceedings of the Fourth National Conference of the Association of College and Research Librar- ies, April 9�12, 1986, ed. Danuta Nitecki (Chicago: ACRL, 1986), 191�93; Charles T. Townley, De- veloping Relationships between Academic Libraries and the State library of Pennsylvania: A Report of the Research with Recommendations, unpublished consultant�s report, 1987, ED 290501; Suzanne Walters, Focus Groups: Linkages to the Community, Denver Public Library (Denver, Colo.: U S West Communi- cations, 1988); Jean Ward, Kathleen A. Hansen, and Douglas M. McLeod, �The News Library�s Contribution to Newsmaking,� Special Libraries 79 (spring 1988): 143�47; Richard Widdows, Tia A. Hensler, and Marlaya H. Wyncott, �The Focus Group Interview: A Method for Assessing Users� Evaluation of Library Service,� College & Research Libraries 52 (July 1991): 352�59. 14. Johnson and Connaway, Arrowhead Library System. 15. Marilyn D. Jacobson, New York State Education Information Centers Program: Summative Evalu- ation Report (Albany, N.Y.: New York State Education Department, 1984), ED 250542. 16. Constance A. Mellon, Naturalistic Inquiry for Library Science: Methods and Applications for Research, Evaluation, and Teaching (New York: Greenwood Pr., 1990). 17. Liz Bishoff, Public Access to the Library of Congress Subject Headings, paper delivered at the annual meeting of the American Library Association, San Francisco, June 29, 1987. 18. Kenneth W. Berger and Richard W. Hines, �What Does the User Really Want? The Library User Survey Project at Duke University,� Journal of Academic Librarianship 20 (Nov. 1994): 306�9; Sharon W. Schwerzel, Susan V. Emerson, and David L. Johnson, �Self-Evaluation of Competen- cies in Online Searching by End-Users after Basic Training,� in Proceedings of the Forty-Fifth ASIS Annual Meeting, 1982, eds. Anthony E. Petrarca, Celianna I. Taylor, and Robert S. Kohn (White Plains, N.Y.: Knowledge Industry Publications, 1982), 272�75. 19. Debra W. Johnson and Lynn S. Connaway, Cataloger as Decision-Make, paper delivered at 420 College & Research Libraries September 1997 the annual meeting of the Wisconsin Library Association, Appleton, Nov. 14, 1990. 20. Donna M. Senzig and Franklyn F. Bright, �The Network Library System: The History and Description of an Evolving Library-Developed System,� Library Hi Tech 5 (spring 1987): 67�75. 21. Merton, Fiske, and Kendall, The Focused Interview, 5. 22. Clifford Geertz, The Interpretation of Cultures: Selected Essays (New York: Basic Bks., 1973), 6. 23. Robert Philip Weber, Basic Content Analysis, 2nd ed. (Newbury Park, Calif.: Sage, 1990), 18. 24. Krueger, Focus Groups, 41. 25. Weber, Basic Content Analysis, 18. 26. Ibid. 27. Ibid., 17. 28. Ibid. 29. Ibid. 30. Morgan, Focus Groups as Qualitative Research, 64. 31. Hill, User�s Guide to the Library Computer Catalog.