020,715 A434 no ,32 BRYCE : ! UNIVERSITY OR ILLINf iBRARY AT URB/ -HAMPAIGK NOTICE: Return or renew all Library Materials! The Minimum Fee for each Lost Book is $50.00. I I ~ I I ~Q I The person charging this material is responsible for its return to the library from which it was withdrawn on or before the Latest Date stamped below. Theft, mutilation, and underlining of books are reasons for discipli- nary action and may result in dismissal from the University. To renew call Telephone Center, 333-8400 UNIVERSITY OF ILLINOIS LIBRARY AT URBANA-CHAMPAIGN L161 0-1096 ALLERTON PARK INSTITUTE Number 32 Papers Presented at the Allerton Park Institute Sponsored by University of Illinois Graduate School of Library and Information Science held October 28-30, 1990 Allerton Conference Center Robert Allerton Park Monticello, Illinois Evaluation of Public Services an Public Services Personnel BRYCE ALLEN Editor University of Illinois Graduate School of Library and Information Science Urbana-Champaign, Illinois 1991 by the Board of Trustees of the University of Illinois ISBN 0-87845-086-6 020.715 A434 No. 32 CONTENTS INTRODUCTION 1 Bryce Allen CAN WE GET THERE FROM HERE? 3 James Rettig SCOUTING THE PERIMETERS OF UNOBTRUSIVE STUDY OF REFERENCE 27 Thomas Childers THE DIAGNOSTIC EVALUATION OF REFERENCE SERVICE IN AN ACADEMIC LIBRARY 43 F. W. Lancaster Cheryl Elzy Alan Nourie GATHERING AND USING PATRON AND LIBRARIAN PERCEPTIONS OF QUESTION-ANSWERING SUCCESS 59 Charles A. Bunge USER-CENTERED EVALUATION OF INFORMATION RETRIEVAL 85 Prudence Dalrymple MINIMUM STANDARDS AS A FIRST STEP TOWARD EVALUATION OF REFERENCE SERVICES IN A MULTITYPE SYSTEM 103 Mary Goulding ASSESSING SERVICE TO SPECIAL POPULATIONS 125 Betty J. Turock EVALUATION OF REFERENCE PERSONNEL 147 Richard Rubin PEER PERFORMANCE APPRAISAL OF REFERENCE LIBRARIANS IN A PUBLIC LIBRARY 167 Geraldine B. King Suzanne H. Mahmoodi CONTRIBUTORS 205 INDEX .209 Introduction On three magnificent late fall days in October 1990, 150 librarians met at Allerton Park to grapple with the issues of evaluation of public services and public services personnel. These proceedings are the formal record of the 1990 Allerton Institute although, clearly, they cannot fully convey the experiences shared by the participants. The papers document the formal presentations, but they do not reflect the atmosphere of intense debate inside the Allerton conference buildings that contrasted so strongly with the lazy sunshine and the beauty of the late fall foliage outside. Keynote speakers are supposed to start the debate by outlining the issues. James Rettig certainly was effective in starting the process of creative dialog. He reminded us that evaluation cannot begin until we have clearly understood goals and objectives. He then raised a number of objections to one of our more cherished ideals and objectives in reference work: that of providing bibliographic instruction. Reading his paper may provide a partial insight into the discussion that it generated. After the keynote presentation, papers presented theories and practical examples, overviews and individual experiences. This range of coverage was planned, as was the balance between speakers from library education and from the practice of public service librarianship. Tom Childers gave an overview of the history and capabilities of unobtrusive evaluation; then Wilf Lancaster, Alan Nourie and Cheryl Elzy presented a specific instance of unobtrusive testing in which they expanded the boundaries of the method by evaluating individual service Evaluation of Public Services & Personnel providers. Charles Bunge spoke about a thoroughly tried-and-tested mechanism for evaluating what goes on in a reference encounter; following him, Prudence Dalrymple discussed ways in which information science research can point out new directions for evaluating information services. Mary Goulding's paper described a classic approach to objectives-based evaluation, while Betty Turock suggested six or seven additional kinds of evaluation that might be attempted. Finally, Rick Rubin gave a masterful survey of personnel evaluation for public service librarians, and Geraldine King provided a specific example of peer evaluation. This constant juxtaposition of how things are being done now and how they might be done in the future provided the basis for a great deal of debate and discussion. In three planned discussion periods, and in dozens of informal encounters, real-life problems were discussed, ideas were generated, and librarians' commitment to public service evaluation was renewed. I hope that readers of these proceedings will sense a bit of the heat of that debate from a distance, and that they will benefit from the stimulation and inspiration of those three days. BRYCE ALLEN Editor JAMES RETTIG Assistant University Librarian Reference and Information Services Swem Library College of William and Mary Williamsburg, Virginia Can We Get There From Here? ABSTRACT To evaluate libraries' public services and public services personnel, the library profession as a whole must agree about the purpose and role of public services. The most problematic service is reference service, especially in academic libraries. The bibliographic instruction movement is examined as a factor that puts reference service in academic libraries out of step with other types of libraries. The flaws in the premises of the BI movement are examined, especially in light of changes being wrought by automation and opportunities presented by the emerging concept of information literacy. These are impelling reference service in all types of libraries towards information delivery rather than instruction in document identification and retrieval. Once consensus forms around this idea, a method or cluster of methods for evaluating services can be devised. Desiderata for the method(s) are stated. INTRODUCTION [Author's note: My apologies to the late Walker Percy, a genius whose work can well withstand the occasional frivolous expropriation such as the following introduction. In Percy's Love in the Ruins: The Adventures of a Bad Catholic at a Time Near the End of the World (New York: Farrar, Straus & Giroux, 1971), the central character, Dr. Thomas More, invents More's Qualitative-Quantitative Ontological Lapsometer, "the stethoscope of the spirit," (p. 62) to measure "angelism, abstraction of the self from itself, and . . . the Lucifer syndrome" (p. 236) in individuals in short, a one-stop, simple device for measuring Evaluation of Public Services & Personnel an individual's mental, spiritual, and moral well-being. A more refined version created in the course of the events related in the novel permits a physician to use the lapsometer not just for diagnostic work, but also to adjust ion levels and correct the patient's angelism, etc.] The author is pleased to announce that he has here in hand the solution to the problem set forth at this conference! This instrument renders the rest of these Proceedings superfluous; instead of discussing the issue of evaluation of public services and public services personnel, conferees can spend their time enjoying the late October air and taking in the pleasures of Allerton Park [the Monticello, Illinois location of the conference]! This is the Qualitative-Quantitative Ontological Lapsometer (model MCTK), the instrument that with just one easy reading measures and evaluates all aspects of public service and personnel. Its use is so self-evident that few users will ever remove the manual from its shrinkwrap. It will tell if a librarian fully understands and comprehends a library user's need, if the librarian selects the appropriate information sources and employs the most effective strategies to satisfy that need, if the staff member treats the patron with proper courtesy and care, and if the user is fully satisfied with the results of the encounter. By simply extending the antenna and pointing it in the direction of the library staff member and the patron, not only can a researcher or supervisor measure each of these things, but can also receive a diagnostic printout that assigns numeric values to each of these areas and recommends strategies for improvement. A more advanced model of the lapsometer that will be available in the near future is fitted with two RJ-11 jacks for input and output so that it can be plugged into a telecommunications line and measure these same variables in encounters between library staff and patrons conducted over the telephone or through electronic mail. The lapsometer has been tested in a variety of libraries of all types and sizes; these tests have unequivocally demonstrated the validity of its measurements, its diagnostic capabilities, and its reliability in recommending remedial measures. Whenever staff members have conscientiously followed these recommendations, they have in all subsequent tests registered perfect scores in all areas, including the patron's satisfaction level. The read-out of these measures is not unlike that of a slot machine. When the lapsometer's LCD window simultaneously registers the harmonic convergence of the ions for the right staff member, the right information source, the right patron, and the right time, then one knows that the public service encounter measured has attained the state of perfection. Work is underway on a much more compact model that can be worn inconspicuously under a library employee's clothing. This ultracompact model will, through continuous subliminal tactile electromagnetic ethereal feedback, provide Can We Get There From Here staff members with information on their performance during an encounter with a patron rather than after the encounter's conclusion as with the present MCTK model. This should assure a perfect score in every instance, since the staff member will know immediately whether or not he or she is performing properly to meet a library user's needs. Given the proven capabilities of this instrument, there is really nothing left for anyone else to say about the why and how of evaluating public services and public services personnel in libraries. The lapsometer asks all the right questions, gives all the right answers, and provides all the needed solutions. This author recommends, therefore, that readers abandon the rest of these Proceedings and place orders for as many Qualitative-Quantitative Ontological Lapsometer model MCTK instruments as their libraries need. EVALUATION: WHY, HOW, AND TO WHAT PURPOSE Would that it were so easy! Alas, it is not, and that is why we librarians are involved in the worthy, challenging endeavor of exploring the questions of why and how and to what purpose we should evaluate library public services and public services personnel. Why evaluate these things? The saying attributed to Socrates about the unexamined life not being worth living might in itself be reason enough. But that implies that the public service function of libraries is a matter of importance only to librarians. That is a very narrow, unconstructive view of the matter. We need to evaluate public services and public services personnel because these services also matter to the people who use and who, not just incidentally, support libraries through taxes, tuition payments, or philanthropy. If these services did not matter to these people, they would not use nor would they support them. They deserve good service; librarians have an obligation to deliver it. The title of this paper poses the question, "Can we get there from here?" "Get where?" one must wonder, and from what "here"? Everyone is familiar with the quintessential bit of American folklore about a traveler lost in a strange place who asks a local for directions and receives the perplexing, unhelpful reply, "You can't get there from here." These Proceedings are unlikely to provide clear answers to all of the questions, explicit and implicit, about evaluation of public services and public services personnel, but even answers posed provisionally will be more helpful than that of the local's reply. But before we can reach our hoped- for destination, that is, before we can say how public services and public services personnel ought to be evaluated, we need to agree on what it is that is to be evaluated and what its purpose is. For only if we know that can we judge whether or not it achieves its purpose. Evaluation of Public Services 6- Personnel The existence of public services and public services personnel in libraries in the United States is a given, something taken for granted by librarians and library users. It has not always been so, as Rothstein (1955) has chronicled in his history of reference service's first six or seven decades, a very brief span in a history of institutions that proudly trace their roots back to Alexandria. The role and purpose of public services in libraries can be summarized by a century-old definition of just one aspect of public services, that part of it known as reference service. In 1891, William B. Child of the Columbia University reference department defined reference as "the assistance given by a librarian to readers in acquainting them with the intricacies of the catalogues, in answering questions, and, in short, doing anything and everything in his power to facilitate access to the resources of the library" (Child, 1891, p. 298). This definition, although it rarely peeks out from the pages of the Library Journal, remains as valid today as it was then. The problem is that the phrase, "doing anything and everything . . . to facilitate access to the resources of the library," includes some "weasel" words open to interpretation. One of the undeniable strengths of this definition is its breadth and its ability to accommodate functions and techniques that Child could not possibly have imagined a century ago. Surely the creation of a catalog is one of the things librarians do to facilitate access to libraries' resources. Provision of remote access to these catalogs via telecommunications systems is another, but not one Child could have imagined. Open stacks and classification of materials are also means by which librarians facilitate access to library resources. Perhaps the most popular thing librarians do to facilitate access to resources is loan those resources to individual library users. So fundamental and so important is this service that it eventually expanded to permit the users of one library to borrow the resources of another library either through reciprocal borrowing agreements or through interlibrary loan. Some of the services libraries provide to their publics are much less ambiguous than others. Circulation, although always the subject of mild controversy because some users or groups of users want more generous policies, is probably the least ambiguous service; patrons borrow books and they return them. Probably the most ambiguous service, and therefore the most difficult to evaluate, is reference service. Just what is it that a reference librarian ought to do? Just what is encompassed by "anything and everything" in the effort to "facilitate access to the resources of the library"? "Anything and everything" is admittedly an ideal, and that perhaps explains why Child's definition never became a standard; ideals, after all, are hard to live up to. Furthermore, the library world is not a monolith, as demonstrated by the existence of its various special- Can We Get There From Here interest associations. Some types of libraries have more difficulty defining for themselves (and, therefore, for their unwitting users) what they mean by reference service. Special libraries, it appears, have the least difficulty in establishing the scope of their reference services; their practices show that they have embraced Child's definition and have little trouble accepting anything and everything that a situation calls for in order to find the information a client in the parent organization needs. Public libraries seem to have little doubt about what their reference services should do. Whereas librarians in the public library community have not resolved the controversy about whether the collections, particularly the fiction collections, they build should widely represent various genre, periods, nations, and styles or should be "give- 'em-what-they-want" collections similar to an airport newsstand's paperback rack, they do not appear to have deep conflicts among themselves about the function of reference services, at least not for adults. Public libraries attempt to answer adult patrons' questions. The situation regarding students is murkier. Sometimes, the service children receive is indistinguishable from the service adults receive; sometimes, it is more similar to a mode of service most frequently found in academic libraries: the instructional mode. It is in academic libraries that one finds the greatest ambiguity about reference service and the way in which it should be carried out. Representative statements from the past decade illustrate the problem. On the one hand are statements such as those from the Bibliographic Instruction Think Tank of the Association of College and Research Libraries. This group of six librarians from universities met in July 1981 and "rejected the traditional notion of the academic library as a mere adjunct to the education program, which led to the establishment of a type of reference service borrowed almost unconsciously from the public library model" (Think Tank, 1981, p. 394). This group "further rejected the notion of bibliographic instruction as a secondary activity of library reference departments, and instead viewed it as the very heart of the reference process" (p. 395). On the other hand, Joanne Bessler (1990) has recently argued that "it's time for librarians to stop trying to teach patrons and to focus more on listening" and declared that "it is time for librarians to raise a new banner. Service, not instruction, should be the hallmark of the profession" (p. 77). These two views could hardly be more different, yet they describe the "here" where public services in academic libraries stand and demonstrate a division in the ranks of academic librarians; some see the raison d'etre of public services as service (meaning fulfilling clients' information needs) and others see it as bibliographic instruction. The term bibliographic instruction has not always been with us. Before it came into vogue, library instruction was an important buzzword Evaluation of Public Services b Personnel among academic librarians. This is a point well worth keeping in mind, for library instruction or, more properly, the term library use instruction, lest one think institutions or even buildings were being taught, is more accurately descriptive of instructional efforts during the past several decades than the fuzzier bibliographic instruction. Library use instruction has been promoted vigorously, especially during the past two decades, as a response to some very real problems. The basic problem that it has addressed, whatever its professed aims, has been that of physical access to library materials. North American academic libraries' prevalent open stacks arrangement provides great convenience to users; once they have identified an item they want, they can retrieve it immediately and begin using it. Combined with the practice of classifying materials and shelving them by classification numbers, open stacks also permit browsing, a not-infrequently useful information search-and-retrieval technique that ought not be scorned. Open stacks also undeniably serve libraries' convenience since they do not have to hire pages to retrieve books for patrons. But before a library user can retrieve anything from open stacks, he or she first has to identify the item(s) to be retrieved. The principal tool for identifying items is the library's catalog. However, because the students on whom library use instruction has been concentrated freshmen and sophomores do not have a strong knowledge base nor a strong bibliographic base in whatever discipline they need library resources for, periodical indexes are equally important. Taking these factors into consideration, library use instruction has devised a template for successful library use for students to follow. This template, promoted as a one-process-fits-every-discipline tool, guides the student to a general-purpose encyclopedia or a subject-specific encyclopedia as a first step. The purpose of this step is to compensate for the student's lack of knowledge on the topic he or she has chosen to write about. The next step guides the student to the Library of Congress Subject Headings and then to the catalog to identify books on the topic. The next step guides the student to a periodical index to identify recent journal articles on the topic. This strategy culminates with a trip to the stacks to retrieve the books and articles identified in its various steps. In other words, it was designed largely to enable students to take advantage of the convenience the open stacks arrangement offers all users regardless of their level of sophistication. This basic approach remains the foundation of bibliographic instruction (BI) programs in countless academic libraries. The pattern is repeated and promoted in classroom lectures, audiovisual programs, workbooks, and computer software. For example, the user's manual for Research Assistant (Bevilacqua, 1989), a bibliographic instruction program for the Macintosh computer, includes a generic "Library Can We Get There From Here Research Flow Chart" that suggests checking a subject encyclopedia for a general overview, checking the catalog for books, checking periodical indexes, and consulting other reference books such as almanacs and dictionaries (p. 3). This is also essentially the model promoted by Gemma DeVinney (1987, pp. 13-23). Yet this template is seriously flawed, especially when one examines the claims and justifications often made on its behalf by BI advocates. BI has been promoted by some for its promise to turn callow, ignorant freshmen into independent lifelong learners. Nobody can argue that it is not one of the ideals of a college education, including the role the library plays in it, to teach students to become independent lifelong learners. The question then becomes, how can the library best play its role in that noble effort? BI as it has been practiced at most institutions has yet to prove that it has a significant contribution to make. A truly independent lifelong learner must be able to make independent judgments about the value, the truth, and the accuracy of information regardless of how that person came into possession of that information. This applies to all types of information to the editorial in the morning newspaper delivered to one's doorstep, to the articles in a magazine one subscribes to, to the direct mail appeals delivered to one's mailbox, to the news bulletin one hears on the car radio while driving to work, to correspondence one receives from a business associate, and to the diagnosis of an illness made by one's physician, as well as to books one borrows from a library. BI programs, especially those promoting a universally applicable search strategy, have been very weak instruments for instilling the critical thinking skills needed to judge all of these forms of information. Their emphasis, sadly, has been on the mechanics of retrieving documents. This is a necessary skill, but not one that makes those who possess it independent lifelong learners. Miriam Drake (1989) has noted that "Librarians continue to be more concerned with delivery of documents and have not focused on delivery of content or the data and information contained in the documents" (p. 523). This is a serious shortcoming. Theoretical discussions of the purpose of BI and its foundations have for many years transcended the document retrieval level. However, the programs as practiced, by and large, have not transcended it. Instead, they have been judged successful if students in them have demonstrated mastery of the behavioral objectives of being able to find a citation in an index and retrieve the cited article or to identify a book through the catalog and retrieve that book from the stacks. This is far too little to settle for in return for all of the fiscal resources, time, effort, and energy librarians have invested in these programs. Furthermore, given the vagaries of organization and architectural design in libraries, it is 10 Evaluation of Public Services & Personnel questionable how transferrable these skills are from one library to another. Unless they can be transferred in toto, they make little or no contribution to independent lifelong learning skills. Because many in the BI movement and the BI Think Tank of 1981 declared itself "a political movement within academic librarian- ship" (Think Tank, 1981, p. 395) have cited as one of its goals the development of independent lifelong learners, BI has been promoted as vital to every college student (Association of College and Research Libraries, 1987, p. 257). As a result, a favored structure has been to incorporate a library use instruction component in freshman English courses. These courses have been targeted because in the cafeteria-style curricula of American universities in recent decades, English composition has often been the only course every student takes. When these courses have taken as their purpose the teaching of writing skills, the library component has been largely superfluous. When these courses have taken as a part of their purpose teaching students how to write a research paper, the library component has been able to resonate sympathetically with the courses' broader purposes. If properly designed and taught, these composition courses have focused on critical thinking skills and the ways in which students can judge the validity of a text and its use of logic, its presentation of evidence, its rhetorical devices, etc. In comparison to this, instruction in the mechanics of document retrieval is insignificant in the long run. At their worst, these courses focus on the mechanics of a style manual and proper forms for citing documents. In comparison to this dull stuff, instruction in the mechanics of document retrieval is simply one more incentive for students to daydream or cut class. Perhaps one of the reasons the BI movement has not succeeded in carrying out its 1981 manifesto is that it has made poor choices in seeking political allies. Within any university, one can hardly think of a less politically powerful group than English composition teachers, frequently an assortment of a few junior, non tenured faculty; several adjunct instructors; and many graduate teaching assistants. That does not, however, explain the failure of the BI movement to make reference librarians the equals of faculty in shaping and carrying out the university's academic mission. The shortcomings of the bread-and-butter approach employed by most BI programs give a fuller explanation. The universal search strategy is inherently flawed and its limitations have been made evident towards the conclusion of many a BI session when a student has asked a librarian what subject heading to search in the catalog or what specialized encyclopedia to consult for information on a topic that made headlines in that day's newspaper. Since reference works such as specialized encyclopedias are late products of the process by which knowledge is generated and spreads, they are useless as sources Can We Get There From Here 1 1 of information for some topics. This model assumes that students are seeking information on a topic that is well-established and has, therefore, become equally well-established in the bibliographic chain. But topic selections and needs are simply too individual for the cookie-cutter search strategy to work for every student in a class, much less for every freshman in every course. The literature of every discipline has its own structure. Freshmen and sophomores generally take courses in many different disciplines simultaneously. To offer them one approach and to suggest that it will be equally useful in all courses in all disciplines is a gross oversimplification of the way information is stored in documents and can be retrieved. As Tom Eadie (1990) recently summarized it, "Information gathering made simple is information gathering made superficial" (p. 45). Furthermore, as Stephen K. Stoan (1984) has argued, efforts to introduce students to the library in the first two years are probably premature, for as Linda K. Rambler (1982) has shown, even in a research university, less than 10 percent of the courses require heavy library use and more than half require none. Furthermore, Rambler demonstrated that requirements for library use are lightest in introductory courses and most intense in graduate courses. Many courses in Rambler's study relied on lectures and textbooks to impart information to students. Some courses augmented these with reserve readings; few did much more. In most courses, then, even the minimal document retrieval skills conveyed in bibliographic instruction are not needed. And those students introduced to the search strategy model who remember it long enough to apply it when they begin upper-division courses in their major may be using a tool better suited to some other discipline. Why, then, attempt to instruct every student in library use techniques? And why, furthermore, focus those very labor-intensive efforts on lower division students whose need for library resources is minimal or less? The model has run into additional problems in recent years with the introduction of nonprint information retrieval systems. So long as this process was something carried on online and carrying unpredictable costs, BI librarians could largely ignore it and omit it from the model since it was done not by the users but by the librarians. However, the introduction of optical disk information products with predictable fixed costs and software intended for use by the general public challenged that. Some librarians, so confirmed were they in their belief in the validity of the search strategy model, responded to these new systems, particularly those easiest to use, by rejecting them. They chose not to introduce "an attractive and easy-to-use, but limited, searching tool into an undergraduate environment" (even though students "were eager to use the automated system") rather than suffer 12 Evaluation of Public Services & Personnel the sight of "the undergraduate user who prints out whatever results from the search term [entered], circles the journals cited, finds the journals left on the shelves, and thinks that the topic has been fully researched" (Van Arsdale 8c Ostrye, p. 515). One cannot help but think of Macaulay's statement that "the Puritan hated bear-baiting, not because it gave pain to the bear, but because it gave pleasure to the spectators" (1899, Vol. 1, p. 159). Some librarians rejected InfoTrac not because it was initially ridiculously overpriced, but because it made the process of identifying relevant documents easy. However, a deeper problem indicated by the statement quoted above is that some librarians have equated research and the tried-but-not-always-true model search strategy. Stoan (1984) has convincingly drawn the distinction between these two activities. Library use can be a part of the research process, but it is not the same as the research process. The way in which a researcher identifies library materials has little to do with the search strategy model. Yet nobody can deny that scholars are, if nothing else, independent lifelong learners. How, then, have these scholars managed to become independent lifelong learners and yet not use the library as outlined by the model search strategy? They succeed because they have developed a deep knowledge base of their discipline through extensive exploration of its literature. In the process, they have also developed a deep knowledge base of the discipline's bibliographic structure. That literature, as Stoan points out, indexes itself very effectively through citation chains. These, far more than secondary reference sources, enable scholars to identify documents relevant to their work. The search strategy model has thrived because its proponents have failed to understand that research is not the same as a prescribed pattern for library use; indeed, research thrives without following this pattern. Another reason the goals of the BI manifesto have not been realized is that the agenda BI librarians have had for library users has not been the agenda these users have for themselves. As Robert Taylor observed in 1957, "Most librarians approach the library by way of the book (form) while the user, often unconsciously, approaches the library by way of information (content)" (p. 303). For the most part, BI programs have emphasized form over content and document retrieval over document use. People, whether they are faculty members, students, business people, homemakers, etc., approach the library looking for answers to questions, not for lessons in retrieving those documents that might answer their questions. Instruction in library use would unquestionably be a valid approach if all patrons used their library as many hours each week as librarians work in it. This is not the case (especially in academic libraries between semesters); library use by most people is intermittent. BI programs have been an attempt to solve problems some librarians in academic libraries have perceived but have failed to convince library users they (i.e., the users) have. All of these are reasons why the BI movement has been misdirected. However, two reasons stand out. The first is that its practice, in spite of BI's rhetoric and its theoretical discussions about teaching critical thinking skills and the like, has not progressed significantly beyond the teaching of a simple strategy to students who may or may not have any immediate or even long-term use for it. The second is that it is not what people want when they seek library service. To get answers to questions from documents stored in libraries and organized for (relatively) easy retrieval, one must know how to identify those documents and how to find them in their storage locations. From this undeniable basic need sprung bibliographic instruction programs. Wedded to a simple model search strategy and a limited set of behavioral objectives, the practice has not changed dramatically even though the literature and discussions about BI have grown increasingly sophisticated. Earlier, BI was likened to puritanism. There is, it seems, a strong streak of puritanism in some reference librarians, at least among some in academic libraries. Puritanism strongly distrusts personal freedom and individual judgment. It seeks, therefore, to impose uniform behavior on all members of a society so that all will conform to standards that the society's leaders have judged to be the best. Nothing illustrates this streak more dramatically than the strong reaction to and rejection of InfoTrac because it allegedly made the process of identifying documents too easy. Stoan (1984) notes that "the logic of using . . . access and synthetic sources seems so evident to librarians that they are alternately critical, bemused, or amused when they observe that faculty members fail to use them consistently" (p. 100). And Bessler (1990) notes that "while Katz claims that 'the user should have the option to learn how to use the library or not and still expect an answer,' many practicing librarians resent choosers of the second option" (p. 77). The effort to teach every student the model search strategy and the claims sometimes made for the model strategy's adaptability to any and every discipline are nothing less than a puritanical attempt to control behavior. The crucial question for evaluation of reference services and any other library public service is: Who judges? A puritanical approach says that only the librarians may judge, for only they know what is best for others. One of the things that is good for library users is conformity in their approach to library resources; hence the importance of bibliographic instruction programs designed "to build better users" (Bessler, 1990, p. 77). In contrast, a democratic or laissez-faire approach says that each individual user may judge for herself or himself. Applied to reference 14 Evaluation of Public Services b Personnel service, this means that not only can patrons choose whether to be instructed in library use or to have their questions answered, but also that they can decide how much information is enough for their purposes and which documents identified in a search are relevant and useful to them. Indeed, these are decisions that ultimately only the patron can make for himself or herself. This latter model emphasizes document use rather than document retrieval. It was relatively easy to impose a single approach on users when all resources and all finding tools were paper-based. But the situation is changing. Several forces are (or at least should be) impelling academic librarianship towards a reassessment of the role and purpose of reference service and user instruction. The first is automation. Most academic libraries today have implemented or are on the brink of implementing an online catalog. In the wake of this, some have been able to go beyond closing their card catalogs to removing them. A common result of the implementation of an online catalog is an increase in circulation. While no OPAC (many given a variety of local "-CAT" names) is perfect and none is as user-friendly as one's own dog, all make it easy to identify cataloged documents. In one library, circulation of its Dewey books, none of which initially were in the OPAC database, dropped dramatically after the OPAC was implemented while circulation of its LC books, most of which were initially in the database, soared. This correlated with the librarians' observations that use of the card catalog had dropped almost to none, whereas use of OPAC terminals was nearly constant. The patrons of that library used the OPAC because it was easier to use than the divided card catalog; the increase in circulation, greater than that which was expected as a part of a perennial trend, indicated that, through the OPAC, users were identifying more books than they had identified through the card catalog and were, therefore, borrowing more. (Incidentally, when records for the Dewey books were eventually loaded into the OPAC's database en masse, circulation of those books increased.) One observation often heard from interlibrary loan librarians after InfoTrac was introduced was that it increased the number of interlibrary loan requests. At the same time, these librarians rightly complained about the lack of inclusive pagination in InfoTrac citations. The significant point, however, is that, thanks to the ease with which patrons were able to identify documents, they sought to use a greater number of documents. Whether or not these documents were the best possible for the users' various purposes is mostly a moot consideration; these users had judged them to be good enough. Although they are more user-friendly than their printed counter- parts, OPACs and CD-ROM systems have a long way to go before they are truly user-friendly. To be truly so, the next generation of these systems Can We Get There From Here 15 needs to develop hypermedia user interfaces that are conversational in nature. These need to offer users options and explain, perhaps even model, the implications of the options and then allow each user to choose the path he or she judges best. Advances in telecommunications necessitate progress in this direction. When OPACs were first installed in libraries, access to their contents was available only from dedicated terminals in the libraries. Users who needed assistance could always turn to a library staff member for personal help. OPACs are now accessible from outside the libraries. This means that conversational interfaces and help devices are badly needed to compensate for the absence of the library staff. The ideal would emulate a system that reportedly is already in service at Disney World. A Disney World visitor can turn to a computerized information system for advice on restaurants, lodging, or other area attractions. If the user asks the system questions that it cannot answer, it switches to real-time video to link the user to a real human for a real, face-to-face conversation. Meanwhile, however, we have the systems we have and users are using them, often without formal training, to identify more documents than they identified when they had to rely exclusively on manual systems and laborious manual transcription of citations. These systems increasingly are stealing the thunder of the typical BI program. When the process of identifying a document has been simplified through automation, when keyword search capabilities in OPACs and CD-ROM databases make it easy to find some things, even if not the best things, there is no need for students to be taught the model search strategy process. When libraries mount additional databases searchable through their OPACs, there is even less reason to teach this process. Students do, however, need to learn the very skills that the literature of BI has promoted but that its practice has rarely imparted critical thinking and how to judge a document's validity and relevance. The experiment OCLC has announced for enhancing bibliographic records by including tables of contents of monographs illustrates the need to emphasize critical thinking skills rather than document retrieval skills. An ever-increasing number of libraries are offering access to additional bibliographic databases through the software used to drive their OPACs. Projects like this will give library users more access to more information; and they must make judgments about all of it lest, in the words of T. S. Eliot (1963), they be left to ask: Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information? (p. 147) Furthermore, as more and more users search OPACs and other databases from outside the library, pressure will build for document delivery systems more convenient than a trip by the user to the library. As these 16 Evaluation of Public Services fa Personnel systems become common, the document retrieval skills emphasized in BI programs will become completely irrelevant. If the Leviathan automated book retrieval system scheduled to go into service at California State University at Northridge in 1991 is a success, document retrieval will be reduced to issuing a command from the same terminal or PC used to search the OPAC (Hirsch, 1990). If the Northridge installation is a success, it will be imitated widely and BI designed to teach students to retrieve books will be reduced to one tap on a function key. Progress in library automation is one of the forces necessitating a welcome reassessment of reference service. Another is the development of computer-based information systems marketed directly to consumers, some of whom, of course, are students and faculty. While every computer system, like the organization of any library, makes demands upon its users to conform to certain protocols, there is no sign that system and software developers intend to arrest or reverse the trend towards making the use of their products more intuitively self-evident. The almost rabid loyalty of Macintosh microcomputer users to their machines and the Mac's graphic interface despite, until October 1990, the machine's relatively high price indicates how important these features are to people. Vendors promote systems such as PRODIGY as "your personal one-stop source for information" (personal communication, September 1990). Relatively few people in the country use these systems thus far, and none of them can offer access to the many information riches stored in libraries' vast collections of printed documents, but their convenience and increasing ease of use will gradually change library users' perceptions of how libraries ought to deliver their information services. If libraries ignore this, then users may well decide to make these systems their one-stop sources. In using these systems, of course, critical thinking skills and the ability to evaluate and make judgments, to find knowledge in information, are just as important as with any library system, automated or manual. While automation is both enabling and forcing librarians to rethink the purpose of reference service, a relatively new concept may offer libraries an opportunity to revamp hollow instructional programs. In its search for a name for itself, the phenomenon now most commonly known as bibliographic instruction once flirted with the label library literacy. Fortunately, this did not catch on, for if it had there would almost certainly be confusion between library literacy and the newer, much more meaningful term information literacy. The American Library Association's Presidential Committee on Information Literacy (1989) defined information-literate people as people: who have learned how to learn. They know how knowledge is organized, how to find information, and how to use information in such a way that Can We Get There From Here 17 others can learn from them. They are people prepared for lifelong learning, because they can always find the information needed for any task or decision at hand. (p. 1) It is significant that this definition says the information literate "know how knowledge is organized," not that they know how libraries are organized. In other words, this is a quantum leap from the typical behavioral objectives of BI programs, objectives made increasingly obsolete by automation. Miriam Drake (1989) has explained the implications of this: When dealing with students, we have a large agenda that goes beyond traditional courses in library usage. We need to extend our programs to develop information awareness and instill the practices of information finding and lifelong learning. . . . While bibliographic instruction has helped students find books and articles for term papers, it has not increased information awareness or significantly changed general information skills. (P- 527) Like Bessler, Drake calls upon academic libraries to shift their focus from teaching skills whose importance is being diminished by automation; she says they must begin by "shifting emphasis from product (book, journal, etc.) to process and from access to the provision of information" (p. 529). Until one has information in hand, it is impossible to judge its value. This is when one needs to apply critical thinking skills, the very skills used constantly by researchers in their information searches even though they rarely use the reference tools promoted in the search strategy model. Despite prognostications, this is not a paperless society, although more and more information is becoming available in electronic media, some of it exclusively so. Automation efforts take time and involve transitional periods. Library users will still need to use some manual processes to identify documents. For example, a reference librarian responding to Bessler's call for a shift from the BI paradigm to a service paradigm for academic reference librarianship, while agreeing with her basic argument, notes: I would love to have a self-explanatory serials list. But I don't, so I explain it over and over again; I teach it every chance I get, despite the fact that I have yet to find a way to make it the least bit interesting. My serials list is a public service problem that begs for a technical services solution. (Lewis, 1990, p. 80) Unfortunately, not all reference librarians see it this way. Some see automation as merely another cause or reason to teach patrons the mechanics of various processes. Clark N. Hallman (1990), discussing reference librarians' need to master new computer hardware and software, says that new and ever-changing information technologies . . . make it paramount that students, faculty, and staff, and others are taught to cope with the 18 Evaluation of Public Services s particular book, serial, etc in our collection? O 3 Any Ol.hr. O Libra Qothe JOT DOWN QUESTION B SHORT ANSWER WANTED (AND IS APPROPRIATE) (What, when, where, who, which, yes or no. etc.) (Answer of i words. Includes verification and meaning of citations, bibliographical form, recommendations, etc., etc., etc.,) GENERAL EXPL. OF CATALOG, LIBR., OR PRINTED REF. SOURCE WANTED (Rather than short answer) n TYPE MATERIALS OR LONGER DESCRIPTIVE ANSWER WANTED (OR APPROPR.) (Answer usually in the form of printed materials) 1. SUBJECT (Mark one) Qa Single subject(s) O b Relate 2 subj or concepts 2. ASPECTS (MARK ALL THAT APPLY) (_/ a Something, anything, everything O l> Must be cert, time period, currentness. . publ.. (D il Focus on aspect, biog., hist., othe O Requests factual inf. in general lo {names, addr.. definitions, statisti rce containing it) tings, rankings, etc O <:. Must bo cert type rot O * Criticism, reviews, interpr. etc O h Requests thai you compile list of references on a : 2A RESULTS (MARK ONE) 2B RESPONSE (MARK ONE) 2C TIME (MARK ONE) O 1 Found O 2 Partly found O 3. Not found 4 Don't know O 1 Directed and suggested only O 2 Helped with or made search O 3. Deferred Ql Referred O I- 0-3 minutes O 2. 3-6 minutes O3 5- 16 minute O 4 Over 16mir 3. SPECIAL FACTORS. DO NOT OMIT MARK ALL THAT APPLY QUESTION AND PATRON CONDITIONS (_) 2 Concerned with foreign counlr./lang. (_) 3. Concerned with yovt. docs. O 4. Inf. needed lor citat.) very recent O 6 Difficult citation O 7. Patron in hurry O 9A Needs extra help O 9B Returns freq. O 10. Difficult to think of source O " - Difficult to find sub, headings O '2 Books off shelf O '3. Source difficult to consult Ql4A. Busy Ql4B Very busy (3 1 5. Cataloging or tech. problem O <7. Need bks. in another area or location 4. LIB. INSTRUCT. MARK ALL THAT APPLY O 1- Expl. sources, citations, search stn O 2 Expl. cat . computer, holdings, loci 5. NUMBER OF 12345 SOURCES USED, REC.. OR INTERP. - QOOOO TYPE: MARK ALL THAT APPLY 6. QUESTION DIFFICULTY (as perceived! O Mediun 7. ASPECT (only if applicable) OB> Oo 01 Oi 02 02 Q3 Q3 O* O O' 07 OB Qa Qa Qa Oo Oi 02 O2 03 O3 O" O5 O5 OB Oe O' O' OB O9 O ! Indexes to pern O 2 Ref books O 3. Cat (card, onlir O 4. OCLC. RUN. et( 5. Comp databasi or CO-ROM Use separate guidesheet and select subj. Mark boxes with no. of your subject. ~v| EXAMPLE O 1 O 1 2 Q2 O 1 O ' O2 IOI IOOOOBBBOOOO MAKE NO MARKS IN THIS AREA 14454 FOR O00 OFFICE USE O ONLY O 78 Evaluation of Public Services b Personnel APPENDIX B Patron's RTAI for Reference Questions Academic Libraries FILL IN DOT LIKE THIS The Reference Department is doing a survey of reference use and would appreciate it if you would mark the following brief checksheet. Thank you! (Deposit checksheet UNFOLDED in container on leaving this area or on leaving the library.) THANKS AGAIN FOR YOUR HELP! _U8E>IO. 2 PENCJL ONLY STATUS O Freshman O Sophomore Ojnn,o, OSenio, s-\ Graduate student or OContinuifiQ education or nondegree student OAlumn, Faculty Os..,ii Not affiliated with Umv MAJOR OR TEACHING/ _ RESEARCH AREA O Arts or Humanities Ou O Business/Management O Other Social Sci O Medicine/Health O Agric /Biological Sci O Math /Physical Sci LJTedtnelaw Enumccsring O Interdisciplinary/Other Major not declared O O O O O O 1. Did you locate Yes, |ust Yes. with Yes. not what 1 asked Yes. but Only partly No what you asked what 1 limitations for. but oth information not really about at the wanled or materials that will be what 1 reference desk? he pful wanted 2. If yes, how did you find the O Librarian found information or materials? hulped find or QFollowed sugges- f) Didn't follow sugges Aliens and found ^ lions but found on 3. Were you satisfied with the information or materials found or suggested? O O O Yes Pailly No 4. If partjy or not satisfied, why? MARK ALL THAT APPLY. Ofi'imil nothing Q Too much O Wanl different viewpoint O Need more simple O Not relevant enough Q Not sure if information given me is correct 5. How important was it very to you to find what important Important you asked about? Q Q Moderately Somewhat Not imnoMan, in.poM.m, impoM Yes Partly No 6. Was the librarian busy (e.g., phone ringing, others waiting)? O O O 7. Did the librarian understand what you wanted? 000 8. Did you get enough help and explanation? 000 9. Were the explanations clear? O O O 10. Did the librarian appear knowledgeable about y our question? O O O 11. Was the service you received courteous and considerate? O O O 12. Did the librarian give you enough time? O O O 13. Did you learn something about reference sources or use of the library as a result of consulting the reference librarian? O O O hadn't previously known about, as a result of consulting the reference librarian? Yes^one Yes. mo^than one No.^one OBOBOBOOOflHHOOOO 15914 MAKE NO MARKS IN THIS AREA FOR 0j OFFICE USE O000000 ONLY O000 Perceptions of Question Answering Success 79 APPENDIX C Patron's RTAI for Reference Questions Public Libraries FILL IN DOT LIKE THIS The Reference Department is doing a survey of reference use and would appreciate it if you would mark the following brief checksheet. Thank you! (Deposit checksheet UNFOLDED in container on leaving this area or on leaving the library.) THANKS AGAIN FOR YOUR HELP! OCCUPATION (Mark one) (j Homemaker O Skilliid l.ihur /traili'S O Secretarial/clerical /ntlri! O Sates/marketing O ProiesMonnl/technH:al Q Unri MI jli r r ril at present O Retired 00 O 1. Did you locate Yes. just Yes. with Yes. not what 1 asked Yes. but Only partly No what you asked what 1 limitations for. but oth information not really about at the wanted or materials that will be what 1 reference desk? helpful wanted 2. If yes. how did you find the O Librarian found information or materials? helped find or ()FoMowed sugges- f~) Didn't follow sugges- tions and found ^ lions but found on on my own my own 3. Were you satisfied with the information or materials found or suggested? O Yes Partly No 4. If partly or not satisfied, why? MARK ALL THAT APPLY. O Found nothing Q T much O Wan < different viewpoint AGE O Under 18 O in -to O 41-64 Qwj O Need more simple O Not relevant enough O Not sure ,f information g.ven me is correct 5. How important was it very to you to find what important Important you asked about? Q Q Moderately Somewhat Not important important impoMan, Yes Partly No 6. Was the librarian busy (e.g., phone ringing, others waiting}? O O O SEX OMO* O Female 7. Did the librarian understand what you wanted? O O O 8. Did you get enough help and explanation? O O O STUDENT O Hjli School OCA* 3 Graduate school [3 Continuing education 9. Were the explanations clear? O O O 10. Did the librarian appear knowledgeable about your question? O O O 1 1 . Was the service you received courteous and considerate? O O O ^2. Did the librarian give you enough time? 000 SOURCE OF QUESTION 3 Work related O S:lmnl/liKami r.taud O Recreation related Other personal project 2) (In.lilin--. selt-duvelopment. 13. Did you learn something about reference sources or use of the library as a result of consulting the reference librarian? O O O 14. Did you become acquainted with any reference sources you hadn't previously known about, as a result of consulting Yes^one Y.s.mc,hanone No.^on, 1 1 1 OBBIOOOOHBOOOO 14454 MAKE NO MARKS IN THIS AREA FOR O00 OFFICE USE Q ONLY Q 80 Evaluation of Public Services b Personnel APPENDIX D Librarian's RTAI for Directional Questions Public and Academic Libraries IOOBOBBOBBOOOOOOO MAKE NO MARKS IN THIS AREA I i ! I la la OOO 3 @@ FOR OFFICE USE ONLY O O o O O O O O O o Perceptions of Question Answering Success 81 APPENDIX E Patron's RTAI for Directional Questions Academic Libraries I OH Of I ICE USE ONLY DiBOOMOOOMBOHOOOO MAKE NO MAIIKS IN THIS AHEA "S M ~ (U a n fc " E o (5 -C o) u o -0 CO "C 3 "c b i' I S5 ^ I S i '| a JB 3 r- C U O g o Q- Q. - " "ro O co c ~ 1 i 1 .6 ~ c> i> ro S ra" 0) < ^ 1 ^ 'o O O O O O 0) _. i c. -. a [j] LU OC LU 1* I 1 O 3 O H g O CC o Q z IK a O LU CO HI _J " oc u. 5) Z o ^ < z *- o 5 I I u < OJ r^ O O A <5 LU (3 ^ n CL) til ^_ c o 5 1- c (^ PH > g > o oc O i ,. ^ S i 1 o w 2 'S c Lo co o i 3 C QC O CD /- a> flj 2 isle, : '0 E ^ T3 ^ TO "4, uJ CO 1 O LJJ ' U ^ 000000 'A Is jj i i j E SL r ra - \ Q. ID o F Q a a, ; c c O O 0) *-. *- "D u Q c ^ 5s %z S 0! 0) * to CD t; "^ "g i <1) in CC ^ ,J h- o s 10 ^ ro o) c> m ^ rn ^ o ^ >- 3= i"c -c -c o o -B -S 5 fi S & g ss S^ it co ^ co o2(jt5 _3 o 2 ^ < l2 CO Z o o o o o o 00 82 Evaluation of Public Services 6- Personnel APPENDIX F Patron's RTAI for Directional Questions Public Libraries CD FOR OFFICE USE ONLY IOOBOBBOBBOOOOOOO MAKE NO MARKS IN THIS AREA ,_ fe - i W j= 'n JO V) o D I 1 s a -g |> -I " 9 & i-o ? = s B?^ V* Z c" LUS'gg"'" A tu LUti a, 5 3 (Jj ^ ;aS2 3 "> Of K "r OJ T3 i: ^i *QJnO "3rT> := (TJ Jo -^ UJIoC 2 i- O ll O 1^5 U 0)^-25 Hi O O O <->$ U) {TOd o w o o o o w o o o o 5 3 O ra V) 'o z |5 F Q. UJ LU -Q O | "i = 00 z S S S 2 S 1 _J g m a * + x -s E Dn!S Uj|^ LL 2 < o o o o w o o 5 t Q) GJ 1 o 2 > o 3 . o ie Refer leckshe 5o ^ * o c = X * w> ^ i^ ^ >> c ro ro S- + 00 OO CM O lO OO Osioo tio ^-oorocri iiouoro vo 4-> I-H ro CM 00 _i CM CMCMCMCSJ 00^,^1 r-LT)OO^r LT> f- 0. at > E (J S- 1 4J H C ro QJ QJ >> ! 01 c c: *- -M O C JO C 01 +J O-r +-> u O) O inre**-c o o ' ro -i o JO C >i C E - D.-*-^ 4-QjaJ in^-> m**-i/i jZr cui_O'^ O) O O) 01 in Oin-roEm Ololf -COS- O-fOi-SOJO E -i- s_ L. 1/1 in JZ D-4^0JZJ i -i ro OO d> QJOrOQ)-*-> ex. 01 r- QJ Q. in t ZJ in l_ o +J i O t *--r-CCX i Q-OOOJ o o <*- zs in o J^roT? -r-JZ: -f OmC4->Qro CUQjc>>JC^->(/) r CX Q) cr jc S-COIr ZJI +J OJ r .(-> -r- ro >- CX 4->-^ro-l->Q. ro 0) S- (U C in O O r O" C U C !M4->in rO4J inJO > !Z i. o -^ t- S-f-Oiroo)-'- l *-c l t-ai'*--f-C2 oE-^OOmro u. QJ OJ *t- i- S- 1- me S- OQIOBOCQIOI ** ' r< ~ -^ -* ' QJ 4^ O 4-> 01 *-> o ro4JC OO Oro U ro S -4-> U ai 3 i/l -o caicoccnai m^JCLininij-D . - * oo 4- - C -r- Z> +J in O l -CCXij rji O'*-OOOS-S- t *- in ommro O UJ O S- O > X> C 2 s-O"-ozjzc -r- a) "o -^ o o ai>5i-aiai (_>' II OO S - S- QJ QJ CU r^ ZJ S- 4-> JZ ^J O '*~ 4-* S 4-* 4-> "O O 4-> | * O O QJ '^ ^** oo o m 4-> QJ f- -i- r- OO-Cl/lrD-^-c -r- -r- -f- r 0; to O'l-rjJOUl/lJZZ LU Q- L. (_> i- -r- IO -r- l_ <- > -^ U >- f O O ro E OJ i- OO 4-> S_ru O4->>S_OOZJO3OZ>OZ>-^> >>JZC>,>,jZi-OO cj a: !_ ro > 1 C O 1 OS- <: U O O QJ LUroZI roroC H >- Cu ZC UJ LL. oo i u_ oo O o_ " i ^ ^ ^ cc: y uj ^ uj LU < U 1 u- OC. 0. et UJ c ... _l r _lL-H CM ro ec 1 CM 0-> ^ LO ior^ h- r i CM oo ^r 331 ICM 00*3-00 _J I r 1 "- Q_ | 00 c o- ^ LU 118 Evaluation of Public Services & Personnel APPENDIX (Cont.) 6. (com.) Additional Note to Chart: For some items, it seemed valuable to tally ratings of a subset of libraries which had to adjust scheduling, budgets, space, or procedures in order to meet the particular requirements (see Question #5). Half of the ratings were surprisingly similar, using either the total responses or the subset as a base. Those which indicated more than a 10% difference in the "vitally/very important" rating are: Vitally/Very No/Negative Important (%) Effect (%) Total/Subset Total/Subset Requirement for Formal Education 69/80 8/0 Interview Workshops 63/42 3/0 Reference Sources Workshops 60/42 6/0 Continuing Education Workshops 70/59 2/0 Local Government Documents 43/58 5/0 Information on Local Organizations 55/68 5/6 Retention of Newspapers 36/47 12/15 Most of the percentages unaccounted for above were rated in the "Important" column; a few had no opinion. The numbers of libraries which had to acquire equipment were too small to make valid comparisons. The only exception was the 15 libraries which acquired equipment to access online databases. 82% of them rated that element as vitally or very important, exactly the same as the rating from total responses. 2 libraries which had to write a policy felt it had no effect on their services. 1 library which had to purchase titles on the Core List felt it had no effect on service; another felt it had a negative effect because of cost. OTHER COMMENTS WRITTEN ON SURVEYS: In response to "Which of the following did your library have to do IN ORDER TO MEET reference standards?": "We purchased a few titles to meet the standards but also used the list to expand our holdings further." 'The best method of evaluating reference service is yet to be decided. Do you plan to recommend something system-wide?" "As a small suburban library, I appreciate having a standard to measure against our reference collection-even though it can be a juggling act to cover the cost." Minimum Standards as a First Step 119 APPENDIX (Cont.) "We have always sent people to workshops, but now we are making more of an effort to make sure that everyone goes to at least 2 a year-and of course there are more available now." "Writing a reference policy was very worthwhile in terms of deciding just what we will do and standardizing how we treat patrons. It was also a good cooperative project for the Adult and Young People's Services Department." "Obtaining local documents has been far more difficult than we expected. It took nearly two years to receive current minutes of City Council meetings from XXXXXX, and we still don't have a complete set of ordinances from either XXXXXX or XXXXXX. Apparently, neither City Hall believes the library does more than just hand out Danielle Steel novels and host Story Hours for preschoolers." "We don't do very much reference work at this library, but all the various steps the staff has gone through to meet the standards has made them more aware of the importance of reference service and more familiar with our reference collection." "An essential aspect of reference to emphasize in continuing education is familiarity of electronic reference sources available, best utilization of such, budget concerns, management of such services and current display and information about such as electronic encyclopedias, video-audio technology, etc. Which is most cost effective? Which is the best to use to fulfill information requests? Update on a nationwide standard of information format. A budget plan to introduce electronic sources each year in a long range plan. Helping our youth to be aware of electronic availability of resources. When and how to use electronic information and critical decision making of which is best to use and digest at critical points of needs. Helping our youngsters become computer literate in knowing what to use, when, how?" "We changed staffing and scheduling so that there is a more even distribution of those trained in reference." "Writing the Reference Policy was the most difficult part. The whole staff contributed and it made us all more aware of our policies and able to be more consistent in our answers to patrons." "Many of the books required have proven totally irrelevant to a library of our size and a community of our type." "We held staff inservice training to use the new reference material and make better use of what we already had." (from a library which does not meet standards) "We are always informally evaluating our reference service. The standards now make us do so formally." "Since XXXXXX came aboard as our new director, we have added a reference desk complete with telephone, CLSI terminal with DIALOG compatability, increased our core reference collection substantially, added MLS trained librarians for around the clock coverage. I believe we have made enormous strides toward meeting reference standards in the past three years!" "Frankly, I always fear something on the core reference list has gotten out of date with me forgetting to replace it. Chases 's Annual Events remains for me the most delightful and important discovery on the list." 120 Evaluation of Public Services & Personnel APPENDIX (Cont.) In response to "Even if there were no reference standards, which of the following SLS- provided reference aids would you want continued?": This is a loaded question. All of the above are or have been helpful to a degree. But the 'Core List,' for example, as a requirement is different than a 'Suggested List,' that might be just as helpful." "We have become increasingly aware of the depth of our own collection. With the new additions to our collection and the training of the staff we are able to answer the reference questions that are asked of us. Please keep in mind that the role we have chosen is a Popular Materials Library." (from a library which does not meet standards) "Serving as a member of the 1990 Core Reference Committee was a pleasant and profitable experience for me. I learned so many things from my colleagues about reference sources and methods of service. It is an ideal way to up-date and develop one's own reference collection." "Rather than just continue the requirements should be strengthened." "Bibliographies in various subect fields of recommended titles to help small libraries in adding depth to their collection--the opinion of SLS peers would be more valuable to us than many printed bibliographies in books." "Besides Core Reference Lists I would like to see suggestions for reference material that you have found useful, even though not required." "This year's workshops had few of relevant value. Perhaps more on basic reference sources and tips and less on hi-tech and interviews." "Evaluation of Reference Services for Youth Services Dept." "We love all SLS-provided reference aids." "Workshops are fine if they are on a subject you need-but to take a workshop for a requirement has a negative effect." The existence of written Reference Standards makes it easier to justify the Reference budget to library trustees; one can defend expenditures by arguing that Ve have these system reference standards to uphold..." "Reference workshops should be provided, but workshops should be offered IN SEVERAL SESSIONS for professional staff. So far, I've seen little of this." "Even more meaningful than educational requirements is the hands on experience of staff-whether through SLS workshops or in house training." "How often is a library visited? How is the schedule of visits decided?" In response to "In general, do you think SLS reference standards have improved reference service in your library?': They keep us from losing sight of some basic things we need to maintain." "Yes, but we were thinking along the same lines anyway. However, the workshops provide CE that we couldn't do on our own." Minimum Standards as a First Step 121 APPENDIX (Cont.) "Found several good titles on core list." "Seem to apply to small libraries." "No, we have no SLS backup." (from a library which does not meet standards) "Cannot evaluate, as very few changes were needed." "If nothing else, just looking at and thinking about reference service is a great exercise. But the SLS standards have value beyond that. We're lucky to have them, even if we all do complain a little." In response to "Whether you answered Yes or No to the above, would you recommend some type of reference standards as a good idea?": "Guidelines yes/standards no!" "Maybe called guidelines." "Yes, I merely disagree with penalty. Knowing the norm is valuable; following like sheep is thoughtless." (from a library which does not meet standards) In response to "How do you think SLS standards could be more effective?": "By SLS helping (financially, if necessary) those libraries who do not meet standards. I strongly disagree with the process of denying service to any SLS library. SLS was founded to help libraries-not to punish them." "Continue revisions of core list (two year intervals). Help libraries evaluate their reference service. A uniform method would be of greatest value." "When I first dealt with the many pages of the core reference list I wished it could be published on interactive software for much greater ease of maintaining and upgrading the collection as well as budgeting! I still think it's a great idea..." The best way would be in terms of available consultation with SLS personnel so our standards could be better updated. Perhaps we could reserve at least 1 session annually of the Zone Reference Librarians' meetings for standards and one annual session (at least) for evaluation stats." "Provide fewer workshops of higher quality and help the instructors by providing an outline of what to cover in workshops. There is an uneveness in the quality unfortunately." "As long as member libraries are relatively autonomous, I doubt there is much more that can be done. I worry a bit about running out of new workshops for long-term SLS librarians, but continuing education (or just battery-recharging) is a real need. The fact that patrons are still being referred to us for help or materials they could have gotten in their own libraries bothers me, so may need to look at ways to reinforce training." "They will be effective if they are enforced. Each library should assume its own responsibility in seeing that the SLS standards are met. Yet, we still need reminders that we are keeping in step with the standards." 122 Evaluation of Public Services & Personnel APPENDIX (Cont.) "Insist that academic libraries meet the same standards as the SLS public libraries." "Standards should be re-evaluated for fairness to smaller libraries. Cutting them off from Reference Service assistance is a double punishment-they are the libraries who need it most. Also, the original concept was to set up standards to strive for and guidelines to good service- 'what should we be doing?'--not what must we do." "Sensitivity to the limitations in staff and reference materials of smaller, poorer libraries." "Youth reference questions are a very important aspect of reference service. Consideration in training, input, etc. should always have a youth services librarian representative." "The Head of Reference reports that the workshops are especially useful. She also recommends that workshops be offered on the subjects of business and legal resources, the two areas where staff have most expressed a need." "Certain portions should be based on population and budget. The truly 'poor' library in a small population certainly doesn't require as much as a larger population needs." "I think a workshop in 'writing winning proposals' would be valuable in helping us make our case with our boards." "Basic Standards should be expanded. For example: long distance phone calls, large core list, immediate access to SWAN terminal, etc. It might be useful to have some standards cover the quality of the actual reference work, in addition to the collections and equipment." "Divide standards by size of population served with varying degrees of standards." "For those libraries that rely on Reference Service, the comments that I hear are that the service is slow and sometimes inadequate or nonexistent." (from a library which does not meet standards) "They would be more effective if they took the conditions of the small libraries into consideration, e.g. Reference person on duty all open hours; on-line capability; and core reference." "Reference standards currently require that the Reference Role be one of the top three roles for every library-it is not one of ours by action of the Board." (from a library which does not meet standards) "I think continuing education for all professionals is a necessity. Technology is moving so rapidly~we all need help in keeping up-to-date." "Perhaps if there were more distinction between the size of a library and the specific requirements." "Provide more reference workshops pertaining to public libraries." Minimum Standards as a First Step 123 REFERENCES Crawford, W. (1986). Technical standards: An introduction for librarians. White Plains, NY: Knowledge Industry. Dubberly, R. (1988). Questioning public library accreditation. Library Journal, 113(9), 56-58. Mackay, H. (1988). Swim with the sharks without being eaten alive. New York: William Morrow and Co. Suburban Library System. (1989). Reference evaluation manual for public libraries. Oak Lawn, IL: SLS Reference Service. White, H. S. (1989). The value-added process of librarianship. Library Journal, 114(1), 62-63. BETTY J. TUROCK Chair, Department of Library and Information Studies Director, MLS Program Rutgers University School of Communication, Information and Library Studies New Brunswick, New Jersey Assessing Service to Special Populations ABSTRACT Over the past twenty years, librarianship has promulgated quantitative evaluation through the application of output measures to a goal-based model, even in the face of evidence that such an approach makes difficult the fair assessment of services to special populations. While outside librarianship the emphasis is on outcome measurement, we have failed to move into that realm, even when it is most appropriate. In the future, the way in which evaluation is conducted must be determined by the questions it seeks to answer, the model that will best supply the answers, and the design that will uncover an accurate reflection of the program. That requires a combination of qualitative and quantitative measure- ment rigorously applied. Eight models are suggested that can provide the valid, reliable evaluations that have to date eluded us. INTRODUCTION Not unlike other professions, librarianship has resisted evaluation. At the federal level, even with legislation like the Library Services and Construction Act (LSCA) Title I, which has as its major focus service to special populations the aging, handicapped, disadvantaged minorities, the illiterate, and those for whom English is a second language hard-hitting comments have become part of the record on library efforts (Shavitt, 1985, pp. 124-25). Although assessment is required to receive LSCA funding, the consensus of recent studies, 125 126 Evaluation of Public Services 4r Personnel including a 1989 meta-evaluation, is that library program evaluation stands now where educational program evaluation stood fifteen years ago (Roberts, 1985, p. 1; Turock, 1990, p. 50). Why Is Evaluation Resisted? Given this negativity, why do librarians continue to resist evaluation? Frequently, that question is answered by citing a tradition of limited interest which, in turn, is blamed on a limited knowledge and understanding of evaluation processes and techniques. But that supposition is not only condescending, it also reinforces the unrealistic expectation that minimal knowledge of the evaluative process will not harm the validity of the resulting product. At a Midwinter Conference held during January, 1989 at the United States Department of Education's Office of Educational Research and Improvement, where eighty participants from forty-seven states analyzed the national status of evaluation in service programs funded by LSCA Title I, it became clear that ascribing resistance to lack of skill alone is too simplistic. Even when librarians are knowledgeable, they may not evaluate. Some of the conferees' reasons for abstinence had a philosophical basis, such as, "What we do can't be reduced to numbers"; others had an operational basis, such as, "Costs are too high and evaluation consumes more time than we have to give it." With some probing, however, two prevalent underlying reasons were brought forth. First, librarians have little faith in the usefulness of evaluations. For all of the effort assessment requires, they believe no one pays attention to the results. Second, all too frequently, evaluation militates against demonstrating the worth of nontraditional services for nontraditional populations. Taken together, these reasons pointed up the perceived lack of utility of evaluation, and the misinterpretation of evaluation as synonymous with currently practiced output measurement. Expanded Options In the last decade a shift has taken place in evaluation, from the dominance of numbers in quantitative assessments toward the addition of narratives in qualitative approaches. That shift is only now beginning to have an effect on library programs. Until twenty years ago, minimum standards for public libraries and public library systems issued by the Public Library Association (PL A) concentrated on the resources supplied to provide service, such as income, number of staff, volumes owned, and volumes added (Public Library Association, 1966). The major problem uncovered with these assessments was that putting standard inputs into a library did not necessarily assure standard levels of Assessing Service to Special Populatios 127 activities, such as circulation or the number of reference questions answered per questions asked, i.e., input and service did not necessarily go hand in hand (Chelton, 1987, pp. 463-84). In the 1970s, with a grant from the U.S. Department of Education, Ernest DeProspo at Rutgers University began building the case for support from a more systematically developed and tested set of quantitative measures that emphasized outputs, i.e., measuring performance through services used, such as library visits, in-library materials circulation, and program attendance (DeProspo et al., 1973). By 1982, PLA had sponsored the publication of Output Measures for Public Libraries (Zweizig 8c Rodger, 1982), which was revised in 1987 (Van House et al., 1987). As adoption grew, problems were uncovered. Today, although output measurement may be managerially necessary, stressing it without regard for its limitations has retarded the development of library program evaluation, especially with regard to demonstrating the worth of services for special populations. Studies over time have revealed that when measures of use are compared, the differences discovered may not be due so much to service performance as they are to the social and educational characteristics of the library's public (D'Elia, 1980, pp. 410- 30; D'Elia fe Walsh, 1983, pp. 109-33; D'Elia & Walsh, 1985, pp. 3-30; D'Elia & Rodger, 1987, pp. 5-20). Even in the face of evidence that applying output measures may make difficult the fair assessment of services to special populations, particularly those situated in economically disadvantaged communities, they are still the only approach widely recommended. The use of input and output measurement has also been called into question because it does not reflect on the quality of service provided. It makes no distinction between technical quality what is delivered and functional quality how it is delivered (Shaughnessy, 1987, pp. 5-10). While currently outside librarianship the emphasis is on outcome measurement, we have failed to move into that realm even where it is most appropriate. The focus of output measurement is the library, but the focus of outcome measurement is the library's users. The shift is to determining impacts, that is, what happens as a consequence of a program. This approach takes a marketing rather than an institutional stance by asking such questions as: How well did the service meet the magnitude of the need uncovered? Did it have the intended effects? Did it reach the target audience? What changes occurred in them? Were their skills enhanced? Were they able to reach a personal goal which improved the quality of their lives or the lives of their family members? What values did they derive from library use? 128 Evaluation of Public Services cr Personnel The answers to these questions give a better picture of the merit of services for special populations than traditional measures such as circulation per capita. Common constraints put boundaries on the course undertaken in all evaluations. The aim is to conduct a credible assessment for affordable costs within the available time. Staff expertise also determines the design implemented; it cannot be more intricate or complex than staff can handle. When design demands a level of skill that is not available, options include hiring consultants, giving staff short, intensive training courses, or isolating complex or difficult portions of the design for performance under contract (United States General Accounting Office, 1984, pp. 12-13). The self-diagnostic approach to library evaluation currently in vogue has led to librarians assuming the role of evaluator in addition to other roles demanded of them. Indeed, that not only requires time unavailable, but it may not be worthwhile in the long run. A study of the U.S. Department of Education's National Diffusion Network (NDN), established to recognize and disseminate information and training on exemplary programs of educational innovation, has shown that most of the programs deemed outstanding were assessed by expert outside evaluators (Lynch, 1987, pp. 20-24). Librarians can stop the self-flagellation because they are not authorities in the craft of evaluation and realize that there are some things experts should be hired to do. Measurement and measures have held the spotlight. But the application of measures alone does not ensure the systematic process that is a hallmark of rigorous evaluation. The demand for evidence that something good is happening can exert pressure to decide program merit on the basis of what is readily measured. This rush to quantify can damage progress in developing sound library programs for special populations aimed at long-term outcomes (Schorr, 1988). Ultimately, the way in which the evaluation of a program is conducted must be determined, not by the application of a few measures, but by the questions it seeks to answer, the model that will best supply the answers, and the design that will uncover an accurate reflection of the program under scrutiny. In some cases, qualitative data is needed first to better understand and measure what will adequately assess impact, particularly where services to special populations are concerned. But qualitative evaluation is rarely discussed and even more rarely implemented. Two Perspectives on Rigor Qualitative strategies frequently supply the only means to fairly and accurately assess what is occurring in services aimed at special populations. Perhaps they have largely been ignored because they are Assessing Service to Special Populatios 129 mistaken for a return to the conventional wisdom or because their rigor is questioned. But neither quantitative nor qualitative evaluations has a corner on rigor. They seek to answer different questions. Qualitative strategies are directed toward descriptive questions. Quantitative strategies are directed toward normative and cause-and effect questions (United States General Accounting Office, 1984, pp. 1-2). Descriptive questions provide data on the condition of program participants, why they need the program undertaken, how to reach them and provide them with service. For example, an English-as-a- second language program for older adults will have limited access to previously gathered systematic data to guide program implementation. The first evaluative step, then, is to collect information that will lead to an understanding of what is going on in the lives of the elders and how that will affect the way in which the service is designed and delivered. Normative questions provide data that compare what is observed to what was expected, a standard of performance, or a performance objective. For example, the influence of a homework hotline for disadvantaged youths may have been discovered by comparing scores on high school assignments before and after program participation. As the number of scores mounts up over time, the program will develop a standard for improvement by which continued program success can be measured and by which the effectiveness of this program can be compared to other similar programs. Cause-and-effect questions collect data that reveal whether an observed result can be attributed to the program's operation, for example, determining what part of the change observed in the quality of research papers submitted by disadvantaged high school students is attributable to the effects of the public library user instruction program they attended. The proof may be determined by comparing a group who participated in the program with a group who did not. That is not to say quantitative strategies should be cast aside. Michael Quinn Patton (1987) has created a series of questions to guide the determination of the appropriate approach. Quantitative strategies are preferred when: 1. Standards exist by which to judge the merit and worth of a program. 2. Program goals are specific and measurable. 3. Concentration is on comparing participants of the program on standardized, uniform measures. 4. Instruments are available to measure important program results. 5. Instruments can be developed that measure important results. 6. Emphasis is on aggregating information so that uniformities are highlighted. 7. Causes of change in the target audience are the focus of the evaluation. 130 Evaluation of Public Services employment decisions. For this function, three had not observed a relationship and two others rated peer appraisal as no more effective than other evaluation processes in making employment decisions. The fifteen performance appraisal functions which the respondents used to rate the effectiveness of the peer appraisal were: 1. learning about others with whom one works, 2. providing performance feedback to colleagues, 3. eliciting feedback from colleagues, 4. supporting job development, 5. providing ideas about learning and personal development needs, 6. improving work relationships, 7. acknowledging work that was done well, 8. creating a base for modification of behavior, 9. improving work focus, 10. putting fears to rest, 11. facilitating personnel planning, 12. improving communication skills, 13. improving productivity, 14. identifying work that could have been done better, and Peer Performance Appraisal of Librarians 177 15. making employment decisions, i.e., identifying candidates for branch and project management, etc. They were also asked for further comments on the process. They identified being better able to co-assess personal plans, understanding the relationship of their and others' individual goals and objectives and their day-to-day activities, being able to relate better with coworkers by giving and receiving feedback and support, and being able to benefit from group problem-solving through the peer appraisal process. Two individuals reported, however, that they remained hesitant about participating in peer appraisal, one giving as a reason a personal crisis and the other, the possibility of "group think." Some noted that the group could improve in identifying needed changes, in confronting one another about problem behavior, and in being critical about each other's work. They added that more experience with the peer appraisal process might increase open communication. Second Interviews After a second round of peer appraisal discussions, the participants were also interviewed by phone. Once again, they expressed positive reactions to the process. They identified the following as benefits: a positive change in relationships related to increased understanding of one another; being able to use personal strengths, identified through the process, in facing new challenges such as the implementation of an automated system; greater and more accurate self-knowledge; willingness to be more open; peer evaluation of personal accomplishments; solving problems productively and with quality, e.g., developing strategies for working with staff shortages and being overworked; seeing the wholeness of one's job and of one's role in the institution; identifying personal needs, e.g., for additional training in supervisory skills; understanding nonverbal expressions; understanding that compensating for others' shortcomings is more stressful than confrontation; learning to whom to speak to get things done; reorganizing work in own area of responsibility; becoming positive about the process, especially because of its problem- solving potential; and learning how to compensate for personal uneasiness in confrontations. 178 Evaluation of Public Services if Personnel When asked about changes needed in the process, the majority saw no change needed at that time but were open to the idea of possible future change. Two suggested that other procedures to provide one- on-one feedback, such as coaching and mentoring, should be used in conjunction with peer appraisal. When asked about scheduling of the process, some wanted the rounds of discussions to be scheduled every six months. Having the benefit of group problem-solving focused on their own problems and objectives, they wished that opportunity to be made available more often than once a year. CONSULTANT OBSERVATIONS The consultant observed both the first and second rounds of peer discussion. She and the group discussed her observations on content and process at the end of each individual peer discussion. The group learned from this evaluation to recognize and interpret nonverbal gestures, to gain confidence in situations which demanded confronta- tion, to assign problems to appropriate groups for solving, and to assume various group process roles as needed. In her evaluation of a group without a designated leader, the consultant observed various leadership behaviors. The discussions were led by the interviewee or any other member of the group. If the interviewee appeared to want to be in control of the discussion, he or she would lead the discussion. In this case, other group members would assure that all sections of the self-assessment form were covered by intervening when needed. If the interviewee seemed to not want to initiate the discussion, another group member, usually self-selected, would take the lead by asking the first question. By the second round, interviewees tended to lead their own discussions. As they became more experienced in the process, all became more skilled in isolating problems, working through conflict, giving specific examples of an individual's strengths and needs for improvement, and understanding when certain personal traits needed to be accommodated. Members of the group were supportive of one another. They helped those lacking self-confidence to make statements about themselves; they were empathetic about problems and frustrations shared; they helped one another analyze problems; they offered options for solutions; they listened; they reminded one another that the five competencies needing improvement were to be provided for in the objectives; they cajoled, if needed. Phrases that became part of each discussion included "What could help you?" and "How could we help you improve?" Each discussion ended with "Have we discussed everything you wanted to discuss?" Peer Performance Appraisal of Librarians 179 CONCLUSION The peer evaluation technique met all four objectives: honest co- assessment, constructive criticism, problem-solving for individual and group, and clarification of functions, objectives, and priorities. The technique offered both self and group assessment of factors that affected an individual's performance. Participants voiced satisfaction with both their self-assessment, which involved responding to the questions on the self-appraisal sheet, and the group discussion, which assessed, validated, and corrected the individual's responses. Two-thirds considered what was said of them to be accurate and specific. Both parts the self-appraisal and the group discussion were necessary to achieve this objective. As the group accepted and used confrontation and conflict resolution, they reported even more openness and satisfaction with their personal appraisals by others and their appraisals of others. The technique met the objective of constructive criticism. The participants viewed the criticism of themselves as accurate, helpful, and specific. A few wished for even more specific comments. The group adopted the practice of helping individuals incorporate changes which needed to be made into the following six months' objectives. The problem-solving portion of peer evaluation is one of its outstandingTeaTures. The group problem solving for the individual with its clarification of issues, examples of specific situations, and suggestions of strategies and resources the individual may use led to evaluative comments such as the following: Changes in technology are accompanied with changing relationships and procedures. Peer evaluation makes us aware of how to use our strengths in such situations. We need peer evaluation every six months for problem solving. I have everyone's attention on me and my problems for an hour wonderful! Alleviates feeling of being overworked when we know we will have an opportunity to problem-solve. The clarification of functions, objectives and priorities were achieved by use of the self-appraisal form as well as within the discussion. The individual was asked to self-identify these on the form, and the group used these self-identified items as a basis for their discussion. Perceptions were clarified and changed by the discussion. The peer evaluation process as used by the Ramsey County staff accomplished the objectives set by the group as well as fulfilled the functions of performance appraisal identified in the literature. Its success might be attributable in part to the Ramsey County staff's having experience with various participative techniques, trusting in their leader, being accustomed to innovative approaches used in the library, and having other experiences with team organization. However, the use of 180 Evaluation of Public Services & Personnel this technique clearly improved the problem solving of the group, the self-knowledge of the individual, and the clarification of functions, j objectives, priorities and perceptions. This system of employee performance evaluation strengthens and fits in with participatory management in two valuable ways: (1) the process contributes to establishing an environment of trust, and (2) the process provides the opportunity for the communication and discussion needed for coordination and setting of priorities. The work of individual professionals and the various group projects involving some but not all of the group can be integrated into a logical departmental plan. Trust is established and fostered by the repeated experience of discussing their individual jobs and objectives and their commitment to providing information for people. The individuals' self-esteem is enhanced by seeing themselves as their colleagues see them and by having their colleagues validate their self-assessment. They are "empowered" by this enhanced self-esteem, by the support of and the help given them by their colleagues, and by their acknowledged commitment to the projects each manages. They and their colleagues benefit from reaching mutual agreement on each person's objectives and priorities. As an additional benefit, the clarification of objectives and functions and agreement upon priorities allow the individual to ask for and receive help without worrying about personal image and status as perceived by others. Discussing activities and projects as they relate to each individual and that person's work objectives allows for continual readjustment of priorities and refinement of projects as each person's perspective is taken into account by the group. A work group such as the Ramsey County reference department can truly become a team which minimizes the effect of individual shortcomings, which benefits from the strengths of each individual, and which creates a whole greater than the sum of its parts through using management techniques that recognize coworkers as co-equals, such as the peer appraisal process described herein. Subsequent Experience with the Peer Appraisal Process Five years after the initial experience with peer appraisal, the Ramsey County reference librarians are still using this process. In the intervening five years, the process has been expanded so that the majority of Ramsey County Library employees are evaluated in this way. Five years have resulted in few changes in the way the reference department practices peer appraisal. The list of competencies has proved remarkably stable. Twice in five years, a formal process of revision of the list has been carried out. The first revision resulted from the Peer Performance Appraisal of Librarians 181 introduction of an integrated automated system into the library. Changes and expansion were needed in the Information Technology section of the list; changes in database searching procedures also provided some new competencies for this section. An example of an added competency is "is able to teach users to do computerized searching." At the time these changes were made, the entire list was reviewed by all the reference librarians but no other changes were made. The second revision was handled with the assistance of the state library consultant. She provided the librarians with some readings about the future of library service and then met with them to brainstorm new or revised competencies. Except for some editorial changes to clarify meaning, the established list was not changed. A carefully prepared list of competencies appears to need little revision except for the changing terminology of new technologies; the Information Technology section seems to need revision about every two years. Working out the optimum scheduling and timing of the appraisals has been a continuous tension between what would make the process work best, how much time can be devoted to the formal process, and the annual requirements of the County. During the first year and a half, the feeling was that individual objectives needed to be monitored by the group about every nine months. Since the nine-month interval did not fit the County's requirements, in 1988, a midterm process was skipped. However, letting a whole year go by made it too easy to forget one's objectives and priorities, whereas a six-month interval seemed too onerous on the work schedule. An alternate midyear process was tried in 1989 in which each librarian did a one-page written report on his/her personal objectives. Copies of these reports were handed out to each person in advance and then a one-hour group planning conference was held. Group planning and priority-setting were accomplished in this session, but there was not time to deal with each individual. And the kind of trusting atmosphere necessary for individual concerns could not be established in so short a time period. Again in 1990, no midyear formal review has been held. What seems to be the best compromise between the needs of the library and the annual requirement of the County is for each individual to monitor her/his own objectives. He/she can then request informal feedback from the group or other individuals as desired. The most significant aspect of long-term use of the appraisal process is dealing with new librarians coming into the group, both transfers and job rotations from other Ramsey County Library branches and newly hired reference librarians. While the reference librarian group had been quite stable prior to the introduction of peer appraisal and 182 Evaluation of Public Services b Personnel for the first two years of its use, approximately one-half the work group changed as a result of the introduction of a job-rotation plan near the end of 1987. Integration of these new group members was handled in two ways. New employees were eased into the peer appraisal process. The County requires a six-month probationary period for them and they must be evaluated on the County form at three months and at six months. During their first six months, new employees participated in the discussion periods for their colleagues but were not evaluated by that process themselves. For those coming from other branches, individual educational meetings were held where the background and rationale for the process were explained, the experience with it and its evaluation were described, and the various forms were studied. These transfers were scheduled for their evaluations after they had participated in the peer appraisal sessions for the "veterans." This caused only the usual initial apprehension at the first time one is the subject of the peer discussion. Another effect of job rotation was that those former members of the reference group wanted to expand peer appraisal to the Ramsey County branch libraries. The change in group composition had some effect on the openness and trusting atmosphere of the group discussion sessions. A person's initial appraisal by this process is often primarily an informative or educational process for one's peers. The second time that person is evaluated in this way, more help is usually provided for solving one's problems. As the individuals get to know each other better, more trust f is established and confrontation of difficult issues is easier to do. To i some extent, changing the group membership temporarily sets back the effectiveness of the peer appraisal process for team building and individual development. However, as the process has expanded to other branches, individuals who have had experience with it in one location transfer their understanding of the process and tolerance for stress in the group discussion to the new location. In the long run, the rotation plan strengthens the process throughout the Ramsey County Library system. Expansion of the Peer Appraisal Process to Less Homogeneous Groups The first attempts at using this peer appraisal process by work groups composed of professional, paraprofessional, and clerical employees took place at about the same time in a local college library and in a branch library of Ramsey County. The staffs of both of these libraries were about the same size. The branch library had acquired Peer Performance Appraisal of Librarians 183 some of the original reference librarian group as a result of job rotation. They were eager to continue using peer appraisal but wanted to expand it to include the paraprofessional and clerical branch staff; the nature of branch library work is such that the various types of employees work more closely together and their roles overlap more than in a larger or headquarters library. To have a group of a reasonable size for discussion eight to twelve rather than three to five it was necessary to include more than the librarians. This same rationale applies to the college library. The major piece of work which needed to be done before the process could be expanded was to provide competency lists for other levels of staff. The college library asked the state library consultant to work with them in setting up their process. The consultant met with them and presented an overview of the process and the steps needed to start using it. They were able to begin their construction of competency lists with the King Study (1984) lists which were compiled for academic libraries. The branch public library needed to go back to some of the competency lists from the earlier Minnesota studies (1980) which included public library competencies in addition to the reference ones already mentioned such as "Staff-Patron Relations" or "Staff Communications." At the same time as these experiments were getting underway, the authors were also working with other public library assistant and associate directors to draft a "top management team" competency list for public libraries. This project started with generic management competencies as well as selecting the management competencies from the various librarian lists already mentioned. Part of this list of library management competencies, combined with some of the competencies from the Minnesota studies mentioned above, formed the basis for expanding the peer appraisal process to the management team of the headquarters library, which includes the nonlibrarian supervisors of the circulation services. This became the basis for developing lists for branch library managers by including the reference librarian competency list. This list also was used as a starting point for other circulation clerical workers at both the headquarters and branch libraries. It was possible to begin using the process with draft lists, refining them in use and combining them with training and orientation checklists which were also being developed for the library system. Another kind of job rotation plan led to the technical services staff adopting the peer appraisal process. Each librarian and paraprofessional and all full-time clerical workers in technical services work one day each week in public services. As a result of their public service day 184 Evaluation of Public Services & Personnel in the various libraries, some of them were participating in the peer appraisal processes in those public service libraries. They requested that the technical services department adopt the process. In 1989, the technical services staff was divided into two- to three-member subcommittees to work on the parts of competency lists for their work. Some of the subdivisions such, as management, had many ready-compiled lists. Others, such as processing, had to start from the very beginning, modelling their lists on the style of those already completed and using their training manuals as guides to the competency content. They developed lists for five competency areas which were used for their 1989 evaluations: acquisitions, cataloging, processing, management and communications, and computer/automation skills. The process is now sufficiently well-developed at Ramsey County Public Library that it can be carried on with only a little more time than a traditional appraisal system. Beginning in October each year, each week someone in the work unit is scheduled for an appraisal- discussion period to precede the weekly work-unit meeting. The group discussions often cover topics that would need to be discussed in the work-unit meetings. By the end of the year, most units have completed the individual appraisals and developed and prioritized their list of unit objectives for the coming year. They have a clearer understanding of how the unit and the individual objectives fit together than they would had they had traditional individual appraisals. . i li Peer Performance Appraisal of Librarians 185 APPENDIX A Self-Appraisal Form RAMSEY COUNTY PUBLIC LIBRARY Peer Performance Appraisal Summary Form Date I. Competencies. List the five that you think are your outstanding competencies or the five most significant competencies on which you rate yourself highly. Also list the five that are your top priority for improvement. Outstanding competencies Needs improvement II. List factors, both positive and negative, which affect your level of performance, and which you would like to discuss with the group. III. List your previous year's objectives. Write one or two sentences about your achievement for each one. IV. List your next 12 months objectives as suggestions for consideration by the group. IVb. List your long-range objectives. V. List your present duties and try to put them in priority order. DEPT. HEAD'S SIGNATURE SIGNATURE 186 Evaluation of Public Services is- Personnel APPENDIX B Reference Librarian Competency List O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) Nl - Needs improvement NET = No experience or training A. RESEARCH SKILLS Of critical importance (essential) Is able to analyze information needs with careful attention to detail. Understands how library materials and information sources are organized. Is able to match the best available information resource to the information need. Is able to use various search strategies. Is able to decide whether a manual or an on-line search is more appropriate. Is able to use information networks as appropriate. Is able to interpret information sources as appropriate Very important (should) Is able to use Boolean logic in conducting on-line searches. Is able to use print thesauri and on-line indexes to develp search strategies for on-line searches. Is able to compile bibliographies. B, COMMUNICATION SKILLS Of critical importance (essential) Is able to accurately comprehend the oral communications of others. Is able to remember, evaluate and use data obtained through listening. NI NET Peer Performance Appraisal of Librarians 187 APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI Needs improvement NET = No experience or training COMMUNICATION SKILLS Of critical importance (essentiaD(cont'd) Is able to orally express or present ideas and factual information clearly and effectively. Is able to use interviewing techniques to determine the individual's information needs. Is able to teach individuals how to use information sources. Is able to use bibliographic instruction techniques appropriate for groups. Is able to interpret library policy, goals, services, and procedures for individuals or groups. Is able to give directions clearly. Is able to translate between users, their needs, and information sources, translating information into terms used by both. Is able to work with users of all ages appropriately and fairly. Is able to convey to the public knowledge of materials and services. Establishes initial climate that facilitates open communication. Is able to balance the need for efficiency and friendliness in telephone reference transactions. NI NET 188 Evaluation of Public Services tr Personnel APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training COMMUNICATION SKILLS (con't) Very important (should) (cont'd) Is able to interpret and/or sum- marize information accurately for telephone reference transactions. Is able to perceive and react to the feelings and needs of others. Is objective in perceiving own impact on others. Is able to clearly express concepts and informatioa in writing, in well- organized and good grammatical form. Is able to use non-verbal communication effectively. Is able to evaluate the individual user's response to information provided. Of moderate importance Is able to work with individuals, local media and other groups using appropriate techniques to promote reference service. Is able to use questionnaires and discussion techniques. Is able to conduct meetings with individuals and groups both within and outside the library. Is able to convey the image of friendly, professional library service in contacts with others. NI NET Peer Performance Appraisal of Librarians 189 APPENDIX B (Cont.) Q = Outstanding S = Superior (aBove average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training KNOWLEDGE OF COMMUNITY Very important (should) Is familiar with community demographic, social, economic, and political informatioa Is able to identify specific needs of clientele groups. Is able to anticipate future needs based on knowledge of the community. Knows current events in the community. Is familiar with institutions, organiz- ations, agencies and industries within a community. Knows history of the community. Is aware of the relation of a community's political structure to a library. Participates in community organizations. Is able to work with community groups and agencies on cooperative projects. MANAGING SKILLS AND KNOWLEDGE Of critical importance (essential) Knows when to accept or delegate responsibility. Is able to identify problems, research relevant information, identify possible causes of problems, and suggest workable solutions. Is able to set, modify and follow through priorities. NI NET 190 Evaluation of Public Services b Personnel APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training MANAGING SKILLS & KNOWLEDGE Of critical importance (essential) (cont'd) Is able to develop and maintain good working relationships with personnel in other library areas. Knowledge of the operation of other sections in the library and how they work. Is able to perform effectively under pressure with frequent interruptions, and when faced with difficult tasks. Is able to manage personal and task time effectively. Has political skills, e.g., planning strategies for accomplishing objectives. Very important (should) Formulates and interprets reference policies. Is able to organise the available personnel resources to optimize strengths and compensate for weaknesses. Is able to utilize appropriate inter- personal styles or methods in order to effectively guide individuals (sub- ordinates, peers, supervisors) or groups toward task accomplishment. Is able to train and develop staff. Is able to establish procedures to monitor and/or regulate processes, tasks, or job activities and responsibilities of subordinates. NI NET Peer Performance Appraisal of Librarians 191 APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training MANAGING SKILLS AND KNOWLEDGE (confd) Very important (should) (cont'd) Is able to evaluate personnel, using appropriate standards, measures and methods. Is able to develop alternative and appropriate courses of action based on logical assumptions and which reflect factual information and rational and realistic thinking. Is able to develop new and innovative services. Is able to measure and evaluate reference service. Knowledge of evaluation methods and techniques to evaluate systems, services and products. Is able to collect, analyze and interpret data. Is able to manage a budget. Is able to anticipate long-range needs of the library. Is able to design systems and procedures to improve library operations. Is able to arbitrate and negotiate among staff. Has knowledge of statistical description, analysis, interpretation and presentatioa Has knowledge of standards, measures and methods for evaluating personnel. NI NET 192 Evaluation of Public Services fr Personnel APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NT = Needs improvement NET = No experience or training MANAGING SKILLS AND KNOWLEDGE (cont'd) Very important (should) (cont'd) Has knowledge of alternative management structures and their implications for the operation of the library. Encourages innovation and new ideas of others Is able to work as a member of a group to reach decisions & accomplish tasks. Has knowledge of the costs associated with library resources (materials, personnel, space, etc.) Has knowledge of cost analysis and interpretation methods. KNOWLEDGE OF INFORMATION RESOURCES Of critical importance (essential) Is familiar with the mission, goals and objectives of the library. Knows the policies and procedures relevant to the library. Is able to use the expertise of the entire staff. Is able to identify and use community information or referral sources. Has a broad generation knowledge in order to interpret patrons' questions and information sources. Is familiar with the expanding information community, its participants and their interrelationships (social, economic technical, etc.) NI NET Peer Performance Appraisal of Librarians 193 APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training KNOWLEDGE OF INFORMATION RESOURCES Of critical importance (essential) (cont'dl Knows the characteristics & use of the most commonly used information resources. Very important (should) Is familiar with the entire library collection. Is familiar with alternative approaches to the organization of information, e.g., classification schemes. Is familiar with the literature of various subject areas, both fiction and non-fiction, especially those of primary interest to users. Is familiar with authors and titles, both current and standard. Knows the arrangement (structure) of information resources in all formats. Is able to identify appropriate resources of other libraries. Is familiar with the operations of other sections of the library and how they relate to reference. Of moderate importance Understands the relation of the publishing industry to libraries. Is familiar with the contracting process. NI NET 194 Evaluation of Public Services b Personnel APPENDIX B (Cont.) O - Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training ATTITUDES Of critical importance (essential) Is aware of the purpose of library service in society. Is committed to promoting libraries and library services. Is committed to equal service for all patrons. Willingness to draw upon and share knowledge and experience with others. Maintains a nonjudg mental attitude toward patron questions. Has the persistence to obtain requested information or to locate a correct source for information. Is alert toward recognizing and respond- ing to patron needs. Is committed to maintaining a high stan- dard of personal and professional ethics. Is tolerant of individual differences. Is sensitive to others' feelings in dealing with people. Participates in educational activities to improve her/his job performance. Is committed to protecting the patron's right to privacy in his search for information. Is willing to learn to use equipment necessary for library service. NI NET Peer Performance Appraisal of Librarians 195 APPENDIX B (Cont.) O = Outstanding S = Superior (above average) A = Acceptable (average/moderate) NI = Needs improvement NET = No experience or training ATTITUDES Of critical importance (essential) fcont'd) Is committed to defending the right of patrons to intellectual freedom in their pursuit of knowledge. Listens objectively to other people's ideas and suggestions. Accepts responsibility for decisions and their consequences. Is committed to participating in professional organizations. Is receptive and adaptable to change. Is committed to achieving user satisfaction. Very important (should) Is interested in and seeks to become better educated in a wide variety of subjects. PERSONAL TRAITS Of critical importance (essential) Respects others. Is tactfuL Is cheerful. Shows self-confidence. Has a sense of humor. Has self-control. NI NET 196 Evaluation of Public Services