College and Research Libraries Letters To the Editor: Recent articles such as those by Whitlatch 1 and Douglas2 have discussed the use and ap- propriateness of unobtrusive testing of reference services. Unobtrusive testing, like any other methodology, merits rigorous review and assessment. The purpose is to better un- derstand situations in which a particular methodology is most appropriate, while at the same time identifying and addressing, where possible, methodological weaknesses. Un- fortunately, misconceptions about, and incorrect attributions to, unobtrusive research per- sist. Unobtrusive testing is only one of many possible research/evaluation methodologies. Articles suggesting that unobtrusive testing is an inferior technique for evaluating ''refer- ence services" distort the research record. First, unobtrusive testing only examines specific aspects of reference service. Although data resulting from unobtrusive testing can provide useful diagnostic information regarding a range of other reference service activities, the method focuses attention on reference staff responses to factual and bibliographic ques- tions. Second, Murfin and Gugelchuk3 and Benham and Powell, 4 among others, have used other methodologies to assess staff responses to reference questions. The findings of these studies have supported those resulting from unobtrusive research. Studies that apply unobtrusive testing or any other methodology must incorporate stan- dard research practices. The Hernon and McClure study5 as well as the work of Crowley and Childers6 provide specific information about the criteria guiding the development of test questions and about the steps taken to ensure the collection and analysis of quality (reli- able and valid) data. These reliability and validity controls were carefully implemented and reported in the studies. In fact, Hernon and McClure summarizes these methodological refinements. 7 In contrast, the Douglas study provides no information about the reliability and validity of the data he reports. Similarly, Whitlatch gives no information on the reliability and valid- ity of her obtrusive data. Rather, the reader must assume that there is reliability and that the research design permits the drawing of conclusive and widely generalizable findings. Whitlatch's discussion of content validity misses the point. Obviously, factual and biblio- graphic reference questions do not represent all reference questions received at the refer- ence desk. It might be noted that content validity can be viewed from other perspectives . . An informed discussion and assessment of unobtrusive testing necessitates careful at- tention to methodological issues and questions centering around reliability, validity, and (perhaps) utility of the data and findings. Neither Douglas nor Whitlatch referred to the article ''Quality of Data Issues in Unobtrusive Testing of Library Reference Service'' by Hernon and McClure. In fact, that article discusses a number of their concerns. Thus, re- cent criticism of unobtrusive methods and quality of data issues tends to be simplistic and repetitive and ignores existing research on the topics. Additional examples of misunderstandings about, and incorrect representations of, un- obtrusive research dot the literature. And, despite some problems, Whitlatch's piece is a 701 702 College & Research Libraries November 1989 good example of exploring some of these issues in a constructive manner. However, con- trary to her assertion, the literature already contains examples of constructive uses of unob- trusive testing in specific libraries, for example, Stephan and others8 and Williams and We- dig.9 The key issues with unobtrusive testing are to (1) advance knowledge and the discourse regarding the evaluation of various types of reference services, (2) increase the profession's awareness of evaluation overall and within specific library organizations, and (3) develop strategies for improving the quality of reference services. Our experience suggests that while there is much discussion about the evaluation of ref- erence services, actual formal evaluation (regardless of type of evaluation), overall, occurs much less frequently. Thus, we are pleased that unobtrusive testing has increased profes- sional attention on issues related to the quality of reference service, "correct answer fill rates," and techniques for assessing the quality of various aspects of reference services. Such discussions, however, are better served when the participants are fully informed on the issues; when they carefully craft their studies to address specific research questions, especially if they attempt to prove/disprove a particular hypothesis; and when they de- scribe indicators of reliability and validity for the data they report. PETER HERNON, Simmons College CHARLES R. McCLURE, Syracuse University REFERENCES 1. Jo Bell Whitlatch, "Unobtrusive Studies and the Quality of Academic Library Reference Services," College & Research Libraries 50:181-94 (Mar. 1989). 2. Ian Douglas, "Reducing Failures in Reference Services," RQ 28:94-101 (Fall1988). 3. Marjorie E. Murfin and Gary M. Gugelchuk, "Development and Testing of a Reference Transaction Assessment Instrument," College & Research Libraries 48:314-38 Ouly 1987). 4. Frances Benham and Ronald R. Powell, Success in Answering Reference Questions (Metuchen, N.J.: Scarecrow, 1986). 5. Peter Hernon and Charles R. McClure, Unobtrusive Testing and Library Reference Services (Norwood, N.J.: Ablex Publishing Corp., 1987). 6. Terrence Crowley and Thomas Childers, Information Service in Public Libraries (Metuchen, N .J.: Scarecrow, 1971). 7. Peter Hernon and Charles R. McClure, "Quality of Data Issues in Unobtrusive Testing of Library Reference Service: Recommendations and Strategies," Library and Information Science Research, 9:77-94 (Apr.-June 1987). 8. Sandy Stephan and others, "Reference Breakthrough in Maryland," Public Libraries 27:202-3 (Win- ter 1988) . . 9. Saundra Williams and Eric Wedig, ''Improving Government Information and Documents Refer- ence Skills through a Staff Development Program," RQ 24:143-45 (Winter 1984).