College and Research Libraries I · By ROLLAND E. STEVENS The Reference Survey as an Administrative Tool A s MORE and more attention is paid to the university library survey and to the pressing need for uniform statistics among libraries, it would seem that the problem of evaluating reference service would come in for its share of the discussion. As a matter of fact, however, there has been little treat- ment of this phase of evaluation in the li.; brary literature published during the last three years. The general note of dis- couragement on which earlier discussions of this problem have almost invariably ended may account in large part for the apparent lack of interest in it in recent years. Various methods of measuring the effect- iveness of reference service I]ave been pro- posed, practiced, described, criticized, and abandoned. Stone1 lists ten methods that have been employed, none of which is wholly satisfactory. Hutchins/ in a few para- graphs under the heading "Inadequacy · in Statistics Gathered," surveys previous at- tempts to evaluate reference service. Mc- Diarmid3 describes various methods that have been used, pointing out the limitations of . each. Guerrier 4 qualifies the use of a time-unit in measuring reference service, listing several factors that may make such a unit variable. Hutchins and Stone are also critical of the time-unit. ·Henry5 points 1 Stone, E. 0. "Methods of · Evaluating: Reference Service." Library Journal 67 :2'96-98, Aprtl 194~. 2 Hutchins, Margaret. Intt:oduction to Reference Work. Chicago, Amencan Lrbrary Association, 1944, p. 199·201, 3 McDia·rmid, E.W. The Library Survey, Problems and Methods. Chicago, American Library Association, 1940, p. 138-41. 4 Guerrier, E . "The Measurement of Reference Serv· ice.'' Library Journal 61:529-31, July 1936. 5 Henry, Edward A. "Judging Reference Service." Library Journal 64:358-59, May x, 1939. out an internal contradiction in the phrase, "measurement of quality of service." Rid~r6 questions the possibility of finding a criterion of reference work. However, in a survey of the literat~re of the evaluation of reference service, two facts stand ' out. In the first place, nearly every study of this kind that has been re- . ported has been made in a public library and has had as its purpose the justification of the reference department to the com- munity by which it is supported. 70 Secondly, 1 criticism of the adequacy of current methods of measuring reference work has been uni- formly directed at their inability to measure the value of this service to the community or to measure the amount of this service for purposes of comparison between institutions. The point to be made is this: the fact that these methods have been used without suc- ess in previous studies by no means invali- dates their use in other studies having dif- ferent purposes. The current trend in the literature of this type of measurement seems to indicate a renewed faith in the adequacy of statistical measurement of reference serv- ice, where the goal of the survey is to answer specific questions about the reference department for administrative purposes or to study further certain aspects of reference work in general rather than to justify this service to the community or to the support- ing body. In support of this observation, 6 Rider, Fremont. "Library Cost Accounting." Li- brary Quarterly 6:331-81, October 1936. 7 Hutchins, op. cit., p. 198; Guerrier, E. "Measure- ment of Reference !::lervice in a Branch Library." A.L.A. Bulletin 29:632-37, September xo :l5· 252 COLLEGE AND RESEARCH LIBRARIES the recent studies of Cole, 8 Erlandson, 9 and Christ10 are cited. Each of these has as its end the study of one or more features of reference service, either in its general nature or from the point of view of specific administrative problems arising in a par- ticular library. For such purposes, refer- ence service can be measured adequately. However, as McDiarmid has pointed out, 11 the method and techniEIUes to be used must be determined in each instance by the spe- cific problem. The study reported in the following pages was planned by the writer from an adminis- trative point of view, following his appoint- ment as head of the reference department at the University of Rochester. The re- sults of the survey were intended, in gen- eral, to serve as guides of administrative policy: to indicate the extent and variety of the use of the reference service, the adequacy of the library's resources for refer- ence work, possible improvement of meth- ods, and lines of future development. Specifically, it was hoped that the survey would provide answers to the following questions: (I) What classes of users receive the bene- fit of reference service, and to what degree? ( 2) What books and materials are most used irl reference work? (3) What are the most frequent causes of failures in the reference service? (4) · What classes of inquirers are most fre- quently confronted by failures in the reference service caused by the lack of adequate sources? ( 5) What are the relative amounts of time being spent, at present, on the various duties performed by the department? ( 6) In what fields is the reference collec- tion weak? 8 Cole, D. E. "Some Characteristics of Referet:Jce Work." College and Research Libraries 7:45-51, Janu- ary 1946. 9 Erlandson, Ruth M. "An Analy~is o! the Wor)<: C?f the Information Desk at the Umverstty of Ilhnots Library." College and Research Libraries 5:36-43, December 1943. "'• 1·° Christy Robert W. "Recording Reference Service." College and Research Libraries 8 : 23-27, January 1947. 11 Op. cit ., p. 140·4x, 144. JULY, 1947 It should be noted that these questions apply only to a particular reference department, in which the study was made. No attempt is made here to outline, as a result of this study, administrative policies for reference departments in general. However, the gen- eral method used has a universal range, being applicable to any department in which ad- r:t:Iinistrative problems anse. Three-Part Survey In order to find adequate answers to these questions, the survey was planned in three parts. ·These were: ( I) a record of requests for information, ( 2) a record, dur- ing sample weeks of the year, of the time spent by each member of the staff on various duties, and ( 3) a check of library holdings against Mudge's Guide and against the three supplements covering the period I935- 43· The methods used in each of these parts will first be described in some detail. Following this will be given a very brief account of the , specific results obtained. Finally, a criticism will be given of the methods used and some general suggestions for further studies of reference service. Before I946 only a daily count of refer- ence questions was kept, plus a fuller state- ment for future reference of some of the more difficult questions, including the sources in which information was found. Since the beginning of the survey (in July I 946) , a regular form for recording refer- . ence questions has been used. Questions that can be answered immediately, either from the reference worker's knowledge of the library or with the aid of a ready refer- ence book, are not recorded. Questions re- quiring any search are recorded on cards, giving the following data: (I) Short subject heading (2) Date ( 3) N arne and position of the inquirer (4) Manner of receiving the request: phone, mail, personal 253 ( 5) Statement of question as received ( 6) Sources consulted ( 7) Time spent (8) Remarks: inaccuracy of original in- formation or .of statement of question, reason for not finding, difficulty in finding, etc. For questions not answered or only partially answered, a red line is drawn across the upper right-hand corner of the card. Several objections to this method of re- cording questions may be anticipated here. For the immediate purposes of the survey, as well as for long-range measures of refer- ence work, it is unnecessary to count "fact" type questions. It will be shown that ques- tions of this group take only a small fraction of staff time and, from an adminis- trative point of view, are negligible. Fur- thermore, there is no need of keeping a record of the source in which this informa- tion is found for answering similar questions in the future, since it can always be located readily. It may be objected further that the data included on the record for each question are more than the purposes of the administra- tive survey, listed above, would require. For example, the short subject heading and the date of request are not needed for answering the questions posed in the survey. But by the addition of these items, the record cards may be filed for convenient location of the information for a similar request. This use of the records has been made repeatedly. Time Study In order to supplement the information anticipated from an analysis of the records of reference questions, a time study was planned as the second part of the survey. This was to be a tabulation of the amounts of time devoted to various duties regularly performed by the department. The record- ing form used for this tabulation is a sheet ruled into vertical columns. In the left- hand column is entered the date. Each of the other columns is designated by one of the principle duties. In keeping the record it is necessary only that the staff member fill in the date and the number of minutes devoted to each duty in its proper ·column. Duties other than those for which columns are provided are entered in a final column headed "Miscellaneous," which is wide enough to permit a description of the duty. An instruction sheet; giving a detailed out- line of the specific operations to be entered in each column, accompanies the form. Since it was deemed impracticable to keep the record daily throughout the year, four sample (not consecutive) weeks, each cover- ing a different part of the month and of the school term, were selected. Analyses of these records as well as those for refer- ence questions, 'is described below. It is hoped that these records of time may later serve other purposes than that of the survey, such as providing data for position classifi- cation in the department. The third part of the study was to ascer- tain the adequacy of the library's collections for refere~ce work. The raw data for this part had previously been obtained by check- ing Mudge's Guide and its supplements against the library card catalog. Conse- quently, it was necessary only to assemble these data, compare, and interpret them. Analysis of recorded reference questions was made in various ways. The data were records of 127 reference questions involving some search for information. These ques- tions had been collected over a period of seven months and represented an· aggregate of time spent amounting to nearly one hun- dred hours. In the first analysis, the ques- tions were sorted according to class of in- quirer, as enumerated below: ( 1) Faculty ( 2) Administrative officers 254 I COLLEGE AND RESEARCH LIBRARIES . ' (3) Students (4) Other departments of the library ( 5) Local firms ( 6) Individuals not directly connected with the university ( 7) Other: municipal officers, institutions, etc. For each class, the number of questions asked, the percentage of the total number of questions, the time spent in finding the desired information, and the percentage of total time spent, were listed in tabular form. This same information was given in sum- mary form, combining groups I to 4 (those associated directly with the university)' and groups 5 to 7 (those not directly associated with the university). This table provided at a glance the answer to the first question for which the survey was planned: "What classes of users receive the benefit of refer- ence service, and to what degree?" The I 2 7 reference questions were now re-examined, and the sources used in locat- ing the information were tabulated accord- ing to L.C. classification. This table provided the answer to the second question of the survey: "What books and materials are most used in reference work?" Questions Not Fully Answered Those questions that had not been fully answered were grouped according to the following four sources of difficulty : I ( 1) Questions based on inaccurate informa- tion-unanswerable ' ( 2) Information probably not published ( 3) Not thoroughly searched (4) Sources of information not in the li- brary (if published). The relative numbers of unanswered ques- tions in each group supplied a direct answer to the third question: "What are the most frequent causes of failure in the reference service?" The related question, "What JULY~ 1947 classes of inquirers are most frequently con- fronted by failures in the reference service caused by the lack of adequate sources?" was also directly answered by dividing this group of unanswered questions according to the class of inquirer. In addition to the full answers furnished to the first four questions, a partial answer was found also to the sixth question, "In what fields is the reference collection weak?" This was obtained by classifying according to subject and counting those sources (not in the library) in which the information asked for in the · fourth group of unanswered questions may have been located. Although the weak points of the collection were directly indicated in this way, the evidence was too scanty to be• ac- cepted as final. Thus, by means of these various analyses of the recorded reference questions, answers, or at least partial answers, have been sup- plied to all the questions originally posed, except: "What are the relative amounts of time being spent on the various duties per- formed by the department?" For an answer to this question it is necessary to turn to the time study described above. In analyzing this study, a table was made, giving the total time spent on each study and the percentage of that time to the whole time of the sample period. For greater ease of comprehension, duties were grouped by their nature (ad- ministrative, reference, clerical, etc.) and totals were given for the various groups. A similar table was made to show the strength of the reference collection on the basis of the titles listed in Mudge and its three supplements. This table listed various subject divisions and gave for each subject the number of titles in Mudge and its sup- plements, the number of those titles in the reference collection, and the percentage of holdings. A summary table, giving the same information for broad subjects, was also made. The same information was ~lso bar- 255 graphed for easier comprehension. By this means, more detailed information was sup- plied in answer to the question, "In what fields is the reference collection weak?" The evidence of unanswered questions was confirmed by this information. Specific Answers The specific answers obtained by the sur- vey are probably of little interest to li- brarians in other universities. A few brief paragraphs will suffice, therefore, to des- cribe in general terms the picture of the reference service afforded by the survey and the nature of the policies deduced from it. The reference collection was found to be well fitted to the demands made on it, particularly the demands made by those users whom it is the library's primary func- tion to serve. This was ascertained by . a comparison of the subject classes of sources used in answering reference questions with the classes in which the library has a large proportion of · the titles listed in Mudge. Further corroboration was found in ·the fact that, of the questions not fully answered be- cause of the lack of needed sources in the col- lection, 94 per cent were questions originat- ing off-campus. Policies for future growth will be of two sorts: Current publications will be selected with equal emphases on all fields in which the users of the library have shown interest. On the other hand, want lists are being compiled of older, important publications which may be acquired in the secondhand market. These lists are being compiled first for those subjects in which the collection has been shown to be weakest and for which there has been the greatest de- mand. Second, the survey showed that more than a third of the recorded reference questions ' were asked by Rochester firms, municipal officers, libraries, associations, and individu- als having no direct connection at present with the university. It has long been the policy of the university to serve these groups insofar as that service does not conflict with the university's primary aims. It would seeru desirable to support this spirit of co- operation by continued aid to the com- munity, as long as it can be carried out without curtailing service given to students and faculty. Touching the staff of the department, the survey showed a satisfactory balance of professional and nonprofessional duties be- ing performed. It also pointed up the unduly large proportion of time being spent on one duty that is not ordinarily con- sidered a function of the university library, the compilation of a weekly calendar of events. It is interesting to note in passing the relative amounts of time spent in answer- ing questions of the "search" type (those recorded for the survey) and questions of. the "information" type (not recorded). Of the total time spent during the sample period in answering questions of both types, 88 per cent was accounted for by the former type while only 12 per cent was spent in answering "information" type questions. This figure accounts only for those reference questions that reach the reference depart- ment and does not represent the total num- ber of questions serviced by the library. Thus, it is affected by the relative positions · of the circulation department, the refer- ence department, and the card catalog, and by other factors. Nevertheless, for the ·ad- ministrative purposes which the . survey was intended to fulfill, these · figures are adequate in indicating the actual nature of the work performed by the reference de- partment. The significance of this propor- tion is that, from an administrative point of view, the search type question should be heavily weighted, or considered alone with- out _reference to the 1'information" type question. 256 COLLEGE AND RESEARCH LIBRARIES 11 Conclusion In these concluding paragraphs, I offer criticism of some features of the method described above and some recommendations for future studies of reference work. Much has been said in criticism of the use of the time unit in measuring reference service. Admittedly, this unit is inadequate when it is · the value of reference service to the reader that is being measured. The time spent in finding the answ~r to a question has no relation to the value of that informa- tion to the inquirer. It is also obvious that the time unit is of llttle value in attempts to measure the efficiency of the reference departmenf. The many variable factors that affect the amount of time necessary to locate requested information detract from its reliability as a device for measuring the efficiency of an individual staff m.ember or of the department as a whole . . But to the librarian who is planning the work of his department, a consideration of the amount of time spent on the various kinds of work and in serving various groups, is of impor- tance. With this information, he may cur- tail certain phases of the work that do not justify the time that is being spent or find some 1 other means of continuing the wor~ with the expenditure of less time. Used in this connection, the time unit affords an . objective criterion to correct or support a previous intuition. It is recogni.zed, of course, that additional, or different, information-than that included in this study may be recorded, suitable to the particular aim. Additional analyses may be mad~ even with the data of this study: for example, the analysis of subject classes of materials needed for the demands of the various groups of readers or a classification of the questions asked by the · various groups. It goes without saying that the determining factor in selecting the data to be recorded must always be the need by JULY) 1947 which the survey was called forth. Studies have been made in which readers' questions were classified and counted. In an adminis- trative study, this information has little or no value, as such. The administrator needs to know, not what questions are being \ asked, but what materials aQd resources are needed for answering these questions. , Thus, in the study described abbve, not the questions asked, but the reference materials used, were classified and counted. In the time record kept by each member of the departmen~, a direct attempt was made to learn how reference time is being employed. The form used for recording time was designed in such a way as to take the least possible time and effort to fill in. An average of about two minutes per day was required to keep this record. How- ever, the composition of this form presup- poses an accurate knowledge of each and all of the duties being performed and their relative importance. The column headed "Miscellaneous" should not be a catch-all for duties not provided for in other columns. The instruction sheet accompanying the form should tell what duties are to be entered under "Miscellaneous," as well as those to be entered in other ·columns. If the work is not thus well defined, it would be preferable to use a simpler form, con- sisting of only two columns: on the left a narrow column for the number o'f minutes spent on any single operation, and on the right a wide column in which a brief des- crip/tion of the operation may be written. This is, of course, a , more time-consuming record to keep, but under analysis it gives a more <~;ccurate picture of the work per- formed. It does not impose a form upon the description of duties. This time record was kept by each mem- ber of the reference department for a sample period of four weeks ranging over a period of sixteen weeks. Percentages of 257 total time spent at each duty were computed for each of the four weeks. They were also computed for the whole sample. The average deviations, when computed for each group of related duties, were too high (ranging up to g.r, or 24 per cent) to as- -sure reliability to the averages given. It is doubtful that ~ larger sample would give better resul;s, The solution see~s to be to report a range of values rather than a , single value for each duty. Though un- reliable for close predictions, the results of this study are still valuable, nevertheless, for planning the work of the reference depart- ment. A Special Bibliography A final word of caution pertains to the practice of evaluating the collection by the m~thod of checking against a general bibli- ography. Mudge's Guide was used for this purpose in the study described above, be- United Nations Documents I N THE SPRING of 1946 arrangements were made with the American Library Association for twenty-five leading univer- sity and public libraries to function as de- positories for United Nations documents. Accordingly, the United States delegation to the United Nations has been sending the mimeographed documents of the United Nations and the American delegation to these libraries several times a week. Each library also receives on subscription the com- plete printed documeittation on the United Nations. Each of these depository libraries has agreed to service this c