College and Research Libraries CAROL A. SALVERSON The Relevance of Statistics to Library Evaluation There has been increasing concern about standardizing library statis- tics. Yet conventional statistics are inner-directed and static measures. A library system should be evaluated in terms of function related to use, i.e. its efficiency and effectiveness. Several methods used in eval- uation are examined. The concept of the library as a system is de- veloped and an approach to evaluation is outlined. In conclusion this paper considers the relevance of statistics, and indicates briefly what types of data should be collected. L mRARY STATISTICS have been gathered systematically since the nineteenth cen- tury. Categories traditionally covered in- clude sources of income; expenditure in terms of personnel, books, materials, and other expenses; size of collection; circu- lation; and type of personnel. There has been increasing concern about improv- ing this type of data by establishing more exacting definitions. Yet these statistics have mainly focused on the internal as- pects of the library and emphasize its more static characteristics. There is need for inquiry into the meaning of statistics, as they relate to evaluation of the library system, comparisons between systems, and developing overall concepts of li- brary service. This paper proposes the formulation of statistical methods and standards which are more relevant to the goals and function of libraries. Miss Salverson is Assistant Librarian in charge of Readers' Services in Jefferson Community College, Watertown, New York. 352/ Statistics are quantitative measure- ments of stated facts or processes. The meaningfulness of a statistic depends on the purpose for its use, the significance of that purpose, the relevance of the statistic, the accuracy of compilation; and the interpretation. Statistics are used to give a concise and objective repre- sentation of facts. They provide a sim- plified summary of complex data, for ease of comprehension and objectivity. They can be used for cross-comparison. Statistics are necessarily an oversimpli- fied description of facts, however, and should be thus interpreted. It is highly important in statistical analysis that the elements to be quantified be selected carefully, to provide a truly representa- tive statistical schema.1 Library statistics are compiled for var- ious purposes-record keeping, support- ive material for influencing management, reporting to national agencies, indication of individual progress, and to provide 1 "Numerical News" (editorial), Library Journal, XCI (May 1, 1966), 2288. Relevance of Statistics to Evaluation I 353 objective means for comparing libraries. This paper is concerned with the rele- vance of statistics to the purpose of eval- uating libraries. The validity of a meas- urement depends on the purpose. celt is important, therefore, to avoid compla- cency because standard measures will soon be agreed upon, when in reality such measures have no real meaning in relation to our objectives."2 For example, the number of items circulated in a year is useful for a permanent record of circu- lation. If the circulation increases great- ly, it may be taken as a general indica- tion of expanded activity, and vice versa. Circulation figures are weak criteria of the library's use. Library patrons also read books in the library; obtain informa- tion by telephone and .from the refer- ence librarian. Evaluation is intrinsic to the library field. It is necessary in assessing the needs for library service, in appraising the library operation, and _ .in weighing resources so as to give insight into the significance of libraries; it is necessary for purposes of improving library prac- tices, and to aid in the formulation of objectives. Some librarians are opposed to the use of statistics · as being largely in- adequate for library evaluation: They feel that the quality of library service cannot be represented by numbers; oth- .ers have proposed mathematical formu- las for assessing a library. These positions must be considered. It is evident that some sorts of criteria are needed for evaluation, comparison, and establish- ment of principles. Subjective Evaluation Evaluation of a library by subjective means has the advantage of permitting a view of the system as a whole in terms of its objectives, of allowing considera- tion of all aspects of operations and 2 James Krikelas, "Library Statistics and the Meas- urement of Library Services," ALA Bulletin, LX (May 1966), 498. subtle distinctions involved in quality of service. It requires extreme perceptive- ness by the librarian, as well as detach- ment from daily routines. Also, it is only possible where the librarian has close contact with all staff and operations. Thus this method can be useful only in the smaller libraries. It can, however, also result in a biased estimation on the part of the librarian. He may think the library is ccsuccessful," when actually many patrons have ceased coming be- cause of lack of information, or other such unrecognized factors. Furthermore, few libraries can continue to operate in isolation, and those which do are not a benefit to the field. Such a library's ac- tivity and progress can only be stated in subjective terms. This does not pro- vide objective verification for the librar- ian's findings and does not easily lend itself to comparison by others . Mathematical Formula Evaluation by a well based mathemat- ical formula might provide verifiable re- sults in an objective form. It would rate the library on a quantified scale, and would be useful to other libraries for comparisons. Beasley3 suggested the fol- lowing measure: ~ = fi·s ~=books per capita Where: B =books (resources) C = measure of users (broader than circulation) 3 From a presentation b y Kenneth E. Beasley, direc- tor of the Kansas Legislative Council, and a member of the faculty, Department of Political Science at the University of Texas, at the Conference on "Meas- urement and Evaluation in Library Research," Sep- tember 10-13, 1967, Champaign, Illinois. Described in a memorandum from Mary D. Quint to Robert W. Levesque, Syracuse University Research Corporation, September 28, 1967. 354 I College & Research Libraries • July 1969 P = population S = research/ study factor (deter- mined · by another ratio of physical facilities, non-fiction, periodicals, personnel, hours of service) The original model is more detailed.4 Extended criticism could be made of this formula. It could be said that the number of books to be provided per per- son is inadequate as a formula for "meas- urement of library service."5 This does not indicate how well all the services are functioning to bring the user in contact with needed information. It does not consider quality of the collection, but as- sumes that there is a "positive correla- tion between quantity and quality."6 "There is, however, no known evidence to demonstrate that size is correlated to quality or service in any way."7 Further, the v- was used because circulation is less signi_ficant .as it increases. Assuming this to be true, why not use 3 v - or 4y- ? Each of these would likewise in- dicate that circulation has less influence as it increases. It seems to be a rather ar- bitrary operation. It is difficult to assess critically this formula without more knowledge of how Beasley would sup- port it. However, it is being used here as an example of how completely quanti- tative formulas are greatly inadequate for evaluating the complexity of a library situation. They tend to be poorly found- ed in principle, do not take all variables into account, and are unsophisticated in design and mathematical operations. The result is not a meaningful assess- ment of library service. Conventional Library Statistics Conventional library statistics are largely inadequate for measuring the • Ibid. 5 Ibid. 8 Ibid. 1 Krikelas, op. cit. , 497. - system, since they are static and inner- directed. The library as a system should be measured in terms of function related to use. Expense is measured in terms of budget categories, yet it does not relate costs to time, space, and other variables. The collection is measured in number of volumes, often divided by form of material. This count is some indication of potential of the collection for provid- ing service, but it does not show quality, subject distribution, and how much it is used. The figures could be related to those for circulation, but it at best would give only a crude estimate of use. The circulation statistic has been discussed previously. Statistics for type of person- nel describe them as professional or cler- ical without relation to their activities. The standard statistics do not measure many aspects of library service, such as in-house use of materials, how many peo- ple actually come to the library, the ways in which patrons seek information, and the role of the reference staff. Use Studies Emphasis in recent years has been shifting toward learning the .aspects of library use-how many users, their char- acteristics and needs. There have been a great number of use studies, often du- plicative, using non-standardized meth- odologies. Often sampling techniques and data gathering are· poorly conceived. Comparison of studies is difficult. Some use studies have tended to focus on the expressed desires and opinions of the pa- trons without integration with the sys- tem objectives. The studies to date have been generally unsophisticated in experi- mental design; also, few libraries have undertaken this type of analysis. Some statistics that are of real value in assess- ing library activities and clientele have been obtained by these methods. For example, Behling, at Ohio State U niver- sity, used a questionnaire to ascertain Relevance of Statistics to Evaluation I 355 who was using what facilities in the li- brary on a particular day. It was found that nearly 55 per cent of the persons completing the questionnaire were us- ing the library only for study.8 This sort of statistic has important implications for the particular library, and it should also encourage other librarians to con- sider such assessment. Systems Analysis Systems analysis is another method of evaluation which is receiving more at- tention in the literature. "One wonders why systems analysis techniques, so helpful in developing information sys- tems themselves, are not applied to the development of tools for evaluating these systems."9 Most studies of systems at present are undertaken largely in con- nection with advanced techniques of in- formation retrieval and dissemination. This includes a great number of rele- vance assessments, which center on one aspect of a specialized system. Studies have taken place largely on an indi- vidualized basis, without standardized methods. They may not be based on the system objectives. Only a limited num- ber of studies have focused on the prob- lems of the conventional library system. For example, some studies have been made of work flow in technical process- ing. At this point, it is desirable to look at the library as a system. The Library System A library provides access to informa- tion for a group of users. The system concept implies an organization of facil- ities, resources, and personnel in a de- sign which implements the institutional s Orlando Behling and Kermit Cudd, "A Library Looks at Itself," CRL, XXVIII (November 1967), 416-22. 9 Alan M. Rees, "Evaluation of Information Sys- tems and Services," Annual Review of Information Science and Technology, II ( 1967), 8. objectives. Library service is a process; as such it should be dynamic and flexi- ble, adapting to the need for change. Prior to the establishment of the library, and contributing to its reason for exist- ence, there are two elements: a group of people with a need for information and the information. There is a barrier between the two, i.e., the people who need something cannot get what they need. The librarian surmounts this inter- face by providing an approach for the user and bringing the information to him. The broad function is shown in the first diagram. The librarian must define the user population, its characteristics, and needs, in order to select from the broad cate- gory of information those materials which are most relevant. He must organize these materials and provide channels of ac- cess. Means should also exist for the use of materials beyond the library collec- tion. A librarian does not have direct knowledge of why a reader desires in- formation, what he does with it, or how it affects him. The librarian has contact with materials and with users making re- quests. Yet, for efficient operation the li- brary needs feedback from outside its domain. The organization is managed by ad- ministrators. The remainder of the staff can be roughly divided into technical services (from selection of material through preparation) and readers' serv- ices (circulation and reference). Proc- esses are coordinated within depart- ments, and ultimately are directed to- ward service, as shown in the second diagram. The collection (and availability of out- side resources ) is central to the internal workings of the library. Its materials, ar- rangement, and indexing are provided by the technical services; direct access is provided through readers' services. The collection should be dynamic, expand- ing, shifting in emphasis, and decreas- 1. 356 I College & Research Libraries • July 1969 Information I I Int~rface I I I I I I Library Staff I I I . r I I I Users 2. r-----------------''------------------1 ------ ----- 3. Technical Services I I I ~ I I I I I Administration I • I I I I ; .. Readers Services --- ~ I Ind~xes I ~ Technical/'----;.'---....... ............_ · Servkes I "- \\ 1 i ij Collection I I I I I I Administration I ~----~ ~---- Readers Services ---~ ~--- Relevance of Statistics to Evaluation I 357 ing in some areas, according to demand, as in the third diagram. The schema is becoming increasingly complex. This provides for some conceptualization of the problem of evaluating. Criteria for Evaluation In evaluating the library we must con- sider the efficiency of its internal op- eration and its effectiveness in imple- menting the goal of service to users.10 These two criteria are necessarily inter- related. A library is established on the basis of a service objective. The organi- zation of physical facilities, resources, · and staff provides a means for the ful- fillment of this objective in practice. Both aspects are important in maximiz- ing the potential of the library. An in- efficient library creates a waste of ener- gy and cost; even though it may be user- oriented, the possible extent of service is necessarily decreased. Likewise, a library which is ineffective in serving users may function smoothly and efficiently, but the continuation of such a system has little value. The most basic formula for determin- ing efficiency of an operation is: 0 output E = - efficiency = --- 1 input Lipetz translated this into terms useful for communication systems. 11 useful results ( services ) E = --------- costs involved The cost factor includes expenditures in terms of materials, supplies, person- nel, and equipment. If the cost is re- 10 This distinction was made clear by Karen Breed- ing, a student at Syracuse University School of Li- brary Science, in a preliminary draft p~per on Re- search Libraries and a National Informatwn Network, May 1968. 11 Ben-Ami Lipetz "The Evaluation of Commu- nication Systems: S~me Thoughts on Effectiveness," Sci-tech News (Winter 1966), 110-13. duced, for example, by shortening the time required to catalog a book, effi- ciency is increased. There are many var- iables involved; considerations of time, space, and motion affect cost in various complex library operations. Thus a de- termination of the cost factor cannot be arrived at by means of a simple formula. We can change one small variable, how- ever, keeping others constant, and then observe whether expenditures have de- creased over a period of time. The service factor is less tangible. It includes all products of the library op- eration which can be useful to patrons: reference service, cataloging and index- ing, classification, preparation of bibli- ographies, the book collection, circula- tion. If these results are improved or in- creased, the efficiency ratio increases. This area is also subject to many vari- ables, but, in addition, it is less quantifi- able. For example, if we expand the col- lection, we may improve the services of- fered; however, this requires a qualita- tive judgment of what makes better serv- ices. The interrelation between these two variables is not always considered by li- brary planners. In the past there was ex- treme conservatism about expanding services; emphasis was placed on cost. Now there is a more general availability of funds .and much emphasis on research in developing new technologies for im- proved service. There is a tendency to think that an innovation is justified if it can provide more sophisticated service, e.g., selective dissemination of informa- tion, computerized file search, or long- distance xerography. The practical im- portance of the cost factor is often neg- lected. The efficiency ratio is suggested only as a guideline for aid in evaluation. It cannot be quantified as a simple per- centage; methods at present are too un- sophisticated for the defining of signifi- cant data, compiling, quantifying re- 358 I College & Research Libraries • July 1969 suits, determining of mathematical rela- tionships, and assigning of constants. Thus there is an indication that we may be able to analyze statistically more in future than at present. The J"atio is di- rectly concerned with certain qualitative evaluations, however, such as the judg- ment of the value of a collection, which cannot be wholly quantified without los- ing our perspective. · The ratio implies that a resulting per- centage will be less than ideal. In a sys- tem we do not seek to achieve perfec- tion, but a workable organization. We must determine what degree of evalua- tion is desirable in terms of improved service. This must be seen as a long- range and cooperative investment also. A detailed evaluation by a few libraries can benefit a great number of others. The latter can eventually learn from the experience of the former, without as great an expenditure. According to Lipetz12 the ratio of 0/1 consists of unknowns: an infinite variety of factors may be substituted. It only as- sumes meaning in the context of the li- brary's objectives. What is the library at- tempting to achieve? Are its aims valid? Then we must consider existing opera- tions or new planning in relation to how the institutional goals can best be im- plemented, in terms of both cost and service. If a library provides a high type of services (defined in terms of its objec- tives) at a low cost, it has a high effi- ciency. Although this is a desirable stand- ard, it is not sufficient, since it does not indicate how or to what extent the li- brary is, in effect, serving users. The cri- terion of effectiveness must also be ap- plied. We must determine the relation- ship between level of service offered and actual use. The goal is that the ex- pressed institutional objectives be imple- mented in practice. 12 Ibid. As stated previously, there is a great emphasis at present on improved serv- ices, and all types of libraries are gain- ing access to federal funds. As a result librarians may stress the need for ac- quiring new books, facilities, or person- nel without considering the total cost in relation to how much this will increase actual use. Thus, a costly change which only moderately increases the use of the system is not worthwhile. It is evident that the elements involved in assessment of services are necessarily complex. At this point it is interesting to con- sider the criteria of efficiency and effec- tiveness in relation to Daniel Gore's ar- ticle, "The Mismanagement of College Libraries."13 Mr. Gore stresses his belief that libraries exist to provide books for readers, but that our college libraries do not provide the number of books need- ed for good service. Meanwhile, he claims, a great deal of money is being spent on librarians performing much clerical work. He maintains that by keep- ing funds constant, we can replace many professional personnel with less expen- sive clerical help, further reduce expense by instructing students in use of the li- brary, and use the increased funds made available for the purchase of books. All this is accomplished "without sacrificing anything of value."14 In our formula of efficiency Mr. Gore has kept expense constant; therefore, we must consider services. The student has been instructed in the use of the library: this has "virtual- ly eliminated the need for the ... serv- ices of a reference librarian."15 The serv- ice factor has been improved by mak- ing instruction available to students, but it has also been limited by omitting a regular reference librarian. Let us as- sume, as apparently Mr. Gore does, that these two changes in variables balance 1 3 Daniel Gore, "The Mismanagement of College Libraries," AAUP Bulletin (Spring 1966), 46-51. 14 Ibid., 51. 1 5 Ibid., 47. l Rele..;a,nce of Statistics to Evaluation I 359 each other out, and the service factor remains constant. Also, the book collec- tion is being rapidly expanded; this will then increase the service factor and re- sult in a more efficient system. Now let us apply the effectiveness criterion. The increased efficiency has resulted from an increase in the number of books, but how effective is the result- ing situation? To quote from "Guidelines for quality in college libraries" : 1 6 However, the presence of a supportive col- lection is, in itself, not enough. The de- gree to which faculty and students use the library as a means of extending their cul- tural frontiers is very definitely one meas- ure of the calibre of the instruction that takes place on campus, as well as an in- dication of the amount of emphasis the administration places on intellectual mat- ters. (italics added here) What indications are there that the ex- panded collection has proved more use- ful to the student? Further, it is doubt- ful that a student can master the use of the library to the extent that he does not need reference assistance through a single course in bibliography. At present, it is indicated that even one year of full- time study for librarians may be insuffi- cient preparation for understanding the many and complex approaches to find- ing information (granted that much of this training has other objectives, e.g., administration). It is possible that the larger quantity of books has had little benefit and that the curtailing of reader assistance has greatly minimized the ef- fect of the library on education. If this is true, then Mr. Gore's measure of ef- fectiveness has decreased . Or, perhaps the effectiveness has remained the same· i.e., the students have learned to do o~ their own all that they needed help for previously from the reference librarian. If such a low level of reference assist- ance is required in practice, it would ap- • 16 New York State Education Deparbnent, " Guide- lines for Quality in College Libraries," ( Mimeo) , 5 p. pear that the educational goal of ccself- education after college, and during it as well"17 is far from being effectively im- plemented. Statistics and Evaluation Evaluation of a system should be based on the scientific method. First, we must determine what the object of the system is; this has been discussed here in the most general way as service to users, but this must be better delimited in terms of explicit objectives. N ext, data must be gathered on the functioning of the system and its actual benefit to the users. One could recommend here the methods of systems analysis for internal library operations and the use study for obtaining data on the population. Statis- tics are to be used insofar as data are measurable or would be useful in quan- titative form. Statistical analysis is a tool for quantifying and classifying certain facts. Both selection of the particular ele- ments of the system to be analyzed and statement of the relationship between these elements are judgments which must be determined by the librarian. This relates to the purpose of the sys- tem. "Af.ter the librarian makes this de- cision, statistical analysis can be used to evaluate the most likely values of vari- ous constants."1 8 Work in this area is largely unsophisticated, at present. Therefore, it is best to begin with the most simple application of statistics. They should be used to the extent that is presently feasible , along with non- quantified information. In the final inter- pretation of a study the librarian will at- tempt to integrate the statistical data compiled with general considerations of other factors, and relate these to the ini- tially stated objectives. 17 Daniel Gore, op. cit., 50. 18 Marianne Cooper and H erbert W . Cooper, " In- t erpreting Results of Statistical Studies," CRL, XXVIII (July 1967) , 266. 360 I College & Research Libraries • July 1969 For controlled analysis of an existing system, it is advisable to concentrate on single aspects in relation to the whole. If we were to analyze the procedures in- volved in cataloging a book, and some- how shortened this operation, we could observe what effect this had on general efficiency, since other factors have been generally kept constant. As more knowl- edge of library systems is developed and methods are standardized, evalua- tion should become a regular part of sys- tem procedure. The methods of systems analysis will be used to study routine technical serv- ices operations, considering time, output, cost, space, skill of personnel, physical facilities , .and organization. This would make use of work-How charts and flow- process charts. Statistics would be com- piled in conjunction with this on factors determined to be significant, e.g., cost of cataloging a book, time required to veri- fy a title, how frequently a worker must resort to a certain cataloging tool. Simi- lar analysis can be used for studying the users and reference service. Steps re- quired to perform searches will be dia- grammed and compared. Time and mo- tion studies can be made for reference staff and users. A study was made at Wayne State University School of Medi- cine library "to determine the How pat- tern in the use" of the library for plan- ning of physical facilities. Locations oc- cupied were noted and the amount of time spent there. 19 Questionnaires and other techniques can also be utilized to gain information on user activities. Ref- erence requests can be noted and clas- sified. Again, selected statistical analyses could be made for such factors as num- ber of users, classified by group, by fa- cilities used, or by intent; number of re- 19 Saul Herner and Mary Herner, "Information Needs and Uses in Science and Technology," An- nual Review of Information Science and Technology, II (1967) , 13-14. quests for extensive bibliographies; and so forth. There is a need for standardization of statistics useful for system evaluation. Conventional cumulated statistics are not directly relevant to effective library service. The recent handbook published by American Library Association recom- mends that circulation figures for college and university libraries not be reported nationally-they represent only a seg- ment of library use, and involve so many local variables, that they are useless for national comparison. 2 0 It might further be suggested that standard methodolo- gies and type of data be developed so that libraries can be compared to some degree in actual use. In addition, sound criteria for technical services and readers' services are needed. Many aspects of li- brary operation, notably building of the collection and provision of reference service, cannot be completely quantified. Library service is affected by a complex- ity of factors, and also judgments of val- ue are n ecessary. If relevant objective system data can be provided to some ex- tent, then other librarians can relate this to their own services, and make their own interpretations. Society is now changing dramatically as a result of population expansion, vol- ume and diversity of information need- ed by it, increased interdependence of functions , and improved technology in communications, types of media, and electronic data processing. Librarianship needs to be reevaluated in terms of these changes, so that libraries can effective- ly serve the population. This requires re- vision of principles in the profession. Fundamental principles must be based on a broad range of data from the field. The data available at present are of lit- tle use in interpretation, since measures 20 American Library Association, Statistics Coordi- nating Project, Library Statistics: a Handbook of Concepts, D e finitions and Terminology (Chicago: American Library Association, 1966), p. 22. Relevance of Statistics to Evaluation I 361 of income, expenditures, collections, staff, and number of borrowers have lit- tle relevance to how libraries are func- tioning or what patrons do in relation to them. The field at present needs much more relevant and better coordinated statistical data. ERRATUM In the article by Richard W. Trueswell en- titled "User Circulation Satisfaction vs. Size of Holdings at Three Academic Libraries" in the May issue part of the key to Figure 5, page 210, was inadvertently omitted. Symbols shown on the graph relate as follows to the institutions listed: • Mount Holyoke 0 Goodell (U. Mass.) Q9 Deering (Northwestern) ••