College and Research Libraries JAMES MICHALKO and TOBY HEIDTMANN Evaluating the Effectiveness of an Electronic Security System The article updates a 1971 study that recommended the installation of an electronic security system in the Van Pelt Library at the University of Pennsylvania. A pre- and postinstallation comparison of estimated loss rates is given. Also presented are an analysis of costs and benefits based Qn the new estimated loss rates and a discussion of factors affecting loss rate esti- mates and additional benefits. THE PROBLEM OF PROVIDING efficient physical access to documents has become one of major concern to librarians. Research in document delivery, user frustration, and manipulation of loan policies has demon- strated that physically accessible collections are fundamental in any consideration of li- brary effectiveness. The problem of book theft, as an obstacle to physical access, has attracted much attention and considerable study. The purpose of the present study was to update a 1971 analysis · of the value of an electronic security system (ESS) for a library collection. The 1971 study was conducted at _ the Van Pelt Library of the University of Pennsylvania to determine the yearly loss rate for documents that were permitted to circulate. Based on these estimated loss rates, a cost-benefit analysis was made to determine if the installation and operation · of an ESS was justifiable. The results of this 1971 study appeared in an article by Michael Bommer and Bernard Ford in this journal. 1 Acting on the positive results of this analysis, the University of Pennsylvania li- brary administration installed an ESS in August 1974. Preparatory to the actual in- James Michalko is business administrator and Toby Heidtmann is assistant to the associate director in the University of Pennsylvania Librar- ies, Philadelphia. stallation of the detection hardware, the li- brary had begun inserting sensitive pieces into new materials as they were processed and, for a six-month period, into older ma- terials as they were returned to the circula- tion desk. Thus all materials acquired since early 1974, reference materials , and mate- rials actually borrowed have sensitive pieces placed in them . With the system in operation for just over two years, the present study was under- taken to assess the impact of an ESS on the loss rate of protected materials. In addition, a loss rate for the entire collection was es- timated using the techniques of the 1971 study. METHODOLOGY Other studies of book loss have been conducted over longer periods of time2 and using a two-stage sampling procedure. 3 However, the method used here closely fol- lows the second method outlined in the 1971 study. 4 To determine the loss rate of recently acquired, protected documents and to estimate the loss rate for the circulating collection, a methodology based on certain assumptions and sample data collected at a particular time was used . In summary, the assumptions are (1) that the loss rate for documents correlates with the publication date of documents, (2) that the loss rate in the immediate past has been constant, (3) that the loss rate for the group /263 264 I College & Research Libraries • July 1978 of most recently acquired documents (1974 on) is similar, (4) that the number of docu- ments lost from those acquired in a particu- lar year is a direct function of the length of time these documents have been on the shelf, and, finally, (5) that the document loss rate is in direct proportion to the use rate. 5 Table 1 summarizes the calculations that result in the loss rate estimates. These loss rate estimates were derived as follows: An estimate was first made of the number of documents allowed to circulate in various publication classes by drawing a random sample from the shelflisr' and coordinating the proportions of cards in the sample classes with the estimated number of cards representing circulating materials. This was accomplished using the following algorithm: the estimate of total shelflist cards, derived from total shelflist inches times 100, was multiplied by the ratio of cards in the sample representing documents that are allowed to circulate to the total number of cards in the sample to yield the estimated number of cards representing documents permitted to circulate. Then the following formula was applied to each publi- cation class to yield the estimated number of documents in each publication class that are allowed to circulate, which results ap- pear in column 4 of table 1: ESTIMATED CARDS THAT CIRCULATE NUMBER OF --------X SAMPLE DOCUMENTS = SAMPLE CARDS THAT CIRCULATE IN EACH CLASS ESTIMATED NUMBER OF DOCUMENTS IN EACH CLASS (column 4, table 1} A second sample of the shelflist was taken to obtain a sample of documents acquired in 1974 and 1975, then a search was conducted during this two-week period for these documents to determine the number unac- counted for and presumed lost. The results appear in table 2. Following the Bommer/ Ford assumptions, a calculation was made to . estimate the annual rate at which these most recently acquired documents were being lost. Estimated yearly proportion of documents lost for the most recently acquired documents (1974, 1975) = 10/2 + 11/3 = .0102 849 A yearly document loss rate for each pub- lication date class was then computed by first computing the circulation rate of documents in each date class relative to class I (e. g., the circulation rate of class II divided by class I is .00659/.00595 = 1.108), based on a one-week sample of ·circulation data (column 3, table 1). Then these relative circulation rates were used to estimate the annual document loss rates for each class . Thus if documents in class I are being lost at the rate of .0102 per year (as determined above) then class II documents are being lost at the rate of (.0102) (1.108) = .0113 per year. These annual document loss rates are shown in column 7 of table 1. Finally, the annual loss rate for each class of documents is multiplied by the es- timated number of documents in each pub- lication date class. This yields an estimate of the annual number of lost documents and is shown in column 8 of table 1. Summing this column indicates that the estimated number of documents lost from the Van Pelt collec- tion during 1976 is 7,488. However, based on reported lost books searched for by li- brary staff, approximately 13 percent of lost documents eventually turn up. Thus 7,488 - (.13) (7,488) = 6,515 documents are ap- parently lost as a result of theft. CONCLUSIONS Perhaps the most reliable comparison that can be made between the two studies is the loss rate of the most recently acquired documents. This comparison comes closest to the one-group pretest-posttest design out- lined by Campbell and Stanley7 and is, of course, subject to the problems outlined by them. In 1971 the loss rate of the most re- cently acquired documents was estimated at • \)228 per year while in 1976 the estimated rate had dropped to .0102-a reduction of approximately 55 percent. Further, the es- timated loss rates for the circulating collec- tion as a whole dropped 39 percent-from .0079 in 1971 to .0048 in 1976. ]]l s ~ :S:J""' go:Z c '" ~]~:; ~~8 ~ .5 "'" c ~ i]l t- ~~~ "' ... c ... ....-! E~~~ p:; ~ ~§~ Q!l:l .§00 00 ~~ ·= lZ~ ~9 ll-~ ~5 ~~ -s ~ "'<:> 0 .... ~a~~ .... o E-<: u~ Q) Q) ~..:I Q)- ~ ~'-' ~~ ~ oz ~ Ucij < E-< ~< ~"' OE-< s;: ~< ::l Q)"' E-<0 ~~~~ (I)~ ~ Q)~ - c oE-< '" ·- ~u ~'o ~ ~ V"J...:I E-<...:1 zo ~u ~< "8 ::>E-< ·;::: 'o Q) u< ~ 8c c;,j -" Q) Q) ~"' ~ a ~ 0~ ::l v :z c p:;V) 0 ~~ ·= !l:lQ s~ z~ ~ (I) §:~ ll ~ 8 (I) < ~~~&;lns~~M~~~ ~ ...-f~"<1'~c<)...-f...-t ........................ r-: 00)~...-f...-f...-!~000 ~c-)00 ~~~~~§ ...-t~t- 8c<)...-f...-f C1C1C1~~ -~~~ . ~~"'£5~c:j~&;~~ 0~...-t """'"""'~"""' • ...-t~t-~"<1'c<)...-t...-t~ .....;,....;,....; . . . . . . . ~...-tO O)...-t000)...-t~"<1'~c<)~ ;;t;~£5~~"<1'&> ~rn v5ct:ict:i~c-:i~,....; . .. O)~~O)~c<)c<)O)t-~ f;s ~~~~ ~~~~~ 0 oor--:- ~ ei co\6 g riCO)~ 0 r-: t-0"<1'~"<1'0) 0...-t...-t ~ ...-f...-f...-f...-f """'~"""'"""' ........ ~S:~;;t;&;~~~~~ "<1't-0~"<1'~c<)...-t ........ "' ~~n=;~;~~ "'"' "' ~ &it-...-f...-f...-f...-t...-f...-f...-f...-t...-t...-f --=22:;;..;>~~~:< Electronic Security System I 265 COST-BENEFIT ANALYSIS The 1971 analysis of costs and benefits an- ticipated a 90 percent reduction in the loss rate due to the ESS. The present study shows that estimate to be overly optimistic. Nevertheless, an analysis of actual costs and benefits clearly demonstrates the beneficial impact of the ESS . A summary of costs ac- tually incurred for the purchase and opera- tion of the ESS in Van Pelt Library is: Initial one-time costs: Purchase and installation $11,622 Sensitive pieces 19,986 Annual costs: Sensitive pieces Maintenance $31,608 $ 7,894 523 $ 8,417 No account is taken of the cost of instal- ling the sensitive pieces in each book be- cause that job replaces the pasting of bookplates and, therefore, involves no additional expense . The analysis of benefits to the library is based on cost-saving estimates derived from the ongoing lost book replacement program in Van Pelt Library. For these cost-benefit calculations , one refinement in the earlier study has been attempted; the number of books assumed stolen has been separated from the total population of lost books , which also includes books borrowed and not returned. To determine the number of documents that would have been stolen had the ESS not been installed, we applied the 1971 col- lection loss rate to the 1976 estimate of the number of documents in the collection. This yields (.0079) (1,347,053) = 10,642 docu- ments that might have been stolen without the ESS. The difference between this number and the 1976 estimate of lost documents (10,642 - 7,488) gives 3,154 documents presumably not lost because of the ESS . Naturally, not all of the estimated lost books come to the attention of the library staff and become candidates for replace- ment. In the most recent full year, the li- brary actually replaced only 864 books that were lost as a result of theft. This is approx- imately 12 percent of the 7,488 books esti- 266 I College & Research Libraries • july 1978 TABLE 2 SAMPLING OF DocUMENTS LoST Document September~ober 19'11• Number of Number of Document September~ober 1976 Number of Number of Acquisition Documents in Documents Lost Acquisition Documents in Documents Lost Date Sample as oE Sept. -{)d. Date Sample as of Sept.~. 1971 1976 1971 313 5 1975 332 10 1970 388 7 1974 511. ll 1969 365 17 TOTAL 849 21 1968 ~ a TOTAL 1391 43 •From table 2 in Michael Bommer and Bernard Ford, "A Cost-Benefit Analysis tor Determining the Value oE an Electronic Security System," Colkge & Resean:la Ubrarlea 35:274 Ouly 1974). mated as stolen during 1976. If the library did not have an ESS, then presumably 12 percent of 10,642 documents or 1,228 books would have had to be replaced during the same period. Thus the library avoided re- placement of (1,228 - 864) or 364 docu- ments. The average total cost of searching, evaluating, and replacing a stolen book over the last full year was $22.63. Consequently the library avoided spending ($22.63) (364), or $8,237, because of the ESS. The Benefit/Cost (B/C) calculation used in the 1971 study is duplicated here with changes in only those values that reflect ac- tual operational experience. The subjective values associated with estimated benefits to the library (L) and estimated benefits to the user (U) as well as the time frame for con- sideration are carried over without change from the 1971 study for purposes of com- parison; i.e., (L + U = $10,000, n = 15). The Benefit/Cost ratio is then: B c n I (8,237 + L + U) t = 1 --------- = $1.73 n 31,608 + I (9,417) t = 1 A break-even analysis shows the pay-back period to be 3.2 years. The B/C ratio and pay-back period compare favorably with the 1971 values particularly in light of the high degree of effectiveness that was anticipated but not realized for the ESS in the prelimi- nary study. DISCUSSION As mentioned above, subjective values (L + U) were held at the levels estimated by the first study in the calculation of B/C and pay-back for purposes of comparison with this earlier study. However, this leaves out several important factors in the present analysis that result in a conservative com- parison. For instance, it seems that benefits to the library and the user are larger than estimated in 1971. The ESS has, in its physical presence alone, a public relations value by proclaim- ing to the community that the library is ac- tively taking steps to prevent book loss. A similar message is conveyed to staff mem- bers, particularly the door guards, who now have an effective control tool, and the stack personnel, who appreciate the efforts to protect the collection. In addition, the user is rewarded by a higher success rate in re- covering the popular materials that might otherwise be stolen. Raising the value of L and U would, of course~ increase the $1.73 B/C ratio. In addition to these subjective factors af- fecting the valuation of the ESS, there are many factors that affect the critical estimate of the loss rate that cannot be identified in a study such as this-collection policies, course requirements, student population, user attitudes, staff thoroughness, and so on. Two factors should be mentioned that seem to impact on the loss rate more di- rectly. First, the percent of the collection Qutfitted with sensitive pieces. In a large collection, such as the University of Pennsylvania's, it is prohibitively costly to protect the entire collection. With only a portion of the books protected, a burden is shifted hack onto the thoJ;l<)ughness of exit checks. Moreover, the user population could, with some effort, discover which vol- umes are protected and which are unpro- tected and concentrate their efforts on steal- ing these. Essentially this problem is a var- iation on the truism that determined thieves can beat the system, electronic or other- wise. For the purpose at hand, the second factor-the circulation rate-may be even more important. Over the five-year period between studies, a dramatic increase in cir- culation activity occurred. In the two areas examined in this study-most recent acqui- sitions and the whole collection-both ex- perienced a heavy rise in demand. In .1971 the circulation rate of most recent acquisi- tions was 2.415 per 1,000 documents while in 1976 it was 5.949 per 1,000 documents. This is an increase of more than 240 per- cent. As a whole, the collection had a circula- Electronic Security System I 267 tion rate of .97 per 1,000 documents in 1971. In 1976 despite a large increase in the population of circulating documents, the rate had soared to 3.18 documents per 1,~a rise of more than 320 percent. Be- lieving that book loss and book use are cor- related directly, it is justifiable to assert that the ESS has had a positive effect i~ limiting the increase in book theft that circulation rates would lead one to expect. SUMMARY RESULTS The effects of an ESS on the Van Pelt Li- brary collections were to reduce the loss rate of most recently acquired items by 55 percent and reduce the overall collection loss rate by 39 percent. For every dollar the ESS has cost, approximately $1.73 in ben- efits have been received. At this rate, the system will be paid for in thirty-eight months. REFERENCES 1. Michael Bommer and Bernard Ford, "A Cost-Benefit Analysis for Determining the Value of an Electronic Security System," Col- lege & Research Libraries 35:27~79 Guly 1974). 2. Powell Niland and William H. Kurth, "Es- timating Lost Volumes in a University Library Collection," College & Research Libraries 37:128--36 (March 1976). 3. Bruce Miller and Marilyn Sorum, "A Two- Stage Sampling Procedure for Estimating the Proportion of Lost Books in a Library," jour- nal of Academic Librarianship 3:74-80 (May 1977). 4. Bommer and Ford, "A Cost-Benefit Analysis." 5. Ibid., p.273. I 6. The sampling technique used relies on the method explained in Herman Fussier and Ju- lian Simon, Patterns in the Use of Books in Large Research Libraries (Chicago: Univ. of Chicago Pr., 1961), p.151. 7. Donald. T. Campbell and Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research (Chicago: Rand-McNally, 1963), p.7.