College and Research Libraries On the Merits of Direct Observation of Periodical Usage: An Empirical Study Marifran Bustion, John Eltinge, and John Harer Measurement of current periodical usage is one of several tools used by librar- ians to make decisions regarding placement or deselection of serial titles. In an open-stack current periodical room, librarians may make such measurements through either direct observation of periodical usage or through less direct measures, e.g., reshelving counts or voluntary user responses. In principle, the direct-observation method is attractive because it may include broader classes of use than those covered by other measurement methods. Extended use of direct observation, however, is prohibitively expensive, and, if taken to an .extreme, may itself intrude on total periodical use. The authors conducted an empirical study to assess the practical merits of · direct observation work. The resulting data indicate that observational errors, aggregation issues, and costs limit the feasibility of long-term, direct-observa- tion use studies. With careful training of observers, however, direct-observation work may be useful in calibrating the results of other less expensive methods of measuring periodical use. uring the past few years, li- braries have had to manage unprecedented increases in serials costs with limited or shrinking budgets. The effect on collec- tion development has been a decrease in the number of monographs added, can- celed serials subscriptions, increased inter- library loan transactions, and decreased institutional research efforts. In addition, cancellation of serials sub- scriptions invariably lead to some con- cerns among faculty and students. In some cases, specific departments or administra- tors may reasonably ask libraries to justify certain cancellation decisions. To provide such justification, and to ensure that cancellation decisions are indeed equi- table and reasonable, it is useful to have available several objective measures of the Marifran Bustion is Head, Acquisitions Department at the Gelman Library, George Washington University, Washington, DC 20052; John Eltinge is Assistant Professor, Department of Statistics; and John Harer is Head, Circulation Division at the Sterling C. Evans Library,Texas A&M University, College Station, Texas 77843-3143. The authors express appreciation for the assistance of Irene Hoadley, Bill Hobson, Jane Oliver, Tommy Shaeffer, Richard Sellers, Tanya Wiggins, and Jo Ann Williams in the study. The authors also thank R. J. Bartlett and E. D. Alexander for computational work reported in the statistical analysis section. This work was supported in part by grants from the Sterling C. Evans Library Research Committee and the Texas A&M University Mini-grant Committee. The work of the second author was also supported in part by grants from the National Institutes of Health (GM39015-01A1) and the A. C. Nielsen -Company. 537 538 College & Research Libraries use of specific periodicals and groups of periodicals, and of the value of specific periodicals to a given institution. Use studies are often designed to ex- amine user questionnaires, interlibrary loan requests, photocopier use, reshelv- ing counts, and direct and indirect obser- vations of user behavior. All types of use studies have merit and rely to greater or lesser degrees on the cooperation or par- ticipation of users and library staff. User questionnaires rely heavily on user par- ticipation and cooperation; interlibrary loan and photocopier use depend on staff records; reshelving counts rely on user cooperation, but are more depen- dent on staff efficiency in record keeping and returning material to the shelves; and direct observation of user behavior relies heavily on staff involvement. , The authors designed the auxiliary s~dy to assess the degree to which direct observation permits effective measurement of periodical use not recorded in reshelving counts. Use studies are generally designed to address several questions, including estimation of total use and cost-per-use of specific periodicals; assessment of pe- riodicals through faculty ratings and ci- tation analyses; development of proce- dures for storage, cancellation, duplica- tion, and acquisition of back volumes; and development of issue binding and retention schedules. Each of these issues plays a role in periodical cancellation decisions required by reductions in ac- quisitions budgets. 1 Questions involving shelf space or use density are of second- ary interest.2 In the authors' use study, the principal measure of periodical use was a daily count of the number of reshelvings of each periodical. Some controversy exists regarding the use of a reshelving count as a proxy for the total use of a given periodical. For example, Colin R. Taylor reported a case in which reshelving counts represented only 22 to 40 percent of an alternative measure of journal use.3 November 1992 The implications of a similar undercount rate for the Texas A&M University study are threefold. First, such an undercount may lead to a discouraging and mislead- ing picture of the value of the library periodical collection to the university community. Second, the undercount may lead certain rarely used periodicals to be classified as unused. This misclassi- fication might lead to removal of such rarely used periodicals under a deselec- tion policy that employs zero use per se as an important criterion for cancella- tion.4 Third, and most important, differ- ences in the undercount rate across disciplines or periodicals may lead to reported reshelving counts that underre- present or overrepresent the true pro- portionate use of specific periodicals or groups of periodicals. This result in tum may lead to inequitable or inefficient cancellation decisions. These issues led the authors to con- sider direct observation of periodical use as an alternative to reshelving counts. In principle, a direct-observation study is more attractive than a reshelving-count study because the investigation may de- sign a direct-observation study to measure specific use types of interest and avoid the differential undercount issues described above. More practically, however, obser- vational errors, aggregation issues, inter- vention effects, and costs may limit the value of direct observation of periodical usage. To address these issues, the authors conducted an auxiliary study based on direct observation of current pe- riodical usage in the Evans Library. This auxiliary study permitted the authors: (1) to assess the merits of the direct-observa- tion method and (2) to compare the direct- observation use counts and reshelving counts for specific groups of journals on the same days. The present article addresses the first issue; subsequent papers will discuss the second. The remainder of this paper is orga- nized as follows. The Methodology sec- tion presents the basic methodology of the auxiliary study, including a descrip- tion of the periodicals studied, an outline of the randomized design for the direct observation of stacks, and the definition of "use" employed in the direct-observa- tion work. Methodology also discusses the assessment of observational errors and reports the costs of the direct-obser- vation and reshelving-count studies. The Statistical Results section reports some summary statistics and standard errors for the direct-observation method and also analyzes the structure of the observational errors encountered in this study. The same section discusses the degree to which these results may be generalized to other time periods and to titles and volumes not contained in the Current Periodicals Department. The section concludes with a discussion of some aggregation issues which arise in direct-observation studies. Finally, the Discussion section presents some general implications and limitations of the re- sults of this study. METHODOLOGY The authors designed the auxiliary study to assess the degree to which direct observation permits effective measure- ment of periodical use not recorded in re- shelving counts. The design involved observation of periodical use at randomly selected locations and times, a detailed operational definition of periodical use, and an assessment of the bias and variance of observational errors. Cost constraints were an important additional factor in the design of the auxiliary study. The Scope of the Study The investigators restricted the auxili- ary study to the Current Periodicals De- partment (CPD) of the Evans Library. Recent issues of approximately 7,500 pe- riodical titles were stored in the CPD. Throughout the study, the titles had an average age of approximately nine months (from date of receipt) and had an average of eight issues stored in the CPD. During the week of the study, approximately 1,600 issues were received and added to the CPD, and 660 volumes (approxi- mately 2,640 issues) were removed for binding. The auxiliary study did not consider titles or volumes outside the CPD. For one reason, differential undercount rates Observation of Periodical Usage 539 in reshelving measures were considered to be of particular concern for issues in the current periodical room. Different discip- lines may make different uses of recently published periodicals. Some such uses, e.g., careful reading of articles, are more likely to be measured by a reshelving count than are other uses, e.g., casual browsing. In addition, uses like casual browsing are more prevalent in the Cur- rent Periodical Room than in the general stacks. Thus, direct observations of peri- odical use may be more appropriate for use in a current periodical room. For another reason, the direct-observa- tion method used in this study would be less efficient in the general stacks area, where periodicals and regular books are interspersed according to Library of Con- gress call numbers. For an auxiliary use study in the general stacks area, user-re- sponse methods may be preferable.5 Observational Design and the Direct-Observation Method The auxiliary study recorded certain types of periodical use on randomly selected shelves at randomly selected times on three randomly selected days (Tuesday, April 17; Thursday, April 19; and Sunday, April 22) in the week of AprillS-22, 1990. The direct observation method employed was similar to direct observation methods reported by Jo- hanna Ross, Charles Wenger and Judith Childress.6 The week chosen to be studied was not randomly selected, however. It occurred one month after the use study was half completed. Also, it was neither the first nor last week of a semester, but was at a time students were completing term papers and study- ing for exams. The authors considered it a typical week during the academic year. Figure 1 presents the layout of the Evans Library CPD. In the discussion below, range refers to the side of a book- case that contains periodicals, and aisle refers to the floor area adjacent to or between ranges of periodicals. Both the general and the auxiliary use studies ex- cluded newspaper holdings. Thus, figure 1 displays a total of 51 relevant ranges arranged in 27 horizontal aisles. RE5HELVING AREA- OE-Z N/1/59 - NX/765/A7 L8/1 051/C678 - N/1/P83 K/25/N53 - LB/1051/838 JA/4/C2 - K/25/N5 HM/291/1595 - JA/3/P7 HG/11/M3 - HM/263/P767 HG/11/89 • HG/11/E2 HD/9000.1/578 • HF/6/843 HC/186/Al - HD/9000.1/J6 H8/1/E56 • HC/167/567/L37 GV/975/G7 • H8/1/E525 G8/841/J6 • GVI975/G6 E/185.5/J8 • G8/841112 D/839/C87 • E/184/575/H5 8F/233/P4 - D/839/CS37 RE5HELVING AREA - A-OD OP/501/A 18 - OR/360/J62 OL/801/J9 - OP/474/VS OL/461115 - OL/8011A45 OK/1/577 - OL/461/G49 · oH/505/A1/H4 • OK/1/568 OH/7/P495 • OH/505/A1/88 OE/1/J8 - OH/5/86 OD/271/J66 - OE/1/17 OD/1/A36 - OD/271/J65 OC/461/J63 - OD/1/A35953 OC/1/P45 • OC/451/15 OA/801/J682 • OC/1/P43 OA/76.5/NS - OA/801/A7 OA/1/J45 - OA/76.5/M522 0/11115 • OA/1/192 PR/91 OO/C25 • 0/111F65 PN/2091/E4 - PR/8700/J68 PF/3001/54 - PN/2081/R4/L5 P/1/A1/C22 - PF/3001/G3 FIGURE1 T511 080/C34 - Z/278/56 TP/757/P55 - T5/940/L4 268 TN/677/A1/0412 - TP/700/53 TK/7870/E543 - TN/672/J68 T J/825/W5 • TK/7870/E54 TD/201/W35 • TJ/810/593 TA/168/59 - TD/2011W345 T/1/A66 - TA/168/15 5/604.8/R39 - 58/599/F52 RK/71/JS - 5/601/A37 R/11/87 • RK/1/82 New York Times· Die Zeit El Paso Herald- New York Times AFL-CIO News - El Diablo Layout of the Evans Library Current Periodicals Department Observation of Periodical Usage 541 Sunday, April 22, 12:00 - 12:05 Observer: ________ _ 58 --Inside: ---------- 59 --Outside: --------- 6A --Inside: ---------- 6A -Outside: --------- Sunday, April 22, 12:05 - 12:10 Observer: ________ _ 11 B --Inside: ---------- 11 B --Outside: --------- 12A --Inside: 12A --Outside: --------- FIGURE2 Time and Aisle Assignments Twenty-four of the aisles adjoin 2 ranges (~.g., 7B and SA), while the remaining 3 aisles adjoin a single range (e.g., 1A). Since time of day is an important fac- tor in periodical use, the investigators partitioned the CPD operating hours into the following blocks: Tuesday, April 17 and Thursday, April 19 (8:00a.m. to midnight), 6 periods of 160 minutes each; Sunday, April 22 (noon to midnight), 4 periods of 145 minutes each, and 1 period of 140 minutes. Within each time block, each aisle was assigned at random to 1 or 2 five-minute segments. The columns of figure 2 reproduce an example of the resulting time segments and aisle assign- ments. The statistical literature describes this type of stratified sample design as controlled selection.7 Student workers observed the selected .aisles in the indicated time segments. Preliminary work indicated that an ob- server could distinguish clearly between use in the right and left ranges of an aisle and between use of periodicals in the inside and outside of each range. (The rules define the inside and outside of a range to be the halves of ranges closest to specific walls in the room.) The rules, listed below, explained to observers how to identify a use and record it: 1. Enter name in space marked Ob- server. 2. Make a single hash mark, /,each time you observe one of the types of count described below. 3. Be sure to mark your counts on your data form separately for the inside and outside half of each range, as indicated on the form. "Inside" is the end of the range closest to the wall by Acquisitions; "outside" is the end of the range closest to the window. Unless told otherwise, please observe from the outside end of the range. 4. Count any periodical issues that are returned to the shelf by any per- son who does not work in CPD. If a single person returns more than one issue to the shelf, write down your best estimate of the number of 542 College & Research Libraries issues which that person returned to the stacks. 5. Count each time a person uses a pe- riodical, even if he or she just picks up an issue and looks at the cover. 6. Count as one use a single person's use of one or more issues from a single stack of issues. If the patron removes an issue after picking up a stack, count as one use each issue removed and do not count the is- sues returned. If the patron does not remove any issues from a stack, but returns the entire stack to the shelf, count that stack as one use. 7. Count separately each time the same person uses periodicals from separate stacks of issues. 8. Count only the uses completed during the five-minute period that you observe a particular stack. 9. Do not count periodicals a person carries away from the aisle you are watching. Following the rules reproduced above and discussed in the Definitions section, the observers recorded separately the number of uses in the inside and outside halves of the right and left ranges of the observed aisle. These observations were recorded in the indicated columns of figure 2. To reduce the effect of observer fatigue on recorded use counts, no stu- dent worker observed aisles for more than two hours at a time. Definitions of "Use" and Observational Rules In defining use of periodicals for the direct-observation method, the authors assumed that patrons reviewed issues in the aisles to locate an issue containing a specific article and to locate information in a specific article or as a result of a random search (browsing). A casual re- view or browsing of issues may yield some useful information. Since obser- vational techniques are generally unable to distinguish actual value of each re- view, each casual review was considered important and was counted. For the pur- poses of this study, casual use was de- fined as any look at the cover, contents page, or information contained in the November 1992 body of the issue. A glance at the spine for volume number or date or handling of any issue to retrieve another issue was · not considered a use. To implement the direct-observation method, the investigators developed the direct-observation counting rules repro- duced above. The focus of these rules was to have an observer count every use of periodicals in an observed aisle, ex- cept for uses which subsequently would be recorded in a reshelving count. Some distinctions were required from the oper- ational definition of use outlined in the rules. For example, rule six gives a single count to one patron's use of one or more issues stored in a single stack. Otherwise, a patron's brief perusal of all of the issues in one stack would lead to an inflated count of use. Also, rule four directs the observer to estimate and record the number of issues returned to an ob- served aisle by a patron. The investiga- tors reasoned that patron removal and return of several issues of the same title indicated a greater intensity of use than the casual scanning of several issues con- sidered in rule six. Moreover, if the pa- tron had not reshelved the issues, each of the issues would have been included in the reshelving count. Thus, rule four is intended to parallel the implicit count- ing rule employed in the reshelving count. Finally, rules eight and nine address the limited observation times al- located to each aisle. Some preliminary observations indicated that some uses of a single issue in the aisles lasted one minute or more. In the absence of rule eight, the random observational design would give such long uses a higher prob- ability of being recorded than the short uses. To parallel the reshelving counts, the investigators chose to use rule eight to ensure that all patron uses of issues have the same probability of being re- corded, regardless of the length of use. Similar reasoning applies to rule nine. Assessment of Observational Errors In assessing the merits of a direct-ob- servation study of periodical use, it is important to measure the magnitude of observational errors associated with ~----------------------------------------------------------------------------- misinterpretation of rules by the student workers. To address this issue, the three investigators observed the same aisles as student workers in some randomly selected five-minute segments. Student workers recorded each observed use with a single hash mark (/), and the faculty members recorded each use with one of several symbols: S (scan), C (care- ful use), orR (reshelved by patron). A fourth use, T (taken away by patron), was also recorded by the investigators but is not directly relevant to the student workers' observations. Note that obser- vational rule nine for student workers specifically excludes the T usage re- corded by faculty members. Separate re- cording of these four use types allowed the investigators to study whether ob- servational errors were associated pri- marily with a particular use type. For example, an investigator might specu- late that careful use (C) is more clearly defined, and thus less subject to obser- vational error than scanning (S). The Statistical Results section provides an empirical discussion of this issue. For the remainder of this paper, the sum of the faculty S, C, and R counts for a given half-range will be defined as the true count of use (excluding staff re- shelving of issues). The difference be- tween the student use count for a given half-range and the corresponding true count equals the observational error in the student count. From a statistical point of view, this definition is restrictive, because it excludes the possibility that the investi- gators' observations may also contain measurement error. Nonetheless, this ap- proach appears to be reasonable. The fac- ulty members had a vested interest in the project; one generally associates such in- terest with greater attention to detail. In addition, the faculty observers were more familiar with the observational rules and the study's purpose. Consequently, the investigators expected fewer errors in faculty observations than in student ob- servations. Use of a more elaborate er- rors-in-variables model, which would allow for both student and faculty obser- vational errors, is beyond the scope of the present work.8 Observation of Periodical Usage 543 Costs of the Direct-Observation and Reshelving-Count Studies The direct-observation study used a total of 53 student-worker hours. At a pay rate of $3.80 per hour, the direct cost in student wages was $197.60. In addi- tion, the authors spent 20.5 hours in direct observation. If a direct-observation study were conducted using one student worker- hour for each of the 111 CPO operating hours per week, the cost, at $3.80 per hour, would be $421.80 per week. During the same week, the CPO used 50 student hours to record reshelving counts. These 50 hours were in addition to the 22 hours per week otherwise re- quired to reshelve periodicals in the CPO. Using the same pay rate, $3.80 per hour, the marginal student worker cost for recording reshelving counts was $190, a daily cost of $27.14. STATISTICAL RESULTS Assessment of the statistical results of this study requires considering basic de- scriptive statistics for the observed use counts; the distribution, bias, and variance of errors in the observed-use counts; limits on the generalizability of the reported re- sults; and aggregation issues. A paper by Bartlett and Eltinge presents a more detailed statistical analysis of the data from this auxiliary use study.9 In th_at paper Bar- tlett and Eltinge discuss some regression and variance component models for the faculty and student worker observa- tions; consider alternative models based on logarithmic data transformations and trim- ming of extreme observations; and evaluate least-squares methods of pre- dicting true usage counts based only on student worker observations. Counts by Student Workers and by Faculty Under the random design outlined above, student workers observed a total of 615 aisle-time combinations. Observa- tion of a given aisle led to use counts for two or four half-ranges in the specified time segments. For 792 of these half-range student observations, there was a match- ing faculty observation for the same half- 544 College & Research Libraries November 1992 TABLEt FREQUENCY DISTRIBUTION OF FACULTY AND STUDENT COUNTS Observed count 0 2 3 4 5 6 7 8 9 10 or more Faculty frequency 734 21 7 5 0 6 2 4 0 5 8 Student freguency 739 17 6 3 5 1 4 5 2 0 10 TABLE2 FREQUENCY DISTRIBUTION OF THE ERRORS, (STUDENT COUNT)- (FACULTY COUNT) FOR THE 58 NONZERO FACULTY OBSERVATIONS Observed error -9 or less -8 -7 Frequency 4 2 Observed error 2 3 Freguency 6 0 2 range at the same time. Table 1 presents the frequency distributions of the stu- dent counts and faculty counts for these matched observations. Note that for both student and faculty observations, a large majority of the use counts equalled zero. Table 1 also indicates substantial differ- ences between the frequencies of student and faculty counts. As noted in the Methodology section, the authors used the working hypothesis that faculty observa- tions were the true use counts. Thus, obser- vational error is defined as any difference between a student count and a corre- sponding faculty count. The authors ex- pected the observational error process to be fundamentally different for zero and nonzero true counts. For example, with a true count equal to zero, a student obser- vation cannot be less than the true count. Consequently, the authors discuss below the differences between student and faculty ob- seiVations for the 58 paired obseiVations with a nonzero faculty observation; some possible explanations of variability in the same set of observational errors; student observations for the 734 paired observa- tions in which the faculty observer re- corded no auxiliary periodical use; and other statistical aspects of this study. Observational Errors for Nonzero Faculty Counts Table 2 reports the frequency distribu- tion of the differences, (student observa- -6 4 3 -5 -4 -3 4 2 2 5 6 7 1 -2 5 8 0 -1 13 9or more 2 0 8 tion) - (faculty observation) for the 58 matched observations with a nonzero faculty count. For these 58 pairs, the fac- ulty counts had a sample mean of 5.41 and a sample standard deviation of 6.15; the student counts had a sample mean of 4.39 and a sample standard deviation of 8.01. The differences between the stu- dent and faculty counts had a sample mean of -1.02 and a sample standard deviationof5.29.Aformal testofthenull hypothesis of no difference between the overall mean student count and the overall mean faculty count, against the alternative hypothesis of a nonzero difference between these two means,led to at-statistic equal to -1.47 on 57 degrees of freedom. This test statistic was not significant at the 0.10 level of signifi- cance. In practical terms, this means that sufficient evidence does not exist to con- clude that the student observations were systematically higher or lower than the corresponding faculty observations. Given this conclusion, we may study the variance of the differences between the matched student and faculty counts. One useful measure of this variability is an estimated reliability ratio, defined in this case as equaling the sample variance of the true observations, divided by the sample variance of the student observa- tions.10 For the 58 matched pairs under consideration, this estimated reliability ratio equals 0.59, with an estimated standard error equal to 0.19. In practical terms, this point estimate of the reliabil- ity ratio suggests that about 59% of the variability of the student counts is at- tributable to variability in the true counts, and the remaining 41% of vari- ability in student counts is attributable to measurement error. However, the large standard error indicates the rela- tively poor precision of this point esti- mate. The estimated reliability ratio of 0.59 is not entirely encouraging, but also is not entirely out of line with reliability ra- tios for some social-science measurements based on complex concepts.11 Relationships between Observational Errors and True Use To study further the observational error question, figure 3 presents a plot of the 58 student observations against the corresponding nonzero faculty observa- tions. Strike-overs in the lower left section of the plot indicate multiple observations at the same points. The plotting symbols are letters representing the different stu- dent observers. If there were no measure- ment errors, the matched student and faculty observations would be equal, so that all plotting symbols would fall on a straight line with a slope of one and an intercept of zero. The distance of a given point above or below this straight line in- dicates the magnitude of measurement error in a given student observation. Define Yijk to be the use count recorded by the ith student for the kth half-range in the jth aisle, and define Xijk to be the corre- sponding faculty count. Then a linear re- gression model (see chapter 1 of the textbook Applied Regression Analysis by N. R. Draper and H. Smith,12) for the Yijk and Xijk observations is, Yijk = bo + btXijk + eijk MODELl where bo and b1 are the fixed intercept and slope of a straight-line model relat- ing Yijk to Xijk, and eijk is a residual term accounting for random variation in the student observational errors. The conjec- ture that there was no systematic under- Observation of Periodical Usage 545 count .or overcount in the student obser- vations is equivalent to the null hypothe- sis that bo = 0 and b1 = 1. An F test of this null hypothesis against the alternative hypothesis that (bo, b1) does not equal (0,1) had a test statistic F = 1.07 on 2 and 56 degrees of freedom, and was not sig- nificant at the 0.10 level of significance. An estimated generalized least squares fit of model1 with a variance component model for random student, aisle, and half-range effects led to an estimate ofb1 = 0.98, with a standard error of 0.10; and an estimate of bo = -1.11, with a standard error of 1.63.13 In addition, note that in figure 3 obser- vations by certain students appear to be systematically higher (e.g., student A) or lower (e.g., student B) than the corre- sponding faculty observation. However, for a variance component model for the residuals of the regression model de- scribed above, a formal analysis of vari- ance test of a "student effect'' led to an F test statistic equal to 1.96 on 9 and 7 degrees of freedom. This test was not sig- nificant at the 0.10 level of significance. As noted in the Methodology section, the true use count is the sum of observed S (scan), C (careful use), and R (re- shelved by patron) counts. Review of the counting rules listed above suggests that errors in the student observations differ across different types of use. Estimation of two regression relationships helps evaluate this suggestion. First, an ordi- nary least squares regression of the er- rors Yijk - Xijk on the corresponding faculty Sijk, Cijk, and Rijk counts led to the estimated equation, Yijk- Xijk = -1.00- 0.24 Sijk + 1.10 Cijk - 0.44 Rijk + dijk MODEL2 where the error term dijk will be dis- cussed further below. The intercept and coefficients for &jkt Cijk, and Rijk had esti- mated standard errors equal to 0.88, 0.12, 0.33, and 0.31, respectively. An in- formal interpretation of model2 is that, after accounting for the effects of the other variables in the equation, and after 546 College & Research Libraries November 1992 H g- H (/) c: 0 ~ 2: Q) H (/) ..0 0 'E ~-Q) "0 :::J Ci5 F ~- J 8 8 J R D J F K H D J c D 8 J A J 8 8 F D D D D D F 0- 88 B B 8 I I I I I 0 5 10 15 20 25 Faculty Observations FIGURE3 Plot of Student Observations against the Corresponding Nonzero Faculty Observations Observation of Periodical Usage 547 TABLE3 FREQUENCY DISTRIBUTION OF STUDENT OBSERVATIONS WHEN THE CORRESPONDING FACULTY OBSERVATION IS ZERO Student observation Frequency 0 719 2 4 2 accounting for an overall undercount or overcopnt rate, a significantly positive coefficient for a given independent vari- able (Cijk, at the 0.01 level) indicates that the student workers tended to overcount this type of use, while a negative coeffi- cient (for Sijkt at the 0.05 level) indicates that student workers tended to under- count the corresponding type of use. By contrast, if there were no differen- tially systematic undercount or over- count of any use type, we would expect each of the slope coefficients in model 2 to equal zero. A test of the null hypothe- sis that the S, C, and R coefficients were all equal to zero had an F test statistic equal to 11.24 on 3 and 54 degrees of freedom; this test was significant at the 0.01 level of significance. Second, we may expect the variability in observational errors to vary across types of observed use. To address this issue, the squares of the residuals dijk from model 2 were regressed on the fac- ulty Sijk, Cijk, and Rijk counts. The result- ing estimated model was, dijk2 = -0.06 + 4.18 Sijk + 6.95Cijk - 0.09 Rijk + error MODEL3 with the estimated standard errors for the intercept, and the coefficients of Sijkt Cijk, and Rijk equal to 6.2, 0.89, 2.3, and 2.2, respectively. An informal interpreta- tion of model 3 is that independent vari- ables with coefficients significantly greater than zero (Sijk and Cijk, each at the 0.05 level of significance) are the use types associated with greater variability in nonsystematic observational errors. In addition, if variability in nonsys- tematic observational errors was the same for each use type, then we would expect each of the slope coefficients in 3 4 5 6 7 8 3 0 2 2 1 model 3 to equal zero. A test of the null hypothesis that the S, C, and R coeffi- cients were all equal to zero in model 3 had an F test statistic equal to 13.94 on 3 and 50 degrees of freedom. This test was significant at the 0.01 level of significance. Observational Errors for Zero Faculty Counts Table 3 reports the frequency distribu- tion of the student observations matched with faculty observations that equalled zero. Note that for 719 of the 734 faculty zero counts, the student also recorded a zero count. The relatively low frequency of student overcounts in this case is not surprising because, for most faculty counts equal to zero, there were no patrons in the aisle under observation. For the remainingl5 cases, the student recorded a use count between 1 and 8. It appears that most of these overcounts were associated with misinterpretation of rules four, six, and nine for times in which a patron was in the observed aisle. Generalization of Results This study, like many use studies, does not permit one to make formal statistical inferences to populations other than the CPD and period studied . Generalization of the results reported here depends on the comparability of the Evans Library to other libraries in terms of definitions and measures of periodical use, training of direct-observation workers, and peri- odical use patterns. First, variability in administrative inter- ests may lead to legitimate differences in definitions and measures of total periodi- cal usage. For example, as noted in the Methodology section, the definition of total use of current periodicals depends substantially on the degree to which one recognizes casual use. Second, the specific statistical results on observational errors 548 College & Research Libraries apply only to observers with general back- grounds and observational-rules training similar to that of the student observers. Since the student workers employed in the direct-observation study were regularly assigned to other duties in the Evans Li- brary CPD, they had some familiarity with library patrons and periodical use. Their training in the direct-observation rules, however, involved only a brief (thirty minutes) review of the observation rules. Given the relatively small reliability ratio reported in the Statistical Results section, it is clear that more extensive training for work- ers in a long-term direct-observation study is preferable. A separate reliability study could then assess error magnitudes for these more extensively trained observers. Third, periodical use patterns may themselves influence the extent to which observational errors are of practical con- cern in a direct-observation use study. As noted in the Methodology section, obser- vation of a single careful use is fairly straightforward and is generally un- likely to result in a measurement error. Casual use, however, requires the ob- server to exercise some judgment and thus is more likely to result in measure- ment errors. Thus, an investigator is more likely to be concerned about obser- vational errors for libraries and times in which casual use is a substantial com- ponent of periodical use. Aggregation Issues Aggregation places an additional con- straint on the utility of a direct-observa- tion study. Some measures, such as reshelving counts or voluntary user re- sponses, associate a use count with a specific periodical title or issue. Other measures of value, such as citation ana- lyses or faculty ratings, are also title- specific. Thus, such measures may be used fairly directly in a journal place- mentor deselection decision. As noted in the introduction, however, reshelving counts, user responses, or other non- direct measures may not reflect several important forms of periodical use. By contrast, an investigator may de- fine direct-observation rules to include browsing or other forms of casual use, November 1992 but direct-observation counts are re- corded only for total use within a given half-range in a given time period. Obser- vations recorded for a finer level of aggre- gation, such as for a quarter-range or for individual titles, do not appear to be feasible. Since each half-range includes sixty-five to eighty titles with consecutive Library of Congress call numbers, a single direct-observation count generally meas- ures aggregate use within one or more re- lated subdisciplines, but gives no specific information about the use of specific journals. Thus, direct-observation use counts are most likely to contribute to journal acquisition, placement, and deselection decisions by indicating the relative jour- nal use intensities in different subdiscip- lines and by permitting librarians to assess the differential undercount issue raised in the introduction. A detailed dis- cussion of these two is beyond the scope of the present work. Another paper will present some specific data analyses for these issues. DISCUSSION Training and Attentiveness of Observers The results of the auxiliary use study indicate that student observations of pe- riodical use contain a substantial com- ponent of observational error. The question· arises as to how such errors could have occurred. The first and most obvious hypothesis is that the training was not adequate. As noted, the training session was brief, about one-half hour at most. During the training sessions, most observers appeared to have understood the rules of the project. The majority of the student observers were quite famil- iar with library practices and pro- cedures. Many were shelvers in the CPD. Nonetheless, the short training period and the training method may not have been sufficient for the absorption of a complex set of rules. Second, boredom may have contributed to measurement error problems. As indi- cated in table 1, the direct-observation method required an observer to spend substantial amounts of time recording zero use. Naturally an observers' atten- tion wandered during such periods. For the short periods in which periodical use did occur, the observer needed to record use counts carefully, according to some fairly complex rules. This combination of inactivity and need for careful attention to detail may have contributed substan- tially to observational error problems. The observers attempted to be unobtrusive in as many instances as possible while still holding some confidence that the patron could be observed accurately. Intervention Effects When the study was designed, there was some concern that the direct-ob- servation method would have some deleterious effect on the patrons using the CPD. If so, such an effect would not only bias the results of the study but, more importantly, would destroy the ability of the CPD to deliver its service effectively. If the intervention made pa- trons self-conscious to the point that they chose not to search the CPD shelves at all, then direct observation would have created a negative effect that would not have been present otherwise. The observers attempted to be unobtrusive Observation of Periodical Usage 549 in as many instances as possible while still holding some confidence that the patron could be observed accurately. While this was not always possible, the investigators did not note any evidence to suggest a negative intervention effect on patron use of periodicals. For ex- ample, the investigators did not note any cases of patrons staring at the direct-ob- servation staff or moving out of an aisle when a patron saw an observer. CONCLUSION The direct-observation method earned mixed reviews. Direct observation of peri- odical use is attractive because obser- vational rules may be tailored to satisfy specific administrative definitions of use. However, observational errors, aggrega- tion issues, and costs may limit the admin- istrative utility of direct-observation use counts. Librarians may reduce the obser- vational errors problem through addi- tional training of observers. Aggregation and cost issues, however, are fundamen- tal constraints on the value of the direct- observation method. For the Evans Library use study, the authors now plan to use direct-observa- tion counts primarily to assess differential undercount rates for other use measures, such as reshelving counts. Another article will present details of this assessment. REFERENCES AND NOTES 1. Studies addressing concerns of use studies include Robert N. Broadus, "The Measure- ment of Periodicals Use," Serials Review 11:57-61 (Summer 1985); Maurice B. Line and Alexander Sandison, "Practical Interpretation of Citation and Library Use Studies," College & Research Libraries 36:393-96 (Sept. 1975); Allen Kent et al. Use of Library Materials: The University of Pittsburgh Study (New York: Dekker, 1979); Katherine Konopasek and Nancy Patricia O'Brien, "Undergraduate Periodicals Usage: A Model of Measurement," Serials Librarian 9:65-74 (Winter 1984); and Marifran Bustion and Jane Treadwell, "Reported Relative Value of Journals versus Use: A Comparison," College & Research Libraries 51:142-51 (Mar. 1990). 2. Studies addressing shelf space and use density questions are Dianne C. Langlois and Jeanne V. Von Schulz, "Journal Usage Survey: Method and Application," Special Librar- ies 64:239-44 (May/June 1973); Charles B. Wenger and Judith Childress, "Journal Evaluation in a Large Research Library," Journal of the American Society for Information Science 28:293-99 (Sept. 1977); and Li~e and Sandison, "Practical Interpretation," p.394. 3. Colin R. Taylor, "A Practical Solution to Weeding University Library Periodicals Collections," Collection Management 1:27-45 (Fall/Winter 1976-77). 4. Ibid, p.34-41. 5. Langlois and Von Schulz, "Journal Usage," p.240; Taylor, "A Practical Solution," p.31-32. 550 College & Research Libraries November1992 6. Johanna Ross, "Research Notes: Observations of Browsing Behavior in an Academic Library," College & Research Libraries 44:269-76 (July 1983); Wenger and Childress, "Journal Evaluation," p.294-95. 7. For a general discussion of controlled selection designs, see W. G. Cochran, Sampling Techniques (New York: Wiley, 1977), p.124-27; and J. J. Waterton, "An Exercise in Controlled Selection," Applied Statistics 32:150-64 (1983). 8. For a discussion of errors in variable models, see W. A. Fuller, Measurement Error Models (New York: Wiley, 1987). 9. R. J. Bartlett and J. L. Eltinge, "Variance Component Approaches to Measurement Errors in Count Data: An Illustrative Example," unpublished manuscript, Department of Statistics, Texas A&M University. 10. For a more detailed discussion of the reliability ratio and related issues, see Fuller, Measurement Error Models, Section 1.1.2. 11. For example see table 1.1.1 in Fuller, Measurement Error Models, p.8. 12. N. R. Draper and H. Smith, Applied Regression Analysis, (New York: Wiley, 1981), Chapter 1. 13. Generalized least squares fit discussed in Draper and Smith, Applied Regression Analy- sis, Section 2.11.