525

The Scholarly Impact of Books Acquired via 
Approval Plan Selection, Librarian Orders, and 
Patron-Driven Acquisitions as Measured by 
Citation Counts

David C. Tyler, Brianna D. Hitt, Francis A. Nterful, and McKenna R. 
Mettling*

Patron-driven acquisition has been an important, if contentious, topic for decades, with 
numerous programs having been piloted, adopted, and reported on, largely favorably, in 
the library literature. Still, questions and doubts persist for academic libraries, especially 
where the composition of vendor plans and packages and the judgment of patrons are 
concerned. Past literature has approached the assessment of patron-driven acquisition 
by analyzing circulation/usage, comparing peer-library holdings, seeking patrons’ or 
librarians’ judgments of utility and suitability, looking for evidence of collection imbal-
ances, and testing for overlap in patrons’ and librarians’ purchases. To contribute to this 
literature, this study addresses scholarly impact and examines whose selections—ap-
proval plans’, librarians’, or patrons’—have been most heavily cited. For the social sci-
ences, the sciences, and the humanities, the authors gathered topic-matched random 
samples of books acquired via approval plans and librarian orders during the first five 
years of operation of their institutions’ interlibrary loan purchase-on-demand patron-
driven acquisition program and compared their citation counts to the counts of books 
acquired via the program. Google Scholar was employed to tally citations.

Introduction
Academic librarians with collection development responsibilities and library administrators who 
have followed the library literature on acquisitions and collection development since the 1999 
publication of Perdue and Van Fleet’s study of the successful program at Bucknell University1 
could hardly be unaware of the explosion of interest in patron-driven acquisition (henceforth, 
PDA)—or, alternately, demand-driven (DDA), user-driven, and/or evidence-based acquisition. 
In 2010 and in 2012, respectively, the ACRL Research Planning and Review Committee identi-
fied collection growth driven by patron demand and patron-driven e-book acquisition as top 
ten trends in academic libraries.2 As well, several authors contemporaneously named PDA 

* David C. Tyler is Collections Analyst & Strategist in the University Libraries at the University of Nebraska-
Lincoln; email: dtyler2@unl.edu. Brianna D. Hitt is a PhD student, and Francis A. Nterful and McKenna R. Met-
tling are MS graduates, all in the Department of Statistics at the University of Nebraska-Lincoln. ©2019 David 
C. Tyler, Brianna D. Hitt, Francis A. Nterful, and McKenna R. Mettling, Attribution-NonCommercial (http://
creativecommons.org/licenses/by-nc/4.0/) CC BY-NC.

mailto:dtyler2@unl.edu
http://creativecommons.org/licenses/by-nc/4.0/
http://creativecommons.org/licenses/by-nc/4.0/


526  College & Research Libraries May 2019

one of the most widely discussed acquisition models in the library literature.3 In fact, Walker 
declared US research libraries to be on a PDA “tipping point” in 2011.4

The PDA approach would certainly seem to have been quickly and widely adopted. In 
a national survey reported on in 2010, Lenares and Delquie found just 32 respondents with 
a current program, 42 with plans to implement a program within a year, and 90 institutions 
indicating that they would like to implement a program within the next three years.5 The 
following year, Osorio presented a survey of librarians in Illinois that found that a mere 12.6 
percent and 23.1 percent of respondents had a PDA or DDA program for print books or e-
books, respectively.6 In a 2013 article adapted from a 2012 report to The Andrew W. Mellon 
Foundation, Esposito, Walker, and Ehling estimated that there were just 400 to 600 programs 
in existence.7 More recently published survey results, however, have shown that between 85 
and 95 percent of respondents report that their libraries have PDA/DDA programs in place.8 

The field’s high level of interest and the increasing number of programs piloted and ad-
opted by academic libraries have produced a veritable avalanche of literature. Caminita, in 
a chapter published in 2014, reported that searches of Google Scholar and of EBSCO’s Library, 
Information Science and Technology Abstracts (LISTA) employing a selection of terms associated 
with college libraries and PDA found more than 2,000 and more than 400 items, respectively, 
had been indexed since 2009.9 A search of Google Scholar employing Caminita’s terms performed 
by the lead author of this study during the fall of 2017 for items indexed since the beginning 
of 2013 found nearly 12,000 items.10 Thus, it would seem that the field’s interest in PDA has 
been and continues to be quite keen, and adoption of the model has been widespread. In 
fact, several in the field have argued that this “new” patron-driven approach has or will soon 
become standard library practice,11 although perhaps it should be noted that several authors, 
both PDA critics and supporters, have taken pains to declare that patron-driven collecting is 
actually nothing new and is part of what libraries have always done.12 One can only speculate, 
but perhaps this reflects a desire on the part of the model’s critics to call into question the 
boosterish enthusiasm of some of PDA’s promoters and a desire on the part of the model’s 
promoters to reassure that a seemingly new and potentially risky acquisition practice is not 
actually new and is, therefore, not risky.

Regardless of whether one enthusiastically embraces or unenthusiastically laments the 
development and adoption of PDA, the authors of this study would caution that it is early 
yet to draw firm conclusions concerning the model and its effects. Thus far, the published 
literature has cast PDA as a new and exciting disruptive technology/practice that moves aca-
demic libraries away from a wasteful and moribund “just-in-case” model and toward a lean 
and responsive “just-in-time” model;13 as a subversion of library hegemonies, hierarchies, 
and authority/power relations;14 as a potential revolution in library work;15 as a major stra-
tegic change;16 as a “powerful shift in how libraries acquire materials”;17 and as a continua-
tion of or a return to past practice in a new guise …something both new and not.18 From the 
midst of this promotional whirl, several authors have thoughtfully called into question the 
PDA model and argued that it has several failings that will produce numerous unfavorable 
outcomes. Early studies, though, regardless of the metrics employed or the issues addressed 
and assessed, have largely produced findings and conclusions favorable to the PDA model.19 
Many of these findings have not, though, been entirely generalizable or particularly robust. As 
Walker has noted, the published research on, as well as the criticism of, PDA “has consisted, 
in the main, of opinion pieces debating the merits of PDA and case studies of libraries’ expe-



The Scholarly Impact of Books Acquired   527

riences implementing PDA.”20 Likewise, Dewland and See have noted that a shortcoming of 
the model and its literature has been the absence of consistent, objective, and standardized 
metrics for success.21 

Remedying this entirely would be well beyond the current authors’ abilities or the scope 
of a single article, yet the authors do hope that they may here introduce an approach that will 
address one of the more frequently raised criticisms of PDA: its potential for acquiring books 
of low scholarly merit. Unfortunately, “scholarly merit” is something of an amorphous and 
multifaceted measure with a very high potential for subjectivity. Unsurprisingly, the library 
literature thus far has tended to approach the idea obliquely and fractionally rather than di-
rectly and comprehensively, most commonly through assessments of PDA-purchased books’ 
appropriateness for academic library collections. One facet of scholarly merit that has yet to 
be addressed is scholarly impact. So far as the authors have been able to ascertain, no one 
has yet examined whether the books supplied/ordered via the various selection/acquisition 
modes—approval plan selection, librarian firm order, or PDA—tend to be more heavily cited. 
To begin to remedy this, the authors propose to examine the citation counts of the books of 
the three disciplines—the social sciences, the sciences, and the humanities—acquired via the 
three main acquisition methods (henceforth, “Order Type[s]”) employed by the University of 
Nebraska-Lincoln Libraries (UNL Libraries). The books in question were added to the circulat-
ing collection during the first five years (2003/04 to 2007/08) of the UNL Libraries’ interlibrary 
loan-driven purchase-on-demand (ILL-PoD) program for print books. For those unfamiliar 
with the terminology, Carrico, Leonard, and Gallagher have defined ILL-PoD as follows: 
“ILL-PoD is a patron-driven purchase model whereby patrons request books not owned by 
their local library, and based on various criteria…, the book is purchased rather than bor-
rowed from another library.”22 For the criteria employed by the UNL Libraries, excepting the 
list of barred publishers, please refer to appendix A. The citation counts for the books were 
collected via Google Scholar from January to April of 2016, roughly 8 to 13 years after they had 
been added to the collection and 12 years, on average, after they had been published. In this 
manner, the authors hope to discover which Order Type, if any, attracted the most citations, 
which the authors hope will serve as an at least partial measure of the Order Types’ scholarly 
impact and merit.

Review of Literature
As was noted above, a great deal has been written about the PDA model and about PDA/
DDA programs. Much of the published research literature that has assessed and evaluated the 
outcomes of PDA programs has supported the approach,23 but some of the library literature 
has been cautionary, if not outright critical. The criticisms of PDA pertinent to this study can 
be distilled to two issues: first, the difference in the objectives of the traditional, collection-
building, “just-in-case” model and the more patron-oriented, “just-in-time,” PDA model; and 
second, the different motivations and aims of traditionalist academic librarians, of vendors 
and publishers, and of library patrons. When developing collections in accord with the tra-
ditional model, academic librarians act as experts, agents, and gatekeepers, in which roles 
they identify, collect, manage, and preserve the best and most significant scholarship of their 
day and of the past, as identified by themselves and other authorities, such as the teaching 
faculty and reviewers of academic books. The intent behind the model is to build a collection 
to meet institutions’ educational missions, to preserve the scholarly record, and to meet not 



528  College & Research Libraries May 2019

just the needs of the day’s patrons but the needs of future scholars working 10, 20, 50, and 
even 100 years hence.24 

By way of contrast, the PDA model was derived at least in part from an approach to 
manufacturing and warehousing developed by Toyota that was designed to meet customers’ 
needs as they arose, rather than anticipating them.25 As Costello explains in the preface to her 
recent book on the subject, “Demand-driven acquisitions (DDA) describes any acquisitions 
process that is driven by the desires of patrons or their usage of materials rather than predic-
tive processes like package plans and librarian selection.”26 Thus, as one of its most vigorous 
proponents, Rick Anderson of the University of Utah, has suggested, the PDA model is not 
intended for building collections in the traditional sense at all.27 

For PDA’s critics, the fairly recent, widespread, and enthusiastic adoption of PDA/DDA 
approaches by academic libraries marks an abdication of one of academic librarians’ main 
professional responsibilities,28 the “ultimate source of pride of many collection specialists,”29 
through a turning away from collection-building as an end in itself, what Anderson has 
termed the “monument to Western Civilization” collections mindset.30 For PDA’s advocates, 
it instead marks a positive turn toward the adoption of what several authors have termed a 
facilitated-collection or collections-as-service mindset centered on present patrons’ expressed, 
rather than anticipated, needs.31 Not incidentally, it also marks something of a reduction in 
academic librarians’ control over the composition of their libraries’ collections and a corre-
sponding empowerment of patrons.32 As such, it is not difficult to suppose why traditionalist 
academic librarians who understand collection building to be one of their core duties find 
PDA worrisome and upsetting, especially when some of its more radical champions have sug-
gested it ought to become the primary, if not the sole, method for adding books to academic 
libraries’ collections.33

In addition to being concerned with the lessening of collection building as academic 
librarians’ raison d’être, PDA’s critics have also called into question vendors’/publishers’ and 
patrons’ motives, expertise, and knowledge. With respect to vendors/publishers, they have 
warned that book jobbers and publishers are primarily in the business of selling books, so 
academic libraries ought to be wary of their PDA plans and e-book DDA packages lest their 
library catalogs come to resemble online shopping malls crowded with inappropriate and 
low-quality materials.34 Similarly, with respect to PDA patrons, several authors have warned 
against, or have reported hearing colleagues opine against, allowing patrons too much freedom 
to add books to academic libraries’ collections. Critics have cautioned that patrons tend to 
be too narrowly focused on their immediate needs, that patrons request/access books whose 
utility and scholarly worth is unknown to them, that patrons will unbalance collections in 
their pursuit of idiosyncratic or trivial popular topics, and so forth.35 

The criticisms of vendors strike the authors of this study as somewhat problematic. It has 
been a generally accepted professional truism that good collection development is a product 
of good partnerships and the sharing of expertise between librarians on the one hand and 
vendors and publishers on the other, and one can certainly see this attitude expressed in the 
library literature from both sides.36 Yet, in the critics’ support, there has been some recent, 
empirically supported counter-literature that has found that approval plan selections pro-
duce fewer circulations than do librarians and/or patrons’ purchases37 and offer worse value 
in terms of cost-per-circulation,38 although it should also be noted that a number of studies 
have found that approval plan selections slightly outcirculated books ordered by individual 



The Scholarly Impact of Books Acquired   529

selectors.39 Approval plans also have been found to do a poor job of providing libraries with 
books from small presses and from small societies, with highly specialized books, and with 
books whose subject matter or approach fall outside traditional disciplinary boundaries.40 
There also does seem to be some evidence that DDA e-books packages warrant scrutiny. Two 
recent studies have both reported their authors having discovered nonacademic materials 
in their e-book packages despite both pairs of authors having worked closely with vendors 
to exclude such materials.41 An internal study of e-book subscription packages conducted at 
the UNL Libraries similarly found questionable materials in the vendors’ packages, as well 
as books from suspect publishers. Thus, cautions against giving over collection development 
too much to vendors/publishers would certainly seem worth considering.

The library literature’s response to criticisms of patrons as selectors has been varied, and 
the authors would be hard-pressed to do justice here to the voluminous scholarship on this 
subject or to summarize it succinctly. For a lengthy overview of the approaches taken in the 
literature, the authors recommend Costello’s recent Evaluating Demand-Driven Acquisitions.42 
For the authors’ current purposes, it should suffice to note that the attitudes apparent behind 
much of the criticisms of patrons have been chided for exhibiting a snobbish paternalism43 and 
that much of the early research on PDA programs would seem to support the model’s ability to 
deliver collection-appropriate books of acceptable scholarly merit. As Tyler et al. have noted: 

The primary worry about inappropriate books has largely been answered in the 
existing body of literature. Authors have reported small numbers or percentages 
of their programs’ requested or purchased items as being inappropriate to their 
libraries, with subject, Dewey call number or Library of Congress (LC) subclass, 
genre, readership-/content-level, publisher, or material type variously employed 
as criteria. However, librarians who have directly reviewed the requested or pur-
chased items or who have reviewed requests by subject criteria, publisher, material 
type, readership-level, reviewers’ recommendations, or against peer institutions’ 
holdings, have found PDA items to largely have been worthy and appropriate 
purchases for their collections.44 

Still, although critics’ dire predictions against patrons as selectors have yet to find much 
solid support in the research literature, they should not be dismissed entirely. There does 
seem to be some scattered evidence in the critics’ favor. For example, with respect to the prob-
lem of “hot topics,” there is, of course, the infamous “banana books” incident, recounted by 
Polanka at the Charleston Conference, wherein a library’s PDA budget was quickly spent in 
response to a class assignment.45 Additionally, concerning patrons’ much-feared fondness for 
less-scholarly materials, Goedeken and Lawson have discovered that their patrons appear 
to have an affection for books in Wiley’s Dummies series.46 Of interest to critics questioning 
patrons’ motivations for accessing/requesting books would be the finding of Reynolds et al. 
that patrons’ second-most-frequent reason given for using the “Suggest a Purchase” service 
at Texas A&M was for recreational reading.47 

A few critics, rather than call into question all patrons, have limited themselves to sug-
gesting that perhaps only undergraduates ought to be barred from adding books to academic 
libraries collections and that only faculty and, perhaps, graduate students should be allowed. 
With respect to the issue of undergraduates’ potential for being problematic selectors, one 



530  College & Research Libraries May 2019

recent study of a PDA program for undergraduates did find that the books undergraduates 
requested tended not to be widely held by peer institutions,48 whereas studies of programs that 
were less restricted by patrons’ status have found PDA/DDA books to be held widely by peer 
institutions or held widely in general.49 The same study on undergraduates as selectors also 
reported that selections made by undergraduates appeared not to obtain as great a circulation 
advantage as has been reported elsewhere for less restricted programs.50 In yet another study, 
Purdue University’s librarians found that books requested by undergraduates tended to be 
subsequently checked out most frequently by other undergraduates.51 These studies raise the 
strong possibility that undergraduates’ PDA/DDA purchases may be qualitatively different 
from purchases made by faculty, graduate students, and other types of patrons, which has 
understandably made placing the patrons on an equal footing to selectors an issue of concern 
for some PDA critics.52 

In light of the uncertainties recounted above, and owing to the fact that no one has yet 
looked into the scholarly impact of PDA books, it would seem worthwhile to examine whether 
books provided by vendors or publishers via approval plans or e-book packages, purchased 
by librarians via firm ordering, or acquired via PDA/DDA programs produce a greater schol-
arly impact. As well, given the uncertainty surrounding the various types of academic library 
patrons as purchasers of scholarly works, it would also seem worthwhile to examine whether 
PDA books acquired by the various types of patrons are of commensurate interest to scholars. 

Background, Data Collection, and Method of Analysis
Toward the end of the 2002/03 school year, the manager of the UNL Libraries’ ILL department 
brought Perdue and Van Fleet’s article on Bucknell’s successful PDA program to the attention 
the Collection Development Committee and suggested that it might consider a pilot project 
involving ILL PoD. As a result, since the start of the 2003/04 school year, the UNL Libraries 
has operated an ILL PoD PDA program for recently published nonreference academic books. 
After five years’ operation, in response to a call for papers for a special issue of the journal 
Collection Management, data were drawn for the PDA-acquired books and for all print books 
added to the circulating collection during the interval via traditional acquisition modes. The 
resulting data set produced several articles concerning the circulation performance of this 
set of books, the cost-per-circulation, whether the books were being monopolized, and the 
selection patterns of librarians and patrons at the level of Library of Congress (LC) subclass. 

For the current study, the authors gathered records for PDA-acquired books in the data 
set with LC subclasses associated with the social sciences, the sciences, and the humanities. 
From the contemporaneous approval plans’ selections and librarian firm-ordered titles, the 
authors then drew equal numbers of books for each LC subclass via simple random selection 
without replacement. Subclasses for which not all three Order Types had purchases were not 
included in the draw. Thus, the authors produced three “matched” samples for each discipline’s 
purchases (for example: if PDA patrons had purchased 10 books in subclass QA—Mathematics, 
then 10 books for that subclass would have been randomly drawn from the approval plans’ 
and from the librarians’ books, resulting in a pool of 30 books for purposes of analysis and 
comparison). Very small numbers of books were purchased by the PDA patrons in some LC 
subclasses, so the authors, to facilitate analysis, found it necessary to merge several of the 
LC subclasses into topical groups (in other words, “Topic”). For the composition of the Topic 
groups of each discipline, please see table 1.



The Scholarly Impact of Books Acquired   531

TABLE 1
Composition of the Samples by Discipline and Topic  

(Grouped Library of Congress Subclasses)
Topics with ILL PDA Purchases Books per Order Type
Social Sciences:
Anthropology & Recreation (GF–GN, GT–GV) 29
Business & Economics (HB–HG) 106
Education (LB, LC, LD, LJ) 47
Geography & Environmental Sciences (G–GE) 9
Law (K, KB–KBU, KF–KFZ, KJ–KKZ, KL–KWX, KZ) 43
Military & Naval Sciences (U, UG) 5
Political Science (JA–JF, JK, JN–JQ, JV, JZ) 45
Psychology (BF) 46
Social Sciences (General) & Statistics (H–HA) 9
Sociology & Related Fields (HM–HX) 152

Books per Order Type 491
Total Social Sciences Books 1,473

Sciences:
Agriculture (SB, SF) 10
Arts & Crafts (TT) 1
Building Construction (TH) 1
Chemical Technology & Manufacturing (TP, TS) 18
Dentistry (RK) Life Sciences (QH–QR) 55 1
Engineering & Technology (General) (T–TD, TN) 23
Engineering (Mechanical, Electrical, & Automotive [TJ–TL]) 17
Life Sciences (QH–QR) 55
Medicine (Clinical & Internal) (RC–RD, RF–RJ) 81
Medicine (General, Public Health, & Pathology) (R–RB) 40
Physical Sciences (QB–QE) 34
Science (General) & Mathematics (Q–QA) 59
Therapeutics & Pharmacy (RM–RS) 3

Books per Order Type 343
Total Sciences Books 1,029

Humanities:
Architecture (NA) 37
English Language & Literature (PE, PR, PS) 82
Fine Arts (N, NB–ND, NK–NX) 70
General Works, Biography, Library & Information Science (AZ, CT, Z–ZA) 21
History (CB–CC, CN, D–DA, DC–DG, DJK–DK, DR–DU, E, F) 166
Music (ML–MT) 22
Non-English Languages & Literatures (PA, PJ, PL, PQ, PT) 18
Philology, Linguistics, & Literature (General) (P, PN) 65



532  College & Research Libraries May 2019

In addition to being curious in general about the performance of the selection modes that 
comprise the three Order Types, the authors’ attention, as was noted above, was also drawn 
to the performance of the library’s patrons by the literature critical of PDA/DDA programs for 
failing to exclude certain sorts of patrons. Therefore, the authors further grouped the books in 
the UNL Libraries’ PDA data sets into books ordered by one of five “Patron Types” by status. 
As can be seen from table 2 below, PDA at UNL has mostly been a means for graduate stu-
dents and for members of the faculty to make additions to the circulating collection, which is 
consistent with results reported elsewhere in the library literature for PDA programs for print 
materials,53 although not necessarily for e-book DDA programs.54 Unfortunately, the UNL Li-
braries’ ILL department’s data collection form allowed patrons to self-report their status, which 
reporting was not mandatory, so it would appear from the table that roughly 11.4 percent of 
PDA patrons did not sufficiently identify themselves and had to be classed as “Unknown.”

To gather data on the apparent scholarly interest in/impact of these books, as indicated 
by cumulative citation over the study’s interval, the lead author searched Google Scholar from 
January through April of 2016 for the books’ index entries and citation counts. In a number 
of instances, there were multiple index entries for books due to variations in how they had 
been cited, so each book was searched for with three different combinations of search terms 
drawn from the author, title, and publisher fields of their catalog records. Upon discovering 
the books’ index entries, the books’ citations were recorded and, if necessary, summed. In a 
few instances, books in the data set proved to be earlier or later editions or to be books from 
numbered series. In such instances, only citation counts for the appropriate editions or series 

TABLE 1
Composition of the Samples by Discipline and Topic  

(Grouped Library of Congress Subclasses)
Topics with ILL PDA Purchases Books per Order Type
Philosophy (B–BD, BH–BJ) 44
Photography (TR) 19
Religions (BL–BP, BR–BX) 71

Books per Order Type 615
Total Humanities Books 1,845

Note: No PDA books were purchased in LC subclasses not listed, such as GR—Folklore, so they were not 
included in the study.

TABLE 2
Percentage of PDA Books Purchased by Patron Type

Patron Types
Disciplines Faculty Graduates Undergraduates Staff Unknown
Social Sciences 21.0% 53.4% 9.4% 6.1% 10.2%
Sciences 19.8% 50.4% 7.3% 8.7% 13.7%
Humanities 31.5% 44.2% 8.8% 4.4% 11.1%

Totals 25.2% 48.8% 8.6% 6.0% 11.4%
Note: Percentages rounded.



The Scholarly Impact of Books Acquired   533

numbers were included in the tallies. In a very few instances, index entries for the books were 
not discoverable. In those instances, the lead author conducted a fourth search that employed 
the books’ first authors’ last names and a few words from the books’ titles as a phrase. The lead 
author then manually tallied the number of times the books appeared in the first 10 pages of 
search results. The author found, when forced to employ this approach, that none of these 
books appeared beyond the fifth or sixth page of search results.

The collected data produced highly right-skewed distributions of citation counts (see 
appendix B). As an initial, informal testing of the waters, the authors attempted within each 
discipline to test for differences in performance by Order Type, by Order Type within Topic 
groups, by Patron Type within the PDA subset, and by Patron Type within Topic groups using 
Poisson regression, but this initial test run produced results that were unlikely, with nearly 
every comparison proving to be significant, and the Poisson models were poor fits. Subsequent 
model fitting suggested that the negative binomial distribution was a much better fit for the 
data, so the authors reanalyzed the performance of the Order Types and Patron Types using 
negative binomial regression analysis with p value adjustments for multiple comparisons.

Research Questions
As should be evident from the foregoing, the foci of this study are the citation count perfor-
mances of the three Order Types and, within the PDA data sets, the performances of the five 
Patron Types. For each of the three disciplines, the authors first asked a pair of questions: 
If the three Order Types had been given an opportunity to select/purchase equal numbers 
of books on the same Topics for the UNL Libraries, would there be differences in the books’ 
performance as generators of citation activity, and, if so, which Order Type’s books would 
have been cited more? As a follow-up research question, the authors asked: Would there be 
differences in Order Types’ citation count performances within Topic groups? If there were no 
significant differences between Order Types’ performances in general, the authors reasoned, 
perhaps there would be performance differences concealed within particular subject areas. 
Alternately, if there were differences between the Order Types’ performances in general, these 
differences might be present across all Topics or might be limited to outsized differences within 
just a few. For the study’s final research question, the authors repeated Research Question #1 
but turned their attention to the PDA data sets and to the five Patron Types: Within the PDA 
data sets of each discipline, did any of the Patron Types outperform any of the others?

Results
Research Question #1: For each discipline, were there significant differences in 
performance by Order Type? If so, which Order Types were cited more?
For this first question, the authors tested within disciplines the three Order Types’ citation 
counts for statistically significant differences and then performed post hoc tests with adjust-
ments for multiple comparisons to determine which Order Types, if any, outperformed one 
or more of the others. As the F values for the three tests of fixed effects in table 3 show, there 
would fairly definitely seem to be at least one significant difference in performance present 
in each discipline. To determine the nature and direction of the differences between Order 
Types, the authors performed the aforementioned post hoc comparisons, whose results are 
to be found in the “Differences of Order Type Least Squares Means” portions of the table.



534  College & Research Libraries May 2019

TABLE 3
Citation Count Performance within Disciplines by Order Type:  

Approval Plans, Librarian Orders, and PDA Patrons
Social Sciences: Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F
Order Type 2 1,470 33.26 < .0001

Order Type Least Squares Means
Order Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean
Appr. 5.0816 0.07150 1,470 71.07 < .0001 161.03 11.5145
Libs. 5.8916 0.07145 1,470 82.45 < .0001 362.00 25.8663
PDA 5.6199 0.07147 1,470 78.64 < .0001 275.86 19.7149

Differences of Order Type Least Squares Means 
with Adjustment for Multiple Comparisons (Tukey-Kramer)

Order Type Order Type Estimate S.E. DF t Value Pr > |t| Adj. p
Appr. Libs. –0.8100 0.1011 1,470 –8.01 < .0001 < .0001
Appr. PDA –0.5383 0.1011 1,470 –5.32 < .0001 < .0001
Libs. PDA 0.2717 0.1011 1,470 2.69 0.0073 0.0198

Sciences: Type III Tests of Fixed Effects
Effect Num DF Den DF F Value Pr > F

Order Type 2 1,026 68.61 < .0001
Order Type Least Squares Means

Order Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean
Appr. 4.6697 0.08742 1,026 53.42 < .0001 106.67 9.3254
Libs. 6.0116 0.08731 1,026 68.86 < .0001 408.13 35.6321
PDA 5.8110 0.08732 1,026 66.55 < .0001 333.95 29.1529

Differences of Order Type Least Squares Means 
with Adjustment for Multiple Comparisons (Tukey-Kramer)

Order Type Order Type Estimate S.E. DF t Value Pr > |t| Adj. p
Appr. Libs. –1.3418 0.1236 1,026 –10.86 < .0001 < .0001
Appr. PDA –1.1412 0.1236 1,026 –9.24 < .0001 < .0001
Libs. PDA 0.2006 0.1235 1,026 1.62 0.1046 0.2357

Humanities: Type III Tests of Fixed Effects
Effect Num DF Den DF F Value Pr > F

Order Type 2 1,842 31.27 < .0001
Order Type Least Squares Means

Order Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean
Appr. 3.8868 0.06229 1,842 62.40 < .0001 48.7561 3.0371
Libs. 4.1691 0.06223 1,842 67.00 < .0001 64.6585 4.0234
PDA 4.5788 0.06216 1,842 73.66 < .0001 97.3967 6.0539



The Scholarly Impact of Books Acquired   535

For the social sciences, as one can see from the reported t values and adjusted p values 
presented in the top third of the table, the librarians significantly outperformed both the ap-
proval plans and the patrons, and the patrons significantly outperformed the approval plans. 
(Note: In reading this and the tables to follow, positive t values indicate that the Order Type 
or Patron Type in the left-hand column outperformed the one in the right-hand column; a 
negative value indicates the reverse.) How great were the differences in citation-count perfor-
mance? To answer this question, one will need to subtract the mean values found in the table. 
For example, the difference in citation counts between the approval plans and the librarians 
in the social sciences was 161.03–362.00 = –200.97. That is to say that the book jobbers’ selec-
tors underperformed the UNL social sciences librarians by roughly 201 citations per book. 
Similarly, the librarians outperformed the PDA patrons by roughly 86 citations per books, 
and the patrons outperformed the approval plan selectors by roughly 115 citations per book.

For the sciences, the post hoc comparisons show that there were two statistically signifi-
cant performance differences to be found among the Order Types. Both the librarians and the 
patrons handily outperformed the approval plans. The librarians outperformed the approval 
plans by approximately 301.46 citations per book, and the patrons outperformed the approval 
plans by roughly 227.28 citations per book. The librarians did outperform the patrons by 
approximately 74 citations per book. Were one to multiply this difference by the number of 
books in question, one would find that the librarians’ books attracted more than 25,000 more 
citations than did the patrons’ books; that certainly seems like a large amount, but, when one 
consults the table, one can see that this difference was not statistically significant. In fact, it 
was not significant even prior to the adjustment for multiple comparisons (p = 0.1046). Still, 
it is difficult not to see it as substantial.

Finally, for the humanities, as was the case with the social sciences, all three post hoc 
comparisons reveal statistically significant differences. The librarians again outperformed the 
approval plans’ selectors. Unlike with the social sciences, the PDA patrons outperformed both 
of the other Order Types. If one consults the mean values displayed and/or consults appendix 
B, one will notice that the parameters and counts for the humanities tended to be quite a bit 
smaller than those of the social sciences and of the sciences, although the shape of the cita-
tion counts’ distributions tended to be quite similar. Despite these differences between the 
disciplines, statistically significant differences were still very much present in the humanities. 
The humanities librarians attracted almost 10,000 more citations than did the approval plans, 
and the patrons garnered roughly 20,000 more than the former and almost 30,000 more than 
the latter.

TABLE 3
Citation Count Performance within Disciplines by Order Type:  

Approval Plans, Librarian Orders, and PDA Patrons
Differences of Order Type Least Squares Means 

with Adjustment for Multiple Comparisons (Tukey-Kramer)
Order Type Order Type Estimate S.E. DF t Value Pr > |t| Adj. p
Appr. Libs. –0.2823 0.08805 1,842 –3.21 0.0014 0.0039
Appr. PDA –0.6920 0.08800 1,842 –7.86 <.0001 <.0001
Libs. PDA –0.4097 0.08795 1,842 –4.66 <.0001 <.0001
Note: Statistically significant effects/differences appear in bold.



536  College & Research Libraries May 2019

Research Question #2: For each discipline, were there significant differences 
in performance between Order Types within Topics? If so, which Order Types 
were cited more? 
As a follow-up to the first research question, the authors parsed the data sets a bit more finely 
and again tested the performance of the Order Types, this time within the Topic categories 
presented in table 1. The authors tested the Order Types’ performances within the 10 Topic 
categories of the social sciences; the 13 categories of the sciences, although only the results of 
9 will be presented; and the 11 Topic categories of the humanities. The four missing sciences 
categories—Arts & Crafts, Building Construction, Dentistry, and Therapeutics & Pharmacy—were 
included in the modeling and testing of the data, but they comprised so very few books that 
the authors felt that it was not worthwhile—and would perhaps be misleading—to report 
their results.

As was the case above, the authors looked for the presence of statistically significant differ-
ences in the three disciplines’ data sets and then followed with a battery of post hoc comparisons 
to determine where and in what direction the significant differences in performance within 
Topics, if any, occurred. Because of the sizable number of comparisons being made, the authors 
unfortunately found it necessary to split the table employed above into three parts, one for each 
discipline. In the upper portion of each discipline’s subtable, the authors once again will report 
the results of the tests of fixed effects, this time looking for Order Type and Topic effects and, 
most important, for Order Type*Topic interaction effects. In the remainder of each subtable, the 
authors will again report on the post hoc comparisons performed, as in table 3 above.

As one can see from a quick review of tables 4a-c below, there were statistically significant 
Order Type*Topic interaction effects for all three disciplines, with the social sciences and the 
humanities producing roughly equal F values; but, with the sciences, despite its enormous F 
value displayed in table 3 above, producing a value slightly less than one half of either. With 
Topic taken into account, perhaps the sciences exhibited fewer or less extreme differences 
than the other disciplines. To obtain a clearer picture of what occurred in the data sets, one 
will need to turn to the post hoc comparisons.

First, for the social sciences, the post hoc testing determined that performance differences 
between Order Types were not present across all Topic groups. As the t values and adjusted p 
values in table 4a show, there were statistically significant differences in performance in just 
8 of the 30 comparisons made, after the adjustment for multiple comparisons. For example, 
in the Anthropology & Recreation Topic group, there was no statistically significant difference 
in performance between the approval plans and the librarians, despite the latter having at-
tracted roughly 48 more citations per book. The PDA patrons, however, outperformed both 
in a statistically significant manner, garnering roughly 219 more citations per book than did 
the approval plans and roughly 171 more per book than did the librarians. The close reader 
of the tables may notice that some of the differences in means appear to be quite large, yet the 
adjusted p values show the differences not to be significant at all, such as was the case with 
the Topic Social Sciences (General) & Statistics. In such cases, an explanation will likely be found 
by referring back to the composition of the samples presented in table 1 and referring to the 
standard error of the means column (“S.E. Mean”) in the table under review. If one were to 
do so with this example case, one would see that the number of books being tested was small 
and that the standard errors, partially as a result, were quite large. Thus, one could speculate 
that had more books been purchased in the LC subclasses H and HA, then this Topic would 



The Scholarly Impact of Books Acquired   537

TABLE 4A
Social Sciences: Comparison of Citation Counts by Order Type and Topic:  

Approval Plans, Librarian Orders, and PDA Patrons
Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F
Order Type 2 1,443 4.04 0.0178
Topic 9 1,443 7.95 < .0001
Order Type*Topic 18 1,443 3.85 < .0001

Differences of Order Type Least Squares Means with
Least Squares Means Adjustment for Multiple Comparisons (Tukey-Kramer)

Topic Order Type Mean S.E. Mean Order Type Order Type t Value Pr > |t| Adj. p.
Anthropology 
& Recreation

Appr. 59.931 17.0728 Appr. Libs. –1.47 0.1422 0.3064
Libs. 108.24 30.7864 Appr. PDA –3.82 0.0001 0.0004
PDA 279 79.2585 Libs. PDA –2.36 0.0186 0.0488

Business &
Economics

Appr. 153.86 22.876 Appr. Libs. –0.97 0.3337 0.5979
Libs. 188.55 28.0264 Appr. PDA –2.99 0.0028 0.0079
PDA 288.71 42.8978 Libs. PDA –2.03 0.0428 0.1061

Education Appr. 100.74 22.5114 Appr. Libs. –4.99 < .0001 < .0001
Libs. 487.66 108.78 Appr. PDA –4.63 < .0001 < .0001
PDA 434.34 96.8954 Libs. PDA 0.37 0.7137 0.9284

Geography &
Enviro. Science

Appr. 157.89 80.5612 Appr. Libs. 0.22 0.8229 0.9727

Libs. 134.33 68.5585 Appr. PDA –0.73 0.4666 0.7468
PDA 267 136.16 Libs. PDA –0.95 0.3412 0.6073

Law Appr. 121.3 28.3275 Appr. Libs. –1.64 0.1010 0.2287
Libs. 208.53 48.6629 Appr. PDA 0.30 0.7633 0.9512
PDA 109.81 25.6493 Libs. PDA 1.94 0.0523 0.1273

Military &
Naval Science

Appr. 190.6 130.45 Appr. Libs. 0.84 0.4032 0.6806
Libs. 84.8 58.1182 Appr. PDA 1.02 0.3096 0.5666
PDA 71.2 48.8207 Libs. PDA 0.18 0.8569 0.9822

Political 
Science

Appr. 214.44 48.9161 Appr. Libs. –1.30 0.1923 0.3928
Libs. 326.62 74.4786 Appr. PDA –0.21 0.8357 0.9765
PDA 229.29 52.2985 Libs. PDA 1.10 0.2728 0.5160

Psychology Appr. 193.22 43.5969 Appr. Libs. –0.53 0.5928 0.8542
Libs. 229.17 51.701 Appr. PDA 0.75 0.4512 0.7315
PDA 151.91 34.2874 Libs. PDA 1.29 0.1978 0.4019

Social Sciences
& Statistics

Appr. 800.56 408.03 Appr. Libs. 0.04 0.9649 0.9989
Libs. 775.56 395.29 Appr. PDA –1.07 0.2858 0.5342
PDA 1728.44 880.84 Libs. PDA –1.11 0.2664 0.5069

Sociology Appr. 151.01 18.7498 Appr. Libs. –7.71 < .0001 < .0001
Libs. 584.71 75.5247 Appr. PDA –2.56 0.0105 0.0282
PDA 236.82 29.3901 Libs. PDA 5.15 < .0001 < .0001

Note: Statistically significant effects/differences appear in bold. In order to fit the table to the page, some 
statistics could not be displayed. Full tables are available upon request.



538  College & Research Libraries May 2019

TABLE 4B
Sciences: Comparison of Citation Counts by Order Type and Topic:  

Approval Plans, Librarian Orders, and PDA Patrons
Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F

Order Type 2 990 7.39 0.0007
Topic 12 990 12.56 <.0001
Order Type*Topic 24 990 1.61 0.0322

Differences of Order Type Least Squares Means with
Least Squares Means Adjustment for Multiple Comparisons (Tukey-Kramer)

Topic Order Type Mean S.E. Mean Order Type Order Type t Value Pr > |t| Adj. p.

Agriculture Appr. 50.5000 24.4422 Appr. Libs. 0.26 0.7985 0.9647

Libs. 42.4000 20.5383 Appr. PDA –1.21 0.2261 0.4467

PDA 115.60 55.8174 Libs. PDA –1.47 0.1428 0.3076

Chemical Tech.
& Manufacturing

Appr. 82.3333 29.6536 Appr. Libs. –1.25 0.2101 0.4217

Libs. 155.89 56.0767 Appr. PDA –2.31 0.0209 0.0544

PDA 267.22 96.0706 Libs. PDA –1.06 0.2895 0.5394

Engineering &
Tech. (Gen.)

Appr. 191.78 61.0151 Appr. Libs. –0.79 0.4271 0.7065

Libs. 274.17 87.1982 Appr. PDA –2.11 0.0347 0.0874

PDA 496.48 157.84 Libs. PDA –1.32 0.1870 0.3841

Engineering
(Mech., Elect.,
& Auto.) 

Appr. 292.47 108.19 Appr. Libs. –1.23 0.2206 0.4382

Libs. 555.29 205.34 Appr. PDA –0.95 0.3444 0.6113

PDA 479.71 177.40 Libs. PDA 0.28 0.7797 0.9578

Life Sciences Appr. 68.2364 14.0671 Appr. Libs. –4.32 <.0001 <.0001
Libs. 240.33 49.4328 Appr. PDA –3.45 0.0006 0.0017
PDA 186.60 38.3915 Libs. PDA 0.87 0.3846 0.6595

Medicine
(Clinical & Int.)

Appr. 68.1358 11.5746 Appr. Libs. –3.75 0.0002 0.0006
Libs. 167.54 28.4084 Appr. PDA –3.28 0.0011 0.0031
PDA 149.56 25.3623 Libs. PDA 0.47 0.6359 0.8837

Medicine (Gen.) Appr. 109.07 26.3363 Appr. Libs. –2.20 0.0277 0.0709

Libs. 231.47 55.8319 Appr. PDA –2.53 0.0116 0.0311
PDA 258.55 62.3563 Libs. PDA –0.32 0.7458 0.9437

Physical 
Sciences

Appr. 119.15 31.1984 Appr. Libs. –4.52 <.0001 <.0001
Libs. 634.09 165.79 Appr. PDA –2.79 0.0054 0.0149
PDA 334.44 87.4711 Libs. PDA 1.73 0.0840 0.1946

Science (Gen.) 
& Mathematics

Appr. 122.44 24.3370 Appr. Libs. –7.69 <.0001 <.0001
Libs. 1062.08 210.78 Appr. PDA –6.48 <.0001 <.0001
PDA 754.78 149.80 Libs. PDA 1.22 0.2239 0.4434

Note: Statistically significant effects/differences appear in bold. In order to fit the table to the page, 
some statistics could not be displayed. Full tables are available upon request.



The Scholarly Impact of Books Acquired   539

also have produced a batch of statistically significant results. Unfortunately, as things stand, 
this can only be a speculation.

With respect to the sciences, if one were to peruse the mean values reported for each 
Topic, one would see that the librarians’ books attracted more citations in eight of the nine 
Topic categories reported on than did the approval plans’citations, with the sole exception 
being Agriculture. Likewise, the patrons’ mean citations were higher than the approval plans’ 
citations in all nine Topics. Both results would certainly be in line with the more general results 
reported on in table 3. If one were to review the t and adjusted p values for the approval plans’ 
post hoc comparisons, one would see that the approval plans were statistically significantly 
outperformed in 9 of the 18 comparisons involving them, and they were outperformed by 
nearly statistically significant amounts in an additional three comparisons—versus librarians 
in Medicine (General) and versus patrons in Chemical Technology & Manufacturing and Engineer-
ing & Technology (General). In reviewing the Topic means for librarians and for patrons, one 
can see that the performance results were more mixed. Librarians outperformed patrons by 
substantial amounts in four Topics, patrons substantially outperformed librarians in three 
topics, and the performance of both was roughly equal for Medicine (Clinical & Internal) and 
just slightly favorable for patrons in Medicine (General, Public Health, & Pathology). Although 
some of the performance differences appear substantial to the eye, however, none of the post 
hoc comparisons for librarians and patrons produced differences that were statistically signifi-
cant. In fact, just one, the comparison for Physical Sciences, was even close to being significant.

TABLE 4C
Humanities: Comparison of Citation Counts by Order Type and Topic:  

Approval Plans, Librarian Orders, and PDA Patrons
Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F
Order Type 2 1,812 15.52 < .0001

Topic 10 1,812 31.13 < .0001
Order Type*Topic 20 1,812 3.58 < .0001

Differences of Order Type Least Squares Means with
Least Squares Means Adjustment for Multiple Comparisons (Tukey-Kramer)

Topic Order Type Mean S.E. Mean Order Type Order Type t Value Pr > |t| Adj. p.
Architecture Appr. 26.8649 6.3963 Appr. Libs. 1.36 0.1724 0.3596

Libs. 16.9459 4.0556 Appr. PDA –0.06 0.9528 0.9981
PDA 27.4054 6.5239 Libs. PDA –1.42 0.1546 0.3287

English 
Language &
Literature

Appr. 28.7195 4.5906 Appr. Libs. –2.14 0.0322 0.0816
Libs. 46.5854 7.4225 Appr. PDA –4.70 <.0001 <.0001
PDA 82.8293 13.1675 Libs. PDA –2.56 0.0106 0.0287

Fine Arts Appr. 23.5857 4.0877 Appr. Libs. 0.55 0.5792 0.8441
Libs. 20.5857 3.5730 Appr. PDA –1.97 0.0492 0.1205
PDA 38.1714 6.5901 Libs. PDA –2.52 0.0117 0.0315

General Works,
Biography, &
LIS

Appr. 67.5714 21.2405 Appr. Libs. 0.07 0.9461 0.9975
Libs. 65.5714 20.6141 Appr. PDA 0.86 0.3875 0.6629
PDA 46.0 14.4839 Libs. PDA 0.80 0.4257 0.7051



540  College & Research Libraries May 2019

Finally, let us turn to the humanities. A review of the reported means shows the per-
formance of the approval plans to be quite mixed vis-à-vis the librarians, with the approval 
plans having higher means in six Topics and with the librarians having higher means in five. 
The approval plans’ performance against the PDA patrons was not as good, with the latter 
having higher means in 11 of the 12 Topic categories. Against the PDA patrons, the librarians’ 
performance was similarly poor, with the patrons having higher mean values for nine Topics. 
A review of the post hoc comparisons supports this sense of the Order Types’ performances 
within Topics. Of the 33 post hoc comparisons, 15 proved to be statistically significant, with 
the patrons outperforming the approval plans or the librarians in 12 of them. Additionally, 
the patrons outperformed the approval plans by a nearly significant amount in the Topic Fine 
Arts and the librarians in the Topic Non-English Languages & Literatures. Of the three remaining 
comparisons whose results were statistically significant, the approval plans outperformed both 
the librarians and the patrons in Photography, and the librarians outperformed the approval 
plans in Philology, Linguistics, & Literature (General). The librarians also nearly outperformed 
the approval plans by a statistically significant amount in English Language & Literature. Thus, 
the results reported above for the general samples presented in table 3, which showed the 
patrons strongly outperforming both the approval plans and the librarians and which showed 

TABLE 4C
Humanities: Comparison of Citation Counts by Order Type and Topic:  

Approval Plans, Librarian Orders, and PDA Patrons
History Appr. 60.1988 6.7334 Appr. Libs. 0.91 0.3634 0.6346

Libs. 52.1325 5.8348 Appr. PDA –3.00 0.0027 0.0076
PDA 96.7711 10.8078 Libs. PDA –3.91 <.0001 0.0003

Music Appr. 33.5 10.3256 Appr. Libs. –1.56 0.1182 0.2620
Libs. 66.1364 20.3130 Appr. PDA –2.93 0.0034 0.0096
PDA 119.73 36.7128 Libs. PDA –1.37 0.1716 0.3583

Non-English
Languages &
Literatures

Appr. 22.0 7.5246 Appr. Libs. 1.12 0.2633 0.5025
Libs. 12.7778 4.4043 Appr. PDA –0.88 0.3764 0.6501
PDA 33.7222 11.4905 Libs. PDA –2.00 0.0454 0.1119

Philology, 
Linguistics & 
Literature (Gen.)

Appr. 76.8923 13.7325 Appr. Libs. –4.11 <.0001 0.0001
Libs. 216.89 38.6572 Appr. PDA –3.00 0.0027 0.0077
PDA 163.97 29.2351 Libs. PDA 1.11 0.2673 0.5083

Philosophy Appr. 79.75 17.3093 Appr. Libs. –0.77 0.4389 0.7189
Libs. 101.14 21.9370 Appr. PDA –3.85 0.0001 0.0004
PDA 259.45 56.1951 Libs. PDA –3.07 0.0021 0.0061

Photography Appr. 43.0526 14.2566 Appr. Libs. 4.97 <.0001 <.0001
Libs. 3.9474 1.3774 Appr. PDA 2.54 0.0111 0.0300
PDA 13.0 4.3600 Libs. PDA –2.46 0.0139 0.0370

Religions Appr. 43.8732 7.5148 Appr. Libs. –0.53 0.5991 0.8587
Libs. 49.831 8.5297 Appr. PDA –3.25 0.0012 0.0034
PDA 96.1831 16.4256 Libs. PDA –2.72 0.0066 0.0181

Note: Statistically significant effects/differences appear in bold. In order to fit the table to the page, some 
statistics could not be displayed. Full tables are available upon request.



The Scholarly Impact of Books Acquired   541

the performance difference between the approval plans and the librarians to have been slightly 
less pronounced, would seem to have been well supported here.

Research Question #3: For each discipline, were there significant differences in 
performance by Patron Types? If so, which types were cited more? 
The study’s third research question is, essentially, the application of Research Question #1 to 
the citation counts of just the PDA-acquired books, with the five Patron Type categories as-
suming the place of the three Order Types. As such, tables 5a–c below should each be read in 
much the same manner as were the disciplines’ sections of table 3 above. In the upper portion 
of each discipline’s section of the table, one will again find the tests of fixed effects, which 
indicate whether or not at least one statistically significant difference was present in the data 
sets being analyzed. As one can see via a quick review of the three pertinent sections of the 
tables, there was strong evidence of at least one statistically significant difference in the social 
sciences data set, considerably weaker evidence of at least one difference in the sciences data 
set, and strong evidence for at least one difference in the humanities data set.

TABLE 5A
Social Sciences: Comparison of Citation Counts by Patron Type: Faculty, Graduates, Staff, 

Undisclosed-status Patrons (Unknown), and Undergraduates
Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F
Patron Type 4 486 12.62 < .0001

Order Type Least Squares Means
Patron Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean
Faculty 5.0817 0.1386 486 36.67 < .0001 161.05 22.3204
Graduates 5.6126 0.08684 486 64.63 < .0001 273.86 23.7826
Staff 6.5879 0.2565 486 25.68 < .0001 726.23 186.27
Unknown 6.0071 0.1987 486 30.23 < .0001 406.32 80.7487
Undergrads 4.6898 0.2075 486 22.60 < .0001 108.83 22.5863

Differences of Order Type Least Squares Means 
with Adjustment for Multiple Comparisons (Tukey-Kramer)

Patron Type Patron Type Estimate S.E. DF t Value Pr > |t| Adj. p
Faculty Graduates –0.5309 0.1636 486 –3.25 0.0013 0.0109
Faculty Staff –1.5062 0.2915 486 –5.17 < .0001 < .0001
Faculty Unknown –0.9254 0.2423 486 –3.82 0.0002 0.0014
Faculty Undergrads 0.3920 0.2496 486 1.57 0.1169 0.5171
Graduates Staff –0.9753 0.2708 486 –3.60 0.0003 0.0032
Graduates Unknown –0.3945 0.2169 486 –1.82 0.0695 0.3636
Graduates Undergrads 0.9229 0.2250 486 4.10 < .0001 0.0005
Staff Unknown 0.5807 0.3245 486 1.79 0.0741 0.3870
Staff Undergrads 1.8981 0.3299 486 5.75 < .0001 < .0001
Unknown Undergrads 1.3174 0.2873 486 4.58 < .0001 < .0001
Note: Statistically significant effects/differences appear in bold.



542  College & Research Libraries May 2019

For the social sciences, as the t and adjusted p values in the table show, there were several 
statistically significant differences in performance between Patron Types, and every Patron 
Type was involved in at least one such pairing. The surprise in the social sciences PDA data 
set for the authors was the performance of Staff, which outperformed all of the other Patron 
Types and outperformed three of the other four by statistically significant amounts. Its out-
performance of the Unknown patrons was also nearly significant prior to the adjustment of 
p values. The next-best-performing Patron Type among the categories whose members were 
identifiable was Graduates, who significantly outperformed Faculty and Undergraduates. 
Surprisingly, Faculty did not significantly outperform Undergraduates, although it should 
be noted that the Faculty mean was higher.

Before proceeding to the next discipline, the authors thought it would be worthwhile 
to note the magnitude of some of the differences in performance in the social sciences. The 
top-performing Patron Type, Staff, outperformed Faculty by roughly 565 citations per book, 
Graduates by roughly 452 citations per book, and Undergraduates by roughly 617 citations per 
book. Graduates, by way of comparison, slightly outperformed Faculty and Undergraduates by 
roughly 113 and 165 citations per book, respectively, and Faculty outperformed Undergraduates 
by just 52 citations per book. The performance of social sciences’ Staff was quite remarkable.

TABLE 5B
Sciences: Comparison of Citation Counts by Patron Type: Faculty, Graduates, Staff, 

Undisclosed-status Patrons (Unknown), and Undergraduates
Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F
Patron Type 4 338 2.66 0.0327

Order Type Least Squares Means
Patron Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean
Faculty 5.9401 0.1833 338 32.40 <.0001 379.97 69.6523
Graduates 5.9496 0.1149 338 51.77 <.0001 383.61 44.0860
Staff 5.2818 0.2761 338 19.13 <.0001 196.73 54.3238
Unknown 5.3368 0.2206 338 24.19 <.0001 207.85 45.8511
Undergrads 5.5868 0.3024 338 18.48 <.0001 266.88 80.7034

Differences of Order Type Least Squares Means 
with Adjustment for Multiple Comparisons (Tukey-Kramer)

Patron Type Patron Type Estimate S.E. DF t Value Pr > |t| Adj. p
Faculty Graduates –0.00952 0.2164 338 –0.04 0.9649 1.0000
Faculty Staff 0.6582 0.3314 338 1.99 0.0478 0.2750
Faculty Unknown 0.6033 0.2868 338 2.10 0.0362 0.2210
Faculty Undergrads 0.3533 0.3536 338 1.00 0.3185 0.8557
Graduates Staff 0.6678 0.2991 338 2.23 0.0262 0.1702
Graduates Unknown 0.6128 0.2487 338 2.46 0.0143 0.1015
Graduates Undergrads 0.3628 0.3235 338 1.12 0.2628 0.7951
Staff Unknown –0.05497 0.3534 338 –0.16 0.8765 0.9999
Staff Undergrads –0.3049 0.4095 338 –0.74 0.4570 0.9458
Unknown Undergrads –0.2500 0.3743 338 –0.67 0.5047 0.9631
Note: Statistically significant effects/differences appear in bold.



The Scholarly Impact of Books Acquired   543

In the PDA data set for the sciences, the Patron Type performances were more in keeping 
with what one would expect from having read the cautionary PDA literature. Faculty and 
Graduates had roughly equal means and handily outperformed the other three Patron Types 
by more than one hundred citations per book. Surprisingly, however, none of the post hoc 
comparisons proved to be statistically significant after the adjustment of p values. Faculty and 
Graduates did significantly outperform the Staff and Unknown types prior to adjustment, 
but not the Undergraduates. A closer inspection of the data set revealed that Undergraduates 
books had the greatest performance variability, so this may in part account for the lack of a 
detectable difference. The bulk of Undergraduates books performed no better than did Staff 
and Unknown books, but the best-performing of these books performed as well as the best-
performing Graduates books and nearly as well as the best-performing Faculty books. From 
this, one would expect that, were Undergraduates acquisitions to be increased, eventually a 
statistically significant difference in performance would develop.

The performances of the Patron Types in the humanities’ PDA data set were more prob-
lematic than they were in the social sciences or sciences. The tests of fixed effects showed the 
presence of at least one likely real difference in the humanities’ PDA data set; but, after the 
adjustment for multiple comparisons, the only significant differences that remained were 

TABLE 5C
Humanities: Comparison of Citation Counts by Patron Type: Faculty, Graduates, Staff, 

Undisclosed-status Patrons (Unknown), and Undergraduates
Type III Tests of Fixed Effects

Effect Num DF Den DF F Value Pr > F
Patron Type 4 610 5.89 0.0001

Order Type Least Squares Means
Patron Type Estimate S.E. DF t Value Pr > |t| Mean S.E. Mean
Faculty 4.2792 0.1083 610 39.52 <.0001 72.1804 7.8161
Graduates 4.6096 0.09137 610 50.45 <.0001 100.44 9.1776
Staff 3.8774 0.2907 610 13.34 <.0001 48.2963 14.0397
Unknown 5.1714 0.1826 610 28.33 <.0001 176.16 32.1623
Undergrads 4.5852 0.2051 610 22.36 <.0001 98.0185 20.1018

Differences of Order Type Least Squares Means 
with Adjustment for Multiple Comparisons (Tukey-Kramer)

Patron Type Patron Type Estimate S.E. DF t Value Pr > |t| Adj. p
Faculty Graduates –0.3304 0.1417 610 –2.33 0.0200 0.1362
Faculty Staff 0.4018 0.3102 610 1.30 0.1957 0.6942
Faculty Unknown –0.8922 0.2123 610 –4.20 <.0001 0.0003
Faculty Undergrads –0.3060 0.2319 610 –1.32 0.1875 0.6792
Graduates Staff 0.7322 0.3047 610 2.40 0.0166 0.1159
Graduates Unknown –0.5618 0.2042 610 –2.75 0.0061 0.0479
Graduates Undergrads 0.02442 0.2245 610 0.11 0.9134 1.0000
Staff Unknown –1.2940 0.3433 610 –3.77 0.0002 0.0017
Staff Undergrads –0.7078 0.3558 610 –1.99 0.0471 0.2723
Unknown Undergrads –0.5862 0.2746 610 –2.14 0.0332 0.2065
Note: Statistically significant effects/differences appear in bold.



544  College & Research Libraries May 2019

those where the Unknown type outperformed the Faculty, Graduates, and Staff Patron Types, 
which information is, clearly, not useful. If one were to review the preadjustment p values, 
one would see that Graduates nearly significantly outperformed Faculty and Staff and that 
Undergraduates nearly outperformed Staff. Though Graduates and Undergraduates had 
nearly equal means, the latter’s performance did not as closely approach significance, likely 
because of the small number of books purchased by Undergraduates and because of the com-
paratively high variability in their citation performance, as was the case with the sciences’ 
Undergraduates above.

Analysis
For the social sciences, the results displayed in table 3 suggest that some of PDA’s naysayers’ 
and critics’ concerns about potential differences in the quality of vendors’, librarians’, and 
patrons’ book selections may be justified and that their arguments in favor of librarian title-
by-title selection were warranted. Librarians’ selections during the interval had more of an 
impact, as measured by citation counts, than did book vendors’ and patrons’ selections by 
statistically significant amounts. Patrons’ selections, of course, also had more impact than did 
book vendors’ selections for the approval plans, also by a statistically significant amount. The 
librarians’ sample of 491 books garnered 42,293 more citations than did those of the patrons, 
for a difference of roughly 86 citations per book. When looking at these numbers, it would 
be difficult to argue that librarian selection did not add value or that librarian selection and 
patron selection were equivalent where scholarly impact is concerned.

That said, these results would suggest to the authors that the more important story was 
the dismal performance of the approval plans’ book selectors. Over the study interval, the 
sample of titles that the book vendors sent to the UNL Libraries garnered almost 100,000 fewer 
citations than did the titles in the librarians’ sample and slightly more than 56,000 fewer cita-
tions than did the patrons’ purchases. However correct one may count PDA’s critics concern-
ing patrons’ failings as book selectors for the ages, the citation counts would suggest that the 
critics would be substantially more justified in concluding that book vendors deserve close, 
constant, and careful scrutiny.

As was noted above, to produce a more nuanced picture of the three selectors’/purchas-
ers’ performance, the authors elected also to compare the performance of the Order Types 
within Topics; but, where the approval plans were concerned, there was very little nuance to 
be found. In table 4a, the librarians’ purchases had higher mean citation counts in 7 of the 10 
Topic groups, and most of these differences showed the librarians’ books to have collected two, 
three, or four times as many citations. Likewise, the patrons outperformed the book vendors 
in 6 of the 10 Topic groups, with an additional two near ties. The sole Topic group where the 
approval plan selectors outperformed the other Order Types was “Military & Naval Sciences,” 
and even there the differences in performance were not statistically significant.

There may be some room for nuance where Librarians and PDA Patrons are concerned. 
The librarians had higher mean counts in half of the Topic groups, the patrons had higher 
mean counts in 4 of the 10 Topic groups, and the two had roughly equal performances in one 
group. Of the performance differences, each Order Type had just one that was statistically 
significantly different. Thus, of the three Order Types, the authors would suggest that firm-
ordering librarians and ILL-PoD-requesting patrons were much more nearly comparable in 
their performances than were the book vendors and the librarians or the book vendors and the 



The Scholarly Impact of Books Acquired   545

patrons, although, again, the librarians were clearly the superior selectors when the samples 
were looked at as a whole.

For both PDA’s critics and its advocates, the results presented by table 5a must have 
produced the most bafflement, in that they both confirmed and countered expectations. Of 
the five Patron Types, for example, Undergraduates performed the worst, just as PDA’s critics 
have contended they would. However, of the two Patron Types that both critics and advo-
cates esteem, only one, Graduates, outperformed Undergraduates by a statistically significant 
amount. The Faculty decidedly did not. A review of the results for a Patron Types within 
Topics analysis performed by the authors but not reported on did suggest that Faculty books 
pretty consistently produced higher mean citation counts than did Undergraduates books in 
Topic groups where purchasing overlapped, but, again, the differences in Faculty and Under-
graduates performance were largely statistically indistinguishable. Librarians critical of PDA 
have suggested to the lead author that this unexpected finding should be read as the result of 
undergraduates and faculty having purchased books with different qualities that both attract 
low citation counts. On the one hand, the former, in accordance with the anti-undergraduate 
and anti-PDA argument, obviously must have purchased books with low scholarly merit. On 
the other hand, the latter, in accordance with librarians’ generally pro-faculty attitude, must 
have purchased books of high scholarly merit, but these obviously were so advanced, abstruse, 
and/or esoteric as to attract less widespread attention. This may well be the case, but, without 
empirical support, this argument seems to the authors to be tainted by motivated reasoning.

And what should PDA’s critics and advocates make of the results for the university staff? 
This is a group of library patrons that has largely been ignored in the PDA/DDA literature, 
yet here they clearly outperformed the other three groups with identifiable patrons. The so-
cial sciences’ Staff purchases attracted roughly 5,200 more total citations than did Faculty’s, 
despite their having made 30 purchases to Faculty’s 103. In essence, the Staff books attracted 
4.5 citations for every citation that the Faculty books attracted. Perhaps one ought to expand 
Walter’s argument against letting undergraduates add to the collection and make a point of 
including university staff in PDA/DDA purchasing.

How ought one to interpret this surprising result? Unfortunately, because of policies 
intended to protect patron privacy, the authors were unable to delve into just what was be-
hind the outsized performance of the university staff, but we are able to offer some plausible 
speculation based upon our knowledge of local culture that may suggest that the staff’s 
performance may be read as a local effect. First, some university administrators at UNL may 
be listed in the patron database as members of the university staff rather than as members 
of the faculty. Second, it is not entirely uncommon at UNL for recently graduated graduate 
students, especially students in education and in a handful of the social sciences, to take 
various sorts of staff positions on campus. Once in these positions, some of them spend a 
few years conducting research and building their résumés prior to applying for faculty and 
administrative positions at UNL or elsewhere. Thus, it might be that some sizable percentage 
of the Staff performance could more accurately be characterized as having been produced 
by administrators with active research agendas and by lingering, highly motivated graduate 
students intent on improving their chances in a tough career market. Thus, if one’s institution 
has a culture similar to that of UNL, one may see a similar performance from one’s staff; if 
not, then likely not. Regardless, this study’s findings for Research Question #3 suggest that, 
in the social sciences at least, some of PDA’s critics’ assumptions concerning the intersection 



546  College & Research Libraries May 2019

between academe’s natural hierarchy and the quality of what academic PDA patrons purchase 
may be inaccurate and misleading.

Where the sciences were concerned, the statistics presented in appendix B and the results 
of the first batch of statistical tests reported on in table 3 suggest that PDA’s critics’ opinions 
in favor of librarian title-by-title selection were, again, justified. Their concerns with PDA, 
however, do not appear to have been supported by the data analyzed here. Far and away, the 
concerns that were most clearly and substantially supported by the UNL Libraries’ sciences 
data were, once again, critics’ concerns over the motivations and/or performances of vendors. 
Both the librarians’ and the patrons’ science books had higher citation counts by statistically 
significant amounts than did the book jobbers’ approval plan selections. Librarians’ books gar-
nered just over 103,000 more citations than did the plans’ books, and Patrons’ books garnered 
almost 78,000 more. Librarians’ books did outperform Patrons’ by more than 25,000 citations, 
but this difference, again, proved not to be statistically significant. 

Thus, in reviewing the Order Types in general, it would also be difficult to argue that 
librarian selection in the sciences does not add value, and it would certainly be appropriate 
to see the results of this study as supporting the contention of several of the more caution-
ary voices in the PDA debate that librarian selection should not be abandoned in favor of 
universal PDA just yet.55 The results of this study do not, so far, support the contention of 
reactionary voices in the debate that have suggested PDA/DDA ought not to be employed 
in academic libraries. The citation counts of the PDA Patrons books were nearly as good 
as were Librarians, so there would seem to be little justification for excluding academic 
library patrons in the sciences from collection development or for closely monitoring and 
mediating their collecting, especially if the distribution of one’s library’s PDA requests tends 
to resemble the UNL Libraries’ skew in favor of Faculty and Graduates. With minimal and 
reasonable restrictions in place (again, see appendix A), the UNL Libraries’ patrons per-
formed quite well.

The same, of course, cannot be said for the book vendors’ contributions to the collection 
via approval plans. The results of the table 3 tests for the sciences, as they did for the social 
sciences, suggest that critics of vendors may be more right than they know. Approval plan 
selectors for the UNL Libraries have not only proven themselves to be poor at selecting books 
that meet local needs,56 but they have, so far, proven themselves to be poor at identifying and 
selecting books that meet the interests of researchers generally, as well.

Again, to produce a more nuanced picture of the three Order Types’ performance, with 
Research Question #2 the authors also compared the sciences’ books’ performance by Order 
Type within Topic groups. Of the 27 head-to-head, within-Topic comparisons reported on, 
the 9 involving Librarians versus PDA Patrons showed no statistically significant differences 
in performance, even prior to the adjustments made to their p values to account for multiple 
comparisons. In fact, the preadjustment p values showed only a single difference in perfor-
mance being nearly significant. Thus, these results support the conclusion that there is little 
to choose from where Librarian and PDA Patron selection in the sciences are concerned. 

The same, once again, cannot be said for the approval plan selections. Half of the 18 post 
hoc comparisons showed Approval Plans as having been outperformed by either the Librar-
ians or the Patrons by statistically significant amounts, even after adjustment for multiple 
comparisons. An additional three comparisons—against Patrons in Chemical Technology & 
Manufacturing and in Engineering & Technology (General) and against Librarians in Medicine 



The Scholarly Impact of Books Acquired   547

(General, Public Health, & Pathology)—were also nearly significant. Thus, Approval Plans’ per-
formance proved to be pretty consistently poor across the several Topics.

With respect to the criticisms and cautions raised and aired concerning which sorts of 
patrons ought or ought not to be allowed to add books to academic libraries’ collections, the 
results for the final research question would appear to offer some support for the contention 
that members of the faculty and graduate students can be relied upon as worthy selectors, but 
that undergraduate students and others might not be as reliable. In the set of tests reported 
on in table 5b, Faculty and Graduates citation counts were statistically indistinguishable, and 
there was also little to distinguish one from the other in additional Patron Types within-Topics 
post hoc tests performed later by the authors but not reported on. In the general tests of per-
formance, both well outperformed Undergraduates, Staff, and the Unknown Patron Types, 
although none of the post hoc head-to-head comparisons’ results proved to be statistically 
significant. Still, Faculty and Graduates would seem a safer and more consistent bet in the 
sciences.

The humanities data—including the statistics in appendix B, the general tests of the 
Order Types presented in table 3, and the Order Types within-Topics tests presented in table 
4c—offer very strong support for the PDA approach. For advocates of librarian title-by-title 
selection, this data set also offers some welcome, if not entirely unproblematic, support as well. 
Consistent with the results presented for the social sciences and for the sciences, the humani-
ties’ results for the general testing of the three Order Types support both PDA and librarian 
title-by-title selection approaches to some extent and call into question the value of approval 
plans for meeting researchers’ needs. The humanities PDA books attracted tens of thousands 
more citations than did the librarians’ firm-ordered books and the approval plans’ selections, 
and they, unsurprisingly, strongly and statistically significantly outperformed both in table 
3. As well, in the 22 Order Types within-Topics comparisons reported on in table 4c, the PDA 
books pretty consistently outperformed the other two Order Types, with 18 of the 22 PDA 
mean citation counts being higher than either the Librarians’ or Approval Plans’ mean counts 
and with 12 of the 22 post hoc comparisons in table 4c favoring PDA by statistically signifi-
cant amounts. Thus, it would seem worthwhile to allow humanities patrons a fair amount of 
freedom to add books to an academic library collection. 

Librarians, as was the case with the other disciplines, also outperformed Approval Plans 
by a statistically significant amount in table 3, although the humanities’ difference in perfor-
mance was not as outsized as were those of the social sciences or the sciences. In table 4c, 
which presented the Order Types within-Topics comparisons, however, the Librarians versus 
Approval Plans relationship was shown to be much less clear. The within-Topics comparisons 
actually would appear to have been something of a draw. The two Order Types had higher 
means in roughly equal numbers of Topic categories, and the majority of the post hoc com-
parisons strongly support failing to reject the null hypothesis. In essence, table 4c suggests 
that the liaison librarians for languages and literatures were able to outperform the approval 
plans, but none of the other humanities librarians were able to distinguish themselves in any 
meaningful way.

Finally, for those critics who have taken a position against allowing undergraduates to 
add books to academic library collections via PDA, the UNL Libraries’ humanities data would 
offer little to no support. None of the known Patron Types performed better or worse than any 
of the others by statistically significant amounts, although there was some slight performance 



548  College & Research Libraries May 2019

variation between types of patrons. Thus, the results reported here would support letting 
humanities students and faculty, and perhaps even university staff, to order what they like 
within the guidelines and constraints of the program. 

Limitations to the Study
The first limitation to this study of which to be cognizant is one that affects much of the library 
literature generally and, as Walker noted, the PDA literature particularly: this was a single 
study conducted with data from a single site. This, of course, limits its generalizability, as 
it is impossible to determine whether or not it was merely characteristics of the setting that 
produced the effects presented. Thus, the results reported and conclusions drawn may not 
be universal, and studies of this sort should be conducted at other sites to reduce or eliminate 
the impact of local factors.

The issue of the study’s generalizability naturally leads to the second important limita-
tion to the study: the issue of its robustness. Recently, the robustness of scientific findings in 
a myriad of fields has been called into question,57 and this has led to something of a crisis 
in validity and to a renewed interest in replication. Recently, researchers have had difficulty 
reproducing the results of highly cited papers in clinical research,58 the findings of classic 
studies in psychological science,59 the conclusions medical studies and drug trials,60 findings 
concerning structural brain-behavior correlations,61 and so forth. Reproducibility is a defin-
ing feature of science, the final arbiter of findings’ validity, and the cornerstone of cumulative 
science,62 and this should be no less the case in library science. The current study, as far as 
the authors are aware, has been the first of its kind. Without additional studies of this sort, it 
will be difficult, if not impossible, to determine how much the results of this study actually 
explain rather than merely describe, to determine whether the authors have observed system-
atic regularities, idiosyncrasies in the data, or mere products of chance.63 Thus, it would be 
incorrect to conclude that the issues raised by PDA’s critics and advocates have been firmly 
and entirely settled here.

The third important limitation to this study centers upon the type of PDA program stud-
ied. As has been noted, previous research on PDA has shown that Order Type has been a good 
predictor of circulation.64 At the UNL Libraries, ILL PoD purchasing has been associated with 
greater circulation than librarians’ orders and approval plans’ selections.65 In essence, that an 
academic library patron has a need, perceived a book to have the potential to meet that need, 
was willing to go through the UNL Libraries’ ILL process to request it, and was then willing 
to wait for a few days to receive the requested item has predicted that the requested item 
will circulate significantly more in the short term. Thus, ILL-PoD–derived findings may not 
hold for other sorts of PDA programs that have lower barriers to patron satisfaction, such as 
catalog-integrated DDA programs for e-books that allow one to merely click on a link to ac-
cess a book.66 The ILL-PoD findings also may not hold for programs that merely encourage 
patrons to recommend books for purchase, such as mediated “suggest-a-book” programs that 
direct patron input to selector librarians for vetting.67 

A fourth limitation to the study involves the research questions and the metric employed 
to answer them. As was noted above, this study only addresses scholarly merit via scholarly 
impact as measured by citation counts. As such, it offers the conclusion that Order Type may 
be a useful predictor of scholarly attention, at least in the short term. This study has nothing 
to say about the quality of the scholarship in the books under examination, about how last-



The Scholarly Impact of Books Acquired   549

ing their influence will be in the long run, about how well received they were by scholars, or 
about any of the other, more qualitative conclusions one could come to about the books and 
their impact. 

A fifth limitation to the study that the authors would like to note involves the samples 
employed. For the first research question, depending upon the discipline, the study had be-
tween 300 and 500 books per Order Type to analyze. These samples should be sufficiently 
large to draw and support some meaningful conclusions. As the study progressed to its second 
question, however, the authors had to “slice” the samples more finely. For the third question, 
the authors employed just the books that comprised the three PDA samples. Thus, the results 
for these latter questions should be treated a bit more cautiously, and the conclusions drawn 
should be treated as being less certain. 

Finally, the reader should keep in mind that this study did not control for the disciplinary 
affiliations of the librarians and PDA patrons. The study’s samples were drawn and divided 
solely on the basis of Order Type and Topic. As a result, one cannot conclude that librarians 
or patrons with various disciplinary affiliations were or were not particularly good at select-
ing books of high interest in their areas of specialization. For example, the samples for the 
“Education” Topic contained books purchased by librarians with liaison assignments to de-
partments in the College of Education and Human Sciences, but the samples also contained 
books purchased by librarians with assignments to other departments and programs, such 
as anthropology, sociology, ethnic studies, and so forth. In most Topics, pertinent disciplin-
ary affiliations did account for the majority of purchases in most Topics, but one should still 
not conclude too much from this study about the value of narrow and specific disciplinary 
knowledge.

Conclusions
The conclusions to be drawn from this study would seem to be several. For critics who have 
warned against a too eager, uncritical, and comprehensive adoption of PDA as a collections 
panacea and for advocates of the value of librarians as selectors, this study has provided 
some much-needed empirical support. Librarians performed very well in the social sciences 
and in the sciences, as well as or better than did the PDA patrons and much better than did 
the approval plans’ selectors. In the humanities, of course, the librarians were substantially 
outperformed by the patrons, but they did still significantly outperform the approval plans 
in their turn. Thus, it would seem far too soon to worry over whether selection is dead, as a 
panel discussion at a recent Charleston Conference provocatively inquired. The results of this 
study would support the myriad panelists’ arguments that librarian selection is still valuable 
and vital.68 

For PDA critics who have loudly opined against allowing patrons in general or against 
undergraduates in particular to add titles to academic libraries’ collections, this study pro-
vides some much-needed evidential discouragement. The PDA patrons performed well in all 
three disciplines, and the cases for the faculty and against the undergraduates were nowhere 
near as clear or strong as PDA’s critics would apparently have them. Graduate students were 
consistently good performers across all disciplines and within many Topics, and Faculty 
performed very well in the sciences, but the results of this study certainly do not support a 
blanket disparagement or lionization of any one type of patron. Certainly, concerned academic 
librarians could more closely monitor undergraduate students’ requests if doing so felt neces-



550  College & Research Libraries May 2019

sary, but if their institutions’ undergraduate students perform as well and make up as small 
a percentage of PDA purchases as do UNL’s, then the authors would be inclined to suggest 
that such heightened gatekeeping would largely be a waste of time.

Which of the final two conclusions to this study will prove the most pertinent will depend 
upon the nature of the reader’s library and the principles guiding its collecting, as well as 
where one’s library falls along what Nardini, in an early article on the history of the adoption 
of approval plans, posited as the depth and breadth versus precision divide.69 Some number 
of this journal’s readers are employed by large research libraries with budgets and mandates 
oriented more toward comprehensiveness. Many more, no doubt, are employed by academic 
libraries with constrained budgets whose collection-building focus is directed more toward 
meeting patrons’ current needs.

For the large research library whose intent is to collect more comprehensively and to build, 
as Anderson would have it, a cultural monument, this study should provide some reassur-
ance that operating a PDA program will not precipitate the arrival of a “bibliopocalypse” that 
destroys the library, regardless of what some PDA critics appear to have been prophesying. 
The UNL PDA books in this study attracted good amounts of scholarly attention during the 
study interval, and nothing in this study would suggest that PDA polluted or debased the 
UNL Libraries’ collection with low-quality dross, beneath the interest of scholars. Past surveys 
of PDA programs have found that early adopters were devoting 1 to 5 percent of their book 
budgets to PDA.70 Given the results of this study and of past studies of PDA books’ circulation 
and collection suitability, it would be difficult to imagine that a large research library, however 
monumental its collection, could not profitably devote at least such a small percentage of its 
book budget to meeting its current patrons’ expressed needs.

For the more typical sort of academic library, the library hoping to direct limited col-
lection dollars toward impactful titles, the results of this study suggest that the PDA critics’ 
cautions against vendors’ and publishers’ plans and packages could profitably be heeded. 
Nardini, in a recently published chapter, positioned approval plans as an important part of 
academic libraries’ selection infrastructure, one that provides libraries with numerous ben-
efits and advantages, especially in the areas of efficiency and collection balance.71 From the 
results of this study, however, it would be difficult not to conclude that, as prices continue to 
increase and as collection dollars grow ever more dear, approval plans’ advantages may be 
outweighed by their failings. The results of this and past studies at the UNL Libraries incline 
the authors to wonder whether, in the librarian-vendor relationship, the shadowy outlines of 
a classic principal-agent problem may not be discerned, the sort of problem where one party 
believes the other to be acting to advance its interests when the second party is in fact acting 
to advance its own.72 Contrary to the rosy picture advanced by Nardini in the aforementioned 
chapter, much of the early criticisms of the approval plan model warned of just such potential 
conflicts of interest.73 Approval plans and e-book packages may well deliver the efficiencies 
and savings promised, but at UNL the approval plans appear to have been doing so with 
books that comparatively few researchers want. The not infrequent response in the field to 
this dilemma is to suggest that vendors’ products require monitoring and modification to 
function perfectly,74 but employing this approach at UNL actually produced worse perfor-
mance on several occasions. If academic libraries with similarly constrained book budgets 
find themselves experiencing similarly unsatisfactory results with vendor plans or packages, 
they may wish to investigate the benefits of divesting.



The Scholarly Impact of Books Acquired   551

To close: as was noted above, the library literature on PDA has so far been largely posi-
tive and supportive of the model, and a number of studies have substantiated the claims of its 
advocates, especially where circulation/usage and collection suitability have been concerned. 
The counter literature skeptical of or cautioning against PDA, to this point, has warned of 
the dangers and shortcomings of the model almost entirely speculatively. The authors hope 
that this study has contributed to a continued, evidence-driven, and empirically supported 
conversation on the PDA model and furthered the examination of its benefits and pitfalls 
for academic libraries. Effective and efficient materials selection is a difficult process, and 
academic librarians engaged in collection building often find themselves striving to satisfy 
multiple and sometimes contradictory masters. Concerning the struggle to purchase books 
that meet current patrons’ needs and the simultaneous struggle to purchase books of high 
and sustained scholarly interest, this study suggests that PDA’s advocates may be right: Giv-
ing the patrons what they want today may very well be supplying the collection with what 
it needs for tomorrow.



552  College & Research Libraries May 2019

APPENDIX A. UNL University Libraries’ ILL Purchasing Project 
Criteria (2010–2011)
All books requested via ILLiad, published between 2007 and 2010 and not owned by UNL 
Libraries must be considered for purchase (books in IRIS that are counted as missing, lost, 
or on search cannot be purchased). Selected books will be judged by the following criteria. 
Books must be available from Barnes and Noble online and must arrive in a timely manner. 
We cannot order any book that takes longer than 2–3 days to ship. We also cannot order books 
that are marked as “Pre-orders.” The price limit on purchasing project books is $175; we may 
not order books that cost more than this amount. Books purchased have academic merit. The 
following type of books may not be purchased:

• Textbooks (any book designated by OCLC or Barnes & Noble as a textbook)
• Foreign language books
• Fiction, poetry, plays
• Music scores
• Lab manuals, workbooks, field guides
• Solutions manuals
• Popular interest (nonfiction, best sellers, self-help, and the like)
• Popular biographies (biographies can be decided on a case-by-case basis)
• Journal volumes/serials
• Computer books
• Anthologies
• Older editions of a book that still fall within the 3-year period

Most books not marked as textbooks published by university presses are appropriate for the 
ILL purchasing program. Another way to judge a book that may be published by an unknown 
publisher but is not disqualified by any of the above criteria is to look in the OCLC holdings 
to see what other libraries own this item. If it is owned by many GWLA and ARL libraries, 
it should be purchased. If it is owned by mostly public libraries, it should be obtained via 
interlibrary loan. If it is a new book not owned by many libraries or you think it would be a 
good addition to the collection, decide which subject area the book falls under and consult the 
library liaison. Ask if they feel the book is a worthwhile purchase. If they feel that it is, you 
may buy it. If the liaison is not available or the book falls under the prohibited categories but 
still seems like a good addition to the library collection, ask [the Chair of Technical Services] 
for permission to purchase it.



The Scholarly Impact of Books Acquired   553

APPENDIX B. Sample Distributions and Parameters
Social Sciences: Distributions and Parameters of Citation Counts by Order Type:  

1) Approval Plans, 2) Librarians, and 3) PDA Patrons

1)

Pe
rc

en
t

Sum = 79,068, Range = 0–6,050, Mean = 161.035, Median = 43, 
St. Dev. = 498.968, Skew = 8.491, Skew S.E. = .110

2)

Pe
rc

en
t

Sum = 177,741, Range = 0–28,479, Mean = 361.998, Median = 47,
St. Dev. = 1,728.174, Skew = 10.791, Skew S.E. = .110

3)

Pe
rc

en
t

Sum = 135,448, Range = 0–11,758, Mean = 275.862, Median = 71,
St. Dev. = 975.865, Skew = 8.224, Skew S.E. = .110

Times Cited
Curve Normal

Note: Values rounded to the nearest 1/1,000th where appropriate.



554  College & Research Libraries May 2019

Sciences: Distributions and Parameters of Citation Counts by Order Type:  
1) Approval Plans, 2) Librarians, and 3) PDA Patrons

1)

Pe
rc

en
t

Sum = 36,588, Range = 0–3,381, Mean = 106.671, Median = 36, 
St. Dev. = 241.988, Skew = 8.290, Skew S.E. = .132

2)

Pe
rc

en
t

Sum = 139,987, Range = 0–23,877, Mean = 408.125, Median = 41,
St. Dev. = 1,840.102, Skew = 9.860, Skew S.E. = .132

3)

Pe
rc

en
t

Sum = 114,545, Range = 0-12,252, Mean = 333.950, Median = 77,
St. Dev. = 1,044.363, Skew = 8.937, Skew S.E. = .132

Times Cited
Curve Normal

Note: Values rounded to the nearest 1/1,000th where appropriate.



The Scholarly Impact of Books Acquired   555

Humanities: Distributions and Parameters of Citation Counts by Order Type:  
1) Approval Plans, 2) Librarians, and 3) PDA Patrons

1)

Pe
rc

en
t

Sum = 29,985, Range = 0–1,800, Mean = 48.756, Median = 17, 
St. Dev. = 113.845, Skew = 8.029, Skew S.E. = .099

2)

Pe
rc

en
t

Sum = 39,765, Range = 0–3,995, Mean = 64.659, Median = 14, 
St. Dev. = 216.044, Skew = 11.726, Skew S.E. = .099

3)

Pe
rc

en
t

Sum = 59,899, Range = 0–5,042, Mean = 97.397, Median = 26, 
St. Dev. = 314.550, Skew = 9.480, Skew S.E. = .099

Times Cited
Curve Normal

Note: Values rounded to the nearest 1/1,000th where appropriate.



556  College & Research Libraries May 2019

Notes
 1. Jennifer Perdue and James A. Van Fleet, “Borrow or Buy? Cost-effective Delivery of Monographs,” Journal 

of Interlibrary Loan, Document Delivery & Information Supply 9, no. 4 (1999): 19–28.
 2. Association of College and Research Libraries, Research, Planning and Review Committee, “2010 Top 

Ten Trends in Academic Libraries: A Review of the Current Literature,” College & Research Libraries News 71, no. 
6 (2010): 286–92; Association of College and Research Libraries, Research, Planning and Review Committee, 
“2012 Top Ten Trends in Academic Libraries: A Review of the Trends and Issues Affecting Academic Libraries 
in Higher Education,” College & Research Libraries News 73, no. 6 (2012): 311–20.

 3. Cristina Caminita, “E-Books and Patron-Driven Acquisitions in Academic Libraries,” in Customer-based 
Collection Development: An Overview, ed. Karl Bridges (Chicago: ALA Editions, 2014), 1–12; Karin J. Fulton, “The 
Rise of Patron-driven Acquisitions: A Literature Review,” Georgia Library Quarterly 51, no. 3 (2014): Article 10, 
available online at http://digitalcommons.kennesaw.edu/glq/vol51/iss3/10 [accessed 28 November 2017]; Denise 
A. Garofalo, “Tips from the Trenches,” Journal of Electronic Resources Librarianship 23, no. 3 (2011): 274–76; Rick 
Lugg, “Collecting for the Moment: Patron-driven Acquisitions as a Disruptive Technology,” in Patron-driven 
Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 7–22; 
Norm Medeiros, “King of the Hill,” OCLC Systems & Services 28, no. 2 (2012), 64–66.

 4. Kizer Walker, “Patron-Driven Acquisition in US Academic Research Libraries: At the Tipping Point in 
2011?” Bibliothek Forschung und Praxis 36, no. 1 (2012): 126–30.

 5. Deborah Lenares and Emilie Delquié, “Give the People What They Want: Patron Driven Acquisition: Re-
sults and Reflections on a Survey Completed by Publishers Communication Group” (presentation, STM Spring 
Conference, Le Meridien Cambridge-MIT, Cambridge, MA, April 21–29, 2010).

 6. Nestor L. Osorio, “A User’s Participatory Selecting Model: Librarians Point of Views” (presentation, Illi-
nois Library Association Annual Meeting, Rosemont, IL, October 19, 2011), available online at http://hdl.handle.
net/10760/16507 [accessed 26 January 2018].

 7. Joseph Esposito, Kizer Walker, and Terry Ehling, “PDA and the University Press: A Report Prepared for 
The Andrew W. Mellon Foundation,” Journal of Scholarly Publishing 44, no. 3 (2013): s1–s62.

 8. Steven Carrico, Michelle Leonard, and Erin Gallagher, Implementing and Assessing Use-Driven Acquisitions 
(Lanham, MD: Rowman & Littlefield, 2016); Tony Horava and Michael Levine-Clark, “Current Trends in Col-
lection Development Practices and Policies,” Collection Building 35, no. 4 (2016): 97–102.

 9. Caminita, “E-Books and Patron-Driven Acquisitions in Academic Libraries,” 1–12.
10. Google, Google Scholar, n.d., s.v. “college, libraries, patron, driven, acquisitions,” available online at http://

scholar.google.com/ [accessed 31 October 2017].
11. Dee Ann K. Allison, The Patron-driven Library: A Practical Guide for Managing Collections and Services in the 

Digital Age (Oxford: Chandos Publishing, 2013); Rick Anderson, “Collections 2021: The Future of the Library 
Collection Is Not a Collection,” Serials 24, no. 3 (2011): 211–15; Faye A. Chadwell, “What’s Next for Collection 
Management and Managers? User-centered Collection Management,” Collection Management 34, no. 2 (2009): 
69–78; Joseph Esposito, Kizer Walker, and Terry Ehling. “The New Supply Chain and Its Implications for Books 
in Libraries,” Educause Review 47, no. 5 (2012): 58–59; Esposito, Walker, and Ehling, “PDA and the University 
Press,” s1–s62; Fulton, “The Rise of Patron-driven Acquisitions,” Article 10; Edward A. Goedeken and Karen 
Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” College & Re-
search Libraries 76, no. 2 (2015): 205–21; Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 
126–30.

12. John Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions: A Summary,” in Customer-
based Collection Development: An Overview, ed. Karl Bridges (Chicago: ALA Editions, 2014): 159–76; Barbara Fister, 
“Problematizing Patron-driven Acquisitions,” Library Journal: Peer to Peer Review (blog, November 11, 2010); Barbara 
Fister, “Puzzled by Patron-driven Acquisitions,” Library Babel Fish (blog, November 11, 2010), available online at 
https://www.insidehighered.com/blogs/library-babel-fish/puzzled-patron-driven-acquisitions [accessed 12 April 
2019]; Mary Gilbertson, Elizabeth Chadbourn McKee, and Lutishoor Salisbury, “Just in Case or Just in Time? 
Outcomes of a 15-month Patron-driven Acquisition of e-Books at the University of Arkansas Libraries,” Library 
Collections, Acquisitions & Technical Services 38, no. 1/2 (2014): 10–20; William Miller, “Patron-driven Acquisitions 
(PDA): The New Wave in Book Acquisitions Is Coming,” Library Issues: Briefings for Faculty and Administrators 31, 
no. 5 (2011): 1–4; Scott A. Smith, “PDA: Driving Off the Cliff or, New Wine in Old Bottles,” Against the Grain 23, no. 
3 (2011): 40; Suzanne M. Ward, Guide to Implementing and Managing Patron-Driven Acquisitions (Chicago: American 
Library Association, 2012).

13. Esposito et al., “The New Supply Chain and Its Implications for Books in Libraries,” 58–59; Esposito et 

https://digitalcommons.kennesaw.edu/glq/vol51/iss3/10/
http://hdl.handle.net/10760/16507
http://hdl.handle.net/10760/16507
http://scholar.google.com/
http://scholar.google.com/
https://www.insidehighered.com/blogs/library-babel-fish/puzzled-patron-driven-acquisitions


The Scholarly Impact of Books Acquired   557

al., “PDA and the University Press,” s1–s62; Goedeken and Lawson, “The Past, Present, and Future of Demand-
driven Acquisitions in Academic Libraries,” 205–21; Lugg, “Collecting for the Moment,” 7–22.

14. Kathleen A. Lehman, “Collection Development and Management,” Library Resources & Technical Services 58, 
no. 3 (2014): 169–77; Miller, “Patron-driven Acquisitions (PDA),” 1–4; Jane Schmidt, “Demand-driven Acquisi-
tions: The Hegemony of the Canon Interrupted,” in Creating Sustainable Community: The Proceedings of the ACRL 
2015 Conference (Portland, Oregon, March 25–28, 2015), 168-73, available online at www.ala.org/acrl/sites/ala.org.
acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf [accessed15 November 2017]; Elena S. Smith, 
“Power and Practice in Academic Library Materials Selection Paradigms,” SLIS Student Research Journal 1, no. 2 
(2011): Article 6, available online at http://scholarworks.sjsu.edu/slissrj/vol1/iss2/6/ [accessed 1 May 2012]; William 
H. Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” Library Resources 
& Technical Services 56, no. 3 (2012): 199–213.

15. Medeiros, “King of the Hill,” 64–66. 
16. Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30.
17. Carrico et al., Implementing and Assessing Use-Driven Acquisitions, xiii.
18. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Esposito et al., 

“PDA and the University Press,” s1–s62; Gilbertson et al., “Just in Case or Just in Time?” 10–20; Smith, “PDA: 
Driving Off the Cliff or, New Wine in Old Bottles,” 40; Walker, “Patron-Driven Acquisition in US Academic 
Research Libraries,” 126–30.

19. David C. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials: 
A Chronological Review and Summary of Findings,” Library Philosophy and Practice (2011): Article 635, available 
online at http://digitalcommons.unl.edu/libphilprac/635/ [accessed 26 January 2018]; David C. Tyler, Joyce C. 
Melvin, MaryLou Epp, and Anita M. Kreps, “Patron-driven Acquisition and Monopolistic Use: Are Patrons at 
Academic Libraries Using Library Funds to Effectively Build Private Collections?” Library Philosophy and Practice 
(2014) 11, Article 1,149, available online at http://digitalcommons.unl.edu/libphilprac/1149/ [accessed 26 January 
2018].

20. Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 129. 
21. Jason C. Dewland and Andrew See, “Notes on Operations: Patron Driven Acquisitions: Determining the 

Metrics for Success,” Library Resources & Technical Services 59, no. 1 (2015): 13–23.
22. Carrico et al., Implementing and Assessing Use-Driven Acquisitions, 95.
23. Laura Costello, Evaluating Demand-driven Acquisitions (Cambridge, UK: Chandos Publishing, 2016); Tyler, 

“Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635; Tyler et 
al. “Patron-driven Acquisition and Monopolistic Use,” Article 1,149.

24. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic 
Libraries,” 205–21; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 
199–213; Aaron Wood, “Mainstream Patron-Driven Acquisition: Topicality Over the Scholarly Record… and the 
Cello Suites,” Against the Grain 25, no. 5 (2013): 22.

25. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic 
Libraries,” 205–21.

26. Costello, Evaluating Demand-driven Acquisitions, ix.
27. Joseph Esposito, “A Dialogue on Patron-Driven Acquisitions,” The Scholarly Kitchen (blog, January 3, 2012), 

available online at https://scholarlykitchen.sspnet.org/2012/01/03/a-dialogue-on-patron-driven-acquisitions/ [ac-
cessed 15 November 2017].

28. Sheila Corrall, “Community Engagement in Collection Development: Social Responsibility or Professional 
Abdication?” (presentation, Community Engagement & Social Responsibility: ALISE’17 Annual Conference, 
Atlanta, GA, January 17–20, 2017). 

29. James A. Cogswell, “The Organization of Collection Management Functions in Academic Research Li-
braries,” Journal of Academic Librarianship 13, no. 5 (1987): 270.

30. Rick Anderson, “What Patron-Driven Acquisition (PDA) Does and Doesn’t Mean: An FAQ,” The Scholarly 
Kitchen (blog, May 31, 2011), available online at https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven-
acquisition-pda-does-and-doesnt-mean-an-faq/ [accessed 15 November 2017]; Anderson, “Collections 2021,” 
211–15.

31. Anderson, “Collections 2021,” 211–15; Lorcan Dempsey, “Library Collections in the Life of the User: Two 
Directions,” Liber Quarterly 26, no. 4 (2017): 338–59; Julie Linden, Sarah Tudesco, and Daniel Dollar, “Collections 
as a Service: A Research Library’s Perspective,” College & Research Libraries 79, no. 1 (2018): 86–99; Doug Way, 
“Transforming Monograph Collections with a Model of Collections as a Service,” portal: Libraries and the Acad-
emy 17, no. 2 (2017): 283–94.

32. Miller, “Patron-driven Acquisitions (PDA),” 1–4; Schmidt, “Demand-driven Acquisitions: The Hegemony 

http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf
http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf
http://scholarworks.sjsu.edu/slissrj/vol1/iss2/6/
http://digitalcommons.unl.edu/libphilprac/635/
http://digitalcommons.unl.edu/libphilprac/1149/
https://scholarlykitchen.sspnet.org/2012/01/03/a-dialogue-on-patron-driven-acquisitions/
https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven-acquisition-pda-does-and-doesnt-mean-an-faq/
https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven-acquisition-pda-does-and-doesnt-mean-an-faq/


558  College & Research Libraries May 2019

of the Canon Interrupted,” 168–73; Smith, “Power and Practice in Academic Library Materials Selection Para-
digms,” Article 6; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 
199–213.

33. Douglas Jones, “On-demand Information Delivery: Integration of Patron-driven Acquisition into a Com-
prehensive Information Delivery System,” Journal of Library Administration 51, no. 7/8 (2011): 764–76; Michael 
Levine-Clark, “Developing a Multiformat Demand-driven Acquisition Model,” Collection Management 35, no. 
3/4 (2010): 201–07; Michael Levine-Clark, “Developing a Model for Long-term Management of Demand-driven 
Acquisitions,” Against the Grain 23, no. 3 (2011): 24–26; Michael Levine-Clark, “Building a Demand-Driven Col-
lection: The University of Denver Experience,” in Patron-driven Acquisitions: History and Best Practices, ed. David 
A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 45–60; Peter Spitzform, “Patron-driven Acquisition: 
Collecting as if Money and Space Mean Something,” Against the Grain 23, no. 3 (2014): 20–24; Peter Spitzform and 
Pongracz Sennyey, “A Vision for the Future of Academic Library Collections,” International Journal of the Book 4, 
no. 4 (2007): 185–89.

34. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Fister, “Problema-
tizing Patron-driven Acquisitions” (blog); Fister, “Puzzled by Patron-driven Acquisitions” (blog); Jean-Mark 
Sens and Anthony J. Fonseca, “A Skeptic’s View of Patron-driven Acquisitions: Is It Time to Ask the Tough 
Questions?” Technical Services Quarterly 30, no. 4 (2013): 359–71.

35. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Chadwell, “What’s 
Next for Collection Management and Managers?” 69–78; Candice Dahl, “Primed for Patron-driven Acquisition: 
A Look at the Big Picture,” Journal of Electronic Resources Librarianship 24, no. 2 (2012): 119–26; Lindsey Reno, “Out 
of the Shadows: A Public Face for Acquisitions in Academic Libraries,” Against the Grain 24, no. 6 (2012): 22–26; 
F.K. Rottmann, “To Buy or to Borrow: Studies of the Impact of Interlibrary Loan on Collection Development in 
the Academic Library,” Journal of Interlibrary Loan & Information Supply 1, no. 3 (1991), 17–27; Sens and Fonseca, “A 
Skeptic’s View of Patron-driven Acquisitions,” 359–71; Smith, “PDA: Driving off the Cliff or, New Wine in Old 
Bottles,” 40; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213; 
Ward, Guide to Implementing and Managing Patron-Driven Acquisitions; Aaron Wood, “Mainstream Patron-Driven 
Acquisition,” 22.

36. Audrey Fenner, “The Approval Plan: Selection Aid, Selection Substitute,” The Acquisitions Librarian 16, no. 
31/32 (2004): 227–40; Sherry E. Gelbwasser, “Vendor-Library Relationships in ARL Libraries,” The Acquisitions 
Librarian 8, no. 16 (1996): 109–17; Alison Griffin and Sarah Forzetting, “Behind the Scenes: Approval Profiles 
from a Vendor’s Perspective,” in Library Collection Development for Professional Programs: Trends and Best Practices, 
ed. Sarah Holder (Hershey, PA: IGI Global, 2012), 16–31; Bob Nardini, “Approval Plans and Patron Selection: 
Two Infrastructures,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and 
Boston: Walter de Gruyter, 2011), 23–43; Robert F. Nardini, “Approval Plans: Politics and Performance,” College 
& Research Libraries 54, no. 5 (1993): 417–26.

37. Wenli Gao, Irene Ke, and Cherie Turner, “Programming Plus Subject Expertise: A Combined Approach for 
Approval Profile Modification,” Collection Management (forthcoming); Irene Ke, Wenli Gao, and Jackie Bronicki, 
“Does Title-By-Title Selection Make a Difference? A Usage Analysis on Print Monograph Purchasing,” Collection 
Management 42, no. 1 (2017): 34–47; David C. Tyler, Christina Falci, Joyce C. Melvin, MaryLou Epp, and Anita M. 
Kreps, “Patron-driven Acquisition and Circulation at an Academic Library: Interaction Effects and Circulation 
Performance of Print Books Acquired via Librarians’ Orders, Approval Plans, and Patrons’ Interlibrary Loan 
Requests,” Collection Management 38, no. 1 (2013): 3–32; David C. Tyler, Yang Xu, Joyce C. Melvin, Marylou Epp, 
and Anita M. Kreps, “Just How Right Are the Customers? An Analysis of the Relative Performance of Patron-
Initiated Interlibrary Loan Monograph Purchases,” Collection Management 35, no. 3/4 (2010): 162–79.

38. David C. Tyler, Joyce C. Melvin, Yang Xu, Marylou Epp, and Anita M. Kreps, “Effective Selectors? Inter-
library Loan Patrons as Monograph Purchasers: A Comparative Examination of Price and Circulation-Related 
Performance,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 21, no. 1/2 (2011): 57–90.

39. Denise Brush, “Circulation Analysis of an Engineering Monograph Approval Plan,” Collection Building 
26, no. 2 (2007): 59–62; Erin L. Ellis, Nikhat J. Ghouse, Monica Claassen-Wilson, John M. Stratton, and Susanne 
K. Clement, “Comparing Approval and Librarian-Selected Monographs: An Analysis of Use,” in Library Data: 
Empowering Practice and Persuasion, ed. Darby Orcutt (Santa Barbara, CA: Libraries Unlimited, 2010), 53–68; Marcie 
Kingsley, “Circulation Statistics for Measuring Approval Plan Effectiveness,” Against the Grain 8, no. 4 (1996): 1, 
16–17; James Cory Tucker, “Collection Assessment of Monograph Purchases at the University of Nevada, Las 
Vegas Libraries,” Collection Management 34, no. 3 (2009): 157–81; Michele Wolff, “Evaluating the Art Approval 
Plan: A Case Study,” Art Documentation 18, no. 1 (1999): 5–9.

40. John S. Brantley, “Approval Plans, Discipline Change, and the Importance of Human Mediated Book 
Selection,” Library Collections, Acquisitions, and Technical Services 34, no. 1 (2010): 11–24; Mary Eldredge, “Major 



The Scholarly Impact of Books Acquired   559

Issues in Approval Plans: The Case for Active Management,” The Acquisitions Librarian 8, no. 16 (1996): 51–59.
41. Kady Ferris and Tina Herman Buck, “An Ethos of Access: How a Small Academic Library Transformed 

Its Collection-building Processes,” Collection Management 39, no. 2/3 (2014): 127–44; Goedeken and Lawson, “The 
Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21.

42. Costello, Evaluating Demand-driven Acquisitions.
43. Esposito, Walker, and Ehling, “PDA and the University Press,” s1–s62.
44. Tyler et al., “Patron-driven Acquisition and Monopolistic Use,” 11.
45. Jason Price and John McDonald, “Beguiled by Bananas: A Retrospective Study of the Usage & Breadth of 

Patron vs. Librarian Acquired eBook Collections,” in Proceedings of the Charleston Library Conference (Charleston, 
South Carolina, 2009), doi:10.5703/1288284314741; Gabrielle Wiersma and Yem Fong, “Patron-driven e-Book Solu-
tions: Moving beyond the Banana Books Incident,” in Proceedings of the Charleston Library Conference (Charleston, 
South Carolina, 2011), doi:10.5703/1288284314973.

46. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic 
Libraries,” 205–21.

47. Leslie J. Reynolds, Carmelita Pickett, Wyoma Vanduinkerken, Jane Smith, Jeanne Harrell, and Sandra 
Tucker, “User-driven Acquisitions: Allowing Patron Requests to Drive Collection Development in an Academic 
Library,” Collection Management 35, no. 3/4 (2010): 244–54.

48. Jeffrey H. Waller, “Undergrads as Selectors: Assessing Patron-driven Acquisition at a Liberal Arts Col-
lege,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 23, no. 3 (2013): 127–48.

49. Gilbertson et al., “Just in Case or Just in Time?” 10–20; Doug Way, “The Assessment of Patron-initiated 
Collection Development via Interlibrary Loan at a Comprehensive University,” Journal of Interlibrary Loan, Docu-
ment Delivery & Electronic Reserve 19, no. 4 (2009): 299–308.

50. Waller, “Undergrads as Selectors,” 127–48.
51.. Judith M. Nixon and E. Stewart Saunders, “A Study of Circulation Statistics of Books on Demand: A 

Decade of Patron-Driven Collection Development, Part 3,” Collection Management 35, no. 3/4 (2010): 151–61.
52. Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213.
53. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 

635.
54. Emily K. Chan and Susan L. Kendall, “Identifying Users of Demand-driven E-book Programs: Applica-

tions for Collection Development,” in Imagine, Innovate, Inspire : Proceedings of the ACRL 2013 Conference (Chicago: 
Association of College and Research Libraries, 2013), 301–08.

55. Rick Anderson, “Is Selection Dead? The Rise of Collection Management and the Twilight of Selection,” 
Against the Grain 23, no. 2 (2011): 40–44.

56. Tyler et al., “Patron-driven Acquisition and Circulation at an Academic Library,” 3–32; Tyler et al., “Just 
How Right Are the Customers?” 162–79.

57. John P.A. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (2005): 
e124, available online at http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 [accessed 
23 April 2018].

58. John P.A. Ioannidis, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Re-
search,” JAMA 294, no. 2 (2005): 218–28.

59. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 349, no. 
6251 (2015): aac4716; Harold Pashler and Eric-Jan Wagenmakers, “Editors’ Introduction to the Special Section 
on Replicability in Psychological Science: A Crisis of Confidence?” Perspectives on Psychological Science 7, no. 6 
(2012): 528–30. 

60. C. Glenn Begley and Lee M. Ellis, “Drug Development: Raise Standards for Preclinical Cancer Re-
search.” Nature 483, no. 7391 (2012): 531–33; Florian Prinz, Thomas Schlange, and Khusru Asadullah, “Believe It 
or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” Nature Reviews Drug Discovery 10, 
no. 9 (2011): 712.

61. Wouter Boekel, Eric-Jan Wagenmakers, Luam Belay, Josine Verhagen, Scott Brown, and Birte U. Forstmann, 
“A Purely Confirmatory Replication Study of Structural Brain-Behavior Correlations,” Cortex 66 (2015): 115–33.

62. Jennifer Crocker and M. Lynne Cooper, “Addressing Scientific Fraud,” Science 334, no. 6060 (2011): 1182; 
Jeremy Freese, “Replication Standards for Quantitative Social Science: Why Not Sociology?” Sociological Methods 
& Research 36, no. 2 (2007): 153–72; Gary King, “Replication, Replication,” PS: Political Science & Politics 28, no. 3 
(1995): 444–52; Barbara R. Jasny, Gilbert Chin, Lisa Chong, and Sacha Vignieri, “Again, and Again, and Again…,” 
Science 334, no. 6060 (2011): 1225; Matthew C. Makel and Jonathan A. Plucker, “Facts Are More Important Than 
Novelty: Replication in the Education Sciences,” Educational Researcher 43, no. 6 (2014): 304–16; Stefan Schmidt, 
“Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences,” Review 

https://doi.org/10.5703/1288284314741
https://doi.org/10.5703/1288284314973
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124


560  College & Research Libraries May 2019

of General Psychology 13, no. 2 (2009): 90–100.
63. Nassim Nicholas Taleb, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (New 

York: Random House, 2005).
64. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 

635.
65. Tyler et al., “Patron-driven Acquisition and Circulation at an Academic Library,” 3–32; Tyler et al., “Just 

How Right Are the Customers?” 162–79.
66. Costello, Evaluating Demand-driven Acquisitions.
67. Rebecca Schroeder, “When Patrons Call the Shots: Patron-driven Acquisition at Brigham Young Univer-

sity,” Collection Building 31, no. 1 (2012): 11–14.
68. Anderson, “Is Selection Dead?” 40–44.
69. Nardini, “Approval Plans: Politics and Performance,” 420.
70. Lenares and Delquié, “Give the People What They Want.”
71. Nardini, “Approval Plans and Patron Selection,” 23–43.
72. Michael C. Jensen and William H. Meckling, “Theory of the Firm: Managerial Behavior, Agency Costs 

and Ownership Structure,” Journal of Financial Economics 3, no. 4 (1976): 305–60.
73. Association of Research Libraries, Approval Plans in ARL Libraries, SPEC Kit, no. 83 (Washington, DC: ARL, 

1982), 79–84; Margaret Dobbyn, “Approval Plan Purchasing in Perspective,” in Background Readings in Building 
Library Collections, 2nd ed., eds. Phyllis Van Orden and Edith B. Phillips (Metuchen, NJ: Scarecrow Press, 1979): 
194; Jane Maddox, “Approval Plans—Viable?” Journal of Academic Librarianship 1 (1976): 22; Rose Mary Magrill 
and Mona East, “Collection Development in Large University Libraries,” in Advances in Librarianship, vol. 8, ed. 
Michael H. Harris (San Diego, CA: Academic Press, 1978): 6; Norman D. Stevens, “The Resources and Technical 
Services Division of ALA,” Library Resources & Technical Services 13 (1969): 334. 

74. Leslie Czechowski, “Edging toward Perfection: Analysis of a New Approval Plan in a Health Sciences 
Library,” Library Collections, Acquisitions, & Technical Services 32 (2008): 107–11; Eldredge, “Major Issues in Ap-
proval Plans,” 51–59.