81 Maximizing Academic Library Collections: Measuring Changes in Use Patterns Owing to EBSCO Discovery Service Kristin Calvert Kristin Calvert is Electronic Resources Librarian at Western Carolina University; e-mail: kcalvert@wcu.edu. © 2015 Kristin Calvert, Attribution-NonCommercial (http://creativecommons.org/licenses/by-nc/3.0/) CC BY-NC. Despite the prevalence of academic libraries adopting web-scale dis- covery tools, few studies have quantified their effect on the use of library collections. This study measures the impact that EBSCO Discovery Service has had on use of library resources through circulation statistics, use of electronic resources, and interlibrary loan requests. Additionally, the data were compared against data from similar academic libraries without discovery systems. Findings include a strong growth in e-journal use but a sharp decline in circulation statistics. Implications of discovery on use of the print collection are discussed along with suggestions for improving integration of book and article data. estern Carolina University (WCU) is a regional comprehensive university, and part of the University of North Carolina System, serving the western part of the state. In the spring of 2011, the library undertook a review of current web-scale discovery solutions and resolved to purchase EBSCO Discovery Service (EDS). The decision favored EBSCO’s product over others on the market for two primary reasons: first, WCU subscribes to a very large number of data- bases on the EBSCOhost platform and wanted to maximize use of those databases and leverage users’ familiarity with the interface; and second, the confidence the library has in EBSCOhost’s relevancy rankings. The implementation process for EDS occurred over the summer and the service launched to the public in August 2011, under the branding Find It! The key aims of the current study are to measure the direct and indirect effects of a discovery tool on the use of library resources through circulation numbers, use of electronic resources, and interlibrary loan requests. The library seeks to maximize the value of the resources it purchases and to ensure that the resources are being fully used. When WCU selected and implemented EDS, the librarians imagined it would be the new go-to resource for less-experienced undergraduate users. In addition, the advantage to the consolidated index of a discovery tool offered would be to support interdisciplinary research and undergraduates who cite journal titles “across a broad spectrum of subject areas.”1 Specifically, it would replace EBSCOhost’s Academic Search doi:10.5860/crl.76.1.81 crl13-557 82 College & Research Libraries January 2015 Complete as the patron’s first stop for information and most highly used resource. The content searched in EDS includes EBSCOhost databases, records loaded from EBSCO’s data partners, library catalog and institutional repository holdings, and any resources set up as a federated search option. Uncertainty over continued funding and the relative worth of discovery led the library to consider the subscription to EDS to be a one-year “pilot” test. The emergence of funding for a second year and the realization that there would be insufficient data available before the renewal decision had to be made, forestalled the library’s assess- ment of the “pilot.” While the product received mixed reviews from library staff, the library continued the subscription for a third year and undertook processes to mea- sure the benefit of the product on the undergraduate user experience. The library is constructing usability testing procedures to gauge user success and behavior, but the first question always asked is “how much use are we seeing”? Literature Review Much of the literature on discovery tools to date has focused on the selection and implementation of a discovery layer. The special issue, “Discovery Tools: The Next Generation of Library Research” in College & Undergraduate Libraries, coedited by John S. Spencer and Christopher Millson-Martula, touches on a variety of issues libraries faced when adopting discovery tools: selection, impact on information literacy programs, marketing campaigns, and case studies.2 Published studies on the assessment of dis- covery tools are still emerging, but they typically involve usability studies, transaction log analysis, or analysis of usage statistics. Studies of federated search engines and other similar systems, which focused on many of the same issues, are included here to provide additional context for research on later products. Usability testing provides researchers with the opportunity to observe how actual users interact with a system. Testing scenarios typically ask participants to perform predetermined tasks. The observers assess the ability of the user to independently use features and the interface, judge the user’s understanding of terms used by the product, and measure how efficiently the user was able to complete the tasks, if at all. The outcomes of these studies suggest how satisfied users are with the product or service and inform how libraries can respond by making improvements to the product in response to how well participants performed. Libraries have reported on usability tests for all currently available discovery systems. For many libraries, these tests ask what improvements discovery tools present to users over the library catalog, Google Scholar, or federated search tools. Studies report that the major developments to discovery systems over federated search tools come from ease of use and improved navigation.3 Discovery systems generally support better relevancy rankings, deliver higher quality resources, and offer a variety of material types within a single interface.4 But for all the gains made toward improved usability, users still stumble when performing basic tasks in discovery systems. Studies for every system have reported that users have problems identifying or distinguishing between the material types of results.5 Additionally, discovery systems can overwhelm users with the sheer volume of search results and the myriad ways to refine or redo a search.6 Students continue to struggle to find books or ascertain the availability or call number of the item.7 Many users found it to be just as difficult to find a book in the discovery system as they did in the traditional library catalog, failing especially when they need to locate a specific edition of a book.8 In fact, usability testing at Bucknell University and Illinois Wesleyan University found that users who searched discovery tools chose fewer books from their results lists than those who searched Google Scholar, the library catalog, or used no tool at all.9 On the road to creating a simpler search environment for library users, Maximizing Academic Library Collections 83 some aspects of search become more complicated through the combination of print and electronic resources. The challenges revealed in usability testing question whether a discovery system gets in its own way and interferes with users being able to connect to the resources they need. Transaction logs also reveal information on the hidden interactions between a user and the discovery system. From this type of analysis, the library can ascertain the types of subjects most often searched, identify spurious requests (searches for URLs or for library hours), and identify which features and search types are most often used. For discovery systems, a majority of users still rely on basic keyword searches and rarely take advantage of advanced search options or conduct searches using Boolean opera- tors.10 Academic users (over those as other types of libraries) rely more on abstracts to judge the relevance of results prior to accessing the full text.11 When the user employs a less sophisticated search methodology, it forces the responsibility to produce high- quality search results onto the relevancy ranking algorithm of the discovery tool. In such a paradigm, the question of what effect single-search has on the discovery and use of highly relevant resources becomes paramount. Studies quantitatively examining the effects of federated search or discovery tools on the usage of library resources, however, are limited. Ruth Stubbings surveyed Lough- borough University’s implementation of MetaLib, Ex Libris’ federated search product.12 Stubbings and Hamblin followed up on MetaLib at Loughborough and reviewed the usage statistics from two academic years.13 For two similar studies on discovery tools, Kemp and Way measured use library resources after the implementation of Summon. 14 Kemp reviewed data for one year of print and e-resource use at the University of Texas at San Antonio, while Way relied on a semester’s worth of e-resource data at Grand Valley State University. The four studies shared some common methods for assessing collection use. Each study used COUNTER15 compliant database search and login statistics from individual databases. Neither MetaLib nor Summon could provide built-in statistics on the prod- uct itself or the content it searched. Instead, the authors relied on statistics from other sources and extrapolated the effect the federated search tool or discovery layer had on use patterns. In the studies focusing on Summon, both Kemp and Way used the number of link resolver click-throughs to track whether users accessed the full text; however, Way went on to include full-text downloads from COUNTER journal reports. The sole study to consider the effect on print collections with circulation statistics was Kemp’s at UT San Antonio. Use patterns across all materials and metrics were not uniform in these four stud- ies. E-journal use rose markedly in the Summon studies when measured through both click-through statistics and full-text downloads in COUNTER JR1 reports.16 Circulation of print materials also increased, though at a more modest rate compared to e-journals. Circulation statistics of the UT San Antonio’s print materials increased 1.2 percent, but Kemp noted the increase was smaller than the enrollment growth for the same year and the effects reported may not be indicative of the benefits of the discovery system.17 While e-journal and book use benefited positively in the aforementioned studies, the overall findings on database use require a more nuanced summation. After the first year of MetaLib at Loughborough University, Stubbings reported an increase in use for all databases, but a more dramatic increase for cross-searched databases, ranging from 10 to 300 percent.18 When Stubbings and Hamblin followed up a year later, they observed a 609 percent increase in the number of searches across databases, but noted a corresponding 13 percent drop in the number of logins.19 At UT San Antonio, the number of searches decreased by 5 percent (when Summon was excluded), but full- text accesses rose 23 percent,20 with similar trends in use observed at Grand Valley 84 College & Research Libraries January 2015 State University. Way, additionally, reported a drop in use of abstracting and indexing (A&I) databases. A clearly defined relationship between discovery systems and the number of database searches cannot be inferred from these studies with any certainty, though the data strongly imply use of a discovery tool that displaces use of databases on their native platforms. Putting the existing research into perspective, there are several outstanding issues that future research can address. The limited data from these fairly young discov- ery systems hinder the conclusiveness of the research on the effects of Summon on library resource. It is quite common for there to be peaks or dips in use in any given semester at academic libraries based on course offerings and fluctuations in incoming class size. As libraries employ discovery systems for longer stretches, the data will be available for longitudinal studies to help minimize any variance in the results that may be present in the existing studies. The same is true for use of print collections. As libraries consider how to address decline use of print, further studies on what role discovery has in exposing all of the libraries’ collections are needed. Another disadvantage of these studies is the reliance on purely external markers of use. It is difficult in the best of cases to eliminate intervening variables, but tracking data and use within the discovery system, as is possible with EDS, should work to isolate the role of the discovery tool. Each discovery system presents different strengths and weak- nesses, and each vendor comes at the problem from a slightly different background. To come to terms whether their effects can be generalized to all cases, research into each discovery system is needed. Hypotheses and Methods The present study posits three effects of web-scale discovery on library resource us- age: (1) EDS will result in increased use of the library’s e-resources; (2) EDS will result in increased use of the library’s print collection; and (3) EDS will result in decreased use of interlibrary loan for materials. Discovery layers counteract the effect that information silos have on patrons by eliminating the need for the user to assess his or her information needs and then identify an appropriate library resource. Currently, a student has to know to start with the online catalog when looking for print materials, search a number of different journal databases when looking for articles, wade through countless pathfinders or topic-specific wikis set up by the librarian, and explore the institutional repository when looking for local content.21 EDS presents a simplified search experience through a single search box, which exposes the patron to a far greater number of the library’s electronic, media, and print resources. Consolidated access to the library’s resources facilitates the connection of users with an article or book in the library to meet their needs. In a study examining the bibliographies of students at SUNY Oneonta, Hendley counted the number of academic journals cited by the students.22 Only 1 out of the 60 journals cited by un- dergraduates was not held by the library. If users, particularly undergraduates, find something sufficient to meet the research needs, they will use that resource instead of seeking out a (potentially) better source, because availability counts more than specificity for many assignments. “If students have used something successfully in the past, they will return to that resource for other research questions.”23 WCU can leverage students’ preference for Academic Search Complete and have their EBSCO brand loyalty carry over to EDS. Use of print and electronic resources should increase in this environment. Moreover, requests for material from other libraries should Maximizing Academic Library Collections 85 drop as a result of the strengthened use of locally held materials. Users with more specific research needs will still want to get the exact article, whatever the source, but less experienced undergraduates will find something from the library to use for their assignments. Methods for Hypothesis 1: Measuring Increased Use of the Library’s E-Resources The first objective is to demonstrate a positive, direct relationship between imple- menting a discovery tool and use of the library’s licensed e-resources. To measure the use and changes in use patterns for licensed resources, the current study uses three methods. First, EBSCOhost database use will be measured through abstract views and full-text retrievals. The second will compare the change in use of EBSCOhost databases at WCU to the change in use of EBSCOhost databases at three other institutions in North Carolina: Appalachian State University (ASU), University of North Carolina at Asheville (UNCA), and NC LIVE. The third method for measuring use of e-resources is in full-text retrievals in COUNTER JR1 reports for publishers where the library subscribes to a journal package. This study omits individual journal subscriptions to limit the number of data points and to simplify data gathering. E-books are not considered because of insufficient usage data available from the period prior to the implementation of EDS. The current study examines metrics on databases available on the EBSCOhost platform where the subscription began on or before January 2010. Abstract views and full-text retrievals more accurately reflect use of EDS content because of the way EBSCO records its session information. When a user performs a search in EBSCO Discovery Service, a search and session are recorded for all content included in the EDS profile. Thus, sessions and search counts for each content source searched by EDS are roughly equivalent to the total number of searches and sessions performed on the EDS platform and therefore become unuseful standards of measure. The cur- rent study prefers abstract views and full-text retrievals because these metrics reflect active use of the specific database content on the part of the user. For this reason, the study must rely upon non-COUNTER statistics for database-level use, which are not currently available for databases from vendors besides EBSCO. Data from the period starting January 2010 through December 2012 will be examined in six-month blocks: spring semester (January to June) and fall semester (July to December). The study period provides three semesters of use prior to the implementation of EDS and three semesters of use with EDS in place. Next, NC LIVE, ASU, and UNCA contributed their aggregated, site-level EBSCO- host statistics on the total number of abstract views and full-text retrievals by month to this study. Comparative use data are included here to provide a big picture view of trending use of EBSCOhost resources and, by extension, all electronic resources in North Carolina. The current study uses these comparative data from these institu- tions as a control group for change in use patterns due to other factors (including the growth of online resources, or year-over-year increases in use) and not caused by the implementation of a discovery layer.24 Western Carolina University and the other schools in the University of North Carolina System participate in NC LIVE, North Carolina’s statewide online library service, which provides electronic resources to all North Carolina residents. Due to the large number of EBSCOhost databases avail- able to North Carolina’s universities through NC LIVE, database holdings are more similar across institutions than they might otherwise be. The relatively even playing field provided by the NC LIVE common resources should allow any effect EDS has on increasing e-resource use to become readily apparent. 86 College & Research Libraries January 2015 Methods for Hypothesis 2: Measuring Increased Use of the Library’s Print Collection The current study will demonstrate a positive, direct relationship between implement- ing a discovery tool and use of the library’s physical collections with three data sets: (1) the total number of checkouts by WCU patrons; (2) in-house use statistics; and (3) consortial borrowing requests. WCU shares an integrated library system, Innovate Interfaces Inc.’s Sierra, with ASU and UNCA, the other members of the Western North Carolina Library Network (WNCLN). There is a cooperative borrowing agreement within the consortium to loan most library materials through a service called ABC Express. To encourage use of the service, catalog records for borrowable materials from ASU and UNCA are included in WCU’s catalog loads to EDS. Sierra web circulation reports provide the number of checkouts (grouped by patron type) and a count of in-house use. The WNCLN network office provides the count of the number of ABC Express requests placed. Historical reporting in the ILS is limited to the last 36 months, restricting the period of study for the data to July 2010–June 2012. The data are grouped into fiscal years 2010–2011 and 2011–2012. Comparisons of use statistics for the print collections at WCU to those at ASU and UNCA are also considered to rule out unidentified intervening factors. Trends in use at the three universities have followed a similar trajectory historically partly because they share a public catalog. Any variation from the trends seen at the other institutions is based on the effect of the discovery system on WCU patrons. Methods for Hypothesis 3: Measuring Decreased Use of Interlibrary Loan to Borrow Material The third hypothesis is predicated on the first two hypotheses holding true. There exists an inverse relationship between use of the library’s collections and the number of requests for material through interlibrary loan. The rise in use of print and online resources resulting from implementing EDS, as demonstrated by hypotheses 1 and 2, fulfills the research needs of WCU users. By satisfying the research needs of users with locally held materials, EDS reduces the demand for material held by other in- stitutions. The reduction in demand will be demonstrated through fewer borrowing requests created in ILLiad during the time study period where EDS is used. ILLiad Web Reports will provide interlibrary loan statistics for borrowing requests. Under borrowing reports, the study will use Requests Finished to measure the change in demand for books and journal articles via interlibrary loan. This report displays the number of requests finished within the given time period. The current study will col- lect data for the period starting with January 2010 and ending December 2012. Data will be separated into six-month blocks: spring semester (January to June) and fall semester (July to December). Results Hypothesis 1: EDS will result in increased use of the library’s licensed e-resources measured through EBSCOhost research database and e-journal use Hypothesis 1 is supported by the data. Tables 1 and 2 show the number of abstracts viewed by EBSCOhost database. The databases are separated into two groups: one group for databases where at least some full-text coverage is included for the peer-reviewed journals, full-text periodicals, reports, and books indexed in the databases. The shorter list in table 2 includes those EBSCOhost databases where only A&I are provided. For each database, the average number of abstracts viewed per semester was calculated. The change in this average after EDS was implemented is shown in the percent change column. For both groups of databases, the overall use as measured by abstracts viewed Maximizing Academic Library Collections 87 grew (see figure 1). While the databases with full text dominate in raw numbers for both numbers of databases and overall use, the biggest percent increase came from the A&I databases. Of the thirty-seven EBSCOhost databases included here, ten showed declined use of abstracts. Of note is the use of Academic Search Complete, the library’s most used resource, dropping 15 percent. Not the largest percentage drop, but the decrease corresponds to more than 13,000 fewer abstract views for this content source. Caution should be exercised when drawing conclusions from changes in specific databases. TABLE 1 Average Number of Abstracts Viewed Per Semester for EBSCOhost Databases with Full Text Full-Text Databases Pre-EDS Post-EDS Percent Change MasterFILE Complete 1,006.00 3,298.67 228% Consumer Health Complete 24.00 75.67 215% MAS Complete 73.00 190.00 160% Literary Reference Center Plus 739.67 1,911.33 158% Newspaper Source Plus 610.00 1,539.33 152% Communication & Mass Media Complete 473.33 1,175.67 148% Middle Search Plus 72.33 136.67 89% Library, Information Science & Technology Abstracts with Full Text 478.33 885.00 85% NoveList Plus K–8 639.00 1,114.67 74% SPORTDiscus with Full Text 3,490.33 6,086.33 74% MEDLINE with Full Text 5,033.00 8,679.33 72% CINAHL with Full Text 7,909.00 13,127.00 66% SocINDEX with Full Text 4,415.67 7,048.67 60% GreenFILE 85.00 110.33 30% ERIC 17,102.00 18,630.00 9% Business Source Complete 7,388.33 7,986.00 8% PsycARTICLES 2,355.00 2,535.33 8% Regional Business News 143.33 153.00 7% Health Source: Consumer Edition 286.00 299.00 5% Military & Government Collection 255.33 260.33 2% Hospitality & Tourism Complete 247.67 215.67 –13% Academic Search Complete 85,433.33 72,419.67 –15% Science Reference Center 329.33 278.33 –15% Health Source: Nursing/Academic Edition 405.33 234.67 –42% Mental Measurements Yearbook with Tests in Print 786.67 358.00 –54% TOPICsearch 80.33 31.00 –61% NoveList Plus 2,353.33 581.67 –75% Total 142,214.67 149,361.33 5% 88 College & Research Libraries January 2015 Some records are present in multiple EBSCOhost databases but deduplicated in EDS. When the user views the abstract for the deduplicated results, the action is recorded once for the database of the record displayed, but the choice of which record the system shows can be arbitrary, based on the orders of records in the index. The other metric this study selected to measure database use is full-text retrievals in EBSCOhost databases. The list of EBSCOhost research databases in table 3 is identical to TABLE 2 Average Number of Abstracts Viewed Per Semester for EBSCOhost Abstracting and Indexing Databases A&I Databases Pre-EDS Post-EDS Percent Change GeoRef 72.00 631.00 776% Historical Abstracts 116.33 401.00 245% America: History & Life 287.00 827.00 188% Philosopher’s Index 260.67 712.33 173% MLA International Bibliography 2,387.00 3,303.67 38% Biological & Agricultural Index Plus 168.33 216.67 29% PsycINFO 12,526.00 14,850.67 19% Teacher Reference Center 226.00 206.00 –9% Applied Science & Technology Abstracts 277.00 190.00 –31% MLA Directory of Periodicals 235.67 146.33 –38% A&I Databases 16,556.00 21,484.67 30% FIGURE 1 Total Abstracts Viewed in EBSCOhost Databases with Pre- and Post-EDS Semester Averages 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000 200,000 Spring 2010 Fall 2010 Spring 2011 Fall 2011 Spring 2012 Fall 2012 FT Databases A&I Databases Average Pre-EDS Average with EDS Maximizing Academic Library Collections 89 the list in table 1. The average number of full-text retrievals per semester was calculated before and after the EDS implementation date in the same manner as the calculations for tables 1 and 2. The overall 6 percent increase in full-text retrievals is comparable to the 5 percent increase in number of abstracts viewed for EBSCOhost databases including full text. While the overall number of documents retrieved increased, fewer databases individually showed positive growth. The number of full-text retrievals is an under- TABLE 3 Average Number of Full Text Documents Accessed Per Semester Per EBSCOhost Database Database Pre-EDS Post-EDS Percent Change MasterFILE Complete 1,744.00 3,966.67 127% Literary Reference Center Plus 1,141.33 2,488.33 118% GreenFILE 1.67 3.33 100% MEDLINE with Full Text 2,897.67 5,151.00 78% Middle Search Plus 124.33 209.33 68% CINAHL with Full Text 4,977.00 7,932.67 59% SocINDEX with Full Text 4,245.33 5,836.33 37% Library, Information Science & Technology Abstracts with Full Text 408.33 559.00 37% Communication & Mass Media Complete 1,179.00 1,564.33 33% SPORTDiscus with Full Text 3,907.33 4,948.67 27% MAS Complete 193.67 234.00 21% Consumer Health Complete 268.00 323.33 21% PsycARTICLES 7,175.00 7,433.67 4% Science Reference Center 437.00 424.67 –3% Newspaper Source Plus 2,831.00 2,736.67 –3% Hospitality & Tourism Complete 178.33 170.00 –5% Academic Search Complete 58,424.67 54,602.67 –7% Business Source Complete 8,098.00 7,129.33 –12% Military & Government Collection 266.33 225.00 –16% Health Source: Consumer Edition 528.00 343.33 –35% Regional Business News 327.00 185.33 –43% Health Source: Nursing/Academic Edition 501.67 277.67 –45% Mental Measurements Yearbook with Tests in Print 1,028.00 447.67 –56% TOPICsearch 95.00 36.00 –62% NoveList Plus 10.00 0.33 –97% ERIC 32.00 — –100% NoveList Plus K–8 1.67 — –100% Grand Total 101,021.33 107,229.33 6% 90 College & Research Libraries January 2015 representation of actual full-text use in EBSCOhost databases. The full-text use in table 3 counts full text (PDF and HTML) available in EBSCOhost but does not include full text of articles available through EBSCO’s linked full text (via EBSCO Journal Service) or set up through custom linking to publisher websites. These data are included in the analysis of COUNTER JR1 reports. The current study compared statistics gathered from EBSCOhost from other institu- tions that did not have a discovery tool in place to assess whether the increasing use observed in the previous section is correlated to use of the discovery tool. A strong positive effect of a discovery tool in use at WCU should be apparent in the data over and above any growth in use at the other institutions. Tables 3 and 4 compare the abstracts viewed and full-text retrievals from all EBSCOhost content at NC LIVE, ASU, UNCA, and WCU. The data include use from more databases than are listed in tables 1, 2, and 3. Tables 4 and 5 calculate the percent change in use for the variable from calendar year 2011 to 2012. 25 All institutions have seen a steady increase in full text. At WCU, use of full text increased at a comparable rate to UNCA and NC LIVE, but by only half as much as at ASU. In contrast, abstract views at WCU increased at a rate 15 percent greater than the rate of growth in full-text retrievals, while the rate of increase for the metrics at the other institutions is within 1 to 5 percentage points of each other. The data suggest EDS resulted in a significant increase in abstract views, but the evidence for EDS strongly affecting full-text retrievals in EBSCOhost databases is weak. The final method for considering the validity of first hypothesis comes from data from COUNTER JR1 reports for WCU’s major journal providers (see table 6). WCU has big deal packages with Wiley, Sage, Elsevier, and Emerald. The remaining publishers in table 6 supply WCU with smaller journal collections. The average number of full- text downloads per semester was calculated for the period before and after EDS was implemented. Overall, there has been a 16 percent increase in full-text retrievals from e-journals (see figure 2). Major problems with linking to Ovid journal titles through TABLE 4 Comparison Across Institutions of Full-Text Documents Accessed on EBSCOhost Spring 2011 Fall 2011 Spring 2012 Fall 2012 Percent Change from 2011 to 2012 NC LIVE 2,736,175 2,979,340 3,184,340 3,489,079 17% UNCA 24,536 44,031 36,687 39,811 12% ASU 137,750 179,729 178,108 218,944 25% WCU 93,259 109,257 108,593 117,595 12% TABLE 5 Comparison Across Institutions of Abstracts Viewed on EBSCOhost Spring 2011 Fall 2011 Spring 2012 Fall 2012 Percent Change from 2011 to 2012 NC LIVE 6,651,710 7,296,527 7,422,980 8,222,916 12% UNCA 41,280 70,758 57,450 69,130 13% ASU 228,223 293,846 276,732 371,546 24% WCU 158,165 200,629 216,950 238,778 27% Maximizing Academic Library Collections 91 TABLE 6 Average Number of Full-Text Downloads per Semester by Journal Package (from COUNTER JR1 Reports) Pre-EDS Post-EDS Percent Change Ovid 439.67 1,294.33 194% Elsevier ScienceDirect 18,504.00 21,736.00 17% Cambridge University Press 734.33 839.00 14% Oxford University Press 1,392.67 1,552.33 11% Emerald 760.00 822.33 8% Sage 7,831.00 8,346.33 7% JSTOR 24,520.00 25,634.33 5% Springer 1,723.33 1,757.00 2% Mary Ann Liebert 94.67 95.67 1% Wiley 4,654.00 4,245.67 –9% Project Muse 2,148.67 1,918.67 –11% American Chemical Society 1,392.00 921.67 –34% Totals 59,534.67 69,163.33 16% the link resolver suppressed use through July 2011. Those problems were corrected, and the correction is more likely the cause for the 3-figure percentage change in use rather than EDS. If the Ovid journals are excluded, the overall increase stays strongly positive; the remaining 15 percent growth in use in e-journals is attributed to the likely effect of EDS. FIGURE 2 Total Full-text Downloads from COUNTER JR1 Reports 0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 Spring 2010 Fall 2010 Spring 2011 Fall 2011 Spring 2012 Fall 2012 92 College & Research Libraries January 2015 Hypothesis 2: EDS will result in increased use of the library’s print collection and consortial borrowing Hypothesis 2 is rejected. Circulation of monographs and print materials at Hunter Library declined over the last several years. After the implementation of EDS, check- outs at WCU continued to decline, decreasing 28 percent from the previous year. To judge to what extent this decrease is resultant from EDS and not a continuation of the downward trend, the change in use patterns of print materials was compared to those at UNCA and ASU. Figure 3 displays the total number of checkouts for the study period broken out by school and by patron type. The data show a more precipitous drop in circulation numbers at WCU when compared to ASU and UNCA. The effect is more pronounced when considering undergraduates alone. WCU undergraduate checkouts decreased 39 percent after EDS was implemented. From these data, the null FIGURE 3 Change in Number of Check-outs by Institution and Patron Type –28% –17% –15% –39% –17% –18% 0 50,000 100,000 150,000 200,000 250,000 WCU All Users UNCA All Users ASU All Users WCU Undergrad UNCA Undergrad ASU Undergrad FY10–11 FY11–12 FIGURE 4 In-house Use Statistics by Semester — 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 Fall 2010 Spring 2011 Fall 2011 Spring 2012 Fall 2012 Maximizing Academic Library Collections 93 hypothesis is rejected; a relationship exists between EDS and the use of print resources, but it is an indirect relationship. The current study finds EDS had a negative effect on the number of checkouts of library material. In-house use was considered here as an alternate use of the print collection, which would not be reflected in the number of checkouts. In-house use has increased slightly over the same two fiscal periods as the circulation data. However, in-house counting has not been performed consistently or systematically across all library collections over the years, making suspect any claim of a measurable change due to EDS. While in-house use is trending upward, the rate of increase is small compared to the decline in the number of checkouts. If patrons are discovering library material through EDS and using the materials in the building, the use is not great enough to counteract the decline in checkouts. Total use of WCU print collections continues to decrease. The study considered whether including catalog records for materials available through the consortial borrowing agreement with UNCA and ASU in EDS had any effect on WCU’s use of the service. The number of ABC Express requests to share material between the schools has declined in recent years. Figure 5 illustrates that the decline in total number of requests continued after EDS was in place. The implementa- tion of EDS, however, did coincide with changes made in January 2012 to the public display of the library catalog. The change to the public display prioritizes locally held material and is partly responsible for patrons requesting consortial items less often. While WCU’s use of the ABC Express service has declined at a comparable rate to its partners, it is interesting to note the borrowing patterns across the schools have shifted to make WCU the net borrower. Historically, the distribution pattern of requests has been in direct relation to collection size, where WCU is in the middle of the pack. The data suggest EDS may promote consortial items sufficiently to ameliorate some of the decline in use caused by the changes to the library catalog. A 30 percent decrease in total requests, however, still prevents the current study from claiming EDS positively affects use of print collections. FIGURE 5 Change in Number of Consortial Items Requested by Institution –30% –42% –27% 0 5,000 10,000 15,000 20,000 25,000 WCU UNCA ASU 2010/11 2011/12 94 College & Research Libraries January 2015 Hypothesis 3: EDS will result in decreased use of interlibrary loan for materials The current study cannot conclude with any confidence whether hypothesis 3 holds. As stated previously, the effect that EDS had on interlibrary loan borrowing was predicated on the data affirming the first two hypotheses. Table 7 shows that the change in ILL borrowing requests is mixed; requests for monographs and media have increased 12 percent, while article requests have decreased 15 percent. The total number of requests per year decreased slightly. The results do align with predicted outcomes when taking the two categories of ILL requests separately. Implementing EDS did increase the use of e-resources, especially the library’s e-journals, which cor- relates with the decline in ILL requests for articles. The data suggest that the more users used local resources, the less they relied on interlibrary loan for articles. On the other hand, non-article ILL requests rose. This rise would appear to prove the existence of an inverse relationship between use of print materials and number of ILL requests. There is insufficient data to rule out an intervening variable to explain the rise in monograph and media requests at this time. Discussion Databases, e-Journals, and Interlibrary Loan With the introduction of EDS, WCU experienced a demonstrable increase in the use of abstracts and A&I databases in contrast to other studies.26 While use alone is an in- complete measure of the value of a resource, for now it is still one of the major factors in renewal decisions. In 2011, WCU underwent an extensive serials review following a cut to the collections budget. In at least one case, an A&I database was cancelled due to overlapping subject coverage with a full-text resource. The decision was partly based on amount of use but also the assumption that students prefer full text. Insofar as the library relies on usage statistics as the primary measure of utility, the 30 percent increase seen in A&I databases may forestall questions of whether indexes should still be a collection development priority. However, Asher et al. found “reading abstracts was the most commonly used method of evaluating resources” for students. The data do not indicate whether students sought out resources after viewing the abstract.27 Further qualitative data from users on the usefulness of abstracts are necessary. EDS did not appear to increase use of EBSCOhost full text to any great extent, es- pecially when considered with the upward trend in use observed in North Carolina. While total full-text retrievals in EBSCOhost databases rose 12 percent in 2012, the growth is below other peer institutions. Increased EBSCOhost database offerings or growth in student populations in 2012 could have resulted in increased referrals. One of the stronger factors to consider is the current shift in student populations of the university. WCU is growing its online and distance education programs, leading to a larger number of students who will undoubtedly be dependent upon e-resources to support them even though the print collection is available to them through other means. The current study did not undertake any analysis to rule out any of these factors as an alternate explanation for the observed results. Perhaps it is not unexpected that EDS would not play a large role in increasing exposure of its native, full-text content, TABLE 7 Average Number of Interlibrary Loan Requests Finished per Semester Pre-EDS Post-EDS Percent Change Article 1,523.00 1,291.67 –15% Books and Media 1,053.00 1,179.33 12% Maximizing Academic Library Collections 95 given how EBSCOhost already shares subscribed full-text links across databases. With such functionality already inherent in the EBSCOhost platform, the benefit of EDS comes from centralizing indexing and standardizing metadata rather than improving the mechanisms involved in the delivery of full text. These conclusions may not be generalizable to other discovery tools where the vendor is not also a content provider or does not have the same background as EBSCO in servicing journal content. Concerns remain about the impact of discovery tools on database content not avail- able on the discovery platform. It is not possible for the current study to assess the effect of EDS on the library’s non-EBSCOhost databases until standardized use data are available across platforms. As Condit Fagan et al. found through usability testing, users incorrectly assume that all library content is available through the discovery interface.28 Such a misunderstanding could adversely affect the users’ exploration and use of other electronic resources. Specialized databases provide discipline-specific search features and metadata, which cannot be offered with the same depth of fo- cus in a unified discovery environment. Without improvements to discovery tools to direct users to other resources, the onus is on the library to instruct users on the limitations of the systems. The findings from the current study on the use of the library’s journal packages concur with results from the studies conducted by Way and Kemp for Summon.29 The data confirmed that the introduction of a discovery tool has a strong positive effect on e-journal use. The corresponding decline in article requests through interlibrary loan was another beneficial outcome of the increased use of the library’s e-journals. Unfortunately, it is difficult to rule out other factors that may affect interlibrary loan use without a thorough review of each transaction. The functionality of the library’s link resolver, for instance, can greatly influence the delivery of resources and, in turn, ILL demand. Activating unmediated book requesting in ILLiad may also have played a role in increasing ILL book transactions by decreasing the time it takes to process those requests. With those caveats, discovery tools do appear to maximize the value of e-journal subscriptions and lower the reliance on interlibrary loan for articles. Satisfying the user’s research needs with local material saves the user time and saves the library the costs associated with interlibrary loan transactions. The semester-to-semester data in figure 2 also suggest another implication for e-journal use not yet addressed in the current or previous studies; e-journal use appears to plateau after an initial upswing. Future studies should continue to track usage for e-resources over a longer period to determine if the rate of growth can be maintained. Books and the Print Collection Many intertwining factors, including the introduction of EDS, have affected patrons’ use of the library catalog at WCU. The resultant circulation data paint a worrying picture for discovery and the library’s book collection. At WCU, checkouts have dropped to the point where WCU is now the university with the lowest circulation numbers in the consortium. In terms of total circulation, WCU had 5 percent fewer checkouts than UNCA despite having a significantly larger student body.30 And, while WCU does purchase a growing number of e-books, they tend to be in support of distance and online programs and less so in other areas of the collection where the approval plan and selectors do the majority of their purchasing. The root cause of the steep decline in checkouts could be due to one or several of the following factors. The first, and sim- plest, explanation is that EDS replaced the library catalog as the main search box on the library’s home page and made the catalog more difficult to access. Whether users are unable to find the new location on the website, or are less exposed to the catalog now, is a question WCU will need to address through usability testing. 96 College & Research Libraries January 2015 Problems with website design do not address why users are not finding and check- ing out books from EDS instead. Certainly, overall circulation at academic libraries has declined as user behavior shifts away from print.31 Circumstances at WCU may additionally depress circulation. Outside of subjects like health care, where material ages out within five years, the monograph collection at WCU has not been weeded to any great extent. As the collection ages, users may be discouraged from using books when the books they browse are outdated or irrelevant. One further possibility is that users are not being instructed in how best to use available features in EDS to target books. At WCU, librarians admit to preferring to use the library catalog to conduct book searches because of the ability to browse subject headings and the perception of retrieving more precise results from known-item searching. Furthermore, the approach WCU took to adopting a discovery tool as a “pilot” where ongoing funding for EDS was not assured dissuaded instruction librarians from immediately teaching EDS to freshmen and undergraduates; instruction librarians were hesitant to teach a new tool if it would not be available in the future. The “pilot” is now in its third year, and the instruction librarians do introduce students to EDS, but the delay of instruction has probably hindered student facility with book searching in EDS to some extent.32 Such factors are specific to this institution, and the results should not be generalized to all discovery tools or all libraries with EDS; but we suggest that future research studies ought to examine how library instruction impacts patron behaviors with discovery tools. Another factor to consider is the limitations of EBSCO’s search algorithm for book searching. In EDS, if users select facets that apply only to article metadata, book content is excluded from the results list without users being aware they have done so. Subject headings are often the cause of this. Library of Congress Subject Headings (LCSH) in catalog records are generally unique from specific subject terms in other thesauri, less likely to be included as author-supplied keywords, and occur less frequently in results list. EDS can only display the top 50 subject headings in the facet panel by hit count. Users attempting to narrow their search using these subject terms may be disadvantag- ing book results because LCSH terms do not occur often enough to make the cut. One solution to the problem would be for EBSCO to work on cross-mapping EBSCOhost thesauri with LCSH for popular subject headings in a manner similar to the existing EBSCO feature where related words are applied to search results.33 Then, if a user se- lected a subject term originating from a journal article, EDS would retain book results with the corresponding LCSHs. Other discovery systems, including Encore Synergy, may treat book and article results separately. It would be advantageous to perform comparison studies against EDS to ascertain whether there is a positive impact on use of books and other print material in other discovery systems. A future area of study will be to also look at the effect of EDS on e-book use. Pres- ently, WCU has 400,000+ e-books in its collection, all of which are included in EDS. It is not clear yet if e-book use would mirror that of print materials or e-journals. The picture may not be uniform across e-book collections. EBSCO e-books may benefit more from EDS through searching within the full text and including records with more robust metadata, while e-books from providers where the only access point in EDS is the MARC record suffer the same limitations as print materials. Conclusion EBSCO Discovery Service has undoubtedly changed user behavior to better connect the patron to the library’s e-journals and to increase use of full text and abstracts in EBSCOhost databases. Unfortunately, the negative outcomes on use of the library’s print collection undermine one of the primary reasons for adopting a discovery tool. Maximizing Academic Library Collections 97 There needs to be an incentive to subscribe to a product that combines book and article content, and improving the integration of the library catalog would automati- cally increase the utility and value of EDS. Much of the benefit to increased e-resource and e-journal use from cross-database searching could be achieved through a custom EBSCOhost search box that included all EBSCO content subscribed to by the library. Many of the added-value features of a discovery tool would be lost (that is to say, data from additional publishers and content providers, as well as a more robust and unified subject index); but the custom search box could provide an adequate solution in the future as library funding comes under additional pressure, given ongoing budgetary strain to the university. The library revised its guidelines for including content in the discovery system after considering the strengths (and weaknesses) of EDS. During implementation, the setup team elected to include as much content as possible to create a comprehensive search experience. The inclination to load as much content as possible into EDS was aban- doned, and the library became much more selective about what resources to include. First and foremost, the library wants to direct users to its own resources within EDS and to deliver highly relevant, scholarly materials to students. Both anecdotal data from public services and the published usability studies indicate too many irrelevant results frustrate users.34 This process began with the removal of several open source datasets, such as the Harvard Library Bibliographic Dataset and Government Publica- tions Office catalog, which returned a large number of results not held by the library. But now, even new library e-resources are scrutinized before they are added to EDS and may be excluded if the types of records do not integrate with EDS well (such as market research data), or would not strongly align with the type of research the typi- cal EDS user would perform. For instance, the library chose not to include recently purchased Gale digital historical collections. In many ways, the library treats the success or failure of the discovery system as a sign that the library is able to keep up with the evolving research needs of the uni- versity. Improvements to technological tools are important for providing a seamless research experience, but that will not solve larger collection development problems. The attention that WCU has placed on growing enrollment with distance and online programs over traditional residential students means that the library must grow its collection in alignment with changing campus demographics. And, while EBSCO Discovery Service helps to facilitate the use of the library’s e-resources, the decline in print circulation may need to be addressed not only through improvements to EDS but also through collection decisions. Until now, librarians have been cautious and placed much higher standards on e-books to prove their usefulness through usage statistics than they do for print books. And while e-book usage was not examined here, the circulation statistics presented suggest that the library should undertake further study and consider adopting a more aggressive shift to e-books for all collection areas. Acknowledgements The author would like to thank Rachel Fleming, Appalachian State University, for her technical assistance with this paper, as well as the following people for their assistance with data collec- tion: Brandy Bourne, UNC Asheville; Leslie Farison, Appalachian State University; Catherine Wilkinson, WNCLN; and Tim Rogers, NC LIVE. Notes 1. Michelle Hendley, “Citation Behavior of Undergraduate Students: A Study of History, Political Science, and Sociology Papers,” Behavioral & Social Sciences Librarian 31, no. 2 (2012): 109. 98 College & Research Libraries January 2015 2. Discovery Tools: The Next Generation of Library Research, eds. John S. Spencer and Christopher Millson-Martula, vol. 19 (2–4), College & Undergraduate Libraries (2012): 121–397. 3. Sarah C. Williams and Anita K. Foster, “Promise Fulfilled? An EBSCO Discovery Service Usability Study,” Journal of Web Librarianship 5, no. 3 (2011): 179–98. 4. Ibid.; Andrew D. Asher, Lynda M. Duke, and Suzanne Wilson, “Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources,” College & Research Libraries 74, no. 5 (2013): 464–88; Abe Crystal, “Final Summon User Research Report,” North Carolina State University Libraries, available online at www.lib.ncsu.edu/userstudies/studies/2010summon [accessed 12 July 2013]. 5. David J. Comeaux, “Usability Testing of a Web-Scale Discovery System at an Academic Library,” College & Undergraduate Libraries 19, no. 2–4 (2012): 189–206; Jody Condit Fagan et al., “Usability Test Results for a Discovery Tool in an Academic Library,” Information Technology & Libraries 31, no. 1 (2012): 83–112; Crystal, “Final Summon User Research Report”; Sue Fahey, Shannon Gordon, and Crystal Rose, “Seeing Double at Memorial University: Two Worldcat Local Usability Studies,” Partnership: The Canadian Journal of Library & Information Practice & Research 6, no. 2 (2011): 1–14; Williams and Foster, “Promise Fulfilled?” 6. Asher, Duke, and Wilson, “Paths of Discovery,” 473; Condit Fagan et al., “Usability Test Results,” 84. 7. Fahey, Gordon, and Rose, “Seeing Double at Memorial University,” 9–10. 8. Crystal, “Final Summon User Research Report.” 9. Asher, Duke, and Wilson, “Paths of Discovery,” 471. 10. Kelly Meadow and James Meadow, “Search Query Quality and Web-Scale Discovery: A Qualitative and Quantitative Analysis,” College & Undergraduate Libraries 19, no. 2–4 (2012): 163–75; Asher, Duke, and Wilson, “Paths of Discovery.” 11. Iris Xie and Dietmar Wolfram, “A Longitudinal Study of Database Usage within a General Audience Digital Library,” JODI: Journal of Digital Information 10, no. 4 (2009): 1–19. 12. Ruth Stubbings, “Metalib and Sfx at Loughborough University Library,” VINE: The Journal of Information & Knowledge Management Systems 33, no. 1 (2003): 25–32. 13. Ruth Stubbings and Yvonne Hamblin, “How to Make Your E-Resources Earn Their Keep,” Serials 17, no. 1 (2004): 25–30. 14. Jan Kemp, “Does Web-Scale Discovery Make a Difference? Changes in Collection Use after Implementing Summon.,” in Planning and Implementing Resource Discovery Tools in Academic Libraries, eds. Mary Pagliero Popp and Diane Dallis (Hershey, Pa.: Information Science Reference, 2012), 456–68; Doug Way, “The Impact of Web-Scale Discovery on the Use of a Library Collection,” Serials Review 36 (2010): 214–20. 15. COUNTER (Counting Online Usage of Networked Electronic Resources) is an initiative to, in part, set standards for credible and reliable e-resource statistics. 16. Kemp, “Does Web-Scale Discovery Make a Difference?”; Way, “The Impact of Web-Scale Discovery,” 217. 17. Kemp, “Does Web-Scale Discovery Make a Difference?” 462. 18. Stubbings, “Metalib and Sfx at Loughborough University Library,” 31. 19. Stubbings and Hamblin, “How to Make Your E-Resources Earn Their Keep,” 28. 20. Kemp, “Does Web-Scale Discovery Make a Difference?” 464–65. 21. William H. Weare Jr., Sue Toms, and Marshall Breeding, “Moving Forward: The Next-Gen Catalog and the New Discovery Tools,” Library Media Connection 30, no. 3 (2011): 54–57. 22. Hendley, “Citation Behavior of Undergraduate Students,” 100–02. 23. Amy Fry and Linda Rich, “Usability Testing for E-Resource Discovery: How Students Find and Choose E-Resources Using Library Web Sites,” Journal of Academic Librarianship 37, no. 5 (2011): 397. 24. All institutions adopted discovery systems following the study period. ASU and UNCA implemented Encore Synergy in January 2013. NC LIVE implemented EDS in August 2013. 25. Unfortunately, at the time of the request, data for all institutions were not available for the full study period (back to January of 2010). 26. Way, “The Impact of Web-Scale Discovery,” 218. 27. Asher, Duke, and Wilson, “Paths of Discovery,” 473–74. 28. Condit Fagan et al., “Usability Test Results,” 100. 29. Kemp, “Does Web-Scale Discovery Make a Difference?” 463–65; Way, “The Impact of Web-Scale Discovery,” 218. 30. In spring 2013, UNCA had 3,400 students enrolled (from http://ierp.unca.edu/ir/enrollment- reporting), compared to WCU with an enrollment of 9,608 students in fall 2012 (from www.wcu. edu/about-wcu/leadership/office-of-the-chancellor/chancellors-division/oipe/index.asp) [accessed 12 July 2013]. 31. Will Kurt, “The End of Academic Library Circulation?,” ACRL TechConnect (blog) (Feb. 1, Maximizing Academic Library Collections 99 2012), available online at http://acrl.ala.org/techconnect/?p=233 [accessed 10 December 2013]. 32. Book searching is not the only aspect of searching in discovery tools where the absence of instruction would have a negative effect on students’ ability to use EDS critically. Issues of functionality as well as the limitations of discovery systems, including what content is being searched by them, would also factor into instructional outcomes. 33. The feature to “apply related words” is available in EBSCOhost and EDS to expand search results to include plurals and known synonyms of the search terms. See the EBSCO help page for more information, available online at http://support.epnet.com/knowledge_base/detail. php?topic=999&id=1278&page=1 [accessed 10 December 2013]. 34. For a discussion of newspaper records in results lists, see Asher, Duke, and Wilson, “Paths of Discovery,” 471–72.