Evidence Based Library and Information Practice

Commentary

Building Evidence of the Value and Impact of Library and Information Services: Methods, Metrics and ROI

Carol Tenopir

Chancellor’s Professor

University of Tennessee

Knoxville, Tennessee, United States of America

2013 Tenopir. This is an Open Access article distributed under the terms of the Creative Commons‐Attribution‐Noncommercial‐Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

Those who work in libraries or have a close understanding of their historical and contemporary roles know that libraries bring value to the individuals and institutions they serve. Yet, when there are decreasing resources and increasing alternatives for information and attention, libraries of all types find that they need to measure and demonstrate the value of all of their collections and services. And, sometimes, measuring value means choosing to eliminate some traditional roles in order to take on or re-emphasize new ones (Tenopir, 2012).

The history of building evidence of the value of libraries is long, particularly for public and special libraries (Griffiths & King, 1993, 2011; Fraser, et al, 2002; Matthews, 2002, 2007). In academic research libraries, the LibQUAL+ initiative provides an international base of comparison for patrons’ expectations and how individual libraries meet those expectations (Association of Research Libraries, 2013). Oakleaf (2010) summarized years of work on value of academic libraries. She emphasized the need for academic libraries to demonstrate their value and offered multiple suggestions for how to do so, including surrogate measurements for value and impact. As important as demonstrating value, however, is being valuable: the line of inquiry about how valuable libraries are and how to increase their value derives from inquiring about the demonstration of value (Oakleaf, 2010).

Yet, much remains to be done in building a culture of measuring value in the specific context of each individual academic library. The LibValue project, sponsored by the U.S. Institute of Museum and Library Services, is a multi-year project focusing on testing multiple methods for measuring the value, outcomes, and return on investment (ROI) of academic library products and services (LibValue.cci.utk.edu, 2013). Principals in LibValue include the University of Tennessee; University of Illinois; Syracuse University; Byrant University; and the Association of Research Libraries. Other participants have included, in the UK, JISC Collections; Cranfield University; University of Dundee; University of East Anglia; Durham University; Imperial College London; and University of Manchester. In the US, participants include Seton Hall University; the University of North Carolina-Wilmington; the University of Colorado; Baruch College; Buffalo State University; and Brooklyn College. In Australia, participants include the John Metcalfe Memorial Foundation; the University of New South Wales; and University of Queensland.

The value of an academic library is complex, because total value is composed of many separate values for each type of collection or service and because the value differs for different constituents and over time. The value of the entire library may be a composite of dozens (or even hundreds) of separate services and collections, each of which has a different value. Value to first year undergraduates may be the library’s role in encouraging them to continue in school (thus factoring into retention rates), which is enhanced by having a welcoming physical space with friendly assistance. By the time those students are seniors, the value of the library may be more in helping them find high quality resources in a timely manner to improve their research papers. For a faculty member, the value of instructional services may be to help them improve the courses they teach and, at the same time, help their students do better in class assignments. For a graduate student or a faculty member’s role as researcher, the value of the library may be to get access to the widest range of resources or the use of an institutional repository as a place to deposit their research datasets for long-term preservation and shared access. Value in all of these cases, however, revolves around outcomes – how the library contributes to the academic work and success of the people it serves.

In a broad sense, library and information resources value can be considered as either “exchange” value or “use” value (Machlup, 1979). In the information context, exchange (or “purchase”) value is what one is willing to pay for information in money or time, and use value is the favorable consequences derived from using the information. Use value can also be looked at as outcomes from using information products or services. Bruce Kingma of Syracuse University, principal in the LibValue project, describes three different types of library value, including: 1) Economic (private) value, e.g., what is the value to an individual to use the library resources?; 2) Social (public) value, e.g., what is the value to the institution of the library?; and 3) Environmental (externality) value, e.g., what is the value of the environmental savings of library provision of electronic resources?

Another way to categorize types of value measurements in libraries is to consolidate these measures into: 1) measures of “implied” value, such as the value that is measured from usage statistics (people download the library e-resources, therefore value of these collections is implied); 2) “explicit” measures of value that come directly from testimonials (for example, how access to a special collection helped an author complete a book) or value that constituents ascribe to the library and describe in interviews, focus groups, or questionnaires; and 3) “derived” measures such as ROI or contingent valuation measures that are calculated on a combination of implied, explicit and other sources of evidence (Tenopir, 2012). Some examples of types of value and measurements are given below.

Exchange value for a journals collection can be calculated using questionnaires to gather information on the amount of reading by academic staff and faculty members (or students), how much time they spend on average per reading, and extrapolating up to an annual basis and to the total university faculty population. For example, in six universities in the UK, faculty members report reading on average per month, 22 scholarly articles at 49 minutes per reading, 7 books or parts of books at 106 minutes per reading, and 10 other scholarly publications at 42 minutes per reading. This adds up to a commitment of 37 hours per month just on reading (not including the time spent identifying and obtaining the readings) (Tenopir & Volentine, 2012). The percent of that time that comes on readings from the library is one way to measure exchange value; in the case of the UK, respondents reported that 67.3% of article readings, 27.6% of book readings, and 15.2% of other publication readings come from the library. That means they spend over 16 hours per month or an estimated 192 hours per year just in reading items from their libraries. Bringing in average faculty salaries and extrapolating to the total faculty numbers, and adding in time identifying and locating the readings, completes this exchange value calculation (King, 2012).

Another value that can be derived from this type of study is contingent valuation (what faculty or students would expect or be willing to pay if the library collections or services were not available to them). At Bryant University in the US, LibValue principal Donald W. King conducted an in depth cost and value study of multiple library services. For journal collections he estimated that at the average salary of Bryant University faculty members, the cost of not having access to library-provided articles would carry a total cost of nearly a half million dollars or a cost per faculty member of $1,200 or cost per reading of $27 (King, 2012).

Return on investment (ROI) in a strict sense is a quantitative measure expressed as a ratio of the value returned to the institution for each monetary unit invested in the library. In other words, for every amount of money spent on the library, the university receives ‘X’ amount in return. ROI can be used with contingent valuation or it can be used to demonstrate that library collections and services contribute to income-generating activities.

In a study funded by Elsevier, the ROI of research university libraries in eight countries in helping the research grants process succeed, through access to high quality materials to cite, was estimated to be between just under 1:1 for a humanities/social sciences based teaching institution to over 15:1 for a scientific research university system. Most comprehensive research and teaching universities had a library ROI in grants in the range of 3-5:1 (Tenopir, 2010).

The National Network of Libraries of Medicine has an ROI calculator that helps libraries demonstrate: how much benefit does the institution, your user, receive for every dollar spent by the library?; value of benefits and costs for each service; and total value of library use (2013). Simple public library ROI calculators are also available (see, for example, http://www.lrs.org/data-tools/public-libraries/return-on-investment/ and http://www2.library.lapeer.org/library-return-on-investment-calculator.html).

In a softer sense, ROI is also values of all types and outcomes that come to stakeholders and the institution from use of the library’s collections, services, and contribution to its communities. Use or outcomes value can be gathered explicitly in a survey or interview by asking respondents to focus on a critical incident of information use. In the example of scholarly reading, we ask respondents to focus on the last scholarly article they read (whether or not it was particularly typical or important), thus giving us a second stage random sample of readings in addition to the first stage sample of readers. In the study in the UK, faculty identified many outcomes from article reading, including in rank order: 1) inspire new thinking or ideas; 2) improve the results of work; 3) narrow/broaden/or change the focus of the principal purpose; 4) resolve technical problems; 5) save time or other resources. Negative outcomes were offered as choices, but were seldom selected.

Open-ended questions are another source of explicit outcomes to support decision making (Volentine & Tenopir, 2013). Comments include simple praise or complaints about the level of service provided or can offer hints on how to improve service. For example, from one U.K. academic: “Electronic access to the university library system from off-site is crucial for swift access to articles to support my teaching and research activities.” Another (this time from our studies in Australia) offers: “I prefer reading hard copy to online books and articles. I particularly dislike ebooks and articles which I’m only able to read for a limited time.” Comments can be used to gather personal evidence of the value of the library’s collections and services; they can also lead to specific courses of action such as negotiating contracts that include off-site access and better e-book lending policies.

Putting together quantitative data and open-ended comments in surveys allows libraries to put a personal face on their value to constituents by building data-based personas (Volentine, et al 2012). Personas are fictional characters based on actual respondent data. For example, in the UK, segmenting the data to select the most successful academics (defined as those who have won an award in the last two years and published four or more items), builds a portrait of a successful academic, who: reads more of every type of material; spends more time per book and other publication readings; uses the library for articles; more often buys books and obtains other publications from the Internet; occasionally participates and creates social media content. Another persona in the U.K. study is an Associate professor in physics who reads 30 articles, 2 books, and 11 other publications per month and has not visited a physical library in many years. He needs current issues of articles, off-site access to collections, access to search engines and online resources without a distinction between library and non-library resources, and a wide range of dates to see trends over time.

It should be clear that value of the library to its constituents can be demonstrated in many ways – by time invested, by value to purpose, by outcomes of use, and by ROI. Multiple methods should be used to measure value, including quantitative, qualitative, and a mixture of both. No one method stands alone and the choice of methods must be tied to the mission of each specific institution. Quantitative data can show ROI and trends, while qualitative data can tell a story or put a personal face on data. Whatever methods are used, libraries need to focus on measuring outcomes, not inputs, and use this evidence to demonstrate the role of the library in helping with the success of faculty, graduate students, and undergraduate students.

This commentary is based on a presentation for the LIS DREaM Project Conference, British Library, 9th July 2012, UK.

Acknowledgements: The studies reported here involve many librarians around the world, who are too numerous to mention specifically, but who collectively have made these projects successful. Special thanks to Paula Kaufmann, Dean of Libraries, University of Illinois-Urbana-Champaign; Bruce Kingman, Professor at Syracuse University; and Donald W. King, Research Professor at Bryant University and all of the LibValue researchers. LibValue is funded by a grant from the U.S. Institute of Museum and Library Services. Additional funding for international participation comes from, in the U.K. JISC Collections and, in Australia, the John Metcalfe Memorial Foundation. An international study on ROI in grants was funded by Elsevier.

References

Association of Research Libraries (2013). LibQual+: Charting library service quality. Retrieved from https://www.libqual.org/home

Fraser, B.T., McClure, C.R. & Leahy, E.H. (2002). Toward a framework for assessing library and institutional outcomes. portal: Libraries and the Academy, 2(4), p. 505-528.

Griffiths, J.M. & King, D.W. (1993). Increasing the information edge. Special Libraries Research Series. Washington, DC: Special Libraries Association.

Griffiths, J.M. & King, D.W. (2011). A strong future for public library use and employment. Chicago: American Library Association.

King, D.W. (29 October 2012). Assessment of the use, value, and ROI of academic library services. Paper presented at the ARL Library Assessment Conference, Charlottesville, VA.

Libvalue.cci.utk.edu (2013). LibValue: Value, outcomes, and return on investment of academic libraries. Retrieved from http://libvalue.cci.utk.edu

Machlup, F. (1979). Uses, value, and benefits of knowledge. Science Communication, 1(1), p. 62-81.

Matthews, J.R. (2002). The bottom line: Determining and communicating the value of the special library. Westport, CT: Libraries Unlimited.

Matthews, J.R. (2007). The evaluation and measurement of library services. Westport, CT: Libraries Unlimited.

National Network of Libraries of Medicine (2013). Library value calculator: What is your library worth? Retrieved from http://nnlm.gov/mcr/evaluation/roi.html

Oakleaf, M. (2010). The value of academic libraries: A comprehensive research review and report. Association of College and Research Libraries, Chicago, IL.

Tenopir, C. (2010). University investment in the library, phase II: An international study of the library’s value to the grants process. Library Connect White Papers. Retrieved from http://libraryconnect.elsevier.com/university-investment-library-phase-ii-international-study-librarys-value-grants-process-2010

Tenopir, C. (9 July 2012). Building evidence of the value and impact of library information services: Methods, metrics and ROI. Keynote presentation at the DREaM Project Conference, London, UK.

Tenopir, C. (2012) Beyond usage: Measuring library outcomes and value. Library Management, 33(1/2), p. 5-13. DOI: 10.1108/01435121211203275.

Tenopir, C. & Volentine, R. (1 February 2012). UK scholarly reading and the value of library resources: Summary results of the study conducted Spring 2011. JISC Report. http://www.jisc-collections.ac.uk/Reports/ukscholarlyreadingreport/

Volentine, R. & Tenopir, C. (2013). “Value of Academic Reading and Value of the Library in Academics’ Own Words”. Aslib Proceedings. 65 (4).

Volentine, R., Whitson, L., & Tenopir, C. (2012). “Portraits of Success: Building Personas from Scholarly Reading Patterns”. QQML 4th annual conference proceedings.