424 A Multilevel Approach for Library Value Assessment David Schwieder and Lisa Janicke Hinchliffe* As the volume of academic library value research has continued to in- crease, the resulting literature has grown complex and sprawling. This article takes stock of this body of work, using a review of the published research literature and an analysis of its organization and structure. This investigation reveals that the research on library value follows a “levels of analysis” framework, with three types of studies—small group, single institution, and multi-institution—comprising the major dimensions. We discuss the strengths and weaknesses of studies at each level and pro- pose a multilevel framework designed to improve the practical utility and scholarly rigor of academic library value assessment. Growing awareness of the need to demonstrate academic library value received formal recognition with the 2010 release of the ACRL Value of Academic Libraries report,1 and it has inspired a large scholarly literature. Many studies have examined the library’s impact on student and institutional success, applying divergent methods across numer- ous colleges and universities. The resulting body of work is richly informative, but so complex and sprawling that it can be hard to get a handle on. This presents a serious challenge. Most library value studies function not only as research exercises, they also serve practical purposes. They can document library contributions, thus demonstrating value to key institutional stakeholders, and inform matters like budget expenditures, the allocation of spaces, and creation of new posi- tions and programs. Accordingly, making sense of this literature has both scholarly and applied importance. This paper provides a framework to clarify and advance work in this area. Using a review of the academic library value literature, we show how this work is organized by a “levels of analysis” structure. This concept, drawn from several social science disciplines, posits that most social phenomena can be analyzed at multiple levels of aggregation. In ascending order, these library value levels involve small-group stud- ies, which examine library impacts on small, typically class-sized groups of students; single-institution studies, which examine larger student groups at a single college or university; and multi-institution studies, which analyze how libraries impact students across multiple academic institutions. * David Schwieder is Political Science Liaison and Coordinator of Humanities and Social Science Data Services in the George A. Smathers Libraries at the University of Florida; e-mail: dschwieder@uflib.ufl. edu. Lisa Janicke Hinchliffe is Professor/Coordinator for Information Literacy Services and Instruction in the University Library at the University of Illinois at Urbana-Champaign; e-mail: ljanicke@illinois.edu. ©2018 David Schwieder and Lisa Janicke Hinchliffe, Attribution-NonCommercial (http://creativecommons. org/licenses/by-nc/4.0/) CC BY-NC. doi:10.5860/crl.79.3.424 mailto:dschwieder@uflib.ufl.edu mailto:dschwieder@uflib.ufl.edu mailto:ljanicke@illinois.edu http://creativecommons.org/licenses/by-nc/4.0/ http://creativecommons.org/licenses/by-nc/4.0/ https://doi.org/10.5860/crl.79.3.424 A Multilevel Approach for Library Value Assessment 425 We then build upon this review by showing that these three approaches can be combined into a “multilevel” research and assessment program. Each level has offset- ting strengths and weaknesses; thus, combining them can promote the broadly based, systematic study and documentation of library value. We conclude by integrating all three approaches into a coordinated framework for academic library value assessment. The Structure of the Academic Library Value Literature By definition, library value studies typically seek to show that library usage or library programs provide benefits to users or contribute to key goals of the library’s parent institutions. Most commonly, these impacts include promoting user information literacy, advancing student learning in subject fields, and increasing student graduation and retention rates. Other studies, typically conducted at large research universities, have focused on impacts like supporting faculty research and contributing to institutional imperatives like the securing of grants. The resulting body of scholarship is large and diverse, and it has produced a great deal of knowledge about library value. However, little attempt has been made to outline or distinguish between various approaches and methods or to consider this body of work as a whole. Literature reviews in most library value studies cite a small selection of previous work without taking a broader perspective, and even the Value of Academic Libraries report does not offer a means to fully conceptualize and analyze the literature in this field. Accordingly, it would be useful to have a general conceptual framework for library value studies. Such a framework could help to reduce the seeming complexity of this literature and more clearly reveal the elements (in other words, categories) within it. Categorization is key; it plays a central role in human cognition, serving to organize and simplify the world and various kinds of phenomena within it.2 The benefits of effective categorization are quite familiar to librarians—library collections could not be readily organized or understood without the Dewey and Library of Congress clas- sification systems. Levels of analysis represents an obvious choice for a categorization scheme. This concept, well established in several social sciences, posits that most social phenomena can be analyzed at multiple levels, ranging from smaller to larger units of aggregation.3 For example, psychologists might view individual behavior as resulting from individual factors like a person’s motivations or emotions. This seems obvious and natural, but it is not the only level that can be employed. Behavior can also be influenced by lower- level biological factors, such as chemical influences on the brain. In other words, an individual’s actions can be explained through the effects of his or her constituent parts. Alternatively, individual behaviors can be shaped by factors at a higher level of ag- gregation (for example, by the dynamics of membership in a larger group). Research at all of these analytical levels can help to illuminate human behavior.4 A levels of analysis framework can usefully characterize the academic library value literature. This is true for a fundamental reason: because this literature has implicitly developed along a levels of analysis structure. In the course of this development, most academic library value studies have taken one of three basic approaches: small-group studies, which examine library impacts on small groups of students, usually from a single course; single-institution studies, which examine larger student groups at a single college or university, such as a year-class, like sophomores, or a particular eth- nic group; and multi-institution studies, which analyze how libraries impact students across multiple academic institutions. In the following sections, we outline the nature of these three levels of analysis and briefly review the literature in each. 426 College & Research Libraries April 2018 Small Group Studies Our literature review shows that the most popular approach to academic library value studies involves small group, or “microlevel,” studies.5 These examine some sort of library service, typically provided to students—through one-shot instructional sessions, provision of LibGuides or online modules, or some other form of student-librarian interaction—and they seek to show that this produces learning via gains in student knowledge or skills. With their emphasis on learning, most of these studies take place within the context of a particular subject course (although exceptions do exist—most notably, for infor- mation literacy skills). The research designs vary in rigor and complexity, ranging from simple before-and-after comparisons to formal, controlled experiments. Student gains or improvements are commonly assessed by surveys, tests, self-reports, citation analyses, or rubrics measuring student paper or project quality. The large volume of small-group studies makes it difficult to succinctly sum- marize the work in this area, but major emphases can be illustrated. Most studies have examined student success, using on-campus undergraduates for their study population. Some studies examine specific groups, such as first-year students6 or students from low socioeconomic status backgrounds.7 Other studies have broadened the literature by examining other groups, including graduate students and students enrolled in online programs.8 A smaller number have examined more specialized populations, like medical students9 or physicians and medical researchers.10 To understand these small-group studies, it is useful to focus on the library factors involved and their impacts upon students and parent academic institutions. Many of these studies have compared the effectiveness of different instruction modalities or forms of librarian involvement. The most common thrust has compared different in- structional formats (such as in person vs. online),11 or examined the efficacy of blended instruction programs that combine both approaches.12 Other studies have compared the effectiveness of different online presentation modalities.13 Still others have compared traditional one-shot sessions with more persistent modes of course-based library- student interaction, such as embedding.14 Finally, the instructional efficacy of learning objects has also been examined.15 Dependent variables, or impacts, have taken several forms. Some studies have examined user self-perceptions, seeking to show that library experiences have led users to perceive improvements in their learning-related skills and information self- efficacy. However, most studies have employed more direct, and valid, measures of actual learning and skills. The formulation and application of formal rubrics to assess student work has been a topic of particular interest. Given that many librarians have held that rubrics represent a particularly valid and reliable way to assess student work—and thus also student learning16—rubric-based studies represent some of the most thorough work in this area.17 A limited number of small-group studies have deviated from this general pattern, examining more novel types of library impacts (for example, by assessing whether library experiences impact occupational success). One study, taking advantage of a university philosophy that encouraged undergraduate students to secure co-op jobs, investigated whether library contact was associated with higher self-perceptions of search skill adequacy when looking for a co-op position.18 Another study examined graduate students and found that MLS students who participated in a paid, library- based internship found early professional employment at higher rates than peers who had not secured such an internship.19 A Multilevel Approach for Library Value Assessment 427 Single Institution Studies A second type of study takes a broader focus, examining larger student groups at a single academic institution and seeing how they are affected by engagement with library resources or services. These studies typically focus on a subset of undergradu- ates: specific demographic groups or an entire year-class of students (such as first-year students). Statistically analyzing hundreds or thousands of individual cases, single institution studies seek to demonstrate correlations between individual students’ library experiences and factors like grade point average (GPA) or retention and graduation rates. (While both small-group studies and these broader studies take place at a single college or university, this paper uses the term “single institution studies” to refer to the broader studies rather than the small-group studies that typically focus on students enrolled in a particular class.) Single-institution studies show a great deal of homogeneity in their analytical meth- ods, relying on statistical correlation measures, which range from simple comparisons of percentages to bivariate correlations to various regression procedures. They vary in their choice of research subjects (in other words, the particular group of students examined) and in the specific library factors that are employed as independent variables (usually some combination of library materials usage, catalog or workstation logins, reference services, or participation in instructional sessions). Conversely, there is little variation in the broader institutional impacts that constitute the dependent variables; overwhelmingly, these single-institution studies have focused on undergraduates’ year-to-year retention rates, graduation rates, and GPA.20 Several well-done studies convey the general approach used in this kind of work. In a representative example (though a methodologically advanced one, with its inclu- sion of numerous control variables and its reliance on multivariate linear and logistic regression procedures), Soria, Fransen, and Nackerud assembled a wide range of library usage and engagement measures (e-resource and website logins, ILL transactions, loans, workstation logins, chat reference usage, and participation in library workshops, course-integrated instruction sessions, or peer resource consultations), along with several categories of demographic control variables, and found positive correlations between several library variables and first semester GPA and first- to second-semester retention for more than 5,300 first-year students at a large Midwestern university.21 Largely similar results were obtained in a follow-up study.22 Similarly, Stemmer and Mahan used a logistic regression analysis to examine how student use of library resources, services, and spaces impacted their rates of first- to second-year retention, graduation, and several measures of GPA. Tracking the impacts of these library usage habits for several hundred students across all four years of the undergraduate experience, they found that the early impact of “library as place” tends to fade and that library resources and services become more consequential in the junior and senior years. This study was noteworthy not only for its longitudinal design but also because it controlled for a wide variety of nonlibrary factors, including students’ ACT scores, high school GPA, several demographic variables, first-generation status, and estimated family financial contribution.23 Multi-Institution Studies A smaller number of studies have taken a multi-institution approach. Like the single- institution studies, these seek to show correlations between library activities and desirable college or university outcomes. Using large data sets from organizations like the Association of Research Libraries (ARL) and the Association of College and Research Libraries (ACRL), and surveys like the Academic Libraries Study (ALS) and the Integrated Postsecondary Education Data System (IPEDS), these studies typi- 428 College & Research Libraries April 2018 cally employ institutional characteristics of libraries as their independent or causal variables. They seek to show that these factors have a positive impact on important outcomes for their parent colleges and universities. Most commonly, these college and university impacts, or dependent variables, have involved undergraduate retention and graduation rates. Library—that is, independent—variables differ across these studies. One study examining more than 500 U.S. colleges and universities used ARL, ACRL, and IPEDS data to examine bivariate correlations (in other words, correlations between one independent and one dependent variable) between library staff size, five types of library expenditures, and undergraduate year-to-year retention rates. Using the pre-2005 Carnegie Classification System, the analysis found fairly strong relation- ships between library staff size and retention at baccalaureate and doctoral-granting institutions. Moderately strong relationships were observed between total library expenditures, total library materials costs, and serials costs, but only at baccalaure- ate colleges and universities.24 A more recent study used ALS and IPEDS data to assess bivariate correlations for undergraduates at more than 1,300 four-year colleges and universities. Independent variables included library expenses, several types of parent institution expenses, and library usage (measured by circulation, interlibrary loan, gate count, instructional session attendance, and reference transactions), with dependent variables including year-to-year retention rates and four-, five-, and six-year graduation rates. Among all variables, library expenditures had the strongest overall correlation with retention and graduation, with per-FTE usage having a more moderate association.25 Controlling for a variety of spending and institutional factors, one fiscally focused study found that total library expenditures per student were associated with higher undergraduate graduation rates at more than 400 four-year, public American colleges and universities.26 The study relied primarily on IPEDS data, but it also used data retrieved from U.S. News & World Report college selectivity rankings and employed a multivariate regression methodology. In an interesting variation, Mezick (2015) used IPEDS and ARL SPEC survey data for 65 U.S. research universities to investigate how a commitment to library assess- ment programs—as measured by assessment training for library staff, adoption of librarywide assessment plans, duration of assessment activities, use of student learn- ing outcomes evaluations as an assessment tool, and public accessibility to assessment data and analysis—correlated with undergraduate year-to-year retention rates. None of the relationships analyzed were statistically significant.27 While studies focusing on retention and graduation have been most common, a few authors have analyzed library effects on other institutional priorities. A U.K. study examined library impacts on undergraduate GPA.28 In a methodologically sophisti- cated study, employing linear and logistic regression and a cumulative logit model, Womack used data from ARL, IPEDS, and the NSF’s National Center for Science and Engineering Statistics (NCSES) to evaluate library impacts on grant funding received by 99 ARL-affiliated universities.29 Another study analyzed the relationship between academic libraries and faculty research productivity at 103 ARL-affiliated universi- ties; no effects were found.30 At least one study examined academic library impacts on undergraduate students’ development of critical thinking skills. Kuh and Gonyea used the College Student Experiences Questionnaire (CSEQ) to examine library ef- fects on two types of self-reported student improvements: gains on an information literacy scale and overall gains in college. None of the library experiences correlated with either type of student improvement.31 A Multilevel Approach for Library Value Assessment 429 Library Value Studies and Levels of Analysis These three categories of library value studies illustrate the concept of “levels of analy- sis.” While there is no inherent or “correct” number of levels for studying any topic or phenomenon, in practice, analysts have often settled on three. These can be labeled micro, middle, and macro.32 As is evident from the analysis already presented, this model maps well to the library value literature. Small-group studies are conducted at the micro level, usually showing how library services impact students in a single course. Conversely, multi-institution analysis, which evaluates library impacts across many academic institutions, represents a broad, macro-level approach. Single institu- tion studies then occupy the middle level. The Multilevel Study of Academic Library Value The levels of analysis concept not only serves as a useful framework for understanding the library value literature, it can also help to advance scholarship on library value. A research or assessment program that combines all three types of studies—small group, single institution, and multi-institution—into a coordinated, multilevel approach can provide a coherent and convincing approach for demonstrating academic library value. This multilevel framework draws conceptually on a more familiar approach: multi- method research. Multimethod research employs two or more research methodologies and most commonly seeks to “triangulate” results—that is, to show that divergent methods point toward similar findings. If these different methods produce similar results, the researchers can usually be more confident in their conclusions.33 Similarly, a multilevel approach can employ studies at more than one level of analysis. If one can demonstrate library value with some combination of small-group, single-institution, and multi-institution studies—that is, at multiple levels—then this can make a strong case for library contributions and value. The multilevel approach borrows a key element from the multimethod approach. Multimethod approaches are stronger when their individual methodologies are complementary (that is, when these methods can compensate for one anothers’ weak- nesses).34 Perhaps the classic example involves combining qualitative and quantitative approaches. Qualitative approaches can provide rich, detailed findings, but their results usually are not readily generalizable. Quantitative work typically includes few details but can offer more general conclusions. Thus, the two methods complement each other well. Similarly, our multilevel approach strengthens the design of library value studies by drawing on the complementary nature of the different levels. To illustrate this, we build on our earlier reviews of these three approaches by considering the strengths and weaknesses of each. Small-Group Studies Small-group studies, which typically provide library instruction or services to students and then evaluate whether such interventions are associated with gains in skills or knowledge, can offer two main virtues. First, student learning plays a key part in the college and university environment. It comprises the central focus at smaller, teaching- oriented institutions and is one of the central concerns in larger, more research-oriented colleges and universities. Accordingly, by demonstrating facilitation of learning, librar- ies contribute to a core aspect of the institutional mission. Second, if they employ well-designed experimental methodologies, small-group studies can permit strong inferences about causal impacts. If students who receive assistance show gains in skills or knowledge, with other factors controlled, one can conclude that the library instruction or other library stimulus was responsible. More- over, this strong internal validity can be accompanied by a strong external validity as 430 College & Research Libraries April 2018 well. In contrast to psychology experiments, which commonly involve highly unusual conditions—presenting an unfamiliar stimulus in an artificial laboratory setting, thus raising concerns about whether findings will actually apply to the “real world”—li- brary interventions can be conducted in the student’s actual classroom setting, using authentic instructional materials, and assessing learning via evaluation of actual student academic work. This fortuitous situation increases confidence in the ultimate validity and reliability of small-group results. Of course, small-group studies also have weaknesses. First, the emphasis on learn- ing may be something of a double-edged sword. Colleges and universities are largely in the business of learning, in one form or another, and thus libraries may not stand out or offer unique value in this area, although learning of information literacy skills represents an exception. Second, small-group studies may be too “micro” to make a strong impression on key institutional stakeholders. College and university administrators, perhaps the most important audience for library value studies, take a broad view and presumably are more impressed by institution-scale impacts and payoffs. Moreover, the small-group studies literature is highly scattered, with little replication or coherence; as the 2010 Value report noted, “most of the published evidence on the impact of libraries on student learning is sporadic, disconnected, and focused on limited case studies.”35 Single-Institution Studies While small group studies are defined by their micro focus, single-institution studies are distinguished by a wider breadth and inclusion of larger numbers of cases—typically all or part of a major cohort or demographic group like first-year students. As noted earlier, single-institution studies typically examine undergraduates and seek to dem- onstrate correlations between library services and desirable institution-level outcomes. Single-institution studies have several attractive features. Most notably, they typi- cally demonstrate library value at the overall college or university level—as noted, they often define impacts in terms of graduation and undergraduate retention rates. Factors like retention and graduation are important to accreditation and university prestige and rankings. These measures speak a broad institutional language, and thus they can offer what the ACRL’s Value of Academic Libraries report calls “the broader, more coherent demonstrations of value that librarians need to articulate the importance of information literacy learning in an institutional context.”36 Availability of data tends to be another strength. Most single-institution studies rely primarily on “transactional” data for their independent and dependent variables: library usage measures, and institutional measures of GPA, retention, and graduation, respectively. These are generated in the course of normal library and parent institu- tion operations, which avoid some of the problems that can be encountered when collecting data with user surveys: time and expense, low response rates, and a limited number of suitable variables. Once privacy concerns are satisfied, and data security safeguards are in place, many kinds of transactional data can be accessed at many or most colleges and universities. Of course, these studies have their downside as well. Single-institution studies have difficulty analyzing the effects of institutional characteristics of libraries. These characteristics usually involve “input” factors like collection size or expenditures, col- lected via institutional surveys sponsored by the ARL, ACRL, or other organizations. Since there will be little variance in many or most of these independent variables at a single school—for example, library hours and collection sizes will not change much over the short to medium term—these factors will effectively be a constant; thus, they cannot be correlated with library value measures like student success or performance. A Multilevel Approach for Library Value Assessment 431 Accordingly, single-institution studies are typically unable to show that institutional characteristics have an impact. Another acknowledged weakness involves the difficulty of determining the direc- tion of causation.37 For example, a finding that student library usage correlates with student GPA could be interpreted either way; that is, it could suggest that higher library usage is leading to better grades, or, alternatively, that students who get better grades are using the library more often. Mediating factors—which can produce spurious cor- relations—may also be present. Multi-Institution Studies Like single institution studies, multi-institution analysis demonstrates library value at the overall college or university level. However, multi-institution studies can rig- orously assess the effects of institutional library characteristics and show how these library characteristics impact library value outcomes. Institutional characteristics like expenditures, collection sizes, and number of open hours will vary across libraries; thus, they can be correlated with impact variables. Perhaps the most significant weakness of this method involves data limitations. Multi-institutional statistical analysis requires one or more data sets offering a suf- ficient—and preferably extensive—set of dependent, independent, and control vari- ables. Unfortunately, these can be difficult to find, as evidenced by an examination of several key data sets. For variables measuring the impacts libraries have on a college or university level, the Integrated Postsecondary Education Data System (IPEDS) is perhaps the leading choice, as its heavy usage in the literature suggests. However, a paper that canvassed a recent version of the IPEDS (2008) found only two suitable impact variables: graduation and retention rates.38 As for library factors, the widely used Academic Libraries Survey also enjoys attractive qualities: broad participation by colleges and universities and, at the time the studies in our literature review were undertaken, the broadest range of library related variables. Since 2014, however, the survey has been radically circumscribed; numerous variables were eliminated, and the ALS now survives only as a component of the IPEDS, with a small fraction of the original variable set.39 A second fundamental limitation derives from the basic nature of multi-institutional work; since these studies encompass a number of colleges and universities, the results rarely enable scholars to draw conclusions about individual cases. Most correlational results represent averages calculated across most or all cases in a study, and it is often unclear how these averages apply to any particular library. Obviously, this can be problematic. Practicing librarians as well as college and university administrators must care most about their own library and institution. While there are ways to partially ameliorate this difficulty (for instance, by selecting a small set of peer institutions for comparison), the inability to obtain precise results for one’s own institution will rep- resent an issue for librarians seeking to demonstrate library value. The Complementarity of Multilevel Library Value Studies Having outlined the strengths and weaknesses of each research approach, we can more explicitly address our question: can these three methodologies be productively combined into a broader, multilevel library value assessment framework? Collectively—in the spirit of multimethod research designs—can they compensate for each other’s weak- nesses? We consider this from the perspective of each of these three types of approaches. Although small-group learning studies cannot document value at the crucial insti- tutional level, the other two approaches are well positioned to compensate. Both the single-institution studies and the multi-institution studies address value at this overall 432 College & Research Libraries April 2018 college or university level. Moreover, one can argue that learning forms the foundation for library benefits seen at these higher levels; better learning of information skills or subject material results in better grades, and ultimately better grades result in higher retention and graduation rates. Thus, these convincing institutional impacts ultimately reflect the impact shown in the small-group studies. While single-institution studies can have difficulty establishing the direction of causation or impact—that is, showing that library usage positively impacts academic success, rather than that already successful students more frequently use the library— the other two methods can offer useful supporting context. Since most small-group studies involve some library treatment or intervention that impacts subsequent learn- ing or skill development, they face no ambiguity about causal direction. The same point holds for some—though not all—multi-institution studies. While some multi- institution studies are open to alternative interpretations of causation—for example, a finding that library expenditures are correlated with graduation rates might actually reflect the effect of a third variable, such as college or university status—the dependent variables most commonly employed, student GPA and graduation and retention rates, will tend to guard against reverse causation, since they will not plausibly drive library policies. Thus, even if the direction of impact is uncertain for single-institution studies, confidence that libraries have an impact at this level is bolstered when the other two approaches can show that library contributions clearly do affect student academic suc- cess. Furthermore, if single-institution studies have difficulty analyzing the impact of institutional library factors like library programs, spending, and policies, then multi- institution studies can fill this gap. Conversely, if multi-institution studies have difficulty demonstrating library impacts for an individual library, the other two methods can do so; obviously, they are conducted within a single institution. If the institutional surveys used for multi-institution studies present difficulties with data availability, the other methods are affected less by this limitation. A single-institutional study can draw on transactional data from its library and parent institution, and small-group studies typically generate their own data in the course of their research processes. Accordingly, these divergent methods can work well as a set. And, importantly, they do more than just offset one another’s deficiencies; they combine to comprise an ordered structure that can provide a growing, building sense of academic library value. A Multilevel Framework for Demonstrating Academic Library Value In the previous section, we argued that the main three types (or levels) of value studies can be productively combined into a single assessment program. Here we outline such a program, in the form of a multilevel assessment framework. This framework reflects the library impacts evaluated in the studies covered in our literature review. For the main focus—students and their success—these impacts comprise a sequence. This sequence begins with course-based learning, continues with higher GPA (presumably at least partially the result of such learning), follows with retention (presumably resulting, in part, from learning and good grades), and ends with graduation. Accordingly, such an assessment program will document the library’s contributions to students at key points in their academic careers. Such a program will also employ a mix of methods. Learning impacts tend to be assessed with small-group studies, as we have noted, while the other stages use some combination of single- or multi-institution studies. As noted earlier, these methods combine in a complementary manner—characteristic of multimethod research—with the strengths of some methods offsetting weaknesses in others. For example, small- group experimental studies that rigorously document causal “micro” learning impacts A Multilevel Approach for Library Value Assessment 433 from one-shot instructional sessions can be combined with single- or multi-institution statistical studies, which can demonstrate broader associations between libraries and higher student GPA, as well as higher rates of graduation and retention. By specifying methods, this framework provides a guide for conducting research studies at each particular step. More broadly, it also provides a structure for an overall assessment program and shows how studies at multiple levels of analysis can be com- bined. It is important to note that not every library assessment program will necessarily include all of these elements. But, since there are a variety of elements to choose from, any given academic library can use this framework to create a tailored program that includes multiple levels and diverse methodologies. In addition to illustrating a levels of analysis structure for library value work, the framework can also offer two other contributions. First, its sequential nature provides a convenient blueprint for expanding the assessment of library value. By considering the sequence students follow as they move through their college careers, other key stages can be identified, along with potential library impacts. For example, to “back up” to the very start of the college education process, one interesting possibility involves the recruitment of new students. Recruitment is a crucial institutional concern, especially at smaller, tuition-driven schools; thus, it may repre- sent a potentially overlooked area for demonstrating library value. Library impacts upon recruitment could be assessed with an experimental approach. Since prospec- tive students commonly tour campuses in small groups, students could be randomly assigned to groups, with some groups receiving a standardized library element, like a library tour, while other groups omit this. Follow-up surveys and actual enrollment figures could be used to compare the two types of groups, therefore determining if the inclusion of a library element has a favorable impact. If it could be shown that library involvement in the recruiting process can improve outcomes like yield (the percent of admitted students who decide to enroll), the enroll- ment rates for a desirable demographic group, or the academic quality of an entering class, it would send a powerful signal about library value. As the symbolic “heart of the university,” libraries may have some strong cards to play in terms of attracting students (and their parents). Moving one step forward in the sequence, remedial classes might provide another opportunity. At many institutions, a substantial percentage of incoming students require remedial assistance to perform college-level work.40 As with regular classes, libraries potentially can contribute to student success in this area. As with most course-based approaches, small-group studies could provide an appropriate study methodology, serving to compare learning and grades in remedial classes that contain—versus lack—a library component. If students in remedial classes have been underserved by library services in high school, library impacts here might be particularly strong. The recruiting and remedial stages suggested here could be supplemented with oth- ers. Assessing a wide range of steps could help to better document impacts from a wide range of library resources and services, as strongly advocated by the ACRL Value report.41 Envisioning library impacts as a sequence also highlights a second attractive feature of the framework; it can readily lend itself to a narrative format for reporting library value. Demonstrations of sequential library impacts—ranging, ideally from recruit- ment to graduation—can be woven into an iterating, cumulating story, explaining how libraries benefit students—and thus the parent college or university—at virtually every step of their college and university careers. As Steve Hiller and Betsy Wilson note, combining data and narratives into evidence-based narratives provides a potent way to communicate library value to college and university administrators.42 434 College & Research Libraries April 2018 Conclusion Our approach supplies two things that have previously been absent from library value scholarship: 1) a general scheme to understand the structure and organization of the existing library value literature; and 2) a framework for a systematic academic library assessment program. As this literature continues to grow, it is crucial to foster coher- ence in this field, so library value programs can continue to mature in sophistication and increase their practical utility. As such, our multilevel framework meshes well with the Value of Academic Libraries report, and it can help to provide a conceptual template to accompany a formal assessment management system. Given its ability to establish library value at three distinct but interrelated levels, this approach comprises more than the sum of its parts. It draws validity from its close mapping to the structure and organization of existing academic library value scholarship, and it increases the scholarly rigor of the field through its reliance on the fundamental, widely recognized concept of levels of analysis. Ultimately, we believe that practical assessment programs will be the most important beneficiary. If the Value report is correct in calling for “systematic, coherent, and connected evidence to establish the role of libraries in student learning,” then this multilevel approach and framework would seem well equipped to provide it.43 Notes 1. Association of College and Research Libraries (ACRL), Value of Academic Libraries: A Com- prehensive Research Review and Report, researched by Megan Oakleaf (Chicago, Ill.: Association of College and Research Libraries, 2010), available online at www.acrl.ala.org/value/?page_id=21 [accessed 12 September 2012]. 2. Jerome Bruner, J.J. Goodnow, and G.A. Austin, A Study of Thinking (New York, N.Y.: Wiley, 1962), 12–21; D.L. Medin and C. Aguilar, “Categorization,” in The MIT Encyclopedia of the Cogni- tive Sciences, eds. R.A. Wilson and F.C. Keil (Cambridge, MA: MIT Press, 2001), 104–05; Jerome S. Bruner, “On Perceptual Readiness,” Psychological Review 64, no. 2 (1957): 123–52; C. Neil Macrae and Galen V. Bodenhausen, “Social Cognition: Thinking Categorically About Others,” Annual Review of Psychology 51 (2000): 93–120. 3. Pat Lauderdale, Steve McLaughlin, and Annamarie Oliverio, “Levels of Analysis, Theoreti- cal Orientations, and Degrees of Abstraction,” American Sociologist 21, no. 1 (1990): 29–40. 4. J. David Singer, “The Level-of-Analysis Problem in International Relations,” in The Inter- national System: Theoretical Essays, eds. Klaus Knorr and Sidney Verba (Princeton, N.J.: Princeton University Press, 1961); Kenneth N. Waltz, Man, the State, and War: A Theoretical Analysis (New York, N.Y.: Columbia University Press, 1959); Eleanor Rosch, “Principles of Categorization,” in Cognition and Categorization, eds. Eleanor Rosch and Barbara B. Lloyd (Hillsdale, N.J.: Lawrence Erlbaum, 1978). 5. David Streatfield and Sharon Markless, “Evaluating the Impact of Information Literacy in Higher Education: Progress and Prospects,” Libri 58, no. 2 (2008): 102–09. 6. Char Booth, M. Sara Lowe, Natalie Tagge, and Sean M. Stone, “Degrees of Impact: Analyz- ing the Effects of Progressive Librarian Course Collaborations on Student Performance,” College & Research Libraries 76, no. 5 (2015): 623–51. 7. Anne Horn, Alexia Maddox, Pauline Hagel, Michael Currie, and Sue Owen, “Embedded Library Services: Beyond Chance Encounters for Students from Low SES Backgrounds,” Australian Academic and Research Libraries 44, no. 4 (2013): 235–50. 8. Barbara A. Shaffer, “Graduate Student Library Re¬search Skills: Is Online Instruction Ef- fective?” Journal of Library & Information Services in Distance Learning 5, no. 1/2 (2011): 35–55. 9. Laura M. Schimming, “Measuring Medical Student Preference: A Comparison of Classroom versus Online Instruction for Teaching,” Journal of the Medical Library Association 96, no. 3 (2008): 217–22. 10. Susan C. Smith and Emma C. O’Hagan, “Taking Library Instruction into the Online Envi- ronment: One Health Science Library’s Experience,” Journal of the Medical Library Association 102, no. 3 (2014): 196–200. 11. James Nichols, Barbara Shaffer, and Karen Shockey, “Changing the Face of Instruction: Is Online of In-class More Effective?” College & Research Libraries 64, no. 5 (2003): 378–88; Brighid M. http://www.acrl.ala.org/value/?page_id=21 A Multilevel Approach for Library Value Assessment 435 Gonzales, “Online Tutorials and Effective Information Literacy Instruction for Distance Learners,” Journal of Library & Information Services in Distance Learning 8, no. 1/2 (2014): 45–55. 12. Qinqin Zhang, Maren Goodman, and Shiyi Xie, “Integrating Library Instruction into the Course Management System for a First Year Engineering Class: An Evidence-Based Study Mea- suring the Effectiveness of Blended Learning on Students’ Information Literacy Levels,”College & Research Libraries 76, no. 7 (2015): 934–58. 13. Cindy L. Craig and Curt G. Friehs, “Video and HTML: Testing Online Tutorial Formats with Biology Students,” Journal of Web Librarianship 7, no. 3 (2013): 292–304. 14. Booth, Lowe, Tagge and Stone, “Degrees of Impact.” 15. Jacline L. Contrino, “Instructional Learning Objects in the Digital Classroom: Effectively Measuring Impact on Student Success,” Journal of Library & Information Services in Distance Learn- ing 10, no. 3/4 (2016): 186–98; Shaffer, “Graduate Student Library Research Skills.” 16. Megan Oakleaf, “Using Rubrics to Assess Information Literacy: An Examination of Methdology and Interrater Reliability,” Journal of the American Society for Information Science and Technology 60, no. 5 (2009): 969–83; Lorrie A. Knight, “Using Rubrics to Assess Information Literacy,” Reference Services Review 34, no. 1 (2006): 43–55; Elizabeth Choinski, Amy E. Mark, and Missy Murphey, “Assessment with Rubrics: An Efficient and Objective Means of Assessing Student Outcomes in an Information Resources Class,” portal: Libraries and the Academy 3, no. 4 (2003): 563–75; Davida Scharf, Norbert Elliott, Heather A. Huey, Vladimir Briller, and Kamal Joshi, “Direct Assessment of Information Literacy Using Written Portfolios,” Journal of Academic Librarianship 33, no. 4 (2007): 462–77. 17. Indeed, this interest is reflected in the Rubric Assessment of Information Literacy Skills (RAILS), an IMLS-funded project that sought to investigate and develop rubric-based assessments of IL skills. See http://railsontrack.info [accessed 12 January 2016]. 18. Elise Ferer, Mary Ellen Taggart Ford, Stephanie Sullivan, and Joe Hawk, “Putting the Library to Work: Information Literacy Skills in Co-Op Job Search” (poster presented at the 2016 American Library Association Annual Conference, June 24, 2016), available online at https://apply. ala.org/aia/docs/project/13829 [accessed 15 February 2017]. 19. Tyler Scott Smith and Linda A. Kopecky, “Preparing the New Professionals: Assessing the Long-Term Impact of Library Internships on Graduate Student Success” (poster presented at the 2014 American Library Association Annual Conference, June 27, 2014), available online at https:// apply.ala.org/aia/docs/project/5503 [accessed 15 February 2017]. 20. This narrowness in dependent variables has been evident since the earliest single- institu- tion studies. A 1965 article found that the number of books checked out correlated positively with GPA for undergraduates: Patrick Barkey, “Patterns of Student Use of a College Library,” College & Research Libraries 26, no. 2 (1965): 115–18. Similarly, a 1974 article found that freshmen scores on the Library Orientation Test correlated with GPA for these students: Donna Corlett, “Library Skills, Study Habits and Attitudes, and Sex as Related to Academic Achievement,” Educational and Psychological Measurement 34, no. 4 (1974): 967–69. 21. Krista M. Soria, Jan Fransen, and Shane Nackerud, “Library Use and Undergraduate Stu- dent Outcomes: New Evidence for Students’ Retention and Academic Success,” portal: Libraries and the Academy 13, no. 2 (2013): 147–64. 22. Krista M. Soria, Jan Fransen, and Shane Nackerud, “Stacks, Serials, Search Engines, and Students’ Success: First-Year Undergraduate Students’ Library Use, Academic Achievement, and Retention,” Journal of Academic Librarianship 40, no. 1 (2014): 84–91. 23. John K. Stemmer and David M. Mahan, “Investigating the Relationship of Library Usage to Student Outcomes,” College & Research Libraries 77, no. 3 (2016): 359–75. 24. Elizabeth M. Mezick, “Return on Investment: Libraries and Student Retention,” Journal of Academic Librarianship 33, no. 5 (2007): 561–66. 25. Gregory A. Crawford, “The Academic Library and Student Retention and Graduation: An Exploratory Study,” portal: Libraries and the Academy 15, no. 1 (2015): 41–57. 26. Florence A. Hamrick, John H. Schuh, and Mack C. Shelley II, “Predicting Higher Education Graduation Rates from Institutional Characteristics and Resource Allocation,” Education Policy Analysis Archives 12, no. 19 (2004): 1–23. 27. Elizabeth M. Mezick, “Relationship of Library Assessment to Student Retention,” Journal of Academic Librarianship 41, no. 1 (2015): 31–36. 28. Graham Stone and Bryony Ramsden, “Library Impact Data Project: Looking for the Link between Library Usage and Student Attainment,” College & Research Libraries 74, no. 6 (2013): 546–59. 29. Ryan P. Womack, “ARL Libraries and Research: Correlates of Grant Funding” (2016), avail- able online at http://arxiv.org/pdf/1601.05104.pdf [accessed 1 April 2016]. 30. Dean Hendrix, “Relationships between Association of Research Libraries (ARL) Statistics and Bibliometric Indicators: A Principal Components Analysis,” College & Research Libraries 71, http://railsontrack.info https://apply.ala.org/aia/docs/project/13829 https://apply.ala.org/aia/docs/project/13829 https://apply.ala.org/aia/docs/project/5503 https://apply.ala.org/aia/docs/project/5503 http://arxiv.org/pdf/1601.05104.pdf 436 College & Research Libraries April 2018 no. 1 (2010): 32–41. 31. George D. Kuh and Robert M. Gonyea, “The Role of the Academic Library in Promoting Student Engagement in Learning,” College & Research Libraries 64, no. 4 (2003): 256–82. 32. Hubert M. Blalock, Social Statistics (New York, N.Y.: McGraw-Hill, 1979); Amy Blackstone, Principles of Sociological Inquiry: Qualitative and Quantitative Methods (2012). 33. John W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 4th ed. (Thousand Oaks, Calif.: Sage, 2013). 34. Ibid. 35. ACRL, Value of Academic Libraries, 14. 36. Ibid., 39. 37. Soria, Fransen, and Nackerud, “Stacks, Serials, Search Engines, and Students’ Success,” 7. 38. David Schwieder and Lisa Janicke Hinchliffe, “NCES Datasets and Library Value: An Ex- ploratory Study of the 2008 Data,” Proceedings of the 2012 Library Assessment Conference, available online at http://libraryassessment.org/bm~doc/proceedings-lac-2012.pdf [accessed 13 May 2016]. 39. National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, “Comparing the ALS to IPEDS AL Component” (n.d.), available online at https:// nces.ed.gov/ipeds/resource/download/IPEDS_ALSComparing.pdf [accessed 3 February 2016]. 40. Jamie P. Merisotis and Ronald A. Phipps, “Remedial Education in Colleges and Universi- ties: What’s Really Going On?” Review of Higher Education 24, no. 1 (2000): 67–85. 41. ACRL, Value of Academic Libraries, 101–39. 42. Steve Hiller and Lizabeth (Betsy) Wilson, “Making the Case for Institutional Investment in Libraries: The Value of Evidence-Based Narratives,” available online at http://libraryassessment. org/bm~doc/Hiller_Steve_2012.pdf [accessed 12 March 2016]. 43. ACRL, Value of Academic Libraries, 117. http://libraryassessment.org/bm~doc/proceedings-lac-2012.pdf https://nces.ed.gov/ipeds/resource/download/IPEDS_ALSComparing.pdf https://nces.ed.gov/ipeds/resource/download/IPEDS_ALSComparing.pdf http://libraryassessment.org/bm~doc/Hiller_Steve_2012.pdf http://libraryassessment.org/bm~doc/Hiller_Steve_2012.pdf