Investigating the Relationship of Library Usage to Student Outcomes 359 Investigating the Relationship of Library Usage to Student Outcomes John K. Stemmer and David M. Mahan John K. Stemmer is Director of Library Services at Bellarmine University; e-mail: jstemmer@bellarmine. edu. David M. Mahan is Director of Institutional Research and Assessment at Manhattan College; e-mail: david.mahan@manhattan.edu. © 2016 John K. Stemmer and David M. Mahan. This study connects library user surveys, a common library assessment technique, to institutional data to demonstrate the value an academic library brings to student learning and student outcomes. Using regression techniques, the study identifies multiple significant correlations, both posi- tive and negative, between student use of the library and student learning and outcomes as measured by retention, graduation, and grade point av- erage (GPA). The library factors associated with student outcomes change over the course of the four-year undergraduate experience. Methods used in this study could be a model for other institutions seeking a means for assessing the library’s relationship to student learning and outcomes. hile libraries were once “the heart of the university,” their value is no longer taken for granted. Librarians are increasingly facing the task of demonstrating the value they provide to the university communities they serve. Universities are expected to provide information on assessment and evaluation of their educational efforts. In this environment, it is not surprising that the academic library, as a creature of higher education, finds itself working to identify measures to more clearly demonstrate the library’s value by its influence on institution- ally relevant outcomes—faculty productivity or student outcomes. This study gathered data from user surveys on three different areas of library influ- ence—resources, services, and place (or space)—and how these areas relate to student success across the undergraduate years. This research has developed a model of library usage that considers multiple variables at various points in time during the undergradu- ate experience. Specifically, how does the student’s reported usage of the library’s services and resources relate to their success after controlling for exogenous characteristics? The Spellings Commission Report of 2006 influenced American higher education to more closely examine accountability and assessment.1 The economic recession of 2008 has intensified these efforts as higher education institutions have worked to dis- tinguish themselves and demonstrate their benefit to the wider community. Following the federal government’s lead, organizations, institutions, and researchers began an ongoing effort to create a “culture of assessment.” Colleges and universities often as- sess every unit and academic department, measuring both efficiency and effectiveness in support of institutional goals.2 doi:10.5860/crl.77.3.359 crl15-704 360 College & Research Libraries May 2016 In light of this ongoing change in the world of higher education, library leadership has begun to search for data that demonstrate the value of an academic library in institutional terms. The appearance of a conference dedicated to library assessment is one indication of the increasing importance of assessing the library in relationship to institutional goals. The first Association of Research Libraries Assessment Confer- ence was held in 2006 and has continued biannually. Joseph Matthews’ 2007 work on Library Assessment in Higher Education also stressed the need for libraries to look at how they impact student learning and faculty research.3 At the 2010 Library Assess- ment Conference, keynote speaker David Shulenburger highlighted that libraries have considerable data on their operations and that these “data can be used to evaluate the impact of library services and resources on outcomes of value to the university.”4 He recommended the ACRL published report The Value of Academic Libraries, written by Megan Oakleaf. She emphasizes the need to place library assessment in an institu- tional context, stating, “Not only should academic libraries align themselves with the missions of their institutions, they should also allow institutional missions to guide library assessment.”5 Literature Review Assessment is not a new topic for libraries, but it has historically focused more on inputs, such as collection size, journal subscriptions, and materials expenditures rather than institutional considerations such as student outcomes. Jon Hufford’s review article on the literature of assessment in academic libraries provides a valuable and wide-ranging overview of library assessment from the year 2005 to 2011.6 The focus of this review is on studies and reports showing the library’s relationship with institutionally relevant goals, such as student outcomes in learning and retention. While there have been a few studies raising the question of the library’s connection with student outcomes, Oakleaf’s 2010 report serves as a signpost, marking the start of the growth of a number of projects that address the question of the value of the academic library to institutional outcomes. She encourages a research agenda that will allow academic libraries to present their value to appropriate stakeholders.7 In 1986, Jane Hiscock was one of the first to try to establish a connection between the library and academic performance. Her results were generally disappointing in this regard; though she was able to identify two areas, previous experience of libraries and usage of the library catalog, that were associated with positive academic performance.8 Several years later, Ronald Powell again urged librarians to consider how student use of the library affects their academic performance and suggested methods that could be used to address this question. He indicated that a user panel may be an effective means to do so.9 A decade later, in 2002, Ethelene Whitmire addressed the need to connect the library and student use and educational outcomes. Using the IPEDS Academic Library Survey and the College Student Experiences Questionnaire, she used the self-reported gains in critical thinking as a measure of library usage’s relationship with student outcomes. She found no effect of the library’s services or resources on this measure, suggesting instead that stronger links between the library and educational outcomes may exist in other outcomes, such as graduation rates or graduate school entrance exams.10 A number of more recent studies have looked at the relationship between the library and various student outcomes and identified statistically significant correlations. Several articles have considered differing ways an academic library can influence institutionally relevant outcomes, exploring whether or not there is a correlation be- tween a particular aspect of the library and student outcomes. While not causal, they do indicate that a relationship does exist between the library and student success. In two different studies, S.H.R. Wong and her colleagues demonstrated a positive correla- Investigating the Relationship of Library Usage to Student Outcomes 361 tion between the library and student outcomes. They first found a positive correlation between checking out materials and a student’s graduating grade point average (GPA)11 and then a relationship between library instruction and a student’s graduating GPA.12 In the United Kingdom, the Library Impact Data Project also reported “a statistically significant relationship between student attainment” and two indicators: using the library’s electronic resources and borrowing its books.13 Soria, Fransen, and Nackerud investigated the influence of the library on retention and academic success of first-year students at the University of Minnesota and reported higher GPA and higher retention rates for library using students.14 Other researchers are attempting to create appropriate assessment models that will demonstrate library impact on student outcomes. Derek Rodriguez developed an Understanding Library Impact protocol, expanding efforts in evaluating the library’s impact by focusing on individual academic programs. Examining the library’s impact through capstones, for example, that are a natural part of an academic program’s as- sessment efforts links the library’s efforts to ongoing programmatic and institutional efforts, thus making them more understandable to stakeholders.15 While Margie Jantti and Brian Cox have reported on the development of the “library cube” to bring together library resources, student demographic data, and student academic performance as a means to support continuous improvement of the library.16 The ACRL has supported the development of other efforts through its Assessment in Action Program. This effort has generated a number of projects whose results will soon be reported.17 One of the longstanding tools in the librarians’ assessment toolkit has been user surveys. Jim Self and Steven Hiller point out that user surveys have strengths and weaknesses. Among their strengths are highlighting a user’s perspective on the library and examining developments over time.18 User surveys, by their nature, focus on library users, not on other library stakeholders, and so are more operationally than value oriented. A multi-institutional user survey, the MISO, examined trends and identified the library as place and its online services as valuable to user groups.19 One effort has been made to apply user surveys to student success assessment. Ying Zhong and Jo- hanna Alexander relied on user survey data to examine how library services make a difference in student success. They identified the importance of the library’s facilities and its electronic resources as the most significant factors in students’ views.20 These surveys are limited, however, because data collected are anonymous and can only be used to address aggregate trends, not individual impact. Institutional Context Bellarmine University is a private, Catholic university located in a large Midwestern city, Louisville, Kentucky. Total university enrollment is approximately 2,000 under- graduate students (approximately half living on campus) and nearly 800 graduate students. The institution offers more than 50 undergraduate degree programs and more than 20 graduate programs, mostly master’s degrees but also two doctorate practitioner degrees and one research doctorate. More than 80 percent of Bellarmine’s undergraduate students attend full-time and are under 25 years of age. Each fall, approximately 600 new full-time freshmen matriculate at Bellarmine University. Measures of demographics and academic preparedness are consistent across freshman cohorts: • 24–25 ACT Composite average • 65 percent from Kentucky, 35 percent out of state • 40 percent first-generation students (defined as neither parent earning a bach- elor’s degree) • 20 to 30 percent Pell recipients 362 College & Research Libraries May 2016 • 10 to 15 percent students of color • 1 percent international students A goal of the administration at Bellarmine University is to build a culture of assess- ment. In light of this new culture of assessment, the library developed a collaborative relationship with the Office of Institutional Research and Effectiveness (OIR&E). One key goal of the collaboration was to consider the relationship, if any, between the library and student outcomes. These can be defined in a number of ways such as retention and graduation rates, GPA, student satisfaction surveys, and direct measures of learning. The possibility of investigating if there is a connection between the library and these student outcomes would require combining data from both areas. The library’s student user survey provides data on how students used the library and how often they do so; the OIR&E maintains student data on specific student outcomes such as retention, GPA, and graduation status. Methods The conceptual framework used for the study was the Astin Input-Environment- Outcome model, which serves as a tool to better understand student development while in college. In this model, inputs represent the student characteristics at the time of entry into an institution. Environment addresses various educational and cocurricular experiences to which a student is exposed—including use of the library. Outcomes are the model’s results and focus on student characteristics after the student experiences the collegiate environment. Because of the complexity of the student ex- perience during college, the model is a tool to assess student growth holistically. The premise of the model involves understanding students at point of entry to determine whether the collegiate experience influences student outcomes.21 Statistical methods were used to assess measurements of the students’ overall four-year experience. The models specifically were built using multiple regression techniques to identify predic- tor environmental (engagement) variables, which may have a significant relationship with GPA and retention/graduation rates. The Astin I-E-O model simplifies some of the complexity of the student experience. The basic premise of the Astin model is to determine the input characteristics of stu- FIGURE 1 Astin Model (1970): Four-Year Student Experience Inputs Environment Outcomes Genera�on Status Pell Status Freshmen EFC ACT composite HS GPA Race Gender Cohort Year HS Private or Public Reasons for coming to the Library Frequency of coming to the Library How the Library materials were accessed Class: FR, SO, JR, SR Athlete Major Reten�on in 2nd year Gradua�on in 4,5,6 years Grade Point Average (x/4.0) - Session GPA - Cumula�ve GPA - Gradua�ng GPA Investigating the Relationship of Library Usage to Student Outcomes 363 dents to assess if the college experience has made a difference in student development. While many input student characteristics and control variables were used in this study, there were many more that could not be considered because of sample size and the scope of the study. The researchers considered whether library usage played a small but important role in the successful undergraduate student. No claim of causation is intended, but if a significant relationship were found, it would indicate that the use of the library was one piece in the many experiences of a successful Bellarmine student. The researchers collected data from undergraduate students participating in a library survey in the years of 2007, 2008, 2010, and 2012. Feedback from the survey is tied to a unique and confidential student ID, which can be tied to institutional research metrics at the student unit level. Since the survey is not anonymous, the library can identify how students report using the library at an individual level. This is an impor- tant change to the way many user surveys are conducted, and it allows for individual student results to be linked to institutional student information. Oakleaf called for just this type of development in her 2010 report.22 Mathews reiterated that it is essential to identify the library’s individual student unit data if the library is going to be able to link its programmatic efforts to student outcomes.23 The survey provides data on why students come to and how often they use the li- brary. The library survey asked questions in two groupings. First, the survey addresses reasons a student came to the library, using a list of 21 reasons with checkboxes. Re- spondents were asked to check all reasons that applied. The second group of questions requested information on how often a student came to the library and how often the student used the library online, with response options ranging from daily to never. (See appendix A for the relevant sections of the most recent library questionnaire.) The student survey runs in the spring semester, usually in late February or early March. More recent years have used a small gift card prize randomly selected from respondents to incentivize student responses. Due to the manageable number of full- time undergraduate students at the institution, sampling was unnecessary because the entire population could easily be requested to participate. All undergraduate students were invited by e-mail to respond to the web-based survey regardless of whether they enrolled as traditional freshmen, transfer, or readmitted students. Response rates for the survey were from 20 to 26 percent. While all students were sent the survey, only traditional, cohorted student re- sponses were used in this study to maintain similar demographics and to minimize variations among the respondent experiences. The library data would be viewed in collapsed undergraduate classification cohorts; all first-year, sophomore, junior, and senior responses from all years of the survey were combined into their appropriate undergraduate classification cohort. This ensured that the time spent at the institution was consistent in comparing class levels (freshmen, sophomores, juniors, and seniors). For example, researchers studied the first-year experience by combining students who responded to the library survey in their freshman year, from cohorts 2006, 2007, 2009 and 2011. Thus all students in this analysis were traditional-aged freshman in their first year of college at Bellarmine University. Combining cohorts was possible because little variance was found in terms of the entering demographic and academic preparedness characteristics of freshman cohorts. The library survey data was then linked with Office of Institutional Research student data to first determine if there were any significant relationships between a student’s self-reported library usage and known student outcomes. This was done using a cor- relation analysis of all the library input variables against a selected student outcome including retention, graduation, and GPA. In addition to the library variables, several demographic factors drawn from institutional research data were also tested against 364 College & Research Libraries May 2016 selected student outcomes to identify the most significant demographic factors for the cohorts. Using these control factors and the library variables, specific research questions were developed: 1. Does library usage have a significant positive correlation with whether a fresh- man student returns in his/her next year of undergraduate study? 2. Does library usage have a significant positive correlation with whether a fresh- man, sophomore, junior, or senior student graduates in undergraduate study? 3. Does library usage have a significant positive correlation with grade point average (GPA) for freshmen, sophomore, junior, or senior students? Limitations of the Study It is important to note the self-reporting nature of the study, as this method has been questioned recently, especially in regard to self-reported learning gains (SRLG). However, this survey is not pursuing self-reported gains in learning, but more fac- tual information on specific activities of the respondents. According to the Center for Postsecondary Research at the Indiana University School of Education, the validity of self-reported data is contingent on whether respondents can respond accurately. The NSSE User Guide reports that research indicates that self-reported data are valid under five conditions: 1. The requested information is known to respondents. 2. The questions are phrased clearly and unambiguously. 3. The questions refer to recent activities. 4. The respondents take the questions seriously. 5. The questions do not threaten, embarrass, or violate respondents’ privacy.24 All of these were considered during the survey design and the collection of data. Another limitation to consider is the independence of sample, as the survey is not proc- tored; however, an individually specific web link is sent to requested respondents. In TABLE 1 Population by Cohort Year and Classification Used for Analysis Freshman Sophomore Junior Senior Full-time Freshman Cohort Census Undupl. Count Resp. Count *Libr. Surv. Yr. Resp. Count *Libr. Surv. Yr. Resp. Count *Libr. Surv. Yr. Resp. Count *Libr. Surv. Yr. Fall 2003 451 89 2007 Fall 2004 409 96 2007 73 2008 Fall 2005 435 100 2007 76 2008 Fall 2006 430 83 2007 58 2008 51 2010 Fall 2007 568 127 2008 94 2010 Fall 2008 567 111 2010 93 2012 Fall 2009 602 145 2010 132 2012 Fall 2010 577 129 2012 Fall 2011 600 134 2012 **Total 498 23% 398 25% 398 27% 306 24% *The library survey is administered biannually in the spring semester. ** Response rates are based on fall census counts in semester before the library survey is administered. Investigating the Relationship of Library Usage to Student Outcomes 365 addition, a limitation of single institution studies at small private schools is the number of student participants relative to the number of variables considered for analysis. This study’s sample was representative of the small population at the university. Statistical Analysis Freshman Retention Researchers conducted a forward-entry logistic regression using freshmen student data only (n = 488), considering any significant library variables related to the outcome variable, Return (indicating that the student returned for his or her second year). There were 448 freshmen who returned in the second year (92%) and 40 students who did not (8%). A logistic regression was conducted in two steps. First, the control variable, Session GPA, was entered. This was selected because correlation analysis revealed significance relationship to Return (p <.05). Second, forward-entry technique was used to consider library variables after the control variables. The omnibus test of model coefficients was significant: chi-square was 53.770 (df = 3, p < .001). Session GPA significantly predicted Return (p < .001) and the Nagelkerke R squared was 0.168. After the control variable was entered, the library variable Access library online loaded first as a significant predic- tor (p < .05). The Nagelkerke R squared was 0.226. In step 2, Use computers for personal use was the next significant predictor (p < .05). However, it was a negative predictor of return—students who reported that using the library’s computers for personal use was a reason to come to the library were less likely to return for their second year. The final Nagelkerke R squared was 0.243. Freshman GPA To examine first-year GPA, the study started by conducting an analysis of freshman student data, considering any significant nonlibrary variables related to the outcome variable, Session GPA. The method used was a forward-entry ordinary least squares (OLS) regression to consider what if any variables predict Session GPA. Four signifi- cant variables were identified: ACT, HS GPA, first-generation status and estimated family contribution (EFC). A second regression was conducted in two steps. First, the significant nonlibrary control variables (p < .05) were entered: ACT, HS GPA, first-generation status, and EFC. The adjusted R squared was 0.339. Then forward-entry technique was used to TABLE 2 Second Year Retention of First Year Students Significant Variables B S.E. Wald df Sig. Exp(B) Step 1a SessGPA 1.293 .227 32.469 1 .000 3.642 howoftenaccesslibraryonline .558 .162 11.918 1 .001 1.748 Constant –2.780 .802 12.009 1 .001 .062 Step 2b SessGPA 1.327 .229 33.541 1 .000 3.771 Usecomputersforpersonaluse –.737 .372 3.925 1 .048 .479 howoftenaccesslibraryonline .600 .165 13.238 1 .000 1.822 Constant –2.595 .815 10.134 1 .001 .075 a. Variable(s) entered on step 1: howoftenaccesslibraryonline. b. Variable(s) entered on step 2: Usecomputersforpersonaluse. 366 College & Research Libraries May 2016 consider library variables after the control variables. Two significant library predictors were identified: Study alone and Check out books. Both variables were positive predic- tors; agreement with them was associated with a higher session GPA. The adjusted R squared was 0.0364 for the first-step entry of study alone. It was 0.0369 for the second- step entry of Check out books. Similar analysis was done for both Cumulative GPA (at end of freshman year) and Graduating GPA (when considering those freshmen who ultimately completed an un- dergraduate bachelor’s degree). Cumulative GPA had significant nonlibrary variables of HS Type, Session GPA, Minority status, Pell2, ACT, and HS GPA. The library variable How often access the library online had a very small, but still significant, effect on the adjusted R squared, increasing it to 0.0899 from 0.0897 for just the nonlibrary variables (Sig = 0.001). One library variable had an effect on Graduating GPA—how often use the study room after hours. The change in adjusted R squared moved from 0.688 to 0.694, but it was a negative association. Students who studied more often in the 24-hour room their freshman year had lower GPAs (Sig = 0.015, n = 244). TABLE 4 Freshman GPA ANOVAd Model Sum of Squares df Mean Square F Sig. 1 Regression 75.108 4 18.777 62.176 .000a Residual 142.845 473 .302 Total 217.953 477 2 Regression 80.745 5 16.149 55.553 .000b Residual 137.208 472 .291 Total 217.953 477 3 Regression 82.189 6 13.698 47.522 .000c Residual 135.764 471 .288 Total 217.953 477 a. Predictors: (Constant), EFC2, HS GPA, 1st Gen, ACT b. Predictors: (Constant), EFC2, HS GPA, 1st Gen, ACT, study alone c. Predictors: (Constant), EFC2, HS GPA, 1st Gen, ACT, study alone, check out books d. Dependent Variable: Sess GPA TABLE 3 Freshman GPA Model Summaryd Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .587a .345 .339 .54954 .345 62.176 4 473 .000 2 .609b .370 .364 .53916 .026 19.392 1 472 .000 3 .614c .377 .369 .53689 .007 5.009 1 471 .026 a. Predictors: (Constant), EFC2, HS GPA, 1st Gen, ACT b. Predictors: (Constant), EFC2, HS GPA, 1st Gen, ACT, study alone c. Predictors: (Constant), EFC2, HS GPA, 1st Gen, ACT, study alone, check out books d. Dependent Variable: Sess GPA Investigating the Relationship of Library Usage to Student Outcomes 367 TA B L E 5 St ud en t R et en ti on a nd G ra du at io n by C la ss ifi ca ti on C oh or t N B Si g N ag el R 2 N B Si g N ag el R 2 N B Si g N ag el R 2 N L V S es sG PA 0. 16 4 St ep 1 ho w of te n ac ce ss li b on lin e 4 88 0. 55 8 0. 00 1 0. 22 6 N O S IG 24 3 N on e N O S IG 34 7 N on e St ep 2 us ec om pu te rs fo rp er so na lu se 48 8 –0 .7 37 0 .0 48 0. 24 3 N L V Pe ll2 , S es s G PA 0. 21 7 Se ss G PA , H S G PA , S ex 2 0. 25 2 St ep 1 N A us ec om pu te rs fo r a ca de m ic pu rp os es 24 3 0. 74 6 0. 04 7 0. 23 9 st ud y al on e 26 1 1. 23 6 0. 02 3 0. 29 0 St ep 2 m ee t f ri en ds –0 .7 98 0. 05 3 0. 26 0 us e a la pt op –1 .4 25 0 .0 36 0. 32 1 N L V Se x2 0. 03 5 St ep 1 N A N O S IG 35 4 N on e lo ca te a rt ic le s 39 0 0. 88 5 0. 03 4 0. 06 4 N L V 0. 18 1 St ep 1 N A N O S IG 30 8 N on e N O S IG 30 8 N on e Sr NN B NN B 4 Y ea r G ra d NN B G ra d Y /N 1s t Y r R et ur n So ph Jr N LV =s ig ni fic an t n on li br ar y va ri ab le s 368 College & Research Libraries May 2016 TA B L E 6 St ud en t G PA b y C la ss ifi ca ti on C oh or t N B Si g A dj _R 2 N B Si g A dj _R 2 N B Si g A dj _R 2 N L V EF C 2, H S G P A , 1s tG en , A C T 0. 33 9 H S T yp e, S es s G PA , M in or it y, P el l2 , A C T H S G PA 0. 89 7 Pe ll 2, A C T , H S G P A , Se ss G PA 0. 68 8 St ep 1 st ud y al on e 47 8 0. 29 7 0. 00 0 0. 36 4 ho w o ft en a cc es s li b on li ne 47 7 0. 02 3 0. 00 1 0. 89 9 ho w o ft en u se 2 4h r st ud y ro om a ft er h ou rs 24 4 –0 .0 27 0. 01 5 0. 69 4 St ep 2 ch ec ko ut bo ok s 0. 11 1 0. 02 6 0. 36 9 N L V H SG PA , A C T 0. 23 1 Se ss G PA , M in or it y, A C T , H S G PA 0. 69 9 Se ss G PA , E FC 2, A C T , H G PA 0. 66 1 St ep 1 us e a la pt op 38 5 –0 .1 52 0. 04 7 0. 23 7 m ee tf ri en ds 38 5 –0 .0 67 0. 02 3 0. 70 2 m ee tf ri en ds 23 8 –0 .1 18 0. 00 0 0. 67 7 St ep 2 ho w of te n2 4h rr oo m –0 .0 23 0. 04 6 0. 70 5 us eg ro up s tu dy rm s 0. 06 3 0. 03 1 0. 68 2 St ep 3 us eg ro up s tu dy rm s 0. 06 0. 03 0 0. 70 7 ho w of te n2 4h rr oo m –0 .0 31 0. 02 2 0. 68 8 St ep 4 co m pu te rs fo r p er so na l us e –0 .0 62 0. 03 5 0. 69 3 St ep 5 vi si t t he A R C 0. 05 6 0. 04 4 0. 69 7 N L V 1s t G en , H S G PA A C T 0. 07 7 H S G PA , S ex 2, S es s G PA , A C T 0. 52 H S G PA , S ex 2, S es s G PA , A C T 0. 41 5 St ep 1 us e a la pt op 38 6 –0 .2 15 0. 00 8 0. 09 2 vi si t t he A R C 38 6 –0 .1 07 0. 00 1 0. 53 4 vi si t t he A R C 35 6 –0 .0 71 0. 04 3 0. 42 0 St ep 2 st ud y al on e 0. 27 2 0. 01 9 0. 10 2 ch ec ko ut bo ok s 0. 08 2 0. 00 8 0. 54 1 St ep 3 ho w of te n 24 hr rm –0 .0 63 0. 03 7 0. 11 0 us e pr in te r o r ph ot oc op ie r 0. 10 7 0. 01 7 0. 54 7 N L V Pe ll 2, A C T , H S G PA 0. 10 9 H S G PA , S es s G PA , A C T 0. 58 7 H S G PA , S es s G PA , A C T 0. 55 8 St ep 1 ch ec ko ut bo ok s 31 1 0. 20 2 0. 00 3 0. 13 2 lo ca te a rt ic le s 31 1 0. 09 8 0. 00 2 0. 59 9 lo ca te a rt ic le s 30 6 0. 08 7 0. 00 4 0. 56 9 St ep 2 re se ar ch C on su lt in g –0 .1 84 0. 00 4 0. 15 3 vi si t h el p de sk 0. 07 5 0. 01 8 0. 57 5 St ep 3 us e co m pu te rs fo r pe rs on al u se –0 .0 59 0. 04 5 0. 57 9 1s t Y r So ph Jr Sr Se ss G PA C um G PA G ra d G PA Investigating the Relationship of Library Usage to Student Outcomes 369 Similar statistical analyses were done for each classification-based cohort consider- ing the following student success measures: whether the student was retained in his/ her second year, whether the student graduated, and the student’s GPA. Significant results for these cohorted classifications are reported in tables 5 and 6, in which the student outcomes are grouped as either retention and graduation or a measure of academic performance (GPA). Results for Freshman Students A first-year student’s use of the library is related to both his or her second-year retention and GPA, but it is not associated with whether the student graduates in four years or at all. Library usage is positively associated with freshman retention and GPA, whether measured as session GPA (the term when the survey was taken), cumulative GPA (at the end of the first year) or graduating GPA (for those students who graduated). The library variables associated with first-year students are in two areas: resources and place. The library’s information resources have a positive cor- relation, with students who use the library’s book collection and online resources being more likely to return for the second year and have better session or cumula- tive GPAs. The library as place has a split result, with students using the library as a place to pursue academics, such as studying alone, earning a better session GPA, while students using the library as a place for late-night studying (using the “after hours” 24-hour study room) has a negative relationship on their graduating GPA. Another likewise negative connection is the use of computers for personal use. Students indicating that this is one of the reasons they came to the library were less likely to return for the second year. Generally, first-year students who used the library to facilitate learning, either as a place to study or as an information resource, had stronger academic outcomes and retention rates. Students who used the library’s computers to facilitate personal interests were less likely to return for the second year. Likewise, students who used the library’s provision of late-night study space were also negatively affected. The library provided resources and space, but it was individual usage decisions that determined whether these things had a positive or negative impact on a student’s outcomes. Sophomores In examining the sophomore student results, their use of the library is again related to their GPA. It is also related to whether the sophomore graduates. The relation- ship between library usage and student outcomes shifts somewhat. The significant relationship variables, both positive and negative, are mostly when sophomores use the library as a place. The library had a positive relationship when students used it as a place to study alone, used group study rooms, or used the computers for academic purposes. Using the library as a place to meet friends, study late, use computers for personal interests, or use a laptop were negatively related with student outcomes. Library information resource variables are not significantly associated with session, cumulative, and graduating GPAs. One service measured by the survey did show up as a positive relationship, Visiting the Academic Resource Center (ARC), which is Bel- larmine’s combined advising, tutoring, and writing center. Sophomore students who visited the ARC had a positive relationship with Graduating GPA (p < .05). The use of a laptop is a negative association with a student’s Session GPA and whether or not the student graduated at all (p < .05). Unlike freshman year, library usage is related to whether sophomores graduate, either in four years or at all. Sophomores using the library to pursue academic concerns are more likely to graduate. 370 College & Research Libraries May 2016 Juniors In the junior year, certain uses of the library as place or as a technology support continue. Using the library as a late-night study spot makes its third straight appearance as a negative association. Likewise, studying alone makes its third straight positive-factor appearance on student outcomes. Using a laptop continues to present as a negative relationship. The library’s technological support shifts from providing computers for academic purposes to using printers or photocopiers, but it is still positive. One service continues to appear but switches from a positive to a negative on both cumulative and graduating GPA. Visiting the ARC in a student’s third year has a negative association with cumulative and graduating GPA. The role of the library seems to shift in an important way in the junior year. The library support shifts away from being place-based and toward providing informa- tion resources. Using the library for information is significantly related with two of the output variables: Cumulative GPA and Graduate at all. Checking out books reappears as a positive on a student’s Cumulative GPA, as does using library equipment such as a printer or photocopier. Appearing for the first time is Locating articles, which is a positive factor in regard to whether a student graduates at all. Seniors The shift of the library’s role toward information resources is completed in the senior year. The three output variables significantly associated with the library are all student learning outcomes: Session GPA, Cumulative GPA, and Graduating GPA. These have a positive relationship with a senior’s use of the library’s information resources, Checking out books for Session GPA and Locate articles for Cumulative and Graduating GPA. The library as a place completely disappears; both study alone and study rooms are gone as significant library factors. The library’s role with student outcomes is derived from its resources (informational and technological) and services. Results The study’s results indicate that, in many cases, library usage is positively associ- ated with student outcomes. The significant library factors change over time, as the traditional-aged student progresses through the undergraduate years of study. First- year students have positive benefits from using the library to Study alone, Check out books and How often access the library online. These findings provide some confirmation for the earlier studies of Soria et al.25 and Stone et al.26 Negative effects seem to stem mostly from choices made by first-year students to use the library’s resources for personal interests, rather than academic ones. This trend continues in the sophomore year. Using the library as a social space or late-night cram center is negatively associ- ated with student outcomes. Using the library for academic purposes (Study alone, Study rooms, and Academic use of computers) has a positive relationship with student outcomes. In many ways, the role of the library in the underclassman experience is using it as a place; a place used to study alone, with a group, or using technology. Even the negative association of the library is place oriented—technology for personal use and a late-night study center. There are some information resource significant findings in the first year, and they are beneficial; checking out books by first-year students is positively associated with GPA and using online resources with retention. There are no information-resource significant findings in the sophomore year. The sophomore year does have a positive relationship with information service; the ARC and its services positively relate to Graduating GPA. The library’s role starts shifting away from place and toward information resource in the junior year. While the place-oriented study alone, using technology, and using Investigating the Relationship of Library Usage to Student Outcomes 371 the 24-hour room continue to be influential in their previously established positive and negative manners, information resources reappears as a significant factor. Check- ing out books and Locating articles have positive relationships with student outcome, while information services (Visiting the ARC) shifts from being a positive to a nega- tive. One possible explanation for the inversion is that the student has already fallen behind. While the visit may be a beneficial learning experience, it is too late to have a significant positive effect on cumulative or graduating GPA. This would seem to indicate that tutoring and writing interventions have their best effect when done early in a student’s college career, which is supported by the research of the first-year experience.27 By senior year, the library as place almost completely disappears—only nonacademic use of computers continues with a negative relationship. Otherwise, the library’s role is information services (mostly negative) and information resources (all positive). This includes a positive association for seniors who Check out books and Locate articles. While not an exact match, the Check out books connection does agree with the finding of Wong regarding the relationship of resource use with GPA.28 It is interesting that frequency of use seems to have limited connection on student outcomes with one exception. Three frequency questions were asked—how often a student came to the library building, used the library online, and used the library’s 24-hour facility after hours. How often a student visited the library never appeared as a significant factor in any of the tests. How often a student accessed the library online was the only frequency factor that had a positive correlation to a student outcome, namely first-year return. One frequency factor does have a repeated connection to student learning—how often a student uses the library’s 24-hour space has a negative relationship with the student’s GPA. It affects at least one measure of GPA every year except senior year. Reviewing the library factors significantly associated with student outcomes across all four years provides a snapshot of the programmatic role of the library on student outcomes. Grouping student outcomes into persistence (retention and graduation) or learning (GPA) indicates that the library has a much more consistent relationship with student learning than with persistence. Library use had a connec- tion with every measure of student GPA, but with only four of the nine measures of student persistence. Almost all of the links to persistence are in a student’s first two years. The library is used differently by students depending on where they are in their academic careers. In examining the library’s role on student outcomes, what the library provides could often be broken down into three components: library as place, library as resource, and library as service. In examining the study results, two of these three components are well represented. The library as place and library as resource appear throughout the four years reviewed. Library services and other services offered in the library appear less frequently. Note that two of the library’s services—interlibrary loan and, perhaps the most significant library programmatic service, instruction—are not considered in the user survey, so there is no data on this relationship. Underclass students are more place oriented, finding a place to study or do other academic work. Upperclass library usage is significantly related to the library’s information resources. Freshman and, to a lesser extent, sophomores are adjusting to a new environment and experience. Thus, it makes sense that their physical surroundings are perhaps most important. By the time students are upperclassmen, they typically are much more focused on their selected academic discipline. Success in this requires using appropriate information resources, so the library’s information resources become more significant in their success. 372 College & Research Libraries May 2016 Conclusions The results of this study are similar to findings from other studies using much more labor- and technology-intensive data collection. This is an issue noted in studies, such as “Analyzing Demographics” by Nackerud et al.29 Most of these studies have taken place at medium- to largesized institutions. The assessment model in this research uses a research tool common to a wide range of institutional types and sizes because it is less labor and technology intensive. Simply changing a user survey protocol from anonymous to user identifiable may allow even smaller academic libraries to generate local data to demonstrate their value to stakeholders while maintaining its usefulness as an operational management tool. As underclassmen students are adjusting to a new life and new responsibilities, the library’s role is largely place oriented. It provides a study space (whether individually or in groups) as well as technology access. How students use these, and the decisions they make adjusting to their new environment, largely determines whether the library has a positive or negative relationship with student outcomes. As an information resource, the library is limited to the first-year experience, but this could still be indicative of underclassmen adjusting to their new environment. Those who start using the library’s information resources quickly have beneficial impacts on their outcomes. After students have transitioned to upper-division work, aspects of the library’s programmatic activities (space, service, and resources) begin to significantly associate with student outcomes. While space-oriented variables dominate in sophomore year, in a student’s junior year they represent less than half of the library’s role. After students have adjusted to their new environment, selected an academic discipline, and moved on to advanced-level work in their major, the library’s information resources rise to significance across almost all the student outcome variables as influential. This use of books and articles makes sense. As students begin to focus on their majors, books are more valuable to them as a broader-based introduction to the discipline, one that has more space to develop ideas than a scholarly journal article. At the same time, students at an advanced level, in major specific courses, are those most likely to benefit from and understand scholarly journal literature. If the library has done its information fluency in- struction well, it would make sense that these students focus on and use these resources. Implications for practice from this study’s findings cover several areas. While the library’s role is traditionally seen as one related to information seeking and knowledge building, this study seems to indicate that, early in a student’s career, more emphasis should be placed on the role of the library as place, whether it is for study and learning or for technology access. It is also interesting that one service function covered by the survey flipped from a positive connection early in a student’s time to a negative one as the student became an upperclassman. Should the service aspects of the library, such as reference, put more emphasis on reaching out to students and creating relation- ships earlier in their time on campus? Looking at library instructional efforts, many libraries take a developmental approach to information fluency and try to focus on appropriate information resources depending on the topic and level of the course and program. Is this information-seeking model based too much on librarians’ experience with faculty researchers’ information habits and insufficiently addressing the different needs of young college students for a place to focus on academic pursuits with ap- propriate support services? It may be that the development of the learning/academic commons model and more recently the “personal librarian” concept are an organic response of experienced academic library leaders to highlight the library as place and ongoing service relationship and not just an information resource. The changing nature of student engagement with the library over the course of a student’s undergraduate years identified in this study needs further study and consideration. Investigating the Relationship of Library Usage to Student Outcomes 373 When broken down into classification-based cohorts, the data identify multiple significant correlations, both positive and negative, between student use of the library and student learning and outcomes. These statistical analyses provide evidence for the value provided by the library in support of institutionally important student outcome goals. Since this study looks at cohort use of the library across four years of activity, it allows a more holistic view of the library and its role in the undergraduate experience. Not surprisingly, the library factors that are associated with student outcomes change over the course of the student’s time at the university. Appendix A 2012 Library Usage Survey Reasons you come to the library (check all that apply): q To check out books q To check out media (CDs, DVDs, etc.) q To locate journal/newspaper articles q To get help with research papers or other course assignments q To read newspapers or current magazines q To use items (books/articles/videos) placed on reserve by your professor q To use media equipment (such as video cameras, digital cameras, scanners, video editing, video viewing) q To study alone q To study with a group q To use the group study rooms q To use a printer or photocopier q To use the computers for academic purposes q To use the computers for recreational/personal use q To use the Mac lab (Apple Macintosh Computers) q To visit the Help Desk q To visit the Academic Resource Center (ARC) q To visit the Merton Center q To use a laptop q To meet friends q To look for information in online databases (EBSCOhost, ProQuest and so on) q Other (please specify) If you never use the library, why don’t you? 374 College & Research Libraries May 2016 Please rate the following: On average, how often do you use the library in person? Daily 2 to 4 times a week Once a week 2 to 3 times a month Once a month or less Never On average, how often do you access library materials, services, and databases (such as ProQuest and EBSCOhost) without visiting the library? Daily 2 to 4 times a week Once a week 2 to 3 times a month Once a month or less Never On average, how often do you use the 24-hour study room after regular library hours? Daily 2 to 4 times a week Once a week 2 to 3 times a month Once a month or less Never Notes 1. United States Department of Education, A Test of Leadership: Charting the Future of U.S. Higher Education: Report of the Commission Appointed by Secretary of Education Margaret Spellings (Washington, D.C.: U.S. Dept. of Education, 2006). 2. Michael F. Middaugh, Planning and Assessment in Higher Education: Demonstrating Institu- tional Effectiveness (San Francisco: Jossey-Bass, 2010). 3. Joseph R. Matthews, Library Assessment in Higher Education (Westport, Conn.: Libraries Unlimited, 2007). 4. David Shulenburger, “The Relationship between University and Library Assessment” (paper presented at the Library Assessment Conference, Baltimore, 2010), 4. 5. Megan Oakleaf, The Value of Academic Libraries: A Comprehensive Research Review and Report (Chicago, Ill.: Association of College and Research Libraries, 2010), 30. 6. Jon R. Hufford, “A Review of the Literature on Assessment in Academic and Research Libraries, 2005 to August 2011,” portal: Libraries and the Academy 13, no. 1 (2013): 5–35. 7. Oakleaf, The Value of Academic Libraries. 8. Jane Hiscock, “Does Library Usage Affect Academic Performance?” Australian Academic and Research Libraries 17, no. 4 (1986). 9. R.R. Powell, “Impact Assessment of University Libraries: A Consideration of Issues and Research Methodologies,” Library & Information Science Research 14, no. 3 (1992). 10. Ethelene Whitmire, “Academic Library Performance Measures and Undergraduates’ Library Use and Educational Outcomes,” 24, no. 2 (2002). 11. Shun Han Rebekah Wong and T.D. Webb, “Uncovering Meaningful Correlation between Student Academic Performance and Library Material Usage,” College & Research Libraries 72, no. 4 (2011): 361–70. 12. Shun Han Rebekah Wong and Dianne Cmor, “Measuring Association between Library Instruction and Graduation GPA” College & Research Libraries, 72 no. 5 (2011): 464–73. 13. Graham Stone, David Pattern, and Bryony Ramsden, “Library Impact Data Project,” Sconul Focus 54 (2012): 25–28. 14. Krista M. Soria, Jan Fransen, and Shane Nackerud, “Library Use and Undergraduate Stu- dent Outcomes: New Evidence for Students’ Retention and Academic Success,” portal: Libraries and the Academy 13, no. 2 (2013): 147–64. 15. Derek Rodriguez, “Answering Questions About Library Impact on Student Learning,” in In the Library with the Lead Pipe (2012), available online at www.inthelibrarywiththeleadpipe. Investigating the Relationship of Library Usage to Student Outcomes 375 org/2012/answering-questions-about-library-impact-on-student-learning/ [accessed 3 September 2012]. 16. Margie Jantti and Brian Cox, “Measuring the Value of Library Resources and Student Academic Performance through Relational Datasets” (paper presented at the Library Assessment Conference, Baltimore, 2010), 525–32. 17. Association of College and Research Libraries, “Assessment in Action: Academic Libraries and Student Success” (2013). 18. Jim Self and Steven Hiller, “Two Decades of User Surveys: The Research Library as Seen through the Eyes of Its Customers,” in 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, eds. Ian Hall, Stephen Thornton, and Stephen Town (York, UK: University of York 2011), 377–83. 19. Laurie Allen et al., “Analyzing the Miso Data: Broader Perspectives on Library and Com- puting Trends,” Evidence Based Library and Information Practice 8, no. 2 (2013): 129–38. 20. Ying Zhong and Johanna Alexander, “Academic Success: How Library Services Make a Difference,” in ACRL Thirteenth National Conference (Baltimore 2007), 141–57. 21. Alexander W. Astin, “The Methodology of Research on College Impact, Part One,” Sociol- ogy of Education 43, no. 3 (1970): 223–54. 22. Oakleaf, The Value of Academic Libraries. 23. Joseph R. Matthews, “Assessing Library Contributions to University Outcomes: The Need for Individual Student Level Data,” Library Management 33, no. 6 (2012): 389–402. 24. National Survey of Student Engagement, “NSSE Data User’s Guide 2014” (Bloomington, IN: Indiana University, 2014), 9. 25. Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes.” 26. Graham Stone and Bryony Ramsden, “Library Impact Data Project: Looking for the Link between Library Usage and Student Attainment,” College & Research Libraries 74, no. 6 (2013): 546–59. 27. George D. Kuh et al., “Unmasking the Effects of Student Engagement on First-Year College Grades and Persistence,” Journal of Higher Education 79, no. 5 (2008): 540–63; Ernest T. Terenzini and Patrick T. Pascarella, How College Affects Students: A Third Decade of Research (San Francisco: Jossey-Bass, 2005). 28. Wong and Webb, “Uncovering Meaningful Correlation.” 29. Shane Nackerud et al., “Analyzing Demographics: Assessing Library Use across the Institu- tion,” portal: Libraries and the Academy 13, no. 2 (2013): 131–45.