Introduction

At Bellarmine University the administration is creating a culture of assessment. As part of this effort, Bellarmine University Library conducts user surveys of faculty or students as a matter of course, starting in 2007. These surveys are a direct result of the need to provide assessment measures to outside stakeholders, in this case for the Southern Association of Colleges and Schools accreditation review. They are also used as a focused supplement to benchmark studies that had already been conducted. In many ways this practice reflects the state of traditional library assessment, in that it is generally input/output oriented and, more recently, operationally oriented. In both of these assessment efforts, benchmarks of inputs/outputs or user satisfaction surveys, whether locally or nationally (e.g., LibQUAL), the assessment has remained focused on the library and its internal operations. They do not provide particularly compelling evidence for other interested parties.

The library’s satisfaction survey is used to gauge the effectiveness of the library’s operations as seen by its users. Traditional library assessment has little correlation with the outcomes and positive work of the institution; rather it provides data that indicates strengths and weaknesses on the library’s part that the library administration can then take efforts to further develop or correct as the case may be. Operationally these surveys can be helpful to a library administration; they provide valuable data that the library and its resources and services are being used and even appreciated. However, they do not demonstrate to institutional stakeholders the impact of the library on institutional goals and objectives.

Increasingly, academic libraries are being asked to provide data and assessments that demonstrate the library’s connection with the institution’s desired outcomes. They are facing increasing demands for accountability to a broader audience than ever before. Libraries have moved from being the only information resource to being a quality information resource. In the current economic climate, parent institutions seek to save every dollar they can and face difficult budget choices, often including decreased library budgets. Meanwhile, the political climate calls for increased accountability and greater connections between the university’s efforts and student learning and student success. This is the changing environment faced by higher education institutions and therefore academic libraries.

In response, librarians are more frequently seeking to identify areas where they can demonstrate a relationship exists between a student’s library interactions and student outcomes, in contexts relevant to a wider audience at the university and beyond. To accomplish this Oakleaf (2010) indicates that the library must link its data to individual students. Matthews (2012) has likewise recently noted that success in efforts to link the library and student outcomes will require the use of individual student data. One of the reasons that student unit data is such a constant issue in these efforts by librarians is that generally libraries do not track what resources are used by which students, or how students use library resources and services. As a matter of professional ethics, personally identifiable information is often deleted or not even collected to maintain an individual’s privacy. So while offices of institutional research have access to large amounts of individually identifiable outcome data, most libraries do not.

The need for library assessment efforts that show the influence of the library in terms of institutionally relevant outcomes, such as student learning and student success, is highlighted in the ACRL report The Value of Academic Libraries (Oakleaf, 2010). Much work and many attempts have been made to demonstrate the library’s direct impact on student learning and other student outcomes. Recent work includes studies by Wong and her colleagues to examine the connections between library material usage and student GPA (Wong & Webb, 2011), and library instruction and graduating GPA (Wong & Cmor, 2011). Emmons and Wilkinson (2011) examine the library’s impact on student persistence. Tenopir (2012) reports on multiple methods being used to try and measure library value in her status report on the lib-value project. More thought and effort has been put forth to developing appropriate assessment measures that address the library’s impact on students, particularly the impact on student outcomes and student learning. Oakleaf (2011) addresses the challenges librarians face in trying to assess the library’s connection with student learning and outcomes. Rodriguez (2011) reports on his efforts to develop one such tool, the protocol for Understanding Library Impacts, to show library impact on student learning, and his first results are promising (Rodriguez, 2012). In the UK, the Library Impact Data Project has been examining the issue of developing appropriate measures to assess the library’s role in student learning and outcomes, finding significant correlation between library usage and student attainment of the final degree (Stone, Pattern, & Ramsden).

Aims

This exploratory study seeks to identify areas where relationships exist between a student’s library usage and relevant student outcomes at Bellarmine University. The two primary aims are to see if an operationally oriented user survey can be used to provide evidence of the library’s role in institutionally important student outcomes, and to move beyond the simple, one to one, correlations and use a regression model that provides a bigger picture with multiple variables to determine if, when significant demographic factors are controlled, library factors are still significant in student outcomes.

Methods

Like many academic libraries, Bellarmine University library uses a student user survey as part of its assessment efforts. Between 2007 and 2010 it conducted three such surveys, and now does one every other year. These surveys provide data on why students come to the library and how often they use the library. The library survey asked questions in two groupings (see Appendix A). The first grouping asks the reasons that students come to the library, with a list of 18 possible reasons with checkboxes. Respondents were asked to check all reasons that applied. The second group of questions requests information on how often a student came to the library and how often the student used the library online, with response options ranging from daily to never. Most importantly, the survey is not anonymous. The library can identify how students report using the library at an individual level.

Researchers paired the library survey data with the university’s Office of Institutional Research student data to first determine if there were any significant relationships between a student’s self-reported library usage and known student outcomes. They used a logistic regression of all the library input variables against selected student outcomes: retention, graduation, and GPA. They also determined that the data would be viewed in undergraduate class-based cohorts to minimize variations among the respondent experiences. In addition to the library variables, researchers tested a number of demographic factors drawn from institutional research data against selected student outcomes to identify the most significant demographic factors for the individual cohorts. Using these control factors and the library variables the researchers developed three research questions:

Does library usage influence whether a freshman student returns in his/her next year of undergraduate study?
Does library usage influence whether a freshman student graduates within four or five years in undergraduate study?
Does library usage influence cumulative GPA for freshmen and seniors?

Undergraduate students participated in library surveys in the years 2007, 2008 and 2010. Bellarmine University is a small, private, Catholic university located in Louisville, Kentucky. Total university enrollment is approximately 2,000 undergraduate students and nearly 800 graduate students. There were over 1,000 students living in the residence halls. The institution currently offers over 50 undergraduate bachelor degree programs and over 20 graduate programs. Over 80% of Bellarmine’s undergraduate students attend full-time and are under 25 years of age.

Bellarmine University has an incoming class of about 600 students every fall. The following demographic and academic preparedness measures are consistent across freshman cohorts:

24-25 ACT composite average
65% from Kentucky, 35% out of state
40% first-generation students (defined as neither parent earning a bachelor’s degree)
20-30% Pell Grant recipients
10-15% students of colour
1% international students

Due to these similar demographics across the student body, researchers collapsed the survey responses for 2007, 2008, and 2010 and studied the data as class-based cohorts.

Due to the limited number of undergraduate students at the institution, sampling was unnecessary because the entire population could easily be requested to participate. On designated years, the institution invited the undergraduate populations to participate in the library survey. Most members of the freshmen and senior populations start at the university as first-time, traditional age freshmen; however, all freshman and senior students were invited to participate in the survey, regardless of whether they enrolled as traditional freshmen, transfer students, or re-admitted students. The response rate for the three surveys ranged from a low of 20% to a high of 26%.

Results

The variables considered, Full-time/Part-time Status, Session GPA, Race, and ACT composite score, either singly or in combinations, were identified by a logistic regression as having a significant relationship to outcome variables, whether a student graduated or the cumulative GPA (p<.05). These variables were controlled in our analyses to consider whether library usage variables independently influenced student outcomes (see Appendix B).

Analysis One: Second-year Retention of Freshman Students

Does library usage influence whether a freshman student returns in his/her next year of undergraduate study?

Researchers conducted a forward entry logistic regression using freshmen student data only (n=370), considering any significant library variables related to outcome variable, retained in a students’ second year. There were 336 freshmen returning in the second year (91%) and 34 students did not (9%). The omnibus test of model coefficients was significant, with a chi-square test result of 8.227 (p<.05). One library variable was determined to be a significant positive predictor of returning for the second year: Access library online (see Table 1). The Nagelkerke R-squared was 0.048.

Researchers conducted second logistic regression in two steps. First, by identifying the control variables Full-time/Part-time Status, ACT composite score, and Session GPA, which were selected because correlational analysis revealed significance relationship to Return (p<.05). Second, they used a forward entry technique to consider library variables after the control variables. The omnibus test of model coefficients was significant, with a resulting chi-square of 31.021 (df=3, p<.001). They found that Session GPA significantly predicted Return. After the control variables were entered, the library variable Access library online again significantly predicted whether a student returned in their second year using the same forward entry method (p<.05) (see Table 2). The Nagelkerke R-squared was 0.206.

Table 1

Significant Library Variables Related to Retention


Measure	B	S.E.	Wald	Df	Sig.	Exp(B)
Access library online	.432	.158	7.504	1	.006	1.54
Constant	1.242	.388	10.236	1	.001	3.463

Table 2

Significant Control and Library Variables Related to Retention


	B		S.E.	Wald	Df	Sig.	Exp(B)
FT or PT		21.868	40192.7	.000	1	1	.000
ACT		.074	.067	1.238	1	.266	1.077
Session GPA		.998	.224	19.859	1	.000	2.712
Access library online		.311	.157	3.947	1	.047	1.365
Constant		18.721	40192.7	000	1	1	1.4E+08

Table 3

Significant library variables related to graduation

	B	S.E.	Wald	df	Sig.	Exp (B)
Access Library Online	.245	.117	4.411	1	.036	1.277
Constant	.240	.335	.513	1	.474	1.271

Analysis Two: Graduation of Freshman Students from the Same Institution

Does library usage influence whether a freshman student graduates within four or five years in undergraduate study?

Researchers conducted a forward entry logistic regression using freshmen student data only (n=220, after combining freshmen from surveys in 2007 and 2008), considering any library variables related to outcome variable to identify any significant library variables related to graduation. This included respondents who graduated within four years for 2008 freshman surveys or within five years for 2007 freshman surveys. Respondents were predominantly traditional-aged freshmen, and primarily full-time students. There were 156 freshmen that graduated (71%) and 64 students did not (29%). The omnibus test of model coefficients was significant, with a resulting chi-square of 4.584 (p<.05). Researchers found that one library variable, Access library online (p<.05), was a significant predictor of graduation It was a positive predictor, and confirmation of this was associated with students graduating (see Table 3). The Nagelkerke R-squared was .029.

Researchers conducted a second logistic regression in two steps. First using the following control variables: Full-time/Part-time status, ACT score, Session GPA, and Race, which were selected because correlational analysis revealed significance relationship to Graduate (p<.05). Second, using a forward entry technique to consider library variables after including the demographic control variables. The omnibus test of model coefficients was significant, with a chi-square result of 37.943 (p<.001). After entering the control variables, the library variable Access library online still significantly predicted whether a student graduated, using the forward entry method (p<.05) (see Table 4). The Nagelkerke R-squared was 0.255.

Analysis Three: First-year Cumulative GPA

Does library usage influence cumulative GPA for freshmen?

Researchers conducted an analysis of freshman student data only (n=370), considering any significant variables correlated with the outcome variable Cumulative GPA. They employed a forward entry ordinary least squares (OLS) regression to consider what variables, if any, predict Cumulative GPA. Researchers identified two significant library variables, Study alone and Use of printer or photocopier, as positive predictors, for agreement with them was associated with a higher Cumulative GPA. The adjusted R-squared was .030. Controlling for the significant non-library variable, ACT Composite, entering it in the model before library survey items, the R-squared is .15 for the ACT variable; however Study alone and Use of printer and photocopier still present themselves as significant factors for Cumulative GPA (see Tables 5 and 6).

Table 4

Significant Control and Library Variables Related to Graduation

	B		S.E.	Wald		df	Sig.	Exp(B)
FT or PT		22.999	40193.2	.000		1	1	9.74E+09
ACT		0.065	0.054	1.44		1	0.23	1.067
Session GPA		1.09	0.282	14.892		1	0	2.974
Race		0.954	0.437	4.766		1	0.029	2.595
Access library online		0.262	0.131	3.986		1	0.046	1.299
Constant		-28.586	40193.2		.000	1	0.999	.000

Table 5

Control and Library Variables Related to First-year GPA

				Std. Error of the Estimate	Change Statistics
Model	R	R Square	Adjusted R Square	Std. Error of the Estimate	R Square Change	F Change	df1	df2	Sig. F Change
ACT	0.396	0.157	0.154	0.561	0.157	67.240	1	362	0.000
ACT, study alone	0.426	0.182	0.177	0.553	0.025	11.014	1	361	0.001
ACT, study alone, use printer or photocopier	0.437	0.191	0.184	0.551	0.009	4.005	1	360	0.046

Additional library survey items were not significant in the model, but had significant positive correlation with students’ cumulative GPA at the end of the their freshman year (p<.01):

· Access library online

· Use computer for academic purposes

· Consult a reference librarian

Analysis Four: Senior Cumulative GPA

Does library usage influence cumulative GPA for seniors?

Researchers conducted an analysis of senior student data only (n=360), considering any significant library variables correlated with the outcome variable Cumulative GPA. Researchers used a forward entry ordinary least squares (OLS) regression to consider what variables, if any, predict Cumulative GPA. They identified two significant library predictors: Check out books and Use of group study rooms. Check out books was a positive predictor, and confirmation of this was associated with a higher Cumulative GPA. Use of group study rooms was a negative predictor. The adjusted R-squared was .045. When the control variable ACT Composite is entered in the model before library survey items, the R-squared is .275 for the ACT variable; however while Check out books still presents as significant, Use of group study room is no longer a significant factor (See Tables 7 and 8).

Table 6

First-year GPA Model Coefficients

	Unstandardized Coefficients		Standardized Coefficients
Model	B	Std. Error	Beta	t	Sig.
(Constant)	1.443	0.218		6.617	0.000
ACT	0.071	0.009	0.396	8.200	0.000
(Constant)	1.243	0.223		5.565	0.000
ACT	0.070	0.009	0.391	8.198	0.000
study alone	0.265	0.080	0.158	3.319	0.001
(Constant)	1.201	0.224		5.372	0.000
ACT	0.069	0.008	0.385	8.099	0.000
study alone	0.231	0.081	0.138	2.845	0.005
use printer or photocopier	0.133	0.066	0.097	2.001	0.046

Table 7

Control and Library Variables Related to Senior GPA

				Std. Error of the Estimate	Change Statistics
Model	R	R Square	Adjusted R Square	Std. Error of the Estimate	R Square Change	F Change	df1	df2	Sig. F Change
ACT	0.525	0.275	0.273	0.353	0.275	104.147	1	274	0.000
ACT, check out books	0.540	0.292	0.287	0.350	0.016	6.333	1	273	0.012

Table 8

Senior Year Cumulative GPA Model Coefficients

	Unstandardized Coefficients		Standardized Coefficients
Model	B	Std. Error	Beta	t	Sig.
(Constant)	1.780	0.155		11.456	0.000
ACT	0.063	0.006	0.525	10.205	0.000
(Constant)	1.743	0.155		11.276	0.000
ACT	0.062	0.006	0.513	10.029	0.000
Check out books	0.111	0.044	0.129	2.516	0.012

Discussion

The analysis indicates that student usage of the library manifests itself in a number of ways, including use as an academic or information resource, use as an information service, as a place to study alone or in a group, a place to use equipment available in the library, and for social reasons. Many of these have previously appeared in the library literature as recognizable parts of a library’s offerings to the university community. The results illustrate that library factors consistently show a significant relationship to student outcome variables, even when control factors are considered.

The study’s findings confirm other studies that link the use of library services and resources to a student’s learning and success outcomes. Wong and Webb (2011) identified a single aspect of library usage (checking out books) that correlates with a student outcome (GPA). For seniors, checking out books was a significant predictor in the model for their GPA. Emmons and Wilkinson (2011) identify that the ratio of professional staff to full-time students has an impact on student persistence. This study also found that the library has an influence on freshman retention and graduation, through the use of online resources.

In examining the library’s influence on student success outcomes of second-year retention and graduation, this study indicates that the library’s support does favourably influence student success. In looking at second-year retention of freshmen, accessing the library online was identified as the most significant library factor and even after controlling for significant demographic variables, this factor was still identified by the model as significant influence on retention. For freshman students the same factor influenced both retention and graduation: the availability of online access to the library. Freshman students that accessed the library online more frequently were more likely to return for their second year and to graduate. Using the library’s academic information resources appears to favourably influence a freshman’s student success outcomes.

In looking at student learning outcomes represented by GPA, the results again indicate that the library does have a consistent and positive influence on a student’s GPA, though specific library factors change depending on where in their academic career the student is located. For first-year students the library’s influence appears through two variables that highlight the library as a place: providing a place to study alone and as a place that has specialized equipment available to students. In a sense, as new students they are getting acclimated to the place of the library in their academic efforts. Interestingly, the impact of studying alone repeats a finding by Arum and Roska (2011) showing that time spent studying alone is more academically beneficial than time spent studying in groups. Seniors cumulative GPA was influenced by the library differently than first-year students. Checking out books had a positive impact on senior’s GPA. Interestingly Using group study rooms, which had a negative effect when considering the library variables alone, was not significant when control variables were entered. The consistent presentation of library variables after the use of significant control variables shows that library factors do have significant connections to student outcomes and success.

Limitations

Researchers employed a number of different non-library control variables in this study, but there are many that were not considered. The researchers wanted a “bigger picture” view of the library’s relationship with student outcomes more than a one-to-one correlation, but this is not intended to imply causation. It should be noted that this study relies on self-reported data, the value of which has recently been challenged (Porter, 2011). It is also worth noting that this study focuses on acquiring factual information on specific activities of the students. This study does not seek information on students’ perception of ability or growth over time, merely what students were doing the year the survey was taken. It was designed within the parameters outlined by Gonyea (2005) as an appropriate means of creating and using self-reported data.

Conclusion

Not surprisingly, not all correlations identifying the library’s influence on student outcomes were found to be significant when using regression analysis to further examine the relationship in conjunction with non-library control variables. The study confirms previous correlation studies that identify the library as contributing to student outcomes, and indicates that the library’s relationship with student outcomes is not eliminated by control variables. The library has an influence on first-year and senior student outcomes but does not affect them in the same manner. The study indicates that the library provides support for students in a number of ways: as a resource, through services, and as a place. All of these aspects of the library’s operations are shown to be influential on student outcomes, but no single aspect is consistent from year to year as an individual students progress through their studies. Rather, as individual students develop they seem to rely on different aspects of the library offerings and their use of the library in order to succeed.

This study indicates that the library does have an influence on freshman and senior level outcomes, whether expressed as learning outcomes represented by cumulative GPA, or more typical student success outcomes, such as retention and graduation rates. This is true even when controlling for certain demographic characteristics, including the student’s ACT score, whether the student is part-time or full-time, and their session GPA. The factors that impact a student’s outcomes changes depending on where in their academic career a student is located. These statistical analyses provide significant evidence for the value provided by the library in support of institutionally important student outcome goals. Additional study into what students identify as different library factors over time would be useful. Another area for further research would be to consider if there are any differences in library usage factors based on the student’s field of study.

References

Arum, R., & Roksa, J. (2011). Academically adrift: Limited learning on college campuses. Chicago, Ill: University of Chicago Press.

Emmons, M., & Wilkinson, F. C. (2011). The academic library impact on student persistence. College & Research Libraries, 72(2), 128-149. http://dx.doi.org/10.5860/crl-74r1

Gonyea, R. M. (2005). Self-reported data in institutional research: Review and recommendations. New Directions for Institutional Research, 2005(127), 73-89. http://dx.doi.org/10.1002/ir.156

Matthews, J. R. (2012). Assessing library contributions to university outcomes: The need for individual student level data. Library Management, 33(6/7), 389-402. http://dx.doi.org/10.1108/01435121211266203

Oakleaf, M. (2010). The value of academic libraries: A comprehensive research review and report. Chicago, IL: Association of College and Research Libraries.

Oakleaf, M. (2011). Demonstrating the value of the library: Assessment tools & techniques. Paper presented at the American Library Association annual conference New Orleans, LA. http://meganoakleaf.info/ala2011valuepanel.pdf

Porter, S. R. (2011). Do college student surveys have any validity? Review of Higher Education, 35(1), 45-76. doi: http://dx.doi.org/10.1353/rhe.2011.0034

Rodriguez, D. (2011). Understanding library impacts on student learning. In the Library with the Lead Pipe. Retrieved from: http://www.inthelibrarywiththeleadpipe.org/2011/understanding-library-impacts-on-student-learning/

Rodriguez, D. (2012). Answering questions about library impact on student learning. In the Library with the Lead Pipe. Retrieved from: http://www.inthelibrarywiththeleadpipe.org/2012/answering-questions-about-library-impact-on-student-learning/

Stone, G., Pattern, D., & Ramsden, B. (2011). Does library use affect student attainment? A preliminary report on the library impact data project. LIBER Quarterly, 21(1), 5-22. doi: http://liber.library.uu.nl/index.php/lq/index

Tenopir, C. (2012). The lib-value project: Status report. Paper presented at the American Library Association annual conference, Anaheim, CA. http://libvalue.cci.utk.edu/sites/default/files/ARL%20ALATenopir20126%2022%20REV%20Final.pdf

Wong, S. H. R., & Cmor, D. (2011). Measuring association between library instruction and graduation GPA. College & Research Libraries, 72(5), 464-473. doi: http://dx.doi.org/10.5860/crl-151

Wong, S. H. R., & Webb, T. D. (2011). Uncovering meaningful correlation between student academic performance and library material usage. College & Research Libraries, 72(4), 361-370. http://dx.doi.org/10.5860/crl-129

Appendix A

2010 Library Usage Survey

Reasons you come to the library (check all that apply):

To check out books

To check out media (CDs, DVDs, etc.)

To locate journal/newspaper articles

To get help with research papers or other course assignments

To read newspapers or current magazines

To use items (books/articles/videos) placed on reserve by your professor

To use media equipment (e.g. video cameras, digital cameras, scanners, video editing, video viewing)

To study alone

To study with a group

To use the group study rooms

To use a printer or photocopier

To use the computers for academic purposes

To use the computers for recreational/personal use

To use the Mac lab (Apple Macintosh Computers)

To visit the Help Desk

To visit the Academic Resource Center (ARC)

To visit the Merton Center

To use a laptop

To meet friends

To look for information in online databases (EBSCOhost, ProQuest etc.)

Other (please specify)

If you never use the library, why don't you?

Please rate the following:


On average, how often do you use the library in person?	Daily	2 to 4 times a week	Once a week	2 to 3 times a month	Once a month or less	Never

On average, how often do you access library materials, services and databases (such as ProQuest and EBSCOhost) without visiting the library?	Daily	2 to 4 times a week	Once a week	2 to 3 times a month	Once a month or less	Never

Appendix B

Variables considered

Demographic and Academic Preparedness

ACT Composite (max. 36)
1^st Generation College Student (Y/N)
Session Grade Point Average (GPA)
Pell recipient (Y/N)
Athlete (Y/N)
Race (white, black, Hispanic, Pacific Islander/Native Hawaiian, Asian, American Indian/Alaska Native, Multi-race, Unknown)
Sex (F, M)
High School Type (public or private)
Radius (miles from home)
Full-time or Part-time Student

Library Usage Variables

Check out books
Locate articles
Read newspapers or magazines
Use items on reserve
Study alone
Study with a group
Use the group study rooms
Use printer or photocopier
Use computers for academic purposes
Visit the Help Desk
Visit the Academic Resource Center (ARC)
Visit the Merton Center
Use a laptop
Use the computers for personal use
Meet friends
Use library at all
Use library in person
Access library online

Outcome Variables

Graduated in four or five years (Y/N)
Returned in the second year (Y/N)
Cumulative GPA in freshman year (0-4.0)
Cumulative GPA in senior year (0-4.0)