459 Library Usage, Instruction, and Student Success across Disciplines: A Multilevel Model Approach Linda L. Anderson and Susan A. Vega García* To better understand the library’s role in student success, this research used student survey data to explore students’ reported library building use, library resource use, library instruction, as well as student perceptions of the library’s role in their success, and how these factors may vary by academic discipline. Cumulative GPAs were also matched with respondents. Quantifying the relationship between library usage and student success is one of the six areas recommended for further research by ACRL. This study focuses on the variation and the uncertainty of the measurement of this relationship across disciplines, using Bayesian multilevel regression methods. Lev- els of library resource usage and percentages of respondents believing the library contributes very much to the respondents’ academic success vary quite a bit by discipline. In this study, cumulative GPA is higher on average for students who use library resources more frequently, but not for students who receive library instruction. Although, for undergraduates, higher frequency of building usage predicted higher probability of believing the library contributes very much to academic success, it did not predict higher GPA. Introduction Libraries have long been interested in analyzing user behaviors and attitudes toward library use. The idea of measuring library service impact as a means of communicating library value to stakeholders gained significant ground in 2010 through the release of ACRL’s Value of Aca- demic Libraries report.1 That report recommended that academic libraries seek to track “library influences on increased student achievement,” and specifically to seek ways to link ostensibly objective indicators of success, such as student grade point averages and test scores, with their library usage experiences. In addition, the report noted heavy investments from academic li- braries in library instruction to students but a general lack of programwide and longitudinal assessments that measured real learning resulting from those instruction sessions that might collectively in turn communicate library value more concretely to external stakeholders.2 An additional area of interest from the report focused on seeking to measure students’ experiences with the library, including their “attitudes, and perceptions of quality” through the collection of data. * Linda L. Anderson is Library Assessment Specialist in Parks Library at Iowa State University; email: landerso@ iastate.edu. Susan A. Vega García is Assistant Dean for Inclusion and Equity in Parks Library at Iowa State Univer- sity; email: savega@iastate.edu. ©2020 Linda L. Anderson and Susan A. Vega García, Attribution-NonCommercial (https://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:landerso@iastate.edu mailto:landerso@iastate.edu mailto:savega@iastate.edu https://creativecommons.org/licenses/by-nc/4.0/ 460 College & Research Libraries April 2020 Quantifying the relationship between library usage and student success is one of the six areas recommended for further research in the 2017 ACRL report, Academic Library Impact: Improving Practice and Essential Areas to Research.3 While many studies have focused on estimat- ing an average relationship across disciplines, while adjusting for disciplinary differences, this study focuses instead on the variation and the uncertainty of the measurement of this relationship across disciplines, providing a fuller set of information than averages, and one that reflects reality better. This approach should help replication when applied in studies at multiple universities. For example, averages at universities focused on science and technol- ogy may differ a great deal from averages at universities focused on humanities and social sciences. Describing the relationship as it varies based on group affiliation is accomplished by multilevel regression modeling, which combines a model for the individuals with a model for the groups. The group-level information derived from this study can be used for two purposes: map- ping the correlation between library instruction, library resource use and positive attitudes and cumulative GPA; and communicating the value of library instruction and resource use to university administration and faculty, who have observed the relationship between the library and their students within their own discipline, but perhaps not so much for other disciplines. It may be more convincing to see results that reflect their own experience, and it is more informative to study the variations than to limit the results to an average. Although the results from most published work about the impact of library usage and instruction center around statistical significance, this paper follows the recommendations of the American Statistical Association4 to present estimates with measures of uncertainty and of McShane, Gal, Gelman, Robert, and Tackett5 to “abandon statistical significance.” Literature Review Subsequent to the publication of the Value of Academic Libraries Report, an increasing number of library service impact studies began to emerge. In one early study, Wong and Webb6 sought empirical evidence of the positive impact of library service usage on student GPA. The research- ers conducted a massive study of more than 8,500 students at Hong Kong Baptist University that analyzed students’ graduation GPA and the number of times these students had checked out material from the library. Results were organized by academic discipline and student rank (undergraduate or graduate), with the finding that 65 percent of the academic subject areas considered showed positive correlations with library checkouts and higher graduation GPA. Subsequent research7 has also concluded that book checkout, library database usage, and e-journal access/usage have positive correlations with student GPA and other forms of academic achievement. It is noteworthy that these library-use indicators are all student self- directed library research activities that require engagement, activity, and often some sort of procedural follow-through from the student. Other library value studies have often focused on the relationship of library instruction with student success, but with varying results. Wong and Cmor8 investigated the relation- ship of attending discipline-relevant library instruction workshops and student graduation GPA, using data from a previous study9 of 8,500 students at Hong Kong Baptist University. Overall, this research found that results varied by student academic disciplines and by num- ber of instruction workshops attended. Data from students in only 24.4 percent of the subject area groups considered showed positive correlations (at varying levels of significance) with a Library Usage, Instruction, and Student Success across Disciplines 461 higher graduation GPA. The researchers also found that attending more sessions of discipline- relevant library instruction increased the likelihood of a stronger positive correlation with higher GPA at graduation. In a study analyzing 4,489 transcripts, Bowles-Terry10 found a very small yet statistically significant positive difference in GPA among students who had been enrolled in an upper- level course that received course-related library instruction (CRI) and whose attendance at that instruction session was inferred, while Soria, Fransen, and Nackerud11 found that prob- able CRI attendance by 5,368 first-year undergraduates was correlated with a lower GPA; little explanation was given for this outcome other than noting the difficulties of assessing instruction effectiveness. However, a subsequent study by Soria et al.12 used different analysis methods and found opposite results, that “students who engaged in libraries instruction” by attending library workshops, CRI, or a library class had a significantly higher GPA than did those students who had not been involved in library instruction. Gaha, Hinnefeld, and Pellegrino13 sought to normalize data to account for probable disciplinary grading variations (a multilevel model would make this step unnecessary, even with a small data set). While they found some positive correlations between higher normal- ized GPA among students who had taken at least one course with library instruction, again there was no control for whether a student had actually attended that instruction session, so conclusions are somewhat speculative. Krieb14 analyzed the impact of CRI and reference desk usage on student retention and GPA at a community college, finding some positive impact of CRI completion on retention but minimal impact on GPA. Statistical Approaches in Previous Studies Many big data library value studies have relied heavily on identifying correlations and signifi- cance to show which library activities, behaviors, resources, or service usage seem to matter most in terms of supposedly objective student success variables such as GPA. Most of the analyses accounted for the average differences in GPA across majors or colleges. Soria et al.15 did so by allowing varying intercepts for each college but did not allow for varying relation- ships with the library (different regression slopes for the library usage variables). Two early library value studies recognized that not only the average GPA varies by major, but that different majors have different relationships with the library;16 for that reason, these studies conducted analyses (Pearson correlation and chi-square tests, respectively) for each discipline group separately. Because some discipline groups had few students, these groups were left out of the analysis. A multilevel regression could have combined both of these analy- ses into one, while allowing use of data from all discipline groups and using the number of library sessions per discipline as a group-level predictor. Another challenge faced in a number of big-data instruction-value studies is how to determine whether a student has indeed received library instruction. One common method has been to cross-check the institution’s student course enrollment data with library instruc- tion program records of CRI sessions for that course,17 while other studies18 may be able to verify completion of an initial mandatory library orientation session but not attendance at subsequent and potentially more relevant CRI sessions. These methods do not guarantee that the student was actually present at a CRI session. In addition, some researchers state that instruction program records may not be complete and are particularly problematic for high-enrollment, multisection courses for which CRI is offered for some but not all sections. 462 College & Research Libraries April 2020 Krieb19 described an alternate method of asking students after CRI sessions for their verbal consent to be tracked for research purposes. This method allowed solid identification of both the course and CRI completion, as well as close tracking of individual student retention and GPA success measures. While this method ensures that a student did receive CRI, larger col- lege and university libraries may find it difficult to replicate the individual student consent process for reasons of scale. Methods After IRB approval, a survey including undergraduate and graduate students was conducted in 2016 for the purpose of service improvement. Information about the development of this survey, based on the previous 2012 survey, are available in the reports.20 The survey sample was stratified on college and student type (undergraduate or graduate) combination, with higher sampling rates specified for smaller strata. Within each stratum, a systematic sample was drawn, sorting by gender and residency (Iowa, non-Iowa US, International) and using a random start, selecting each nth student to reach the specified percentage for that stratum. All regression models included the college and student type sampling strata. Survey sample size and response rates are given in table 1. Details of the statistical analyses are in appendix A. Three outcome variables were modeled: 1. Students’ library resource usage index (LRU) calculated from responses to survey questions about frequency of use of various library resources and services 2. Students’ perception of library contribution to academic success (LCS) 3. Students’ cumulative GPA Library Resource Usage Index (LRU) The LRU index was constructed from survey questions (see appendix B) about the frequency of use of the library website, the library search, article indexes and databases, electronic jour- nals and articles, e-books, physical items (examples: books, DVDs, CDs, maps, microforms), and of materials from Special Collections and University Archives. Each of these survey questions had response categories of daily, weekly, monthly, once a semester, less often, and never. The LRU, simulating an estimate of a total average monthly library usage count was created by equating survey responses to estimated counts, as depicted in table 2. These estimated counts were summed over the set of questions, with types of library resources weighted equally. If a student did not answer at least five of these questions, the case was deleted. Sixteen cases were deleted, leaving 1,595 cases for analysis. TABLE 1 Survey Sample Size and Response Rates Undergraduate Students Graduate Students Postdocs Eligible Sample 2,952 3,451 318 Completed Surveys 494 1,116 117 RESPONSE RATE 16.7% 32.3% 36.8% Library Usage, Instruction, and Student Success across Disciplines 463 Evidence that this is a fairly reliable way to estimate the frequency of usage comes from a previous analysis where an estimate of gradu- ate and undergraduate website usage derived in this way was similar to an estimate of under- graduate and graduate website usage derived from a time series regression of web log data.21 The LRU should be regarded as an index or ar- tificial construct. Summing the estimates from the individual resource questions may inflate an estimate of total resource usage; for instance, the respondent may use the library website to find a database, which is then used to find an article, counting the use three times. Perception of Library Contribution to Success (LCS) The LCS variable is the binary outcome variable constructed from the response to this ques- tion: “To what extent have the library’s resources and services contributed to your academic success?” A response of “very much” was set to 1, and missing or other responses (“some,” “very little,” or “not at all”) were set to 0. Most responses were either “very much” or “some.” Making it a binary variable was a way to simplify the analysis. Since not many answered “very little” or “not at all,” we focused on the respondents who answered “very much.” Cumulative GPA There are fewer observations (1,551) for the cumulative GPA analyses22 than for the other analyses because the GPA data was requested retroactively in 2018 and not all cases were matched. The average of the GPAs of the undergraduate survey respondents is 3.05, while the mean for all undergraduates at this institution is 3.03. Because the range of the GPAs is bounded by 0 and 4 and the distribution is extremely left-skewed, a beta regression with a logit link was used to model the GPAs.23 A beta distribu- tion is a distribution of probabilities between 0 and 1 and can take many shapes, from sym- metrical to skewed. The cumulative GPA variable was divided by 4 and transformed so that values were between 0 and 1 and not exactly 0 or 1. The beta regression fits much better and does not predict impossibly high values (see figure 1). The logit link means that, instead of the GPA itself being modeled, a function of the GPA (the logit function) is modeled instead. Multilevel Regression In a linear regression, each regression coefficient (slope) may be interpreted as the predicted difference in the outcome variable when the predictor variable is one unit greater (or, for cat- egorical variables, present or not present) while the value of all other predictors remains the same. Predictive does not imply causal, but simply compares the averages of the outcomes for different groups, given different conditions.24 Respondents from various demographic, discipline, class-level, and graduate students with and without thesis/dissertation requirements may respond differently to questions about the library and have varying relationships with the library. The variation arising from these TABLE 2 Conversion of Survey Responses to Estimated Usage per Month How often do you do… Estimated count Daily 16 times per month Weekly 4 times per month Monthly 1 time per month Once a semester 0.3 times per month Less often 0.2 times per month Never 0 464 College & Research Libraries April 2020 groups needs to be accounted for in the model so that the importance of other predictors may be compared more directly.25 A common method is to include a dummy variable equal to 1 if a respondent is a mem- ber of a particular group and equal to 0 if not. This allows estimates of a different mean or regression intercept for each group. The intercept for each group is estimated separately, so, for a group with few members, the estimate may be unreliable. The result is a set of parallel regression lines (same slope but different intercepts). An alternative method is a multilevel model, where separate intercepts can be estimated using all the data. Where there are fewer members, the group mean will be closer to the overall mean. This is called partial pooling; it offers a conservative estimate of the differences between groups, alleviating the noisiness and unreliability of small group sample sizes. In addition to differing means of the outcome variable for different groups, there may be different slopes for predictor variables within each group. Although these different slopes for different groups (called interactions) can also be estimated with the dummy variable method, again there is the advantage of partial pooling offering a more conservative and reliable estimate.26 McElreath27 argues that multilevel regression should become the default method of regression, to account for differences between individuals and groups that can’t be measured. FIGURE 1 Comparison of Linear Regression to Beta Regression for Modeling Cumulative GPA The upper right histogram displays the extreme left-skewness of the cumulative GPA distribution, which is bounded at the upper end at 4.0. The two plots on the left compare the densities of replications (light blue lines) from a linear regression and a beta regression with the distribution of the observations (dark blue line). The replications from the linear regression (top) follow a symmetrical (normal distribution) distribution matching the observations very poorly, while the replications from the beta regression match the observations fairly well. Library Usage, Instruction, and Student Success across Disciplines 465 Three possible sets of groups were available for use in the multilevel models. The first, college and graduate/undergraduate type combination, were the sampling strata for the sur- vey and are included to account for the varying probability of selection. The second group, discipline and graduate/undergraduate type combina- tion, overlaps with college and type. Some colleges at the authors’ institution are represented by one discipline, other colleges contain multiple disciplines whose library usage and attitudes toward library use may differ significantly, and some disciplines are found in multiple colleges. Disciplines may be a more precise grouping to describe this varying rela- tionship with the library than are col- leges. In addition, disciplines make it easier to compare our results to results from other universities, where the col- lege organization may be quite differ- ent. The number of respondents in each discipline/type group are given in table 3. The level of the student (classification 1, 2, 3, and 4 for undergraduates, and master’s de- gree with no thesis, master’s degree with thesis, and PhD for graduate students) is the third overlapping grouping. Bayesian methods were used in this study as an alternative to the conventional null hypothesis significance testing and to present ranges of values for parameter estimates. The ranges, or credible intervals, can be used to estimate the probability that an effect is greater or less than zero, or greater or less than the effect for another group, given the data. Statistical significance calculations are based on a null hypothesis; in real-life social sci- ence, treatment effects or differences between groups are highly unlikely to be zero. They are likely to be small and varying among individuals, groups, and situations. Dividing results into categories of significant or nonsignificant by the arbitrary .05 p-value level often leads to a misunderstanding that a p-value greater than .05 is proof that there is no treatment effect. On the other hand, too much confidence is placed in low p-values, as studies with low p-value results often fail to replicate.28 Recommendations include presenting estimates with Bayesian uncertainty or credible intervals, which present the probability that the value is really within the interval, given the data, with the uncertainty of the estimate.29 Bayesian estimation starts with a “prior” belief about the values of the parameters to be estimated. These priors may come from previous research. The data are then used to update that prior estimate. The result is a distribution of plausible estimates, given the data, called a posterior distribution. In this study, we assume we know nothing and do not use previous research to set priors. The priors used in this analysis are distributions centered at zero, called weakly informative priors,30 which allow the data to have most of the influence while restraining unrealistic estimates and stabilizing the computation. Again, this produces conservative estimates since the starting TABLE 3 Number of Respondents per Discipline Number of Respondents Discipline Graduate Undergraduate Agriculture & Life Sciences 123 121 Arts & Design 58 63 Business & Economics 77 50 Humanities 30 21 Math & Computer Science 54 12 Physical Science & Engineering 393 117 Social Sciences 155 99 Interdisciplinary 7 11 Veterinary Medicine 219 0 Total 1,116 494 466 College & Research Libraries April 2020 point is the area near zero, while leaving room to be a reasonable distance from zero as well. The most plausible values of the parameters we are trying to estimate, given the data, are the most frequent values in the posterior distribution. Newer advanced software, such as the rstanarm R package31 used in this study, and more computing power allow more computationally stable estimation of these complex models than were possible using non-Bayesian methods. Muth, Oravecz, and Gabry32 provide a tutorial for using rstanarm and a brief review of Bayesian data analysis, with a more complete explana- tion of Bayesian methods from Gelman and Hill,33 Kruschke,34 and McElreath.35 To estimate average levels of library resource use (LRU) and of library contribution to suc- cess (LCS) within each discipline and type, multilevel regression and poststratification (MRP) was used to adjust for differential nonresponse rates by gender and international status. The multilevel model estimates the average for each college, discipline, type, and demographic combination, which is then weighted by the population for each combination.36 Females made up 43% of the university student population and 54% of the response, while international students made up 12% of the university student population and 31% of the response. Average Predictive Comparisons Each of our three outcome variables were modeled with nonlinear models and interactions between variables, making the direct interpretation of regression coefficients difficult, un- FIGURE 2A Average Library Resource Usage Index (LRU) by Discipline Average levels of the LRU (sum of estimated uses per month) by discipline for graduate and undergraduate students. Wider bars indicate more uncertainty or a wider range of plausible values for the estimate, caused by fewer respondents in the group or a wider spread in responses. The thicker segments represent the middle 50 percent of the plausible values. Partial pooling shrinks the estimates toward the mean, especially for groups with few respondents. Library Usage, Instruction, and Student Success across Disciplines 467 like in linear regression. In these situations, average predictive comparisons allow intuitive comparisons to be made. To make average predictive comparisons, the values at which to compare an input are chosen, the input variable is set to the first value over all observations, predictions made, then the input value is set to the second value overall and predictions made again. The average difference between the set of predictions at the two different values is the average predictive comparison. For two-way interactions, the values of both inputs can be changed to clearly show the predicted differences (Gelman and Hill,37 Gelman and Pardoe38). Results of the models are communicated primarily in graphical form. This allows the easiest comparison of the size of the effects of the predictors between each other and across groups; it also communicates the amount of uncertainty in the estimates at a glance. Graphs are constructed in the R package ggplot2.39 Results Levels of library resource usage (LRU) and percentages of those believing the library contrib- utes very much to the respondents’ academic success (LCS) varied quite a bit by discipline. Both were quite a bit higher for graduate students than for undergraduates. LRU and LCS tended to be high in the same groups, with Humanities the highest in usage and nearly the highest in success (see figures 2a and 2b). FIGURE 2B Library Contributed to Your Success? (by Discipline) Average levels by discipline for graduate and undergraduate students for the proportion who believe that the library’s resources and usage contribute “very much” to their academic success (LCS) (right). Wider bars indicate more uncertainty or a wider range of plausible values for the estimate, caused by fewer respondents in the group or a wider spread in responses. The thicker segments represent the middle 50 percent of the plausible values. Partial pooling shrinks the estimates toward the mean, especially for groups with few respondents. 468 College & Research Libraries April 2020 Our study defined library instruction as course-related instruction (CRI) or, as the survey put it, a librarian coming to one of their classes. Reporting library instruction in a class pre- dicted higher library resource use (LRU) (see figure 3). International students use the library more than US residents, with or without CRI. Receiving CRI predicts higher usage for both international students and US residents, but more so for international students. For graduate students, LRU was predicted to be 15 percent higher for US residents re- porting course-related instruction (CRI) than for those without CRI, and 49 percent higher for international students. The difference in LRU for undergraduates reporting CRI was larger than for graduate students, with estimates of 64 percent for US residents and 56 percent for international students (subtracting the two dots on the plot). The estimates for undergradu- ate international students are very uncertain since there are few of them, so it is not clear that there is a difference in LRU between the two conditions, since the 50% credible intervals denoted by the thick bars overlap. Gender and perception of library contribution to success (LCS) are held constant in these calculations, and the differences do not vary much by discipline. Female graduate students use the library about 5 percent less than male graduate students, while female undergradu- ates have about 5 percent higher LRU than males. Average predictive comparison plots, such as figures 3 and 4, show the size of the pre- dicted difference in the outcome as the value of one predictor is changed, for each predic- tor listed on the y-axis. The median of the plausible values for the predicted difference is presented as a white dot. The thick gray bars represent the 50 percent credible interval (the middle 50% of the plausible values from the posterior distribution). There is a 50 percent chance that the unobserved parameter is within that range, given the data. The thin lines represent the 90 percent credible interval. The longer the bars and thin lines, the greater the uncertainty of the estimate. FIGURE 3 Average Predicted Difference for Library Resource Usage Index with Course-Related Instruction (CRI) CRI predicts a larger increase in LRU for international graduate students than for US residents. Without CRI, LRU of international students is higher than for US residents (shown as baseline at 0%). CRI also predicts a larger increase in LRU for undergraduates than for graduate students. Library Usage, Instruction, and Student Success across Disciplines 469 In figure 4, the average difference in LRU between those who believe the library con- tributes very much to success and those who do not varies quite a bit by discipline, but it is always positive. Veterinary Medicine students with LCS average 200 percent higher LRU than those without, while Physical Sciences & Engineering, Math & Computer Science, and Humanities graduate students with LCS have about 75 percent higher LRU than those with- out. Among undergraduates, Physical Sciences & Engineering is one of the disciplines with the largest differences (150% higher LRU with LCS), and Business & Economics among the lowest (around 40% higher LRU). International students tended to report both higher LRU than US residents and were also more likely to say the library contributes “very much” to their academic success (LCS), for both undergraduate and graduate students (8% for undergraduates and 10% for graduates, if no CRI is reported). (See figure 5.) Female students were more likely than male students (7% for undergraduates and 9% for graduates) to say the library contributes “very much” to their success. While receiving course-related instruction predicted higher probability of LCS for US residents, this was not true for international students with LRU held constant at four library FIGURE 4 Difference in Library Resource Usage Index Predicted by “Library Contributes Very Much to Success” (by Discipline) Average differences in library resource usage (LRU) when the perception of library contribution to success (LCS) is “very much.” The ranking of disciplines by differences does not match between graduate and undergraduates. For example, LCS predicts 77 percent higher LRU for graduate students in physical sciences and engineering, in the lower end, and 148 percent higher LRU for undergraduates, the highest among the undergraduates. 470 College & Research Libraries April 2020 resource uses per month. International undergraduate students who reported no CRI were more likely to respond that the library contributes “very much” to their success than those who reported CRI. There was no difference for international graduate students (see figure 5). In the line in figure 5 labeled “International with CRI,” undergraduate column, the thick bar is on the left side of the red zero line, indicating that there is 75 percent probability that international undergraduates who receive CRI are less likely to report LCS than international undergraduates who do receive CRI. Higher frequency of building usage predicts higher probability of LCS for undergraduates (by 11%), but not for graduate students, comparing 4 times per month to 1 time per month. For undergraduates, higher building usage predicts more of a difference in LCS than does LRU. For graduate students, building usage predicted only a slight difference in LCS (see figure 5). For most undergraduate disciplines, both higher LRU and higher building usage predicted a greater probability of LCS, a little more than 10 percent for building usage and a little less than 10 percent for LRU. Humanities had a larger difference, about 12 percent for LRU and 17 percent for building usage, comparing 4 times per month to 1 time per month (see figure 6). Building usage seemed to be slightly more important for international undergraduates than for US residents. For graduate students, the predicted difference in probability for LCS with higher LRU ranged from about 20 to 25 percent for Interdisciplinary, Humanities, and Arts & Design down FIGURE 5 Predicted Differences in Probability of Responding “Library Contributes Very Much to My Academic Success” Comparing the average predicted probability of saying the library contributes “very much” to the respondents’ academic success (LCS), at 4 times per month to once per month for library resource usage index (LRU) and building usage; for international students compared to US residents, without CRI; and for receiving CRI compared to no CRI, averaged over respondents’ LRU, for both international and US residents. Library Usage, Instruction, and Student Success across Disciplines 471 to 10 percent for Business & Economics and Math & Computer Science graduate students (see figure 6). The predicted difference was a bit higher for international students than US residents, by about 2 to 3 percent in most disciplines. FIGURE 6 Average Predicted Comparisons for “Library Contributes Very Much to My Academic Success” For each discipline, the predicted probability of saying that the library contributes “very much” to the respondents’ success, compared at lower and higher levels of LRU and building usage of once per month and 4 times per month. Squares indicate US residents and triangles indicate international students, which shows a little higher LCS with higher LRU for international graduate students. Otherwise, the two groups are very close. Building use seems to be slightly more important for international undergraduates than US resident undergraduates, and for international graduate students in the Arts & Design, Business & Economics, and Veterinary Medicine disciplines. 472 College & Research Libraries April 2020 Model of Cumulative GPA Cumulative GPA was higher on average for students who used library resources more frequently, but not for undergraduate students who received library instruction. An under- graduate with a library resource usage index (LRU) of 4 times per month averaged a higher GPA by 0.1 than an undergraduate with an LRU of 1, if they had not received course-related instruction (CRI). The difference was greater than 0 with at least 95 percent probability, given the data. This is seen in figure 7, as the entire 90 percent credible interval (thin bar) is to the right of 0, together with the other 5 percent to the right of the bar. Little to no difference in GPA was predicted for higher LRU for those undergraduates who reported CRI. The average predicted graduate student GPA was higher by about 0.05 at an LRU of 4 times per month vs. 1 time per month. The difference was greater than 0 with at least 95 per- cent probability, whether or not the student reported receiving library instruction. More frequent building usage predicted lower GPA, comparing 4 times per month to 1 time per month, for both undergraduates and graduate students (see second line of figure 7). On average, both graduates and undergraduates who had received library instruction in class had the same or slightly lower GPAs than those who had not. Receiving CRI predicted lower GPAs by 0.03 for US undergraduates, although this was estimated with a great deal of uncertainty. There was little to no difference between international students and US residents (see third and fourth lines of figure 7). FIGURE 7 Cumulative GPA Compared at Different Values of Each Predictor Comparing the average predicted cumulative GPA at different levels of library resource usage (LRU, once per month vs. 4 times) with course-related instruction (CRI) vs. none; building use (once per month vs. 4 times); and received CRI compared to no CRI for international students and for US residents. Increased LRU predicts increased GPA for both graduate students and undergraduates with at least 95 percent probability (90 percent credible interval is above zero), but with less than 75 percent probability for undergraduates who received CRI, while increased building usage predicts lower GPA. Receiving CRI predicts lower GPA for both international and US residents by the same amount. Library Usage, Instruction, and Student Success across Disciplines 473 In the left column of figure 8, the predicted difference in GPA between those with an LRU of 4 and those with an LRU of 1, averaged over the conditions of receiving CRI or not, varied by discipline. Veterinary Medicine students’ difference of 0.11 was the highest for graduate students. Veterinary Medicine students had a wider range of GPA than other graduate students. Other graduate student disciplines ranged from .03 higher GPA (Business & Economics and Interdisciplinary) to 0.05 higher GPA. There was more variation for undergraduates, ranging from 0 for Math & Computer Sciences to 0.09 for Physical Sciences & Engineering. Because exploratory plots showed a distinct grouping with 0 usage, and because the aver- age of the GPAs of that group seemed to be higher than those with a small amount of usage, a dummy variable for 0 usage was included in the model, which could vary by discipline. For this reason, the lower level of LRU in the comparison in figures 7 and 8 was 1 rather than 0 usage. FIGURE 8 Cumulative GPA Compared at Lower and Higher Levels of Library Resource Usage and Receiving CRI vs. No CRI Average difference in predicted cumulative GPA, comparing an LRU of 1 time per month (red baseline at zero) to 4 times per month (left column), and comparing CRI to no CRI, while averaging over range of LRU. Thick segments show the middle 50 percent of plausible values, while thin segments show the middle 90 percent. If the entire 50 percent interval is greater than zero, then there is at least 75 percent probability that the difference is positive (higher GPA). If the entire 90 percent interval is greater than zero, then there is at least 95 percent probability that the difference is positive. 474 College & Research Libraries April 2020 The differences in figures 7 and 8 adjusted for gender (females average higher GPA); inter- national status (difference varied by discipline, but was mostly neutral to negative); class level for undergraduates and whether a graduate student was in a no-thesis, thesis, or dissertation pro- gram; the sampling strata; building use; and average cumulative GPA for class level and college. A simplified model was used for the scatter plots in figure 9, not adjusting for CRI, build- ing usage, residency, gender, and class level, and using only the zero use indicator and log2 of LRU as predictors, with varying slopes for both in the discipline/student type groups, and including the college/type sampling strata (see appendix A for details). The points in figure 9 indicate the combination of the cumulative GPA (y-axis) and the library resource usage index (LRU) (on a logarithmic scale x-axis) for each individual respon- dent. The regression lines show how the GPA on average increased as the LRU doubled, with FIGURE 9A Regression of Cumulative GPA on Library Resource Usage (LRU), 95% Uncertainty Intervals of the Predictor (Graduate Students) Scatter plot of cumulative GPA vs. LRU on a logarithmic x-axis for graduate students, with points representing each respondent’s combination. Lines are from the predictions from the model, drawn over the range of data for each discipline. Different intercepts for the college sampling strata result in multiple parallel lines in some disciplines. A zero-use indicator results in a higher prediction of GPA for students who report zero library resource usage and a steeper slope for students who do use library resources. Gray bands represent uncertainty in the estimate of the slope (95% credible intervals). Disciplines are ordered by slope, with the largest increase in GPA predicted by doubling of LRU at the end. Library Usage, Instruction, and Student Success across Disciplines 475 the slope of the line varying by discipline. The gray band indicates the 95 percent credible interval for the estimate of the slope of the line. The wide scatter of the points above and below the line at every level of LRU demonstrate that library resource usage did not explain very much of cumulative GPA and that there was a rather weak correlation. The angle of the line upward to the left at 0 for the zero-use indicator varied by discipline, showing that there were students who reported no library resource usage yet still achieved high grades, more so in undergraduate disciplines, but also in the graduate disciplines of Business & Economics, Math & Computer Science, and Physical Sciences & Engineering. The Arts & Design and Humanities disciplines were the only graduate disciplines, along with Humanities undergraduates, in which all respondents reported at least some library resource usage. FIGURE 9B Regression of Cumulative GPA on Library Resource Usage (LRU), 95% Uncertainty Intervals of the Predictor (Undergraduate Students) Scatter plot of cumulative GPA vs. LRU on a logarithmic x-axis for undergraduate students, with points representing each respondent’s combination. Lines are from the predictions from the model, drawn over the range of data for each discipline. Different intercepts for the college sampling strata result in multiple parallel lines in some disciplines. A zero-use indicator results in a higher prediction of GPA for students who report zero library resource usage and a steeper slope for students who do use library resources. Gray bands represent uncertainty in the estimate of the slope (95% credible intervals). Disciplines are ordered by slope, with the largest increase in GPA predicted by doubling of LRU at the end. 476 College & Research Libraries April 2020 The slope of the regression line (upward to the right as LRU increases, in most disciplines) tended to be steeper for most undergraduate disciplines and Vet Med students than for other graduate disciplines, as graduate students tend to have a restricted range of cumulative GPA, 3.0–4.0 in most cases. Undergraduates in Math & Computer Science and Business & Econom- ics were the only groups that showed no increase in GPA at higher levels of LRU. Discussion Librarians would not expect the correlation between students’ success and their use of the library to be uniform across disciplines; multilevel modeling provides a way to estimate the varying strength of those correlations and levels. Rather than trying to reject a one-size-fits-all null hypothesis, this analysis was descriptive, providing estimates along with the uncertainty of those estimates. The descriptive approach allows us to recognize the possibility that some students don’t use the library and yet do just fine in terms of their GPA. Putting this aside, when students do use library resources, the cumulative GPA is higher on average as library resource use doubles, although this is not so for building usage. For undergraduates, this correlation between LRU and GPA is true only if they did not report receiving CRI in the last year. As much as we would like to believe that use of library resources leads to higher GPAs, the widely scattered points in the scatterplots (see figure 9) in this study make clear that LRU does not explain very much of GPA. The results are very similar to other analyses such as those from Soria, Fransen, and Nackerud,40 which show a small association between use of library resources and GPA (an increase of .02 in R2). This is not surprising. As Meehl41 put it, the “crud factor” means that, in the social sciences, “everything correlates to some extent with everything else.” As expected, results varied by discipline. In all disciplines, if students believed the li- brary contributed to their success (LCS), they tended to use resources (LRU) more. In some disciplines, this tendency was more pronounced; for example, LCS predicted 200 percent higher LRU for graduate students in Veterinary Medicine and 150 percent higher LRU for Agriculture & Life Sciences, compared to approximately 75 percent higher LRU for Physical Sciences & Engineering, Humanities, and Math & Computer Science. Results for undergradu- ate students also varied by discipline, again with LCS predicting more of a difference in LRU in some subject areas. The highest difference among undergraduates was 150 percent higher LRU in Physical Sciences & Engineering. We might posit what these results might mean, but that is beyond the scope of what the survey can tell us. Finally, predictions of CRI’s effect for LRU did not vary much across disciplines. The survey respondents who said that LRU contributed “very much” to their academic success [LCS] tended to be in the higher end of both LRU and GPA. This study found that frequency of library building usage is predictive of undergraduates believing that the library contributes “very much” to their success, more than LRU, yet data results indicate building use is not predictive of higher GPA. How do students define success and why do undergradu- ates think that library building use contributes to it? Similarly, why do results from graduate students disagree with these findings? Educational research has found compelling linkages between students’ sense of belong- ing with their retention and success;42 this is particularly so with students of color and other minority communities. For undergraduates, might it be that repeated library building use Library Usage, Instruction, and Student Success across Disciplines 477 leads to their greater familiarity and comfort, contributing to a sense of “belonging” at the library where one can “fit in” and engage in various scholarly behaviors and activities likely to lead to success? Our study found that international students were 6 to 8 percent more likely than other student groups to agree that the library’s resources and services contribute “very much” to their success, and they also reported higher LRU. Instruction (CRI) Contributed to Success (LCS) and Library Resource Use (LRU) Many library value studies have sought to examine the impact of library instruction on student GPA and other success measures. Again, our study defined library instruction as course-related instruction (CRI) or, as the survey put it, a librarian coming to one of their classes.43 The results of this study are similar to others in that they show modest, if hard to explain, positive cor- relations after adjusting for LRU between library instruction and student success, defined as student beliefs in whether they felt the library contributed very much to their success (LCS), and negative correlations with GPA, again after adjusting for LRU. In other words, students who had attended a CRI session were more likely to have a fa- vorable attitude about the library’s role in their success, even though their GPA may not have been similarly impacted, although this was only true for U.S. residents. Are classes that receive library instruction more likely to be harder or easier for students to receive a good grade? Most disciplines showed a slightly negative or neutral result; however, if there was a positive effect of library instruction on GPA, it may be diluted by the effect of all other college experiences the student receives. Receiving CRI did strongly predict higher LRU for many student groups, especially international students. These results may be explained by a strong likelihood for courses receiving CRI to have assignments or expectations that require LRU for student assignments or papers. This may also suggest that students learn through CRI about relevant resources as well as the databases and strategies to explore for finding additional resources on their own. While these results are intriguing and open to interpretation, are these types of library value studies that emphasize big-picture quantitative data the most productive ways of looking at the value of library instruction? For example, it is difficult for quantitative studies to get at the quality issues that matter in learning and teaching, such as the quality of the collaboration between librarian and course instructor; the type of learning strategies employed; the level of instruction offered and how well it aligns with student and course needs; the expertise of the librarian as teacher; and so on. Recommendations It must be recognized that cumulative GPA may be a blunt instrument to measure the effect of library usage or instruction on student success. In what proportion of classes is it plausible that library usage or instruction would help to increase the student’s grade? The effect could be a drop in the pond in a cumulative GPA. To more fully understand and measure the relationship between library building and resource use and student success, it would be helpful to examine how students define success and how the library assists them to achieve it. A longitudinal study such as a diary study or panel survey could gather such qualitative information, while allowing students to be their own controls, using multilevel models with multiple observations nested in each student. 478 College & Research Libraries April 2020 GPAs could also be collected for a more standardized and objective measure to go along with the students’ goals. The study from Whitmire44 of library usage is an excellent example of a much fuller data set than ours. It has a range of variables, including background characteristics, college experi- ences, and library activities; one point that is important to note is that it covered three years with the same students. A multilevel model could combine all the data with the three years’ observations nested within each student and students nested within institutions. Normand45 recommends increasing the number of observations per person over increas- ing the number of people in the study. With students as their own control, all aspects of their background, personality, social/job life, study habits, and abilities can be controlled for. Using outcomes linked more tightly to the treatments and focusing on variations over time in indi- vidual performance rather than averages, such a study could provide much richer information about how and why the library can help student success. Conclusion Student attitudes and beliefs about their own library use matter, whether or not their use can objectively be shown to have been positively linked to success outcomes such as higher GPA. This is particularly so with undergraduates in our study. GPA may not be the best measure of whether the library is an aid to student success, although there is a correlation between LRU and GPA. Higher LRU, building use, and CRI does predict student attribution of aca- demic success to the library; by self-report, it helps some students. It may be of more help to some students than to others either because it is discipline-dependent or because availability of alternatives varies. It’s important also to recognize that students are diverse and do not represent a homogeneous block, as seen with the differences between US residents and inter- national students in this study. Including more student-centered measures of specific ways the library can aid the student, including student-defined measures of success, and including more observations per student might bring the issue of library impact on student success into sharper focus. Library Usage, Instruction, and Student Success across Disciplines 479 APPENDIX A. Statistical Analysis Library Resource Usage Index (LRU) The Library Resource Usage Index (LRU) emulates count data (non-negative integers). Ei- ther Poisson regression or negative binomial regression is usually used to model count data. Since the variance of the LRU was much larger than the mean, a multilevel negative binomial regression was used. Variable name: sumnobldg2 Perception of Library Contribution to Success (LCS) The LCS variable is the binary outcome variable constructed from the response to the ques- tion of “To what extent have the library’s resources and services contributed to your academic success?” A response of “very much” was set to 1, and missing or other responses (“some,” “very little,” or “not at all”) were set to 0. It was modeled by a multilevel logistic regression. Variable name: succ2 Cumulative GPA The cumulative GPA was modeled with the LCU as one of the predictor variables. For the graduate students, the cumulative GPA did not include spring 2016, while spring 2016 was included for the undergraduates. Because the range of the GPAs is bounded by 0 and 4 and the distribution is extremely left-skewed, a multilevel beta regression with a logit link was used to model the GPAs (Smithson and Verkuilen).46 A beta distribution is a distribution of probabilities between 0 and 1 and can take many shapes, from symmetrical to skewed. The cumulative GPA variable was divided by 4 and transformed so that values were between 0 and 1 and not exactly 0 or 1. Variable name: GPA.std.beta Predictor Variables Female = 1 if female, 0 if male International = 1 if international student, 0 if US resident instruct = 1 if “Yes” to Q18 (In the past year, have you … received instruction on using library resources from a librarian in one of your classes), 0 otherwise log2.sumnobldg: base 2 logarithm of LRU log2.bldguse: base 2 logarithm of estimate of count of building uses per month Group-level Variables College.Type: Combination of college and undergraduate or graduate student type. These were the sampling strata. Discipline.Type: Combination of discipline and undergraduate or graduate student type. Discipline groups could be found in multiple colleges or only one college. Some colleges contained multiple disciplines. TypeLev: If the type is “Undergraduate” then TypeLev is the undergraduate classification (1, 2, 3, or 4); if the type is “Graduate” then for a no-thesis master’s student it is 5; for master’s with thesis it is 6, and for a PhD student it is 7. 480 College & Research Libraries April 2020 Auxiliary Variables Auxiliary information at the group level can be added to increase the precision of the esti- mates. This is something that can be done with multilevel modeling but not with a model that uses dummy variables. The average cumulative GPA for all students in that college and classification (obtained from the university): CollAvg_GPA.std or GrpAvg_GPA.std (averages at the class level for under- graduates). Transformed to range from 0 to 1. The average library book/material checkout per student in that discipline: circper (values in table A1). Percentage of course-related instruction sessions in courses in a discipline: cripct (values in table A2). TABLE A1 Values of Auxiliary Variable Circper (number of items checked out per student by discipline and type for FY17) Type Discipline Count Of Items Checked Out Enrollment Average Circulation Per Student (Circper) Graduate Ag & Life Sciences 933 801 1.16 Graduate Arts & Design 1,082 162 6.68 Graduate Business & Econ 575 467 1.23 Graduate Humanities 1,777 182 9.76 Graduate Interdisciplinary 59 238 0.25 Graduate Math & Comp Sci 867 478.5 1.81 Graduate Phys Sci & Engineering 3,385 1656 2.04 Graduate Social Sciences 1,392 844 1.65 Graduate Vet Med 889 747 1.19 Undergraduate Ag & Life Sciences 3261 5,536.5 0.59 Undergraduate Arts & Design 3,243 1,946.5 1.67 Undergraduate Business & Econ 2,058 4964 0.41 Undergraduate Humanities 2,988 1,828.5 1.63 Undergraduate Interdisciplinary 402 1,299.5 0.31 Undergraduate Math & Comp Sci 908 1,398.5 0.65 Undergraduate Phys Sci & Engineering 4,577 8,404.5 0.54 Undergraduate Social Sciences 2,484 4,124.5 0.60 Library Usage, Instruction, and Student Success across Disciplines 481 TABLE A2 Values of Auxiliary Variable Cripct (percentage of course sections with CRI by type and discipline for FY16) Type Discipline CRI Sessions Course Section Count Percentage Of Course Sections With CRI (Cripct) Graduate Ag & Life Sciences 3 1,153 0.26 Graduate Arts & Design 1 303 0.33 Graduate Business & Econ 1 272 0.37 Graduate Humanities 2 181 1.10 Graduate Interdisciplinary 13 176 7.39 Graduate Math & Comp Sci 2 342 0.58 Graduate Phys Sci & Engineering 8 1,055 0.76 Graduate Social Sciences 5 668 0.75 Graduate Vet Med 0 83 0.00 Undergraduate Ag & Life Sciences 30 1,773 1.69 Undergraduate Arts & Design 1 1,126 0.09 Undergraduate Business & Econ 6 952 0.63 Undergraduate Humanities 24 1,448 1.66 Undergraduate Interdisciplinary 2 939 0.21 Undergraduate Math & Comp Sci 0 921 0.00 Undergraduate Phys Sci & Engineering 1 2,752 0.04 Undergraduate Social Sciences 29 1,583 1.83 Undergraduate Vet Med 0 135 0.00 482 College & Research Libraries April 2020 Appendix B: Survey Instrument. ISU Library User Survey 2016 VARIABLE SAMPLE INFORMATION CaseID Respondent’s Case ID# Date Survey Completion Date Status Survey Completion Status Type Graduate Student, Undergraduate Student, or Postdoc G = Graduate Student UG = Undergraduate Student PD = Postdoc College Respondent’s College A = Agriculture ADMIN = Administration C = Design E = Engineering H = Human Sciences M = Business S = Liberal Arts and Sciences U = Graduate V = Veterinary Medicine Department Respondent’s Department Gender Respondent’s Gender F = Female M = Male Residency Respondent’s Residency Status Age Respondent’s Age (Not available for postdocs) RaceEthnicity Respondent’s Race / Ethnicity (Not available for postdocs) FullPart Respondent’s Full or Part-time Status (Students only) Degree Respondent’s Degree Program (Graduate students only) Classification Respondent’s Classification Year (Undergraduate students only) 1 = Freshman 2 = Sophomore 3 = Junior 4 = Senior AdmitType Respondent’s Admission Type (Undergraduate students only) 1 = Direct from High School 2 = Transfer VARIABLE SURVEY Q1. Which library or reading room do you use most? 1 = Parks Library 2 = Veterinary Medical Library 3 = Design Reading Room 4 = I don’t use any library spaces or reading rooms Q2 – Q4. Please indicate how often you do the following activities. Library Usage, Instruction, and Student Success across Disciplines 483 1 = Daily 2 = Weekly 3 = Monthly 4 = Once a semester 5 = Less often 6 = Never Q2. Go to the library building or reading room that you use most Q3. Use the Library website Q4. Use an external website (e.g., Google, Google Scholar, Bing, YouTube, etc.) for your work Q5 – Q7. Please indicate how often you use the following types of materials in your course- work or research. 1 = Frequently 2 = Occasionally 3 = Never 4 = No Coursework Q5. IF UG or G/P (not PD): Online materials provided by your instructor Q6. Online materials discovered by yourself Q7. Physical items (books, DVDs, print journals) provided by the ISU library IF UNDERGRADUATE STUDENT (UG), SKIP Q8-12, GO TO Q13. IF GRAD/PROFESSIONAL STUDENTS/POSTDOCS (G/P/PD), ASK Q8-12. Q8. Do you teach undergraduate students? 1 = Yes (Go to Q9-12) 2 = No (Go to Q13) IF Q8 = YES: Q9 – Q11. Please indicate your overall satisfaction with your students’ performance in the following areas. 1 = Low 2 = Medium 3 = High 4 = Not sure Q9. Undergraduate students’ ability to find and use appropriate information for assigned activities Q10. Undergraduate students’ ability to ethically use information for assigned activities Q11. Undergraduate students’ ability to create a standard citation or bibliography Q12 – Q16. Please indicate your satisfaction with your own skills listed below. 1 = Low 2 = Medium 3 = High 4 = Not sure Q12. IF G/P/PD & Q4 = YES (Teach): Your ability to find and use appropriate information for your teaching Q13. IF G/P/PD: Your ability to find and use appropriate information for your research Q14. IF UG or G/P (not PD): Your ability to identify and find appropriate information for as- signed activities or coursework 484 College & Research Libraries April 2020 Q15. ASK ALL: Your understanding of plagiarism Q16. ASK ALL: Your ability to create a standard citation or bibliography Q17 – Q19. In the past year, have you participated in the following tasks related to your ISU coursework or research? 1 = Yes 2 = No 3 = Not sure Q17. ASK ALL: Asked a library staff member for assistance Q18. IF UG or G/P (not PD): Received instruction on using library resources from a librarian in one of your classes Q19. IF G/P/PD: Used bibliographic management software provided by the library or other resources to manage your sources (e.g., EndNote Web, cite features in databases, Zotero, etc.) Q20. How often have you used Quick Search, the main search bar on the Library website home page, to find library materials? 1 = Daily 2 = Weekly 3 = Monthly 4 = Once a semester 5 = Less often 6 = Never IF Q20 = 1-5, GO TO Q22 IF Q20 = 6 (NEVER), GO TO Q21, SKIP Q22-24, AND GO TO Q25. Q21. IF Q20 = NEVER: How do you usually find the information that you need? [Open text response] Q22. IF Q20 = 1, 2, 3, 4, or 5: What is your primary reason for using Quick Search? 1 = Search for a specific title or author 2 = Search for topics 3 = Both of these equally Q23. IF Q20 = 1, 2, 3, 4, or 5: Overall, how satisfied are you with Quick Search results when you look for a specific title or author? 1 = Low satisfaction 2 = Medium satisfaction 3 = High satisfaction Q24. IF Q20 = 1, 2, 3, 4, or 5: Overall, how satisfied are you with Quick Search results when you are exploring general topics or subjects? 1 = Low satisfaction 2 = Medium satisfaction 3 = High satisfaction Q25 – Q29. How often do you use the Library website for the following purposes? 1 = Daily 2 = Weekly 3 = Monthly 4 = Once a semester 5 = Less often 6 = Never Library Usage, Instruction, and Student Success across Disciplines 485 Q25. Use article indexes and databases (e.g., Academic Search Premier (EBSCO), Web of Science, JSTOR, etc.) Q26. Find and use electronic journals and/or articles Q27. Find and use e-books Q28. Find physical items (e.g., books, DVDs, CDs, maps, microforms, etc.) Q29. Find materials in Special Collections and University Archives (e.g., manuscripts, rare books, etc.) Q30 – Q36. How satisfied are you with the following ISU Library resources and services? 1 = Low 2 = Medium 3 = High 4 = Don’t use Q30. Remote (off-campus) access to library materials and services through the Library website Q31. Assistance or help from library staff through Chat or email Q32. Reserve a library space (e.g., group study room, multimedia production studio, or to practice a presentation) Q33. Research and Course Guides Q34. Interlibrary Loan and document delivery Q35. IF G/P/UG: Course Reserves Q36. IF G/P/PD: Library workshops and seminars supporting your research (e.g., bibliographic management tools such as EndNote, funding resources, using the Digital Repository, etc.) IF UG, GO TO Q40. IF G/P/PD, ASK Q37-39: Q37. Digital Repository @ Iowa State University provides free and open access to scholarly and creative works, research, publication, and reports by Iowa State’s faculty, students, staff and administrative units. Are you familiar with the Library’s Digital Repository? 1 = Yes 2 = No Q38. IF Q37 = YES: Have you used the Digital Repository to search? 1 = Yes 2 = No Q39. IF Q37 = YES: Have you added your publications, presentations, research, or thesis to the Digital Repository? 1 = Yes 2 = No Q40. What suggestions do you have to improve the Library’s website, including Quick Search, functionality, ease of use, services, online collections, etc.? [Open text response] IF P (VET MED), SKIP Q41-61 AND GO TO Q62-68. IF UG/G/PD, ASK Q41-54: 486 College & Research Libraries April 2020 Q41 – Q54. Please indicate your overall satisfaction with each of the following resources provided by Parks Library. 1 = Low 2 = Medium 3 = High 4 = Don’t use Q41. Library staff assistance at the Circulation/Research Help Desk Q42. Computers and printers available for public use Q43. Scanners available for public use Q44. Quality of lighting Q45. Access to electrical outlets and chargers Q46. Availability of space to work collaboratively Q47. IF UG: Reserved group study rooms on third floor Q48. Availability of space to work individually Q49. IF G: Assigned library research study rooms for private research Q50. Multimedia production studios and equipment (Learning Connections Center) Q51. Learning Connections Center Help Desk for technology assistance Q52. Parks Library Bookends Café and Fireplace Reading Room Q53. Special Collections and Archives Q54. Media Center IF DESIGN COLLEGE, ASK Q55-61: Q55 – Q61. Please indicate your overall satisfaction with each of the following resources provided by the Design Reading Room. 1 = Low 2 = Medium 3 = High 4 = Don’t use Q55. Library staff assistance at the front desk Q56. Computers and printers available for public use Q57. Scanners available for public use Q58. Quality of lighting Q59. Access to electrical outlets and chargers Q60. Availability of space to work collaboratively Q61. Availability of space to work individually IF VET MED, ASK Q62-68: Q62 – Q68. Please indicate your overall satisfaction with each of the following resources provided by the Veterinary Medical Library. 1 = Low 2 = Medium 3 = High 4 = Don’t use Q62. Library staff assistance at the front desk Q63. Computers and printers available for public use Library Usage, Instruction, and Student Success across Disciplines 487 Q64. Scanners available for public use Q65. Quality of lighting Q66. Access to electrical outlets and chargers Q67. Availability of space to work collaboratively Q68. Availability of space to work individually Q69. What suggestions do you have to improve library services, spaces, or resources? [Open text response] IF G/P/PD, ASK Q70 & 71: Q70 – Q71. The Library wants to ensure that its services support graduate/professional student and postdoc success. How would you rate the importance of the following future initiatives for graduate students, professional students, and postdocs? 1 = Low importance 2 = Medium importance 3 = High importance 4 = Need more information Q70. Develop a trusted online repository to store and provide continuing access to data created by graduate and professional students and postdocs Q71. Provide advisory services for research data management Q72. To what extent have the Library’s resources and services contributed to your academic success? 1 = Very much 2 = Some 3 = Very little 4 = Not at all IF POSTDOC, ASK Q73-74: *Q73. What is your gender? 1 = Male 2 = Female *Q74. Are you from… 1 = Iowa 2 = Another U.S. state 3 = Another country (International) Q75. Please record any additional comments or suggestions about ISU Library resources or services in the space below. [Open text response] Q76. Would you like to be entered into the drawing for a $20 gift card for any ISU on- campus dining location? (50 gift cards will be awarded.) 1 = Yes (Go to Q77) 2 = No **Q77 IF Q76 = Yes: Drawing winners will be notified electronically. Please enter your preferred email address: [Open text response] 488 College & Research Libraries April 2020 Notes The data set “use2016” contained all student responses from the survey (1,595 responses), except those with missing values for the LRU. The data set “GPAuse” was somewhat smaller (1,551 responses), as not all respondents could be matched with their cumulative GPA. Default values of priors in rstanarm were used. The notation (1|GroupName) indicates that varying intercepts were estimated for each group, while the notation (Variable|GroupName) indicates varying intercepts and varying slopes for the variable. R Code for Models Used for Each Figure library(rstanarm) FIGURE 2 (left) SColType.sumno.disc.MRP.3 <- stan_glmer(sumnobldg2 ~ Female.Internation- al + circper + cripct + (1|College.Type) + (1|discipline.Type), family = neg_binomial_2, adapt_delta=0.95, data=use2016[,c(“sumnobldg2”,”Femal e”,”International”,”Female.International”,”circper”,”cripct”,”College. Type”,”discipline.Type”)], seed = SEED, QR=TRUE) FIGURE 2 (right) SColType.succ.disc.MRP.2 <-stan_glmer(succ2 ~ Female.International + cripct + circper + (1|College.Type) + (1|discipline.Type), family = binomial(link=“logit”), adapt_delta=0.95, data=use2016[,c(“succ2”,”Fema le”,”International”,”Female.International”,”cripct”,”circper”,”College. Type”,”discipline.Type”)], seed = SEED, QR=TRUE) FIGURE 3 Average Predictive Comparisons for LRU SColType.sumno.disc.simple.2 <- stan_glmer(sumnobldg2 ~ succ2*Type + Female*Type + instruct*International*Type + (1|TypeLev) + (1|College. Type) + (1|discipline.Type), family = neg_binomial_2, adapt_delta=0.95, data=use2016[,c(“Type”,”sumnob ldg2”,”succ2”,”Female”,”instruct”,”International”,”circper”,”TypeLev”,”Co llege.Type”,”discipline.Type”)], seed = SEED, QR=TRUE) FIGURE 4 LCS and Received CRI Effect on Library Resource Usage SColType.sumno.disc.simple.3 <- stan_glmer(sumnobldg2 ~ succ2 + Female + instruct*International + circper + (1|TypeLev) + (1|College.Type) + (succ2 + instruct*Internationa l|discipline.Type), family = neg_binomial_2, adapt_delta=0.99, data=use20 16[,c(“sumnobldg2”,”succ2”,”Female”,”instruct”,”International”,”circper”, ”TypeLev”,”College.Type”,”discipline.Type”)], seed = SEED, QR=TRUE) FIGURE 5 and FIGURE 6 Average Predictive Comparisons for “Library Contributes Very Much to Academic Success” Library Usage, Instruction, and Student Success across Disciplines 489 SColType.succ.disc.Intl.All.44 <-stan_glmer(succ2 ~ log2. bldguse*International + Female + instruct*International + instruct*log2. sumnobldg + International*log2.sumnobldg + cripct + circper + (log2. bldguse*International + instruct*log2.sumnobldg|TypeLev) + (1|Col- lege.Type)+ (instruct*log2.sumnobldg + International*log2. sumnobldg|discipline.Type), family = binomial(link=“logit”), adapt_del- ta=0.999, data=use2016, seed = SEED, QR=TRUE) FIGURE 7 and FIGURE 8 Average Predictive Comparisons for Cumulative GPA SColType.GPA.Beta.Zero.Type.34 <- stan_glmer(GPA.std.beta ~ sumno- bldg.0 + International*log2.bldguse + Female + instruct*log2.sumnobldg + instruct*International + International*log2.sumnobldg + GrpAvg_GPA. std + (1|TypeLev) + (1|College.Type) + (sumnobldg.0 + International*log2. bldguse + instruct*International + instruct*log2.sumnobldg + International*log2.sumnobldg|discipline.Type), family=mgcv::betar(link=“l ogit”), adapt_delta=0.99, data=GPAuse, seed = SEED, QR=TRUE) FIGURE 9 Model for GPA Scatter Plot SColType.GPA.Beta.Zero.disc.13 <- stan_glmer(GPA.std.beta ~ sumnobldg.0 + log2.sumnobldg + CollAvg_GPA.std + (1|College.Type) + (sumnobldg.0 + log2.sumnobldg|discipline.Type), family=mgcv::betar(link=“logit”), adapt_ delta=0.95, data=GPAuse, seed = SEED, QR=TRUE) Function for Creating Average Predictive Comparisons Used for the analyses of the LCS and GPA outcome variables, both of which modeled a logit function of the outcome variable. The rstanarm function posterior_linpred with transform=TRUE transforms the outcome variable back to probabilities for LCS and for GPA, back to a 0 to 1 scale (afterwards multiplied by 4). The function compares the outcome at two different values of one predictor, resulting in two posterior distributions which are differenced (ME). The predicted differences are averaged over the values of the other predictors present in the survey responses (AME). Credible intervals are formed by taking the quantiles of the posterior distribution (QME).47 • nd: data set used to make predictions • type: graduate or undergraduate • d: a group such as discipline, or “All” • mfit: model fit used to predict • v: variable for average predictive comparison, such as “instruct” or “LRU” • lo and hi: comparison levels of variable • subgrp: To compare interactions of two predictors, first all cases are set to the first sub- group in a set, the function is applied, then all cases are set to the second subgroup. Sub- group sets include “International” and “US Resident”, or “Received CRI” and “No CRI”. effconplot.fun <- function(nd,type,grp,mfit,v,lo,hi,subgrp){ nd[,v] <- lo ### transform=TRUE returns probabilities 490 College & Research Libraries April 2020 p0 <- posterior_linpred(mfit, newdata=nd, transform=TRUE) nd[,v] <- hi p1 <- posterior_linpred(mfit, newdata=nd, transform=TRUE) ME <- p1 - p0 AME <- rowMeans(ME) QME <- quantile(AME, c(.025,.25,.5,.75,.975)) MADME <- mad(ME) SDME <- sd(ME) eff <- data.frame(grp.label=grp, grp.type = type, L95=QME[1],L50=QME[2],M ed=QME[3],U50=QME[4],U95=QME[5],MADME=MADME,SDME=SDME,treat=v,subgrp) return(eff) } Notes 1. Association of College and Research Libraries (ACRL), The Value of Academic Libraries: A Comprehensive Research Review and Report, prepared by Megan Oakleaf (Chicago, IL: ACRL, 2010). 2. ACRL, The Value of Academic Libraries, 14. 3. Association of College and Research Libraries (ACRL), Academic Library Impact: Improving Practice and Es- sential Areas to Research, prepared by Lynn Silipigni Connaway, William Harvey, Vanessa Kitzie, and Stephanie Mikitish of OCLC Research (Chicago, IL: ACRL, 2017), 1. 4. Ronald L. Wasserstein and Nicole A. Lazar, “The ASA’s Statement on P-values: Context, Process, and Purpose,” American Statistician70, no. 2 (2016): 129–33. 5. Blakeley B. McShane et al., “Abandon Statistical Significance,”American Statistician73, no. sup1 (2019): 235–45. 6. Shun Han Rebekah Wong and T.D. Webb, “Uncovering Meaningful Correlation between Student Aca- demic Performance and Library Material Usage,”College & Research Libraries72, no. 4 (2011): 361–70. 7. Krista M. Soria, Jan Fransen, and Shane Nackerud, “Library Use and Undergraduate Student Outcomes: New Evidence for Students’ Retention and Academic Success,”portal: Libraries and the Academy13, no. 2 (2013): 147–64; Graham Stone and Bryony Ramsden, “Library Impact Data Project: Looking for the Link between Library Usage and Student Attainment,”College & Research Libraries74, no. 6 (2013): 546–59. 8. Shun Han Rebekah Wong and Dianne Cmor, “Measuring Association between Library Instruction and Graduation GPA,”College and Research Libraries72, no. 5 (2011): 464. 9. Wong and Webb, “Uncovering Meaningful Correlation between Student Academic Performance and Library Material Usage,” 361–70. 10. Melissa Bowles-Terry, “Library Instruction and Academic Success: A Mixed-Methods Assessment of a Library Instruction Program,”Evidence Based Library and Information Practice7, no. 1 (2012): 82. 11. Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes.” 12. Krista M. Soria, Jan Fransen, and Shane Nackerud, “Beyond Books: The Extended Academic Benefits of Library Use for First-Year College Students,”College & Research Libraries78, no. 1 (2017). 13. Ula Gaha, Suzanne Hinnefeld, and Catherine Pellegrino, “The Academic Library’s Contribution to Student Success: Library Instruction and GPA,”College & Research Libraries79, no. 6 (2018): 737. 14. Dennis Krieb, “Assessing the Impact of Reference Assistance and Library Instruction on Retention and Grades Using Student Tracking Technology,”Evidence Based Library and Information Practice13, no. 2 (2018): 2–12. 15. Soria, Fransen, and Nackerud, “Beyond Books,” 8–22. 16. Wong and Webb, “Uncovering Meaningful Correlation between Student Academic Performance and Library Material Usage,” 361–70; Wong and Cmor, “Measuring Association between Library Instruction and Graduation GPA,”464. 17. Bowles-Terry, “Library Instruction and Academic Success,”82–95; Gaha, Hinnefeld, and Pellegrino, “The Academic Library’s Contribution to Student Success,”737; Soria, Fransen, and Nackerud, “Library Use and Un- dergraduate Student Outcomes,”147–64. 18. Wong and Cmor, “Measuring Association between Library Instruction and Graduation GPA,”464. 19. Krieb, “Assessing the Impact of Reference Assistance and Library Instruction on Retention and Grades Using Student Tracking Technology,” 2–12. Library Usage, Instruction, and Student Success across Disciplines 491 20. Sarah Passoneau, 2012 Library User Survey Report (Ames: Iowa State University Library, 2012), available online at https://assess.lib.iastate.edu/sites/default/files/Survey%20Uploads/Final%202012%20Library%20User%20 Survey%20Report.pdf [accessed 15 January 2020]; Linda Anderson et al., 2016 Library Student and Postdoc User Survey (Ames: Iowa State University Library, 2016), available online at https://assess.lib.iastate.edu/sites/default/ files/Survey%20Uploads/2016%20Library%20Student%20and%20Postdoc%20User%20Survey%20Report%20-%20 December%202016%20-%20Final%20(4-5-17).pdf [accessed 15 January 2020]. 21. Linda Anderson, “Library Website Visits and Enrollment Trends,”Evidence Based Library and Information Practice11, no. 1 (2016): 4. 22. For the graduate student respondents, the cumulative GPA did not include Spring 2016, while Spring 2016 was included for the undergraduates. 23. Michael Smithson and Jay Verkuilen, “A Better Lemon Squeezer? Maximum-Likelihood Regression with Beta-distributed Dependent Variables,”Psychological methods11, no. 1 (2006): 54. 24. Andrew Gelman and Jennifer Hill,Data Analysis Using Regression and Multilevel/Hierarchical Models (Cam- bridge, UK: Cambridge University Press, 2006). 25. Richard McElreath, Statistical Rethinking: A Bayesian Course with Examples in R and Stan (Boca Raton, FL: Chapman and Hall/CRC, 2018). 26. Gelman and Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. 27. McElreath, Statistical Rethinking. 28. McShane et al., “Abandon Statistical Significance,”235–45. 29. Wasserstein and Lazar, “The ASA’s Statement on P-values,”129–33. 30. Gelman and Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. 31. Jonah Gabry and Ben Goodrich, “rstanarm: Bayesian Applied Regression Modeling via Stan, R Package Version 2.18. 2” (2018), https://mc-stan.org/rstanarm [accessed 7 March 2020]. 32. Chelsea Muth, Zita Oravecz, and Jonah Gabry, “User-friendly Bayesian Regression Modeling: A Tutorial with rstanarm and shinystan,” Quantitative Methods for Psychology 14, no. 2 (2018): 99–119. 33. Gelman and Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. 34. John Kruschke, Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan (Cambridge, MA: Academic Press, 2014). 35. McElreath, Statistical Rethinking. 36. Gelman and Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. 37. Gelman and Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. 38. Andrew Gelman and Iain Pardoe, “2. Average Predictive Comparisons for Models with Nonlinearity, Interactions, and Variance Components,”Sociological Methodology 37, no. 1 (2007): 23–51. 39. Hadley Wickham and Winston Chang, “Package ‘ggplot2’,” Create Elegant Data Visualisations Using the Grammar of Graphics, Version 2, no. 1 (2016): 1–189. 40. Soria, Fransen, and Nackerud, “Beyond Books,”8–22. 41. Paul E. Meehl, “Why Summaries of Research on Psychological Theories Are Often Uninterpretable,”Psychological Reports 66, no. 1 (1990): 195–244. 42. Terrell L. Strayhorn, College Students’ Sense of Belonging: A Key to Educational Success for All Students (Abing- don-on-Thames, UK: Routledge, 2018); Silas M. Oliveira, “The Academic Library’s Role in Student Retention: A Review of the Literature,”Library Review 66, no. 4/5 (2017): 310–29. 43. Our study asked students themselves to indicate whether they received library instruction. While this approach is not problem-free, as it does not delve into quality or level of instruction and relies as well on student memory, it avoids the problems of inferred CRI attendance. 44. Ethelene Whitmire, “The Relationship between Undergraduates’ Background Characteristics and College Experiences and Their Academic Library Use,”College & Research Libraries 62, no. 6 (2001): 528–40. 45. Matthew P. Normand, “Less Is More: Psychologists Can Learn More by Studying Fewer People,”Frontiers in Psychology 7 (2016): 934. 46. Michael Smithson and Jay Verkuilen, “A Better Lemon Squeezer? Maximum-Likelihood Regression with Beta- Distributed Dependent Variables,”Psychological Methods 11, no. 1 (2006): 54, https://doi.org/10.1037/1082-989X.11.1.54. 47. Ben Goodrich, “Calculating Marginal Effects in Binomial Logit Using rstanarm,” Stackoverflow.com, available online at https://stackoverflow.com/questions/45037485/calculating-marginal-effects-in-binomial-logit- using-rstanarm/45042387 [accessed 24 October 2019]; Ben Goodrich, “Calculating Credible Intervals for Marginal Effects in Binomial Logit Using rstanarm,” Stackoverflow.com, available online at https://stackoverflow.com/ questions/48431893/calculating-credible-intervals-for-marginal-effects-in-binomial-logit-using-rsta [accessed 24 October 2019]. https://assess.lib.iastate.edu/sites/default/files/Survey%20Uploads/Final%202012%20Library%20User%20Survey%20Report.pdf https://assess.lib.iastate.edu/sites/default/files/Survey%20Uploads/Final%202012%20Library%20User%20Survey%20Report.pdf https://assess.lib.iastate.edu/sites/default/files/Survey%20Uploads/2016%20Library%20Student%20and%20Postdoc%20User%20Survey%20Report%20-%20December%202016%20-%20Final%20(4-5-17).pdf https://assess.lib.iastate.edu/sites/default/files/Survey%20Uploads/2016%20Library%20Student%20and%20Postdoc%20User%20Survey%20Report%20-%20December%202016%20-%20Final%20(4-5-17).pdf https://assess.lib.iastate.edu/sites/default/files/Survey%20Uploads/2016%20Library%20Student%20and%20Postdoc%20User%20Survey%20Report%20-%20December%202016%20-%20Final%20(4-5-17).pdf https://mc-stan.org/rstanarm https://doi.org/10.1037/1082-989X.11.1.54 http://Stackoverflow.com https://stackoverflow.com/questions/45037485/calculating-marginal-effects-in-binomial-logit-using-rstanarm/45042387 https://stackoverflow.com/questions/45037485/calculating-marginal-effects-in-binomial-logit-using-rstanarm/45042387 http://Stackoverflow.com https://stackoverflow.com/questions/48431893/calculating-credible-intervals-for-marginal-effects-in-binomial-logit-using-rsta https://stackoverflow.com/questions/48431893/calculating-credible-intervals-for-marginal-effects-in-binomial-logit-using-rsta