623 Degrees of Impact: Analyzing the Effects of Progressive Librarian Course Collaborations on Student Performance Char Booth, M. Sara Lowe, Natalie Tagge, and Sean M. Stone Char Booth is Director of Research, Teaching, & Learning Services, Claremont Colleges Library, e-mail: char_booth@cuc.claremont.edu; M. Sara Lowe is Educational Development Librarian Indiana University- Purdue University Indianapolis, e-mail: mlowe@iupui.edu; Natalie Tagge is Social Sciences Team Leader, Claremont Colleges Library, e-mail: natalie_tagge@cuc.claremont.edu; and Sean M. Stone is Dental Librar- ian at Indiana University-Purdue University Indianapolis, e-mail: smstone@iu.edu. © 2015 Char Booth, M. Sara Lowe, Natalie Tagge, and Sean M. Stone, Attribution-NonCommercial (http://creativecommons. org/licenses/by-nc/3.0/) CC BY-NC. The Claremont Colleges Library conducted direct rubric assessment of Pitzer College First-Year Seminar research papers to analyze the impact of diverse levels of librarian course collaborations on information literacy (IL) performance in student writing. Findings indicate that progressive degrees of librarian engagement in IL-related course instruction and/or syllabus and assignment design had an increasingly positive impact on student performance. A secondary indirect analysis of librarian teaching evaluations and self-perceived learning gains by students and faculty showed no correlation to rubric IL scores, suggesting the importance of “authentic” assessment in determining actual learning outcomes. This mixed-methods study presents findings in each area and examines their implications for effective IL course collaborations. Introduction: Holistic Assessment and its Challenges Modern libraries operate in a climate of rapid organizational, technological, and information change, the demands of which are exacerbated by persistent resource scarcity. In this context, assessment has become central to the practice of determining and communicating the “value” of academic libraries to the communities of higher learning in which they are situated.1 From ethnography to quantitative study to user experience research, numerous strategies are employed to evaluate the effectiveness and relevance of library services, tools, operations, and personnel in order to support the goal of producing “evidence-based reports of measurable impact.” 2 Within this evaluation-focused framework, the established but ever-expanding peda- gogical role of the librarian in higher education necessitates reliable methods for mea- suring not only the teaching performance of library educators, but the actual learning effects of their interventions among students.3 Learner-centered IL evaluation follows global trends in higher education assessment that supplement quantitative analytics with “holistic” evaluation techniques involving authentic student work, and learning doi:10.5860/crl.76.5.623 crl14-621 624 College & Research Libraries July 2015 experiences that involve applied or “real world” elements beyond testing measures based on recall and reasoning.4 Connecting student academic performance to library/ ian pedagogical and course collaboration efforts such as instruction and assignment design in these ways not only cements our role in facilitating the development of a diverse range of concepts and competencies that comprise information literacy (IL), it responds to a broader call for increased accountability in higher education.5 Weiner argues that there is a “history of difficulty in integrating information literacy with the postsecondary educational process.”6 Similarly and by extension, macro and micro environmental factors in higher education present obstacles to holistic library learning assessment. For the librarian working in the context of non–credit-bearing course-integrated or course-related instruction, common impediments include teaching scenarios of limited duration (like the one-shot), lack of access to student coursework, little influence over course and assignment design, minimal faculty-librarian collabora- tion, and differing syllabi and assignment expectations across a unified program. At the program level, nonrequisite credit-bearing IL courses, detachment from institutional governance and accreditation review, and/or frequent turnover among allied academic stakeholders are common challenges. Within one of the most typical programmatic IL instruction scenarios, shared first- year experience courses such as introductory seminars and foundational writing/ rhetoric programs, these macro and micro challenges often meet. Varied degrees of course collaboration (such as buy-in) can occur between individual teaching faculty and librarians, resulting in different levels of instruction and input into assignment design across a program. In the event of administrative buy-in and standardized IL outcomes, diverse teaching strategies and curricula are often still employed by different librarians within a given program based on faculty collaboration level and personal pedagogy. If it is assumed that direct, program-level assessment of student learning is dependent on the evaluation of shared outcomes or interventions, reliable and coordinated evaluation of student learning becomes problematic within these common constraints.7 Developing practical IL assessment frameworks that focus on authentic student output while also allowing for individualized librarian pedagogy and unique course collaboration scenarios will be critical for communicating the value proposition of libraries and their quantitative and qualitative contributions to student learning on a larger scale; it is within this context that the present study is situated. Research Motivation and Context The Claremont Colleges in Claremont, California, are a consortium of seven contigu- ous but independent institutions situated around a common library. The Claremont campuses comprise a total enrollment of roughly 7,000 students across five liberal arts colleges (Claremont McKenna College, Harvey Mudd College, Pitzer College, Pomona College, and Scripps College) and two graduate universities (Claremont Graduate University and Keck Graduate Institute). This consortial scenario presents unique challenges and opportunities for Claremont Colleges Library (CCL) staff involved in IL education, ranging from the need for customized pedagogies that reflect campus cultures to the ability to test creative approaches to resource scalability among differ- ent stakeholder groups. For the last two decades, teaching librarians at CCL have provided instruction to first-year seminar programs at each of the undergraduate colleges to fluctuating degrees of depth—some programs received minimal opt-in coverage, while others featured a requisite course collaboration with an assigned librarian based on subject area “fit.” Since the establishment of its Instruction Services department in 2011, CCL Degrees of Impact 625 has successfully augmented efforts to standardize course interventions and program- matically integrate outcomes-oriented IL instruction into the first-year seminars at all five of its undergraduate colleges. Steps toward greater “institutionalization”8 have been achieved through increasingly close collaboration with seminar program coordi- nators and campus assessment officers, faculty and librarian professional development initiatives, and the library-led creation of a shared colleges IL definition and first-year seminar learning outcomes.9 Expansion of programmatic collaboration from two to five campuses has effectively doubled the amount of first-year instruction conducted by CCL teaching librarians, translating to more than 100 unique course collaborations that reach the vast majority of Claremont undergraduates at a foundational point in their college experience. These courses are distributed among approximately 20 teaching librarians, each collaboration featuring a unique syllabus with minimal topical overlap and an almost total lack of common assignments with the exception of a relatively standard final research paper. Evaluating student learning across these collaborations requires the application of assessment strategies that apply to highly individualized instruction scenarios that inevitably result in what we will describe as “progressive” degrees of IL interaction with faculty and students. At Claremont, despite considerable coordination among teaching librarians and seminar coordinators, student/librarian and faculty/librarian interactions in the first year range from negligible to substantial across courses in the same program. This scenario creates unequal levels of student exposure to IL concepts and, by extension, different learning effects based on the depth of faculty/librarian collaboration. Given this phenomenon, it is essential to examine whether relative degrees of faculty/librar- ian collaboration and/or librarian instructional interactions result in qualitative and/ or quantitative changes in student learning. In the present study, investigators applied a mixed-methods analysis to student culminating research papers and library instruction evaluations produced in 13 course collaborations in Pitzer College’s First-Year Seminar (FYS) program with the goal of determining the student learning effects of differential levels of IL course engagement. Based on the near-universal FYS deliverable of a final research paper with a secondary source integration component, researchers determined that assessment of papers us- ing an established, CCL-developed IL rubric (see appendix A) was the ideal method for conducting authentic evaluation of progressive IL interventions in student work. In order to examine whether perceived effectiveness of library instruction in terms of instructor performance and perceived student learning self-perceptions correlated to student IL performance as established by rubric analysis, researchers conducted a secondary analysis of web-based faculty and student survey evaluations of library instruction across the same course pairings. Rubric evaluation of student work indicates that increased librarian intervention in the form of more intensive IL instruction and/or assignment design collaboration with faculty had a marked and statistically significant positive impact on first-year student IL performance in research-based writing. Secondary analysis of student/faculty evalu- ations of FYS librarian instruction revealed no correlation between IL performance in research papers and self-perceived student learning gains and/or perceptions of librarian teaching effectiveness. This dichotomy underscores the importance of direct, student-focused IL assessment to determine actual librarian intervention effects. This paper presents findings of each analysis (rubric and survey) and discusses their impli- cations for constructing effective individual and programmatic course collaboration frameworks at the first year and beyond. 626 College & Research Libraries July 2015 Literature Review Assessment of information literacy instruction has a long-established presence in library education literature. Following broad trends within higher education assessment, a practice once focused on librarian performance evaluation in instructional contexts has shifted toward to a more direct and student-focused approach to learning assessment using a variety of techniques, including fixed-choice tests, performance assessments, and rubrics.10 This holistic assessment trend is in large part a response to acknowledged limitations of test-based IL assessment; Dunn notes that “librarians have attempted to assess student information competence skills by ‘testing’ students with standard class- room tests based on multiple choice, fill-in-the-blank, and matching questions. Such tests …cannot assess the effectiveness of student search skills in real life situations.”11 Holistic, authentic, and mixed-methods assessment of library instruction is as a result becoming well-traversed territory.12 Among many emerging techniques in the field, rubric evaluation of student work is an increasingly used assessment strategy in libraries and higher education that seeks to address the limitations of test-based and instructor-focused performance evaluation. Rubric evaluation is founded on the prin- ciple of outcomes-based and “student-centered” instruction, and Oakleaf in particular has made the case that rubric assessment is superior to test-based evaluation of student work.13 Similarly, Daniels and Choinski, Mark, and Murphy’s studies demonstrate the effectiveness of rubric evaluation of student learning in an IL instruction context.14 Previous research has shown positive effects of minimal or “one shot” instruction on student IL concept retention and skill performance; Spievak and Hayes-Bonahan conducted a rigorous study that determined significant increases in student IL follow- ing exposure to a single IL instruction session.15 Analyses of the effects of curriculum- integrated IL instruction tend to find that intensive course collaborations result in improved student learning and performance; Gilbert and Gilbert determined that political science students who participated in a three-month lab course demonstrated increased IL skill performance over students who did not participate in the lab.16 Hearn correspondingly demonstrated that a 10-session integrated IL model resulted in the integration of sources of higher quality as determined by citation analysis.17 Positive learning outcomes due to IL instruction are not universal, however; Emmons and Martin found little change in student source use in a rubric-based pre- and post-analysis of papers before and after a program of library instruction.18 Many authors have examined the efficacy of IL instruction within first-year seminar programs, while increased collaboration between librarians and subject faculty has been shown to have significant impact on the quality and breadth of IL instruction.19 Of particular relevance to the current project is Ghandi’s 2005 study in which a five- session, highly collaborative IL instruction model was compared to a one-shot IL instruction model in the same course; the former group of students showed marked improvement in IL concept understanding relative to the one-shot group, as well as greater motivation and satisfaction with the IL course component.20 Significantly, in the five-session model the librarian was granted access to the course syllabus and tai- lored their instructional delivery to assignment needs, while both faculty and librarian emphasized IL within content delivery. The effectiveness of rubric evaluation, outcomes-focused instruction, collaboration with academic faculty/staff, and the impact of IL scaffolding throughout into the cur- riculum are well-established concepts in the literature. However, with the exception of Ghandi who examined the positive learning and motivation outcomes of a collaborative and course-integrated model compared to a one-shot session, there is a dearth of research that investigates the progressive effects of IL instruction in librarian-faculty curricular collaborations of varying intensity levels. 21 In addition, while educational researchers Degrees of Impact 627 have demonstrated low levels of accuracy in student skills self-assessments (particularly among novices), few studies have compared self-perceived student IL learning and/or evaluations of librarian teaching effectiveness to “authentic” student post-instruction per- formance; instead, most extant literature has investigated student skills self-evaluations compared to actual IL skills performance.22 The present study seeks to address these gaps in the literature from a mixed-methods standpoint to 1) gauge the learning effects of IL course collaborations of varying depths and 2) validate the rigor of direct rubric assess- ment relative to student self-perceptions and instructional effectiveness evaluations. Methodology This study employed a mixed-methods design involving 1) rubric-based analysis of student research papers and 2) comparative evaluation of librarian teaching effective- ness and self-perceived student IL learning as reported by students and faculty in the same course collaborations; each methodology is described below in detail, and find- ings are reported in subsequent sections. Differential effects in these areas relative to librarian engagement depth were measured by characterizing each course collabora- tion within the paper sample on two scales: 1) Librarian Instruction Engagement Level (Lib) and 2) Syllabus/Assignment Design Collaboration Level (Syl). These scales describe the range of scenarios and teaching interventions that occur between faculty and CCL teaching librarians in first-year seminar programs: Librarian Instruction Engagement Level (Lib) 1. None = no course collaboration or instruction provided by librarian 2. Low = librarian conducts a one-shot instruction session/class visit, creates a course guide, or students only complete the online Start Your Research Tutorial (SYRT) and associated Sakai Quiz23 3. Moderate = librarian conducts 1–2 instruction sessions, creates a course guide, and students may complete the SYRT and Quiz 4. High = librarian conducts 2 or more instruction session(s)/class visits, creates a course guide, and students complete SYR Tutorial and Sakai Quiz Syllabus/Assignment Design Collaboration Level (Syl) 0. Librarian did not receive/see a syllabus 1. None = no mention of IL/IL skills in syllabus and no librarian input into research assignment(s) design 2. Low = brief mention of IL/IL skills on syllabus and/or modest input into research assignment(s) design by librarian 3. Moderate = IL/IL skills are directly integrated into course but not into a graded assignment, and/or significant input into research assignment(s) design by librarian 4. High = IL/IL skills are directly integrated into course and built explicitly into graded assignment(s), and/or major input into research assignment(s) design by librarian It is important to note that this study does not seek to imply that quantity of engage- ment is the same as quality of instruction. These levels were designed to capture the wide spectrum of teaching configurations and faculty engagement that occur across first-year instruction programs at CCL, and the number and variety of pedagogical and environmental factors affecting student learning performance contained therein are admittedly complex and difficult to isolate. The methodology employed herein is therefore not intended to gauge librarian teaching effectiveness per se; rather, it is 628 College & Research Libraries July 2015 intended to determine the effects of the overall depth of librarian involvement in the holistic student learning experience in each course pairing, which potentially involves interventions spanning from direct instruction to online tutorial/quizzes to one-on-one appointments to research assignment design. While pedagogical methods may differ dramatically across these course pairings due to the individual freedom allowed teaching librarians in the classroom, it is the opinion of the authors that higher and lower Lib and Syl levels are a fair representation of respectively greater and lesser intensity of student engagement with IL concepts over the course of the semester. To ensure representativeness of Lib and Syl at describing the diversity of first-year seminar experiences, levels were critiqued by CCL teaching librarians involved in first-year instruction prior to application in the study. To mitigate concerns about the validity of a study conducted across unstructured pedagogical approaches and multiple individuals with potentially disparate teaching efficacies, it should be noted that a) teaching librarians at CCL share common first-year IL learning outcomes and assessment strategies, engaging in a robust community of practice that continually discusses teaching methodologies in professional development events and makes many lesson and other teaching templates available for common use, and that b) 45 percent (n=4) of teaching librarians in the current study worked with two or more of the FYS course pairings at vastly different Lib and Syl engagement levels (for instance, 4/3 vs. 2/1 in one case, 2/0 vs. 3/2 in another), which decreases the potential of interpretation error related to research design and individualized teaching effects at specific levels. In late spring of 2012, following a collaborative faculty development workshop conducted by the Pitzer FYS program coordinator and three teaching librarians, faculty of record in each of the 13 sections included in the present study were paired with a total of 8 subject-affiliated teaching librarians via an introductory e-mail message from the library’s Head of Instruction Services that outlined resources and potential collaboration scenarios they might pursue. Following this initial contact, responsibility for course communication and planning transitioned to individual librarian instruc- tors. To provide a more granular pedagogical sense of the course collaborations that developed, summary descriptions of the experiences of three different participants in the present study (Librarians A, B, and C) and on-the-ground details of their reported Lib and Syl levels follow below. Librarian A—Lib Level 2, Syl Level 1: In this scenario, the lowest level of collabora- tion reported from courses in the study, the librarian in question followed up on the introductory e-mail several weeks after it was initially sent but received no immediate answer from the faculty member. As fall semester approached, the librarian success- fully established e-mail contact with the faculty member, who expressed interest in an early-term “one-shot” instruction session to introduce students to “library resources” such as article databases and the online catalog on a specific date that was convenient for the faculty member in terms of syllabus progression and inability to attend the session due to travel. No face-to-face discussion occurred between the librarian and the faculty member prior to course-related instruction, although several e-mails were exchanged. The faculty member declined to assign the Start Your Research online tu- torial and quiz to students but did express interest in a course-related research guide, which Librarian A created using LibGuides and linked to through Sakai (the colleges’ course management system). The faculty member’s syllabus was not available for review and discussion until shortly before the first day of classes (therefore preclud- ing a meaningful discussion of assignment design), but it was shared with Librarian A when completed. Librarian A’s 75-minute in-library instruction session focused on basic interdisciplinary information discovery tools such as Academic Search Premier Degrees of Impact 629 and the online catalog featured in their course LibGuide, with a largely demonstration- based curriculum with some time for hands-on practice. A research paper prompt involving minimal description of source use requirements was distributed to students at the beginning of the IL instruction session by the faculty member but had not been previously reviewed by the librarian, leading to on-the-fly and admittedly less effective attempts to address elements of the assignment related to IL during the session than if the prompt had been discussed with students and the teaching librarian in advance. Following the in-library session, student and faculty evaluations were conducted, and several students made individual 1–1 appointments with Librarian A to discuss their research process. Librarian B—Lib Level 3, Syl Level 3: In this scenario, which represents a slightly above-average depth of librarian instruction and faculty collaboration, the librarian in question established early contact with the faculty member following the introduc- tory e-mail and met several weeks before the beginning of the semester to discuss a draft syllabus and assignments as well as the specific needs around IL instruction. The faculty member was new to the institution and had not taught a FYS section in the past; they therefore expressed interest in discussing the course’s major research assignment relative to library instruction to help them frame the course. Based on librarian suggestions, the faculty member made several adjustments to draft research paper design and timeline (including the addition of a proposal and outline step and more specific sourcing requirements) as well as assigned the Start Your Research Tuto- rial to students prior to library instruction for a small course participation grade. The faculty member also requested an online research guide, moved the initial timing of in-library instruction to better align with research paper proposal/outline due date, and expressed interest in using the CCL IL rubric (see appendix A) as a way to gauge student progress on paper drafts related to IL. In spite of enthusiastic buy-in to librar- ian collaboration in the course, the faculty member declined to specifically state IL as a learning outcome in their syllabus, feeling that this language would not be useful to students but was preferable as a back-end principle to inform assignments and librar- ian instruction. Librarian B made an early-semester visit to the course’s classroom at Pitzer to make introductions and explain IL-related upcoming assignments and their role in the class, including demonstrating the Sakai-linked course site–linked research guide and the Start Your Research tutorial. Upon the request of the faculty member, the in-library instruction session (which the faculty member attended) was timed before students had a specifically defined research topic to facilitate “exploration,” so the session itself was highly activity-based and featured a “boring topic challenge” activ- ity wherein groups of students took a vague/general research area connected to the syllabus theme and developed it into highly specific and interesting potential research paper topics, then used article databases to identify potential source materials to help facilitate further research on their brainstormed topic. Students completed the Start Your Research Tutorial following the in-library workshop (an atypical timing of the tutorial, which usually occurs prior to IL instruction) to reinforce IL concepts intro- duced in the workshop. Several students made 1–1 appointments with the librarian to discuss in-progress research over the remainder of the semester. Librarian C—Lib Level 4, Syl Level 4: In this scenario, the highest level of collabora- tion reported in the study, the librarian in question established early contact with the faculty member following the introductory e-mail. The faculty member in question was coordinator for the FYS seminar program itself so was highly motivated to engage in deep course collaboration with Librarian C. This was the first time the faculty member had taught the course. Librarian C and the faculty member met multiple times during the summer before the fall semester to flesh out the syllabus, the research paper assign- 630 College & Research Libraries July 2015 ment, the scaffolding of the assignment, and the best places for the library instruction session(s). Based on these meetings, the faculty member specified IL as a course learning outcome and adopted the CCL IL rubric not only to grade student papers but as an instrument that was given to students so that they were aware of the faculty member’s IL expectations. The faculty member was interested in nontraditional research paper assignments and, after discussion with Librarian C, decided on a “Texts in Conver- sation” research assignment. This assignment asked students to explore an issue to determine both sides of the scholarly conversation. Students would then explore their opinions and viewpoints on the topic in the next assignment, which was a mock fel- lowship proposal. Librarian C met twice with the class. The first session was a typical hands-on session where students were exposed to databases and were able to start to explore them to find relevant articles on their research paper topics. Students com- pleted the Start Your Research tutorial and quiz prior to the session. The second class session delved more deeply into the evaluation of sources and was a discussion-type format. It should be noted that, in this collaboration, the faculty member took IL very seriously and worked diligently with students (including one-on-one appointments) to help develop those skills; the faculty member also attended both IL workshops. Levels of engagement were self-reported by all teaching librarians involved in the present study at the end of fall 2012, then associated with rubric evaluation data of student papers and course evaluations submitted by students and faculty to facilitate data analysis. It should be noted that the research design, methodology, and results employed in this study have recently been confirmed by a larger-scale (and still in progress at the time of this publication) CCL assessment project for the ACRL As- sessment in Action initiative involving first-year seminar paper rubric analysis at all five of our undergraduate colleges; preliminary findings corroborate those described in our results section across multiple learning communities and a wider spectrum of teaching librarians. (A) Rubric Analysis Methodology Investigators used the IL in Student Work Rubric to evaluate the Pitzer FYS first-year student paper sample (see appendix A). This CCL-developed rubric, modified signifi- cantly with permission from an original instrument created by librarians at Carleton College, assesses three of five IL “Habits of Mind” (HOMs) in authentic student writ- ing and other work: “Attribution”; “Evaluation of Sources”; and “Communication of Evidence.”24 The rubric is a widely used evaluation instrument at the Claremont Colleges that has been adopted for accreditation-level student assessment by several schools; it was employed by the same reviewer group on a similar Pitzer FYS paper analysis project in the summer of 2012 as well as numerous other student work as- sessment projects.25 It features four evaluation levels (1–initial; 2–emerging; 3–developed; and 4–highly developed) and was designed to facilitate assessment of IL within any type of student output, regardless of discipline, format, or enrollment status. To collect the student writing sample, investigators collaborated with the Pitzer Assessment Officer and Assistant Dean of Faculty to gather approximately 200 FYS culminating papers with their corresponding assignment prompts produced in the fall of 2012, all from courses that featured some degree of librarian/faculty contact if not substantive course engagement (n=13). In total, 99 student papers were randomly sampled and scored anonymously (including three rubric “norming” papers); the sum- mary of findings in this report reflects those papers identified within the sample that were deemed suitable for IL evaluation by virtue of including some type of external source integration. Sampling and evaluation methods were approved by Pitzer Col- lege’s Institutional Review Board (IRB). Degrees of Impact 631 Prior to grading, the five evaluators involved in this research study conducted a norming session to calibrate the implementation of the rubric, a practice common to all CCL rubric evaluation efforts and strongly recommended to ensure interrater reli- ability.26 Each librarian read three identical sample papers and scored them separately using the common rubric; they then met to discuss scores and come to a consensus on interpreting and applying rubric criteria consistently. Based in part on the group’s significant prior experience using the IL rubric to evaluate first-year student work, interrater reliability among the five reviewers across the three norming papers was strong to extremely strong (using Cohen’s Kappa, Attribution = 1.000, Evaluation = .800, Communication = .700).27 Following the norming exercise, each librarian individually scored 18–20 Pitzer FYS papers. Student rubric scores were then correlated to librarian course intervention level on the two previously described scales using a p-value test of statistical significance, 1) Librarian Instruction Engagement Level (Lib), and 2) Syllabus/ Assignment Design Collaboration Level (Syl). (B) Librarian Teaching Effectiveness and Student/Faculty Learning Perceptions Correla- tion Methodology Librarian collaborations in first-year courses at CCL feature several common methods of assessment, such as web-based summative instruction session surveys of librarian teaching effectiveness and perceived learning effects by students and faculty, as well as a confirmative web-based faculty survey assessment of student IL learning and librar- ian course collaboration impact at the conclusion of the semester (see appendix B for the student end-of-class survey; appendix C for the faculty end-of-class survey, and appendix D for the faculty end-of-term survey). These surveys include items gauging student and faculty perceptions of librarian teaching effectiveness and relevance to course assignments, as well as self-perceived student learning impact in each IL Habit of Mind area. Secondary analysis correlated the results of these summative and confirmative work- shop evaluations by students and faculty to student paper performance as determined by rubric assessment at each of the progressive collaboration levels; 1) Librarian Instruc- tion Engagement Level (Lib), and 2) Syllabus/ Assignment Design Collaboration Level (Syl). Summary of Findings A) Rubric Analysis Across the 99-paper sample, student performance in each IL HOM rubric category (Attribution, Evaluation, and Communication) averaged between the emerging (2) and developed (3): see appendix E for descriptive statistics. The “Communication of Evidence” CCL IL category indicates how well the writer integrates and synthesizes evidence to support his or her claims and/or thesis. This category had the highest student performance mean of 2.64 out of 4. Half of the papers (50%) scored at the developed level, and a similar number (43%) scored at the emerg- ing level. Seven percent scored highly developed, while none were at the initial level. “Evaluation of Sources” gauges the quality and appropriateness of source materials employed. “Evaluation” scores were extremely close to “Communication,” with a mean of 2.60. In this category, a majority of papers (52%) were developed, 38% were emerging, 4% were initial, and 6% were highly developed. “Attribution” indicates how well the writer documents communicated source materials and demonstrates understanding of citation formatting standards. This was the lowest-scoring category, with a mean of 2.32. For “Attribution,” almost half of papers (48%) scored at an emerging level. A similar number (36%) performed at the developed level. Twelve percent scored initial, and only 4 percent were highly developed. 632 College & Research Libraries July 2015 A central goal of rubric analysis was to determine what effects (if any) progressive degrees of librarian instructional and syllabus engagement in first-year classes had on student IL HOM performance as demonstrated in their end-of-semester writing assign- ments, as well as the effect of levels of faculty-librarian collaboration on assignment and syllabus design. Findings described in this section demonstrate an increasingly positive impact of higher levels of library instruction and librarian/faculty course col- laborations on student IL performance. When student paper rubric scores are correlated to Librarian Instruction Engagement Level (Lib), analysis indicates a clear pattern of escalating performance across all HOM areas corresponding to increasing instruction engagement levels (barring one outlier in Communication between 3 and 4). Total student IL performance differential by librarian instruction engagement level for all three HOM areas (Attribution, Evalu- ation, and Communication) is statistically significant between 4 and 2 (p-value .046) and between 4 and 1 (p-value .015).28 The below table shows rubric scores correlated to librarian Syllabus/Assignment De- sign Collaboration Level (Syl) in addition to Librarian Instruction Engagement Level (Lib). When student IL performance is correlated to Syl, gains are similarly significant to Lib. In Attribution, the student IL score differential between librarian engagement level 2 to 4 has a p-value of .046, and 1 to 4 reflects a p-value of .014. In Communication, the student IL performance differential from librarian engagement level 2 to 4 reflects a p-value of .046. Syllabi that included direct mentions of IL and/or reflected librarian collaboration to scaffold IL into assignments at higher levels demonstrate similarly significant gains in student IL performance, indicating that syllabus/assignment design collaboration is also an effective means of improving student IL learning. Areas of particular significance in this analysis include syllabus collaboration levels of 2 v. 4 in Attribution, which showed a .43 point positive differential in student per- formance at a p-value of .008, and at levels 1 v. 4 in Attribution, which showed a 1.16 point positive performance differential at a p-value of 0.000000778. In Evaluation, Syl Levels of 1 v. 4 showed a positive performance differential of .62 points at a p-value of .014 (Evaluation), and Syl Levels of 1 v. 4 showed a .52 point positive performance differential at a p-value of .027 (Communication).29 TABLE 1 Mean IL Rubric Performance in Student Paper Sample (n=99) Attribution Evaluation Communication Mean 2.32 2.60 2.64 TABLE 2 IL Rubric Scores by Librarian Instruction Engagement Level Librarian Instruction Engagement Level (Lib) Attribution Evaluation Communication Mean Score 2.32 2.60 2.64 4 2.56 2.76 2.76 3 2.31 2.54 2.85 2 2.19 2.52 2.52 1 2.10 2.40 2.40 Degrees of Impact 633 When the Librarian Instruction Engagement Level (Lib) is combined with Syllabus/ Assignment Design Collaboration Level (Syl), the proportional increase in student per- formance between the highest and lowest engagement levels is even more pronounced. (NOTE: Lib Level 2/ Syl Level 1 is the lowest combined involvement reported within the sample.) Combined student performance variation across all three HOM areas between 4/4 librarian engagement (highest) and 2/1 librarian engagement (lowest) is statistically significant (p-value .03). Importantly, performance variation in each HOM area are also statistically significant: Attribution p-value .000000778; Evaluation p-value .014; Communication p-value .027. TABLE 3 IL Rubric Scores by Librarian Instruction Engagement Level (Lib) and Syllabus/Assignment Design Collaboration Level (Syl) Librarian Instruction Engagement (Lib) and Syllabus/ Assignment Design Collaboration (Syl) Levels 4 4 3 3 2 2 1 1 0* Lib n=34 Syl n=28† Lib n=13 Syl n=14 Lib n=42 Syl n=19 Lib n=10 Syl n=10 Syl n=28 Attribution 2.56 2.64 2.31 2.50 2.19 2.21 2.10 1.30 2.36 Evaluation 2.76 2.82 2.54 2.57 2.52 2.53 2.40 2.20 2.57 Communication 2.76 2.82 2.85 2.71 2.52 2.63 2.49 2.30 2.54 *Note that level 0 for Syl only indicates no syllabus was shared by the faculty member, not necessarily that there was no discussion of assignments with the course librarian or mention of IL in the syllabus itself—scores at this level are considerably higher than Syl levels 1 and 2. †In this sample, all courses with level 4 syllabus collaborations also reported course engagement levels of 4, indicating a clear student learning impact when there is comparable depth of instructional and pedagogical collaboration between librarians and faculty in first- year courses. TABLE 4 Mean IL Rubric Scores by Combined Lib and Syl Minimum and Maximum Levels Combined Lib and Syl Levels Attribution Evaluation Communication Mean Score 2.32 2.60 2.64 4/4 2.64 2.82 2.82 2/1 1.30 2.20 2.30 634 College & Research Libraries July 2015 B) Librarian Teaching Effectiveness and Student/Faculty Learning Perceptions Corre- lated to Actual Student Performance Of the 13 FYS sections included in the paper evaluation sample, at least one of the three types of survey evaluations (student end-of-class; faculty end-of-class; and faculty end-of- term; see appendices B, C, and D) were received in 11 sections (a representative sample).30 When end-of-session workshop evaluations were compared to rubric scores at their respective course collaboration levels, investigators discovered no correlation between summative student IL instruction evaluations and Syl Level or Lib Level, and by exten- sion no correlation between perceived librarian teaching effectiveness or self-perceived student IL learning gains and actual IL performance in student writing. Of the five student survey questions related to self-perceived learning gains in IL HOM areas, a small percentage of students who received less face-to-face instruction rated librarian instruction as more effective than those who received more face-to-face in- struction (that is to say, there was a slight but insignificantly larger number of “strongly agrees” relative to “agrees” in Lib level 2, or lower, instruction scenarios compared to TABLE 6 Student IL Instruction Quality/Relevance Evaluations by Lib Levels 2 and 4 How relevant was library instruction to your course assignment(s)? (not at all relevant = 1—very relevant = 5) Lib Level 2 (n=3 Classes) Lib Level 4 (n=5 Classes) Rate the Overall Quality of Instruction Lib Level 2 (n=3 Classes) Lib Level 4 (n=5 Classes) 5 54% 53% Excellent 43% 45% 4 26% 32% Very Good 31% 32% 3 14% 13% Good 23% 19% 2 0% 0% Fair 3% 4% 1 6% 0% Poor 0% 0% TABLE 5 Survey Evaluations Received by Syl and Lib Levels Surveys Received by Level Lib = 4 Lib = 3 Lib = 2 Lib = 1 Syl = 4 Syl = 3 Syl = 2 Syl = 1 Syl = 0 Student End-of-Class (n=9 classes) (Rcvd Papers from 8 of 9 Classes) 5 1 3 0 3 1 2 1 2 Faculty End-of-Class (n=6) (Rcvd Papers from 5 of 6) 5 0 1 0 4 0 2 0 0 Faculty End-of-Term (n=7) (Rcvd Papers from 6 of 7) 4 2 1 0 3 2 2 0 0 Degrees of Impact 635 Lib Level 4, or higher, scenarios). This can be interpreted in a number of ways, from an inverse effect wherein higher level of engagement with librarians leads to a more critical and potentially less inflated assessment of librarian pedagogy; students who only experience a “one shot” may be more generous in their evaluations out of a sense of politeness or distance from the instructor. At the same time, students in Lib Level 4 collaborations rated librarian instruction as slightly but not significantly more relevant to their course assignments than in Lib Level 2 collaborations (85% compared to 80%). Similarly, investigators determined no correlation in assignment relevance or qual- ity of instruction ratings submitted by faculty by either Lib or Syl collaboration levels. Conclusion Conducting holistic assessment of student learning across varied IL instruction collabo- ration levels is a complex undertaking, creating the need for progressive assessment strategies that apply to highly individualized scenarios with incremental interventions. The research design employed in this study attempted to analyze the student learn- ing effects of applied IL collaborations as they “authentically” evolved across faculty/ librarian contexts ranging from negligible to intensive. Our analysis demonstrates that, irrespective of individual librarian, perceived teaching effectiveness, and self-perceived learning on the part of the student, course collaborations of disparate intensities have a differential effect on actual student IL learning. These findings strongly suggest that higher levels of faculty/student interaction with librarians (and thus more intensive engagement with IL concepts) improved authentic student IL performance in first- year seminar writing, in some cases dramatically. Simply put, the quantity of librarian engagement was a clear correlate to the quality of student learning. The well-defined pattern of proportional performance increase in both areas of analy- sis (course engagement and syllabus/assignment collaboration) suggests that while a “one-shot” model with minimal faculty engagement may indeed encourage IL concept retention, a course-integrated model with multiple diverse pedagogical interventions at the syllabus and classroom level consistently achieved greater performance gains. The authors recognize that this conclusion raises questions of scalability and resource limitations, particularly where direct librarian instruction is concerned. That said, the demonstrated efficacy of assignment design and syllabus consultation on student IL performance suggests that teaching-based interventions by librarians are not the only means of encouraging meaningful IL learning. When related skills and concepts are integrated formally into syllabus and assignment design through librarian consultation with program faculty (ideally creating a greater awareness of IL concepts on the part of the faculty and better scaffolded research assignments), performance gains can be achieved with a reduced time outlay in librarian-provided instruction. As others have argued, it is not student and faculty perceptions of librarian efficacy that we should be concerned with. Rather, it is devising methods for demonstrating the real effects of librarian contributions to learner understanding. The lack of correla- tion between summative student and faculty evaluations of librarian instruction/self- perceived learning and actual gains in student performance improvement at increasing levels of intervention underscores the need for authentic and direct skills assessment, librarian curricular integration, course-level design collaborations with faculty, and holistic evaluation of student work to determine the efficacy of IL interventions. At the Claremont Colleges Library, an ongoing focus on scaffolded research assignment design and tailored librarian course collaborations have supported increasing student IL performance gains at the first year. Outcomes-focused instruction, professional development for faculty and librarians, and the cultivation of intentional communi- ties of practice have been essential to providing a bridge across diverse syllabi and 636 College & Research Libraries July 2015 subjects while preserving the individuality of librarian pedagogy in course collabora- tions. Faculty development work and concerted relationship building with first-year program coordinators have led to a greater willingness on the part of faculty to engage in the deeper IL collaborations that make a difference in student IL learning, further supported by data that demonstrate success at improving student skills.31 Finally, findings suggest that sustaining library integration in first-year seminars and increasing pedagogical communication between faculty and librarians fosters greater focus on learning outcomes and a shared understanding of information literacy as an institutional priority. Curricular collaborations between librarians and faculty/staff coordinators of first-year and other academic programs can increase the efficacy and relevance of library instruction by facilitating shared outcomes and a “unified front” of skills assessment and messaging around IL competencies and expectations. As we have shown, this shared priority has the demonstrated potential to improve student IL learning and justify greater engagement between librarians and teaching faculty. D egrees of Im p act 637 APPENDIX A Claremont Colleges Library Information Literacy in Student Work Rubric v2013–14 Learning Outcome Level of Achievement Highly Developed 4 Developed 3 Emerging 2 Initial 1 Attribution Shows a sophisticated level of understanding for when and how to give attribution. • Documents sources consistently and completely • Uses in–text citation and notes correctly and consistently • Cites non–textual sources consistently • Names and labels figures and/or graphs clearly and completely. Attribution indicates understanding of the rationale for and various mechanisms of citation. • Documents sources throughout with occasional errors or inconsistencies. • Uses in–text citation and notes with occasional errors or inconsistencies • Cites non–textual sources with relative consistency • Usually names and labels figures and/or graphs clearly and completely. Missteps in attribution interfere with the argument or point to fundamental misunderstandings. • Frequently documents sources incorrectly or leaves out some citations. • Frequent errors and inconsistencies with in–text citation and notes • Does not consistently cite non–textual sources • Names and labels figures and/or graphs inconsistently. Use of evidence and citation is poor, making it difficult to evaluate the argument or sources. • Displays fundamental and consistent errors in source documentation • Does not include or contains significant inconsistencies with in– text citation and notes • Does not name, title, or cite non–textual sources • Does not name or label figures and/or graphs. Evaluation of Sources Source materials employed demonstrate expertise and sophisticated independent thought. • Demonstrates sophisticated awareness of universe of literature and community of scholarship Source materials are adequate and appropriate but lack variety or depth. • Explores supporting sources and community of scholarship but might overlook important avenues Source materials used are inadequate. • Exhibits weak awareness of universe of literature or other sources that could strengthen claim(s) or argument(s) Source materials are absent or do not contribute to claim(s) or argument(s). • No evidence of awareness of universe of literature or other sources that could strengthen claim(s) or argument(s) 638 C ollege & R esearch Lib raries Ju ly 2015 Evaluation of Sources • Uses a variety of appropriate and authoritative sources • Always distinguishes between types of sources (e.g., scholarly v. popular, fact v. opinion) • Demonstrates a thorough critical exploration and knowledge of evidence, theories, and sources selected” • Sources are used support claim(s) but may not be the most authoritative source to make claim • Usually distinguishes between types of sources (e.g., scholarly v. popular, fact v. opinion) • Demonstrates a preliminary critical exploration and knowledge of evidence, theories, and sources selected” • Relies on too few or largely inappropriate sources • Does not consistently distinguish between types of sources (e.g., primary v. secondary, scholarly v. popular, fact v. opinion) • Clearly selected sources out of convenience • Demonstrates little critical exploration and knowledge of theories and sources selected” • When included, sources are too few or badly inappropriate • No distinction between types of sources (e.g., scholarly v. popular, fact v. opinion) • Does not explore outside sources or present evidence when called for • No evidence of critical exploration and knowledge of theories and sources selected” Communication of Evidence Evidence is integrated and synthesized expertly to support claims. • Consistently presents evidence to support claim(s) and argument(s) • Synthesizes and contextualizes evidence appropriately for audience • Uses evidence instrumentally towards rhetorical goals • Distinction between own ideas and ideas of others is consistently clear • Identifies gaps in the literature and contributes creatively and/ or significantly to a scholarly conversation • Does not over– or under–rely on the ideas of others or the work of a single author" Proficient synthesis and integration of evidence. • Generally employs evidence to support claim(s) and argument(s) • May present some evidence without context • Frequently demonstrates using evidence instrumentally toward rhetorical goals • Distinction between own ideas and ideas of others is usually clear • Begins to identify gaps in the literature or contribute to a scholarly conversation • May over– or under–rely on the ideas of others or the work of a single author" Weak attempts at synthesis or integration. • Sporadically uses evidence to support claim(s) or argument(s) • Frequently fails to put sources into context (e.g. ""The World Bank says…"") • Usually does not demonstrate using evidence instrumentally toward rhetorical goals • Consistently blurs distinction between own ideas and ideas of others • Does not identify gaps in the literature or contribute to a scholarly conversation" No evidence of attempt at synthesis or integration. • Claim(s) or argument(s) lack necessary evidence • Fails to contextualize quotes and evidence • No demonstration of using evidence instrumentally toward rhetorical goals • No distinction between own ideas and ideas of others" Degrees of Impact 639 Information Literacy in Student Work Rubric Scoring Sheet—Claremont Colleges Library Identification ID Code ________Reader Name ________ Term/Year ______ Faculty.______________ Could not evaluate information literacy (IL) in this work? Check the box and you’re done. q Assignment A. Does the assignment ask students to use evidence outside of assigned course content? (check one) q Required q Allowed q Discouraged q No explicit mention q Assignment not available q N/A B. This work is a: . _______________________ (e.g., research paper, thesis, report, summary, argument, analysis, reflection, media project, other) Quality of attribution, evaluation, and communication of IL (see rubric for details): Highly Developed (4) Developed (3) Emerging (2) Initial (1) Comments Totals Attribution Evaluation of Sources Communication of Evidence Sum: OPTIONAL This work is a particularly representative example of the following (check any that apply): Elaboration (optional): Identification Fill out any available details regarding student work. q Very robust bibliography q Clear and consistent citations q Chose appropriate sources to support claims q Sources are well-integrated and synthesized q Shows awareness of depth of scholarship in area q Other ____________________________ q Egregious errors in bibliography, in- text citations, notes q Little or no attribution of non-textual elements q Inappropriate source(s) used to support claim q Sources not integrated or synthesized (e.g., “patch writing” or excessive block quoting) q Sources lack breadth or depth q Over/Undercited claims 640 College & Research Libraries July 2015 Can we evaluate information literacy in this work? Even if no sources are cited or the assignment does not call for outside sources, student work may exhibit information literacy if the student is placing their ideas in a broader context using ideas or information from other sources. Assignment A. Expectations about use of evidence outside of assigned course reading or other materials provided by professor (use N/A in the case of thesis or other work without defined assignment parameters). B. Assignment type allows us to determine how to evaluate works that fall outside the “standard” research paper (e.g. a report, thesis, summary, argument, analysis, reflection, media project, or other type of work) Quality of attribution, evaluation, and communication of Information Literacy For each category, check the appropriate box. (Highly Developed, Developed, Emerging, Initial) • Attribution refers to how well and how consistently the student acknowledges sources of evidence, including non-traditional formats such as lectures, e-mails, DVD commentaries, and images/figures as well as non-textual, embodied, reflective, and experiential materials. • Evaluation refers to the appropriateness or quality of source materials the student chooses to use to support their rhetorical goals (claims or arguments). This includes materials and sources in their bibliography (if available) as well as those used throughout the work. Do the sources, examples, and evidence selected match the purpose of the type of work and argument the student is creating? Is the student aware of the differences between primary and secondary sources, popular and scholarly sources, or fact and opinion? Have they selected the variety and quality of sources appropriate for their argument and work type? • Communication refers to the use and integration of sources as well as the quality of composition, e.g., whether the student has integrated the evidence they’re using and has done so in a way instrumental to their claim(s) and argument(s). Does the student paraphrase, summarize, synthesize, use quotes appropriately? Does the student frame quotations using authoritative sources? How are they using sources to ground their claims? This category also addresses how a student integrates their own ideas with those of others. OPTIONAL—This work is a particularly rich example of the following (check any that apply): Check an item when the noted characteristics are present and should be flagged as interesting or rich examples for future analysis or conversation. If you see other rich examples, note them as “Other.” Rubric content adapted for the Claremont Colleges by Char Booth (char_booth@cuc.claremont. edu), Sara Lowe (sara_lowe@cuc.claremont.edu), Natalie Tagge (natalie_tagge@cuc.claremont. edu), and Sean Stone (sean_stone@uc.claremont.edu) from an instrument originally developed at Carleton College—(Gould Library Reference and Instruction Department. "Information Literacy in Student Writing Rubric and Codebook." Northfield, MN: Carleton College. 2012. http://go.carleton.edu/6a). This rubric version (2013/14) was revised Summer-Fall of 2013 and finalized September 2013. Degrees of Impact 641 2/10/14, 10:52 AMFirst-Year Library Instruction: Student Evaluation, 2012-2013 Page 1 of 3https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dFMyd3Z0NVpMa2djZF9YNG1HbkY1WEE6MA#gid=0 Appendix B - 1st-Year Library Instruction: Student Evaluation, 2012 Thank you for taking the time to fill out this brief survey - your feedback will help us improve library research instruction sessions in the future. * Required Professor * Course Name * Campus * CMC HMC Pitzer Pomona Scripps Librarian(s) * How relevant was library instruction to your course assignment(s)? * 1 2 3 4 5 not relevant very relevant Identify one or more skills or other takeaways from today's session that will be useful to you. Could any of the topics/concepts covered have been explained more clearly or explored in greater depth? Appendix B. First-Year Library Instruction: Student Evaluation, 2012 Thank you for taking the time to fill out this brief survey—your feedback will help us improve library research instruction sessions in the future 2/10/14, 10:52 AMFirst-Year Library Instruction: Student Evaluation, 2012-2013 Page 2 of 3https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dFMyd3Z0NVpMa2djZF9YNG1HbkY1WEE6MA#gid=0 Indicate your level of agreement with the following statements: "Library instruction has helped me to..." * Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Find useful and appropriate resources. Evaluate the information I encounter. Use information resources effectively. Cite sources in my assignments. Access Library resources not available on the open web. Rate the overall quality of instruction. * Poor Fair Good Very Good Excellent Do you have any other questions or comments? If you would like your librarian to follow up with you about your comments on this form or if you have other questions for us, please include your NAME and EMAIL ADDRESS: Submit 642 College & Research Libraries July 2015 2/10/14, 10:52 AMFirst-Year Library Instruction: Student Evaluation, 2012-2013 Page 2 of 3https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dFMyd3Z0NVpMa2djZF9YNG1HbkY1WEE6MA#gid=0 Indicate your level of agreement with the following statements: "Library instruction has helped me to..." * Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Find useful and appropriate resources. Evaluate the information I encounter. Use information resources effectively. Cite sources in my assignments. Access Library resources not available on the open web. Rate the overall quality of instruction. * Poor Fair Good Very Good Excellent Do you have any other questions or comments? If you would like your librarian to follow up with you about your comments on this form or if you have other questions for us, please include your NAME and EMAIL ADDRESS: Submit Degrees of Impact 643 Appendix C. First-Year Library Instruction: Faculty Session Evaluation, 2012 Please take a few minutes to evaluate your Library research instruction session(s). These comments will help us assess our work and make improvements in the future. 2/10/14, 10:51 AMFirst-Year Library Instruction: Faculty Session Evaluation, 2012-2013 Page 1 of 2https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dEhoUXhNUFhaenpBNE02TVN3Q2wtNFE6MA#gid=0 Appendix C - 1st-Year Library Instruction: Faculty Session Eval, 2012 Please take a few minutes to evaluate your Library research instruction session(s). These comments will help us assess our work and make improvements in the future. * Required Your Name * Course Title * Campus * CMC HMC Pitzer Pomona Scripps Librarian(s) * Indicate your level of agreement with the following statements: "Library research instruction supported my students' abilities to..." * Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Find useful and appropriate resources. Evaluate the information they encountered. Use information resources effectively. Cite sources in their assignments. Access Library resources not available on the open web. How relevant was this session or sessions to course assignment(s)? * 1 2 3 4 5 not relevant very relevant 2/10/14, 10:51 AMFirst-Year Library Instruction: Faculty Session Evaluation, 2012-2013 Page 2 of 2https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dEhoUXhNUFhaenpBNE02TVN3Q2wtNFE6MA#gid=0 Which of the topics/concepts covered could have been explained more clearly or explored in greater depth? In your opinion, will librarian involvement have a positive impact on students' course experience and/or classwork? Please rate the overall quality of library research instruction. * Poor Fair Good Very Good Excellent Do you have any additional comments or suggestions? Submit Never submit passwords through Google Forms. Powered by Google Docs Report Abuse - Terms of Service - Additional Terms 644 College & Research Libraries July 2015 2/10/14, 10:51 AMFirst-Year Library Instruction: Faculty Session Evaluation, 2012-2013 Page 2 of 2https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dEhoUXhNUFhaenpBNE02TVN3Q2wtNFE6MA#gid=0 Which of the topics/concepts covered could have been explained more clearly or explored in greater depth? In your opinion, will librarian involvement have a positive impact on students' course experience and/or classwork? Please rate the overall quality of library research instruction. * Poor Fair Good Very Good Excellent Do you have any additional comments or suggestions? Submit Never submit passwords through Google Forms. Powered by Google Docs Report Abuse - Terms of Service - Additional Terms Degrees of Impact 645 Appendix D. First-Year Library Research Instruction: Faculty End-of-Term Evaluation, 2012–2013 Please take a few minutes to evaluate the overall impact of the Library-related component(s) of your first-year course this semester. Your comments will help us as- sess our work and make improvements in subsequent semesters. Thank you for your participation! 2/10/14, 10:54 AM Page 1 of 3https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dDRubW1yZ0RLRzhhTHdqczU4SUw4Tmc6MA#gid=0 First-YearFirst-Year LibraryLibrary ResearchResearch Instruction:Instruction: Faculty End-Faculty End- of-Termof-Term Evaluation,Evaluation, 2012-132012-13 Please take a few minutes to evaluate the overall impact of the Library-related component(s) of your first-year course this semester. Your comments will help us assess our work and make improvements in subsequent semesters. Thank you for your participation! * Required Your Name Your Name ** Course Title Course Title ** Campus Campus ** CMC HMC Pitzer Pomona Scripps 2/10/14, 10:54 AMFirst-Year Library Research Instruction: Faculty End-of-Term Evaluation, 2012-13 Page 2 of 3https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dDRubW1yZ0RLRzhhTHdqczU4SUw4Tmc6MA#gid=0 Librarian(s) Librarian(s) ** Please rate the overall quality of Please rate the overall quality of your experience working with librarian(s) thisyour experience working with librarian(s) this term.term. Poor Fair Good Very good Excellent What type(s) of library research instruction did your students participate in orWhat type(s) of library research instruction did your students participate in or use this semester?use this semester? Check all that apply. In-Library instruction In-class instruction Librarian/student appointments Online research guide Sakai resources Web tutorial(s) Other: Do you agree with the following statement?Do you agree with the following statement? "Library instruction had a positive impact on student coursework." Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Do you have additional comments or suggestions for improvement?Do you have additional comments or suggestions for improvement? 646 College & Research Libraries July 2015 2/10/14, 10:54 AMFirst-Year Library Research Instruction: Faculty End-of-Term Evaluation, 2012-13 Page 2 of 3https://docs.google.com/spreadsheet/viewform?usp=drive_web&formkey=dDRubW1yZ0RLRzhhTHdqczU4SUw4Tmc6MA#gid=0 Librarian(s) Librarian(s) ** Please rate the overall quality of Please rate the overall quality of your experience working with librarian(s) thisyour experience working with librarian(s) this term.term. Poor Fair Good Very good Excellent What type(s) of library research instruction did your students participate in orWhat type(s) of library research instruction did your students participate in or use this semester?use this semester? Check all that apply. In-Library instruction In-class instruction Librarian/student appointments Online research guide Sakai resources Web tutorial(s) Other: Do you agree with the following statement?Do you agree with the following statement? "Library instruction had a positive impact on student coursework." Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Do you have additional comments or suggestions for improvement?Do you have additional comments or suggestions for improvement? Degrees of Impact 647 Appendix E. Descriptive Statistics, 2012 CCL FYS IL Rubric Assessment 7 Appendix E – Descriptive Statistics, 2012 CCL FYS IL Rubric Assessment Attribution Evaluation Communication Fall 2012 2.32 2.60 2.64 Attribution Frequency Percent Valid Percent Cumulative Percent Valid 1 12 11.9 12.1 12.1 2 47 46.5 47.5 59.6 3 36 35.6 36.4 96.0 4 4 4.0 4.0 100.0 Total 99 98.0 100.0 Missing System 2 2.0 Total 101 100.0 Evaluation Frequency Percent Valid Percent Cumulative Percent Valid 1 4 4.0 4.0 4.0 2 38 37.6 38.4 42.4 3 51 50.5 51.5 93.9 4 6 5.9 6.1 100.0 Total 99 98.0 100.0 Missing System 2 2.0 Total 101 100.0 Communication Frequency Percent Valid Percent Cumulative Percent Valid 2 43 42.6 43.4 43.4 3 49 48.5 49.5 92.9 4 7 6.9 7.1 100.0 Total 99 98.0 100.0 Missing System 2 2.0 Total 101 100.0 648 College & Research Libraries July 2015 Frequency Percent Valid Percent Cumulative Percent Valid 2 43 42.6 43.4 43.4 3 49 48.5 49.5 92.9 4 7 6.9 7.1 100.0 Total 99 98.0 100.0 Missing System 2 2.0 Total 101 100.0 Degrees of Impact 649 650 College & Research Libraries July 2015 Notes 1. Megan Oakleaf, The Value of Academic Libraries: A Comprehensive Research Review and Report (Chicago: Association of College and Research Libraries, 2010), available online at www.acrl.ala. org/value/?page_id=21 [accessed 30 September 2013]. 2. Association of College and Research Libraries, Connect, Collaborate, and Communicate: A Report from the Value of Academic Libraries Summits, prepared by Karen Brown and Kara J. Malenfant (Chicago: Association of College and Research Libraries, 2012), 10, available online at www.ala. org/acrl/sites/ala.org.acrl/files/content/issues/value/val_summit.pdf [accessed 30 September 2013]; Jon R. Hufford, “A Review of the Literature on Assessment in Academic and Research Libraries, 2005 to August 2011,” portal: Libraries and the Academy 13, no. 1 (2013): 5–35. 3. Debra Gilchrist and Megan Oakleaf, An Essential Partner: The Librarian’s Role in Student Learning Assessment (NILOA Occasional Paper No. 14) (Urbana, Ill.: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment, 2012). 4. Association of College and Research Libraries, Connect, Collaborate, and Communicate, 3; Bruce Frey, “Defining Authentic Classroom Assessment,” Practical Assessment, Research & Evaluation 17, no. 2 ( 2012), 1, available online at http://pareonline.net/pdf/v17n2.pdf [accessed 30 September 2013]. 5. Melissa Bowles-Terry, “Library Instruction and Academic Success: A Mixed-Methods As- sessment of a Library Instruction Program,” Evidence Based Library & Information Practice 7, no. 1 (Apr. 2012): 82–95; Doug Lederman, “Raising the Bar on Quality Assurance,” Inside Higher Ed (Nov. 18, 2011), available online at www.insidehighered.com/news/2011/11/18/western-accreditor- pushes-boundaries-quality-assurance [accessed 30 October 2013]. 6. Sharon A. Weiner, “Institutionalizing Information Literacy,” Journal of Academic Librarian- ship 38, no. 5 (Sept. 2012): 287–93, doi:10.1016/j.acalib.2012.05.004. 7. Brandy Whitlock and Julie Nanavati, “A Systematic Approach to Performative and Au- thentic Assessment,” Reference Services Review 41, no. 1 (Feb. 2013): 32–48. 8. Weiner, “Institutionalizing Information Literacy.” 9. See http://libraries.claremont.edu/informationliteracy/documents/CCL_Information_Lit- eracy_Habits%20of%20Mind.pdf and http://libraries.claremont.edu/informationliteracy/docu- ments/CCL_First-Year_Instruction_IL_Learning_Outcomes.pdf [accessed 29 October 2013]. 10. Hufford, “Review of the Literature on Assessment”; Megan Oakleaf, “Dangers and Op- portunities: A Conceptual Map of Information Literacy Assessment Approaches,” portal: Libraries and the Academy 8, no. 3 (2008): 233–53. doi:10.1353/pla.0.0011. 11. Kathleen Dunn, “Assessing Information Literacy Skills in the California State University: A Progress Report,” Journal of Academic Librarianship 28, no. 1 (2002): 28. 12. Bowles-Terry, “Library Instruction and Academic Success”; Barbara J. D’Angelo, “Integrat- ing and Assessing Information Competencies in a Gateway Course,” Reference Services Review 29, no. 4 (2001): 282–93; Molly R. Flaspohler, “Information Literacy Program Assessment: One Small College Takes the Big Plunge,” Reference Services Review 31, no. 2 (2003): 139; Yvonne Mery, Jill Newby, and Ke Peng, “Performance-Based Assessment in an Online Course: Comparing Different Types of Information Literacy Instruction,” portal: Libraries and the Academy 12, no. 3 (Jul. 2012): 283–98. 13. Lorrie A. Knight, “Using Rubrics to Assess Information Literacy,” Reference Services Review 34, no. 1 (Feb. 2006): 43–55; Megan Oakleaf, “Using Rubrics to Assess Information Literacy: An Examination of Methodology and Interrater Reliability,” Journal of the American Society for Informa- tion Science and Technology 60, no. 5 (May 2009): 969–83. 14. See Erin Daniels, “Using a Targeted Rubric to Deepen Direct Assessment of College Stu- dents’ Abilities to Evaluate the Credibility of Sources,” College & Undergraduate Libraries 17, no. 1 (Jan. 2010): 31–43; Elizabeth Choinski, Amy E. Mark, and Missy Murphey, “Assessment with Rubrics: An Efficient and Objective Means of Assessing Student Outcomes in an Information Resources Class,” portal: Libraries and the Academy 3, no. 4 (2003): 563–75. 15. Elizabeth R. Spievak and Pamela Hayes-Bohanan, “Just Enough of a Good Thing: Indica- tions of Long-Term Efficacy in One-Shot Library Instruction,” Journal of Academic Librarianship 39, no. 6 (2013): 488–99. 16. J.K. Gilbert and C.P. Gilbert, “Assessing Integrated Library Components to Enhance In- formation Literacy in Political Science,” in Collaborative Information Literacy Assessments, eds. T. Mackey and T. Jacobson (New York: Neal-Schuman, 2010). 17. Michael R. Hearn, “Embedding a Librarian in the Classroom: An Intensive Information Literacy Model,” Reference Services Review 33, no. 2 (2005): 219–27. 18. Mark Emmons and Wanda Martin, “Engaging Conversation: Evaluating the Contribution of Library Instruction to the Quality of Student Research,” College and Research Libraries 63, no. 6 (2002): 545–60. Degrees of Impact 651 19. Julie Gilbert, “Using Assessment Data to Investigate Library Instruction for First-Year Students,” Communications in Information Literacy 3, no. 2 (2009): 181–92; John S. Riddle and Karen A. Hartman, “But Are They Learning Anything? Designing an Assessment of First Year Library Instruction,” College and Undergraduate Libraries 7, no. 2 (2000): 59–69; Sue Samson and Kim Granath, “Reading, Writing and Research: Added Value to University First-Year Experience Programs,” Reference Services Review 32, no. 2 (2004): 149–56; Bonnie J.M. Swoger, “Closing the Assessment Loop Using Pre- and Post-Assessment.” Reference Services Review 39, no. 2 (2011): 244–59; Lara Ursin, Elizabeth Blakesley Lindsay, and Corey M. Johnson, “Assessing Library Instruction in the Freshman Seminar: A Citation Analysis Study,” Reference Services Review 32, no. 3 (2004): 284–92; see also Christine Black, Sarah Crest, and Mary Volland, “Building a Successful Information Lit- eracy Infrastructure on the Foundation of Librarian-Faculty Collaboration,” Research Strategies 18 (2001): 215–25; Debra Hoffmann and Kristen LaBonte, “Meeting Information Literacy Outcomes: Partnering with Faculty to Create Effective Information Literacy Assessment,” Journal of Informa- tion Literacy 6, no. 2 (Dec. 2012): 70–85; Larry Sheret and John A. Steele, “Information Literacy Assessment: Keep it Simple, Keep it Going,” Reference & User Services Quarterly 52, no. 3 (2013): 208–15. 20. Smiti Gandhi, “Faculty-Librarian Collaboration to Assess the Effectiveness of a Five-Session Library Instruction Model,” Community & Junior College Libraries 12, no. 4 (2005): 15–45. 21. Ibid. 22. See Danny Kostons, “Training Self-Assessment and Task-Selection Skills: A Cognitive Ap- proach to Improving Self-Regulated Learning,” Learning and Instruction 22, no. 2 (2012): 121–32; Heather Coates, “Exploring the Disconnect Between Information Literacy Skills and Self-Estimates of Ability in First-Year Community College Students,” Evidence Based Library & Information Prac- tice 8, no. 2 (Apr. 2013): 264–66; Lana Ivanitskaya, Ryan Laus, and Anne Marie Casey, “Research Readiness Self-Assessment: Assessing Students’ Research Skills and Attitudes,” Journal of Library Administration 41, no. 1/2 (Aug. 2004): 167–83; Melissa Gross and Don Latham, “Attaining Infor- mation Literacy: An Investigation of the Relationship between Skill Level, Self-estimates of Skill, and Library Anxiety,” Library & Information Science Research 29, no. 3 (Sept. 2007): 332–53. 23. See http://libraries.claremont.edu/howto/researchtutorial/begin.html for the Start Your Research Tutorial. 24. See www.inthelibrarywiththeleadpipe.org/2011/csil-carleton-forensic-librarians-and- reflective-practices/; The Claremont Colleges Library defines IL as “the ability to use critical thinking to create meaningful knowledge from information. The information literate Claremont Colleges student engages in a process of inquiry in order to frame intellectual challenges and identify research needs; strategically accesses and evaluates information; communicates informa- tion effectively; provides clear attribution of source materials used; and develops insight into the social, legal, economic, and ethical aspects of information creation, use, access, and durability.” (IL “Habits of Mind” are emphasized in this definition). 25. To date, the CCL IL Rubric has been officially integrated into accreditation-level IL as- sessment processes by Claremont McKenna College and the Keck Graduate Institute. Numer- ous academic departments and writing programs in the other Claremont Colleges (Claremont Graduate Institute, Pitzer College, Scripps College, and Pomona College) have adopted the IL Rubric for assessment and pedagogical purposes since its initial creation in 2011 and revision in 2012–14. 26. Oakleaf, “Using Rubrics to Assess Information Literacy.” 27. Kappa of <0.2 is considered poor agreement; 0.21–0.4, fair; 0.41–0.6, moderate; 0.61–0.8, strong; more than 0.8, near complete agreement. 28. A p-value below 0.05 is generally considered statistically significant. 29. Declining p-values indicate increasing statistical significance, indicating particularly strong significance in the span between Engagement & Syllabus Levels 4–1. 30. Interestingly, Level 4 Syl and Lib classes were the only classes where librarians received all three evaluations. 31. See Dani Brecher, Char Booth, and Sara Lowe, “Librarians Matter Infographic” (2013), available online at http://libraries.claremont.edu/informationliteracy/images/FYS_Infographic. jpg [accessed 23 October 2013].