Making Strategic Decisions: Conducting and Using Research on the Impact of Sequenced Library Instruction 212 Making Strategic Decisions: Conducting and Using Research on the Impact of Sequenced Library Instruction Kacy Lundstrom, Pamela Martin, and Dory Cochran Kacy Lundstrom is Coordinator of Library Instruction, Merrill-Cazier Library, Utah State University, e-mail: kacy.lundstrom@usu.edu; Pamela Martin is Outreach and Peer Learning Librarian, Merrill-Cazier Library, Utah State University, e-mail: pamela.martin@usu.edu; Dory Cochran is Reference & Instruction Librarian, Merrill-Cazier Library, Utah State University, e-mail: dory.cochran@usu.edu. © 2016 Kacy Lundstrom, Pamela Martin, and Dory Cochran, Attribution-NonCommercial 3.0 (http://creativecommons. org/licenses/by-nc/3.0/) CC BY-NC 3.0. This study explores the relationship between course grades and se- quenced library instruction interventions throughout psychology students’ curriculum. Researchers conducted this study to inform decisions about sustaining and improving program integrations for first- and second-year composition courses and to improve discipline-level integrations. Re- searchers began with transcript analysis but soon incorporated student surveys and a faculty focus group to supplement the data and influence future directions. Findings confirmed that students benefit from meaning- ful collaborations with the library at strategic, sequenced points in their curriculum, including at the discipline level. This research also provided concrete information that brought about change at the classroom and programmatic level. Introduction Librarians at Utah State University (USU) Merrill-Cazier Library have partnered with select academic departments to map research-related student learning outcomes throughout the curriculum. The library instruction program has made large strides in moving toward a cohesive, integrated instruction program that targets students at strategic points in their curriculum, including at the general education level and in the disciplines. These efforts parallel the university’s curriculum mapping goals, including encour- aging departments to map learning outcomes across programs and courses. USU has also been involved in tuning processes in both the elementary education and history programs. These changes support a liberal education that defines outcomes clearly for students and future employers and that develops and reinforces skills throughout a student’s college experience. While the library has strong integrations at the general education level, especially in writing composition courses, its integration at the discipline level varies widely. doi:10.5860/crl.77.2.212 Making Strategic Decisions: 213 One of the departments that strategically integrates with library instruction is the psychology department. Currently, two courses that have research assignments in- corporate the subject librarian for a session to work with students in fulfilling their assignment. In order to expand the program, engage more deeply with departments on map- ping research skills, and ensure that students are more successful as a result of sequenced library instruction, more information was needed about the impact of current practices. A positive relationship between library instruction and improved academic performance would affirm the effectiveness of current practices for librar- ians, students, and stakeholders and would help garner similar collaborations with less integrated departments on campus. In addition, exploring the weaknesses and strengths of library collaborations for departments that did participate more actively would provide needed information about what assumptions might exist regarding those collaborations (e.g., anecdotally they seemed successful) and where improve- ments could be made. The initial research question asked if psychology students who participate in a sequence of courses that include library instruction at the general education (mainly in first- and second-year writing courses) and discipline levels receive higher grades. Later in the process, research questions became more specific, including which parts of a four-course curricular sequence were most (or least) effective and how those integra- tions could be improved in order to prepare students for research in their discipline. This emergent, developmental approach is a characteristic of action research.1 It reveals a responsiveness in the design approach and an orientation of finding meaningful information of the greatest use, even if that means changes to the process conceived at the outset. While the researchers hoped their work would be applicable and useful to other librarians, the initial catalyst for the study was a desire to gain insight to inform deci- sion making, particularly at the library instruction program level, which would also help librarians advocate at the institutional level. According to McMillan and Wergin, “The purpose of action/practitioner research is to go a step further and ask, What, if anything, should change?”2 The authors go on to discuss the standard for action re- search as based on its ability to influence decision making.3 Researchers for this study introduced two phases of assessment largely because after phase one they needed more information in order to make decisions and change their practice. Another important aspect of action research is its “participatory, democratic pro- cess.”4 This study was part of a broader professional opportunity called Assessment in Action, which involved multiple institutions participating in a community of in- quiry focused on conducting effective library assessment involving campus partners outside the library. USU librarians enlisted the help of numerous campus partners, including the Registrar’s Office, which retrieved the data; the Institutional Research Office, which assisted with data analysis; and the psychology department, which par- ticipated in surveys and focus groups. The library team included the Coordinator of Library Instruction (who served as team leader), the Public Services Associate Dean, the Outreach Librarian, and the Psychology Librarian. While not all partners were involved with every stage of the study, partners and librarians were active participants and contributors throughout the process. For example, when the library team met with the head of the psychology department, she expressed particular interest in how the sequence of a student’s program related to the student’s success. For this reason, researchers focused on the impact of taking courses in sequence in their analysis and later when they shared their findings. 214 College & Research Libraries March 2016 Background and Institutional Context/Current Integrations The library instruction program at USU relies on a highly course-integrated approach. Librarians are embedded in curriculum at the general education level in English 1010, “Introduction to Writing: Academic Prose” (ENGL 1010); and English 2010, “Intermediate Writing: Research Writing in a Persuasive Mode” (ENGL 2010); and to varying degrees in courses at the discipline level. Class size for ENGL 1010 and ENGL 2010 ranges from 20 to 24 students depending on the semester (caps for class size vary slightly by semester). The library instruction component of the ENGL 1010 curriculum changes year by year based on the collaborative work of the new graduate instructor assistant directors, the Writing Program director, and the Coordinator of Library Instruction. Together, they create the research assignment and the library instruction component, which is taught to the incoming graduate instructors. Usually, the assignment is a research paper, and the integration may involve a brainstorming session, a library research day session, and emphasis on additional research concepts, such as evaluation, synthesis, and critical thinking. The majority of the ENGL 1010 sections are taught by gradu- ate instructors. Each graduate instructor is paired with a librarian, and generally the same instructor continues to work with the same librarian throughout their teaching experience in both ENGL 1010 and in ENGL 2010, the course they teach their second year as graduate students. Because of this, nearly 100 percent of instructors of both writing courses have participated in the library instruction program for over a decade. Unlike ENGL 1010, ENGL 2010 is required for all students, and the curriculum varies slightly for each instructor, although the culminating assignment for all courses is the persuasive research essay. Rather than following the 1010 model of a set curriculum (including automatic integration with the library), the library instruction component is a collaborative effort between each teaching librarian and graduate instructor or lecturer. As is the case with 1010, these integrations typically consist of two or three visits with a librarian by the class that occur in the writing classroom and in the library instruction rooms. Recent learning outcomes for this class have emphasized helping students learn to synthesize information. Given that each partnership varied and that students included in this study were enrolled in ENGL 2010 between 2005 and 2015, it is likely that the students’ experiences with library instruction varied quite a bit. An online guide provides details and the specific learning outcomes for the current library integration in both courses (http://libguides.usu.edu/lessonplans). For the past two years, the current USU Psychology Librarian has benefitted from a positive relationship and pattern of integration established by the previous Psychology Librarian. The two key courses with librarian integration, offered both online and face- to-face, are Psychology 2010: “Orientation to Psychology as a Career and Profession” (PSY 2010), and Psychology 3500: “Scientific Thinking and Methods in Psychology” (PSY 3500). These courses are required for psychology majors, and students are advised to take PSY 2010 prior to PSY 3500. There is typically one face-to-face section and one to two online sections of PSY 2010 offered each semester. Likewise, there are typically two face-to-face sections and two online sections of PSY 3500 offered each semester. Class size for face-to-face and online sections of PSY 2010 is 100 students; class size is 30 to 40 students for PSY3500. The history of integration, research assignments, and typical library lessons taught differs between the two courses. Over the past two years, one instructor primarily taught PSY 2010, and the major- ity of library integration has been due to a successful partnership with this instructor. Approximately five face-to-face classes and five online courses have received library instruction with this faculty member. PSY 2010 students complete a short worksheet assignment focused on finding and evaluating a data-driven or scholarly article. In addi- tion to highlighting APA citation style, the worksheet asks students to identify different Making Strategic Decisions: 215 characteristics of a data-driven article (e.g., the methodology or purpose of the study). Face-to-face library instruction for this course includes one 50-to-75-minute session where the Psychology Librarian briefly demonstrates a LibGuide targeted towards the assignment and a practice search in the database PsycINFO. The majority of class time is devoted to one-on-one help between the librarian and students and between students and their instructor. Online library instruction for this course consists of embedding the class LibGuide (http://libguides.usu.edu/psy2010) and a short video into Canvas, USU’s learning management system. The video was created by the current Psychology Librarian and is meant to re-create the demonstrations given in an on-campus PSY 2010 library session. One lecturer and a rotating group of doctoral students typically teach the on-campus PSY 3500 sessions. Two professors primarily teach the online version of the class, with lecturers and doctoral students teaching online sessions as needed. As a result of this semester-by-semester change, the Psychology Librarian worked with seven different instructors over the past two years, including 10 face-to-face classes and 40 online courses. The research assignment for PSY 3500 includes an in-depth proposal assign- ment where students generate and develop a research question, review the relevant literature, and design a study to address their research question. Face-to-face library instruction for this course includes one 50-to-75-minute session where the Psychology Librarian demonstrates the class LibGuide (http://libguides.usu.edu/psy3500sperry) and a search in PsycINFO. PSY 3500 instructors requested this emphasis on short demos, as it gives students the majority of class time to search for articles and one-on- one time with the instructor and Psychology Librarian. Library integration with online sessions of PSY 3500 is limited, consisting of a class LibGuide embedded in the Canvas course. With a lack of concrete data on how many students used the LibGuide in either course, this article focuses on assessing library instruction given to face-to-face courses. Literature Review Libraries have long been interested in how to better integrate information literacy (IL) instruction and assessment into higher education. Recent examples of these efforts are Weiner’s suggestions for “Institutionalizing Information Literacy” and Oakleaf’s call for programmatic assessment that is both focused and extensive.5 Librarians are being actively encouraged to reach out beyond the library into the disciplines and ensure that IL is being taught strategically throughout campus. Such institutional integration of IL instruction should be tactical and precise. After discovering repetition and duplication of librarians’ efforts within the curriculum, Charles turned to curriculum mapping to better arrange library instruction throughout classes.6 Many other librarians have reported their attempts to integrate IL instruction strategically within specific disciplines, though strategic assessment of these programs (beyond student evaluations) is not described.7 These approaches to IL instruction allow librarians to teach sequentially, but the authors do not provide answers for program- matic assessment, with the exception of Charles, who ties assessment to IL curriculum maps and institutional “assessment cycles.”8 Assessment of library instruction has varied widely. In her article encouraging librar- ians “to articulate the value of academic libraries within an institutional context,” Oak- leaf reports that past IL assessment has ranged from student evaluations to authentic assessment to large-scale programmatic reviews.9 In particular, librarians have looked into student performance to answer Oakleaf’s call. Grade point average is a popular entry point, with a several studies comparing students’ GPAs across varying levels of library instruction.10 Course grades have also been examined, but not as widely.11 The research study described in this article is an extension of a previous authentic assessment study investigating the IL skills of USU students at various points in the 216 College & Research Libraries March 2016 curriculum.12 Like the one by Coulter and colleagues, the current study uses course grades to measure the impact of library instruction, focusing on a sequential set of Eng- lish and psychology courses first examined in the 2015 study.13 While most of the data comes directly from analyzing student transcripts, the research team also gathered some qualitative data to flesh out our study in a fashion similar to other library studies. Two notable mixed-methods examples are by Bowles-Terry and by Booth and colleagues.14 Methods Quantitative: Transcript Analysis This research project originally relied solely on quantitative methods, or Phase 1, which included transcript analysis of psychology students who had taken any of the four targeted courses. Researchers worked closely with the Institutional Review Board (IRB) office to follow protocol, and the Registrar’s Office retrieved student data on any psychology undergraduates who took any one of the four courses. The usable data retrieved by the Registrar included transcript information for 380 students, providing 761 data points representing students who had enrolled in variations of the four courses. Because library data that included all four courses went only as far back as fall 2005, data was retrieved for courses taken between fall 2005 and fall 2014. See table 1 for a breakdown of courses by number of students included. After the researchers received the student data from the Registrar’s Office, the librarian members of the research team matched it with library data, which included only whether or not the library had pro- vided library instruction for a course and how many sessions a course received. Library data did not indicate whether a particular student was present for or participated in library instruction. Once the library data and student data were matched, the research- ers met with the Analysis, Assessment and Accreditation Office multiple times to run analyses. Researchers parsed data to look for trends, looking at various subpopulations by category, including degree intent, age, race, gender, entry action, and time status. The analysis relied on Tableau software to help organize and visualize data. As the data was analyzed and new questions emerged, the research team decided to augment its quantitative approach with qualitative analysis. The team recognized that certain voices were missing. The researchers believed these voices could help improve the relationship between library instruction and student performance by helping them identify what changes could be made in the classroom and at the assignment level. Qualitative Research In order to flesh out the greater story surrounding the quantitative data findings, and to learn more about what changes might be implemented to strengthen the connec- tions, the research team engaged in two different methods of gathering qualitative data: surveys and focus groups (Phase 2). While the researchers were interested in understanding the library impact at each of the four stages (or courses), the research team was particularly interested in how library instruction at the general education level, that was then reinforced and extended to discipline research within the student’s major, had an impact on that student’s success, especially in their discipline courses. For this reason, they chose to target psychology instructors and psychology students TABLE 1 Total Students in Each Course Category Course # of (Indistinct) Students in Each Class ENGL 1010 189 ENGL 2010 250 PSY 2010 122 PSY 3500 200 Making Strategic Decisions: 217 in the qualitative portion of the study. Researchers also knew less about perceptions of students and instructors in psychology. In contrast, librarians had frequent opportuni- ties during past collaborations and assessments to learn more about student research skills, and perceptions of library instruction from both students and instructors in ENGL 1010 and ENGL 2010. Student Surveys The researchers created two online surveys using Qualtrics software—one for PSY 2010 and one for PSY 3500. The online surveys were administered to students who had taken a face-to-face section of PSY 2010 or PSY 3500 in spring 2015 and had received library instruction. Thirty-nine students responded in PSY 2010, and 31 responded from PSY 3500. See appendices A and B for focus group and survey questions. A member of the research team coded students’ responses for both of the surveys, specifically focusing on questions 4, 5, 7, and 8. These four questions were given greater attention due to their focus on stu- dents’ perceptions of their research skills and the library session, instead of assessing the instruction librarian’s performance. The themes emerged from content analysis and were created inductively. Following an initial coding, the research team member who coded the surveys met with the team member who coded the focus group transcripts. Identified themes were discussed, as well as similarities and differences within the qualitative data. This mixed-methods approach, which included transcript analysis, faculty focus groups, and online student surveys, gives a broad view of possible relationships be- tween the library instruction program and student achievement, as well as providing ideas for how to specifically change practice during instruction sessions. Faculty Focus Groups The focus group consisted of four psychology instructors who invited librarians into their classes; one of the participants teaches PSY 2010 and three teach PSY 3500. The focus group was recorded and later transcribed. A member of the research team then coded the transcriptions with tags to highlight recurrent themes. Themes were created inductively as the investigator listened to and read through the focus group’s conversa- tion, although she did look specifically for ties to the quantitative study. Findings Quantitative: Phase 1/Year 1 The findings indicated possible connections between library instruction and higher grades. They also indicated that the impact of library instruction increases when courses are sequenced and that, regardless of library instruction, students did better when they took their classes in sequence. Originally, researchers had included students who took ENGL 1010 in high school via concurrent enrollment. However, the library does not provide library instruction to most of those courses, and they are often modeled very differently from the courses offered on campus. For this reason, those students were excluded from the data analysis. As the data spans 10 years of ENGL 1010 assignments, there is variation among the type of library instruction students received, though all students received some hands-on research time in the library classroom with a librarian present. It is also important to note that many students do not take ENGL 1010 because they were either a concurrent enrollment student or they tested out of it. According to the data, in ENGL 1010, students who had library instruction were 9 percent more likely to get an A, but 5 percent less likely to receive a grade above C (i.e., either an A or a B). While researchers would have expected the positive relationship to remain true for the A/B category, an argument can be made that there is a significant 218 College & Research Libraries March 2016 difference between an A and a B in this particular course, given its very introductory nature, which does point to a positive correlation between library instruction and improved grades in ENGL 1010. See table 2. Students enrolled in ENGL 2010 who received an A grade showed a neutral rela- tionship to the presence of library instruction (a difference of 1 percent), but there was a positive connection for students who received library instruction. These students demonstrated a 7 percent increase in the A/Bs category. See table 3. Findings for PSY 2010 showed little positive relationship between library instruction and higher grades. Because not all sections of this course work with the library, the total number of students who had library instruction was much lower than anticipated (25). Students who had library instruction received slightly fewer As and the same number of A/Bs as those without library instruction. See table 4. TABLE 2 ENGL 1010 (Excluding Concurrent Enrollment)* ENGL 1010 A A– All As B+ B B– All Bs A/ Bs C or Lower Other Library Instruction (n = 155) 43% 23% 66% 9% 8% 5% 22% 88% 13% 3% No Library Instruction (n = 34) 40% 17% 57% 7% 27% 3% 37% 93% 10% *Percentages calculated do not equal 100 percent because students can take courses more than once across multiple semesters, resulting in duplication across the different dimensions used to break the numbers down. Percentages have been rounded for each category. This applies to all tables. TABLE 3 ENGL 2010 ENGL 2010 A A– All As B+ B B– All Bs A/Bs C or Lower Other Library Instruction (n = 204) 52% 17% 69% 10% 8 % 6% 24% 93% 7% 2% No Library Instruction (n = 45) 51% 19% 70% 9% 5% 2% 16% 86% 14% 2% TABLE 4 PSY 2010 PSY 2010 A A– All As B+ B B– All Bs A/Bs C or Lower Library Instruction (n = 25) 72% 4% 76% 8% 8% 0% 16% 92% 8% No Library Instruction (n = 97) 67% 13% 80% 3% 5% 2% 10% 91% 9% Making Strategic Decisions: 219 The impact of library instruction at the discipline level is most apparent in PSY 3500. Students in this class were 5 percent more likely to receive an A and 4 percent more likely to receive an A or B if library instruction was present. See table 5. Sequenced Library Instruction When the data was parsed to look at the impact of sequence on a student’s grades, the total number of students was too small to be useful, so the researchers decided to focus particularly on the impact of sequenced courses at the discipline level. As noted earlier, the department head was also particularly interested in data on the sequence of PSY 2010 and PSY 3500, which reinforced the researchers’ intention to focus on those two courses. The researchers defined sequenced instruction as the instruction received by students who took PSY 2010 prior to taking PSY 3500; they then looked for variations in grades for PSY 3500. Regardless of library instruction, students who took PSY 2010 prior to taking PSY 3500 received higher scores (10.5 percent more A/Bs and 5 percent more As). See tables 6 and 7. TABLE 5 PSY 3500 PSY 3500 A A– All As B+ B B– All Bs A/Bs C or Lower Library Instruction (n = 98) 37% 22% 59% 11% 15% 6% 32% 92% 8% No Library Instruction (n = 102) 34% 20% 54% 17% 11% 7% 35% 88% 16% TABLE 6 Sequenced Library Instruction (Took PSY 2010 First) PSY 3500 A A– All As B+ B B– All Bs A/Bs C or Lower Library Instruction (n = 50) 42% 20% 62% 18% 14% 4% 36% 98% 2% No Library Instruction (n = 20) 37% 21% 58% 32% 5% 0% 37% 95% 5% TABLE 7 Non-sequenced (Did Not Take PSY 2010 First) PSY 3500 A A– All As B+ B B– All Bs A/Bs C or Lower Library Instruction (n = 48) 31% 25% 56% 4% 17% 8% 29% 85% 15% No Library Instruction (n = 82) 34% 20% 54% 13% 13% 8% 34% 87% 18% 220 College & Research Libraries March 2016 When looking at how sequence impacts library instruction, there was an increase of 4 percent of As, and 3 percent of A/Bs for students who had library instruction. For non-sequenced students, there was a 2 percent increase of As for students with library instruction and a 2 percent decrease for students in the A/Bs category (a 5 percent de- crease for Bs). This indicates that library instruction has a greater positive relationship when classes are sequenced. See table 6 and 7. While some of these connections are slight, the authors believe the feedback they re- ceived from students and faculty in the qualitative portion will increase those connections. Qualitative: Phase 2/Year 2 Survey Results In the surveys, students were asked what they learned in the library lesson, what aspects of research they wanted to know more about, what parts of research they struggle with, and what aspects of research are easier. According to the coded student surveys, students from both classes reported that the library lesson primarily helped them with learning where to search and how to search in psychology databases. While students stated that this was helpful, the identical reports indicate that library instruction for PSY 3500 could tackle higher-order learning outcomes as it is typically taken after PSY 2010. In PSY 2010, students self-identified as mostly struggling with how to find “good research,” yet finding and summarizing information were noted as easier. In comparison, these findings indicate that students can easily find articles, but are uncertain of how to find the best sources for their specific assignment or purpose. PSY 2010 students also indicated a high level of difficulty with properly citing sources. On the other hand, PSY 3500 students indicated citing as an easier aspect of research. Another major theme from PSY 3500 student responses was a concern over the time needed to research. A number of students identified strategies for speed-reading or efficiently finding resources as an aspect of research they wanted to know more about. Overall, struggles identified by students were almost completely flipped in ranking between the two classes. However, aspects of research that students felt were easier remained similar across both courses and primarily focused on summarization. When responding to the question What parts of research do you struggle with the most? PSY 2010 students reported struggling most with knowing where to start (69 percent), followed by finding relevant information (41 percent). PSY 3500 students reported being most concerned about evaluating information and finding relevant information (52 percent). See table 8. TABLE 8 Student-Reported Areas of Struggle What parts of research do you struggle with the most? (Choose up to 3 that best represent you)* PSY 2010 (n = 39) PSY 3500 (n = 31) Knowing where to start 69% 45% Evaluating information 23% 52% Summarizing information 26% 26% Synthesizing information 33% 45% Using library databases 23% 3% Finding relevant information 41% 52% Citing sources 38% 35% Understanding the information 26% 19% Other 5% 10% *Possible answers provided in survey. Making Strategic Decisions: 221 Students in PSY 2010, who may or may not have taken PSY 3500 previously, reported feeling more prepared for conducting research in their discipline than students in PSY 3500. See table 9. This may be because students new to the discipline do not fully understand what constitutes research in their major or even recognize that there are specific research practices particular to their discipline. Seventy-six percent of PSY 2010 students also reported that the library instruction helped a lot with the assignment, versus only 41.94 percent of PSY 3500 students reporting the same (51 percent of PSY 3500 students said it “helped some”). See table 10. Focus Group The focus group consisted of four psychology instructors who have invited librar- ians into their classes. One of the participants teaches PSY 2010; three of them teach PSY 3500. The focus group findings primarily centered on expectations of each class, the role of the library in students’ success, and worries about overloading students. Differences in the classes were discussed, causing one instructor to state: “I am get- ting educated in terms of what they are covering in 2010 ’cause I don’t think I fully understood exactly.” PSY 2010 is an “introduction to the major” of psychology with very limited library research. Students have one assignment requiring them to find a data-driven article and complete a worksheet about the article. PSY 3500 is a com- municative intensive course, a designation that requires extensive verbal and written communication, with a large research assignment included. Students have to write a literature review using scholarly sources. Instructors were concerned about many IL aspects, though they were not certain if the library could address all of them: “I see some roles that certainly the library can play in that process and that can really be helpful, and even more so than what we do right now.” Instructors also commented on low student engagement. They worried that students suffer from information overload and struggle to locate relevant sources. Participants expressed concern that students need to be able to critique articles and become “good consumers of information.” One instructor stated that it’s important to teach “students skills to critique information and consume it in a way that makes it digestible.” Synthesis skills were also an area of major concern, mentioned repeatedly by all of the PSY 3500 instructors. A resounding theme within results of both surveys and focus group was on the topic of synthesis. All of the PSY 3500 instructors agreed that synthesis skills were a major area of concern. In an interesting contradiction, PSY 3500 students noted in TABLE 9 Student Confidence in Discipline Research Skills I feel prepared to successfully conduct research in my discipline. Not Prepared Somewhat Prepared Very Prepared PSY 2010 (n = 39) 7.69% 41.03% 51.28% PSY 3500 (n = 31) 3.23% 54.84% 41.94% TABLE 10 Helpfulness of Session Related to Assignment This session helped me with my research assignment. Not at All Helpful Not Very Helpful It Helped Some It Helped a Lot PSY 2010 (n = 39) 0% 5.13% 17.95 76.92% PSY 3500 (n = 31) 0% 6.45% 51.61% 41.94% 222 College & Research Libraries March 2016 the surveys that synthesizing information was both what they struggle with the most and an easier aspect of research. One explanation for this contradiction could be that some students do not fully understand what synthesis is. For example, one student noted they find it easy to put “information I find into my own words,” while another noted “It’s easy once I get more info on a certain topic to bring everything together for a large summary.” Taken together these comments seem to show students confuse summarizing and synthesizing. Overall, the qualitative data shows that higher-order thinking skills, like synthesis, are what students struggle with the most when researching and writing. While this result is not surprising, it opens up conversations on how the allocation of time and types of library activities conducted in library instruction can better help students develop these complex skills. Discussion Changes in Practice As a result of this study and subsequent conversations between the psychology depart- ment and the library, several instructors expressed greater willingness to devote more class time to library integration and to focus on higher-order thinking skills rather than on database demonstrations. One major finding from this study is that online versions of PSY 2010 and PSY 3500 receive limited library instruction compared to on- campus sessions. In response, a major change for PSY 2010 has been the integration of optional online conference sessions. In the summer of 2015, the Psychology Librarian polled online students about possible meeting times and, based on feedback, held two conferences through Adobe Connect where students could ask questions about their assignment and get direct library help. Out of a class with 18 students, five participated in these online conferences. The instructor was pleased with this new integration, and online conferences are continuing to be offered in PSY 2010 as scheduling allows. Instructors teaching online versions of PSY 3500 still prefer limited library integra- tion delivered through a class LibGuide. Therefore, the Psychology Librarian used feedback from the focus group and surveys to revise the LibGuide. While integration with PSY 3500 online sessions has not changed in its scope or delivery method, each face-to-face PSY 3500 instructor allowed the Psychology Librarian to use the full class session for a new lesson in fall 2015. One major point of change in the face-to-face library instruction for PSY 3500 was a decrease in time spent on demonstrating databases. In fall 2015, the Psychology Librarian did not demonstrate library resources and instead incorporated video tutorials and materials in the class LibGuide for students to view on their own time and at their own point of need. The majority of class time was then devoted to an active learning exercise where students use a synthesis matrix tool to explore similarities and gaps within their research. However, when students were given research time in these fall PSY 3500 sessions, it quickly became clear that stu- dents’ familiarity with library resources varied. An important lesson learned from this experience is that a successful library session for PSY 3500 students needs to address both lower- and higher-order research skills. In response, the Psychology Librarian will employ a flipped learning approach in spring 2016, where students watch a tutorial on how to use PsycINFO and answer related questions prior to their library session. Another major change in practice for the face-to-face PSY 3500 sessions is that the revision of this lesson was a collaborative endeavor between the Psychology Librar- ian and an undergraduate library student worker who is a psychology major and had previously taken PSY 3500. This revised lesson was co-taught by the Psychology Librarian and a student worker. While this collaborative approach has not been fully assessed yet, students’ participation in the class activity and their interactions with both Making Strategic Decisions: 223 the Psychology Librarian and the student worker reflect openness and appreciation of learning in a co-teaching environment and from a fellow student and psychology major. At the end of the focus group with psychology instructors, the library moderators did receive some resistance when they shared preliminary quantitative data from the transcript analysis of student grades. This resistance was evident in questions regard- ing the interpretation of the evidence and the type and strength of claims that could be made from those interpretations. The instructors’ response to this data was helpful to the research team, encouraging them to consider alternative interpretations and to be critical of their own assumptions. Some instructors seemed reluctant to believe that librarian presence once in the semester would correlate to a student’s grade. Also, the connections were fairly small, especially in PSY 2010, so those findings were not as supportive of library impact and therefore not as convincing as the researchers had originally hoped. It might have helped to share the data at a later date instead of during the focus group, which was rushed, and to more clearly explain that researchers were not talking about just one library session in a class but a series of targeted, meaningful integrations across a student’s experience. Overall, this study created opportunities for positive changes within both PSY 2010 and PSY 3500. This collaborative project resulted in ongoing productive conversations with the psychology department head. A discussion with the department head in the summer of 2015 about the results of this study introduced two new possible areas for future library integration: the PSY 4950 Undergraduate Apprenticeship and point-of- need help for graduate students. While there was no integration in PSY 4950 or new integrations with graduate students in fall 2015, the Psychology Librarian is in the process of assessing the needs of these areas and what can realistically be offered, given constraints of the Psychology Librarian’s teaching load and considerations of sustainability. Programmatic Changes One of the major discussions and potential decisions prompted by this research is looking more closely at the impact of library integration with ENGL 1010. While the library’s integrations with both ENGL 1010 and ENGL 2010 continue to comprise a major portion of the library’s information literacy efforts, the number of sessions of- fered by the English department has increased dramatically over the past five years, making it difficult to sustain our highly embedded curriculum in its current model. Librarians are involved in a related program assessment study looking specifically at the impact of library instruction in a student’s first-year experience, which will pro- vide more information on the impact of our current integration and promote more discussion on viable options for changing how we provide library instruction at the general education level. New questions relating to the larger library instruction program emerged that ne- cessitate further action research, including exploring how changing or eliminating the ENGL 1010 library integration would impact the ENGL 2010 collaboration. Librarians are concerned that not building relationships with those instructors (new teachers in their first year as graduate students) may decrease their ability to build rapport. This could jeopardize the continuing of a near-100 percent involvement in library instruc- tion partnerships for second-year writing composition courses. These results are also being used as librarians continue to target courses and build library instruction into programs and departments. Last year, subject librarians presented to departments in order to start conversations relating to targeted instruc- tion and articulating research learning outcomes at the program level. As librarians continue to follow up on these outcomes and suggested courses, the findings of this 224 College & Research Libraries March 2016 study can serve as further evidence of how to think about and design instruction with the most impact. Campus Partners and Communities of Inquiry While the findings of this study have been useful in informing decisions and improv- ing library teaching practices, the process of conducting research as a team was just as useful to the authors in many ways. One of the major results is that more people on campus know librarians do this type of research, especially in the psychology depart- ment and in the Office of Analysis, Assessment and Accreditation, with which librarians also work with on issues relating to accreditation. The team leader (the Coordinator of Library Instruction) developed a community of inquiry with campus partners and fel- low librarian members of the research team by requesting involvement when needed, while also trying to remain sensitive to workloads. She kept communication open by sending updates and heavily encouraged feedback from team members, which helped to facilitate a more democratic, community-centered approach. As a result, there is a strong interest in the research findings and a willingness among our campus partners to pursue future collaborations. And, like other studies and trends in research on USU’s campus, the focus of this study is student learning and integrating into the curriculum. Librarian curriculum mapping efforts have helped in these efforts, which support university-wide efforts to map skills throughout general education and the major. Conclusion The primary contributions of this study to the library are its ability to help the library team make informed decisions that improve the impact of their work on student suc- cess. Research findings confirmed that students benefit from meaningful integrations with the library at strategic points in their curriculum, including at the discipline level. It also revealed that the current library integrations at the discipline level are failing to reach a significant number of students in face-to-face and online psychology courses. As a result, the library has made some significant changes at the classroom and pro- gram already, although decision making, sharing, and use of this research are ongoing. Conducting this study has also increased librarians’ proficiency with conducting action research. The research team was able to identify allies on campus for future as- sessment projects, and they learned a lot about how to improve their research methods and refine their questions. Conducting this study has also put librarians on the radar of new university groups and expanded campus faculty understanding of what librar- ians do and how they impact student success, increasing librarians’ opportunities to be included in conversations relating to the university curriculum. Learning the impact of sequenced library instruction is also valuable to the profes- sion. It can help shape future integrated instruction approaches and will serve as an example of librarians leading institutional curriculum mapping efforts. These findings are important not only for teaching librarians but also for anyone interested in tying li- brary instruction to student learning and to the larger goals of the faculty and university. More research is needed in exploring the impact of sequenced, multi-session embed- ded library instruction in key courses, especially in how to provide these integrations in combination with online and regional campus courses. Additional future areas of research include investigating ways to increase librarian involvement in institutional conversations about the broader curriculum. This study provides one example of how librarians can have and are having a positive impact on student learning and how conducting and sharing these efforts enhances and provides future collaborative relationships. Making Strategic Decisions: 225 Appendix A: Focus Group Questions 1. What research skills do students need to have to be successful in your assignment? • In your class (PSY2010 and PSY3500)? • In your program? • For the workforce? 2. What areas of research do students struggle with the most? 3. What role do you see the library playing in helping students learn those skills? 4. Do you think the sequence of these research skills in these courses is important? Why and how so? 5. Are there other courses in the program that might be useful in helping teach these skills (or are already teaching them)? 6. To what degree does research proficiency relate to academic success (if at all)? 7. What changes would you like to see to the libraries integration with your program? 8. What do you think of the data and the strength of the conclusions thus far? Appendix B: Survey Questions Do you feel prepared to successfully conduct research in your discipline? (Likert) Was this session helpful in meeting the criteria of your research assignment? What did you learn? What do you wish you knew more about? Overall rate this session and teacher? What part of research do you struggle with the most (choose up to 3 that best repre- sent you)? Where to start, credibility, summary, synthesis, exploration, annotated bib, citation, sifting through lots of info, Other, writing in the discipline, using databases What aspects of research come are easier you? (open ended) 226 College & Research Libraries March 2016 Notes 1. Peter Reason and Hilary Bradbury, Handbook of Action Research (London: SAGE Publica- tions, 2006), 2. 2. James H. McMillan and Jon F. Wergin, Understanding and Evaluating Educational Research, 4th ed. (Boston: Pearson, 2010), 164. 3. Ibid, 165. 4. Reason and Bradbury, Handbook of Action Research 1. 5. Sharon A. Weiner, “Institutionalizing Information Literacy,” Journal of Academic Librarianship 38, no. 5 (September 2012): 287–93, doi:10.1016/j.acalib.2012.05.004; Megan Oakleaf, The Value of Academic Libraries: A Comprehensive Research Review and Report (Chicago: Association of College and Research Libraries, 2010), http://www.acrl.ala.org/value/?page_id=21. 6. Leslin H. Charles, “Using an Information Literacy Curriculum Map as a Means of Com- munication and Accountability for Stakeholders in Higher Education,” Journal of Information Literacy 9, no. 1 (June 2015): 47–61, doi:10.11645/9.1.1959. 7. Laurie A. Kutner, “Library Instruction in an Interdisciplinary Environmental Studies Program: Challenges, Opportunities, and Reflections,” Issues in Science & Technology Librarian- ship, no. 28 (2000), doi:10.5062/F4K935HD; Beth Christensen, “Warp, Weft, and Waffle: Weaving Information Literacy into an Undergraduate Music Curriculum,” Notes, Second Series, 60, no. 3 (March 1, 2004): 616–31; Sonia Bodi, “Relevance in Library Instruction: The Pursuit,” College & Research Libraries 45, no. 1 (January 1984): 59–65. 8. Charles, “Using an Information Literacy Curriculum Map.” 9. Megan Oakleaf, “Are They Learning? Are We? Learning Outcomes and the Academic Library,” Library Quarterly 81, no. 1 (January 2011): 77. 10. Shun Han Rebekah Wong and Dianne Cmor, “Measuring Association between Library Instruction and Graduation GPA,” College & Research Libraries 72, no. 5 (September 2011): 464–73; Melissa Bowles-Terry, “Library Instruction and Academic Success: A Mixed-Methods Assessment of a Library Instruction Program,” Evidence Based Library & Information Practice 7, no. 1 (April 2012): 82–95; Jason M. Vance, Rachel Kirk, and Justin G. Gardner, “Measuring the Impact of Library Instruction on Freshman Success and Persistence,” Communications in Information Literacy 6, no. 1 (March 2012): 49–58. 11. Priscilla Coulter, Susan Clarke, and Carol Scamman, “Course Grade as a Measure of the Effectiveness of One-Shot Information Literacy Instruction,” Public Services Quarterly 3, no. 1/2 (January 2007): 147–63, doi:10.1300/J295v03n01_08. 12. Wendy Holliday, Betty Dance, Erin Davis, Britt Fagerheim, Anne Hedrich, Kacy Lundstrom, and Pamela Martin, “An Information Literacy Snapshot: Authentic Assessment across the Cur- riculum,” College & Research Libraries 76, no. 2 (March 1, 2015): 170–87, doi:10.5860/crl.76.2.170. 13. Coulter, Clarke, and Scamman, “Course Grade as a Measure.” 14. Bowles-Terry, “Library Instruction and Academic Success”; Char Booth, M. Sara Lowe, Natalie Tagge, and Sean M. Stone, “Degrees of Impact: Analyzing the Effects of Progressive Librarian Course Collaborations on Student Performance,” College & Research Libraries 76, no. 5 (July 2015): 623–51, doi:10.5860/crl.76.5.623.