The Librarian Leading the Machine: A Reassessment of Library Instruction Methods 286 The Librarian Leading the Machine: A Reassessment of Library Instruction Methods Katie Greer, Amanda Nichols Hess, and Elizabeth W. Kraemer Katie Greer and Amanda Nichols Hess are Assistant Professors and Elizabeth W. Kraemer is Associate Professor in Kresge Library at Oakland University; e-mail: greer@oakland.edu, nichols@oakland.edu, kraemer@oakland.edu. © 2016 Katie Greer, Amanda Nichols Hess, and Elizabeth W. Kraemer, Attribution- NonCommercial (http://creativecommons.org/licenses/by-nc/3.0/) CC BY-NC. This article builds on the 2007 College and Research Libraries article, “The Librarian, the Machine, or a Little of Both.” Since that time, Oakland University Libraries implemented changes to its instruction program that reflect larger trends in teaching and assessment throughout the profession; following these revisions, librarians undertook a new study to assess the effectiveness of online library instruction in face-to-face courses in comparison to the hybrid model that had been in place. The study’s design and its results serve to contribute to discussion of best practices in information literacy pedagogy, online learning, instructional design, and the role of the librarian therein. n 2005, librarians from Oakland University (OU), a doctoral research insti- tution located in metropolitan Detroit, conducted a study to evaluate and compare the effectiveness of three different library instruction delivery methods: all online, all in-person, and a hybrid model. The methodology and in-depth results of this study were published in the 2007 College & Research Libraries article “The Librarian, the Machine, or a Little of Both: A Comparative Study of Three Information Literacy Pedagogies at Oakland University.”1 Since that time, educators and librarians all over the world have conducted research and provided further in- sights into how humans learn and how information literacy can be effectively taught, especially in the online environment. Moreover, new technologies have proliferated that allow for increasingly effective instructional design and assessment of learning. This research study uses the 2007 study as its foundation while accounting for recent assessment trends, the more established online instructional environment in higher education, and the increasing importance of instructional design in information literacy instruction. Moreover, it provides a new perspective on instructional options for aca- demic librarians. This study specifically considers whether online information literacy instruction in an otherwise face-to-face course is effective in helping students achieve desired learning outcomes. While there are many studies that affirm the effectiveness of online learning for learners who self-select online learning as their desired delivery method, this study specifically examines whether online library learning is as effective doi:10.5860/crl.77.3.286 crl15-719 The Librarian Leading the Machine 287 as hybrid learning for students who self-select face-to-face instruction as their desired delivery method. Understanding whether this is the case may aid librarians as they work to employ instructional design principles to create the most effective learning options, and it can provide libraries and librarians with direction for future instructional initiatives. Background At OU, library instruction is integrated into every section of Writing 160: Composition II (WRT 160), with the number of sections ranging from between thirty and forty in the fall semester to between eighty and one hundred in the winter semester, including three to four all-online sections each semester. Bolstered by findings from the earlier study that found no significant difference in student learning outcome attainment between instruction delivery methods, OU librarians have continued to offer this information literacy instruction in a hybrid format for on-campus sections: one hour of face-to-face instruction paired with an online module. To accommodate the increasing number of online classes, the librarians developed a series of virtual lessons that cover the content delivered in the face-to-face session, after which students complete the same online module as the on-campus students. During the past decade, OU has experienced a 32 percent increase in enrollment, currently enrolling just over twenty thousand students.2 During this same period of time, the number of full-time library faculty positions at the university has remained the same (even fluctuating downward at times due to retirements). To offset the increasing library instruction load that comes with rising enrollment, the OU libraries have hired a number of additional part-time faculty librarians during the past several years; their primary job responsibilities are to teach WRT 160 information literacy sessions and to staff the reference desk. In the midst of this growth, the library hired a new dean. With her support and encouragement, OU’s full-time librarians began to strengthen their liaison librarian roles to academic departments by collaborating more closely with academic faculty within their assigned departments, offering dedicated office hours to students in those departments, and even embedding in departmental courses when possible. This shift in focus left the full-time librarians with even less time for WRT 160 information literacy sessions, and soon the part-time librarians were tasked with teaching all the sessions; as a result, they quickly became overloaded with instruction. The time had come to reexamine the way in which the library offers research instruction to WRT 160 students. As at many institutions, OU has rapidly expanded online learning options, thanks in part to the growing robustness of the university’s learning management system (LMS), Moodle, which offers instructors many options for tailoring online learning environments. OU’s “brick and mortar” courses, too, are employing Moodle to enhance in-classroom learning, and today’s students have grown to expect an increased online learning presence in all their courses. Therefore, as the OU libraries’ instructional team considered future directions for WRT 160 library instruction, it became apparent that e-learning had changed so significantly since the 2005 comparative study that it was necessary to once again evaluate the effectiveness of different information literacy pedagogies before any decisions could be made. With this in mind, the authors of this study revisited the original research project and updated it to reflect the library’s current instructional model, contemporary trends in information literacy assessment, the use of instructional design principles in library instruction, and the prevalent use of online learning resources in higher education. For this revised study, the investigators sought to answer the following question: Does the delivery format of information literacy instruction in face-to-face course sections affect attainment of student learning outcomes? 288 College & Research Libraries May 2016 Literature Review Effectiveness of Online Learning Generally, online information literacy instruction has been well established as an effec- tive learning option. Numerous studies have shown that online learning, when well designed and properly implemented, can lead to student learning and achievement comparable to that of face-to-face instruction.3 Dewald’s study of effective online learn- ing objects for information literacy learning, undertaken in 1999, demonstrated that the Internet could offer learners unique opportunities for knowledge acquisition when coupled with sound instructional pedagogy.4 And in considering how to best evaluate the effectiveness of online learning objects, Blummer developed a tiered hierarchy for librarians to consider: 1. At the most basic level, academic libraries can conduct general assessments to measure if a tutorial works correctly and accomplishes the desired goals. 2. At an intermediate level, libraries can also assess whether online learning objects impact student learning and behavior. 3. At the most advanced level, a library may consider if its program-level online learning objects work across the institution to achieve broader goals and assist learning initiatives.5 Since Dewald’s analysis, studies that compare the effectiveness of online instruction to face-to-face instruction have proliferated across Blummer’s three levels of evalua- tion. Some of these studies compare several instructional types to determine efficacy. For instance, in their comparison of instruction delivered through online tutorials and face-to-face learning sessions, Beile and Boote found that online instruction represented a viable way librarians could reach students in online courses.6 Similarly, in a study that considered whether undergraduate students gained the requisite information literacy skills to carry them to graduation and success in their postacademic careers, Johnston found that those students who worked through an online information lit- eracy tutorial “gained an understanding of information literacy concepts” and that the flexible and customizable structure available through the use of the Internet helped students develop these skills.7 These studies serve to confirm what Zhang, Watson, and Banfield found in their analysis of ten studies comparing online and face-to-face learning: almost universally, online library instruction is seen as comparably effective as traditional face-to-face instruction.8 Still other research works from the perspective ascribed by Ifenthaler: among edu- cational theorists, “there is widespread agreement… that educational applications of modern information and communication technologies can be more effective when they are embedded in multimedia learning environments created to enable productive learning” [author’s emphasis].9 For instance, Smale and Regalado conducted a case study on the impact of online instructional modules embedded into a course manage- ment system; these researchers found that, following the use of such a web resource, students’ scores on an assessment improved and their information literacy proficiency increased.10 Similarly, Stiwinter examined the effectiveness of an online tutorial to impact student achievement and ease librarian labor by considering four specific data points: change in scores from pretest to posttest, usage statistics, citation analysis, and student feedback. This research found that the use of online library instructional tuto- rial increased students’ test scores and improved the quality of citations.11 Some research indicates that online learning may actually be preferable to face-to- face instruction. Silver and Nickel, who evaluated students’ library skills and confi- dence in using library resources after face-to-face and online instruction, found that students expressed a preference for online instruction.12 Anderson and May, in their examination of how students in an introductory communications course performed The Librarian Leading the Machine 289 following different methods of instruction, found that there were significant differences in students’ behavior following face-to-face, blended, and online library instruction; in fact, students who had engaged in online learning had significantly higher scores on a persuasive presentation research assignment that required students to evaluate and present sources in the correct citation format.13 Active Learning Online The aforementioned studies, as well as other research in the area, suggest that the most effective online information literacy learning objects include interactivity in the form of active learning.14 This may include instructional modules with practice ques- tions throughout to provide formative assessment or a more formal test or quiz at the conclusion of a learning object.15 Other online learning interactions require students to engage in higher levels of active learning, with resources such as the University of Arizona’s Guide on the Side or other dual-window designs, where students can work through processes to build information literacy competencies, skills, and knowledge.16 Incorporating active learning elements helps students make sense of concepts by try- ing new skills and, in particular, provides assessment opportunities to help students to reflect on their learning and the learning processes.17 Jacklin and Robinson, in their examination of student evaluations from 108 different biological science course sec- tions, found that learners prefer assignments presented online with active learning components and that students find it helpful to interact with classmates, instructors, and librarians to make sense of concepts and ideas.18 Benefits of Delivering Library Instruction Online Online learning offers learners and instructors alike unique options for knowledge acquisition. At the most basic level, the ability for students to control an online re- source—that is, to pause, review, or jump ahead as one needs—can enhance the learning experience.19 This agency or control over the learning process addresses what Ifenthaler calls “the individuality of learning,” (author’s emphasis) through which students can adapt course and instructional materials to meet their needs.20 The convenience fac- tor of online learning is also an important consideration: by making content available when the student can use it, or at points-of-need, online library instruction may engage students affectively as well as cognitively.21 Faculty, too, may find online information literacy instruction more convenient when it is provided online, because it can be done outside class time.22 Feedback features are also beneficial in online learning. Clark and Mayer cite the ability to provide immediate feedback as an essential best practice in e-learning in- structional design,23 and studies on the use of feedback in online library instruction have illustrated that in-lesson feedback encourages students to engage in and deepen their learning.24 Jacklin and Robinson assert that students want as much assistance as possible, whether through a face-to-face interaction with a librarian or through supporting content online; feedback as students work is one way to provide this scaf- folding and to mirror in-person interaction.25 Through the use of feedback in online instruction, then, academic libraries can extend their outreach to a greater number of students and faculty.26 Certainly, different formats of online instruction have different benefits. For instance, instructional videos or screencasts, which have become increasingly popular, can be simultaneously available on a library website, in a course management system, and on a third-party video-sharing resource such as YouTube. In addition to using these multiple avenues of access to increase the findability of learning objects, screencasts and streaming video can also help librarians to deliver information that is consistently 290 College & Research Libraries May 2016 formatted, chunked to reduce students’ cognitive load, and illustrative of skills and processes to be acquired.27 Embedding Library Services in Learning Management Systems There is considerable evidence that asserts that library instruction, whether face-to-face or online, is most effective when connected with an academic course,28 specific learning need,29 or situated at another academic point-of-need.30 For instance, Lindsay, Cum- mings, and Johnson examined the impact of placing online learning objects at various points in the research process where students fail or encounter difficulty. They found that, by locating resources in the context of these various library services, the use of the library’s online learning resources increased.31 So, as the use of course management systems has grown, “point-of-need” more often means embedded within the LMS. Fortunately, increasing library presence within learning and content management systems offers many opportunities for librarians. First, it diminishes the issues students encounter in finding content, which can be a significant barrier for some groups of learners.32 Such integration and embeddedness also represents an outreach tool. In fact, McLure and Munro, in examining the kinds of presences the academic library may have in an LMS, found that instructors and students alike welcome the library’s presence in this arena. Instructors expressed a willingness to collaborate on such work because they saw it as beneficial to student learning, and students felt that there was value in “visible, consistent, user-friendly access to library resources…that specifically assist[ed] them in achieving course outcomes.”33 McLure and Munro recommend that librarians use their institutional LMS as an avenue to further integrate library resources into the student learning experience and engage with the pedagogy of course design at the department and instructor levels.34 Jeffryes and colleagues found similar attitudes at their institution: instructors desired online, course-specific library resources that “would be simple for students to access and simple for instructors and staff to administer.”35 However, in their survey of instructors, Appelt and Pendell found that, while there was general buy-in for embedding library resources in an LMS, there were some caveats. Instructors’ questions about intended audience, how e-resources would fit into existing curriculum, and the level of faculty input in the design and development of these resources illustrate that, on each campus, librarians working to embed in the LMS need to collaborate with instructors to ensure that resources are used and recommended to students.36 Information Literacy Assessment: Development and Trends In academic librarianship more broadly, information literacy skill assessment is of considerable importance regardless of format. Measuring instructional effectiveness has particular significance in campus climates where data-driven decisions form the foundation of funding, staffing, and programming support. The Association of College and Research Libraries (ACRL) acknowledges the essential nature of assessment, noting that it can be used by libraries to serve several purposes: monitoring and evaluating program effectiveness, better identifying teaching and learning practices, and measur- ing learners’ mastery of skills and achievement.37 However, because most librarians are self-taught and do not have formal training in the area, assessment can be challenging.38 As a result, many librarians grapple with effectively and meaningfully assessing information literacy instruction, and they may not be measuring the full range of student learning. For example, Hsieh and Holden, in their research on determining the effectiveness of one-shot library instruction, found that many librarians assessed learning through pre- and posttests, which usually consist of multiple choice and true-or-false questions.39 These test structures are limiting and The Librarian Leading the Machine 291 restrict the kinds of learning that can be measured; moreover, pre- and posttest assess- ments are often limited to undergraduate-level concepts and do not measure library skills development or graduate-level student learning.40 Also, traditional assessment measures often fail to quantify the affective dimension of student learning, which is, as Schroeder and Cahoy define it, an individual’s “attitudes, emotions, interests, motivation, self-efficacy, and values.”41 This dimension represents an important con- sideration in any learning process but is particularly critical for information literacy and inquiry-based learning.42 Because of these challenges, meaningful and practical assessment resources are being developed professionwide. For example, Oakleaf developed an information literacy assessment cycle with seven iterative steps that can be enacted by academic librarians. These steps are as follows: review learning goals, identify learning outcomes, create learning activities, enact learning activities, gather data to check learning, interpret data, and enact decisions.43 Similarly, Miller notes that assessment should begin with designing instruction around learning outcomes and designing assessments to measure these outcomes.44 Farkas and Hinchliffe, each of whom implemented comprehensive assessment programs at their respective institutions, assert that academic libraries need to develop cultures of assessment to effectively measure student learning and the academic library’s impact, which involves “internalizing the value of assessment.”45 And, while many librarians consider a single pre- and posttest set of assessment data at a time, it may be more instructive to consider multiple years of assessment data to see changes and trends over time.46 In this way, librarians can better measure students’ performance based on learning objectives and outcomes, rather than just to determine if student performance has been affected by instruction. WRT 160 Revision In the decades in which the library has partnered with the writing and rhetoric de- partment to offer course-integrated library instruction to every section of WRT 160, the instructional materials for these sessions have been reconsidered, revised, and updated on a mostly ad hoc basis. In response to the myriad developments in assess- ment trends, instructional techniques, and what 21st century “information literacy” encompasses, however, the library’s Committee on Instruction (COI) undertook a complete, systematic overhaul of the WRT 160 library instruction program during the 2010–2011 academic year. The COI is a team of four library faculty members, including the library’s Coordinator of Instruction. To begin the process, members of COI met with faculty members from the Writing and Rhetoric Department to reexamine the goals for WRT 160 library instruction from the writing department’s standpoint. Taking these needs into account, COI then held a series of meetings with all the librarians to reevaluate and reestablish the library’s goals and objectives for the WRT 160 classes. In the course of these meetings, the library faculty identified two goals for WRT 160 library instruction aimed at building students’ research and critical thinking skills: 1) students will learn the importance of using an effective search strategy; and 2) students will gain an understanding of the value of resource evaluation. Under each of these goals, the faculty then defined com- prehensive yet realistic student learning outcomes around which the WRT 160 library content would be built (see appendix A for the full list of student learning outcomes). The COI applied the seven student learning outcomes to the development of all- new instructional content for WRT 160. Based on their literature review, the members of COI determined which content was best suited to be taught face to face and which could be delivered via online learning objects. See table 1 for a breakdown of content modules and instructional methods for brick-and-mortar sections of WRT 160. The COI 292 College & Research Libraries May 2016 also applied the curricular revisions to the lessons for the all-online sections, revising the content to align with the new student learning outcomes. A notable programmatic change was the application of a shared library instruction teaching outline for face-to-face sessions, which prompts the librarians to address the same content from session to session while still allowing for individualized approaches to the delivery of that content. This programmatic change also allowed for the integra- tion of consistent instructional design principles into students’ learning experiences, including chunking of content and opportunities for practice and feedback. The library faculty also agreed to reduce the amount of face-to-face time spent with each WRT 160 section, from approximately two hours to just one hour. As a result of this change, librarians developed new online instructional content to be delivered to students via Moodle. Of significant note is the fact that, where once the librarians required students to complete the online library content before their classes came to the library for the face-to-face session, this curricular overhaul allowed the librarians to complete in-class instruction and then direct students to the LMS at the end of the session. This switch has improved understanding of the process and has increased completion rates. Finally, accompanying these changes was a wholly new approach to the assessment of WRT 160 library instruction. Instead of the long-standing pre-/posttest approach, the overhaul team developed a new library final exam with questions that directly correlate exam performance to student learning outcome attainment (see appendix B for alignment chart). At the end of the first year of implementation, curricular revisions were considered for every SLO with low attainment (which the team defined as fewer than 80 percent correct answers), resulting in some instructional changes and/or the clarification of some exam questions. In the semesters since this instructional overhaul, OU librarians have consistently worked to review SLO data and modify both online and face-to-face instructional components as appropriate. Reinvestigating whether the delivery format of library instruction affects attainment of student learning outcomes represents a natural progression of this ongoing instructional design work. Methodology Study Design Students who were enrolled in on-campus sections of WRT 160 for the winter 2014 semester provided the subjects for this study. The library and writing faculty have part- nered now for more than a decade to provide library instruction within this required TABLE 1 Content Modules and Delivery Method Module Delivery Method Effective keyword brainstorming face-to-face Best practices for database searches face-to-face How to read, interpret, and evaluate database results face-to-face How to access books and articles that you find face-to-face Types of information sources and their uses online via LMS Evaluating resources online via LMS Finding specific items (“known item searching”) online via LMS The Librarian Leading the Machine 293 first-year course, and the winter semester is by far the busiest term, as noted above, in regard to the number of sections offered. Because students self-select into all-online sections—perhaps indicating a preference for that mode of instruction—the researchers determined that it was necessary to compare the effectiveness of the two instructional methods using only students who enrolled in the traditional, in-person course sections. After receiving approval from the university’s Institutional Review Board (IRB), the researchers selected nine faculty members at random from among those teaching at least two traditional sections of WRT 160 for the winter semester, all of whom agreed to have two of their sections participate in the study. The three study librarians were then randomly assigned to work with three each of the participating writing faculty members. One section from each participating writing faculty member was randomly selected to complete the all-online version of library instruction while the other section would complete the standard hybrid version; thus, each librarian taught three hybrid sections and oversaw three online sections. Every study section came to the library, either to use a library lab to complete the online instruction module or to receive face- to-face instruction from a librarian. This approach allowed the librarians to distribute information about the study and its purpose, answer any questions about the study, and provide copies of the consent form. Participation was voluntary, but students were required to take the same final exam in Moodle regardless of study participation. Checkboxes provided on the final allowed students to indicate if they were not yet eighteen years of age (and thus, not eligible) or if they chose to opt out of the study. Because students are allowed to take the final multiple times, students were informed beforehand that only the data from first attempts were collected for analysis. Results Before analysis commenced, data from students who had indicated they were underage or did not wish to have their data included in the study were removed. The remaining, anonymized data resulted in a near-equal sample size for each instructional method: 128 from the blended instruction control group and 129 from the online-only group. The authors used SPSS software to analyze the data. Table 2 provides the descriptive statistics and table 3 provides the results of an independent samples t-test. In regard to the research question “Does the delivery format of information literacy instruction in face-to-face course sections affect attainment of student learning out- comes?” the data suggest, in this case, that it does not. The means of the two groups are nearly identical, as shown in table 2. An independent t-test, which is a generally accepted statistical measure of difference, confirmed that the groups did not differ significantly (p = 0.88; p > 0.05 confirms that there is no significant difference). One concern with running a study such as this is that performance will necessarily be affected simply by participation in the study. To account for this potential, table 4 provides trimmed mean scores for the two study groups and the nonexperimental sections of WRT 160. The trimming was necessary to account for outliers, specifically in the nonexperi- mental data, as students in nonstudy sections were not informed that only their first attempt at the final would be collected; as a result, the nonexperimental group had TABLE 2 Descriptive Statistics Type N Mean Std. Deviation Std. Error Mean Experimental Group 129 20.12 3.798 .334 Control Group 128 20.20 3.775 .334 294 College & Research Libraries May 2016 many students who appeared to use their first attempt as a preview of the quiz con- tent, resulting in a zero score. The nontrimmed mean of 18.67 for the nonexperimental group, although not vastly different from the means of the two study groups, does reflect this practice. Therefore, the trimmed mean of 19.38 more accurately represents the average student’s score, which here is exactly one point lower than the experimental group’s score. Discussion While the mean final exam scores for the two experimental groups are very similar, a closer look at the data indicates that instructional content may need to be more fully developed in some concept areas. Because each final exam question correlates to one of the WRT 160 library instruction student learning outcomes (as detailed in appendix B), a review of the average scores on individual questions related to the study suggests that some of the content proved more effective online than in person, perhaps due to the student’s ability to take his or her time with the material. Figure 1 illustrates the scores for those final exam questions where instructional delivery method varied; some exam questions relate to content delivered online to all students, so those questions are not represented. The online group scored slightly better on each question, with the exception of Ques- tion 1, which relates to the use of quotation marks in keyword searching (see figure 2). This concept has been flagged in previous semesters as being troublesome for stu- dents regardless of instructional format, but the difference in scores here—78 percent in the in-person group compared to 59 percent for the online group—indicates that much improvement still needs to be done, especially in the online module. As with all TABLE 3 Independent Samples Test Levene’s Test for Equality of Variances t–test for Equality of Means 95% Confidence Interval of the Difference F Sig. t df Sig. (2-tailed) Mean Difference Std. Error Difference Lower Upper Equal Variances Assumed .835 .362 –.151 255 .880 –.07 .472 –1.002 .859 Equal Variances Not Assumed –.151 254.999 .880 –.07 .472 –1.002 .859 TABLE 4 Trimmed Means for Experimental and Non-Experimental Groups Group N 5% Trimmed Mean Experimental Group 129 20.38 Control Group 128 20.59 Non-Experimental Group 1160 19.38 The Librarian Leading the Machine 295 of the library’s online instructional content, the WRT 160 modules are reviewed each year and changes are made on an as-needed basis. Nonetheless, the data demonstrate that comparable outcomes are achieved overall when students in face-to-face WRT 160 sections complete library instruction all online or in a hybrid fashion. This finding helps library faculty make a case for moving more of the WRT 160 library instruction content online, or even having more in-person WRT 160 sections complete the course-integrated library instruction content using the all- online module. Such a transition could have several impacts. First, it would allow the library to continue its valuable partnership with the writing department while freeing FIGURE 1 Average Scores Per Final Exam Question Related to the Study 0% 20% 40% 60% 80% 100% 120% Q1 Q3 Q5 Q7 Q10 Q11 Q13 Blended Online FIGURE 2 Screenshot from Final Exam 296 College & Research Libraries May 2016 additional time for library faculty to devote to their ever-expanding liaison-area work. Second, it could provide students with greater agency over their learning by increasing their ability to pause, repeat, or review instructional content as needed. And third, it allows the librarians to focus their instructional design efforts into meaningful and reusable online learning resources. The impacts of the data, as described above, take on further relevance when one considers the new ACRL Framework for Information Literacy for Higher Education, adopted in early 2015.47 The Framework provides a broader, more conceptual perspective on information literacy in comparison to the Information Literacy Competency Standards for Higher Education,48 which provides specific learning outcomes for learners. This new perspective, though, responds to the impact of the Information Age on the way infor- mation is created, accessed, shared, and used. When one considers just how much the process of formal (and informal) research and inquiry has changed since the release of the Information Literacy Competency Standards for Higher Education, it is clear that an evolution has occurred, and instructional practices necessarily must also evolve, no matter the format or setting. The Framework, focused as it is on conceptual exploration rather than specific tasks, can be aligned with the creation of online instructional opportunities. Not only do these experiences allow for learning to be situated within the very environment in which students are accustomed (and often expect) to work with information, but, as noted in the discussion above, well-developed online instructional models enable content delivery that can be more thorough, allowing for more exploration and feedback, than what may be possible in a more traditional face-to-face instructional setting. In fact, the broadness of the frames almost requires such an online instructional presence. Consider the statement that the Framework opens the way for librarians, faculty, and other institutional partners to redesign instruction sessions, assignments, courses, and even curricula; to connect infor- mation literacy with student success initiatives; to collaborate on pedagogical research and involve students themselves in that research; and to create wider conversations about student learning, the scholarship of teaching and learning, and the assessment of learning on local campuses and beyond.49 Each one of these activities, which further librarianship while pushing the boundar- ies of teaching and learning (and its scholarship), increasingly take place in the online environment. If libraries can identify ways to effectively design meaningful instruction online, librarians can integrate information literacy frames across the many initiatives, collaborations, conversations, and assessment that occur therein. Consider, for example, the information literacy frame Searching as Strategic Exploration. This frame emphasizes that the process of seeking out information is “often nonlinear and iterative, requiring the evaluation of a range of information sources and the mental flexibility to pursue alternate avenues as new understanding develops.”50 In a one-shot instructional ses- sion, librarians can inform students that their first search will not be their last and discuss keyword brainstorming and strategies , as is currently done in the WRT 160 face-to-face session. However, putting these concepts into practice while also integrat- ing knowledge of appropriate resource selection and evaluation requires significant conversation and discourse. For WRT 160 students, this occurs in the online tutorials, which solve the issue of instructional time while situating learners in the environment where they are most likely to be searching for information, whether through discovery tools, databases, or search engines. This is just one of the six frames outlined in the Framework; each of the other five concepts also connects to, and is shaped by, online The Librarian Leading the Machine 297 learning and the Internet more broadly. Situating instruction in a context where learn- ers can understand this benefits librarians and students alike. Future Research Based on the results of this study, there are several potential directions for future re- search. First, the online library learning module could be offered more widely to WRT 160 faculty teaching face-to-face courses. While this randomized study provided gen- eralizable results, an expanded sample size might illuminate issues or trends not seen in this particular instance. Also, additional examination could consider the longer-term impacts of the library’s online and hybrid learning options for WRT 160. Within the course itself, the librarians could work with instructors to measure whether students who participated in the hybrid or online library learning experiences demonstrate different levels of successful library usage in their course assignments. Such research could be conducted through a citation analysis of students’ final papers, or by analyz- ing performance and grades on those assignments linked to library use. The effect of the library’s WRT 160 instructional format could also be measured beyond the course itself. This introductory writing course is a graduation require- ment, and all OU students must enroll in subsequent discipline-specific research- and writing-intensive courses. It may be useful to examine how students who enrolled in a face-to-face section of WRT 160 but received all-online library instruction retain basic library proficiencies and perform on future research tasks, especially in comparison to their peers who experienced hybrid library instruction. Such research could more concretely illustrate the long-term efficacy of these two instructional modalities. Conclusion Although some may wish to interpret these study data as an excuse to move all in- struction online and return to simpler times, when librarians did not spend a good portion of their lives on instruction, the authors would be remiss to encourage anyone to immediately do so. The outcomes of this study do not suggest that the computer is mightier than the librarian; rather, the success of the online group as compared to the hybrid group only indicates that carefully crafted online learning objects, which conform to the most recent scholarship of teaching and learning and are responsive to the needs of a specific audience, can be as effective in empowering students to achieve desired learning outcomes as in-classroom instruction. In either case, the librarian plays a pivotal role as the instructional designer. 298 College & Research Libraries May 2016 Appendix A: Student Learning Outcomes Core Concepts: 1. Keyword Searching 2. Broadening/Narrowing Topic 3. What to Use When 4. Evaluating/Analyzing Results Goals: 1. Students will learn the importance of using an effective search strategy. 2. Students will gain an understanding of the value of resource evaluation. Student Learning Outcomes: GOAL 1: Students will learn the importance of using an effective search strategy. At the end of the IL modules, students will be able to… a. apply effective keyword search skills to find results in an online database. b. identify at least two methods for finding resources related to a specific database record. c. demonstrate at least two strategies for broadening or narrowing a research topic based on search results. GOAL 2: Students will gain an understanding of the value of resource evaluation. At the end of the IL modules, students will be able to… a. identify the main uses of, and reasons to use, at least two types of information sources that are available via the library. b. evaluate any source—print or digital—for authority, currency, relevancy, ac- curacy, objectivity, and appropriateness to their level of research. c. read and interpret a database record to determine an item’s publication date, publication type, and general topic. d. evaluate database search results for relevancy to their research topics. The Librarian Leading the Machine 299 Appendix B: Final Exam/SLO Alignment Exam Question SLO Assessed 1 1a use of quotation marks 2 1b known item searching 3 2d determining relevance 4 2c determining source type 5 2c reading a record 6 2d determining relevance 7 1a keyword selection 8 2b evaluating a resource 9 2b evaluating a resource (website) 10 1b finding related items 11 1b finding related items 12 1c narrowing 13 1c broadening 14 2a identifying uses for sources Notes 1. Elizabeth W. Kraemer, Shawn V. Lombardo, and Frank J. Lepkowski, “The Librarian, the Machine, or a Little of Both: A Comparative Study of Three Information Literacy Pedagogies at Oakland University,” College & Research Libraries 68, no. 4 (2007): 330–42. 2. Oakland University, “About Oakland University,” available online at www.oakland.edu/ about [accessed 5 September 2014]. 3. Kelly A. Donaldson, “Library Research Success: Designing an Online Tutorial to Teach Information Literacy Skills to First-Year Students,” The Internet and Higher Education 2, no. 4 (2000): 237–51; Maria J. Grant and Alison J. Brettle, “Developing and Evaluating an Interactive Information Skills Tutorial,” Health Information and Libraries Journal 23, no. 2 (2006): 79–88; David L. Lechner, “Graduate Student Research Instruction: Testing an Interactive Web-Based Library Tutorial for a Health Sciences Database,” Research Strategies 20, no. 4 (2005): 469–81; Leo S. Lo and Jenny McCraw Dale, “Information Literacy ‘Learning’ Via Online Tutorials: A Collaboration between Subject Specialist and Instructional Design Librarian,” Journal of Library & Information Services in Distance Learning 3, no. 3/4 (2009): 148–58; Bryan Tronstad, Lori Phillips, Jenny Garcia, and Mary Ann Harlow, “Assessing the TIP Online Information Literacy Tutorial,” Reference Services Review 37, no. 1 (2009): 54–64. 4. Nancy H. Dewald, “Transporting Good Library Instruction Practices into the Web Environ- ment: An Analysis of Online Tutorials,” Journal of Academic Librarianship 25, no. 1 (1999): 26–31. 5. Barbara Blummer, “Assessing Patron Learning from an Online Library Tutorial,” Community & Junior College Libraries 14, no. 2 (2007): 121–38. 6. Penny M. Beile and David N. Boote, “Does the Medium Matter? A Comparison of a Web- Based Tutorial with Face-to-Face Library Instruction on Education Students’ Self-Efficacy Levels and Learning Outcomes,” Research Strategies 20, no. 1/2 (2004): 57–68. 7. Nicole Johnston, “Is an Online Learning Module an Effective Way to Develop Information Literacy Skills?” Australian Academic & Research Libraries 41, no. 3 (2010): 207–18. 8. Li Zhang, Erin M. Watson, and Laura Banfield, “The Efficacy of Computer-Assisted In- struction Versus Face-to-Face Instruction in Academic Libraries: A Systematic Review,” Journal of Academic Librarianship 33, no. 4 (2007): 478–84. 9. Dirk Ifenthaler, “Learning and Instruction in the Digital Age,” in Learning and Instruction in the Digital Age, eds. J. Michael Spector, Dirk Ifenthaler, Pedro Isaias Kinshuk, and Demetrios 300 College & Research Libraries May 2016 Sampson (New York: Springer, 2010), 5. 10. Maura A. Smale and Mariana Regalado, “Using Blackboard to Deliver Library Research Skills Assessment: A Case Study,” Communications in Information Literacy 3, no. 2 (2009): 142. 11. Katherine Stiwinter, “Using an Interactive Online Tutorial to Expand Library Instruction,” Internet Reference Services Quarterly 18, no. 1 (2013): 15–41. 12. Susan L. Silver and Lisa T. Nickel, “Are Online Tutorials Effective? A Comparison of Online and Classroom Library Instruction Methods,” Research Strategies 20, no. 4 (2005): 389–96. 13. Karen Anderson and Frances A. May, “Does the Method of Instruction Matter? An Experi- mental Examination of Information Literacy Instruction in the Online, Blended, and Face-to-Face Classrooms,” Journal of Academic Librarianship 36, no. 6 (2010): 495–500. 14. Lori E. Buchanan, DeAnne L. Luck, and Ted C. Jones, “Integrating Information Literacy into the Virtual University: A Course Model,” Library Trends 51, no. 2 (2002): 144–66; Paul L. Hrycaj, “Elements of Active Learning in the Online Tutorials of ARL Members,” Reference Services Review 33, no. 2 (2005): 210–18; Donaldson, “Library Research Success”; Nancy H. Dewald, Ann Scholz- Crane, Austin Booth, and Cynthia Levine, “Information Literacy at a Distance: Instructional Design Issues,” Journal of Academic Librarianship 26, no. 1 (2000): 33–44; Elizabeth Blakesley Lindsay, Lara Cummings, and Corey M. Johnson, “If You Build It, Will They Learn? Assessing Online Informa- tion Literacy Tutorials,” College & Research Libraries 67, no. 5 (2006): 429–45; Debra A. Riley-Huff, “Design Insights and Inspiration from the Tate: What Museum Web Sites Can Offer Us,” portal: Libraries and the Academy 9, no. 1 (2009): 79–98. 15. Hrycaj, “Elements of Active Learning,” 210–18; Donaldson, “Library Research Success.” 16. Leslie Sult, Yvonne Mery, Rebecca Blakiston, and Elizabeth Kline, “A New Approach to Online Database Instruction: Developing the Guide on the Side,” Reference Services Review 41, no. 1 (2013): 125–33. 17. Lih-Juan ChanLin and Chwen-Chwen Chang, “Web-Based Library Instruction for Promot- ing Information Skills,” Journal of Instructional Psychology 30, no. 4 (2003): 265–75. 18. Marcie Lynne Jacklin and Keely Robinson, “Evolution of Various Library Instruction Strategies: Using Student Feedback to Create and Enhance Online Active Learning Assignments,” Partnership: The Canadian Journal of Library and Information Practice and Research 8, no. 1 (2013): 12. 19. Melissa Bowles-Terry, Merinda Kaye Hensley, and Lisa Janicke Hinchliffe, “Best Practices for Online Video Tutorials in Academic Libraries,” Communications in Information Literacy 4, no. 1 (2010): 17–28; Joanne Oud, “Guidelines for Effective Online Instruction Using Multimedia Screencasts,” Reference Services Review 37, no. 2 (2009): 164–77. 20. Ifenthaler, “Learning and Instruction in the Digital Age,” 6. 21. Silver and Nickel, “Are Online Tutorials Effective?”; Stiwinter, “Using an Interactive Online Tutorial.” 22. Stiwinter, “Using an Interactive Online Tutorial.” 23. Ruth C. Clark and Richard E. Mayer, E-learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning (Hoboken, N.J.: John Wiley & Sons, 2011). 24. Dewald, Scholz-Crane, Booth, and Levine, “Information Literacy at a Distance”; Silver and Nickel, “Are Online Tutorials Effective?”; Gwendolyn J. Reece, “Critical Thinking and Cognitive Transfer: Implications for the Development of Online Information Literacy Tutorials,” Research Strategies 20, no. 4 (2005): 482–93; Oud, “Guidelines for Effective Online Instruction.” 25. Jacklin and Robinson, “Evolution of Various Library Instruction Strategies.” 26. Stiwinter, “Using an Interactive Online Tutorial.” 27. Bowles-Terry, Hensley, and Hinchliffe, “Best Practices for Online Video Tutorials”; Amanda Nichols Hess, “The MAGIC of Web Tutorials: How One Library (Re)Focused Its Delivery of Online Learning Objects on Users,” Journal of Library & Information Services in Distance Learning 7, no. 4 (2013): 331–48; Oud, “Guidelines for Effective Online Instruction.” 28. Lori E. Buchanan, DeAnne L. Luck, and Ted C. Jones, “Integrating Information Literacy into the Virtual University: A Course Model,” Library Trends 51, no. 2 (2002): 144–66; Paul J. Bracke and Ruth Dickstein, “Web Tutorials and Scalable Instruction: Testing the Waters,” Reference Services Review 30, no. 4 (2002): 330–37; Grant and Brettle, “Developing and Evaluating an Interactive Information Skills Tutorial”; Debra Kimok and Holly Heller-Ross, “Visual Tutorials for Point-of- Need Instruction in Online Courses,” Journal of Library Administration 48, no. 3/4 (2008): 527–43. 29. Dewald, “Transporting Good Library Instruction Practices”; Barbara A. Blummer and Olga Kritskaya, “Best Practices for Creating an Online Tutorial: A Literature Review,” Journal of Web Librarianship 3, no. 3 (2009): 199–216. 30. Laura Baker, “Library Instruction in the Rearview Mirror: A Reflective Look at the Evolu- tion of a First-Year Library Program using Evidence-Based Practice,” College & Undergraduate Libraries 13, no. 2 (2006): 1–20; Stacy A. Anderson and Emily R. Mitchell, “Life After TILT: Building an Interactive Information Literacy Tutorial,” Journal of Library & Information Services in Distance Learning 6, no. 3/4 (2012): 147–58; Grant and Brettle, “Developing and Evaluating an Interactive The Librarian Leading the Machine 301 Information Skills Tutorial”; Shiao-Feng Su and Jane Kuo, “Design and Development of Web- Based Information Literacy Tutorials,” Journal of Academic Librarianship 36, no. 4 (2010): 320–28. 31. Lindsay, Cummings, and Johnson, “If You Build it, Will They Learn?” 32. Helen Gold, “Engaging the Adult Learner: Creating Effective Library Instruction,” portal: Libraries and the Academy 5, no. 4 (2005): 467–81. 33. Merinda McLure and Karen Munro, “Research for Design: Exploring Student and Instructor Attitudes toward Accessing Library Resources and Services from Course Management Systems (CMS),” Communications in Information Literacy 4, no. 1 (2010): 54. 34. McLure and Munro, “Research for Design,” 53. 35. Jon Jeffryes, Kate Peterson, Stephanie Crowe, Elizabeth Fine, and Elena Carrillo, “Integra- tion Innovation: Launching the Library into a Course Management System,” Journal of Library Innovation 2, no. 1 (2011): 25. 36. Kristina M. Appelt and Kimberly Pendell, “Assess and Invest: Faculty Feedback on Library Tutorials,” College & Research Libraries 71, no. 3 (2010): 245–53. 37. Association of College and Research Libraries, “Assessment Issues,” available online at www.ala.org/acrl/issues/infolit/resources/assess/issues [accessed 18 July 2014]. 38. Karen Sobel and Cassidy R. Sugimoto, “Assessment of Learning during Library Instruction: Practices, Prevalence, and Preparation,” Journal of Academic Librarianship 38, no. 4 (2012): 191–204. 39. Ma Lei Hsieh and Hugh A. Holden, “The Effectiveness of a University’s Single-Session Information Literacy Instruction,” Reference Services Review 38, no. 3 (2010): 458–73. 40. Ibid. 41. Robert Schroeder and Ellyssa Stern Cahoy, “Valuing Information Literacy: Affective Learn- ing and the ACRL Standards,” portal: Libraries and the Academy 10, no. 2 (2010): 129. 42. Ellysa Stern Cahoy and Robert Schroeder, “Embedding Affective Learning Outcomes in Library Instruction,” Communications in Information Literacy 6, no. 1 (2012): 91–106. 43. Megan Oakleaf, “The Information Literacy Instruction Assessment Cycle,” Journal of Documentation 65, no. 4 (2009): 539–60. 44. Sara D. Miller, “Learning Outcomes, Instructional Design, and the 50-Minute Information Literacy Session,” Michigan Academician 39, no. 1 (2009): 31–45. 45. Meredith G. Farkas and Lisa J. Hinchliffe, “Library Faculty and Instructional Assessment: Creating a Culture of Assessment through the High Performance Programming Model of Orga- nizational Transformation,” Collaborative Librarianship 5, no. 3 (2013): 177–88. 46. Margaret Fain, “Assessing Information Literacy Skills Development in First Year Students: A Multi-Year Study,” Journal of Academic Librarianship 37, no. 2 (2011): 109–19. 47. Association of College and Research Libraries, Framework for Information Literacy for Higher Education (Feb. 2, 2015), available online at www.ala.org/acrl/standards/ilframework [accessed 31 March 2015]. 48. Association of College and Research Libraries, Information Literacy Competency Standards for Higher Education (2000), available online at www.acrl.org/ala/mgrps/divs/acrl/standards/ standards.pdf [accessed 31 March 2015]. 49. Ibid. 50. Association of College and Research Libraries, Framework for Information Literacy for Higher Education.