948 Parallel Lines: A Mixed Methods Impact Analysis of Co-Curricular Digital Literacy Online Modules on Student Results in First-Year Nursing Fiona Russell, Chris Rawson, Chrissy Freestone, Michael Currie, and Blair Kelly* An important component of the academic health librarian role is the establishment of interventions to support the development of student skills in evidence-based practice. It is important for practicing librarians to evaluate their activities to determine effectiveness and to inform de- cision making and best practice in librarianship. A series of six online modules was developed for a core first-year Bachelor of Nursing unit. Modules were subsequently investigated for impact on student learning of evidence-based practice by determining any correlation between stu- dents’ completion of the modules and their academic performance. The level of students’ interest and engagement was also examined through a survey about their experiences of and attitudes toward the modules. A positive correlation was found between module completion and attain- ment of higher grades by students. Feedback from the survey indicated that students engaged positively with the modules. Introduction The development of innovative and effective learning experiences for the development of student information literacy has been an important remit of academic librarians for decades.1 Determining whether the instruction provided by library practitio- ners is effective in helping to improve student information literacy is important in demonstrating relevance and impact of the library’s work and for the development of best practice. The rise of evidence-based practice in librarianship over the past decade or so2 has led to encouragement across the profession for practitioners to reflect upon their practice3 and to evaluate the effectiveness of information literacy instruction.4 It has been suggested that “real-world” evidence is ideal in determining that stu- dent learning has been achieved5 and practitioners are in a good position to provide Fiona Russell is Manager, Faculty of Health Library Services, email: f.russell@deakin.edu.au; Chris Rawson is Digital Literacy Learning Designer, email: chris.rawson@deakin.edu.au; Chrissy Freestone is Liaison Librarian, email: chrissy.freestone@deakin.edu.au; Michael Currie is Liaison Librarian (retired); Blair Kelly is Medical Librarian, email: blair.kelly@deakin.edu.au. All are (were) at Deakin University Library. ©2018 Fiona Russell, Chris Rawson, Chrissy Freestone, Michael Currie, and Blair Kelly, Attribution- NonCommercial (http://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. mailto:f.russell@deakin.edu.au mailto:chris.rawson@deakin.edu.au mailto:chrissy.freestone@deakin.edu.au mailto:blair.kelly@deakin.edu.au http://creativecommons.org/licenses/by-nc/4.0/ Parallel Lines 949 such evidence. There are many who advocate for the use of evidence for evaluation and decision making in practice,6 and the evaluation by practitioners of their activity contributes toward the growth of an evidence base. To demonstrate impact, evidence in some form is required. Literature Review Historically, instruction of information literacy in academic libraries has tended to take the form of face-to-face one-shot sessions.7 However, in the last decade, there have been moves toward information literacy support, which is both embedded within the course curriculum8 and delivered largely online.9 Kuh & Gonyea10 strongly advocated for higher education institutions to investigate the effectiveness of library activities on student learning. Many practitioners have since investigated and found demonstrable positive impact between their delivery of information and digital literacy instruction and student skill development.11 Munn and Small12 undertook a systematic review into information literacy and academic skills programs for first-year health science students. They found evidence that such programs improved students’ learning and behavior and that students responded favorably to them; they recommended that practitioners aim to embed support into curriculum. Online modules have greater potential for integration into the curriculum than face-to-face one-shot sessions, given there is less competition for time with other priorities within the curriculum. This integration allows for a more authentic student understanding of graduate learning outcomes, rather than information literacy being treated as supplementary content.13 It also allows for more sustained contact than is usually possible in one-shot classes. Where one-shot classes are still delivered, they are often blended with online pedagogies. Carroll adopted the flipped classroom pedagogy with a group of health science students by developing online modules for students to complete prior to attending the class, with the intention of increasing student retention of information literacy skills.14 Online delivery of information literacy instruction is not constrained by the limits of a 50-minute one-shot class, and thus it offers greater scope for student skill development. In his investigation into librarians’ involvement in critical library instruction, Tewell reported that many librarians find limited time is a significant barrier to instruction in a one-shot class; it is difficult to cover material in sufficient depth.15 The interviewees in this study seemed to largely have a shared history of face-to-face instruction rather than online delivery, possibly due in part to faculty expectations and the focus on critical information literacy. On the other hand, Matlin noted that online delivery allows for a greater reach than traditional face-to-face class, as well as an increased capacity for personalization than is possible within a lecture-style environment.16 Student numbers have increased across many institutions worldwide without a corresponding increase in staffing, and online delivery has been seen as both an efficient and sustainable method of instruction.17 The University of Arizona Libraries considered online delivery of instruction a positive solution to supporting large student numbers and accommodating the “anywhere/ anytime” paradigm shift in education.18 In Australia, James Cook University used online delivery for information literacy support for social work students, in part to accommodate distance education students, students already working in the industry, and limited time available in the curriculum.19 Despite these benefits, it has taken time for libraries to adapt to online learning— Mery notes that many libraries still depend upon passive material in their online instruction and, until recently, have not explored interactive options.20 Due to tradition- ally limited time with students, librarians have also tended to overfill their sessions, 950 College & Research Libraries November 2018 leading to information overload.21 With new instructional technologies, online modules can be chunked and made available throughout the teaching period, in contrast with the one-shot classes. There is growing evidence in the literature to support the effectiveness of online instruction. Although it has been considered difficult to ascertain the effectiveness of one-shot classes22 and research in the past has often relied on student self-perceptions of skill development,23 learning analytics available in modern software offers opportunities for objective analysis of the effectiveness of online instruction. Online delivery has been found to be as effective as the traditional form of instruction of information literacy skills, and overall, students do not demonstrate any clear preference for format.24 Beyond the discourse of information literacy instruction formats, there is still scope for more analysis of the effectiveness of the content and design of the modules themselves. In recent years, studies have been conducted to investigate the correlation between stu- dents’ use of library resources and their academic success. The University of Wollongong in Australia developed a Library Cube of student academic performance data with their library usage and found a positive correlation between use of electronic resources and strong academic performance.25 Soria’s 2017 study controlled for numerous possible confounding characteristics of students and found a positive association with library use and academic performance.26 In the information literacy realm, Walton and Hepworth used assignment data to analyze the effectiveness of online delivery of instruction when compared with one-shot face-to-face workshop, finding that students exposed to the online material showed better analysis and synthesis of information than the control groups.27 There has been progress in demonstrating library effectiveness in supporting stu- dent academic performance; however, more research is required into the relationship between student participation in online interactive learning modules for information literacy development and their academic results. Background Evidence-based Practice Evidence-based practice (EBP) healthcare is the use of the best available research evidence combined with clinical expertise and patient values to assist with clinical decision-making.28 It has become one of the key elements of modern healthcare and is considered an important skill in healthcare practice. Both the Australian College of Nursing29 and the Nursing and Midwifery Board of Australia30 have clearly cited the importance of EBP in the nursing profession. It has also been explicitly linked with information literacy in Adams’ 2014 comparison of the EBP framework and the Association of College and Research Libraries (ACRL) Information Literacy Standards.31 Despite the importance of EBP in the workplace, surveys of nursing staff on their information searching habits by Miglus32 and Ruzafa-Martinez33 have both highlighted a lack of awareness of and training in EBP by many nursing staff. Miglus34 noted a preference of nursing staff for simply searching for information on the web and cites key barriers such as a lack of time, research skills, understanding of electronic data- bases, and difficulty in understanding research articles. The Intervention Deakin University is a multicampus Australian university with more than 50,000 students and a strategic focus on online teaching and learning. The library is a dy- namic unit within the university and has a presence within each campus. The library contributes to the development of student capacity in digital literacy, one of the eight graduate learning outcomes of the university. Parallel Lines 951 Digital literacy is defined at the university as using technologies to find, use, and disseminate information.35 Often considered a multiliteracy, information literacy is one of these associated literacies.36 Sharpe and Beetham have explored digital literacy extensively and note that, as a component of digital literacy, information literacy involves several sophisticated capabilities, including the high-level cognitive skills of evaluation, interpretation, and application.37 Deakin Library, like other libraries, has for decades created and delivered vari- ous forms of interventions to support students in their development of information literacy—and now, in digital literacy. Early in 2016, an opportunity arose for Deakin University Library to develop a suite of online modules for a core first-year Bachelor of Nursing unit in collaboration with academic staff. The modules focused on EBP. The Bachelor of Nursing is one of the flagship courses for Deakin University’s School of Nursing & Midwifery, and online, or cloud-based, learning initiatives have been incorporated into the course. The unit, Understanding Research Evidence, introduces first-year nursing students to research and research design. Recognizing the need to engage students while supporting different learning styles and levels of understanding, the creators decided to implement an interactive learning ap- proach. Deakin University had recently licensed Smart Sparrow software. This software is a web-based tool for creating adaptive learning activities incorporating interactive elements and immediate feedback, and was subsequently used as the platform for the modules. The modules were designed to be constructively aligned to the weekly content of the unit, and, while not compulsory, students were strongly encouraged by the unit chair to use the modules as supplementary study tools. All modules were released on the same TABLE 1 Unit Content and Module Alignment Theme Evidence-based Practice Module Introduction to research and evidence- based nursing, ethical principles, and considerations in nursing research. Module 1: Introduction to EBP Overview of the process of EBP, definition of EBP and its usefulness for clinical decision-making. Asking clinical questions and searching for the best available evidence. Module 2: Resources for research Diagnostic quiz assessing students' ability to retrieve information, presented with remedial content where necessary. Module 3: Asking the clinical question Moving from background to foreground to develop a specific, answerable question and a search strategy to match. Qualitative research designs. Module 4: Qualitative research An introduction to different qualitative research approaches, as well as the critical appraisal of qualitative research. Quantitative research designs, data collection, and analysis. Module 5: Quantitative research Builds on the qualitative research module, provides examples of different study designs and what they’re useful for. Also provides practice in critically appraising a study. Translating research into practice. Module 6: Translating research into practice How research can impact on nursing practice, and time-saving techniques for finding and using evidence in practice. 952 College & Research Libraries November 2018 date at the beginning of the trimester and were publicized during relevant weeks of the trimester on the unit site. The unit site was hosted within the university’s learning manage- ment system (LMS). The modules were linked from within the unit site in a separate section labeled “Interactive Learning Modules,” with some modules being individually embed- ded into weekly topics: for example, “Week 1: Introduction to health research,” linked to under “Bonus Material” (Module 1) or “Week 3: Asking a clinical question and searching for the best evidence,” linked to under “Search Databases” (Modules 2 and 3) (Table 1). A variety of activities were used within the modules. Figure 1 is an example of a drop-down exercise and a fill-the-gap exercise used to reinforce the concept of the PICO framework, an aid for framing clinical questions in EBP. The running theme through the unit was one of vaccination. This example study in figure 2 was used to illustrate the design features of randomized controlled trials, and the study topic was contextualized for this theme. FIGURE 1 Screenshot from Module 3 Showing Drop-Down and Fill-the-Gap Activities to Support Student Learning of the PICO Framework FIGURE 2 Screenshot from Module 5 Showing Introductory Text Parallel Lines 953 Students were encouraged to review any incorrect responses to the activities through relevant feedback provided within the modules. The modules were aligned with the unit theme, drawing upon a narrative structure to contextualize the content where possible. FIGURE 3 Screenshot from Module 5 Showing Drop-Down Activities to Assist with Learning Different Types of Study Design FIGURE 4 Screenshot from Module 6 Showing Informational Text 954 College & Research Libraries November 2018 After three incorrect attempts at an exercise, the answer was provided to the students and they could continue to progress through the module. Each module presented information on the different elements of EBP and included a range of quiz questions at the end to test students’ understanding of the topic. Students were able to advance to the next section in the module either once they had selected the correct answer or when they had made a certain number of attempts. Constructive feedback was provided when incorrect answers were selected. Unit assessment incorporated an annotated bibliography and a final exam, some of which covered EBP content. Methods Research Questions The first research question was to investigate the effectiveness of the modules in student learning by determining any correlation between individual student module participation and final unit result. The second question was to determine students’ experiences with and attitudes toward the modules. Data Collection Researchers collected the following data: • Student enrollments in, and withdrawals from, the unit during the relevant time period • Students’ final unit results, including marks and grades FIGURE 5 Screenshot from Module 6 showing feedback following an activity Parallel Lines 955 • Student engagement with the modules measured by number of modules ac- cessed, time spent on the modules, progress toward module completion, and number of modules completed • Student attitudes toward the modules Student enrollment and withdrawal data were collected to determine the number of students potentially exposed to the modules. Students’ final unit results and data on module engagement were collected to determine any correlation between these two data sets. Student attitudes toward the modules were collected to better understand their perception of the value of the modules. Unit enrollment and withdrawal figures and students’ final unit results were col- lected via the university’s student management system. Student engagement data was obtained via the tracking features available in Smart Sparrow and the LMS, with module completion defined as reaching the final screen of the module. Student attitude data were obtained via a 5-item online questionnaire advertised via news items and links posted on the LMS. The questionnaire was made available to students at the conclusion of the teaching period but prior to the final assessment. The complete text for the questionnaire and the news items are available in the appendix. Student enrollment, withdrawal, and final unit results data were collected at the completion of the trimester after final unit results had been released to students. These data were then matched to the data on module engagement to connect individual’s final unit results with their tracked module engagement. Participants were then deidentified prior to the data analysis stage. This process fulfilled research ethics requirements for avoiding participant risk, as the data was used only after its release to participants, and the deidentification process preserved anonymity. Statistical Analysis A Pearson’s correlation was conducted to examine whether there was a relationship between the extent to which individuals engaged with the modules and their overall final unit results. It was hypothesized that greater participation with the online modules (that is to say, a greater number of modules completed) would be associated with higher final unit results. Qualitative survey responses were extracted from the questionnaire platform, Survey Monkey, and analyzed for recurring themes. Emergent coding was used to assign responses to categories and facilitate effective analysis. Results A total of 1,132 students were enrolled in the unit at some point during the trimes- ter, with a number subsequently withdrawing from the unit during the same period for unknown reasons. The final sample of 802 used for analysis included all students eligible for a final result, excluding those who were enrolled but did not complete the assessments or were otherwise ineligible for a final grade. Table 2 shows that the vast majority of these students (79.9%) completed at least one module. A third of the students (33.4%) attempted all modules. TABLE 2 Number and Percentage of Students Who Attempted or Completed Modules One or More Modules All Modules Attempted 756 (90.2%) 280 (33.4%) Completed 670 (79.9%) 94 (11.2%) 956 College & Research Libraries November 2018 The overwhelming majority of students interacted with the modules, spending an average of 16 minutes and 38 seconds per module. Students who spent more than an hour on the modules were removed from this temporal analysis, as it was assumed that students were unlikely to be active on the modules for this period of time. Correlation between Student Completion and Overall Marks There is a weak to moderate positive correlation between students’ final mark for the unit and the number of modules students completed. As the number of modules stu- dents completed increased, students’ final marks also tended to increase. There was a statistically significant positive correlation (r = .24, P < .001). Students who did not receive a final result were excluded from this analysis. Module Completions As shown in figure 6 above, the number of students completing each module declined over the course of the teaching period. Relationship between Number of Modules Completed and Grades The positive correlation between the number of modules completed and students’ final mark naturally extended to the students’ final grades. For this reason, further investigation was undertaken by looking at the dispersion of students’ grades, based on the number of modules undertaken. FIGURE 6 Number of Students Who Completed, or Did Not Complete, Each of the Modules Parallel Lines 957 Figure 7 shows the grades dispersion of people who did no modules, more than half of the modules, and all of them. It also shows the dispersion of student grades for the unit. Table 3 shows that students typically achieved higher grades when they completed more modules. Figure 8 shows a clear upward trend to higher marks for those who completed more modules. TABLE 3 Percentage of Students Receiving a Grade, Organized by the Overall Number of Modules Undertaken Grade 0 Modules 4+ Modules 6 Modules Total Class Fail (N) Mark= below 50% 17% 8% 4% 11% Pass (P) Mark = 50%–59% 41% 23% 26% 29% Credit (C) Mark = 60%–69% 30% 34% 32% 33% Distinction (D) Mark= 70%–79% 10% 25% 26% 20% High Distinction (HD) Mark = 80% and over 1% 11% 13% 6% FIGURE 7 Distribution of Students’ Grades, Based on Number of Modules Undertaken as Well as the Total Distribution of the Cohort (Regardless of Number of Modules Undertaken) 958 College & Research Libraries November 2018 Survey Results The following figures show the outcomes from the survey of student experiences and attitudes toward the online modules. A total of 30 students responded to the survey, 3.6 percent of the students who completed the unit. While this is a small proportion, it is still sufficient to provide a good indication of the experiences and attitudes of the greater student body. Q1: Please indicate the extent to which you agree with the following statements: • The modules improved my understanding and skills in evidence-based practice • I enjoyed the modules • I found the modules easy to use • The modules were useful for preparing me for assessment tasks across the unit Figures 9–12 illustrate that the majority of respondents either agreed or strongly agreed that the modules were useful, engaging, and easy to use and that they improved their skills in EBP. FIGURE 8 Mean Final Unit Marks by Number of Modules Completed. Error Bars Display ± Standard Error FIGURE 9 Students’ Reported Agreement with the Statement: The Modules Improved My Understanding And Skills In Evidence-Based Practice Parallel Lines 959 More than four out of five (83.3%) of respondents agreed or strongly agreed that the modules improved their understanding and skills in EBP. More than three-quarters (80.0%) of respondents agreed or strongly agreed that the modules were enjoyable. FIGURE 10 Students’ Reported Agreement with the Statement: I Enjoyed the Modules FIGURE 11 Students’ Reported Agreement with the Statement: I Found the Modules Easy to Use 960 College & Research Libraries November 2018 Eight out of ten (80.0%) respondents agreed or strongly agreed that the modules were easy to use. That same (or a similar) 80.0 percent of respondents agreed or strongly agreed that the modules assisted in their preparation for unit assessment tasks across the unit. One respondent indicated strong disagreement with all statements; however, sub- sequent comments from this respondent were, on the contrary, positive. Q2: What was most challenging about the modules? Statements from the responses provided by students were categorized into the follow- ing major themes: Technical, Comprehension, and Content. FIGURE 12 Students’ Reported Agreement with the Statement: The Modules Were Useful for Preparing Me for Assessment Tasks Across the Unit FIGURE 13 Students’ Reporting of the Challenging Aspects of the Modules Parallel Lines 961 Figure 13 reveals that the challenges for respondents in their use of the modules were primarily technical in nature. Technical problems identified related to the mod- ules’ display, navigation, and function. Some issues were rectified after release of the modules. Student comments included: if you were unable to answer the question, you couldn’t move onto the next one until complete. Other students commented that they found some of the questions and content confusing, while others cited not having prior exposure to some of the content as a challenge: Some of the questions were a bit confusing at first Not having had a background to the topic Q3: What was the most useful thing about the modules? Statements from the responses provided by students were categorized into the follow- ing themes: Engagement, Consolidation of Learning, and Content. Figure 14 shows consensus among the respondents that the modules assisted them in consolidating their understanding of EBP and provided a useful resource for revi- sion of the unit content. Reinterated [sic] what I was reading and enabled me to apply this information in a practi- cal manner that [sic] just reading and rote learning. The interactive nature, being able to apply the theory to a practice question and know immediately if you were on the right track or not. The modules helped to break down the information we were given and put them into easily identifiable uses FIGURE 14 Students’ Reporting of Useful Aspects of the Modules 962 College & Research Libraries November 2018 Students indicated that they found the modules engaging… It was all super relevant and engaging, which made them good. …and that the content was presented in a simple and concise manner: Definitions or key terms or concepts were made in a way that was simple and easy to understand Simplified content but still extensive. Feedback also indicated that students consolidated their learning: Consolidation of the reading and relaying the key points of the unit. Q.4: How could we improve the modules? Statements from the respondents were categorized into these major themes: Technical, More challenge/content/interactivity, Design/aesthetics, Extend to other units, and Miscellaneous. Figure 15 indicates that respondents mostly saw potential for improvements in technical areas, as well as a desire for increases in challenge, content, or interactivity. Many comments related to technical issues with the use of the modules, for example: They were sometimes a bit buggy, maybe some more technical improvements are needed… Other suggestions focused on the desire for more challenge and interactivity. … More interactive questions, where you get to put the theory into practive [sic] I would have liked more!![sic] as revision More quizzes. FIGURE 15 Students’ Suggestions for Improvements to the Modules Parallel Lines 963 Other suggestions included using the modules more extensively across the course: Have these in EVERY subject : ) These modules should be used in all units as another learning tool to enhance consolidation Other comments related to aesthetics and design of the modules, with a few stu- dents suggesting the addition of more color and animation. Miscellaneous comments concerned the desire for additional support. Q.5: Any other comments of your experience with the modules? Only half of the students responded to this question, and respondents were mostly positive about their experience with the modules. All good, please continue to use them. I have enjoyed learning through interactive modules Eight comments were positive and six were neutral. The only negative comment related to a student’s preference for lectures, a comment that was more pertinent to the unit’s delivery mode than to the experience of the modules themselves. Discussion While it is encouraging to see a positive correlation between the student completion of modules and their results, it cannot definitively be said that students who completed the modules would have had higher final results. A possible correlation between engagement with the modules and student charac- teristics such as motivation, constructive study habits, engagement with other course material or other support services (such as academic skills support) could not be controlled. Nakayama notes that personality, technological skills, and thinking skills are among factors to influence learning.38 These characteristics may have contributed to higher grades and a possible confounding of the results. However, while the correlation is weak to moderate, this reflects the positive find- ings of Walton’s analysis of the effectiveness of online instruction on student learning39 and the study by MacKenzie.40 Furthermore, correlation is considered an important indicator when evaluating the impact of a library activity or interaction on student learning or performance.41 Students who completed more modules than others typically achieved higher grades (see table 3). This indicates an upward trend between completing the modules and stronger grades. Students who did not complete any modules typically received a lower grade than the average, and those who completed half or more of the modules typically received a higher grade than the average. The average time spent on the modules indicates more than a cursory view of the content. The creators designed the modules so that no more than ten minutes would be required for adequate engagement. Sixteen minutes is considered a period for a student to receive a good level of exposure to the material. This average indicates that students spent time reading and interacting with the activities rather than just clicking through the content to reach the end of the module. It can be inferred from this that the students found the modules sufficiently engaging to spend more than the minimum time. Despite the very high proportion of students who either attempted or completed at least one module, there was a decline in student interaction with modules as the 964 College & Research Libraries November 2018 trimester progressed. While this study is not designed to investigate why that may be the case, it is worth noting that the modules were voluntary and that others report a decrease in student access to online resources over the course of a teaching period,42 which would suggest that this is not unusual. The authors believe it is most likely ex- plained by factors other than a decrease in the quality of the modules over time, such as increasing commitments with assessment as the trimester progressed. The experiences survey was useful to determine student reactions to the modules and also to identify technological issues for correction and module iterative development. Of those who responded, the students overall demonstrated very positive experi- ences with the online modules. I enjoyed the interactive modules because you were putting the steps into play as you worked through them. Great job, excellent resource thanks The vast majority of respondents indicated that the modules helped them with course work and were useful for revision. This is encouraging, as the modules were designed to closely align with the unit curriculum and enhance students’ understand- ing of EBP. In addition, students commented that the modules should be used more extensively across the course. I would have like [sic] one for every week… They are good to use as a revision of the weekly info. I wish these were in all units. The majority of students also indicated that they found the modules easy to use, which may reflect the time spent by the library team designing the modules and se- lecting software that facilitated an interactive and meaningful learning environment. They were a quick and easy way to cement learning. Student comments soundly endorsed the usefulness of the modules. They are a really helpful tool to work through the content. Students enjoyed being challenged and engaged, commenting that the modules could have been designed to extend them even further. Potentially consider having an optional component that has more examples/tasks for a given module or one that looks at a more complex example, thanks :) This is certainly an aspect for consideration in subsequent versions of these modules, and it is something that the adaptive learning pathways allow for. Several students noted that the interactive elements in particular were a novel and positive feature of the modules that kept them engaged, helped to consolidate learning, and provided them with an opportunity to review material and test their knowledge. I liked that it was [sic] new and different way to learn…. Parallel Lines 965 This supports findings in the literature where practitioners have reported increased student motivation, persistence, and greater retention of learning when students feel sufficiently engaged in an interactive online learning environment.43 The students also appreciated the clarity with which the more complex concepts were presented. Information was concise, provide [sic] accurate information, guidance and appropriate examples regarding each module. The majority of respondents were positive about the modules as an effective learn- ing resource for understanding the concepts of EBP. This reflects others’ findings that students respond positively to online interventions.44 Where comments indicated room for improvement, including more challenging content and more interactivity, as well as some aesthetic suggestions, these informed subsequent iterations of the modules. Despite some technical issues, students, for the most part, engaged very well with the modules. This suggests that there was sufficient relevant content and good design in the modules to keep the students interested and engaged. The fact that recom- mendations were made by students to extend the usage of the modules within the unit itself as well as to the rest of the course was very encouraging. By keeping each module chunked and relevant to curriculum, it would appear that students learned a lot across the trimester without feeling overwhelmed. Students appeared to respond to the modules’ alignment to unit curriculum. This learning tool helped to consolidate the learning from other sources eg tutes, cloud etc. I wish these were in all units. Consolidation of the reading and relaying the key points of the unit (in response to the question: What was most the most useful thing about the modules?). It is worth noting that many students implied they were unaware that the library had prepared the modules. This indicates that the modules were seamlessly integrated with curriculum and within the online learning environment. Integration with the curriculum is also important to avoid the pressure on workloads of librarians in the one-shot classes and to assist with contextualizing the concepts.45 While the response rate for the survey was small (3.6%) and validity is a concern, this can be explained by the timing of the survey, following from the end of the teach- ing period. This was a requirement of the unit teaching team so that the survey did not interrupt unit evaluation surveys. Despite promotion of the survey, students may also have been less inclined to complete a survey during their exam study period. Data saturation may not have been reached with 30 responses, and these results need to be used with caution, despite useful emerging themes and feedback for modular improve- ments. Future investigation could involve focus groups and interviews of students. Limitations to the Study This study can only show a correlation between participation with the modules and final results; there are other external factors that may have influenced the results, and, as mentioned above, student enthusiasm and motivation could not be controlled. There may also have been a drop-off in engagement with the modules throughout the trimester as student workloads and priorities changed. As the modules were non- compulsory support for study, it is likely that students prioritized their assessment 966 College & Research Libraries November 2018 tasks and may have had less time to work on the modules later in the trimester. There was also a technical issue with completion of Module 4 for a short period of time, which would also have impacted the completion rates until the problem was fixed. Of note, the student who responded that they strongly disagreed with the statements in question 1 may have inadvertently selected this option. The comments associated with this respondent in the subsequent questions were, by contrast, extremely posi- tive. Another respondent who expressed broader concerns may have misinterpreted the purpose of the survey in determining the students’ thoughts and experiences of the online modules. Future Directions Of future consideration are the time and resources involved in the development of modules. There is then the issue of ensuring that the modules are well integrated into the disciplinary knowledge of the unit. Therefore, where units of study are similar enough in focus to reuse the module with limited modifications, the modules will be modified as necessary and reused. Since the implementation of this initial series of modules for nursing in 2016, there has been significant interest from academic teaching staff in other courses. The whole suite of modules was adapted to an occupational therapy scenario for a core third-year bachelor of occupational therapy unit in trimester 1 of 2017, and the unit chair for a new core first-year psychology unit has also requested the whole series for adaptation to the curriculum (Trimester 2, 2017). These modules will also be used again for the nursing unit in 2017. Future research could focus on the effectiveness of these modules upon implemen- tation in other health degrees. Conclusion There was a weak but positive correlation between student completion of the mod- ules and higher student academic results. While this is not conclusive evidence of the impact of the modules, correlation is considered a good indicator of performance of library activity. Very high student engagement was recorded for the completion of at least one module, and while there were far fewer who completed the whole suite of six modules, this indicates that students are receptive to instruction in this format, despite potential competing demands on their time as the teaching period progressed. Student feedback indicated generally very positive experiences with the modules, displaying both enjoyment and a view that the modules were engaging and more useful for learning than passive content. Combined with the findings from the correlation analysis, this indicates that the modules have been effective in assisting students to develop their skills in EBP and, more broadly, in information literacy, and there is value in continuing to offer these modules within the curriculum of this unit to support student learning of EBP. These findings will assist librarians in their discussions with academics in support- ing their students, allowing them to speak the language of their clients and providing evidence of the impact of these modules when presenting these as an option for sup- porting student learning. More broadly, this study contributes to the evidence-base for librarianship, adding to the growing scholarship on support for EBP learning and for the use of online mod- ules as library intervention. This will assist others in the industry with their decision making when designing interventions to support student learning. Parallel Lines 967 Acknowledgements The development of the EBP modules was partially funded through the Australian Commonwealth Higher Education Partnerships and Participation Program (HEPPP). Many thanks to Michael Horn and Michael Krull for their work on the technical and design elements of the modules, to Christine Oughtred from the library’s Learning and Teaching team for advice, training, ideas, and assistance with quality control, and to additional members of the Health Liaison team for their work and enthusiasm in plan- ning, developing, testing, and editing the modules: Dorothy Rooney and Louisa Sher. Thanks also to Dr. Marzieh Asgari and Dr. Nicola Ivory for their contribution to the statistical analysis of the student marks and grades in relation to module engagement. We’d also like to acknowledge the support of the School of Nursing and Midwifery— in particular, Associate Professor John Rolley and Ms. Monica Schoch. Ethics This research was conducted with the approval of the Deakin University Faculty of Arts and Education Human Ethics Advisory Group, approval #HAE-16-165, under the terms of Deakin University’s Human Research Ethics Committee. 968 College & Research Libraries November 2018 APPENDIX. Unit Site News Posts about Questionnaire and Text for Questionnaire Unit Site News Post about Questionnaire (1) [unit code] Evidence-based Practice Modules---Invitation to Contribute to a Research Project Hi all, You are invited to participate in a short, 5- to 10-minute survey regarding your expe- riences and thoughts on the series of evidence-based practice modules available on [LMS] in [unit code] this trimester. The survey is completely voluntary and anonymous and will contribute toward a library research project that will be used in the planning of future support for digital literacy. Results from this survey will help to determine the effectiveness of online interactive modules in developing an understanding of and skill development in elements of evidence-based practice. We are always looking to improve our support to teaching and research staff, and your input is valuable in this process. You can read the plain language statement about the project at the beginning of the survey. The survey will be open from 29 September to 14 October and is available at: https:// www.surveymonkey.com/r/JR327MT. This project has been approved by the ethics committee, project #HAE-16-165. Please let Fiona Russell (XX@deakin.edu.au) know if you have any questions; and if you have any complaints about the project, please contact XX@deakin.edu.au. Many thanks, The Faculty of Health Library Liaison Team Unit Site News Post about Questionnaire (2) Just a reminder that if you would like to make a contribution to the research project on the evidence-based practice modules, the survey module is open until this Friday and should only take 5--10 minutes. The survey is available at: https://www.surveymonkey. com/r/JR327MT (project #HAE-16-165). https://www.surveymonkey.com/r/JR327MT https://www.surveymonkey.com/r/JR327MT mailto:XX@deakin.edu.au mailto:XX@deakin.edu.au https://www.surveymonkey.com/r/JR327MT https://www.surveymonkey.com/r/JR327MT Parallel Lines 969 Text for Questionnaire 1. Please indicate the extent to which you agree with the following statements: □ The modules improved my understanding and skills in evidence-based practice □ I enjoyed the modules □ I found the modules easy to use □ The modules were useful for preparing me for assessment tasks across the unit [Students were presented with a five-level Likert-type scale to indicate their level of agreement with the above statements. The five levels were: □ Strongly disagree □ Disagree □ Neither agree nor disagree □ Agree □ Strongly agree 2. What was most challenging about the modules? 3. What was the most useful thing about the modules? 4. How could we improve the modules? 5. Do you have any other comments about your experience using the modules? Notes 1. Mark Hepworth, “Approaches to Providing Information Literacy Training in Higher Education: Challenges for Librarians,” New Review of Academic Librarianship 6, no. 1 (2000): 21–34; Barbara MacAdam, “Information Literacy: Models for the Curriculum,” C&RL News (Nov. 1990): 948–51. 2. Evidence-Based Practice for Information Professionals: A Handbook, eds. Andrew Booth and Anne Brice (London: Facet, 2004); Joanne Gard Marshall, “Linking Research to Practice: The Rise of Evidence-Based Health Sciences Librarianship,” Journal of the Medical Library Association: JMLA 102, no. 1 (2014): 14–21. 3. Susie Andretta, “Evaluating Information Literacy Educators’ Practices before and after the Course Facilitating Information Literacy Education: From Tutor to Learner Centred,” Health Information and Libraries Journal 28 (2011): 171–78. 4. M. Fain, “Assessing Information Literacy Skills Development in First Year Students: A Multi-Year Study,” Journal of Academic Librarianship 37, no. 2 (2011): 109–19. 5. Dianna E. Sachs, Kathleen A. Langan, Carrie C. Leatherman, and Jennifer L. Walters, “As- sessing the Effectivness of Online Information Literacy Tutorials for Millennial Undergraduates,” College & Undergraduate Libraries 20, no. 3/4 (2013): 327–51. 6. N.E. Adams, “A Comparison of Evidence-Based Practice and the ACRL Information Lit- eracy Standards: Implications for Information Literacy Practice,” College and Research Libraries 75, no. 2 (2014): 232–48; Andretta, “Evaluating Information Literacy Educators’ Practices,” 171–78; Andrew Booth, “Australian Supermodel? A Practical Example of Evidence-Based Library and Information Practice (EBLIP),” Health Information and Libraries Journal 23, no. 1 (2006): 69–72. 7. Anne Horn, Alexia Maddox, Pauline Hagel, Michael Currie, and Sue Owen, “Embedded Library Services: Beyond Chance Encounters for Students from Low SES Backgrounds,” Australian Academic & Research Libraries 44, no. 4 (2013): 235–50; S.D. Squibb and S. Mikkelsen, “Assessing the Value of Course-Embedded Information Literacy on Student Learning and Achievement,” College and Research Libraries 77, no. 2 (Mar. 2016): 164–83. 8. Simon Cavenett and Chris Rawson, Combining Information Literacy Development and Team- work through Authentic Assessment (Queensland, Australia: Griffith School of Engineering, Griffith University, 2013); John Cripps Clark, Chris Rawson, Linda Hobbs, Christine Oughtred, Kathleen Hayes, Leissa Kelly, and Julie Higgins, “A Pedagogy for Epistemic Agency in the Judgment of Ac- curacy and Reliability,” Qwerty-Open and Interdisciplinary Journal of Technology, Culture and Education 11, no. 2 (2016): 27–47; Q. Zhang, S. Xie, and M. Goodman, “Integrating Library Instruction into the Course Management System for a First-Year Engineering Class: An Evidence-Based Study Measuring the Effectiveness of Blended Learning on Students’ Information Literacy Levels,” College and Research Libraries 76, no. 7 (Nov. 2015): 934–58. 970 College & Research Libraries November 2018 9. Yvonne Mery, Erica DeFrain, Elizabeth Kline, and Leslie Sult, “Evaluating the Effectiveness of Tools for Online Database Instruction,” Communications in Information Literacy 8, no. 1 (2014): 70–81. 10. G.D. Kuh and R.M. Gonyea, “The Role of the Academic Library in Promoting Student Engagement in Learning,” College and Research Libraries 64, no. 4 (2003): 256–82. 11. Squibb and Mikkelsen, “Assessing the Value of Course-Embedded Information Literacy,” 164–83; Geoff Walton and Mark Hepworth, “Using Assignment Data to Analyse a Blended In- formation Literacy Intervention: A Quantitative Approach,” Journal of Librarianship & Information Science 45, no. 1 (2013): 53–63; Lydia MacKenzie and Kim Ballard, “Can Using Individual Online Interactive Activities Enhance Exam Results?” Journal of Online Learning & Teaching 11, no. 2 (2015): 262–66. 12. Joanne Munn and Jann Small, “What Is the Best Way to Develop Information Literacy and Academic Skills of First Year Health Science Students? A Systematic Review,” Evidence Based Library and Information Practice 12, no. 3 (2017): 56–94. 13. S.C. Barrie, “A Conceptual Framework for the Teaching and Learning of Generic Graduate Attributes,” Studies in Higher Education 32, no. 4 (Aug. 2007): 439–58. 14. Alexander J. Carroll, Nedelina Tchangalova, and Eileen G. Harrington, “Flipping One-Shot Library Instruction: Using Canvas and Pecha Kucha for Peer Teaching,” Journal of the Medical Library Association: JMLA 104, no. 2 (2016): 125–30. 15. Eamon C. Tewell, “The Practice and Promise of Critical Information Literacy: Academic Librarians’ Involvement in Critical Library Instruction,” College & Research Libraries 79, no. 1 (2018): 10–34. 16. T. Matlin and T. Lantzy, “Maintaining Quality While Expanding Our Reach: Using Online Information Literacy Tutorials in the Sciences and Health Sciences,” Evidence Based Library and Information Practice 12, no. 3 (2017): 95–113. 17. Y. Mery, J. Newby, and K. Peng, “Why One-Shot Information Literacy Sessions Are Not the Future of Instruction: A Case for Online Credit Courses,” College and Research Libraries 73, no. 4 (2012): 366–77. 18. Ibid. 19. Nicole Johnston, “Is an Online Learning Module an Effective Way to Develop Information Literacy Skills?” Australian Academic & Research Libraries 41, no. 3 (2010): 207–18. 20. Mery, DeFrain, Kline, and Sult, “Evaluating the Effectiveness of Tools for Online Database Instruction,” 70–81. 21. Dani Brecher Cook and Kevin Michael Klipfel, “How Do Our Students Learn? An Outline of a Cognitive Psychological Model for Information Literacy Instruction,” Reference & User Services Quarterly 55, no. 1 (2015): 34–41. 22. Tewell, “The Practice and Promise of Critical Information Literacy,” 10–34. 23. Karen Anderson and Frances A. May, “Does the Method of Instruction Matter? An Experi- mental Examination of Information Literacy Instruction in the Online, Blended, and Face-to-Face Classrooms,” Journal of Academic Librarianship 36, no. 6 (2010): 495–500; Alicia Virtue, Ellen Dean, and Molly Matheson, “Assessing Online Learning Objects: Student Evaluation of a Guide on the Side Interactive Learning Tutorial Designed by SRJC Libraries,” Interdisciplinary Journal of E-Learning and Learning Objects 10 (2014): 93–105. 24. A.L. Weightman, D.J.J. Farnell, D. Morris, H. Strange, and G. Hallam, “A Systematic Re- view of Information Literacy Programs in Higher Education: Effects of Face-to-Face, Online, and Blended Formats on Student Skills and Views,” Evidence Based Library and Information Practice 12, no. 3 (2017): 20–55. 25. B. Cox and M. Jantti, “Discovering the Impact of Library Use and Student Performance,” Educause Review (Jul. 18, 2012): 1–9. 26. Krista M. Soria, Jan Fransen, and Shane Nackerud, “Beyond Books: The Extended Aca- demic Benefits of Library Use for First-Year College Students,” College & Research Libraries 78, no. 1 (2017). 27. Walton and Hepworth, “Using Assignment Data,” 53–63. 28. Tammy Hoffmann, Sally Bennett, and Chris Del Mar, Evidence-Based Practice across the Health Professions, 2nd ed. (Chatswood, N.S.W., Australia: Elsevier Churchill Livingstone, 2013). 29. Australian College of Nursing, “Nursing Research Position Statement,” Australian Col- lege of Nursing (2013), available online at www.acn.edu.au/sites/default/files/representation/ position_statements/acn_nursing_research_position_statement_2013.pdf [accessed 10 November 2017]. 30. Nursing and Midwifery Board of Australia, “Fact Sheet: Enrolled Nurse Standards for Practice,” Australian Health Practitioner Regulation Agency (2017), available online at www.nurs- ingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/Enrolled-nurse-standards-for- practice.aspx [accessed 10 November 2017]. http://www.acn.edu.au/sites/default/files/representation/position_statements/acn_nursing_research_position_statement_2013.pdf http://www.acn.edu.au/sites/default/files/representation/position_statements/acn_nursing_research_position_statement_2013.pdf http://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/Enrolled-nurse-standards-for-practice.aspx http://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/Enrolled-nurse-standards-for-practice.aspx http://www.nursingmidwiferyboard.gov.au/Codes-Guidelines-Statements/FAQ/Enrolled-nurse-standards-for-practice.aspx Parallel Lines 971 31. Adams, “A Comparison of Evidence-Based Practice and the ACRL Information Literacy Standards,” 232–48. 32. Jennifer D. Miglus and Robin D. Froman, “Evaluation of an Evidence-Based Practice Tutorial for Nurses: A Useful Tool and Some Lessons Learned,” Journal of Continuing Education in Nursing 47, no. 6 (June 2016): 266–71. 33. María Ruzafa-Martínez, Lidón López-Iborra, David Armero Barranco, and Antonio Jesús Ramos-Morcillo, “Effectiveness of an Evidence-Based Practice (EBP) Course on the EBP Compe- tence of Undergraduate Nursing Students: A Quasi-Experimental Study,” Nurse Education Today 38 (Mar. 2016): 82–87. 34. Miglus and Froman, “Evaluation of an Evidence-Based Practice Tutorial for Nurses,” 266–71. 35. “Deakin Graduate Learning Outcomes,” Deakin University (2017), available online at www. deakin.edu.au/about-deakin/teaching-and-learning/deakin-graduate-learning-outcomes [accessed 20 October 2017]. 36. “Developing Digital Literacies,” Jisc (2014), available online at www.jisc.ac.uk/guides/ developing-digital-literacies [accessed 20 October 2017]. 37. Rhona Sharpe and Helen Beetham, “Understanding Students’ Uses of Technology for Learning: Towards Creative Appropriation,” in Rethinking Learning for a Digital Age: How Learners Are Shaping Their Own Experiences, eds. Rhona Sharpe, Helen Beetham, and Sara De Freitas (New York: Routledge, 2010). 38. M. Nakayama, K. Mutsuura, and H. Yamamoto, “Impact of Learner’s Characteristics and Learning Behaviour on Learning Performance During a Fully Online Course,” Electronic Journal Of E-Learning 12, no. 4 (2014): 394–408. 39. Walton and Hepworth, “Using Assignment Data,” 53–63. 40. MacKenzie and Ballard, “Can Using Individual Online Interactive Activities Enhance Exam Results?” 262–66. 41. M. Oakleaf, “The Library’s Contribution to Student Learning: Inspirations and Aspirations,” College and Research Libraries 76, no. 3 (2015): 353–58. 42. M.H. Benson Soong, Hock Chuan Chan, Boon Chai Chua, and Koah Fong Loh, “Critical Success Factors for On-Line Course Resources,” Computers & Education 36, no. 2 (2001): 101–20. 43. Rebecca A. Croxton, “The Role of Interactivity in Student Satisfaction and Persistence in Online Learning,” Journal of Online Learning & Teaching 10, no. 2 (June 2014): 314–24; A. Espasa and J. Meneses, “Analysing Feedback Processes in an Online Teaching and Learning Environment: An Exploratory Study,” Higher Education 59, no. 3 (2010): 277–92; Lin Zhao and Raida Abuizam, “The Impact of Adaptive Learning in an Introductory Management Distance Education Course: An Empirical Comparison,” Academy of Educational Leadership Journal, no. 3 (2015): 359–65. 44. Matlin and Lantzy, “Maintaining Quality While Expanding Our Reach,” 95–113. 45. Van Houlson, “Getting Results from One-Shot Instruction: A Workshop for First-Year Students,” College and Undergraduate Libraries 14, no. 1 (2007): 89–108. http://www.deakin.edu.au/about-deakin/teaching-and-learning/deakin http://www.deakin.edu.au/about-deakin/teaching-and-learning/deakin http://www.jisc.ac.uk/guides/developing-digital-literacies http://www.jisc.ac.uk/guides/developing-digital-literacies