527 Research in the Real World: Improving Adult Learners Web Search and Evaluation Skills through Motivational Design and Problem- Based Learning Linds West Roberts* How can we better engage adult learners during information literacy ses- sions? How do we increase students’ perception of the relevance and importance of information literacy skills for academic work and life in the real world? To explore these questions, the ARCS Model of Motivational Design and Problem-Based Learning were used to develop activities for a library instruction workshop. Community college students completed a pretest and posttest assessment to measure change in skill level, per- ceived confidence, and perceived relevance of the research workshop. Results show learners’ skill levels, perceived confidence, and perceived relevance increased significantly. Based on the results, suggestions are made for incorporating Motivational Design and Problem-Based Learn- ing into information literacy sessions to increase student engagement. Introduction Workers in today’s information economy are faced with a wide variety of resources to choose from when making decisions about products, services, and processes for their jobs. During the last couple of decades, government leaders and educators have become increasingly concerned about the need to prepare a workforce for these high- skill jobs.1 The ability to find information and statistics on the open web from reliable sources has become an essential skill across many fields.2 Librarians are increasingly looking for ways to build these transferable “lifelong learning” skills, which go well beyond teaching college students how to use subscription databases to find scholarly articles for academic assignments.3 Few studies examine effective research skills among nontraditional college students. The National Center for Education Statistics (NCES) defines nontraditional students as * Linds West Roberts is Education Librarian at the University of Colorado, Boulder; e-mail: lindsay.m.roberts@ colorado.edu. Many thanks to the Library and Information Science faculty at the University of Denver for their feedback and encouragement. In particular: Dr. Krystyna Matusiak, Dr. Mary Stansbury, Christopher Brown, and Martin Garnar. Thanks also to librarians at Arapahoe Community College: Lisa Grabowski, Ann Priestman, Casey Lansinger, and Andrea Reveley. Special thanks to Michael and Theresa Roberts for their support. ©2017 Linds West Roberts, Attribution-NonCommercial (http:// creativecommons.org/licenses/by-nc/4.0/) CC BY-NC. doi:10.5860/crl.78.4.527 mailto:lindsay.m.roberts@colorado.edu mailto:lindsay.m.roberts@colorado.edu http://creativecommons.org/licenses/by http://10.5860/crl 528 College & Research Libraries May 2017 those who have delayed college enrollment for a year or more after graduating from high school, received a GED or certificate instead of a high school diploma, attend classes part-time, work full-time, are financially independent, care for dependents, or are single parents.4 Many of these students are adults with rich life experiences and priorities that may differ substantially from those of traditional undergraduates. Further, NCES points out that 70 percent of all undergraduates meet one or more of these criteria. NCES predicts that nontraditional students aged 25–34 are expected to increase 20 percent between 2011 and 2022, and students aged 35 and over are ex- pected to increase 23 percent between 2011 and 2022.5 Consequently, understanding the needs of nontraditional students is of growing concern to librarians at all higher education institutions.6 The purpose of this study is to examine improvement in nontraditional students’ in- formation literacy abilities using library instruction that employs Keller’s ARCS Model of Motivational Design (Attention, Relevance, Confidence, and Satisfaction), combined with Problem-Based Learning (PBL).7 Community college students, many of whom are nontraditional adult learners, participated in an hour-long research workshop us- ing Problem-Based Learning scenarios and Internet search strategies. Results of the pretests and posttests show significant improvement in learners’ abilities to evaluate types of sources and to create search strings. Students perceived the instruction to be relevant to their interests, and confidence in their search skills improved significantly. This study addresses the gap in literature examining engagement of community col- lege students in information literacy sessions. It also explores how motivational design models such as ARCS may be combined with teaching strategies like PBL. While the experiment described focuses on community college students, these teaching strate- gies, aimed at nontraditional or adult learners, are applicable to academic librarians teaching information literacy at a much wider range of institutions. Literature Review New Conceptions of Information Literacy The concept of information literacy (IL) has evolved over the past forty years in light of new technologies and changing trends in education. Discussion has shifted from the idea of literacy being equated with printed text and the ability to read, to the concept of multiple literacies needed to function in increasingly complex daily life—visual lit- eracy, numerical literacy, digital literacy, media literacy, and others.8 Best practices for improving IL have been moving away from the prescriptive how-to of bibliographic instruction and toward concept-based teaching of information literacy to prepare students for the information economy.9 The current shift from information literacy to the concept of “metaliteracy” encom- passes both metacognitive and self-reflexive elements. Metaliteracy, as described by Jacobson and Mackey, acknowledges that technology, particularly social media, has helped blur the lines among previously distinct literacies.10 Metaliteracy combines ele- ments of previous understandings of information literacy and positions “information literacy” as an umbrella term. Jacobson and Mackey advocate for a unifying under- standing of literacies as overlapping and collaborative, focusing on a set of practices common to multiple literacies.11 Further, they position metaliteracy at the intersection of four domains: behavioral, cognitive, affective, and metacognitive. Jacobson and Mackey collaborated with a small group to develop four goals for metaliterate learners: “1) Evaluate content critically, including dynamic, online content that changes and evolves, such as article preprints, blogs, and wikis; 2) Understand personal privacy, information ethics, and intellectual property issues in changing technology environments; 3) Share information and collaborate in a variety of participatory environments; 4) Demonstrate Research in the Real World 529 ability to connect learning and research strategies with lifelong learning processes and personal, academic, and professional goals.”12 An important component of metaliteracy is metacognition. Defined by John H. Flavell in the 1970s, metacognition is understood to be an individual’s awareness of his or her cognitive abilities.13 The fourth goal for metaliterate learners, listed above, particularly emphasizes the connection between metaliteracy and metacognition. Other scholars have noted that metacognition is crucial for students to apply knowledge from one set of problems to a new context or domain, such as from academic classes to personal or work contexts.14 The notion of threshold concepts, or big picture ideas that are foundational to the field, allows room for building learners’ metaliterate and metacognitive abilities. First synthesized by Meyer and Land, threshold concepts are suggested to be “transfor- mative,” “irreversible,” “integrative,” “bounded,” and “troublesome.”15 Townsend, Brunetti, and Hofer were among the first to suggest threshold concepts’ applicability to IL, particularly as a theoretical underpinning of information literacy in contrast to bibliographic instruction. Threshold concepts also allow IL practices to be situated within the discipline of information science rather than conceived as a more generic set of practices, such as study skills.16 They assert that threshold concepts will bring the focus to “the transformative content that is unique to our field.”17 Threshold concepts may allow learners to experience a deeper conceptual understanding of information practices, as well as an understanding of how these concepts may be applied across contexts, thus contributing to lifelong learning and adaptability. With lifelong learning in mind, we are seeing a growing need for information literacy instruction to address not only academic contexts but also competencies that apply to real-world contexts, such as work or personal environments. While studies on the transfer of knowledge from one domain to another have been present in the literature for decades,18 transfer has been applied to IL skills more recently, as evidenced by Lloyd’s exploration of workplace literacy and Kuglitsch’s work on threshold concepts and transfer as they relate to discipline-specific knowledge.19 Additionally, Lloyd’s conceptualization of “information resilience” as the ability to respond to rapidly evolv- ing information needs in the work environment, based in large part on changing and increasingly multimodal technologies, has direct implications for best practices when teaching information literacy to college students.20 Relatedly, Head’s Project Information Literacy has shown gaps between recent col- lege graduates’ perceptions of their information skills compared with their employ- ers’ expectations, uncovering four areas where employers would like to see greater competencies: “1) engaging team members during the research process; 2) retrieving information using a variety of formats; 3) finding patterns and making connections; 4) taking a deep dive into the ‘information reservoir’.”21 As all of these information literacy trends indicate, the real-world information environment necessitates a skill- set that is highly collaborative, fluent in many changing technologies, and includes metacognitive awareness to be effective. Evolution: Standards to Framework Following the changing vocabulary and conceptions of information literacy, IL stan- dards are also evolving. The 1989 American Library Association (ALA) Presidential Committee on Information Literacy called for librarians and other educators to ensure that students are prepared for the workplace with adequate information literacy skills and prepared to be lifelong learners.22 This report represented a huge step in recogniz- ing the roles that public, school, and academic libraries play in fostering information literacy skills in the United States, distinct from the traditional K–12 public school 530 College & Research Libraries May 2017 system. In 2000, the Association of College and Research Libraries adopted a set of Information Literacy Competency Standards for Higher Education including the ability to “recognize when information is needed” and the ability to “locate, evaluate, and use effectively the needed information.”23 While the Information Literacy Competency Standards helped solidify core com- ponents and provided a useful structure for teaching information literacy at the time of their adoption, Hofer, Brunetti, and Townsend pointed out several limitations for current practice—after fifteen years, the Standards needed to be revised in light of changing technologies. The Standards were often vague or struggled to prioritize the most important material.24 The new Framework for Information Literacy for Higher Educa- tion focuses information literacy teaching around six threshold concepts: “1) authority is constructed and contextual; 2) information creation as a process; 3) information has value; 4) research as inquiry; 5) scholarship as conversation; 6) searching as strategic exploration.”25 Implications of the six frames’ impact on information literacy teaching and assessment are presented in the Discussion and Conclusions sections. ARCS Model of Motivational Design Keller’s Attention, Relevance, Confidence, Satisfaction (ARCS) Model of Motivational Design provides a framework to improve IL sessions for nontraditional students by laying out the components necessary for adult learners to be engaged in new mate- rial.26 His model uses four broad categories to group motivational and teaching strate- gies.27 The “Attention” category focuses on capturing and keeping students’ interest. “Relevance” ensures that students recognize how the content and skills will be useful to them. “Confidence” refers to students’ perceptions of their abilities related to the material, including overconfidence, when students lack awareness of the skills they need or believe they already have them. “Satisfaction” addresses students’ continuing desire to learn as they progress through the material and may come from internal or external factors ranging from grades to personal enjoyment of the material.28 Keller advocates that instructional strategies from all four categories are needed during a learning experience, though these strategies may be best used at various stages during instruction.29 For instance, an instructor may wish to capture students’ attention at the beginning of a lesson, then show the relevance of the material to their interests and goals. Later on, students may practice with hands-on activities that chal- lenge them and provide opportunities to build confidence. Finally, an instructor may wish to wrap up the lesson by reconnecting the material to students’ goals and enjoy- ment of learning, contributing to students’ satisfaction with the learning experience. As Keller describes, the Audience Analysis step of Motivational Design can help an instructor anticipate and identify when each of these categories will be needed across the lesson or longer-term course.30 The ARCS Model has been applied to information literacy contexts. In their 2004 book, Jacobson and Xu relate the ARCS Model to several areas of information literacy teaching, focusing particularly on credit-bearing information literacy courses.31 Jacob- son and Hu connect “authentic assessment”—which involves cognitive and perfor- mance assessment using rubrics, case studies, and portfolio evaluations—with ARCS motivational strategies.32 Gross and Latham combined Keller’s framework with that of Bruce’s concept of informed learning when designing an educational intervention for below-information-proficient community college students, finding that building relevance and autonomy were particularly crucial to increase student engagement.33 Prior research has found community college librarians use a variety of strategies to maintain students’ attention, yet librarians employ strategies that pertain to the relevance of the material only 24 percent of the time and strategies that pertain to Research in the Real World 531 students’ confidence and satisfaction only 20 percent and 4 percent, respectively.34 The ARCS Model suggests that, to increase students’ motivation to learn, librarians need to capture students’ attention and demonstrate the relevance of the material by communicating how research strategies are useful to students. Instructors can also plan more hands-on activities that give students a chance to build confidence and satisfaction as they expand their research and evaluation abilities.35 Problem-Based Learning Problem-Based Learning addresses the relevance and confidence portions of Keller’s ARCS Model. It offers the hands-on experience that students need as well as an authentic context for learning that simulates real-world decision making. The term “Problem- Based Learning” (PBL) came into being in the early 1970s via medical education and case study teaching used in business schools,36 both highly applied contexts. Hmelo- Silver reviewed empirical studies on PBL’s effectiveness for learning; overall, PBL has been shown to be effective for students in the medical field and for undergraduates.37 The primary benefits to students include the ability to adapt concepts to new scenarios, to self-direct their learning, and to collaborate, since PBL frequently involves work in small groups. Barrows defines PBL using four components: 1) the problems given to students must be complex enough that there is no one correct answer; 2) students must be allowed to self-direct in their exploration of causes and solutions and their gathering of informa- tion; 3) the instructor serves as a “guide at the side of the learner instead of a sage on a stage at the front of the class”; 4) finally, the scenarios given to students should be as realistic as possible, such that they could represent types of situations students will face as decision makers in the future.38 The overall effectiveness of PBL is still being debated. In 2009, Strobel and van Barneveld examined eight meta-analyses that compared PBL with traditional teaching to understand the comparative effectiveness.39 Overall, the authors found that the type of assessment—whether a knowledge test or a skills-based instrument—impacted the relative effectiveness of PBL compared with traditional education. In general, traditional education seemed to work best for knowledge tests involving remembering key terms and concepts, whereas PBL students often, though not always, performed better in practice or skill-based environments. The authors conclude, “evidence suggests that PBL works in particular contexts, especially for workplace learning with a focus on skills and long-term retention.”40 As it relates to information literacy, Downing reported successful use of Problem- Based Learning lessons with both first-year undergraduate and graduate students at the University of Michigan.41 Students were asked to work in small groups to find articles related to their class topic and demonstrate their search techniques and the database or tool they used to the larger class. Post-class surveys and interviews with instructors indicated that the hands-on classes were more effective for students than lessons that relied on lecture and demonstrations of library tools. Connecting PBL and ARCS Model Barrow’s definition of PBL pairs well with the six strategies Keller defines to support the Relevance portion of his ARCS Model: 1) building on the learner’s prior experi- ences; 2) showing present worth or how the material will be useful immediately; 3) demonstrating future usefulness of the material; 4) matching the teaching style to the needs or motives of the student, such as a preference for group work or for individual work, or setting tasks that meet a student’s current skill level; 5) modeling new be- haviors or skills; and 6) giving learners choices that encompass how they learn best 532 College & Research Libraries May 2017 or prefer to work.42 The shared goals of Keller’s Model of Motivational Design and Problem-Based Learning support their use in tandem for information literacy teach- ing. This study is the first to explore both ARCS and PBL as combined strategies for information literacy instruction. Engaging Nontraditional College Students Community college populations include students looking to transfer to four-year colleges, returning students, and students seeking two-year, vocational, or technical degrees. The relative size of the community college student population is quite large: in 2013, two-year colleges enrolled just under 40 percent of all undergraduates.43 Many students at community colleges fall under the category of “nontraditional students”: fall 2012 enrollment numbers from NCES show students aged 25 or older made up 29 percent of the full-time and 48 percent of the part-time students enrolled at public 2-year institutions.44 For academic librarians, the varied backgrounds, busy schedules, and ability levels of their nontraditional student populations present special challenges for engaging students in building IL skills. Heery noted that nontraditional students “often show an unusual degree of motivation and commitment” in spite of barriers such as time, family or job responsibilities, inadequate academic preparation for college work, and language or cultural barriers.45 The question for information literacy instructors be- comes, “How can we most effectively teach information literacy skills to nontraditional students during one-shot or noncredit library sessions, when the students are pressed for time and vary widely in their backgrounds and interests?” Knowles, Holton, and Swanson recognized the needs of adult learners in their work on andragogy.46 Adult learners want information they can immediately use and ap- ply to help solve problems or answer questions.47 Focus groups with 64 community college students who had “below-proficient” information literacy skills indicated that these students preferred hands-on practice opportunities and real-life examples in information literacy classes.48 A major challenge, Latham and Gross point out, is that students with below-proficient information literacy competencies frequently overestimate their research abilities.49 This can be difficult for librarians attempting to engage and motivate students. Interviews with 24 nontraditional adults enrolled at an online-only program in Spain found that, while adults might be information-proficient in their work or home contexts, their knowledge and skills did not necessarily transfer to the academic en- vironment beyond a very basic level.50 The authors emphasized that, in the academic environment, students were less proactive than at work or home, perhaps indicating they were not sufficiently motivated to engage their metacognitive capacities or were not given enough autonomy over their learning.51 Hypotheses The following hypotheses are addressed in this study: H1: Community college students’ abilities to evaluate information for reliability will increase as a result of incorporating problem-based activities into an ARCS-designed library instruction workshop. H2: Community college students’ confidence in their research abilities will increase as a result of incorporating problem-based activities into an ARCS-designed library instruction workshop. H3: Community college students will find the PBL/ARCS-designed workshop rel- evant to their information needs after completing the hands-on scenarios. Research in the Real World 533 Methodology Participants The experimental study was conducted at a community college in the Denver met- ropolitan area. At the time of the study, the campuswide average student age was 26.6 years old, including an increasing number of concurrently enrolled high school students. Thus, the target group of students represented a wide age range, from high school students to traditional-aged college students to adults in their 60s and beyond. Some students had already obtained bachelor’s degrees and were returning to refresh their skills, make a career change, or pursue additional certificate programs. Others were first-time college students or were returning to college after an absence of several years. Thus, IL skill levels were quite varied within the sample group. Participants were recruited through convenience sampling. Students were notified of the workshop and the option to participate in the study through flyers posted at the three college campuses and through e-mail announcements to faculty members. Demographics The analysis included forty- one students’ responses. Re- sponses were not included from three underage stu- dents. Twenty-three male and eighteen female students participated. The average age among participants was 25.4 years old. Thirty-one participants self-identified as full-time students and nine as part-time. Only two partici- pants identified themselves as nonnative English speakers. Twenty-eight students indi- cated they had received some form of library instruction in the past, with 19 students having attended a library research session at the institution and 13 having received instruction in high school or another college. Thirteen students had never had formal library instruction. Pre- and Posttest Instruments Pretest and posttest instruments were developed after reviewing the literature for examples of skills and knowledge questions used on information literacy quizzes and tests. Searches for instruments were conducted in Library and Information Science Abstracts (LISA), Library, Information Science and Technology Abstracts with Full Text (LISTA), and Library Literature and Information Science Full Text and Retrospective. Selection criteria limited the search to articles on information literacy teaching, which included assessment instruments as an appendix or posted online. Few freely available instruments that pertained to Internet research skills were found during the literature review. Thus, the majority of the pretest and posttest questions were developed specifically for this study. The author developed multiple-choice, fill- in-the-blank, and short-answer knowledge questions to ask students to demonstrate some of the skills addressed in the workshop (see appendix B). The posttest asked similar skills-related questions but with different examples compared to the pretest. For example, if students were asked to check the library catalog for the call number TABLE 1 Participant Demographics n = 41 Mean (std.) Percentage First Time College Student 19.5 Continuing Student 56.1 Returning Student 19.5 Full-time 77.5 Age 26.0 (10.3) Gender (Male) 56.1 Prior Library Instruction (None) 31.7 534 College & Research Libraries May 2017 of a book, the book title differed between the pretest and posttest. For the workshop objectives, confidence questions were posed using a 5-point Likert scale. Relevance questions on the posttest were adapted from Keller’s Course Interest Survey and In- structional Materials Motivation Survey and also used a 5-point Likert scale.52 Open- ended qualitative questions were designed to spark students’ metacognitive reflections on their skill levels and their perceptions of the relevance of the material covered. Research Design and Workshop Procedures The experimental design is a one-group pretest/posttest design. Students completed a paper-based pretest before the workshop and a paper-based posttest following the workshop to measure their confidence levels, abilities in evaluating information, and perceived relevance of workshop material. The standalone workshop was offered in person to three groups of students during fall 2013. The content of the workshop focused on search techniques such as: • Using Google’s top-level domain search (site: ) • Finding historic news articles through Google News • Evaluating websites for credibility using the C.R.A.P. criteria (currency, reli- ability, authority, purpose/point of view) • Finding open access books and journals • Finding free, full-text scholarly articles via the government’s PubMed and ERIC databases The session content was selected to focus on freely available resources that students might use after graduating or in their work or personal lives. The presentation por- tion of the workshop was brief, with the intention of increasing students’ awareness of many possible channels for free, reliable information rather than mastering all of these techniques or types of content. The practical nature of the content and the differ- ent scenarios used allowed students to explore underlying concepts of IL, such as the production and dissemination of information in web environments and the economics of information—why some information is freely available while other content is behind a paywall. This structure followed a conceptual approach that is closer to threshold concepts than the ACRL Standards.53 After a brief demonstration of the skills listed above, participants completed three short PBL scenarios for hands-on practice, each lasting about five minutes. The PBL scenarios were chosen to encourage transfer of skills from academic to work or personal contexts, as Perkins and Salomon suggest.54 Students were asked to find reliable information quickly to make a decision or solve a problem and to discuss the steps they took to find and evaluate the information. The three PBL scenarios were designed to be open-ended, with no one correct path to complete the scenario, as Barrows suggests.55 Students could choose to discuss their strategies with those sitting nearby or to work individually if they preferred, following Keller’s recommendation that adult learners be able to choose their work preferences and Mackey and Jacobson’s conception of metaliterate learners being able to engage in social learning.56 The session design aimed to increase students’ metacognitive aware- ness of their research abilities through brief discussions after each PBL scenario and posttest questions that asked students to reflect on their own learning.57 Findings H1 Skills Results The pretest and posttest asked students to complete several short information-need scenarios to determine their search and evaluation abilities before and after the workshop. Table 2 shows the results of the scaled performance measures in panel A. The pretest mean for the skills questions was subtracted from the posttest mean to Research in the Real World 535 TA B L E 2 P re -t es t a nd P os t- Te st I m pr ov em en t i n A bi lit y to A ss es s th e R el ev an ce a nd R el ia bi lit y of I nf or m at io n Se ar ch R es ul ts P an el A : I m pr ov em en t o n Sc al ed P er fo rm an ce M ea su re s P re te st M in P re te st M ea n (S td .) P re te st M ax P os tt es t M in P os tt es t M ea n (S td .) P os tt es t M ax M ea n D if fe re nc e (S td .) P er ce nt C ha ng e T P -V al ue –0 .5 6 0. 36 (0 .3 3) 0. 83 0. 35 0. 73 (0 .1 8) 1. 0 0. 37 (0 .3 2) 10 2. 7 7. 5 <. 00 01 P an el B : P er fo rm an ce o n In di vi du al M ea su re s Ta sk M ax im um S co re P os si bl e St an da rd iz ed M in im um St an da rd iz ed M ax im um M ea n (s td .) G lo ba l W ar m in g 8 –0 .1 3 1. 0 0. 55 (0 .2 9) D ia be te s 3 –1 .0 1. 0 0. 16 (0 .5 3) N ew C ar 3 –0 .6 7 1. 0 0. 60 (0 .4 3) W eb R el ia bl e 6 0. 33 1. 0 0. 71 (0 .2 2) U R L R el ia bl e 5 0. 40 1. 0 0. 88 (0 .1 8) L eg en d: P ar tic ip an ts ’ p re te st a bi lit y to e va lu at e th e re le va nc e an d re lia bi lit y of In te rn et in fo rm at io n so ur ce s w as m ea su re d us in g tw o qu es tio ns : r es ea rc h fo r a n ac ad em ic pa pe r o n gl ob al w ar m in g an d re se ar ch fo r a fa m ily m em be r w ith d ia be te s. P os tte st a bi lit y to e va lu at e th e re le va nc e of In te rn et in fo rm at io n so ur ce s w as m ea su re d us in g th re e qu es tio ns : o ne fo r b uy in g a ne w c ar , a ss es si ng th e re lia bi lit y of a w eb si te b as ed o n in fo rm at io n on th e si te , a nd a ss es si ng th e re lia bi lit y of a w eb si te b as ed o n its U R L . 536 College & Research Libraries May 2017 calculate a pretest/posttest difference score for improved skill. Composite variables were developed by 1) combining the total score for a question, then 2) dividing by the maximum possible score for that question, 3) adding those scores across questions measuring the same skill, then 4) dividing by the number of questions to get the aver- age score per question. Responses were coded +2 to –2 for relevance and reliability. Pretest and posttest scale measures are summed responses of the individual measures, standardized by dividing each participant’s score by the maximum possible score for each item, then summing the standardized scores separately for pretest and posttest items. The pretest and posttest difference scores (see table 2, panel A) showed signifi- cant increase, with a percent increase of 102.7 percent for the summed responses to the posttest skills measures. Results of individual question measures are shown in table 2, panel B. Global warming and diabetes questions were asked on the pretest; new car, web reliable, and URL reliable were questions from the posttest. The global warming, diabetes, and new car questions gave students a brief information-need scenario and asked students to identify the most relevant and reliable sources for that topic from a list of sources including .com sites, Wikipedia, scholarly databases, news sources, and books. Web reliable asked students to identify criteria that impact a site’s reliability, such as references, advertisements, an author’s name, and last updated date. The URL reliable question asked students to identify domains that are considered more credible, such as .gov and .edu over .com, .net, and even .org sites. Not all pretest and posttest measures showed significant differences. Students were asked to identify the call number for a book on both the pretest and posttest as a baseline measure for students’ familiarity with library systems. Only 26 out of 41 students were able to provide a correct call number for the book on the pretest and 28 out of 41 students were able to provide a correct call number on the posttest. This is not surprising since the catalog and Library of Congress were not covered directly during the workshop; rather, this measure was used as a benchmark to gauge students’ awareness of library resources. Website evaluation questions on the pretest and posttest showed screenshots of two sites and asked students to choose which site was more reliable and explain why. On the pretest, students had two healthcare sites; on the posttest, screenshots showed two career websites. In the qualitative explanations, students listed a variety of crite- ria used to evaluate the sites’ reliability. On the pretest, 31 qualitative responses were coded as “The site I chose is more reliable.” On the posttest, students demonstrated a wider range of evaluation strategies (shown in table 3) through coded open-ended responses. The posttest responses included a range of criteria students had considered, including authorship, purpose, brand recognition, relevance of scope, and accuracy of the information presented. Posttest comments indicated students used more than one criterion when considering a site, such as “[w]ebsite A was specific to a cover letter, but this website is coming from a job site and it has the author’s name. There’s probably more research to be done on the site.” In contrast, student comments from the screenshot comparison during the pretest indicate that students considered fewer criteria before the workshop. H2 Perceived Confidence Results When asked on the pretest, “What do you struggle with most when searching for information online?” students responded with a range of concerns. Table 4 shows categories coded from students’ open-ended responses. Based on these qualitative responses, the two most frequent concerns among participants were finding relevant information and evaluating information for reliability or credibility. Research in the Real World 537 Both the pretest and posttest asked four quantitative confidence questions related to students’ perceived abilities to use technology, find information online, choose reliable information, and use information from the Internet to make a decision. As shown in table 5, the results of all four statements improved significantly from the pretest to the posttest. When asked open-ended questions on the posttest regarding how they planned to use the workshop skills in everyday life, a student replied, “Knowing what to search and how to search it. Narrowing my search to what I need.” Similar comments indicate students felt more comfortable with searching after the workshop. TABLE 3 Post-Test Responses to Website Screenshot Question Which of these websites is probably a more reliable source of information about writing a cover letter for a job application? Because… Responses Site B has specific, in-depth content about cover letter writing 17 Recognize Monster.com as good for job searching 7 Author’s name is listed on Site B 6 Site B has fewer ads 6 Site A is trying to sell you something 3 Site B looks more professional 3 Site A has sample cover letters 3 Site B more reliable/authoritative 4 Site B has reviewers’ rating for the article 2 Site B has current info 1 Site B has relevant links 1 Site B doesn’t have obvious bias 1 Unsure about “monster.com” 1 Other: thecareerscollege.com 1 No response 7 TABLE 4 Coded Responses for Participants’ Reported Research Challenges What do you struggle with most when searching for information online? Response Count Relevance/specificity 14 Reliability/credibility 13 Finding free, full-text scholarly articles 5 Choosing database/search strategies 2 Not sure where to start 1 Citing sources 1 Finding consensus among sources 1 Choosing search terms 1 http://Monster.com http://monster.com http://thecareerscollege.com 538 College & Research Libraries May 2017 H3 Perceived Relevance Results To determine students’ perceived relevance of the workshop strategies and real-world scenarios, the posttest asked students to rate the workshop based on four relevance statements, shown in table 6. The dependent variables for these analyses are the differ- ences in participants’ responses from the midpoint of the 5-point Likert scale. Results are statistically significant for all four statements. Students were asked an open-ended question on the pretest: “What do you struggle with most when searching for information online?” Table 7 shows coded responses to this qualitative question. Students’ top responses included the relevance and specificity of the information they find and concerns with finding reliable and cred- ible information. Table 8 shows students’ coded open-ended responses to the question, “How might you use the search strategies we talked about in your everyday life?” Students re- sponded by saying they would use the workshop strategies at work to do research before buying something, to vet businesses or organizations, for researching activities or hobbies, for debates with friends, or to find medical information. Individual student comments included, “Diagnosing medical conditions, research for voting in election season, academic work for research papers,” “Searching for information on where to buy a house,” “‘Downtown Denver’ AND restaurant,” and “I never even looked at Google news before. Time to catch up w[ith] the world.” TABLE 6 Average Responses for Workshop Relevance Questions Item Mean (Std) Difference from Midpoint t P-value Content builds on things I already know 4.60 (0.59) 1.60 17.1 <.0001 Content is relevant to my interests 4.65 (0.65) 1.65 16.1 <.0001 Info in lesson will be useful to me 4.90 (0.30) 1.90 40.6 <.0001 Instructor makes subject seem important 4.80 (0.40) 1.80 28.80 <.0001 Legend: Response scale 5 = high; 1 = low. TABLE 5 Paired T-test Results for Pretest and Posttest Confidence Questions Question Pretest Mean (Std.) Posttest Mean (Std.) T P-Value Confidence in using technology 3.7 (1.2) 4.1 (0.8) 4.4 <.0001 Confidence in finding information online 3.4 (1.1) 4.5 (0.6) 7.0 <.0001 Confidence in choosing information that is credible and reliable 3.3 (1.0) 4.6 (0.5) 8.6 <.0001 Confidence in using information from the internet to make a decision 3.5 (1.1) 4.4 (0.6) 6.4 <.0001 Research in the Real World 539 Discussion This study addresses a gap in literature on information literacy sessions and motiva- tional design as identified by Hess58 and integrates both PBL and ARCS for the first time. This study also contributes to research on the effectiveness of information literacy teaching for improving nontraditional or community college students’ knowledge and skills. Skills Strengthening students’ critical thinking abilities as they relate to the reliability of Internet information is important since the transferability of Internet research skills will be useful to students in the workforce as well as in their academic courses.59 The pretest and posttest include both qualitative and quantitative questions in the form of TABLE 7 Coded Responses for Students’ Perceived Search Difficulties What do you struggle with most when searching for information online? Response Count Relevance/specificity 14 Reliability/credibility 13 Finding free, scholarly articles 5 Information overload 5 Choosing database/search strategies 2 Not sure where to start 2 Saving and organizing sources 2 Citing sources 1 Finding consensus among sources 1 Choosing search terms 1 No response 1 TABLE 8 Coded Responses for Everyday Use of Search Strategies How might you use the search strategies we talked about in your everyday life? Response Count For school work 11 Searching more specifically/efficiently 15 Finding accurate/reliable info 9 Medical questions 4 For personal research 4 Questioning/evaluating information 3 Faster searching 3 Using Boolean operators 3 Prove a point in discussions with friends 2 Choosing better search terms 2 No response 1 540 College & Research Libraries May 2017 short scenarios to determine changes in students’ abilities to evaluate information for reliability and relevance. Analyzing composite variables indicates students’ Internet research skills increased after the workshop, resulting in a percentage increase of 102.7 from the pretest to the posttest. In open-ended responses on the posttests, most students demonstrate an under- standing of the critical thinking abilities needed to assess information found on the Internet. Qualitative pretest and posttest responses show the development of students’ evaluation abilities. On the posttest, for instance, students consider several additional evaluation criteria such as brand-name recognition, the presence/absence of an author’s name, advertising, professional appearance, and other criteria as shown in table 3. In contrast, the same measure on the pretest elicited only a handful of evaluation criteria before the workshop. Confidence Building adult learners’ confidence is important for engaging them in instruction. This study examines students’ perceived confidence in their technology skills, their ability to locate and apply information to PBL scenarios, as well as their evaluation of infor- mation sources. Both the qualitative question on the pretest, “What do you struggle with most when searching for information online?” and the quantitative questions on the pretest and posttest show students’ biggest concerns are finding relevant, reliable and credible information online. As table 5 shows, all four confidence measures are statistically significant after the workshop, indicating that students feel more confident in their research abilities after participating in the workshop. Increased confidence in their research abilities contributes to students’ self-empowerment and self-efficacy and may serve as an antidote to “library anxiety.”60 Relevance It can be challenging to show students why the research skills covered in a one-shot information literacy session are useful to them. Four relevance statements show sta- tistically significant differences after the workshop. The open-ended question on the pretest asks students to identify what they struggle with most when finding information online. The top-coded responses match two of the learning objectives for the workshop: finding reliable and specific information online. From these open-ended responses, it can be inferred that the material covered during the workshop is in line with students’ perceived needs for improving their search skills. Qualitative responses on open-ended questions in the posttest support the relevance ratings (see table 8). Students report thinking they would use the research skills from the workshop at school as well as for personal research, including medical needs, to fact-check information, and generally to improve their speed and accuracy when searching. Seeing in their own words how students plan to use research skills illustrates the personal relevance they found in the search strategies covered. Using realistic PBL scenarios during the session is one method to establish relevance and demonstrate the transferability of the skills that students are learning. Findings and the Literature As mentioned above, the new Framework for Information Literacy for Higher Education focuses information literacy teaching around six threshold concepts: “1) authority is constructed and contextual; 2) information creation as a process; 3) information has value; 4) research as inquiry; 5) scholarship as conversation; 6) searching as strategic exploration.”61 Although this study was designed before release of the Framework, the ARCS Model of Motivational Design and PBL aspects of this study directly relate to several of the Framework concepts. Research in the Real World 541 Specifically, the workshop addressed: 1) the construction of authority in various Internet sources, from freely available databases like PubMed to the variable reliability of news sites aggregated in Google News, overlapping with authority is constructed and contextual; 2) the relative value of subscription database information and how to find similar substitutes through top-level domain searching and open access journals and books, overlapping with information, has value; and 3) PBL scenarios as a way of problem solving and providing an iterative context for defining and refining information problems, overlapping with research as inquiry and searching as strategic exploration. PBL allows students to experiment hands-on with the abstract concepts that underlie realistic information problems. Based on the open-ended responses shown in table 8, students clearly made connections between real-life information needs and the every- day use of search strategies covered. Using three short PBL scenarios in one session allows students to see “hugging” of concepts from one scenario to the next, with the hope of encouraging greater transfer to real-world problems.62 PBL’s active learning characteristics are complementary to Meyer and Land’s con- ception of threshold concepts for teaching content, since PBL scenarios are frequently bounded to a particular realistic scenario, troublesome in that there is no one correct answer or way to solve the problem, and integrative in that they allow students to con- nect and apply many abstract concepts.63 While PBL is a teaching method and threshold concepts apply to content, the results of this study suggest they may be used together to strengthen information literacy teaching. PBL also complements the ARCS Model of Motivational Design by presenting an op- portunity for students to participate in the practical applications of information literacy, have more control over their own learning, and use their past life experiences to enrich their formal education, suggesting these applications can build students’ confidence and perhaps their satisfaction with the learning experience.64 When college students are given realistic scenarios and the chance to immediately practice new techniques, they are more likely to find the material relevant to their needs, which Keller identifies as being important for adult learners.65 The results suggest that the Framework’s threshold concepts are perhaps most ef- fective when combined with authentic learning contexts, such as teaching techniques like PBL, so that students can deeply understand how the concepts work in real life. Such techniques help address the gaps in recent graduates’ IL skills as identified by Project Information Literacy, particularly through discussion with peers, use of different formats, and practice making connections to the larger PBL problem.66 Incorporating a reflexive question or element following the PBL scenario can help students recognize the transferability of the skills they have learned and make connections to the bigger picture understandings of information literacy. The workshop also aimed to gently stimulate students’ metacognitive awareness and ability to self-assess their research skills by asking open-ended questions on the pretest and posttest. These questions helped determine students’ perceptions of their current skill levels, areas where they struggle, and how they might use the skills from today’s session in their academic and personal lives.67 The workshop addressed three of the four metaliteracy learning goals related to evaluating multimodal online content, encouraging discussions with peers, and making connections with students’ goals and needs outside the classroom.68 Limitations This study used a convenience sample rather than control groups and random sampling. Because convenience sampling was used, the participants may not be representative of the larger population of students at the institution. A higher number of male stu- 542 College & Research Libraries May 2017 dents and a higher number of younger students were present in the sample than in the general student population. Examining external validity by similar testing at other institutions would be beneficial. There is inherent subjectivity involved when using self-reported measures of con- fidence and relevance. Additionally, as Gross and Latham point out, students’ confi- dence is often overestimated when it comes to their research abilities.69 It would be extremely valuable to have a validated instrument to use for pretests and posttests of one-shot information literacy sessions at the college level, as homegrown instruments have limited scope and applicability outside their native institution. Finally, a delayed postexperimental questionnaire would help indicate the lasting effects of the research workshop on students’ perceived confidence, perceptions of relevance, and skills. Conclusions Nontraditional students are increasingly part of information literacy sessions at the college level. To be prepared for the increasingly multimodal workforce, students need information literacy skills that will transfer from academic to work and personal landscapes.70 To engage students in information literacy sessions, librarians should be aware of the needs of nontraditional students and consider the motivations of these adult learners. This study adds to the literature by combining Keller’s ARCS Model of Motivational Design with PBL scenarios to build students’ Internet research skills. Models such as Keller’s ARCS can help with motivational strategies that increase student engagement. Active-learning techniques such as PBL give students hands-on practice with threshold concepts. The Research in the Real World workshop had significant results across all three hypotheses, which demonstrate how students’ confidence in their Internet research skills increased after the workshops and suggest that students’ knowledge and evalu- ation skills improved as a result of the PBL scenarios. Students also reported that the workshop and scenarios were relevant to their needs. In light of the move to threshold concepts included in the Framework, librarians are challenged to understand how to assess concepts that are acknowledged to be “trouble- some” and likely to stump students.71 Such concepts may not lend themselves easily to large-scale quantitative assessments as the previous Standards did; rather, librarians have an opportunity to work together with learning scientists to develop valid and reliable methods for assessing Framework components in chunks, or perhaps through cognitive and performance-based assessments such as portfolios or outcomes of longer PBL projects, as Jacobson and Hu suggest.72 Through such innovative assessment methods, academic librarians can demonstrate how information literacy skills contribute to greater metacognitive awareness and can be transferred to other contexts. Ultimately, the findings of this study support the con- tinued interest among librarians for finding more interactive and hands-on methods for teaching and assessing information literacy concepts. These findings and future implications are relevant not only to librarians and educators but also to employers and higher education administrators who have an interest in preparing a workforce that can successfully navigate the Internet and make decisions from the wealth of available information. Research in the Real World 543 APPENDIX A. Description of Problem-Based Learning Scenarios • Scenario 1: You are tutoring a friend in ENG 090 who is writing a paper on Martin Luther King, Jr and using websites to find information. Take a look at http://www.martinlutherking.org/ and http://www.thekingcenter.org/. Think about the CRAP criteria for evaluating sites. Which website would you recom- mend your friend use and why? This scenario allowed students to evaluate and compare a reliable site from the non- profit King Center against a hate site sponsored by a white supremacy group. Students needed to explore both sites to learn about them, using the currency, reliability, author- ity, purpose/point of view criteria as a guide. • Scenario 2: You work for a company that helps businesses prepare for natural disasters. Your boss has a meeting with a potential client in 15 minutes. The client mentioned concern about “the disaster that destroyed Tuscaloosa in 2011.” Your boss wants you to quickly find information on this event so she’ll be prepared to talk about it with the client. This needs to be reliable so that she can mention where you found it. Students were able to use Google News and Advanced Search features to limit by date, among other strategies. They also needed to consider the best keywords to use to find articles about the 2011 tornado. • Scenario 3: You are babysitting your niece for a few days, who comes home from school with red, itchy eyes. You need to find out a little information about what this could be and what you should do about it. You want the information to be from a trusted source. The final scenario aimed to get students to find reliable consumer health information, perhaps by using site:.gov in Google or looking for sites like MedlinePlus or WebMD that are familiar and credible. Students briefly discussed the scenarios as a group after each was completed, comparing techniques and sites they had used that worked well and sharing where they had gotten stuck. Students completed a paper posttest after the workshop. http://www.martinlutherking.org/ http://www.thekingcenter.org/ 544 College & Research Libraries May 2017 APPENDIX B. Pretest and Posttest Instruments Pretest Participant ID ___________________ (please use two letters and a two-digit number, such as AR03) 1. How would you describe yourself? Please check: q First-time college student q Continuing student q Returning student (after a gap of two or more years) q Other: 2. Which program are you pursuing? Please check: q full-time student q part-time student 3. Your age: 4. Please check: q Female q Male q Transgender or other 5. Is English your first language? q Yes q No 6. Have you ever been to a library instruction session q at our Community College q at another college q in high school q somewhere else (please specify __________________________________________) q have not attended a library instruction session before 7. How would you rate yourself on the following? Please circle. 1 = low, 5 = high Confidence in using technology 1 2 3 4 5 Confidence in finding information online 1 2 3 4 5 Confidence in choosing information that is credible and reliable 1 2 3 4 5 Confidence in using information from the Internet to make a decision 1 2 3 4 5 8. What do you struggle with most when searching for information online? 9. If you were looking for information on global warming for an academic paper, where would you look? (Please check all that apply.) q Google search for the topic q Wikipedia q Newspaper or magazines q Books q Scholarly journals q Library database q Other (please specify_______________________________________) 10. If you wanted to find diabetes information for a family member on the Internet, where would you look? (Please check all that apply.) q Google search for the topic q Specific website, such as WebMD.com q An online database, such as PubMed q Wikipedia q Other (please specify_______________________________________) 11. Go to the catalog search from the library homepage. Find the book Roadside History of Colorado by James McTighe. What is the call number? ____________________ 12. If you were searching Google for information on autistic children in Colorado, what words would you use to search? _______________________________ http://WebMD.com Research in the Real World 545 13. Which of these websites is probably a more reliable source of information about the new healthcare laws? q Website A because ____________________________________________________ _______________________________________________________________________ q Website B because ___________________________________________________ ______________________________________________________________________ 14. What happens when you “put something in quotes” when searching? q It searches the words as an exact phrase q Quotation marks are the correct format for a title q It broadens your search q Only books will be listed in the results 546 College & Research Libraries May 2017 15. What is an open access journal? q An online diary or blog q A free scholarly journal q An online newspaper q None of the above Posttest Participant ID ___________________ (please use two letters and a two-digit number, such as AR03) 1. Please rate the following statements about the Research in the Real World work- shop 1–5 based on how you agree or disagree with them. 1 = low, 5 = high It is clear to me how the content of this material builds on things I already know. 1 2 3 4 5 The content of this material is relevant to my interests. 1 2 3 4 5 The information in this lesson will be useful to me. 1 2 3 4 5 The instructor makes the subject matter seem important. 1 2 3 4 5 2. After taking the workshop, how would you rate yourself on the following? Please circle. 1 = low, 5 = high Confidence in using technology 1 2 3 4 5 Confidence in finding information online 1 2 3 4 5 Confidence in choosing information that is credible and reliable 1 2 3 4 5 Confidence in using information from the Internet to make a decision 1 2 3 4 5 3. What techniques from today’s workshop will help you when searching for infor- mation online? ________________________________ ________________________________________ ________________________________________ 4. If you were looking for information on buying a new car on the Internet, where would you look? (Please check all that apply.) q Google search for the topic q Specific auto-buying website, such as Kelley Blue Book (kbb.com) or Edmunds.com q An online database, such as Academic Search Premier Plus q Wikipedia q Other (please specify_____________) 5. Go to the catalog search from the library homepage. Find the book Fight Club, by Chuck Palahniuk. What is the call number? ____________________ 6. If you were searching PubMed for information on autistic children in Colorado, what would be the best way to search? (Please check one.) q autism in Colorado q autism children Colorado q autism AND children AND Colorado http://Edmunds.com Research in the Real World 547 7. Which of these websites is probably a more reliable source of information about writing a cover letter for a job application? q Website A because ____________________________________________________ _______________________________________________________________________ q Website B because ____________________________________________________ _______________________________________________________________________ 548 College & Research Libraries May 2017 8. Which parts of a website will help you determine if the site has reliable informa- tion? (Circle all that apply) q An author’s name is listed q The site was last updated in 2002 q A link to Facebook is included on the site q The author includes references q The site has advertisements 9. Which of these urls are considered more reliable? (Circle all that apply) q.org q.com q.net q.gov q.edu 10. How might you use the search strategies we talked about in your everyday life? 11. What part of this workshop will you use outside academic work? 12. What else would you like to learn about searching for information online? 13. Comments, questions, or suggestions? Thank you! Notes 1. American Library Association, “Presidential Committee on Information Literacy: Final Report” (Chicago, 1989), available online at http://www.ala.org/ala/mgrps/divs/acrl/publications/ whitepapers/presidential.cfm [accessed 28 October 2015]. 2. Annemaree Lloyd, “Building Information Resilient Workers: The Critical Ground of Work- place Information Literacy: What Have We Learnt?” in Worldwide Commonalities and Challenges in Information Literacy Research and Practice (Cham, Switzerland: Springer International Publishing, 2013), 219. 3. Shirley J. Behrens, “A Conceptual Analysis and Historical Overview of Information Literacy,” College & Research Libraries 55, no. 4 (1994): 312; Rebecca Z. Kuglitsch, “Teaching for Transfer: Reconciling the Framework with Disciplinary Information Literacy,” portal: Libraries and the Academy 15, no. 3 (2015): 457–70, doi:10.1353/pla.2015.0040. 4. National Center for Education Statistics and Susan Choy, Findings from the Condition of Education 2002: Nontraditional Undergraduates, NCES 2002-012 (Washington, D.C.: National Center for Education Statistics, 2002), available online at https://nces.ed.gov/fastfacts/display.asp?id=16 [accessed 2 September 2015]. 5. National Center for Education Statistics, William J. Hussar, and Tabitha M. Bailey, Projec- tions of Education Statistics to 2022, NCES 2014-051 (Washington, D.C.: U.S. Government Printing Office, 2014), available online at http://nces.ed.gov/pubs2014/2014051.pdf [accessed 15 July 2015]. 6. National Center for Education Statistics et al., Demographic Enrollment Characteristics of Nontraditional Undergraduates: 2011–12, NCES 2015-025. (Washington, D.C.: U.S. Government Printing Office, 2015), available online at http://nces.ed.gov/pubs2015/2015025.pdf [accessed 3 October 2015]. 7. John M. Keller, “Development and Use of the ARCS Model of Instructional Design,” Jour- nal of Instructional Development 10, no. 3 (1987): 2–10, doi:10.1002/pfi.4160260802; John M. Keller, Motivational Design for Learning and Performance: The ARCS Model Approach (New York: Springer, 2010), doi:10.1007/978-1-4419-1250-3. 8. Barbara H. Kwasnik, “Information Literacy: Concepts of Literacy in a Computer Age,” in http://q.org http://q.com http://q.net http://q.gov http://q.edu http://www.ala.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm http://www.ala.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm http://10.1353/pla https://nces.ed.gov/fastfacts/display.asp?id=16 http://nces.ed.gov/pubs2014/2014051.pdf http://nces.ed.gov/pubs2015/2015025.pdf http://10.1002/pfi Research in the Real World 549 Information Literacies for the Twenty-First Century, eds. Virgil L.P. Blake and Renee Tjoumas (Boston: G.K. Hall & Company, 1990), 127–43; Behrens, “A Conceptual Analysis and Historical Overview of Information Literacy”; Loanne Snavely and Natasha Cooper, “The Information Literacy Debate,” Journal of Academic Librarianship 23, no. 1 (1997): 9–13, available online at www.sciencedirect.com/ science/article/pii/S0099133397900665 [accessed 22 March 2013]; David Bawden, “Information and Digital Literacies: A Review of Concepts,” Journal of Documentation 57, no. 2 (2001): 218–59. 9. Lloyd, “Building Information Resilient Workers”; Michele Forte et al., “Metaliteracy: Goals and Learning Objectives,” 2014, available online at https://metaliteracy.org/learning-objectives/ [accessed 3 October 2015]; Alison J. Head et al., “What Information Competencies Matter in Today’s Workplace ?” Library and Information Research 37, no. 114 (2013): 74–104. 10. Thomas P. Mackey and Trudi E. Jacobson, “Reframing Information Literacy as a Metalit- eracy,” College & Research Libraries 72, no. 1 (2011): 62–78. 11. Mackey and Jacobson, “Reframing Information Literacy as a Metaliteracy”; Trudi E. Jacob- son and Thomas P. Mackey, “Proposing a Metaliteracy Model to Redefine Information Literacy,” Communications in Information Literacy 7, no. 2 (2013): 84–91. 12. Forte et al., “Metaliteracy: Goals and Learning Objectives.” 13. Keith Thiede, “Learning to Learn and Metacognition,” Encyclopedia of Education 13 (New York: Macmillan Reference USA, 2003); 1470; Jennifer Livingston, “Metacognition: An Overview,” 1997, available online at http://gse.buffalo.edu/fas/shuell/cep564/metacog.htm [accessed 22 March 2016]. 14. David Billing, Teaching for Transfer of Core/key Skills in Higher Education: Cognitive Skills, Higher Education, vol. 53, 2007, doi:10.1007/s10734-005-5628-5; David N. Perkins and Gavriel Salomon, “Teaching for Transfer,” Educational Leadership 46, no. 1 (1988): 22–32. 15. Jan Meyer and Ray Land, “Threshold Concepts and Troublesome Knowledge: Linkages to Ways of Thinking and Practising within the Disciplines,” in ETL Project Occasional Report 4, 2003, doi:10.1007/978-3-8348-9837-1. 16. Lori Townsend, Korey Brunetti, and Amy R. Hofer, “Threshold Concepts and Information Literacy,” portal: Libraries and the Academy 11, no. 3 (2011): 853, 857–858, doi:10.1353/pla.2011.0030. 17. Amy R. Hofer, Korey Brunetti, and Lori Townsend, “A Thresholds Concept Approach to the Standards Revision,” Communications in Information Literacy 7, no. 2 (2013): 108–13, available online at www.comminfolit.org/index.php?journal=cil&page=article&op=view&path[]=v7i2p10 8&path[]=168 [accessed 28 February 2016]. 18. Billing, Teaching for Transfer of Core/key Skills in Higher Education; Perkins and Salomon, “Teaching for Transfer.” 19. Annemaree Lloyd, “Drawing from Others: Ways of Knowing about Information Literacy Performance,” in Lifelong Learning: Partners, Pathways, and Pedagogies: Keynote and Refereed Papers from the 4th International Lifelong Learning Conference, Yeppoon, Central Queensland, Australia, 13–16 June 2006, eds. D. Orr et al. (Rockhampton: Central Queensland University Press, 2006), 182–92, available online at http://acquire.cqu.edu.au:8080/vital/access/manager/Repository/cqu:589 [ac- cessed 22 March 2016]; Lloyd, “Building Information Resilient Workers”; Kuglitsch, “Teaching for Transfer.” 20. Lloyd, “Building Information Resilient Workers,” 220. 21. Alison J. Head, “Learning Curve: How College Graduates Solve Information Problems Once They Join the Workplace,” Project Information Literacy Research Report, 2012, 24, doi:10.2139/ ssrn.2165031; Head et al., “What Information Competencies Matter in Today’s Workplace ?” 87. 22. American Library Association, “Presidential Committee on Information Literacy: Final Report.” 23. Association of College, Research Libraries, and American Library Association, Information Literacy Competency Standards for Higher Education (Chicago, Ill.: American Libraries Association, 2000). 24. Hofer, Brunetti, and Townsend, “A Thresholds Concept Approach to the Standards Revision.” 25. Association of College & Research Libraries, “Framework for Information Literacy for Higher Education” (Chicago, Ill.: American Library Association, 2016), available online at www. ala.org/acrl/standards/ilframework [accessed 28 February 2016]. 26. Keller, “Development and Use of the ARCS Model of Instructional Design.” 27. Keller, Motivational Design for Learning and Performance. 28. Ibid. 29. Ibid., 55–56. 30. Ibid., 60. 31. Trudi E. Jacobson and Lijuan Xu, Motivating Students in Information Literacy Classes (New York: Neal-Schuman, 2004). 32. Ibid., 101–125. 33. Melissa Gross, Don Latham, and Bonnie Armstrong, “Improving Below-Proficient Infor- http://www.sciencedirect.com/science/article/pii http://www.sciencedirect.com/science/article/pii https://metaliteracy.org/learning http://gse.buffalo.edu/fas/shuell/cep564/metacog.htm http://10.1353/pla http://www.comminfolit.org/index.php http://acquire.cqu.edu.au http://10.2139/ssrn http://10.2139/ssrn http://www.ala.org/acrl/standards/ilframework http://www.ala.org/acrl/standards/ilframework 550 College & Research Libraries May 2017 mation Literacy Skills: Designing an Evidence-Based Educational Intervention,” College Teaching 60, no. 3 (2012): 104–11, doi:10.1080/87567555.2011.645257; Melissa Gross and Don Latham, “Ad- dressing below Proficient Information Literacy Skills: Evaluating the Efficacy of an Evidence- Based Educational Intervention,” Library and Information Science Research 35, no. 3 (2013): 181–90, doi:10.1016/j.lisr.2013.03.001; Christine S. Bruce, The Seven Faces of Information Literacy (Blackwood, South Australia: Auslib Press Pty Ltd, 1997); Christine S. Bruce, Informed Learning (Chicago: As- sociation of College and Research Libraries, 2008). 34. Ruth V. Small, Nasriah Zakaria, and Houria El-figuigui, “Motivational Aspects of Informa- tion Literacy Skills Instruction in Community College Libraries,” College & Research Libraries 65, no. 2 (2004): 96–121. 35. Ibid. 36. Howard S. Barrows and Robyn M. Tamblyn, Problem-Based Learning: An Approach to Medical Education, Vol. 1 (New York: Springer, 1980); Chiu Yin Kwan, “What Is Problem Based Learning (PBL)?” CDTL Brief 3, no. 3 (n.d.): 1–2, available online at www.cdtl.nus.sg/Brief/Pdf/v3n3.pdf [accessed 22 March 2013]. 37. Cindy Hmelo-Silver, “Problem-Based Learning: What and How Do Students Learn?” Educational Psychology Reviewology Review 16, no. 3 (2004): 235–66. 38. Howard S. Barrows, “Is It Truly Possible to Have Such a Thing as dPBL?” Distance Educa- tion 23, no. 1 (2002): 119–22. 39. Johannes Strobel and Angela van Barneveld, “When Is PBL More Effective? A Meta-Syn- thesis of Meta-Analyses Comparing PBL to Conventional Classrooms,” Interdisciplinary Journal of Problem-Based Learning 3, no. 1 (2009): 44–58, doi:10.7771/1541-5015.1046. 40. Ibid. 41. Karen E. Downing, “Using Problem-Based Learning to Facilitate Student Learning,” in Imagine, Innovate, Inspire: The Proceedings of the ACRL 2013 Conference (2013), 621–24. 42. Keller, “Development and Use of the ARCS Model of Instructional Design.” 43. National Center for Education Statistics, “Digest of Education Statistics, 2014” (Washington, D.C.: U.S. Government Printing Office, 2014), available online at http://nces.ed.gov/programs/ digest/d14/tables/dt14_303.50.asp [accessed 15 May 2016]. 44. National Center for Education Statistics et al., “Characteristics of Postsecondary Students,” in The Condition of Education 2014, NCES 2014-083 (Washington, D.C., 2014), available online at http://search.ebscohost.com/login.aspx?direct=true&db=eric&AN=ED545122&site=ehost-live [accessed 3 October 2015]. 45. Mike Heery, “Academic Library Services to Non-Traditional Students,” Library Management 17, no. 5 (1996): 3–13. 46. Malcolm S. Knowles, Elwood F. Holton III, and Richard A. Swanson, The Adult Learner: The Definitive Classic in Adult Education and Human Resource Development (Routledge, 2014). 47. Ibid. 48. Don Latham and Melissa Gross, “Including Student Voices in Instructional Design: Com- munity College Students with Below-Proficient Skills Talk About IL Instruction,” in Canadian Association for Information Science, CAIS/ACSI 2010 Conference, Montreal, Canada, June 2–4, 2010 (Montreal, 2010), 1–5, available online at www.cais-acsi.ca/proceedings/2010/CAIS011_Latham_Fi- nal.pdf [accessed 1 February 2012]. 49. Don Latham and Melissa Gross, “Instructional Preferences of First-Year College Students with Below-Proficient Information Literacy Skills: A Focus Group Study,” College & Research Libraries 74, no. 5 (2012): 430–49, doi:10.5860/crl-343. 50. Núria Ferran-Ferrer, Julià Minguillón, and Mario Pérez-Montoro, “Key Factors in the Transfer of Information-Related Competencies between Academic, Workplace, and Daily Life Contexts,” Journal of the American Society for Information Science and Technology 64, no. 6 (2013): 1118. 51. Ibid. 52. Keller, Motivational Design for Learning and Performance, 278–80; 283–84. 53. Townsend, Brunetti, and Hofer, “Threshold Concepts and Information Literacy.” 54. Perkins and Salomon, “Teaching for Transfer.” 55. Barrows and Tamblyn, Problem-Based Learning. 56. Keller, Motivational Design for Learning and Performance; Thomas P. Mackey and Trudi E. Jacobson, Metaliteracy: Reinventing Information Literacy to Empower Learners (Chicago, Ill.: Neal- Schuman, 2014), 13. 57. Mackey and Jacobson, Metaliteracy, 9. 58. Amanda Nichols Hess, “Motivational Design in Information Literacy Instruction,” Com- munications in Information Literacy 9, no. 1 (2015): 44–59. 59. Head et al., “What Information Competencies Matter in Today’s Workplace ?”; Kuglitsch, “Teaching for Transfer”; Lloyd, “Building Information Resilient Workers.” 60. Marilyn P. Arnone, Ruth V. Small, and Rebecca Reynolds, “Supporting Inquiry by Identify- http://10.1016/j.lisr http://www.cdtl.nus.sg/Brief/Pdf/v3n3.pdf http://nces.ed.gov/programs/digest/d14/tables/dt14_303.50.asp http://nces.ed.gov/programs/digest/d14/tables/dt14_303.50.asp http://search.ebscohost.com/login.aspx?direct=true&db=eric&AN=ED545122&site=ehost-live http://www.cais-acsi.ca/proceedings/2010/CAIS011_Latham_Final.pdf http://www.cais-acsi.ca/proceedings/2010/CAIS011_Latham_Final.pdf http://10.5860/crl Research in the Real World 551 ing Gaps in Student Confidence: Development of a Measure of Perceived Competence,” School Libraries Worldwide 16, no. 1 (2010): 47–60; Nahyun Kwon, Anthony J. Onwuegbuzie, and Linda Alexander, “Critical Thinking Disposition and Library Anxiety: Affective Domains on the Space of Information Seeking and Use in Academic Libraries,” College & Research Libraries 68, no. 3 (2007): 268–79, available online at http://crl.acrl.org/content/68/3/268.abstract [accessed 3 April 2014]. 61. ACRL, “Framework for Information Literacy for Higher Education.” 62. Hess, “Motivational Design in Information Literacy Instruction”; Perkins and Salomon, “Teaching for Transfer,” 29; Kuglitsch, “Teaching for Transfer,” 465; Patricia Senn Breivik, “21st Century Learning and Information Literacy,” Change 37, no. 2 (2005): 20–27; Bawden, “Informa- tion and Digital Literacies.” 63. Meyer and Land, “Threshold Concepts and Troublesome Knowledge.” 64. Knowles, Holton III, and Swanson, The Adult Learner. 65. Keller, Motivational Design for Learning and Performance. 66. Head et al., “What Information Competencies Matter in Today’s Workplace ?” 67. Mackey and Jacobson, Metaliteracy, 2. 68. Forte et al., “Metaliteracy: Goals and Learning Objectives.” 69. Gross and Latham, “Addressing below Proficient Information Literacy Skills.” 70. Lloyd, “Building Information Resilient Workers.” 71. Hofer, Brunetti, and Townsend, “A Thresholds Concept Approach to the Standards Revi- sion.” 72. Jacobson and Xu, Motivating Students in Information Literacy Classes. http://crl.acrl.org/content/68/3/268.abstract OLE_LINK5 OLE_LINK6 OLE_LINK7 OLE_LINK3 OLE_LINK4 OLE_LINK1 OLE_LINK2 _GoBack _GoBack