Zoellner.indd Continuing Assessment of Library Instruction to Undergraduates: A General Education Course Survey Research Project Kate Zoellner, Sue Samson, and Samantha Hines An assessment project was conducted by the Mansfield Library to evaluate the teaching and content of a research module embedded into a popular undergraduate course, Introduction to Public Speaking, part of the University of Montana’s General Education Requirement. The project aimed to develop internal benchmarks and improve the curricula, as needed. Pre- and post-surveys were developed to assess student research confidence, perceptions of information tools, Web evaluation abilities, and assistance-seeking attitudes. Data from 426 student re- sponses were analyzed using qualitative and quantitative methods. The process, results, and analysis of the project are discussed. ntroductory level coursework provides a rich opportunity for libraries to embed the basic elements of information literacy into the academic curriculum. The premise that all lower-division undergraduates should receive basic library instruction prior to entering their major coursework is well grounded and similar for curricula across the disciplines. Efforts to determine the efficacy of the information literacy instructional models in place for lower-division coursework have become increasingly important as librarians seek means to determine the effectiveness of their instruction efforts, and assessment is required by academic accrediting agencies.1 Assessment of li- brary instruction is not new; but current trends are student-centered, focus on learning outcomes, and are challenging to document effectively.2 Building on the model of pre- and post-testing, the information literacy as- sessment project described here focuses on research confidence, perceptions of information tools, Web evaluation abili- ties, and assistance-seeking a itudes of students enrolled in an entry-level public speaking course that is part of the General Education curriculum at the University of Montana. The project builds on pre- Kate Zoellner is Assistant Professor, Education Librarian and Assessment Coordinator; Sue Samson is Professor, Head, Information & Research Services Division and Humanities Librarian; and Samantha Hines is Assistant Professor, Social Sciences Librarian and Distance Education Coordinator in the Mansfield Li- brary at the University of Montana-Missoula; e-mail: kate.zoellner@umontana.edu, sue.samson@umontana. edu, samantha.hines@umontana.edu. Special thanks is extended to Merinda McLure, Assistant Professor and Applied Human Sciences Librarian, Colorado State University Libraries, Fort Collins, for her early participation in research design and implementation. 370 mailto:samantha.hines@umontana.edu mailto:sue.samson@umontana mailto:kate.zoellner@umontana.edu Continuing Assessment of Library Instruction to Undergraduates 371 vious institutional information literacy assessment, expands the reliability of the pre- and post-test model of assessment for libraries, identifies statistically significant levels of confidence building in the use of library tools and perceived importance of those tools, and reveals that students view library instruction as a key component in increasing their research confidence. Review of the Literature Recent literature on pre- and post-test- ing of information literacy initiatives is varied and diverse and underscores the need for further use of this method to strengthen its reliability as an assessment tool. Manuel (2005) provides a thorough review of literature on first-year instruc- tion and addresses the questions of what first-year students know about informa- tion research and what we can teach them. Building on previous pre- and post-test findings, her study assessed the learning of students enrolled in a required English composition course based on a course- integrated library instruction module. Implicit in her findings were that students can provide articulate explanations for searching and evaluating strategies and that misinformation about the use of information persists a er instruction. She emphasized the importance of research design, the piloting of tests prior to their use, and the understanding of how stu- dents frame their own understanding of information and search strategies rather than using a narrow set of choices. Additional pre- and post-test studies have assessed a variety of information literacy initiatives. Although no sta- tistical demographic differences were established, the LUMENS Project demon- strated that interactive multimedia shows were effective teaching tools for library- user education content.3 A study on the impact of librarians in first-year medical and dental problem-based learning pro- viding instruction to small groups found no statistical difference from instruction to large groups.4 In an assessment of the model of teaching the teachers, randomly selected sections of English composition supported the premise that classroom instructors were more effective than either librarians or online tutorials in effective information literacy instruction delivery.5 No statistical significance was found when comparing information literacy instruction in two sections of an under- graduate telecommunications class—one led by an instructor, the other utilizing Web-based content delivery.6 Carter dis- cusses outcome-focused assessments of library research instruction using pre- and post-tests, a itude and usage surveys, and focus groups.7 Incoming provisional students were reported to make sig- nificant gains a er a four-week intensive summer program at Virginia Union University that included an information literacy component.8 Kaplowitz showed statistical significance in a pre- and post- test evaluation of the English 3-Library Instruction Program at the University of California at Los Angeles.9 Methodology This project was designed to develop internal benchmarks and improve the curricula, as needed, of the undergraduate Introduction to Public Speaking (COMM 111) course, offered as a General Education course by the Department of Communica- tion Studies. This class is taught primarily by graduate teaching assistants; the cur- riculum incorporates an integrated infor- mation literacy component developed by librarians in collaboration with the Com- munication Studies Faculty Supervisor. This same collaboration was extended in the design of a pre- and post- survey tool to explore the research confidence, percep- tions of information tools, Web evaluation abilities, and assistance-seeking a itudes of students enrolled in COMM 111 during spring 2006.10 Specifically, the following research questions were addressed: Do Introduction to Public Speaking students experience a change in their research confidence over the duration of the course? To what factors, if any, do students a ribute a change in 372 College & Research Libraries July 2008 their research confidence over the duration of the course? Do students recognize the current integration of library instruction into their Introduction to Public Speaking class as a factor in their research confidence changes? How, if at all, do students evalu- ate Web pages prior to and a er receiving library instruction? Is there a change in how students evaluate the quality of Web pages over the duration of the course? With what comparative importance do students view specific research resources? How comfortable are students asking for assistance at the library desk and from their COMM 111 instructor? What factors, if any, would make students more comfort- able seeking assistance? Research Design All COMM 111 class sections receive a 75-minute session focused on the effective use of information and library resources, inclusive of a hands-on Web site evalu- ation exercise, during week four of the sixteen-week semester. The development of this library instruction component evolved over the course of five years and is built on a student-centered approach to instruction. It extends and supports findings of previous research by Sam- son and Granath (2004) that confirmed both the value of curriculum-integrated instruction and the model of teaching the teachers to provide that instruction. Within this context, the content of the session is cra ed by the First-year Experi- ence Librarian and the Communications Studies Faculty Supervisor to augment the research preparation that students need to develop their assigned course presentations, to explore the content of their subject ma er, and to provide documentation of their sources (see Ap- pendix A). The survey research tool was designed to address the following: 1) research con- fidence levels; 2) perceptions of research resources; 3) evaluation of Web pages; 4) assistance-seeking comfort; and 5) changes in confidence of COMM 111 stu- dents prior to and a er they received the research library instruction intervention (see Appendix B). On both the pre- and post-surveys, research confidence was based on five specific research activities that students were asked to rate on a three-point likert-type scale. These five activities included: 1) overall ability to do research for COMM 111 assignments; 2) using the library catalog to find books on a specific topic; 3) using a search engine such as Google or Yahoo! to find infor- mative Web pages on a specific topic; 4) using a library database such as Academic Search Premier or Academic Index to find articles published in newspapers, maga- zines, or scholarly journals on a specific topic; and 5) determining the quality of information provided on any Web page. Perceptions of research resources were captured by asking students to rate the library catalog, databases, library staff, their COMM 111 instructor, and search engines as not important, somewhat im- portant, or important. To measure student Web evaluation practices and abilities, an open-ended box was provided a er the question, “Briefly describe how you determine the quality of information provided on a Web page. What do you look for? What do you think about?” And assistance-seeking comfort was gleaned through both closed- and open-ended questions that asked students to rate how comfortable they felt asking for assistance at the library desk and from their COMM 111 instructor. The Week 12 post-test in- cluded three additional questions aimed at capturing changes in student research confidence between the pre- and post-test, change factors, and changes to the COMM 111 course that would increase research confidence. Population and Sample All students enrolled in COMM 111 dur- ing the spring 2006 semester comprised the pre- and post-test survey population (n = 635). The majority of students were lower-division undergraduates (79%), with a small number of upper-division and nondegree students (19%, 1%). Stu- Continuing Assessment of Library Instruction to Undergraduates 373 dents were made aware of the survey through a handout from their teaching assistant instructor. Students were offered five extra credit points for participating in each of the pre- and post-test surveys; they were provided an alternate means of receiving these points, too. Variables, Data, and Hypothesis The independent variable in this study was the 75-minute research session stu- dents received during the fourth week of their sixteen-week COMM 111 course, while the dependent variables were the student responses on the survey. The null hypothesis was that there would be no experimentally important or consistent difference between COMM 111 student confidence, perceptions, or Web evalu- ation abilities before and a er receiving the research lesson. Statistical Procedures and Analyses Students completed both surveys online at any time during Week 3 and Week 12 of the semester. Response data were captured in an ACCESS database and then transferred to Excel for analysis. Quantitative data from the pre- and post-tests were compared using t-tests of means for each question and by status of students within each question to determine experimental importance and consistency. Descriptive comparisons were gathered using pivot tables. Open-ended comments were coded using the content analysis method; a TABLE 1 Changes in Student Research Confidence Overall, how confident do you feel about your ability to do research for COMM 111 speech assignments? Not Confident Somewhat Confident Confident –1.9% –13.0% 14.8% How confident do you feel using the library catalog to find books on a specific topic? Not Confident Somewhat Confident Confident –17.6% –5.9% 11.7% How confident do you feel using a library database such as Academic Search Premier or Academic Index to find articles published in newspapers, magazines, or scholarly journals, on a specific topic? Not Confident Somewhat Confident Confident –25.1% –7.5% 17.6% How confident do you feel using a search engine such as Google or Yahoo! to find infor- mative Web pages on a specific topic? Not Confident Somewhat Confident Confident –1.4% –0.3% 1.7% How confident do you feel about determining the quality of information provided on any Web page? Not Confident Somewhat Confident Confident –6.6% –17.7% 24.3% Overall, has your confidence about your ability to do research changed since the beginning of the semester? Yes No 54.0% 46.0% 374 College & Research Libraries July 2008 codebook was developed and comments unitized by the authors.11 Based on the codebook, two coders, independent from each other and the hypothesis, coded the open-ended re- sponses into the categories; the same two coders then met together to reach con- sensus on discrepancies between categories. Initial coder responses were trans- ferred to SPSS to measure inter-rater reliability using Cohen’s Kappa. Results From within the popula- tion, 214 self-selecting vol- untary respondents (34%), age 18 or older, completed the first pre-test survey in Week 3 of the semes- ter, prior to the library instruction class session that occurred during Week 4; 81 percent were lower-division students and 19 percent were upper division or other. From the same population, 212 self-select- ing voluntary respondents (33%), age 18 or older, completed the post-test survey during Week 12 of the semester, follow- ing submission of their final COMM 111 course assignment. These students were 85 percent lower-division and 15 percent upper-division students. While the authors were unable to track responses across time due to loss of iden- tifying data with the post-test survey, the sample size does meet the assumption of normality, enabling generalizations across the population. All t-tests comparing the pre-test data with the post-test data indi- cated a statistically significant difference between the means of the two groups (t [∞], p<.05). Cohen’s Kappa indicated acceptable inter-rater reliability (k = .70, p<.001) for all coded comments (n = 660) on the pre-test. Similarly, all open-ended comments on the post-test (n = 887) had an acceptable inter-rater reliability (k = .78, p<.001). TABLE 2 Student Evaluation of Web Page Quality Briefly describe how you determine the quality of information provided on a Web page. What do you look for? What do you think about? Category Pre-test (mentions) Post-test (mentions) Difference Authority 29% (101) 23% (107) –6% Accuracy 13% (45) 13% (61) 0% Point of View/Bias 6% (21) 8% (35) 1% Reliability 16% (54) 26% (119) 10% Timeliness 10% (35) 19% (89) 9% Validity 8% (29) 3% (13) –6% Design 7% (25) 2% (11) –5% Information Need 3% (9) 3% (12) 0% Referral <1% (3) <1% (2) <0% Research Confidence In all questions related to research confi- dence, students showed a trend toward increased confidence (see table 1). The largest increase in confidence (24%) was in a student’s ability to determine the quality of information on any Web page. Student confidence in using a library database (18% ), the ability to do research for their COMM 111 assignments (15%), using the library catalog (12%), and using a search engine (2%) all increased a er the intervention of library instruction between the pre- and post-test, for all students in all status levels. Results of t-tests comparing the pre-test data with the post-test data indicated a statistically significant difference between the means of the two groups (t [∞], p<.05). Evaluation of Web Pages The following open-ended question was presented to students: “Briefly describe how you determine the quality of infor- mation provided on a Web page. What do you look for? What do you think about?” Comments from both the pre- and post- survey were coded into one of the follow- ing eleven categories: authority, accuracy, point of view/bias, reliability, timeliness, validity, design, information need, refer- http:authors.11 Continuing Assessment of Library Instruction to Undergraduates 375 TABLE 3 Changes in Importance Students Place on Research Tools When you do research, how important are the following tools? Tool Not Important Somewhat Important Important Catalog –1.2% 0.9% 0.3% Database –10.8% –6.0% 16.4% Search Engine –0.5% –3.3% 3.8% Library Staff 9.7% 3.5% –13.2% Classroom Instructor 6.8% 2.1% –9.0% ral, undefined, or other (see table 2). The top five determinants that students listed on the pre-test were authority (29%), reli- ability (16%), accuracy (13%), timeliness (10%), and validity (8%). On the post-test completed a er library instruction, the top five determinants of Web quality students cited were reliability (25%), au- thority (23%), timeliness (19%), accuracy (13%), and point of view/bias (7%). Perceptions of Research Resources When asked to identify research tools as not important, somewhat important, or important when doing research, students increased their important rating of library databases (16%), Web pages (4%), and the library catalog (<1%) between the pre- and post-test survey (see table 3). Stu- dents decreased their important rating for library staff (–13%) and classroom instructors (–9%) as research tools dur- ing this same time period. Assistance-Seeking Comfort Students were asked on both the pre- and post-test about their comfort level asking for help at the library desk and from their COMM 111 instructor when they needed research assistance for an assignment; selecting from one of three options: not comfortable, somewhat comfortable, or comfortable (see table 4). Between the pre- and post-test, students feeling not comfortable at the library declined (<1%), as did those feeling somewhat comfortable (2%), and com- fortable (2%). Students feeling not comfortable asking for help from their COMM 111 increased (5%), as did those feeling comfortable (<1%), while those feeling somewhat comfortable decreased (–6%). In two follow-up open- ended questions, students were asked to indicate what would make them feel more comfortable asking for help at both the library desk and from their COMM 111 instructors. Comments were coded into one of the following ten categories: approachable, available, awareness, comfortable already, direct, knowledgeable, nothing, privacy, stu- dent’s personality, and other. As detailed in table 5, students’ top responses on the pre-test for what would make them feel more comfortable asking for assistance at the library desk were that they were already comfortable (37%), approachable people working at the desk (13%), noth- ing would help (13%), greater availability (9%), and that their personality got in the way of their asking for assistance (8%). On the post-test, the top three responses were the same, followed by awareness TABLE 4 Changes in Student Assistance-Seeking Comfort If you needed help with your research for an assignment, how comfortable would you feel asking for help at the library desk? Not Comfortable Somewhat Comfortable Comfortable –0.5% –2.2% –1.7% If you needed help with your research for an assignment, how comfortable would you feel asking your COMM 111 instructor for help? Not Comfortable Somewhat Comfortable Comfortable 5.3% –5.7% 0.4% 376 College & Research Libraries July 2008 of services at the desk (8%), and other (7%). There were increases in students asking for awareness (2%) and ap- proachability (1%), and de- creases in students mention- ing availability (–4%), having knowledgeable people at the desk (–3%), and directness of workers in making clear they were there to help (–2%). In responding on the pre- test to the open-ended ques- tion, “What, if anything, would make you feel more comfortable asking your COMM111 instructor for help?” students indicated the following: they were already comfortable (40%), nothing would help (21%), awareness of what help the instructor could provide (9%), and greater approach- ability and availability (both 6%). In the post-test, the top two responses were the same, followed by availabil- ity (11%), other (7%), and approachability (6%). There were increases in students asking for other (5%) and availability (4%), while stu- dents’ mention of awareness (–7%), already being com- fortable (–2%), directedness, and students’ personality (both –1%) decreased (see table 5). Changes in Confidence When students were asked on the post-test whether their overall confidence to do research had changed during the semester, 54% (n = 116) indicated they had experienced a change in confidence. Those students who were both more and less confident were asked an open-ended follow-up question to solicit what factors changed their confidence in their ability to do research. The comments TABLE 5 Student Suggestions to Improve Assistance-Seeking Comfort What, in anything, would make you feel more comfortable asking for help at the library desk? Category Pre-test (mentions) Post-test (mentions) Difference Approachable 13% (22) 14% (22) 1% Available 9% (16) 6% (9) –4% Awareness 6% (10) 8% (12) 2% Comfortable (Already) 37% (63) 37% (58) <1% Direct 5% (9) 3% (5) –2% Knowledgeable 5% (9) 3% (4) –3% Nothing 13% (23) 15% (24) 12% Privacy <1% (1) 1% (2) <1% Student’s Personality 8% (13) 6% (10) –1% Other 4% (6) 7% (11) 4% What, if anything, would make you feel more comfortable asking your COMM 111 instructor for help? Category Pre-test (mentions) Post-test (mentions) Difference Approachable 6% (9) 6% (9) <1% Available 7% (10) 11% (15) 4% Awareness 9% (14) 2% (3) –7% Comfortable (Already) 40% (62) 39% (55) –2% Direct 5% (7) 4% (5) –1% Knowledgeable 5% (7) 5% (7) <1% Nothing 21% (32) 22% (31) 1% Privacy 2% (3) 1% (2) <0% Student’s Personality 5% (7) 4% (5) –1% Other 2% (3) 7% (10) 5% (n = 119) were coded into four categories: assignments, experience, library instruc- tion/knowledge, and other (see table 6). The majority of students cited the library instruction session or greater knowledge of the library as changing their confidence (60%), followed by experience (22%), as- signments (11%), and other (8%). Continuing Assessment of Library Instruction to Undergraduates 377 All students were asked on the post- test to identify changes to COMM 111 that would further increase their confidence in their research abilities. Of post-test re- spondents, 80 percent (n = 169) answered this question, providing a total of 337 re- sponses with students able to select more than one of the six options (see table 7). Of the close-ended choices provided, library instruction (42%) received the highest re- sponse level, followed by class discussion (33%), class assignments (23%), research meetings with the COMM 111 instructor (21%), research meetings with a librarian (21%), and an online tutorial (18%). Discussion Research Confidence One of the major findings in this study is the statistically significant increased overall confidence of students in their ability to complete research in prepara- tion for their assignments in COMM 111. Additionally, student confidence in using library databases, doing research for their assignments, and using the li- brary catalog—all elements of the library instruction component that occurred as part of their curriculum—increased. These results indicate that students feel be er prepared to complete research for their assignments a er the intervention of tailored library instruction and provide TABLE 7 Student-Suggested Course Changes to Increase Research Confidence Which of the following changes to COMM 111 would further increase your confidence in your research abilities? Mark all that apply. Changes to COMM 111 Percent of Total Responses to Question (responses) Library Instruction 42% (90) Class Discussion 33% (71) Class Assignments 23% (49) Research Meeting (Instructor) 21% (44) Research Meeting (Librarian) 21% (44) Online Tutorial 18% (39) TABLE 6 Factors That Changed Student Research Confidence What factors have changed your confi- dence in your ability to do research? Category Post-test (responses) Library Instruction/ Knowledge 60% (71) Experience 22% (26) Assignments 11% (13) Other 8% (9) substantive evidence to continue the model currently being used. These results also provide strong evidence in support of the collaborative design of information literacy instruction embedded into the Introduction to Public Speaking course, provided by the teaching assistants with guidance from the First-year Experience Librarian and the Communication Studies Faculty Coordinator. Evaluation of Web Pages The results from the open-ended ques- tion aimed at capturing student ability in evaluating Web pages are varied. Students increased in their use of reliability and timeliness; decreased in their reliance on authority, validity, and focus on design, yet remained relatively constant in looking for signs of accuracy, point of view, content meeting their information need, and following a referral as evaluation strategies (see table 2). At the same time, it is important to note the means of evalu- ation most prevalent and lacking. Students on both the pre- and post-tests did look to authority, accuracy, reliability, and timeliness to a large ex- tent, yet did not look at point of view/bias or 378 College & Research Libraries July 2008 validity—all areas covered by the in- struction session. These results point to the need for further analysis of student Web evaluation abilities and potentially adjusting the Web evaluation activity within the lesson. Perceptions of Research Resources Notable in the analysis is the fact that students entered the project with a high regard for Web pages but indicated that their confidence in evaluating Web pages was low (see table 3). On the pre-test, students also showed that they did not perceive library databases or the library catalog to be important and did not feel confident utilizing either for research. Students increased their view of the importance of library databases—and to a lesser extent the library catalog—and their confidence in utilizing them as re- search tools over the course of the semes- ter. Students also greatly increased their confidence in their ability to evaluate Web pages, which is critical since they value this tool highly. These findings present strong evidence for the value of the library instruction and its curricula focused on library resources and Web site evaluation. They also point to the need for further analysis of student Web evaluation abilities and potentially adjusting the Web evaluation activity within the lesson. Interestingly, the importance students placed on people as research resources— both library staff and classroom instruc- tors—decreased between the pre- and post-tests. One possible reason may be that as students increase their confidence in utilizing electronic research tools and the value they place on them, they become more independent and do not feel they need librarian or instructor support; fur- ther studies would be needed to explore this hypothesis. Assistance-Seeking Comfort Student comfort asking for assistance at the library desk went down slightly between the pre- and post-test. Drop- ping further was student comfort asking for assistance from their instructor (see table 4). This is a cause for concern and for careful analysis of the suggestions students provided when asked what would make them more comfortable (see table 5). Many students said that they were already comfortable or that nothing would help. Others wanted those they ask for help to be more ap- proachable and provide greater avail- ability. Other key suggestions include being made directly aware of the ser- vices librarians and instructors provide. These responses provide helpful insight into student perspectives and needs and offer opportunities for responding with service adjustments and session content. Changes in Confidence Increases were recorded across all re- search confidence questions. When stu- dents were asked later in the survey if their confidence had changed, 54 percent answered that it had. Those respondents cited the library lesson or learning about the library resources as the key factor in changing their confidence, followed by assignments and experience (see table 6). These open-ended responses are corrobo- rated by the outcome of the close-ended question that asked students to indicate what changes would increase their confidence (see table 7). Again, library instruction was the top choice, followed by class discussions and assignments. These results support the importance of library instruction, as students indicate that it does and can increase their research confidence. Null Hypothesis The null hypothesis was rejected for stu- dent confidence and perceptions before and a er receiving the research lesson. There was an experimentally important and consistent difference between these results. The null hypothesis for student Web evaluation abilities was accepted, as no significance was evident. Continuing Assessment of Library Instruction to Undergraduates 379 Implications for Future Research Two aspects of this study would have in- creased the predictive validity of the results and will be implemented in future studies. First, random selection of COMM 111 sec- tions would have eliminated the selection bias inherent in self-selecting participants across the entire population of students enrolled in this course. Second, although our initial research design included track- ing of individuals across time, so ware malfunction prevented directly matching changes to specific participants. However, interesting statistically significant descrip- tive trends were observed with a sufficient sample size to address the scope and intent of this research project. The tacit assumption that student research confidence in completing their research-based assignments and using library research tools correlates to their research abilities needs to be explored and evaluated with an additional or separate instrument that aims to capture learning outcomes. This is especially relevant in light of the responses on the surveys to search engine value, Web evaluation con- fidence, and Web evaluation abilities. Additionally, administering the pre- and post-test surveys to a different set of COMM 111 students, or students in another key lower-division undergradu- ate course, would provide additional insights. Another potential investiga- tion would focus on a key aspect of the research module (for example, library databases) and examine student research confidence, previous use of the tool, per- ception of the tool, learning outcomes, and citation analysis. Conclusion A primary goal of the library instruction program is to establish effective assess- ment techniques that identify learning outcomes and student success as a result of information literacy instruction. This research project provides another level of information toward this goal. First, the results can be applied to further refine the research component for COMM 111. Second, in combination with previous institutional studies, the results can be extrapolated to expand this model into other curricula. Third, the data can be shared with campus administration and used to augment the significance of information literacy instruction as a significant part of general education research. Fourth, the pre- and post-test assessment model used in this project can be further refined and repeated to substantiate learning outcomes in aca- demic libraries. This testing methodology is not an isolated example of assessment but rather part of an overall approach to embed assessment in the library instruction program. As an extension of the pre- and post-testing projects, a project is currently underway to assess learning outcomes of randomly selected first-year students and a comparable number of randomly selected students in capstone courses. Ad- ditionally, assessment of effective teach- ing is built into the instruction process through the use of an online assessment tool used at the end of teaching sessions, the implementation of the Peer Review of Teaching project initiated in fall semester 2006, and the development of teaching portfolios as part of faculty reviews. By continuing to foster and implement assessment initiatives into the library’s instructional program, we evolve our services to meet the needs of our users and, in turn, develop goals to lead our next steps. 380 College & Research Libraries July 2008 Appendix A Outline of COMM 111 Information Literacy Instruction Module Session Time: 75 minutes Accompanying Handouts: Web Page Types & Domains; Web Page Evaluation Activity; Library Self-Guided Tour Other Materials: Research Log (course pack); Web Page Evaluation: URLs (desktop/online) Outline: Library Services & Space Session Introduction & Goals • working session: find book, articles, a Web site to prepare Informative and Persuasive speeches 1 • break research topics down into keywords use library resources to find books and articles • learn about types of Web pages & focused Web searches • importance of critically evaluating information sources Housekeeping Introduce Research Log Introduce Library Web Site Look for Books: Library Catalog Using Databases to Identify Articles Finding a Known Journal/Article Group Web Evaluation Activity Wrap-up Assessment Remaining Minutes Continuing Assessment of Library Instruction to Undergraduates 381 Appendix B Pre- and Post-test Survey Assessment Questions Questions 1–10b appeared on both the pre- and post-test; questions 11–13 were only additional questions on the post-test. In COMM 111, you are required to research the topics that you choose for your speeches. 1. Indicate your student status: 1 Freshman | Sophomore | Junior | Senior | Other 1 The following questions ask you to describe how confident you feel about specific research activities. 2. Overall, how confident do you feel about your ability to do research for COMM 111 speech assignments? 1 not confident | somewhat confident | confident 1 3. How confident do you feel using the library catalog to find books on a specific topic? 1 not confident | somewhat confident | confident 1 4. How confident do you feel using a search engine such as Google or Yahoo! to find informative Web pages on a specific topic? 1 not confident | somewhat confident | confident 1 5. How confident do you feel using a library database such as Academic Search Premier or Academic Index to find articles published in newspapers, maga- zines, or scholarly journals on a specific topic? not confident | somewhat confident | confident 6. How confident do you feel about determining the quality of information pro- vided on any Web page? 1 not confident | somewhat confident | confident 1 7. Briefly describe how you determine the quality of information provided on a Web page. What do you look for? What do you think about? 8. When you do research, how important are the following tools? 1 1 = not important 2 = somewhat important 3 = important 1 • library catalog to find books 1 2 3 • library databases such as Academic Search Premier or Academic Index to find articles 1 2 3 • library staff 1 2 3 • course instructor 1 2 3 • search engines such as Google or Yahoo! to find informative Web pages 1 2 3 1 382 College & Research Libraries July 2008 The following questions ask you to describe how comfortable you feel asking for as- sistance and what, if anything, would make you feel more comfortable. 9a. If you needed help with your research for an assignment, how comfortable would you feel asking for help at the library desk? 1 not comfortable | somewhat comfortable | comfortable 1 9b. What, if anything, would make you feel more comfortable asking for help at the library desk? 10a. If you needed help with your research for an assignment, how comfortable would you feel asking your COMM 111 instructor for help? not comfortable | somewhat comfortable | comfortable 10b. What, if anything, would make you feel more comfortable asking your COMM 111 instructor for help? 11. Overall, has your confidence about your ability to do research changed since the beginning of the semester? • my confidence is the same now as at the beginning of the semester  go to 12 • I feel more confident now than at the beginning of the semester  go to 11 • I feel less confident now than at the beginning of the semester  go to 11 12. What factors have changed your confidence in your ability to do research? 13. Which of the following changes to COMM 111 would further increase your confidence in your research abilities? Mark all that apply. 1 • more than one hands-on library class about how to do effective research • an online tutorial about how to do effective research • more small class assignments to practice effective research • an individual research meeting with a librarian • an individual research meeting with your COMM 111 instructor • more class discussion about how to do effective research 1 • other changes that would increase your confidence 1 Continuing Assessment of Library Instruction to Undergraduates 383 Notes 1. Prudence W. Darlymple, “Understanding Accreditation: The Librarian’s Role in Educational Evaluation,” portal: Libraries & the Academy 1 (Jan. 2001): 23–32; Amos Lakos and Shelley Phipps, “Creating a Culture of Assessment: A Catalyst for Organizational Change,” portal: Libraries & the Academy 4 (July 2004): 345–61; Bonnie Gratch Lindauer, “Defining and Measuring the Library’s Impact on Campuswide Outcomes,” College & Research Libraries 59 (Nov. 1998): 546–70; and Sharon Lee Stewart, “Assessment for Library Instruction: The Cross/Angelo Model,” Research Strategies 16, no. 3 (1998): 165–74. 2. Elizabeth Fuseler Avery, Assessing Student Learning Outcomes for Information Literature Instruction in Academic Institutions (Chicago, Ill.: Association of College and Research Libraries, 2003); Lorrie A. Knight, “The Role of Assessment in Library User Education,” Reference Services Review 30, no. 1 (2002) 15–24; Kate Manuel, “What do First-Year Students Know About Information Research? And What Can We Teach Them?” in Currents and Convergence: Navigating the Rivers of Change, Proceedings of the ACRL 12th National Conference, ed. Hugh Thompson, 401–17 (Chicago, Ill.: Association of College and Research Libraries, 2005). Further citations to this work are given in the text; and Harvey Varnet and Martha Rice Sanders, “Asking Be er Questions: Small-scale Assessment Measures that Inform Ongoing Work,” C&RL News 66 (June 2005): 461–65. 3. Karen Markey, Annie Armstrong, Sandy de Groote, Michael Fosmire, Laura Fuderer, Kelly Garre , Helen Georgas, Linda Sharp, Cheri Smith, Michael Spaly, and Joni E. Warner, “Testing the Effectiveness of Interactive Multimedia for Library-User Education,” portal: Libraries & the Academy 5 (Oct. 2005): 527–44. 4. Denise Koufogiannakis, Jeane e Buckingham, Arif Alibhai, and David Rayner, “Impact of Librarians in First-year Medical and Dental Student Problem-based Learning (PBL) Groups: A Controlled Study,” Health Information and Libraries Journal 22 (Sept. 2005): 189–95. 5. Sue Samson and Kim Granath, “Reading, Writing, and Research: Added Value to Univer- sity First-year Experience Programs,” Reference Services Review 32, no. 2 (2004): 149–56. Further citations to this work are given in the text. 6. K. Russell Jones, Farhad Moeeni, and Paula Ruby, “Comparing Web-Based Content Deliv- ery and Instructor-Led Learning in a Telecommunications Course,” Journal of Information Systems Education 16 (Fall 2005): 265–71. 7. Elizabeth W. Carter, “‘Doing the Best You Can with What You Have’: Lessons Learned from Outcomes Assessment,” Journal of Academic Librarianship 28 (Jan.–Feb. 2002): 36–41. 8. James A. Damron, “In-Coming Provisional Students: Comparing Pre- and Post-Test Scores for a Summer Library Program,” Virginia Libraries 49 (Apr.–June 2003): 21–22. 9. Joan Kaplowitz, “A Pre and Post-test Evaluation of the English 3-library Instruction Program at UCLA,” Research Strategies 4 (Winter 1986): 11–17. 10. At the time of the study, Merinda McLure served as First-year Experience Librarian and Steven Schwarze as Communication Studies Faculty Supervisor; both were instrumental in the implementation of this project. 11. Klaus Krippendorff, Content Analysis: An Introduction to Its Methodology (Thousand Oaks, Calif.: Sage Publications, 2004).