germain.p65 The Effectiveness of Presentation Formats for Instruction 65 A Comparison of the Effectiveness of Presentation Formats for Instruction: Teaching First-Year Students Carol Anne Germain, Trudi E. Jacobson, and Sue A. Kaczor First-year experience (FYE) programs offer librarians opportunities to teach new students in a comprehensive fashion. However, large FYE programs can place demands on user education programs that are dif­ ficult to meet. Instruction librarians at the University at Albany sought to address this dilemma by developing a Web-based instructional module for one class session. The module was used by a segment of students in the Project Renaissance FYE program, whereas another segment re­ ceived instruction by a librarian. The effectiveness of the two instruc­ tional methods was compared using pre- and post-tests, and was found to be equal. Analysis of the test scores also showed that instruction, regardless of format, makes a significant difference (p < .05) in the num­ ber of correct test answers. he User Education Program at the University Library, Univer­ sity at Albany, SUNY, has worked with and supported the university’s curriculum in a variety of ways over the past two decades. Dur­ ing most of the 1980s, librarians were able to teach all students who took a required English 100 course, but when that course was eliminated during the 1987–1988 aca­ demic year, the program had to redefine itself and its role. The beginning of the 1990s was a time of great change in elec­ tronic resources. CD-ROMs were new and attractive to users, and provided a novel approach for reaching patrons. To help students learn how to use new elec­ tronic resources effectively, an electronic information class program was initiated, following the model developed at the University of Texas-Austin. To keep pace with the technological developments of the 1990s, these classes have coexisted with course-related instruction and have provided an opportunity to reach stu­ dents who did not come to the library with their classes. Although course-re­ lated instruction has been a substantial effort, the application was hit or miss de­ pending on the professor or teaching assistant’s interest. Instruction might oc­ cur anytime from freshmen year to gradu­ ate study, and librarians consistently heard from master ’s degree students on Carol Anne Germain is the Networked Resources Education Librarian at the University at Albany; e- mail: cg219@cnsvax.albany.edu. Trudi E. Jacobson is the Coordinator of User Education Programs at the University at Albany; e-mail: tj662@cnsvax.albany.edu. Sue A. Kaczor is the Science Bibliographer at the University at Albany; e-mail: skaczor@cnsvax.albany.edu. 65 mailto:skaczor@cnsvax.albany.edu mailto:tj662@cnsvax.albany.edu mailto:cg219@cnsvax.albany.edu 66 College & Research Libraries January 2000 the brink of graduating that they needed these research skills earlier in their aca­ demic program. In 1996, the University at Albany initi­ ated a first-year experience (FYE) pro­ gram, Project Renaissance, to address stu­ dent desire for a smaller environment within the larger university setting. Project Renaissance was conceived of as a living/learning environment. Students take six credit hours of Project Renais­ sance core courses each semester, and dis­ cussion class sections are divided by dor­ mitory floor so that students who live to­ gether take classes together. The students have closer contact with faculty members, learn technology skills in a concentrated way, and receive instruction from librar­ ians on the basics of doing research. The program began with 200 students in 1996– 1997, doubled the following year, and then was expected to increase by almost 200 more during 1998–1999. In the end, 1998–1999 enrollment was 523. Project Renaissance students are typically seven­ teen or eighteen years old. An interactive Internet-accessible tutorial was developed, due to the ubiquity and popularity of this medium among the students. Each summer, the project librarian is involved in the two-week orientation for new faculty and teaching assistants in or­ der to integrate user education into the program. During that time, she or he gives a presentation on the advantages of library instruction, the possible sessions that might be taught, and methods to integrate stu­ dent learning with coursework. The pro­ gram, and the university library’s involve­ ment with it, has changed over the three years of its existence. Initially, the project librarian and other librarians provided in­ struction upon request, based on the needs of individual teaching assistants (who taught the small discussion sections). This instruction reached many of the sections, but unevenly. The teaching assistants for a handful of sections might request several instructional sessions, whereas others would request only one and a few would forego instruction entirely. It became very difficult to schedule classes because many teaching assistants desired sessions on the same dates. When the program planned to expand to 600 students, the librarians involved and the program director felt that the students needed a common learning experience. The project librarian developed two classes for all students: one initial class focusing on information formats, the online catalog, and electronic databases; the other centering on the evaluation of information resources, particularly infor­ mation found on the Internet. Motivation for the Study The increased enrollment in Project Re­ naissance also prompted the librarians involved to address instructional format. Prior to the 1998–1999 semesters, all li­ brary instruction for Project Renaissance sections was delivered in the traditional in-class manner. Limited resources, in­ cluding librarians and classroom space, became major issues. It was felt that other instructional formats might provide a more consistent level of instruction as well as ease resource demands. The authors considered less conven­ tional teaching formats as possible solu­ tions. Most were electronic-based tools, including e-mail and Web pages. An in­ teractive Internet-accessible tutorial was developed, due to the ubiquity and popu­ larity of this medium among the students. Because Web-based teaching tools are in their infancy, this tutorial needed to be evaluated carefully. The structure of Project Renaissance classes made easy evaluation possible. Ultimately, the au­ thors hoped to find a solution that would address the needs not only of Project Re­ naissance students, but also of all seg­ ments of the university population, in­ cluding distance learners and returning students. Web versus Live Instruction During the fall 1998 semester, all students enrolled in Project Renaissance were scheduled for library instruction sessions. The Effectiveness of Presentation Formats for Instruction 67 Two methods of instruction were used for the initial class, focusing on information formats. One was traditional in-class in­ struction consisting of a lecture accompa­ nied by hands-on computer experience. The second was the Web-based interac­ tive tutorial, which each student navi­ gated individually from computers lo­ cated in the library’s computer classroom. This tutorial was developed by the first two authors and can be found online at http://www.albany.edu/~libclass/. The tuto­ rial contained fifty-three main pages with many containing links to additional in­ formation. Blocks of pages provided con­ cept instruction. At the end of each block were pages dedicated to questions that related to the subject matter. When a stu­ dent chose an incorrect selection, guid­ ance was provided in making a correct choice and the student was linked back to the question page. When the appropri­ ate answer was selected, the user was linked to the next page of the tutorial. Both instructional formats covered the same material, including: why and when to use specific sources (e.g., newspapers, books); accessing and searching the uni­ versity libraries’ online catalog; library re­ sources such as interlibrary loan (ILL); ac­ cessing and searching the library’s elec­ tronic databases; Boolean operators; and search strategies. Instruction took place during students’ scheduled fifty-five minute class session. Students in the Web- based session were permitted to leave class when they had completed the tutorial, which took from fifteen to fifty-five min­ utes to finish. The traditional class filled the entire period. Opportunities for ask­ ing questions were provided in both class formats. Those questions asked at the live session were answered in class, whereas students using the Web-based tutorial sub­ mitted an electronic form and their ques­ tions were addressed at their follow-up session. Students were encouraged to in­ clude their e-mail address with their ques­ tions for a personal response. Instruction aids, such as handouts, were provided to students in print or electronic form, de­ pending on their version of the instruction. To measure the outcome of the differ­ ent instruction formats, students who were enrolled in the basic Project Renais­ sance curriculum were studied. Because each teaching assistant taught two sec­ tions of students, one section received live instruction and the other received the Web-based tutorial. The authors believed this division would minimize differences between target populations. Literature Review Library instruction for college students is an essential component of a rich academic experience. Students need “some method of formal [library] instruction.”1 “As li­ braries become more complex and as more information is available with faster and easier access, it appears the future of library instruction is becoming more im­ portant than ever.”2 Tremendous changes in technology are providing new formats for delivering li­ brary instruction. Computer-assisted in­ struction (CAI), also referred to as com­ puter-assisted learning (CAL), is the use of computers for learning and instruc­ tion.3 Styles of CAI that have been ex­ plored over the past decade include e- mail, hypertext tutorials, and listservs. CAI is attractive because “it is interactive, self-paced, and self-directed.” It delivers consistent content to all students without the variation that occurs when numerous instructors teach.4 In Growing Up Digital, Don Tapscott states that “CAI programs can improve learning performance by one-third,” making CAI advantageous for user education programs.5 Computer-assisted instruction in user education programs can have various out­ comes. A computer-assisted program de­ signed to replace the traditional library tour was implemented and evaluated at Central Missouri State University. Results from pre- and posttests showed that us­ ing the CAI model produced a signifi­ cantly higher posttest mean (p < .0001).6 In 1984, researchers at the University of Wisconsin-Stout Library Learning Center developed a one-hour CAI program. When implemented and tested a year http://www.albany.edu/~libclass 68 College & Research Libraries January 2000 later, pre- and posttest results found that students who received librarian instruc­ tion performed significantly better (p < .05) on the posttest than students who had used the CAI program.7 Two recent stud­ ies, comparing traditional library instruc­ tion with CAI tutorials, concluded that there was no significant difference be­ tween traditional instruction and CAI for­ mats.8,9 At Western Michigan University, librarians developed a HyperCard mul­ timedia tutorial designed to introduce li­ brary services and teach basic library skills. The researchers concluded that the It is possible that some of today’s students find computer-based instruction more effective or more comfortable. students who used the tutorial learned basic library skills as well as those who had received the traditional form of in­ struction.10 Librarians at UCLA’s Louise M. Darling Biomedical Library compared instruction provided via a computer- based module and that presented in lec­ ture format. “Students’ post-test re­ sponses seemed equivalent regardless” of instructional format.11 Because previous studies were not unanimous in their find­ ings, the authors felt that additional re­ search was warranted. Methodology The authors developed a test to use with all sections involved in the study.12 The test was administered at the beginning of the first library session, before students received either form of instruction. The instruments were color-coded to indicate which were from sections receiving Web- based instruction (denoted as “Web”) and which were from sections receiving live instruction (denoted as “live”). The test included four matching and eight mul­ tiple-choice questions that involved the basics of using the library (name of the online catalog, what is needed to check out materials), its OPAC, and entering a search in a database. It also included two questions about previous library instruc­ tion, either in high school or at the uni­ versity. The same instrument was used as a posttest and administered at the begin­ ning of the second session in the library. It was given only to those students who also had completed a pretest. For confi­ dentiality reasons, the authors were un­ able to match the pre- and posttests for each student, which required analysis of the data in the aggregate. Due to the na­ ture of the scheduling of the sections, times between the first instruction session and the posttest varied, ranging from one and a half weeks to six weeks. The result­ ing data were analyzed using SPSS 8.0 for Windows. Hypotheses The authors felt strongly that the results from the data would show that : • Students receiving traditional in­ struction would learn more than those re­ ceiving the Web-based tutorial. • Regardless of instructional format, students receiving library instruction would know how to use library resources better after the instruction. • Students who had library instruc­ tion in high school would know more about using library resources than those who had not. After the pre- and posttests were col­ lected and coded, other questions were proposed that the authors felt the data could answer. These included: • Did students who performed well with certain questions have prior knowl­ edge of the material? • Is one form of instruction more ef­ fective than the other (i.e., searching by keyword) with specific questions? • Is there a relationship between cor­ rect and incorrect answers on two related questions dealing with keywords? In other words, were students guessing? Effectiveness of Both Types of Instruction A total of 303 pretests and 284 posttests were collected for evaluation. Of these, 160 pretests were obtained from stu­ http:study.12 http:format.11 http:struction.10 The Effectiveness of Presentation Formats for Instruction 69 dents who would go on to have Web- based instruction and 143 from those who would be taught by a librarian. Students who took the Web tutorial completed 157 posttests and those who took live instruction completed 127. The average number (mean) of correct responses on the pretests was 7.91 for the Web group and 7.72 for the live group. The average number of correct responses on the posttest was 9.07 for the Web group and 8.67 for the live group (see table 1). When analyzed us­ ing a one-way analysis of variance (ANOVA), a significant difference was found among the four groups (p < .05). Analyses to locate the source of this dif­ ference showed that there was no dif­ ference between the mean number of correct responses of the two pretest groups (p = .335) or, more important, between the two posttest groups (p = .053). From this, the conclusion could be drawn that there is no difference in the effectiveness of the two types of instruction, Web and live, based on number of correct answers. The authors were surprised to find this to be the case. The research hypothesis was that traditional instruction would be more effective. Of course, this may be due to personal bias and experience: the au­ thors have invested a great deal of time and effort in classroom teaching and con­ stantly strive to increase the effectiveness of classes. In addition, this is the way the authors learned as students. It is possible that some of today’s students find com­ puter-based instruction more effective or more comfortable. Or they may have been responding to the novelty of the medium in this session. Although the Project Re­ naissance program consciously intro­ duces students to a wide variety of com­ puter-based systems and tools, classes are not taught through Web-based modules or computer-aided programs. Further analysis also showed a signifi­ cant difference (p < .05) between the mean number of correct answers for the two pretest groups and the two posttest groups. Library instruction did indeed make a difference. Investigating further, the authors tested to see if this also held true if the results were disaggregated based on instructional formats. The re­ sults showed that there was a significant difference (p < .05) between the pretest Web group (mean of 7.91) and the posttest Web group (mean of 9.07). There also was a significant difference (p < .05) between the pretest live group (mean of 7.72) and the posttest live group (mean of 8.67). Thus, library instruction, regardless of instructional format, was shown to in­ crease the number of correct scores be­ tween the pre- and posttests. Effect of High School Instruction The authors hypothesized that students who had library instruction in high school would know more about using library resources than those who had not. To ad­ dress this, data were analyzed that com­ pared high school instruction (or lack thereof) for all pretests, regardless of in­ structional format (see table 2). Using a t- test, it was determined that there was a significant difference (p < .05) in mean number of correct answers. High school TABLE 1 Mean Number of Correct Answers, by Instructional Format Test N Mean # of Correct Answers Standard Deviation Pretest Web 160 7.9125 1.7023 Pretest live 143 7.7203 1.7539 Posttest web 157 9.0701 1.6057 Posttest live 127 8.6693 1.8816 Total 587 70 College & Research Libraries January 2000 TABLE 2 Instruction during High School (Pretest Results) Yes No Test Pretest N Mean (Std. Deviation) 215 8.02 (1.7265) N 82 Mean (Std. Deviation) 7.40 (1.6610) instruction was effective, and high school librarians should be encouraged to con­ tinue to deliver it. Other Observations When analyzing the data, the authors noticed that two questions were answered correctly by a very high percentage of all students, even on the pretest. One ques­ tion asking what identification students needed to check out books was answered correctly by 95.7 percent on the pretest and by 99.6 percent on the posttest. Project Renaissance students attend the same summer orientation program that all Uni­ versity at Albany freshmen do, and as a part of that orientation they are registered for SUNYCards, an identification card that is used not only for checking out li­ brary materials, but also for meal plans, building access, and more. The SUNYCard plays an important role in stu­ dents’ lives, and they became acquainted with it early on. The high percentage of correct answers to the second question, which asked whether the library would obtain mate­ rials it does not own, was less easy to ex­ plain. Although 95 percent answered this correctly on the pretest and 100 percent on the posttest, ILL does not play a sub­ stantial role in student orientation outside the library. It is possible that students cor­ rectly recognized that this question would not be included on the instrument if the library did not intend to get materials it does not own. One of the questions the authors were interested in having answered was whether one form of instruction was more effective in teaching the benefit of key­ word searches. In almost all cases, there was a decrease in the percentage of wrong answers from pretest to posttest when looking at the data by instructional for­ mat (see table 3). This was also reflected in the correct answer, “You don’t need complete information about the item.” There, the percentage of correct answers increased in both instructional formats, going from 40.3 percent of the answers (pretest live) to 50.4 percent (posttest live). Even more striking was the increase from 40.5 percent (pretest Web) to 74.5 percent TABLE 3 Comparison Between Benefit of Keyword Answer and Test Version Process *Don't need Items Get more complete available to thousands Test quickly information check out of items Total Pretest Web 61 (38.6%) 64 (40.5%) 17 (10.8%) 16 (10.1%) 158 Pretest live 60 (43.2%) 56 (40.3%) 10 ( 7.2%) 13 ( 9.4%) 139 Posttest Web 23 (14.6%) 117 (74.5%) 8 ( 5.1%) 9 ( 5.7%) 157 Posttest live 42 (33.6%) 63 (50.4%) 8 ( 6.4%) 12 ( 9.6%) 125 Total 186 300 43 50 Percentages are given for rows * Correct answer The Effectiveness of Presentation Formats for Instruction 71 (posttest Web), an increase of 34 percent. Looking at the answers to other questions, a similar trend was not observable. Clearly, the Web-based tutorial was more effective in teaching the benefits of key­ word searching. The authors theorize that more emphasis was placed on this con­ cept, or it was presented more effectively, in the Web tutorial. Was there a relationship between cor­ rect and incorrect answers to the two questions on keyword searching? The aggregated data from the two posttests showed that when asked, “What is the best way to enter a search topic in a data­ base,” 257 (92.1%) of the students cor­ rectly chose “Use keywords.” Only 7.9 percent chose “Describe it in a short sen­ tence,” the incorrect answer. Looking at the second, related ques­ tion, “What would you use as a search statement to find articles on the effect of distance education on college enroll­ ment,” the data continued to show a strong trend. Indeed, 177 (68.9%) of the students who chose the previous correct answer went on to choose the true key­ word statement, “Distance education and college enrollment.” In comparison, 5.8 percent chose the “or” statement, whereas 14.0 percent chose a sentence answer and 11.3 percent chose yet another option (keywords, but without Boolean opera­ tors). Of the people who picked “Describe it in a short sentence,” 54.5 percent went on to select the option that actually was in the form of a sentence. These numbers clearly show that the students were not guessing. Conclusions Based on the results of this study, the au­ thors have decided to use the Web-based tutorial for the initial instruction session for all Project Renaissance students dur­ ing the 1999–2000 academic year. How­ ever, based on the experience, the authors do not plan to assign students to complete the tutorial independently. Instruction li­ brarians will work with the Project Re­ naissance administration to ensure that teaching assistants set one class period aside during which students will work on the tutorial, much as they did during 1998–1999 when librarians monitored the session. Computer classrooms are avail­ able for this program, which will make this structured approach possible. The university library also will make the tutorial available through the user edu­ cation portion of the university libraries’ Web page and will publicize its accessibil­ ity to the campus community at large and, in particular, to the distance learning co­ ordinator. Now that the module has been shown to be effective, the authors would like it to be more widely used. Although the tutorial saves librarians’ time in that there will be fewer basic classes to teach, it will require extensive updating as changes occur, especially with the move to a Web-based catalog in the near future. However, the results of the study reassure the authors that this is time well spent.13 One concern about Web-based instruc­ tion is that students do not get a chance to meet a librarian in person. Often stu­ dents seek out a particular librarian at the reference desk because they recognize her or him from a class. Will these students fail to ask their questions if they do not see an inviting face? This problem is miti­ gated, with Project Renaissance, in that librarians will still teach the second in­ structional session. It would be interest­ ing to see research results from studies that look at both the effectiveness of Web- based instruction and student attitudes toward asking reference questions. Notes 1. Anthony Stampatoplos and Robert Mackoy, “Effects of Library Instruction on University Students’ Satisfaction with the Library: A Longitudinal Study,” College & Research Libraries 59 (July 1998): 323–34. 2. Verlane J. Herrington, “Way beyond BI: A Look to the Future,” Journal of Academic http:spent.13 72 College & Research Libraries January 2000 Librarianship 24 (Sept. 1998): 381–87. 3. Torsten Husen and T. Neville Postlethwaite, The International Encyclopedia of Education, 2nd ed. (New York: Pergamon, 1994), 988. 4. Joan Kaplowitz and Janice Contini, “Computer-Assisted Instruction: Is It an Option for Bibliographic Instruction in Large Undergraduate Survey Classes?,” College & Research Libraries 59 (Jan. 1998): 19–27. 5. Don Tapscott, Growing Up Digital: The Rise of the Net Generation (New York: McGraw-Hill), 140. 6. V. Lonnie Lawson, “Using a Computer-Assisted Instruction Program to Replace the Tra­ ditional Library Tour: An Experimental Design,” RQ 29 (fall 1989): 71–79. 7. Denise Madland and Marian A. Smith, “Computer-Assisted Instruction for Teaching Con­ ceptual Library Skills to Remedial Students,” Research Strategies 6 (spring 1988): 52–64. 8. Kaplowitz and Contini, “Computer-Assisted Instruction,” 22–23. 9. Patricia F. Vander Meer and Galen E. Rike, “Multimedia: Meeting the Demand for User Education with a Self-Instructional Tutorial,” Research Strategies 14 (summer 1996): 145–58. 10. Ibid., 153–54. 11. Kaplowitz and Contini, “Computer-Assisted Instruction,” 22. 12. One-third of the Project Renaissance sections are linked to course-specific subjects, such as anthropology. Students enrolled in these “linked” sections attend one (rather than two) Project Renaissance core course, as well as an additional class together in their particular field of study. The study did not include these linked sections because of their different courses of study. All students enrolled in the subject-specific “linked” curriculum received in-class instruction. Those readers interested in obtaining a copy of the test may contact Carol Anne Germain. 13. The web-based tutorial was updated for the 1999–2000 academic year. The new URL is http://www.albany.edu/library/divs/usered/tut/. http://www.albany.edu/library/divs/usered/tut