College and Research Libraries Evaluating a Library User Education Program: A Decade of Experience Virginia Tiefel The impact of library instruction on student learning and attitudes has been regularly evalu- ated in the Ohio State University freshman program. Using the pre/posttest method, the eval- uations have measured both cognitive and affective learning. Evaluations document that li- brary instruction produces significant i'!!J'rovement in students' knowledge about libraries, their abili to use librari s and their attitudes toward librarians. The importance of planning reliable and valid evaluations is stressed with recommendations for implementing an evalua- tion. Common student misperceptions about libraries are identified and the importance of li- brarians' participation is noted. ow does a library user educa- ~-bon program measure its effec- tiveness? Does library user edu- cation make a difference in students' cognitive and affective learning of skills? Can evaluation successfully de- termine if students have learned or not? Can a program of long standing (ten years) sustain an ongoing program of evaluation and continue to improve? What does evaluation reveal about a pro- gram and is evaluation worth the expendi- ture of effort and resources? What can be learned from evaluation about freshman students' misperceptions about libraries (at times fostered in their textbooks) and problems with libraries?1 This article focuses on the evaluation of the freshman portion of the library user e ucation program at Ohio State Univer- sity. The hypothesis is that instruction tliat emphasizes a for-credit library assign- ment offered by librarians can account for a statistically significant improvement in students' knowledge about the library, in their skills in using the library, and in in- culcation of more positive attitudes about libraries and librarians. All of these out- comes are, of course, goals or objectives of many library user education programs. More specifically, it is hypothesized that increased knowledge about libraries is re- lated to student understanding of the con- cept of search strategy, of appropriate uses of the Library of Congress Subject Head- ings (LCSH), of the library's catalogs, and of related special services. Specific stu- dent abilities include those · of finding in- formation in journal indexes, interpreting journal index citations, and indentifying complete call numbers. Also examined is the assumption that students' attitudes toward libraries, librarians, the concept of search strategies, the library's catalog, and special services can be improved as a result of instruction. Some studies have indicated that it is easier to demonstrate that library instruction brings about an im- provement in cognitive skills than it is to show a positive change in attitude about libraries and librarians. DEVELOPMENT OF THE LIBRARY INSTRUCTION PROGRAM AT OHIO STATE The Library Instruction Program (LIP) Virginia Tiefel is Director of Library User Education at the Ohio State University Libraries, Columbus, Ohio 43210. 249 250 College & Research Libraries was developed in 1978 at Ohio State. Li- brary user education program is the broad term that encompasses all_ instruction of- fe~d by liorary personnel; LIP applies specifically to the segment of the program for freshman students. Ohio State's small corps of undergraduate librarians planned, implemented, and have regu- larly evaluated LIP, which is designed to reach every entering freshman and trans- fer sophomore student (approximately eight to ten thousand in recent years) with an in-class presentation by a librarian and a for-credit library assignment. Since its beginning, the program has reached ap- proximately 85,000 students, enough to fill cavernous Ohio Stadium! Two articles published in the Journal of Academic Librarianship GAL) in 1981-82 de- scribe the program's initial development and early evaluations. 2 This article will fo- cus on evaluation of the program since tnat tiffie, with particular emphasis- on evaluations performed in 1985 and 1986. The results of the evaluations provide some useful insights for instruction; the evaluation of LIP can be adapted by other librarians who want to assess students' ac- quisition of basic skills and attitude change. LIP functions through a one credit hour course, University Survey (UVC100), of- fered by University College, the initial aca- demic "home" of almost all undergradu- ate students. Most incoming freshman and transfer students are required to en- roll through University College and all take this course. The College has an exten- sive advisement program based in twenty academic specialties, termed CAP (Curric- ular Academic Program) areas. Students who are unsure of their future major area of study are placed in the General Bacca- laureate Curriculum (GBC) area, where they engage in intensive academic and ca- reer exploration. PURPOSE OF THE EVALUATION Evaluations of LIP have had two pur- poses: to de_!ermine how well the pro- gt"am' s goals and objectives are being met and to suggest changes in methods of in- struction to help the program better meet March 1989 those goals and objectives. Thomas Kirk has identified five essential parts of a com- prehensive evaluation of a bibliographic instruction program. They are (1) content (what was learned), (2) w- due (the results), (3) £!OCeSS (how the results were obtained), (4) changes in at itude toward the libraries 'and the librarians, ana (5) over c:i'ij)ects of the program. 3 There is much emphasis in the library instruction literature on the importance of evaluating not only for information reten- tion but also for demonstrated mastery of concepts, changes in students' library be- havior, and ability of students to transfer learning to meet other information needs. The broad goals and objectives of the li- brary user education program at Ohio State are consistent with this approach to evaluation: they are to teach both skills and concepts and to ensure the applicabil- ity and transferability of those concepts and skills to other information needs, i.e., to prepare students for lifelong learning. The freshman program objectives, how- ever, because of the limitations imposed by only one library-related class period and one assignment, focus on teaching very basic skills. (Anticipated changes in the course syllabus in 1989 will provide opportunity for a second library assign- ment and more instruction.) This teaching of basic skills does support an important objective of the overall user education pro- gram: to furnish students with a base of common library skills and knowledge for subsequent library instruction, thus ena- bling librarians to begin any course- related instruction at a more advanced level. Although it is not possible to test the program at the level of concept mastery and transferability, it has always been deemed important to determine how ef- fectively the program is teaching some of the basic skills that are necessary to achieve the desired changes in library use by the students. The evaluation is of the formative type, and results have been used primarily to improve the program and demonstrate effectiveness. Most of the summative evaluation of LIP is more subjective, i.e., in the observation by li- brary staff and librarians of increased use ·of the library and of more complex materi- als. PREVIOUS EVALUATIONS OF LIP The evaluation done in 1978-79 andre- ported in the 1982 article described a ques- tionnaire that was designed to ascertain what level of previous library instruction and experience using libraries was present in the population, obtain reactions to the program's lectures and materials, assess to what extent students retained the con- tent of the instruction, and gauge student attitudes toward libraries and librarians generally. The findings reported in the ar- ticle substantiate that the evaluation ac- complished its objectives. In analyzing the outcomes of subse- quent evaluations, especially those done between 1980 and 1984, certain problems become apparent. For example, in the fall 1980 term only posttests consisting of ten multiple-choice questions were given to 171 students, or 2% of the population of 8,466 students; and in the 1981 winter and fall terms, pre- and posttests were not matched student-by-student. In winter and spring terms 1982, two methods of in- struction, videotape and lecture, were evaluated. Pre- and posttests were given during the same class period to 56% of 1,138 students. Average pretest and post- test success scores (unmatched by individ- ual student) were: 42.3% vs. 77% (in the videotape group); 43.2% vs. 78.3% (in the lecture group); 36.5% vs. 36% (in the con- trol group). In the fall terms of 1982 and 1983, testing of small random samples of students was done before and after a class- room presentation that featured a video/movie and after completion of the for-credit exercise. Unmatched test results of the 1983 evaluation yielded a 64.8% mean success score on the pretest and 74.3% on the posttest. In the fall term 1984, two different evalu- ations were conducted-one the familiar pre/posttest, the other a survey question- naire of student attitudes about the ''Bat- tle of the Library Superstars,'' the video/ movie used in the classroom presentation. The unmatched tests of a 2.7% sampling show a mean success score on the pretest Library User Education Program 251 of 64.5% and 74.3% on the posttest. The survey of student attitudes toward the vi- deo/movie was continued during winter and spring terms 1985. It was an open- ended questionnaire asking students their opinions about aspects of both the vi- deo/movie and exercise assignment seg- ments of the program. Only selected com- ments are available from the fall term questionnaire; the winter and spring sur- veys were designed to produce more quantifiable results. Of a total of approxi- mately 1,000 responses, 39% of the stu- dents said that the video/film was en- tertaining/interesting, 22% found it edu- cational/informative, 22% were generally positive, 14% found it insulting/silly, and 3% said it was outdated. IMPROVING THE VALIDITY AND RELIABILITY OF EVALUATION Obvious problems with the evaluations from these years were no consistent pre- testing, pre- and posttests unmatched by student, untested questions, and an in- adequate number of students in samples taken. While data from these evaluations provided useful indicators about the di- rection of the program and its effective- ness, they cannot be termed scientifically valid or reliable. More rigorous evalua- tions were needed. To rectify this prob- lem, a specialist in evaluation was enlisted (as had been done for the construction of the evaluations described in the earlier JAL article) to assist in designing an im- proved program of evaluation. The Uni- versity's Center for Teaching Excellence provided an evaluation consultant who helped define the study, develop instru- ments, and collect and analyze data. Her contribution to the usefulness of the eval- uations done in 1985 and 1986 was invalu- able. DESIGN In the newly conceived evaluation for the fall 1985 term, students were given identical pre- and posttests designed to measure the effectiveness of the program in achieving its stated objective.4 They in- cluded teaching the definition and de- scription of the search strategy concept; 252 College & Research Libraries the use of encyclopedias, periodical in- dexes, LC subject headings, and complete call number; and some basic information about Ohio State's online catalog and li- brc,rry system. A primary objective of the instruction was to encourage students to be more comfortable about using a large and complex library system. Pre/posttests were administered to over a thousand students in five different groups of approximately equal size. 5 Of these, 337 students were matched in pre/ posttests which were administered about two to three weeks apart. The results of a correlated group's t-test indicated an aver- age increase of 9.3% between the pre- and posttests, from a mean success score of 72.3% to 81.6% (t = 10.94, df = 673, p- > .0001). The 1986 evaluation was planned along the same lines as the 1985 version. Pre/ posttests were identical to those used the previous year. However, a larger, more representative sampling of students was desired. To achieve this goal, it was de- cided to plan on a large sample, 25% of the approximately 7,400 freshman students enrolled in fall term, recognizing that the complexity and size of the effort would result ultimately in a smaller sample. Two major factors complicated the se- lection of the sample: the dispersal of stu- dents over twenty CAP areas (and six hon- ors groups) and the need to test the effectiveness of two different library as- signments. Also, scheduling and adminis- tering the pre- and posttests involved a comparatively large number of librarians and University College personnel (some forty to forty-five people). Because it was deemed essential that no group of stu- dents be deprived of instruction, no con- trol group was planned. Two different assignments were being used in the program because one was be- ing pilot tested as a possible replacement for the approach used in previous years. The older assignment was a short-answer, basic exercise written in the form of a search strategy: it required the use of ref- erence materials and Ohio State's online catalog. The newer assignment being pilot tested was designed to challenge students in a more advanced approach: it required March 1989 them to select an editorial from a newspa- per, research the topic in standard sources and journals, and write a brief analysis of the topic. Like the basic assignment, it ap- plied the search strategy concept and the use of the online catalog. Posttests were administered two to four weeks after the pretest and as close to the completion of the assignment as class scheduling would permit. To determine if the pretest had any effect on the posttest results, three classes that were doing the old assignment (GBC, Business, Arts & Sciences) were not given the pretest. Be- cause of unforeseen difficulties with scheduling, posttests were not adminis- tered to one class in Developmental Edu- cation or to the Pharmacy and Dental Hy- . giene students. PROCEDURES All tests were machine scored, and tore- duce errors were precoded to indicate CAP area, pre- or posttest status, and whether the new or old assignment was at issue. Students were asked to put their student identification numbers on all sheets and to indicate on the posttests whether they had done the assignment or not. With approximately thirty instructors and fifteen librarians involved, testing done in a number of different classrooms, the existence of two different tests, the completion of an assignment and presen- tation scheduled between the tests, and the dictates of a rigid schedule, the logis- tics of doing the evaluation were complex. THE RESULTS Although considerable thought and ef- fort were expended to ensure the validity of the evaluations, some of the results ob- tained are subject to differing interpreta- tion. The total number of tests (1, 702) scored represented about 24% of the Uni- versity College student body, with 862 matched tests representing 12.3% of the student body from nine of the twenty CAP areas. Across CAP areas, the mean success score on the pretest for the 851 matched tests was 69.43%. The posttest mean suc- cess score was 76.64%. This improvement of 7.21% is statistically significant at a 0.001 level. The following analysis of results by CAP area, by posttest only, by honors versus nonhonors, by type of as- signment, and finally by each of the ten test questions provides a more detailed view of the evaluation results. Curricular Academic Program (CAP) By CAP area, the Nursing group (n = 71) showed the greatest improvement (9 .1% ), with a pretest mean score of 69.3% and a posttest of 78.4%. Allied Medicine was second with an 8.3% increase, and Arts & Sciences followed closely with an 8.2% in- crease. The Social Work group (26) showed the smallest increase (6.2%), from 66.9% to 73.1%. Business and General Baccalaureate followed closely with in- creases of 6.4% each, and Developmental Education scored a 6.6% gain. Dentistry registered the highest posttest score (79.2%); Business registered both the low- est pre- and posttest scores (66.5% vs. 72.9%). Posttest Only The three classes that were given the posttest only achieved a significantly lower posttest mean ( 69.7%) than did any of the matched groups. 6 One possible ex- planation, of course, is that the pretest fo- cuses students' attention on some of the key points of the presentation and assign- ment and thus increases their awareness. In effect, a pretest may serve as an ad- vanced organizer of content, or it may be that students simply learn from taking the pretest. Honors Of the 851 matched tests, 109 were from honors students. Their mean success score on the pretest was 76% with a post- testmeanof82.5%. The +6.5% difference is statistically significant. The nonhonors students' (as a group) mean success score on the pretest was 68.7% and 76% on the posttest, with a difference of +7.3%. While performance change from pretest to posttest was statistically significant for both groups, the difference in actual achievement between the two groupings was not. Library User Education Program 253 Two Assignments Of the 862 matched tests, 369 students had done the new assignment. Their mean success scores were 71.1% on the pretest and 79% on the posttest. This rep- resents a positive difference of 7. 9% and is significant at a .0001 level. The mean scores for those doing the old assignment (486) were 68.6% on the pretest and 75.3% on the posttest with an increase of 6.7%. The scores reflect significant improve- ment by students who undertook either assignment but again no significant dif- ference in student achievement between the two assignments. This may be ex- plained by the design of the testing instru- ment, which was to measure acquisition of the more basic skills that were the focus of the old assignment and not the more advanced skills of the new assignment. Questions An analysis by test question revealed some weaknesses in the testing instru- ment (see appendix A). For example, questions 6, 7, and 8 asked students to identify the parts of a journal citation, a skill which few freshman students seemed to have acquired. However, the pretest scores indicated that most stu- dents already knew how to identify the date, title, and volume number of a cita- tion. Closer examination of the question revealed that the example used may have enabled the students, even if unsure, to guess correctly by process of elimination. Despite this weakness, the test's ten con- tent and five attitude questions did pro- vide some good insights. Question 1, which covered the term ''search strategy,'' revealed an increase of 5.9% (from51.8% to57.7%) in the number of correct answers by students (see figure 1). The number and choice of incorrect an- swers indicated that many students were still not clear about the first step of the search strategy. The second question fo- cused on the Library of Congress Subject Headings (LCSH) and reflected the largest increase in performance; 38.6% of the stu- dents were correct on pretest and 61% cor- rect on the posttest, for an increase of 22.4%. Many students, however, were still confusing the Library of Congress as 254 College & Research Libraries March 1989 AVERAGE QUESTION 1 SEARCH STRATEGY QUESTION 2 LCSH QUESTION 3 MAGAZINE ARTICLE QUESTION 4 CHECK TITLE QUESTION 5 CALL NUMBER QUESTION 6 CITATION- DATE QUESTION 7 CITATION- TITLE ·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·•· 4,.. .....--- . PRE , ........................ if ·~:a ..___.-1221 POST . 76.64 I PRE r·:-:.;.;.: ... ;.; ... ; ... ;.;.;.;.: .. ·.·)$1:t-l--5"9 POST . . . . . . . . . . . . . . . . . . . '57.71 ~~~Tr L-·_: -:_-: ·_: -:_-:_· :-_: <_· :-_:-JB-:6_-:_, --~-'-·0~12. 4 PRE I ............................... '?!!:_ ..... , POST ..................... ~ .................................. ; l·.·;·y·c:, -8.2 PRE 1-: · :-:-:-:-:-:-:-: · :-:-:-:-:-: · :-:-:-:-s2:sj ¥' 8.4 POST_ 60.9l ~~~T r-:-:-:-:-:-:-:.:-:-:-:-:. 3l!: 9·1 48.!f 9.8 1.7 ~~~TF .--........ :::: ........ ;:; ........ :;:; ......... :;:; .......... :;: ....... ;:;: ........ ;:;: ........ ;:; ........ :;:; ......... :;:; ....... :;: ....... ;:;: ........ ;:;: ........ ;:;:-;:; ........ :;:; ........... :;:; .......... :;: ......... ;:;: ........... ;:;:...-;:oo::~~::r-::I·.~:"T'"':~! PRE 1:;:;:;:;:;:;:;::: ·:; ;: :; :; :::::::::::::::::::::::::::::::::::::::::::: ;87-~1 j, 4 " 2 POST_ 9t.2l · 4.9 OUEST ION 8 PRE .-1.-.-. -.. -.-.-. -.. -.-. -.. -.-.-. -.. -.-. -. -.. -.-. -.. -.-. -. -.. -. -. -.. -9--2-.4_,1 .L . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ... CITATION- VOLUME POST _.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.· .. ·~7.3 I QUESTION 9 PRE t:·:·>:. >:-:·:·>:<-:.;.; .;.; < ·:·> ;. >:.; · :.:·:·: · ;. ;. : ·: ·: ·: · ;. :s6.0:1 _/ 2'0 LCS (ONLINE CATALOG) POST .................................. SS~O i ~::!~E1 gTR. ~~~T r.;.:.:.:.:.:.:.:.:.;.:.;.:.:.;.;.;.:.:.;.;.: .; . :. :. :. :. 7l :4j 9 ~ 2 0.0 FIGURE 1 Individual Questions: 1986 Mean Scores (in Percentages) an institution with the LC classification, and LCSH, the tool. Of the ten questions, question 3 indicated the only decrease in score performance. Asked to find an arti- 11Some students apparently were still confused about the content of the on- line catalog, thinking that it indexed journal articles." de on child abuse, the correct answer (Readers' Guide) dropped from 80% to 71%. Two of the incorrect answers re- mained at the same level or decreased: the increase (11% to 27%) was in the incorrect choice-the online catalog. Some students apparently were still confused about the content of the online catalog, thinking that it indexed journal articles. This reflects perhaps too much attention given to the online catalog in the classroom presenta- tions and the assignment, which ties find- ing an article in an index to locating the jou17Ull in the catalog. This important dis- tinction is receiving more clarification and emphasis in LIP. The fourth question asked students how to find a book in the undergraduate library by author and title. The correct answer, which was the undergraduate library's author/title card catalog and the online cat- alog, went from 53% to 61%. The answer ''online catalog'' (which was partially cor- rect) went from 12% to 27%, and the ''author-title catalog'' dropped from 33% to 11% (only 1% responded with two to- tally incorrect answers). Some students in learning about the online catalog clearly forgot or disregarded the fact that the card catalog also provided relevant informa- tion for titles acquired before 1982. Two factors may be responsible: both deal with the wording of the question. The question which read "you would check" implies choice and they may have seen the correct answer as redundant. Students also may have been misled by the author-title cata- log answer, which did not specify" card." A third factor may be students' penchant to overemphasize the inclusiveness and power of an online catalog. These results also call for more clarification of the con- tent of the catalog. The fifth question dealt with identifying a complete call number. Students provid- ing the correct answer increased from 39% to 49% of the group. Those identifying only the classification element of the call number decreased from 52% to 41%; 3% on both tests identified the lower part of the call number; and 7% were consistent on both tests in incorrectly identifying the LC card number as the call number. Since the question asked students to identify what would be needed to locate a book, it could be argued that the classification por- tion of the call number would be adequate to locate the right subject section of the stacks, and given enough time (and pa- tience), to find the book. These results in- dicate that many students still lack under- standing about what constitutes a complete Library of Congress call number. Questions 6, 7, and 8 have already been discussed. The difficulty index was high for pretesting. The correct answer for the date went from 97% to 99%, correct re- Library User Education Program 255 sponses to the journal title question went from 87% to 91% (with 6% answering the article title in both tests), and the percent- age identifying the volume number moved from 93% to 98%. The runth question focused on what in- formation is available through the online catalog. The level of correct responses moved from 86% to 88%. The partially cor- rect responses indicated that students (98% of them) understood that book iden- tification and location information are available in the online catalog, but 10% (a drop in only 1% from the pretest) still failed to recognize that the location of journals is listed in the catalog. This dearly calls for a greater attempt to clarify the on- line catalog'~ coverage of journals. The most clear-cut example of success in student learning is manifested in question 10, which refers to Ohio State's "tele- phone center.'' The number of students who marked the correct answer, indicat- ing that they understood that the center can be used for checking out, renewing, and mailing books, and for identification of library-held books, rose from 72% to 91%. Five percent failed to realize that many items could be checked out, 4% didn't know about renewing books, and 1% missed the option that most circulating books could be sent by campus mail. These positive results can be attributed at least in part to students' motivation to learn about this service, as the telephone center provides a very convenient way to locate and retrieve books. Attitude Questions There were no attitude questions in- cluded in the 1985 evaluation, but in 1986 the attitudes of students about five topics-on a five-point Likert-type scale of positive (1) to neutral (3) to negative (5)- were part of both pre- and posttests. The subjects were: the telephone center, the online catalog, search strategies, using Ohio State's libraries, and librarians. Ap- plying a correlated t-test to the results, four of the five questions revealed differ- ences that are statistically significant; the question about using Ohio State's li- braries was the exception (see table 1). The results of the attitude questions 256 College & Research Libraries March 1989 TABLEt AITITUDE QUESTIONS (PERCENTAGE IMPROVEMENT BETWEEN PRE/POSITESTS) Mean* Telephone Center 16.5 LCS (Online Catalog) 13.5 Search Strategy 12.4 Ohio State Umv. Libraries 3.8 Librarians 26.6 strongly suggest that the active participa- tion of librarians in the program has a very positive effect on students. The online cat- alog is viewed favorably, both before and after the freshman experience, and atti- tudes toward using the telephone center improved somewhat as a result of instruc- tion. It is interesting to note that whereas the cognitive evaluation had indicated that the largest increase in posttest scores was related to the telephone center ques- tion, in the affective evaluation the tele- phone center ranked fourth out of five in attitude improvement and fourth out of five in student positive attitude on the posttests (see figure 2). CONCLUSION The 1985 and 1986 evaluations provide evidence that Ohio State's Library In- struction Program has brought about a statistically significant improvement in students' knowledge about the library, their ability to use libraries, and their atti- tudes toward libraries and librarians. Most entering undergraduate students have been successful in learning about the search strategy concept, LCSH, and the li- braries' online catalog and telephone cen- ter. They have become better able to find information in journal indexes, read jour- nal index citations, and identify complete call numbers. The program has also con- tributed to a statistically significant im- provement in students' attitudes toward li- brarians, the libraries' online catalog and telephone center, and the concept of the search strategy. Students' motivation and their perception of the relevance of instruc- tion were, not surprisingly, major factors ' in the success of their learning and reten- tion of instructional content. It has become clear that more instruc- Stan Dev. E> .0001 .988 .00()1 1.109 .0007 1.067 .0012 1.052 .3077 1.1 .0001 tional emphasis is needed on the search strategy (especially on the first step and its importance); on what constitutes a com- plete LC call number and why that infor- mation is important; and on LCSH, stress- ing its importance (and difference from LC). Without overemphasizing the online catalog, the program needs to clarify and better convey what it is and what it is not. Students especially need to understand better that the online catalog provides bib- liographic and location access to individ- ual journals but not to journal articles. A major factor in successfully achieving all of these objectives, is, as indicated, to con- vince students that this information is im- portant to them. In ten years of experience with the pro- gram, Ohio State librarians have observed some common misperceptions on the part of freshman students that were important enough to receive some special attention in the presentation and assignment. Some of these are obvious and others have al- ready been identified in the literature: stu- dents generally fail to realize the substan- tial differences between school/public and academic libraries and therefore overesti- mate the extent of their knowledge of the latter. Equally common is students' per- ception that the library catalog is an index to the entire holdings of the library, and that it is therefore the only source they need to consult. They have little realiza- tion of the number and variety of periodi- cal indexes available. Few freshman stu- dents (and many upper-class students) have any concept of standardized subject headings and are seldom conversant with LCSH. Students are more familiar by far with the Dewey classification than with the Library of Congress classification sys- tem, whose combinations of alphabetic Library User Education Program 257 PRE .................... 53'2:1 . . . . . . . . . . . . . . . . . . . . . . POSITIVE . . . . . . . . . . . . . . . . . . . . . POST 62.3 1 PRE ............... i45'3 :J . . . . . . . . . . . . . . . . . TELEPHONE NEUTRAL . . . . . . . . . . . . . . . . . ·POST 36.o 1 CENTER PRE )1.3 NEGATIVE POST 2.7 POSITIVE PRE POST ONLINE NEUTRAL PRE CATALOG POST NEGATIVE PRE POST POSITIVE PRE SEARCH POST PRE STRATEGIES NEUTRAL POST 34.0 NEGATIVE PRE POST POSITIVE PRE ·:::::::::::::::::::::::::::::::::::::::::::~.~: USING osu LIBRARIES NEUTRAL NEGATIVE POST PRE POST PRE POST POSITIVE PRE >> >>> :-: < · :-:-:-:-:-: < ·>>~:t · LIBRARIANS NEUTRAL NEGATIVE POST PRE POST PRE POST FIGURE2 Attitude Questions: Mean Scores (in Percentages) and numerical characters is sometimes confusing to them. Many students still do not understand what constitutes a com- plete call number and what significance that has for their finding information. What was learned about the science (and art) of evaluation? First, evaluation requires a commitment of time and effort and, equally important, the guidance and advice of an expert to ensure both the va- lidity and reliability of the approach cho- sen. Steps in planning an evaluation begin with examining the primary objectives of the instruction and then determining the best methods to measure whether or not the objectives are met. A very useful tool in doing this is the handbook Evaluating Bibliographic Instruction, which describes 258 College & Research Libraries how to plan an evaluation, select and test an instrument, and administer the evalua- tion. The handbook also strongly encour- ages the evaluator to consult with faculty who have expertise in evaluation and who can advise on how to ensure a valid, reli- able evaluation. It is important to allow enough time for careful planning and, in- sofar as possible, to delegate routine tasks to others. An example of the latter in the LIP evaluation was that the coding of the answer cards and the distribution, collec- tion, and organization of the materials were all done by a student assistant. Nine years of experience evaluating Ohio State's LIP program have reinforced two basic principles. Evaluation, even March 1989 with its cost in time and energy, is well worth the effort. LIP represents a very siz- able investment (especially in personnel) and such a commitment of resources car- ries with it an obligation to make the pro- gram as effective as possible. Such effec- tiveness can be achieved only through careful evaluation and application of the results of the evaluation. The second prin- ciple is to ensure that evaluations reflect accurate measurements. Evaluations of LIP have varied in validity and reliability over the years, but in the future, careful at- tention will continue to be given to plan- ning and implementing only the most credible studies possible. REFERENCES AND NOTES 1. Virginia Tiefel, "Libraries and Librarians as Depicted in Freshman English Textbooks," College En- glish 441:494-505 (Sept. 1982). 2. Sandra Kerka, Deborah Murray, and Arline Rollins, "LIP Service: The Undergraduate Library In- struction Program at The Ohio State University," Journal of Academic Librarianship 7:279-82 (Nov. 1981); Penny Pearson and Virginia Tiefel, "Evaluating Undergraduate Library Instruction at The Ohio State University," Journal of Academic Librarianship 7:351-57 Gan. 1982). 3. Thomas Kirk, "Bibliographic Instruction-A Review of Research," in Evaluating Library Use Instruc- tion, ed. Richard}. Beeler (Ann Arbor, Mich.: Pierian, 1975), p.3. 4. Pre- and Posttests used in 1985 and 1986 evaluations (See appendix A). 5. There were 1,011 tests administered (559 were pretests and 452 were posttests) in five different CAP areas: Agriculture; Allied Medicine; General Baccalaureate; Engineering; and Education. 6. GBC students (104) scored 67.3 and Arts & Sciences (65) scored 72-both with a probability factor of .0001. Business students' (59) mean was 72.5, but a probability factor of .8951 makes this score in- valid. APPENDIX A: LIBRARY SKILLS EVALUATION For each question, circle the letter corresponding to the best answer. 1. Search strategy involves a. Planning your approach to searching for information on a subject b. Using an encyclopedia first to find general information on your subject c. Starting your research by going to a magazine likely to have an article on your topic d. a and c are correct e. a and b are correct 2. The Library of Congress Subject Headings books (red books) a. Indicate which subject headings are used in the subject card catalog and on LCS b. List books in the Library of Congress c. a and bare correct 3. You need to find a magazine article about child abuse. You should go directly to a. Time Magazine b. The card catalog c. Reader's Guide to Periodical Literature d. Library Control System 4. You are in the West Campus Learning Resources Center. To find out if the LRC has a copy of Library User Education Program 259 Ordinary People by Judith Guest, you would check a. Book stacks b. Subject card catalog c. Author-title catalog d. LCS e. c and dare correct 5. What is the call number you would need to locate the book Rock 'n' Roll Woman? a. ML3561 b. MLR62 c. ML3561 R62 07 d. 73-9374 ML3561 R6207 ou ROCK MUSICIANS Orloff, Katherine. Rock 'n' Roll Woman/by Katherine Orloff Los Angeles: Nash. Pub., 1974, 199 p ., ports, 28 em. Interviews with Nicole Barclay, Toni Brown, Rita Coolidge, and others. OSNdc 73-93974 The following is a citation from a periodical index: ..___ ___ FOOTBALL, College b Beautiful Rose, even for Barna: USC vs Ohio State c D.S. Looney. Sports llius 5:28-33 Ja 14, '80 6. Which of the above letters identifies the date of publication? 7. Which of the above letters identifies the title of the magazine? 8. Which of the above letters identifies the volume number of the magazine? 9. In the Ohio State University Libraries, Library Control System (LCS) is a library computer system which can be used to a. Find out if the OSU Libraries have Jaws by Peter Benchley b. Find out at which library or libraries Jaws is located c. Determine which OSU Libraries have Time Magazine for 1970 d. a and b only e. a, b, and c 10. At OSU, you can accomplish the following tasks by calling the library center at 422-3900. a. Check a book out b. Renew a book c. Have a book sent to your on campus address d. Find out if OSU has Mein Kampf by Adolf Hitler e. All of the above are correct Circle the letter that expresses your feelings about OSU Libraries POSITIVE NEUTRAL NEGATIVE 11. Telephone Center a. b. c. d. e. 12. Computerized Card Catalog (LCS) a. b . c. d. e. 13. Search Strategies a. b. c. d. e. 14. Using OSU Libraries a. b. c. d. e. 15. Librarians a. b. c. d. e. ·staying up-to-the-minute with the fast-breaking research that is .shaping our future has never been easier with ... OWKER A& I PUBLISHING' -------~-~-OF-THE-ART DAl ~ ENVIRONMENT ABSTRACTS ~ ~RGY INFORMATION ABSTRA~ ~ACID RAIN ABSTRACTS ROBOTICS ABSTRACTS :===::::::::====-----i ---ARTIFICIAL INTELLIGENCE ABSTRACTS ---~ ~ CAD/CAM ABSTRACTS ~ ~TELEGEN (BIOTECHNOLOGY) ABSTRACTS~ ~TELECOMMUNICATIONS ABSTRACTS~ ... because every month Bowker's eight carefully compiled journals abstract and index thousands of articles on environmental studies, energy, acid rain*, robotics, artificial intelligence, CAD/CAM. biotechnology, and telecommunications. The information found in each publication is drawn from over 1,700 international scientific, economic, technical, academic, and trade journals plus scores of leading general-circulation newspapers, magazines, and wire services. More than just journals. Better still, our journals are part of a sophisticated document retrieval system that lets you order comprehensive microfiche collections carrying the full text of over 80% of the articles we cite. And you can order microfiche or paper copies of individual articles on a per-document basis as well. In addition, all eight databases are available online from the top online services. What's more, Bowker A & I Publishing's journals bring together the "grey" or elusive literature -- conference proceedings, government studies, scientific association reports, even patents -- that often is neither indexed or available in any readily accessible medium. So whether you're looking for the latest findings on gene products, ozone depletion, nonconventional fuels, U.S. acid rain policy, robot locomotion, human-machine interfaces, or any of hundreds of other crucial high-tech issues, don't bury yourself in indexes or burden yourself with serials orders. "Acid rain is abstracted and indexed every two months. Look to Bowker A & I Publishing to keep you current with the literature. You'll not only make a small research breakthrough yourself ... you'll m~e a big _one easier for your patrons, too. Call Now For a Free Sample Journal! TO ORDER - OR FOR MORE INFORMATION ABOUT OUR INTEGRATED SYSTEM OF MONTHLY JOURNALS, ANNUAL INDEXES, MICROFICHE, AND ON!UNE DATABA.SES - CALL TOLL-FREE 1·800·521·8110 (IN NY, AK, OR HI CALL COUECT 1-212-4e3-llll). B 0 w K E "' ~ IIPUBLISHING THE ABSTRAC TING AND INDEXING DIVISION OF R. R. BOWKER Fo uurly fi CIInUllt gu u A.A. Bowker, 245 West 17 Street. New York, NY 10011