Untitled-3 Engaging Conversation 545 545 Engaging Conversation: Evaluating the Contribution of Library Instruction to the Quality of Student Research Mark Emmons and Wanda Martin Mark Emmons is an Assistant Professor and Head of Instruction Services at the General Library, Univer- sity of New Mexico; e-mail: emmons@unm.edu. Wanda Martin is Associate Professor of English and an Associate Dean in the College of Arts & Sciences at the University of New Mexico; e-mail: wmartin@unm.edu. The authors would like to thank Elizabeth Malloy, graduate statistics consultant, for assisting them with their statistical analysis. The study discussed in this article compared research papers before and after implementation of an inquiry-based library instruction program to assess the program’s effectiveness and consider its future develop- ment. The new program appears to have made a small difference in the types of materials students chose and how they found them. Little change was seen in how students used their sources. The evaluation has fos- tered increased collaboration between the library and the English de- partment, and suggests that the programs can be further improved by providing more training for instructors and placing greater emphasis on the rhetorical approach to research. his study was designed to ex- amine the impact of a new li- brary instruction program on the research processes of stu- dents in a first-year writing course. Three years after replacing a tool-based library instruction program with one based on inquiry, the authors ask what difference the new program makes and how the next steps can be designed to help students de- velop information literacy. In 1998, the library instruction program for first-year students at the University of New Mexico consisted of a series of workshops designed to teach users how to use research tools such as the online catalog, the most-used online indexes, and the Web. The program was a remnant of the early days of publicly available in- formation technology when many stu- dents, staff, and faculty, familiar with the card catalog and printed indexes, were eager to learn how to use the new elec- tronic versions of these tools. However, interest ebbed and these tool-based work- shops were sparsely attended. Students who came to the university already famil- iar with computers assumed they could figure out how to work the systems on their own. Except where faculty required attendance, the library’s certificate of completion had little appeal. And because more students used the Internet to do re- search before they came to the university, they saw little need for the tools the li- brary could offer. These students came to campus highly computer literate, not re- alizing they were information illiterate. Across the nation, librarians had no- ticed similar trends. In 1989, the ALA’s 546 College & Research Libraries November 2002 Presidential Committee on Information Literacy issued a report explaining the importance of information literacy to de- mocracy in an information age.1 The re- port defined information-literate people as “those who have learned how to learn. [T]hey know how knowledge is orga- nized, how to find information, and how to use information in such a way that oth- ers can learn from them.” Librarians be- gan to reduce emphasis on research tools and focus more on research processes and critical thinking, eventually publishing the Information Literacy Competency Standards for Higher Education in Janu- ary of 2000.2 It was in this context, with the avail- ability of information resources outpac- ing the students’ ability to sort and evalu- ate them, that the library approached the English department with a proposal to have each English 102 class formally par- ticipate in a library instruction program. Although only a few hundred feet of open plaza separate the library and the English department, initiating this discussion was a bold move. The University of New Mexico is a public, doctoral-extensive university of about 25,000 students. The freshman class numbers about 2,500 each year, mostly traditional-age students recently gradu- ated from high schools located in the state. Every undergraduate must complete En- glish 102, Analysis and Argument, to earn a degree. This course is the second in a two-course composition sequence de- signed to help students develop skills for reading and writing in academic contexts. It requires a researched essay in which students make and support an argument using sources other than the assigned course readings. At the time the library made its pro- posal, some instructors still required their students to attend the library workshops, but English 102 faculty, mostly graduate instructors, found the workshops to be of minimal value. The department’s Fresh- man English Committee, already in the process of redesigning the English 102 curriculum and adopting new texts, readily agreed to help rethink the library instruction program as part of an effort to make the course more effective. The committee and the library faculty agreed that students needed improved research skills. Students were coming to the university with minimal library re- search experience, often having written only a single research or term paper in high school English. They did not know how to locate materials in a large library and typically lacked skill in evaluating the credibility and biases of sources and in comparing and synthesizing material from more than one point of view. Al- though most of the English 102 instruc- tors were fairly adept at performing re- search in their own fields of literature or rhetoric, they were not necessarily equipped to teach library research strate- gies for the broad range of topics typical of freshman writing. What freshman English needed was a library program that would support its emphasis on writing as an inquiry pro- cess. Students should begin their research with a question or a thesis. In addition, they should use periodicals as well as books and other resources to focus their questions more productively and to dis- cover possible answers. Moreover, they should take a critical stance toward sources, regarding authors as rhetorically situated contributors to the conversation about an issue (i.e., people with an inter- est and a point of view) rather than as dis- interested sources of facts to be reported. They should understand and use sources rhetorically to explore new ideas and to inform and support their own claims. Working from early versions of infor- mation literacy goals and a 1993 article by Barbara Fister, the library faculty shared this perspective on research, which encourages students to become a part of the formal conversation being con- What freshman English needed was a library program that would support its emphasis on writing as an inquiry process. Engaging Conversation 547 ducted in the literature about a particu- lar issue.3 An instruction librarian with a background in literature, Fister noted that much library instruction emphasizes “in- formation retrieval as the purpose of re- search,” when, in fact, “most college-level assignments demand an idea that is de- veloped, argued, and supported with evi- dence.” She argued for a “rhetorical” stance toward sources in terms almost identical to those used in English depart- ment documents. The new program de- signed to meet these needs was launched in the spring of 1999. The library lesson plan for English 102 puts library tools in the context of inquiry by organizing the library instruction around the questions students bring from their classrooms. The lesson builds on a framework proposed by Carol C. Kuhlthau, which described “intellectual access to information and ideas, and the process of seeking meaning.” 4 In Kuhlthau’s six-stage model, students ini- tiate the research process by exploring possible topics and developing a question they would like to answer. They become better informed about the field of study by gathering broad background informa- tion from materials such as encyclopedias. As they formulate more-focused ques- tions, they seek more-specific sources of information, developing more-sophisti- cated search strategies to find what they need. Thus, rather than learn to use the online catalog and an index simply for the sake of becoming familiar with those tools, students do so in the service of a real quest for information to increase their understanding. The lesson extends Kuhlthau’s model by addressing distinc- tions such as the one between popular magazines and scholarly journals, invit- ing students to consider where the con- versation is taking place and the kinds of expertise that are involved. An outline of the lesson is shown in figure 1. Of course, students also need to learn the practical skills that will enable them to find the information they need to un- derstand the conversation and participate in it. These skills include the nuts and bolts of searching the online catalog and the EBSCOhost Academic Search Elite in- dex (hereafter, EBSCO). The lesson in- cludes a section on simple Boolean logic. Although students have only brief oppor- tunities for hands-on work with their own topics during the session, the library of- fers an optional second lab in which stu- dents can spend the entire class period researching their topics with the assis- tance of a librarian. This enables students to practice what they have learned with someone who can help them deal with uncertainty, refine their search strategies, and evaluate what they find. Both ses- sions still teach the use of library research tools but focus attention on the research context. In the spring of 2001, the authors set out to evaluate the new program’s effec- tiveness by comparing research papers written before and after it began. Despite earlier assessments in which it was found that English 102 instructors and students overwhelmingly valued the li- brary instruction sessions, the authors were not satisfied because they had not tested student skills. A survey of the lit- erature by librarians Lois M. Pausch and Mary Pagliero Popp5 revealed that the authors were not alone. In their survey, Pausch and Pagliero found that few stud- ies measure student learning and that most assessment of library instruction is relatively informal, often consisting of surveys of student satisfaction. They found that although “the emphasis of many programs has changed from tools to concepts … few programs have adopted methods aimed at assessing whether students gained the cognitive skills for analysis, synthesis, and evalu- ation of information that form the basis for much of the assessment of higher education.” Although formal assessment for program improvement is increas- ingly prevalent in higher education, and increasing attention is paid to the “as- sessment culture” of libraries, little re- search has been published that uses stu- dent research papers in a nonlibrary course in a rigorous way to measure the 548 College & Research Libraries November 2002 effectiveness of library instruction. The authors thought it would be useful to adapt the freshman writing program’s existing outcomes assessment protocol so as to conduct a careful assessment not only of what students heard in their li- brary instruction sessions, but also of how they applied that learning. Methods The Portfolios Students in English 102 complete their semester’s work by submitting a portfo- lio that includes two revised papers. In- structors emphasize the revision process, encouraging students to submit the very best work they can do. In most sections, FIGURE 1 Outline of English 102 Lesson Plan ASSIGNMENT Students in English 102 write a researched essay in which they make and support an argument using sources other than the assigned course readings. In addition to helping students develop skills for reading and writing academically, the pa- per is designed to involve students in the discussion taking place in the scholarly literature, understanding authors’ contributions as rhetorically situated and sub- ject to evaluation. While introducing basic research process and tools, the library becomes a part of the rhetorical process in which students make their paper part of formal conver- sation going on in books and journals. An example based on the reading takes students through the research process. There is time for students to begin their research and a second optional lab session where they can continue researching. The lesson includes the following: RESEARCH PROCESS • Choose a topic. • Develop a research question or a working thesis. • Gather background information, books, and articles. • Evaluate what you find. FINDING TOOLS • Reference sources • Library catalog • Multidisciplinary index (EBSCO) KEY CONCEPTS • Keyword versus subject searching • Boolean logic • Scholarly journals versus popular magazines NAVIGATION • How to access • How to locate • Services available, including help with research SAMPLE RESEARCH QUESTION How does the image of women in the media affect what women believe about themselves? Engaging Conversation 549 this portfolio accounts for half the semes- ter grade, so students generally revise with care. Portfolios are evaluated collaboratively by small groups of instruc- tors. Since the fall of 1996, the freshman English program has conducted an out- comes assessment project designed to dis- cover how effectively instructors teach college composition skills and to help them make the program more effective. During each semester ’s grading pro- cess, the authors randomly selected un- marked portfolios from each grading group and photocopied them, winnow- ing the sample to twenty-five from each course (about two percent of the students taking the course each year). Thus, in the ten semesters from the fall 1996 through the spring of 2001, 250 portfolios were gathered for analysis from students com- pleting English 102. For the purpose of outcomes assess- ment in freshman English, a group of in- structors reads the sample portfolios against an analytical rubric that asks, with respect to each of the objectives of English 102, “What evidence do I see that the au- thor of this paper controls this particular skill?” The features assessed include the presence of an arguable thesis, the use of source texts to support that thesis, the overall organization of the argument, the development and coherence of para- graphs and sentences, and the control of conventional usage and mechanics. This kind of rating scheme, using a rubric to describe the qualities sought in the object, is well established in the field. The best-known uses of such rubrics are in scoring free responses on standardized tests such as the SAT II or Advanced Placement Examination. In those contexts, where the purpose is to rank students’ work for college admission or credit, re- sponses are scored relative to one another and rated holistically for overall quality in terms of several key characteristics. In contrast, an analytic reading looks at each object separately, not as a relative perfor- mance. And rather than give a single score for “overall quality,” it assigns a value to the performance of each characteristic rated. Thus, instead of asking how good one is compared to another, the reader focuses on specific qualities: “On what skills do students do well? On which ones are they weaker?” After six readers have rated each paper, some time is spent de- briefing, comparing notes on what has been seen, and considering how practices might be shifted to make the program work better for students. This program is discussed more fully in Martin.6 The 250 portfolios on file, equally di- vided between those completed before and after the library instruction program was developed, provided a rich source of data about the students’ information lit- eracy and how the library instruction pro- gram may be influencing it. The Rubric The first step was to design a rubric that would allow examination of student work for the qualities the authors wanted to study and to assign them numerical values so that performances could be compared over time. The authors began by reading together samples from the col- lected portfolios, deciding to work with only the researched essay from each port- folio that provided the most consistent and unambiguous examples of students applying their library research skills. Next, the authors sought to identify the features of interest by exploring parallels and divergences between the ACRL Infor- mation Literacy Content Standards for Higher Education7 and the course objectives that drive English 102. The authors looked at the analytic rubric for English 102 out- comes and corresponded with Dennis Isbell, a presenter at an ACRL workshop on developing rubrics for library research.8 They also looked at several examples of rubrics for assessment of research and bib- liographies posted on the Web, including If the study was to be of interest to their colleagues in the library and the English department, the authors also needed to look at the uses to which students put their sources. 550 College & Research Libraries November 2002 the Lowell Research Cycle Rubric, scoring criteria developed by Kim Ranger for a writing course, and a rubric designed by Keith Wetzel to analyze literature re- views.9–11 The authors first decided to de- scribe the students’ bibliographies in terms of number of citations and variety in for- mat, source and time frame of the infor- mation, and accuracy of the citations, and then to evaluate the writers’ use of sources. It was hypothesized that students who had taken the library instruction program would use more varied sources and, given the emphasis on entering an ongoing con- versation, perhaps more current sources. Because the distinction between popular magazines and professional or scholarly journals is an important part of the pro- gram, the authors expected to see more scholarly and professional sources in postprogram bibliographies. And because students can write on many subjects, the program featured EBSCO, which is a multidisciplinary full-text magazine and journal article index. It was thought that the frequency of sources indexed in EBSCO might provide an index of the ef- fect of the instruction program on stu- dents’ research practices. However, student research is not just searching for sources. If the study was to be of interest to their colleagues in the li- brary and the English department, the authors also needed to look at the uses to which students put their sources. This would mean reading the essays carefully to assess how judiciously students chose sources to support their research pur- poses, how well they understood the sources’ arguments and limitations, and how effectively they engaged the sources to construct their own meaning. Tacking back and forth between the ACRL Stan- dards and the English 102 objectives, as well as mining their own experiences with student research (including their own college research experiences), the authors drafted a matrix-style rubric that divided each of these categories, which they called Relevance, Credibility, and Engagement, into three qualities they hoped to see in successful papers and as- signed a numerical value to each of four degrees of success they thought it would be possible to identify. To test this rubric, the authors indepen- dently scored five sample essays chosen to represent the range of final grades earned in English 102. Then, in a long and lively discussion, they compared the as- sessments in order to reach agreement on what marked each level of success on a feature, to flesh out the descriptions of each feature and level, and to sort out overlapping categories. The final version of the rubric is shown in figure 2. The Evaluation Process The authors wanted to describe quanti- tatively all the bibliographies at their dis- posal to develop the most detailed under- standing of how the library instruction program may have influenced students’ research practices and to provide a baseline of what the students are doing now for comparison as the programs con- tinue to evolve. But it would not be fea- sible to conduct the more-detailed read- ings for Relevance, Credibility, and Engagement on all 250 samples, so the au- thors randomly selected thirty samples each from the pre- and postlibrary in- struction pools. A research assistant used a random number table to select the sixty samples for close analysis, giving them reference numbers that disguised their place in the portfolio sequence so that readers would not be biased by knowing whether they were “before” or “after.” Three senior teaching assistants in the Freshman Writing Program were chosen to assist with the scoring. Each reader was an advanced doctoral student in English who had taught English 101 and 102 for several different semesters and partici- pated in the outcomes assessment read- ings. This made the task more manage- able and gave the authors the opportu- nity to discuss with three knowledgeable readers their perceptions of the student work, greatly enriching the analysis. Moreover, this advanced the program’s agenda of continuous assessment and im- provement because the readers, after ex- Engaging Conversation 551 From Library: Did the class come to the library? ¨ YES ¨ NO Did the class take the second optional lab session? ¨ YES ¨ NO FIGURE 2 The Rubric English 102 Portfolios�Research Paper Rubric READER _________ Fall Semester, 2001 Portfolio Number: __________ English 102 section: __________ Fall Spring Summer Course Grade __________ DESCRIPTION Format Total Textbook Reference Book Scholarly Popular Web Other or Reading Journal Magazine List total number of citations followed by total number of each format. Journals and Magazines EBSCO index EBSCO full text Total journals/magazines List number indexed in EBSCO and number available in full text. List total journals and magazines cited. Time Frame 3 2 1 Within five years Five to fifteen years old More than fifteen years old List number published in each time frame. Accuracy 3 2 1 All references in Includes identifying information, Insufficient or incorrect standard MLA style with errors in format information, frequent errors See following pages for relevance, credibility, and engagement scores. Content Currency Level Total Relevance Authority Summary Purpose Total Credibility Evidence Challenge Meaning Total Engagement 552 College & Research Libraries November 2002 Authority Recognizes author�s authority Summary Summarizes main idea Purpose Acknowledges author�s bias and purpose Identifies each author�s credentials � and they are relevant Summarizes main idea of each source and makes explicit connec- tion with the argument Explicitly acknowledges the purpose or bias of each source Identifies author�s credentials Summarizes main idea of most sources Implicitly acknowledges the purpose or bias of each source Provides limited information about authors Summarizes main idea of some sources Acknowledges the purpose or bias of some sources Provides no information about authors Does not summarize main idea of sources Does not acknowledge the purpose or bias of sources CREDIBILITY Student cites authority of author, summarizes main ideas, and recognizes bias. 4/Excellent 3/Adequate 2/Limited 1/Poor Content Source relevant to topic Currency Time frame appropriate to topic Level Level and variety appropri- ate to purpose All sources clearly related to topic. All sources published in appropriate time frame All sources written at level appropriate to purpose Most sources clearly related to topic. Most sources published in appropriate time frame Most sources written at level appropriate to purpose Some sources clearly related to topic. Some sources published in appropriate time frame Some sources written at level appropriate to purpose Sources un- related to topic or relevance unclear. No sources published in appropriate time frame; No attention at all to time No sources written at level appropriate to purpose FIGURE 2 The Rubric (continued) RELEVANCE/SIGNIFICANCE Student selects information that addresses the thesis or question and that helps analyze a problem or propose a solution 4/Excellent 3/Adequate 2/Limited 1/Poor Engaging Conversation 553 FIGURE 2 The Rubric (continued) ENGAGEMENT Student constructs knowledge and makes meaning of information by entering into a conversation and arguing with sources. 4/Excellent 3/Adequate 2/Limited 1/Poor Evidence Supports arguments with evidence Challenge Challenges ideas and recognizes more than one side to an issue Meaning Constructs meaning or presents original idea Elaborates and extends by supporting arguments with evidence throughout Paper as a whole acknowledges more than one perspective; criticizes own perspective Analyzes and synthesizes; provides new interpretations of old ideas; draws connec- tions Does not question the validity of sources. Acknowledges more than one perspective. Interprets and evaluates; compares and contrasts ideas; assesses value of ideas Quotes sources without comment and without critical understanding or evaluation. Acknowledges only own perspective and denies validity of other perspectives. Understands, interprets, and discusses ideas Makes unsup- ported assertions throughout with little or no effort to cite sources Does not acknowledge more than one perspective Presents no original ideas; shows poor understanding amining the students’ work through this different lens, had the opportunity to share their observations with colleagues in the program’s next teaching orienta- tion. Working as a team, the authors con- ducted readings to generate data. First, they completed the Description portion of the rubric for each of the 250 essays. Next, they met for four hours on each of four days to complete the Relevance, Credibility, and Engagement analyses. They began this reading by scoring a sample from outside the pool and dis- cussing it to arrive at a reasonable con- sensus on how to apply the rubric. Each essay then was scored independently by three readers. After the first of these reading sessions, the authors discussed how well the rubric facilitated the scor- ing and agreed to two minor adjustments in wording. The first, in the Relevance category, was to add “level” to the char- acteristic “variety,” recognizing that in a strong paper they were looking not only for an appropriately expert and detailed treatment of the topic, but also for a range of appropriate sources. In the Credibility category, under the charac- teristic “Purpose,” the authors changed “recognizes author ’s bias” to “acknowl- edges author ’s bias,” so as to assert that they sought an overt statement of evalu- ation. Results The rubric poses a number of questions about the quality of the essays, and with the help of a consultant, the authors sub- jected most of the questions to statistical analysis. However, only a handful of these generated statistically significant 554 College & Research Libraries November 2002 findings (as discussed below). The au- thors did not, for example, try to relate students’ grades in English 102 to the outcomes of library instruction because the course grade includes many addi- tional factors. And because the optional second “lab” visit is a recent innovation, it was too soon to measure its impact on more than the most informal terms. As much as anything, the authors sought to establish a method and a body of baseline data that would help them evaluate the program as it develops and changes. Part I: Description of the Bibliography The first part of the rubric described the bibliography of each paper in terms of the number of each format (textbook or class reading, reference, book, scholarly jour- nal, popular magazine, Web, or other), whether journal citations came from EBSCO Academic Search Elite, the time frame covered, and the accuracy of the ci- tations. The pool of 250 yielded 223 usable pa- pers; the remaining twenty-seven portfo- lios included something other than a re- searched essay, such as an annotated bib- liography. Although there was no reason to believe that the distribution would sat- isfy the normality assumption of a stan- dard symmetrical bell curve, the sample was large enough to use a standard t-test. Are There More Citations? The authors did not expect more citations because most English 102 instructors limit the total number of works that students can cite in their papers. The test deter- mined that there were no significant dif- ferences in total number of citations. (See table 1.) Are There More Citations to Any Particu- lar Format? Because of the program’s emphasis on using scholarly literature, it was antici- pated that students would cite more scholarly journal articles. And because of the instructor-imposed ceilings on the total number of citations, the authors be- lieved that students would cite fewer books as a result. The authors were right on the first count, but not the second. (See table 2.) The increase in the number of journal articles cited was the most significant dif- ference in the entire analysis. As the p- value for t-tests reveals, the increase in citations to scholarly journals is the only statistically significant change in format. (See table 3.) Students cited exactly the same number of books before and after; books were, in fact, the most cited format both before and after implementation of the library instruction program. Because the total number of citations was not sig- nificantly larger, the authors wondered whether students were citing fewer works in other formats. (Totals are shown in table 2.) Slightly fewer citations to the classroom textbook and to reference books were found, but the only, even bor- derline, significant difference was unfor- tunately a reduction in the category “other.” Further analysis will be needed TABLE 1 Mean Number of Citations Number Mean St. Dev Before 108 5.17 1.94 After 115 5.17 1.76 TABLE 2 Total Number of Citations by Format Text Ref Book Journal Magazine Web Other Total Before 71 13 202 57 69 66 80 558 After 55 11 202 113 87 83 49 600 Total 126 24 404 170 156 149 129 1,158 Engaging Conversation 555 to see what kinds of materials are included in the “other” category, but the informal impression from the readings is that many are newspaper articles. In fact, a MANOVA test (multivariate analysis of variance) to see what combination of formats had the most impact indicated that the increase in scholarly journal ar- ticles cited, along with the reduc- tion in citations to textbooks and “other,” accounted for the largest difference. (See table 4.) These results suggest that the library instruction program did bring scholarly journals to the attention of stu- dents. However, the persistence of books as the most-cited category begs the ques- tion of what factors led students to make that choice. A scan of the essays suggests that topic may be one factor. Students writ- ing on historical and literary topics, for ex- ample, seem to have used more books whereas those with more contemporary topics, such as body image or environmen- tal pollution, generally cited more journal articles. This suggests, at least, further study to explore how a student’s topic choice may influence his or her research strategies and choice of sources. Are Articles More Recent? Because of the program’s emphasis on students entering an ongoing conversa- tion, the authors thought that students might cite more recent articles than those used by their preprogram counterparts but found no significant difference in the time frame of their sources. (See table 5.) By far, the most-cited works were writ- ten within five years of when the student wrote the paper. (See table 6.) Perhaps a more fine-grained analysis, using shorter time frames, will reveal more about stu- dent writers’ sense of what is appropri- ately current information for a given topic. Are Citations More Accurate? Because citation accuracy is not part of the library instruction program and received no new emphasis from English 102 instruc- tors, the quality of citations was not ex- pected to improve. This category was in- cluded because it was a standard data point in the rubrics the authors examined when they prepared their own and be- cause the authors felt this category might provide interesting data to inform future teaching. There was no significant differ- ence in the accuracy of citations before and after the new program. (See table 7.) TABLE 3 Mean Number of Citations Used (Standard deviation in parentheses) p-value Before After for t- test Text 0.66 (0.97) 0.47 (.063) 0.089 Ref 0.12 (0.38) 0.10 (0.32) 0.603 Book 1.87 (1.76) 1.73 (1.70) 0.548 Journal 0.53 (1.22) 0.97 (1.51) 0.016 Magazine 0.64 (1.27) 0.76 (1.27) 0.491 Web 0.61 (1.35) 0.72 (1.35) 0.566 Other 0.74 (1.28) 0.43 (0.90) 0.036 TABLE 4 Manova Multivariate Analysis of Variance Test Statistic f-value Degree of Freedom p-value Wilk�s 0.93378 2.178 7, 215 0.037 Lawley-Hotelling 0.07091 2.178 7, 215 0.037 Pillai�s 0.06622 2.178 7, 215 0.037 … students do not have a strong grasp of the kind of information a bibliographic citation should provide. 556 College & Research Libraries November 2002 This did prove interesting, though, because an overall mean accuracy of only two on a three-point scale was observed, suggesting that students do not have a strong grasp of the kind of information a bibliographic citation should provide. In scanning the essays to consider why books are such a prevalent category, the authors observed a substantial number of inaccurate citations that characterized, for example, articles from anthologies or ex- cerpts included in course packs as books. Part II: Relevance, Credibility, and Engagement The second part of the rubric asked read- ers to read each paper and examine how students used the works they cited in their bibliographies. The readers were looking for evidence of the rhetorical approach to research taught in the library lesson and the writing program, and called the char- acteristics they sought to observe “rel- evance,” “credibility,” and “engagement.” Relevance is high when sources directly address the student’s question, are timely, are written at a level the writer can use, and display variety suitable to the writer’s purpose. Credibility is high when a stu- dent acknowledges the author’s authority, bias, and purpose, and shows understand- ing of main ideas rather than merely min- ing for “quotes.” Engagement is high when the student supports arguments with evidence, challenges sources’ ideas, recognizes more than one perspective on an issue, and constructs meaning or pre- sents original ideas. Were the Sources More Relevant? It was thought that the sources used by the students might be more relevant be- cause the library instruction program, in concord with the writing pro- gram, focuses attention on formu- lating a question and entering the scholarly conversation around it. In retrospect, the rubric did not ask readers to give higher scores to the kinds of sources empha- sized in the instruction program but, instead, to accept works clearly related to the students’ topic and purpose without regard to format or con- text of publication. And students had no difficulty locating sources that readers found relevant to their topics. On a 12- point scale, the mean relevance score of postprogram papers was 9.6, compared to 9.1 before the new program. (See table 8.) Although the difference in mean scores is of only borderline statistical significance, it does suggest a trend toward greater rel- evance, especially upon examining the distribution of scores. The boxplot in fig- ure 3 separates the scores into quartiles; the gray area shows the middle 50 percent, with the median score indicated by the horizontal line. Before the program, the middle 50 percent clustered between 8.5 and 9.5, and the top 25 percent (vertical line) extended to 11. After the program, the middle 50 percent clustered from 8.5 up to 11, and the top 25 percent extended almost to 12. Interestingly, the bottom 25 percent (lower vertical line) of postprogram scores ranged much lower than those before pro- gram implementation. TABLE 5 Time Frame 0�5 years 5�15 years 15+ years None Before 75 21 22 19 After 80 28 24 11 Total 155 49 46 30 TABLE 6 Time Frame Most Used Before Time Frame 0�5 5�15 over 15 years years years Number 75 21 22 Proportion 0.64 0.18 0.19 After Time Frame 0�5 5�15 over 15 years years years Number 80 28 24 Proportion 0.61 0.13 0.10 Engaging Conversation 557 The authors would like to think that this means more stu- dents (and perhaps some in- structors) are getting the idea of an inquiry-based research pro- cess. But the lower range sug- gests that some students may be more confused than their pre- decessors were. Clearly, this re- sult will require further study. Were Students More Aware of Sources� Credibility? In contrast to the relatively high relevance scores, credibility scores for both groups were approximately 4.8 on a 12-point scale. It was clear from the readings that students, both before and after the new program, viewed the contents of sources as indisputable fact and made little effort to assess authors’ credentials or to exam- ine their biases. (See table 8.) Were the Students More Engaged with the Ideas They Wrote About? Again, because the library instruction program asks students to engage in a conversation with the authors of the works cited, it was thought that engage- ment scores might increase slightly. However, mean engagement scores were 6.60 before and 6.38 after the program, a statistically insignificant difference. (See table 8.) Conclusion The new library instruction program ap- pears to have made a small difference in the types of materials students chose and how they found them. Students used more scholarly journal articles, which they most likely found more efficiently through EBSCO than with the tools that TABLE 7 Accuracy Mean Score on a 3-Point Scale (Standard deviation in parentheses) p-value Before After for t- test Accuracy 2.073 (0.742) 2.087 (0.615) 0.882 FIGURE 3 Distribution of Relevance Scores 558 College & Research Libraries November 2002 were previously available. Books continued to be the most-cited type of material. The relevance scores suggest that students are developing a somewhat greater awareness of the types of sources that might address their research questions. These results sug- gest that, as a group, students are able to understand, inter- pret, and discuss ideas but rarely rise to the level of evalu- ation, analysis, or synthesis. Apparently, neither the writing program nor the li- brary instruction program is thus far mak- ing much headway in helping students understand their source texts or their own research efforts as rhetorically situated. Implications for the Library The analysis indicates that the new En- glish 102 program was a step in the right direction, primarily in that it fostered con- versation between the library and the English department and revealed the common theoretical ground they share. But smaller-than-expected differences in the quality of student research call for several refinements. First, there is a press- ing need to address the consistency of instruction, especially as it relates to the conceptual framework that drives the les- son plan. Second, the lesson plan should make better connections with students’ actual research topics. Third, greater col- laboration is needed among English 102 instructors and library faculty. At the start of each semester, the head of instruction services meets with the English 102 faculty to demonstrate the lesson and talk about effective library as- signments. To ensure that students are motivated and focused when they come to the library, the English 102 faculty are encouraged to schedule their library in- struction classes at a time when students have developed a question they would like to answer. But with more than forty sections in the fall and eighty in the spring, it is not always possible to sched- ule the class at the optimal time. In read- ing the papers, the authors noticed that topics used as examples in the library in- struction program were more highly rep- resented. Although this may be an indi- cation of having chosen very obvious ex- amples based on the readings, it also sug- gests that some students might select top- ics for which the library instruction has given them a head start on research. It would be interesting to experiment with topic examples in the future to see if this does indeed have an impact on student topic selection. In the meantime, however, greater emphasis should be given to scheduling the session at a time when stu- dents are ready to explore research ques- tions of their own devising. The optional follow-up labs appear to be a promising innovation that supports and extends the library instruction pro- gram. This element was introduced in the spring of 2000, and in the following spring, almost a third of English 102 sec- tions made a second library visit. The En- glish faculty praise the additional sessions in which students can work on their own projects with the help of a librarian. The labs provide an opportunity for both the English 102 and the library instructors to comment on the relevance and credibil- ity of the resources that students locate during class time. Although the current TABLE 8 Relevance, Credibility, and Engagement Mean Score on a 12-Point Scale (Standard deviation in parentheses) p-value Before After for t- test Relevance 9.118 (0.897) 9.640 (0.31) 0.083 Credibility 4.79 (1.19) 4.77 (1.31) 0.954 Engagement 6.60 (1.16) 6.38 (1.49) 0.532 And the quite modest scores for Engagement indicate that students continue to view the researched essay as an academic exercise more than as a quest for knowledge. Engaging Conversation 559 sample included too few students who had this experience to meaningfully as- sess its impact on their papers, future studies will be able to do so. The kind of collaboration that takes place in the lab also should be formal- ized during the instructional design pro- cess. One way to do this might be through the English 102 readings: Each class selects one of five books that are chosen to raise the research questions. Library instructors could conceivably become members of the groups that ana- lyze the books and come up with themes to explore. They then could create an array of examples for use in the class in- stead of the one per book that is used currently. An even simpler step would be to require library instructors to attend the English 102 training. Such a large program demands wide- spread participation of library faculty and staff. Each semester, the head of instruc- tion services offers training and refresh- ers to those who will be teaching. With as many as nineteen faculty and staff mem- bers participating, it is essential that the library instructors provide a consistent lesson to each class. However, this is not always easy to ensure. Because of staff- ing constraints, it also has not been pos- sible to consistently observe instructors at work. Limited observation and infor- mal reports, along with the results of the authors’ analysis, indicate that many in- structors continue to emphasize tools over process. Future training needs to empha- size the importance of the rhetorical ap- proach to research. Perhaps the most im- portant step will be sharing the results of this study—the collaboration that began with two program heads interested in assessing a program could very well end up pushing library faculty and staff into greater collaboration. Implications for the English Department The analysis so far suggests that the En- glish department needs to focus more at- tention on the purposes of the researched essay. The low level of accuracy and com- pleteness of citations is not just a cosmetic matter but, rather, indicates that students are not clear on the concept of document- ing source material. It looks as though many students continue to work with broad topics rather than with focused re- search questions. Moreover, from the in- frequency with which students question authors’ assertions or put them into con- text, the authors infer that, despite instruc- tor efforts, most students view sources as places to get facts to support their asser- tions and fail to take into account the sources’ rhetorical situation or to develop one of their own. And the quite modest scores for Engagement indicate that stu- dents continue to view the researched es- say as an academic exercise more than as a quest for knowledge. The authors plan to analyze the data further to gain more insights into how students approach re- search tasks and will take their insights back to the instructors in conversations about curriculum and teaching. In Janu- ary 2002, when the preliminary results of this study were shared with English 102 instructors, the instructors were particu- larly interested in the rubric as a device for teaching students and for evaluating the quality of their research. Many instruc- tors are themselves new to the idea of re- search as a rhetorical process and receive little training in specific strategies for teaching research. Perhaps the rhetorical research process, in conjunction with the rubric, might be used in future training of first-time English 102 instructors. In addi- tion, the authors will use the results to frame some focus groups with students to help clarify why students make the choices they do and how they can be helped to choose more discerningly. Although much remains to be done, this study represents a good start toward implementing a library instruction pro- gram that will support students in an in- quiry-based approach to research. Work- ing with each instructor to provide a li- brary session tailored to the interests of her or his students, at a time when students are ready to begin library work, has greatly increased communication between the li- brary and the English department. 560 College & Research Libraries November 2002 Despite these gains, however, the au- thors are not satisfied that students are finding sources relevant to their topics as long as they show so little awareness of the other questions they should be asking and are so little engaged in the real intel- lectual work of evaluating and analyzing the ideas they use to fill out their papers. The real significance of this project so far has been discovering the convergence of the two programs’ theoretical perspectives. Traditionally, library instruction has been viewed as a “service” to the writing pro- gram in much the same way that the writ- ing program provides “service” to the larger curriculum. But the authors’ work together has shifted that relationship be- cause they realize that the ACRL’s infor- mation literacy standards are remarkably similar to the standards of critical think- ing they try to impart in freshman English. This realization leads to a shift from asking how the library can provide bet- ter service to freshman English to asking how the two programs can extend their collaboration to ensure that all of the university’s first-year students get a re- ally effective introduction to this ap- proach to research. And having started this conversation across the hundred feet of Smith Plaza, the authors wonder how it can be expanded across the whole cam- pus and the whole range of the liberal arts, moving these fundamental strate- gies of inquiry from the edge of the first- year curriculum to its center. Notes 1. “American Library Association, Presidential Committee on Information Literacy, Final Report.” Web page. Available online from http://www.ala.org/acrl/nili/ilit1st.html. 2. “Association of College and Research Libraries, Information Literacy Competency Standards for Higher Education.” Web page. Available online from http://www.ala.org/acrl/ilcomstan.html. 3. Barbara Fister: “Teaching the Rhetorical Dimensions of Research,” Research Strategies 11, no. 4 (Fall 1993): 211-219. 4. Carol Collier Kuhlthau, Seeking Meaning: A Process Approach to Library and Information Services (Norwood, NJ: Ablex Pub Co, 1993). 5. Lois M. Pausch and Mary Pagliero Popp. “Assessment of Information Literacy: Lessons from the Higher Education Assessment Movement.” Presentation at the Association of College and Research Libraries 9th National Conference, Detroit, Michigan, 8-11 April 1997. Web page. Available online from http://www.ala.org/acrl/paperhtm/d30.html. 6. Wanda Martin. “Outcomes Assessment as a Teaching Tool,” in The Writing Program Ad- ministrator as Researcher, ed. Shirley K. Rose and Irwin Weiser (Portsmouth, NH: Boynton/Cook Publishers, 1999), 40-51. 7. “Association of College and Research Libraries, Information Literacy Competency Standards for Higher Education.” Web page. Available online from http://www.ala.org/acrl/ilcomstan.html. 8. Kathleen Collins, Bee Gallegos, Dennis Isbell and Lisa Kammerlocher. “Assessing Infor- mation Competency.” Workshop at the Association of College and Research Libraries 10th National Conference, Denver, Colorado, 17 March 2001. 9. “Lowell Research Cycle Rubric.” Web page. Available online from http:// www.bham.wednet.edu/mod8low.htm, used to evaluate research projects at the Bellingham Public Schools, WA. 10. “Ideas for Incorporating Information Literacy into Writing 150.” Web page. Available online from http://www4.gvsu.edu/infolit/CourseEgs/Eng150KR.htm, includes scoring cri- teria for an English class assignment at Grand Valley State University, MI. 11. “Rubric for Literature Review - Fall 2000.” Web page. Available online from http:// www.west.asu.edu/kwetzel/EMC675/AnBioRu3.html, used to assess an Educational Media and Computers class literature review.