College and Research Libraries Literature Reviews and Inaccurate ~eferencing: An Exploratory Study of Academic Librarians Peter Hernon and Cheryl Metoyer-Duran Using focus group interviewing, this article explores the vie·ws of the Associa- tion of Research Libraries (ARL) librarians about the literature of library and information science. The article touches on how these librarians do their literature searches and conveys their opinions about the professional literature. In conducting literature searches for work-related activities and the preparation of conference papers and manuscripts for journal submission, they tend to search the literature selectively for timely articles. They have ol1served instances of inaccurate referencing in the published literature, even in works of noted authors. A number of the librarians value the literature of subject disciplines more than they do that of library and information science. The article concludes with observations from selected former journal editors and current editorial board members, and identifies topics meriting further study. BACKGROUND: CHARACfERIZATION OF PUBLISHED WRITINGS The literature review, according to many research methods textbooks, places the problem statement in the con- text of previous knowledge, identifies variables that previous investigations have found either significant or insignif- icant, and suggests factors to consider in setting the research design and methodology. The references or foot- notes in published articles tend to in- clude only key, not necessarily all, relevant works in part because some journals do not have the space to publish detailed literature reviews as an intro- duction to an article. The determination of which works to reference is a judgment call. According to G. Nigel Gilbert, authors make in- formed guesses about which sources the intended audience will regard as authori- tative and persuasive.1 Manfred Kochen believes that a paper should "cite every past publication to which it owes an intel- lectual debt."2 The references, among other things, indicate "the author's ac- tual sources of ideas, which may not be the true origin of the idea." They also direct "the reader to further informa- tion" and meet "others' expectations about the content of a scholarly paper.''3 Kochen notes that authors of scientific articles often do not acknowledge their intellectual debt, that the list of refer- ences may contain major omissions or Peter Hernon is a professor at the Graduate School of Library and Information Science, Simmons College, Boston, Massachusetts 02115. Cheryl Metoyer-Duran works at the Graduate School of Library and Information Science, University of California, Los Angeles, California 90024-1250. Tlze investiga- tors would like to thank the Emily Hollowell Research Fund, Simmons Coll<'ge, for travel money to support data collection. 499 500 College & Research Libraries questionable inclusions, and "that at most 10 percent of what is published is a genuine contribution to knowledge."4 Inaccurate references (regarding the mis- spelling of authors' names and initials, in- complete or misleading titles of works, use of abbreviations subject to variations, wrongly cited volume or edition numbers, and incorrect pagination and publication years) plague the sciences and may inhibit the retrieval of potentially useful works.5 Gerald De Lacey, Carol Record, and Jenny Wade found instances in which authors misquoted. They concluded that "inac- curate quotations and citations are dis- pleasing for the original author, mis- leading for the reader, and mean that untruths become 'accepted fact."' 6 According to Jay J. J. Christensen- Szalanski and Lee Roy Beach, some people rely on summaries of research and "do not closely examine the research and results." 7 They might examine and cite, for instance, summaries contained in Dissertation Abstracts, rather than the dissertations themselves. Robert Broadus suspects that some authors ''lift their bib- liographic references from other publica- tions without consulting the original sources,"8 while Michael J. Moravcsik and Poovanalingam Murugesan discovered "redundant references-namely, situa- tions when a reference is made to several papers, each of which makes the same point." 9 Authors might even "select cita- tions to serve their personal goals ... or to advocate their favored hypothesis." 10 Blaise Cronin surveyed psychology journal editors and members of the edi- torial advisory boards. He found that they believed: • Authors frequently fail (intentionally and otherwise) to cite all pertinent work (87%); · • Authors tend to cite those whose views support their own (85%); • References are an expression of intel- lectual indebtedness (58%); • Journal editors and referees could do more to ensure standardization in ci- tation practice (65%); and • Referencing is one way in which the scientific community distributes rec- ognition (95%).11 November 1992 Based on these findings, Cronin wondered: To what extent it is reasonable to expect referees to be alert to all over- sights, omissions, misrepresentations, and instances of undercitation is hard to say, but since 87 percent agreed that authors frequently failed to cite all rel- evant work, then clearly there are cases where the wayward author can be instructed in the correct approach.U James Sweetland concurred. Editors and editorial board members associated with refereed journals have a responsi- bility for detecting and correcting prob- lems with the references of papers submitted for publication. 13 Given the breadth of the literature on inaccurate references (much of which is anecdotal and drawn from the biomedical .literature), editors and editorial board members must give references more than superficial attention. An examination of the letters to the editor appearing in College & Research Libraries, Journal of Academic Librarian- ship, Library & Information Science Re- search, Library Quarterly, and RQ from 1980 through mid-1991 revealed in- stances in which the writers claimed that articles contained inaccurate references. There were instances in which authors perhaps missed relevant writings, cited items that could not be located, apparently drew incorrect conclusions from a study, and printed citations that contained mis- takes, including typographical errors.14 Cronin refers to citation as "a private process ... albeit a private process with a public face. The essential subjectivity of the act of citing means that the reasons why an author cites as he does must re- main a matter of conjecture."15 Further- more, "this privateness ... invests citation relationships with frequent biases." 16 A citation may be the product of a literature review and a decision about what source(s) to reference. Even when authors prepare manuscripts for possible publication, their review of the literature might not involve an extensive or exhaus- tive examination of reference sources and the published writings. One might suspect, though, that those seeking pub- lication draw more on the literature than those who do not. Given the breadth of the literature on inaccurate references (much of which is anecdotal and drawn from the biomedi- cal literature), editors and editorial board members must give references more than superficial attention. Key questions are: • What guidelines do editors of refereed journals transmit to reviewers in eval- uating papers? • How extensively are references checked and presumably revised prior to pub- lication of a paper? • How much time can/ should review- ers devote to refereeing papers? • Why is the role of citations not "taken very seriously by the scientific com- munity?"17 A more basic issue relates to whose responsibility it is to minimize the rate of errors in citations. Sweetland observed: While some complaints are routinely made, there is little consensus even as to who is responsible for correcting citations. Publishers [journal editors?] seem to feel it is up to the author(s) to provide correct citations; the authors seem to feel it is up to referees to double-check them. 18 Clearly, published research focuses on citations themselves as opposed to the literature review or the process by which individuals gather the source material from which they might eventually cite individual works. PROBLEM STATEMENT Numerous studies have focused on the outcome of a literature review, and a few studies have probed librarians' use of internal (e.g., interpersonal contacts) and external (e.g., journal subscriptions and articles read) sources for decision making.19 Published studies, however, have not focused on the literature reviews that academic librarians conduct for them- selves through the literature of library Study of Academic Librarians 501 and information science (LIS) for fact- finding or gathering insights useful for the evaluation of library services, collec- tions, and programs, or in preparation of manuscripts intended for oral presenta- tion at conferences or for publication. If library and information science is to advance as a scholarly field, and further justify the position of its programs within college and university graduate schools, the quality of the research, theoretical, and scholarly literature of the field must increase. Such a study suggests the role and value of the professional literature for academic librarians and identifies topics requiring investigation. If library and in- formation science is to advance as a scholarly field, and further justify the position of its programs within college and university graduate schools, the qu- ality of the research, theoretical, and scholarly literature of the field must in- crease. Academic librarians as well as other librarians must regard their litera- ture as important to their professional development and decision making. Clear- ly, research must examine the impact of professional literature on the practice and development of librarianship. STUDY OBJECTIVES The primary objectives of this explora- tory study were to identify: • Perceptions about LIS literature, its quality and value; • Perceptions about the list of references appearing in LIS journal articles--their adequacy (extent to which all major stud-ies of direct relevance are included) and extent of inaccurate referencing; • How librarians conduct literature re- views and the extent to which they search the literature comprehensively; and • Suggestions for ways to make the con- duct of literature reviews easier while improving access to more source material. 502 College & Research Libraries A secondary objective is to elicit the perceptions of former journal editors and current editorial advisory board members about the adequacy of refer- ences provided by those submitting manuscripts for possible publication. This study examines these objectives primarily as a means to identify re- searchable topics and to direct attention to the role and value of literature reviews and references for placing studies within a broader perspective. STUDY DESIGN AND METHODOLOGY Focus Group Interviews Given the purpose of the study-to address the objectives and identify top- ics for further research-the investiga- tors conducted a series of focus group and individual interviews during the spring and summer of 1991. Group inter- views took place with librarians at five academic institutions that are members of the Association of Research Libraries (ARL). Located in the Midwest, South, Southwest, and West, the libraries se- lected were willing to participate in the study and were geographically acces- sible to the investigators. Three of the uni- versities had graduate schools of library and information science accredited by the American Library Association. Two uni- versities were selected because they did not have such schools, although such schools did exist in each state. Focus group participants included a cross-section of library faculty members: administrators and nonadministrators, public and technical services librarians, those who have been published and those who have not, those new to the profession, and those with years of ex- perience. To guide the discussion, the investigators framed the primary objec- tives as questions. The investigators shared the insights gained from pre- vious interviews with focus group par- ticipants in order to make general comparisons and probe unique areas. A total of forty-three academic librari- ans participated in the focus group inter- views, which generally lasted sixty to ninety minutes. These interviews pro- November 1992 duced qualitative data.20 As such, the in- vestigators were able to probe some simi- larities and differences among participants at the same and different institutions, get feedback on topics for further research, and develop insights into participants' views of library literature and the con- duct of literature reviews. Upon comple- tion of the focus group interviews, the investigators met individually with some of the participants to probe certain observations and experiences. The investigators assured participants that their comments would be kept con- fidential and not be attributed to either a particular institution or individual. Dur- ing interview sessions, the investigators took brief notes summarizing the discus- sion. Later the same day, they reviewed the notes, expanding on the points made by participants. The notes from the group and individual interviews were analyzed together, with the results re- ported in this article. Additional Data Collection In addition to interviewing academic librarians, the investigators also queried three former editors of library and infor- mation science journals and six current members of editorial advisory boards.21 Data collection complemented part of the focus group interviews and pre- sented different perspectives. QUALITY OF THE DATA To increase the reliability of the data collected, the investigators conducted a pretest with Simmons College doctoral students who had worked as academic librarians, and with six librarians at one · ARL library who participated in a focus group interview. The purpose of the two pretests was to preview the procedures for conducting the interviews and tore- view the study's objectives. Further- more, the investigators recorded the responses of the participants, compared their notes, and produced a detailed summary of each focus group interview. A few of the forty-three librarians in- terviewed shared copies of sample litera- ture reviews they had conducted. This way, the investigators could compare what these librarians had said with what they had written. Furthermore, because the research reported in this article builds - from a previous study, the findings of that study can be compared with those made during the conduct of this one.22 The six librarians reminded the investigators that library and information science is 11a practice-oriented profession and is not research based." The more important literature, therefore, must be 11usable and understandable." . The internal validity of the data was enhanced by matching questions within and across the group interviews. 23 Furthermore, the investigators periodi- cally summarized oral comments and asked participants to offer any correc- tions or supplementary information. The investigators sacrificed generaliza- bility of study findings to increase the ·study's reliability and internal validity, to identify propositions meriting further research, and to obtain detailed insights into the phenomenon under investiga- tion. There was insufficient funding to conduct a study that insured broad ex- ternal validity. FINDINGS Academic Librarians This section summarizes the findings for each of the five case studies, while the summary section discusses common themes in relationship to the four objec- tives. The final section briefly highlights the results of the data collection from the journal editors and editorial advisory board members. Library A. The six librarians at this university do not conduct exhaustive lit- erature reviews, even for papers that they prepare for possible publication. In- stead of searching indexes, they use sources found in their offices or homes. they draw on reprint collections, browse issues of easily accessible LIS journals, and perhaps consult published annual reviews of the literature. One rationale Study of' Academic Librarians 503 offered for adopting this approach is "a lack of time affects what I do. I do not have the time to do detailed research." They find that journal editors rarely question their list of references. This practice, they believe, validates their ap- proach to locating those titles which they reference. "Why tinker with success, if nobody challenges . your selection of works to reference?" Half of the librarians have a science background and work with scientific lit- erature. In comparison to the scientific literature, they find the indexing for li- brary and information science to be "in- ferior." They have found the subject headings in Library Literature to be too general. As a result, a subject search pro- duces too many "false drops." They would like a replacement for this index that in- cludes fugitive or gray literature and that offers more subject access points. They suspect that librarians tend to reference opinion pieces and that when many authors have a choice between ref- erencing a research study or a summary of that study, they cite the latter. They also have noticed "numerous instances" in which different journals publish the same work. They noted the fragmentation of the literature of library and information science. For example, a search for source material on artificial intelligence and knowledge-based systems in libraries, they thought, is best approached in- directly from the computer science litera- ture and INSPEC (Institute of Electrical Engineers, 1969- ). They suggested that editors might demand longer lists of references and, for each volume, publishers might issue a companion microfiche that includes the results of a complete search for perti- nent literature. The printed papers might include the following statement: "Sup- plementary material available on micro- fiche." The supplementary material might even reprint the survey instrument. The six librarians reminded the inves- tigators that library and information science is "a practice-oriented profession and is not research based." The more important literature, therefore, must be 504 College & Research Libraries "usable and understandable." They ex- plained that many librarians and library school educators do not write well. These authors might have conducted a more thorough literature review or bet- ter study than is apparent, but they may not know how to present their findings and arguments effectively and as briefly as possible. Library B. The six librarians at this university, none of whom is in a manage- rial position or writes for publication, do not draw extensively on the literature of library and information science. They might consult the literature selectively and scan a couple of journals in their area for titles meriting inclusion in the collection and for occasional articles re- lated ~o their work. They professed to be "client oriented" rather than "personally or scholarly oriented." When they con- sult the general LIS literature, it is to browse the "newsy" information con- tained in American Libraries . They do not want to expand their knowledge of librar- ianship in general, academic librarian- ship per se, or even public or technical services. For current and timely information, they prefer to consult their invisible col- lege (colleagues within or outside the institution) by in-person or telephone conversations, or through the use of elec- tronic mail (e-mail). This information must have practical utility; it cannot be theory-related. They like to use e-mail because they get the opportunity to com- municate with the leaders of the profes- sion; "We see the human side of the names of the profession." They have seen numerous instances of incorrect referencing but surprisingly were not concerned that the same could occur with e-mail. They do not expect articles to report the results of compre- hensive literature reviews. They want "sufficient referencing" to ensure that the articles are not redundant. The refer- encing should be to practical works rather than theoretical studies and other writings lacking practical utility. As they explained, "We are not scholars; we are librarians! Theory is for LIS faculty and doctoral students, not for us." November 1992 The periodical literature, in their opin- ion, stresses "how-we-do-it-here" arti- cles. Such writings may be useful for library school students, new librarians, and those assuming new responsibili- ties. However, they surmised that the availability of electronic bulletin boards will lessen the need for such articles to be published. Although they do not have much need for LIS literature, the six librarians believe the library should collect it for two reasons. First, there is a library school on campus, and, second, they suggest that students in subject disciplines often need access to LIS literature. "Our literature has the best coverage of topics such as the his- tory of libraries, privatization, and free- dom of information acts." They do not read these writings, however. Rather, they want them available for student use. When told that the best literature on topics such as privatization and freedom of information acts may not appear in the library and information science lit- erature, they challenged the accuracy of this statement. When told that other lit- eratures may not cite LIS writings on these topics, they exclaimed that "this is their loss." There is a clear contradiction within their beliefs about the value of LIS literature. Without exception, they believe that Library Literature is inadequate. Because it did not index a number of journals and because a work received "too few subject headings," it was possible to miss rele- vant works. However, they would prefer better indexing for their clientele-stu- dents-not for themselves. The focus group interview, and sub- sequent in-person interviews, could not reveal the source of their discontent with library and information science lib~ra­ ture for their own use. Perhaps their atti- tudes can be traced back to library school. On the other hand, they might be offering a rationale for coping with in- formation overload or their work may, indeed, never (or rarely) require the use of LIS literature. Library C. The ten librarians at this university distinguish between work-re- lated and scholarly information needs. For the former (e.g., source material on moving CD-ROM workstations), they would do exactly the same as the librar- ians at Library A: conduct a selective review of the literature to find a couple of relevant articles, perhaps ones explain- ing "how we did it." The only exception to this might be when a supervisor occasion- ally requests detailed background infor- mation and wants a complete search to be performed. However, unlike their peers at the other institutions, they conduct a thorough review of the current and ret- rospective literature when they prepare a speech or manuscript. They want to reference the literature extensively in their presentations and be able to field any question that might arise. A recent M.L.S. graduate explained that she conducts thorough literature re- views because she wants to know what has occurred previously and does not want to waste her time. ''I can better focus on the problem I want to investigate, while at the same time clarifying the approach I'll take." A reference librarian said that "we must practice what we preach; if we teach students to conduct a thorough search of the literature, we should do the same for ourselves." In searching for writings in support of work-related activities and research, public service librarians might conduct online and/or CD-ROM searches, exam- ine their personal collection of xeroxed works, or browse selected journals. Tech- nical service librarians would not conduct online or CD-ROM searches. Instead, they might browse bibliographic essays such as those appearing in "The Year's Work ... " in Library Resources & Technical Services, but only when they either need quick access to source material or want to in- clude older writings. The librarians might also scan e-mail, have journals routed to them, and pho- tocopy articles. In part, they try to com- pensate for the limited indexing of Library Literature. Scanning tables of contents and articles is essential if they are to cope with information overload. One librarian regu- larly changes her routing profile. She likes to sample different library and nonlibrary journals. She is constantly searching for Study of Academic Librarians 505 writings that might be useful for the com- pletion of a current or potential study. These librarians try to sift through vast quantities of writings to find the few most relevant to their needs. They might chance across an article and use it for an unintended purpose. "We make the most of what we find." Two librarians delegate literature searching to a student assistant enrolled in the university's school of library and information science, who conducts the search under the supervision of a'reference librarian. The two librarians then review the results. Three of the librarians have delegated responsibility for conducting the literature review to a coauthor. The library at this institution has clear guidelines covering promotion and con- tinuing status. For continuing status, the librarians are evaluated for "position ef- fectiveness, scholarship, and service." Al- though position effectiveness is the most important factor, the librarians must supply "evidence of contributions to the field of librarianship" by itself or in com- bination with "other academic discip- lines." They can meet the expectations for scholarship by delivering "presenta- tions at conferences and meetings," by being published, or by "other creative projects that benefit the library or the profession." In-house bibliographies count as service, not scholarship, and the text of any presentation must be sub- mitted to the review committee. The cri- teria for promotion, as well, recognize scholarship and publication. The librarians receive twenty-four days of professional leave per year to attend conferences and work on scholarly projects. It is tempting but undoubtedly inaccurate to attribute the desire to be published and to conduct exhaustive lit- erature reviews solely to the above-men- tioned guidelines. The librarians perceive themselves as self-motivated and believe that it is important to advance LIS as a discipline and to enhance the professional image of the field. As one explained, "It is essential to publish and deliver speeches. This is the scholarly model. To claim to be faculty or scholars, we must follow this pattern." 506 College & Research Libraries Library D. The sixteen librarians at this university, both managers and non- managers as well as tenured and un- tenured faculty, find library literature to be repetitious, poorly written, and hard to read. 24 They often experience prob- lems in extracting major ideas from stud- ies and in following the logic of a presentation, especially one using statis- tics. More literature, in the opinion of some, should apply theory to practice. Others prefer a practical literature, one having utility for decision making and planning. Too often, they all agreed, the literature reports a study investigating a narrow problem at a particular library. Such case studies may omit or gloss over key information concerning resources and costs, and thereby offer "question- able" recommendations and conclu- sions. Furthermore, the insights gained from a study are "rarely" transferable, lacking generalizability within the insti- tution or across institutions. Whether using the literature to under- stand something, gain insights for deci- sion making and planning, or writing for publication, they search online and CD-ROM databases, including Library Literature on CD-ROM, as well as browse recent issues of selected journals. Occasionally, a few of them will search, for instance, the ERIC database, data- bases of the National Library of Med- icine, INSPEC, UnCover which lists the table of contents for journal literature contained in CARL (Colorado Alliance of Research Libraries), and Nursing and Allied Health (Cumulative Index to Nursing and Allied Health Literature).25 Those who have searched Library and Information Science Abstracts have found it to be "dis- appointing; the sources listed are diffi- cult to acquire." They do not attempt comprehensive or exhaustive searches, even when writ- ing for publication. Rather, they seek current articles. "Why search prior to the placement of Library Literature on CD- ROM? We do not conduct historical stud- ies," noted one librarian. This person also commented that "chemists and other scientists do not do retrospective searching," that she will "uncover the November1992 key, current writings," and that she wants access to "timely" articles and in- formation. After all, "we deal with a different set of problems than those prac- ticing librarianship in past years." Focus group participants maintain that there is insufficient time at work to conduct a more thorough search. Yet they find time to read the literature of one or more subject disciplines and to identify key writings from these discip- lines for their colleagues' use. Two librarians noted that "it is more produc- tive for us to know the literature of sub- ject disciplines so that we can speak the language of those disciplines and inter- act with those faculty and students." These librarians also commented that their writings have appeared in different journals within those disciplines. A number of the librarians prefer to identify the "major authors in a field, such as reference and government docu- ments, and to read and reference their works." These individuals serve as opin- ion leaders and exert enormous in- fluence. One librarian noted that history, management, education, and some other disciplines have "well-developed and opposing schools of thought." Library and information science, "except for bib- liographic instruction and perhaps a few other fields," lacks such schools. In some instances, this may complicate knowing what and whom to reference. Some of the librarians were willing to pursue articles not locally held. They might call colleagues at other institu- tions to mail or fax copies. They tend to avoid the use of interlibrary loan (ILL) because the service is "too slow." Most of the focus group participants, how- ever, rely exclusively on the immediate collection, which is limited since the in- stitution does not offer a program in li- brary and information science. Finally, similar to those at Library C, the librarians here have promotion and tenure guidelines requiring the presen- tation of papers at conferences and "some" publication. The untenured librarians "feel pressure" to publish and wish that their master's program had better prepared them to identify re- searchable problems and to conduct a research study. Neither the professional association in the state nor the two li- brary schools had undertaken programs to educate them in research methods, statistics, and other parts of the research process. Library E. The five librarians at this university, all of whom hold administra- tive positions, believe that the quality of research reported in LIS journals is im- proving, but is still substandard in com- parison to other fields. There is too much quantity and too little quality. And, at times, they find the quality works diffi- cult to uncover. Part of the problem, they suspect, is that the existence of so many LIS journals inhibits exercising quality control. When two of them have shown LIS journals to university administrators and faculty in other departments, these individuals dismiss most of the reported studies as being poorly written and re- searched. The faculty have noted in- stances in which the inclusion of other variables in an experiment might have affected the conclusions drawn. The focus group participants find that many librarians have not been trained as researchers, but that the library (and the university) might expect them to conduct research-"publish or perish." Complicat- ing matters, one of them suggested, many academic librarians do not know their own research literature. "In fact, there is little incentive for them to learn it." The others added that many librarians may know the practically oriented literature but only in their particular area. Two of them stressed that "as professionals, it is important to know both the practical and theoretical literature." When journals accept for publication papers containing few references or fail.; ing to include writings from other dis- ciplines, editors reinforce the belief that "it is fine to conduct limited literature reviews and omit key writings." There should be greater expectations, one librarian explained, for authors to place their research in historical context. "Too often, we encourage an ahistorical ap- proach to problem solving." Study of Academic Librarians 507 The librarians at this institution browse the table of contents of selected journals for potentially useful and inter- esting articles, and they peruse book re- views. They might also ask colleagues for recommendations, while those in technical services, like their counter- parts at Library C, rely on ''The Year's Work .. . " columns for the identification of writings they might otherwise have missed. One librarian supplements the perusal of selected journals with news- letters providing current information, including the names of people working on specific issues. None of them consult indexes. The library contains a small LIS collec- tion. If there is time, a need, and the material is conveniently located, two librarians might go to a nearby univer- sity housing a graduate LIS program; otherwise, they will either do without the additional source material, use ILL, or ask a colleague at another institution to mail or fax articles. One librarian sub- scribes to a number of LIS journals and is a resource person for colleagues in her division. When asked about how to improve the conduct of literature reviews and lessen the occurrence of inaccurate referencing, they responded, "there is a need for librarians to be trained in research methods and for libraries, LIS schools, and professional associations to share this responsibility." They would also like to see journals adopt greater rejection rates and higher standards for manu- script acceptance: better writing, more descriptive and interesting titles, the in- clusion of key theoretical and other works, and · the referencing of fewer trivial studies. SUMMARY Drawing upon the five case studies, this section compares the focus group participants' insights regarding the four primary objectives: • Perceptions about US literature, its qual- ity and value. The literature empha- sizes practical or how-to articles, and is of uneven quality. 26 In this respect it is like that of other disciplines, fields, 508 College & Research Libraries and professions.27 Opinions about the value of the literature varied greatly among those interviewed. (Perhaps the culture of a particular library in- fluences how the librarians there use the professional literature.) The librar- ians at Library Bare at the negative extreme; they tend to dismiss the lit- erature as having little value to them personally. Perhaps the value is greatest to librarians who are expected to provide evidence of their own scholarship. • Perceptions about the list of references ap- pearing in LIS journal articles-their ade- quacy (extent to which all major studies of direct relevance are included) and extent of inaccurate referencing. Some librari- ans assume that the authors of articles appearing in prestigious LIS journals conduct comprehensive literature re- views and report all key works. "How else could the references of their papers pass a journal's strict referee- ing process?"28 A number of the librar- ians, however, recognize that pub- lished studies omit key works and make erroneous statements about pre- vious writings. They suggested that some journals, more than others, are likely to have inaccurate referencing. Too often, a problem is placed exclu- sively within a local context, without recognition of the broader ramifica- tions and literature. Long production schedules affect the timeliness of an article's refer- ences. However, this is something that the librarians understand and accept. Some of the librarians believe that authors tend to reference works that support a particular viewpoint and omit opposing viewpoints. At times, there may be a lack of "se- rious work" on a topic. "It is important for editorial boards to recognize this, and that it may be appropriate to refer- ence anecdotal evidence," said one focus group participant. The assumption of those inter- viewed is that the conduct of in- complete or faulty literature reviews, as well as inaccurate referencing, is more the fault of authors than it is of November 1992 journals and their editorial boards. On the other hand, those interviewed tend to believe that editors must clearly explain their policies, know the literature, and expect their reviewers to know it as well, and, if possible, offer guidance for improving the qu- ality of a paper, both substantively and stylistically. Focus group partici- pants would like a better written and more readable literature. Some librarians overlook appro- priate writings when they conduct their literature reviews. These writ- ings might be "hard to understand, dense, and contain statistics; we prefer a 'usable literature."' Regarding whether or not the insti- tution has a graduate school of library and information science, a number of librarians consider accessibility, ease of access, and understandability of the information as the primary factors in- fluencing what they include as refer- ences. Some of them search the writings of the major authors in a field. • How do librarians conduct literature re- views and how comprehensive is their search of the literature? The answer needs to distinguish between informa- tion needs related to daily work and scholarship. Literature reviews for the former tend to be more selective. Only the librarians at Library C suggest that they conduct thorough literature re- views for publications and presenta- tions. The other librarians are selective. They may rely on what others cite and what they can turn up from a cursory examination of the current literature. Technical services librarians may con- sult annual bibliographic essays and review articles appearing in the jour- nal literature. Some librarians noted that even dis- tinguished authors make mistakes. A couple of librarians speculated that students conduct the literature re- views for some well-known authors and that these authors might not have examined the results. • Suggestions for ways to make tlte conduct of literature reviews easier while im- proving access to more source material. Librarians at three institutions recom- mended replacing Library Literature with an indexing and abstracting ser- vice that provides more comprehen- sive coverage and that places writings under more subject headings. Some expressed the need for a reminder that sources such as Resources in Education (RIE) might contain relevant works, while others questioned the value of RIE given the uneven quality of the papers appearing in the database. Yet many of the focus group participants do not anticipate any change in their method of gathering and reviewing the literature. They will limit their search to resources (e.g., journals) lo- cally held and easily accessible. JOURNAL EDITORS AND EDITORIAL ADVISORY BOARD MEMBERS The editors and the advisory board members all reiterated that the refer- ences of papers submitted for publica- tion often exclude key works and that inaccurate references present a serious pr~blem. However, neither the staff nor time exists to check every single reference. There was some disagreement about whether or not every referee can be an authority on the topic of the paper under review.29 One former editor speculated that "a journal limiting its pool of referees to members of the editorial board may not get as thorough analysis of papers as jour- nals pulling in additional reviewers." According to another former editor: I found they [the lists of references] were good to excellent from faculty and practically nonexistent from the field .... I routinely suggested addi- tional references when I was an editor to shore up articles, but this worked only when I or the referees had exper- tise in the area written about. I didn't have time or resources to do more ... Probably our standards for literature reviews are minimal, and it may well be time to develop a study that draws attention to this breakdown of the bib- liographic net. The other two editors echoed the differ- ence in referencing between LIS educa- Study of Academic Librarians 509 tors and practicing librarians. One of them stated that "active researchers did better referencing-included the major works-while practitioners often were unfamiliar with the names of persons who had made a contribution in a partic- ular area." Perhaps the reason is that "the literature ... [and referencing are] not integrated into practitioners' daily work." This. person commented that her "best reviewer" was a doctoral stu- dent in a LIS program who "kept on top of the literature on technical serv- ices." Another former editor noted that the references of papers submitted for publication often did not draw on "re- lated literature." TOPICS FOR FURTHER INVESTIGATION Faced with information overload and the need to peruse journal issues and articles to find a few potentially useful articles, a number of librarians first scan tables of contents. Next, they consult the abstracts of selected articles. If an article has potential value, they might read the introduction and conclusion. In a small number of cases, they might read the en- tire article, unless, as a couple oflibrarians noted, it contained "numbers and was dense reading." Clearly, researchers, to- gether with graduate programs of library and information science, should develop ways to make research methods and statis- tics more understandable and widely used. Other topics meriting investiga- tion include: • What image of library literature do we cultivate in library schools? Does the image of Library Literature presented by instructors of reference courses guarantee that upon entering the pro- fession, the new librarians will avoid that index whenever possible? • How can the profession encourage greater participation in intellectual en- deavors and in producing high-quality articles? How can research appreciation and statistics be best taught as non- threatening endeavors? • How would the findings of this study compare to investigations of librari- ans in other ARL, as well as non-ARL, 510 College & Research Libraries settings? Studies might also examine library school educators and other groups contributing to the literature. Replication of the type of research pre- sented in this article might concentrate on focus group or survey participants from the "most productive institu- tions," those identified in studies that have examined patterns and characteris- tics of authorship by academic librari- ans.30 Do these individuals offer unique insights that might serve as a model for other academic librarians? Research might also compare responses of librar- ians at institutions which either grant or deny them faculty status.31 • How can data collection elicit some- thing other than self-reporting data? Studies might ask participants, for ex- ample, to keep diaries or to be ob- served in conducting a literature search. Other studies might set up an independent panel to review and rate the references in published papers. • Is there a difference in citation patterns, includinginaccuratereferencing, between refereed and nonrefereed journals? • From examining ILL records of librar- ies with graduate schools of library and information science, can it be de- termined which US titles are borrowed, by whom, and for what purpose(s)? As noted at Libraries D and E, though, librarians, like many others, may bypass ILL and directly contact colleagues at other institutions.32 • How many research and other articles contain sections entitled "Literature Review"? What are the characteristics of the reviews? Are they descriptive or analytical? What types of works are cited (e.g., summaries of research stud- ies or the studies themselves)? • What secondary services do librarians . rely on for conducting literature re- views, and should they consult addi- tional ones, e.g., Information Science Abstracts and the online database Trade & Industry Index (Information Access Corp.)? Is it important to reference works produced in other countries? • What are the perceptions of most jour- nal editors and editorial advisory board members about the adequacy of November 1992 references contained in papers sub- mitted for publication? Undoubtedly many journal editors will not open their editorial review process to outside scrutiny. If they did, however, it would be interesting to conduct a content analysis of reviewer comments and to trace editorial decision making. CONCLUSION In carrying out a literature review, many academic librarians rely on source material that is convenient and easily understood. When journals accept the outcome of such a review, these librari- ans come to believe that their approach is acceptable. After all, they think they have turned up the "key" literature. If they have overlooked an important work, they might find that editorial boards will let them know about the ex- istence of that work. They might then have an opportunity to reference that work in a revised paper. With the vast number of US journals in-existence, authors have choices as to where they submit manuscripts. Some journals have high rejection rates and long periods of time before publishing accepted manuscripts, while other jour- nals have low rejection rates and have fewer manuscripts from which to select publishable articles. Furthermore, many librarians prefer practical and anecdotal literature, and increasing circulation for many journals necessitates that they recog- nize and cater to reader preferences. All of this reinforces the notion that there might not be a dramatic change in the literature reviews which academic librarians conduct and the references they report. Nonethe- less, relevant questions become: • To what extent should literature reviews and referencing extend to writings from other disciplines and professions, and from English-language works written in other countries?33 • Will the rate of inaccurate referencing decrease? • What are the rewards for more complete referencing? • How can we increase the role and value of LIS literature for librarians so that they will regard the literature as essential for maintaining and furthering their professionalism? How can we in- crease the ambitions and scholarship of Study of Academic Librarians 511 librarians so that they will search sys- tematically for relevant literature and read it, even when it is dense? REFERENCES AND NOTES 1. G. Nigel Gilbert, "Referencing as Persuasion," Social Studies of Science 7:117 (Feb. 1977). 2. Manfred Kochen, "How Well Do We Acknowledge Intellectual Debts?" Journal of Documentation 43:54 (Mar. 1987). 3. Ibid. 4. Ibid., p.55. 5. See Robert K. Poyer, "Inaccurate References in Significant Journals of Science," Bulletin of the Medical Library Association 67:396-98 (Oct. 1979). He cites eight studies which have examined "the inaccurate reference" (p.396). See also James H. Sweetland, "Errors in Bibliographic Citations: A Continuing Problem," Library Quarterly 59:291-304 (Oct. 1989); and Bert R. Boyce and Carolyn S. Banning, "Data Accuracy in Citation Studies," RQ 18:349-50 (Summer 1979). 6. Gerald De Lacey, Carol Record, and Jenny Wade, "How Accurate Are Quotations and References in Medical Journals?" British Medical Journal291:884 (Sept. 28, 1985). 7. Jay J. J. Christensen-Szalanski and Lee Roy Beach, "The Citation Bias: Fad and Fashion in the Judgment and Decision Literature," American Psychologist 39:75 (Jan. 1984). 8. Robert N. Broadus, "An Investigation of the Validity of Bibliographic Citations," Journal of the American Society for Information Science 34:132 (Mar. 1983). 9. Michael J. Moravcsik and Poovanafingam Murugesan, "Some Results on the Function and Quality of Citations," Social Studies of Science 5:90 (Feb._ 1975). 10. Christensen-Szalanski and Beach, "The Citation Bias," p.77. For an excellent summary of the reasons for citing works as well as the difficulties in attributing motivation to referencing specific works, see Blaise Cronin, The Citation Process: The Rol'e and Signifi- cance of Citations in Scientific Communication (London: Taylor Graham, 1984), p.30-31. 11. Blaise Cronin, "Norms and Functions in Citation: The View of Journal Editors and Referees in Psychology," Social Science Information Studies 2:71 (Apr. 1982). · 12. Ibid., p.76. 13. Sweetland, "Errors in Bibliographic Citations." He notes that there is a "high percen- tage" of citation errors "in evidence from the mid-nineteenth century to the present" (p.291). He speculates on why errors might arise and wonders "why they are not discovered and corrected before publication" (p.297). 14. "Letters to the Editor'' appearing in RQ 28:287-88 (Winter 1988); RQ 27:594-96 (Sum- mer 1988); RQ 30:451-52 (Spring 1991); RQ 29:478-79 (Spring 1990); Journal of Academic Librarianship 6:286 (Nov. 1980); Journal of Academic Librarianship 9:294 (Nov. 1983); College & Research Libraries 50:701-2 (Nov. 1989); College & Research Libraries 45:317 (July 1984); College & Research Libraries 48:368-69 (July 1987); College & Research Libraries 52:296 (May 1991); "Letters to the Editor'' appearing in RQ 30:451-52 (Spring 1991); RQ 26:543-44 (Summer 1987); College & Research Libraries 43:243 (May 1982); "Letter to the Editor" appearing in College & Research Libraries 41:243 (May 1980); "Letters to the Editor" appearing in College & Research Libraries 47:175-76 (Mar. 1986); College & Research Libraries 51:280-81 (May 1990). 15. Cronin, The Citation Process, p.28-29 . . 16. Ibid., p.29. 17. Sweetland, "Errors in Bibliographic Citations," p.300. 18. Ibid., p.301. . 19. See, for instance, Charles R. McClure, Information for Academic Library Decision Making (Westport, Conn.: Greenwood, 1980); and Charles R. McClure and Alan R. Samuels, "Factors Affecting the Use of Information for Academic Library Decision Making," College & Research Libraries 46:483-98 (Nov. 1985). . 20. See David W. Stewart and Prem N. Shamdasani, Focus Groups: Theory and Practice (Beverly Hills, Calif.: Sage, 1990). ,?1. They were asked, "Have (had) you noticed instances in which potential authors omitted works central to the problem being investigated?" "Have (had) you noticed 512 College & Research Libraries November 1992 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. instances of inaccurate referencing?" "Does inaccurate referencing present a serious problem in the literature?" See Peter Hernon, "Academic Librarians and the Library and Information Science Monograph: An Exploratory Study," College & Research Libraries 55:507-19 (Nov. 1992). Internal validity assesses the extent to which data collection procedures actually measure what investigators intend them to measure. Two focus group interviews, each containing eight librarians, were held at this site. See Ward Shaw and Rebecca T. Lenzini, "UnCover: CARL's Article Access Solution," Bulletin of the American Society for Information Science 15:11-13 (June/July 1989). See Patricia E. Feehan, W. Lee Gragg II, W. Michael Havener, and Diane D. Kester, "Library and Information Science Research: An Analysis of the 1984 Journal Litera- ture," Library & Information Science Research 9:173-85 (July /Sept. 1987). They found that two-thirds of the research articles examined could be classified as applied (study practical situations) or dealing with professional concerns (librarianship as a profes- sion, e.g., status, salaries, and education). For a discussion of academic scholarship in general, see Carolyn J. Mooney, "Efforts to Cut Amount of 'Trivial' Scholarship Win New Backing from Many Academics," The Chronicle of Higher Education 37:A1, A13, A16 (May 22, 1991); and Carolyn J. Mooney, "In 2 Years, a Million Refereed Articles, 300,000 Books, Chapters, Monographs," Chronicle of Higher Education 37:A17 (May 22, 1991). For an identification of prestigious journals, see David R Kohl and Charles H. Davis, "Ratings of Journals by ARL Library Directors and Deans of Library and Information Science Schools," College & Research Libraries 46:40-47 (Jan. 1985). For a pertinent discussion, see Mary Biggs, "The Impact of Peer Review on Intellectual Freedom," Library Trends 39:145-67 (Summer/Fall1990). See, for example, John M. Budd and Charles A. Seavey, "Characteristics of Journal Authorship by Academic Librarians," College & Research Libraries 51:468 (Sept. 1990). See Betsy Park and Robert Riggs, "Status of the Profession: A 1989 National Survey of Tenure and Promotion Policies for Academic Librarians," College & Research Libraries 52:281-87 (May 1991). See, for example, Stephen E. Wiberley, Jr., and William G. Jones, "Patterns of Informa- tion Seeking in the Humanities," College & Research Libraries 50:635-45 (Nov. 1989); and Peter Hernon, Use of Government Publications by Social Scientists (Norwood, N.J.: Ablex, 1979). John Budd, "The Literature of Academic Libraries: An Analysis," College & Research Libraries 52:295 (May 1991); and Sydney J. Pierce, "Characteristics of Professional Knowledge Structures: Some Theoretical Implications of Citation Studies," Library & Information Science Research 9:143-71 (July/Sept. 1987). Pierce notes that "the differ- ences in age and format of materials cited and the lack of agreement on what items merit citing are indicative of a lack of consensus on the value of individual research efforts in the professions that lessens the value of research generally. The failure of a literature to develop scientific knowledge structures suggests a failure of knowledge to cumulate and build" (p.165).