College and Research Libraries The Readability of Published, Accepted, and Rejected Papers Appearing in College & Research Libraries Cheryl Metoyer-Duran This study examined the readability of papers that College & Research Libraries accepted, rejected, and published for 1990 and 1991. In addition to showing a statistically significant difference for the text of papers, but not for the abstracts, this investigation reports topics for further investigation and presents a pro- cedure for others to follow in replicating the study. n ndividuals conducting action research and evaluation stu-dies, and wanting library - managers to use the findings of these studies to produce change within the organization, must fit "infor- mation presentation formats to decision- making."1 Clearly, researchers and scholars must know the audience with whom they intend to communicate. A research paper that is difficult to read and comprehend is not likely to be read (and presumably published). Reada- bility offers insights into communica- tion in that it addresses whether an audience will "understand" a paper, read it "at optimum speed," and "find it interesting." 2 Readability, therefore, is one indication of the effectiveness of a piece of writing in conveying the author's intended message to the audience.3 READABIUTY FORMULAE As Marie J. Abram observes, The style of writing (or how the con- tent of the writing is stated) can be measured in such a way that a numeri- cal value can be assigned to each writ- ing style. This qualification of style is an entirely separate dimension from the content of the writing. The numeri- cal value that results from the mea- surement of style quantifies the ease or difficulty of the writing. With most formulas this numerical value has been translated into an educational skill level associated with the material (i.e., ... ninth grade level ... ):4 Abram further observes that "many readability formulas exist."5 Three of the better-known formulae, all of which are available on Grammatik (Reference Software International, San Francisco), include the Flesch Reading Ease, Gunning's Fog Index, and the Flesch-Kincaid Grade Level. Each con- siders the average number of words per sentence and the average number of syl- lables per word.6 Both the Gunning's Fog Index and the Flesch-Kincaid Grade Level formula measure level of education necessary to understand a source, or paper. The level of difficulty of a source increases as the grade level advances. Because scholarly literature requires a higher level of un- derstanding and attracts a specialized Cheryl Metoyer-Duran is the Rupert Costo Chairholder in American Indian History, Departments of History and Anthropology, College of Humanities and Social Sciences, University of California, River- side, California 92521-0132. 517 518 College & Research Libraries audience, a higher readability score may be acceptable up to a certain threshold. The Flesch Reading Ease score falls along a scale ranging from 0 to 100, with lower scores suggesting a more difficult to read source(s).7 In interpreting the Flesch-Kincaid Grade Level and Gunning's Fog Index, a researcher can equate increased read- ability with a lower grade level. However, application of the Flesch Reading Ease measure equates a higher level of read- ability with a higher (i.e., less difficult) score. THE STUDY The articles and features appearing in College & Research Libraries presumably require a higher level of education to understand than articles and features appearing in less scholarly and less re- search-oriented journals. Two questions are: "On an average, what are the levels of readability for the articles and fea- tures contained in specific scholarly and research journals?"; and "Has readabil- ity changed over time?" These questions, together with matters of journal content and policy, are most appropriate for an editor, editorial board, and publisher to address, especially in these times of infor- mation source proliferation and fiscal stringencies. Formal and informal reader- ship surveys ensure that journals under- stand subscriber and reader preferences and learn about these individuals and or- ganizations. Clearly, readability is an im- portant variable to investigate and address, especially if editorial staff and authors rewrite papers to accommodate a specified level of readability. The readability of scholarly or ref- ereed journals might be examined from another perspective. Is there a difference in readability between accepted and rejected manuscripts? Because the editorial staff of College & Research Libraries copyedits all ac- cepted manuscripts, two directional hy- potheses might be ventured: • The text of published papers is more readable than that of either accepted or rejected papers, and the text of ac- cepted papers is more readable than that of rejected papers; and November 1993 • The abstracts of published papers are more readable than those of either ac- cepted or rejected papers, and the ab- stracts of accepted papers are more readable than those of rejected papers. A basic assumption in this study is that most, if not all, of the papers reflect at least a college level education. How- ever, there is a point at which a higher level of difficulty suggests less readability. In effect, there are different shades of diffi- culty, ranging from most difficult to read (rejected papers) to less difficult (accepted but uncopyedited papers) and least diffi- cult to read (published papers). The Flesch-Kincaid and the Gunning's Fog Index will indicate differences in grade levels among the three categories of papers-accepted, rejected, and pub- lished. At the same time, the Flesch read- ing ease score will show differences in the level of reading difficulty. RESEARCH DESIGN The editor of College & Research Librar- ies supplied the investigator with copies of all manuscripts accepted and rejected during 1990 and 1991, excluding the names of the authors and associated edi- torial correspondence. During these two years, 82 refereed papers appeared in print, 70 papers were accepted but not yet published, and 119 were rejected. Given the hypotheses and the large size of a sample necessary to achieve a precision of + 5, with 95 percent confi- dence, the investigator examined all271 papers and did not draw a sample. The research design necessary to investigate the study's hypotheses required analysis of each abstract' and a random paragraph sampling, including the first and final par- agraph of each paper. The investigator numbered the unique paragraphs in each paper and, after counting the number of paragraphs, consulted the Appendix to select the actual paragraphs for word pro- cessing and statistical analysis.8 (The Ap- pendix has been reprinted in part to aid other researchers who intend to do read- ability studies but who do not want to develop their own schema.) Some 9 accepted and 26 rejected papers did not contain abstracts. An ex- Published, Accepted, and Rejected Papers 519 TABLEt GRADE LEVEL OF THE TEXT AND ABSTRACTS Flesch-Kincaid Gunning's Fog Index Mean Five-paragraph text a. Rejected papers 14.34 b. Accepted papers 15.27 c. Published papers 15.16 Abstracts a. Rejected papers 16.69 b. Accepted papers 16.38 c. Published papers 16.49 perienced word processor input the ab- stracts and text of the five paragraphs for each published, accepted, and rejected paper, exactly as presented in the sub- mitted or published paper, including spelling, punctuation, and grammatical errors. The investigator verified the ac- curacy of data entry, ran the Grammatik software on each word-processed ab- stract and five-paragraph file, and com- puted the scores for the Flesch-Kincaid Grade Level, Gunning's Fog Index, and Flesch Reading Ease.9 Next, the scores were entered into StatPac Gold (Walon- ick Associates, Inc., Minneapolis), a statistical analysis software package, and statistical analyses were performed to examine the hypothesis. LIMITATIONS The study does not examine the re- viewing process, the comments of refer- ees, and the decision rendered by the editor. An unaddressed question is ''To what extent does readability affect the decision to accept a paper for publica- tion?" Papers reviewed prior to 1990 were not examined, nor were papers submitted in 1991 for which an editorial decision was not rendered that year. Abram cautions that sentence length and word factors "do not cause reading ease/ difficulty. Rather they are highly correlated with reading ease/ difficulty. As such these variables can be used as indicators of changes that would reduce reading difficulty." 10 Highly readable Median Mean Median 14 18.48 19 15 19.41 19 15 19.21 19 16 21.36 21 16 21.08 21 17 21.07 21 writing may at times be boring to read because simple sentences may not fully convey the complexities of ideas ex- pressed in scholarly writing. 11 FINDINGS Table 1 depicts the grade level for both the five paragraphs from the papers and the abstracts. Although every indicator suggests a readability level of at least college, the Gunning's Fog Index pro- duces higher scores than does the Flesch- Kincaid Grade Level. Both measures do not consider the same number of sylla- bles per word (see note 6). Readability • . . is one indication of the effectiveness of a piece of writing in conveying the author's intended message to the audience. It appears that the first hypothesis concerning readability of papers is not supported since the scores for published and accepted papers are higher than those of rejected texts. Without applying the higher-ordered statistical tests used in the next section of this paper, the sec- ond hypothesis regarding the abstracts appears to be partially supported, as re- jected abstracts score higher than pub- lished or accepted abstracts. The table further indicates that abstracts require a higher level of readability than do the extracts from the text. 520 College & Research Libraries November 1993 TABLE2 FLESCH READING EASE SCORES Mean Median Low to High Score Five-paragraph text a. Rejected papers 30.77 31 9 to 53 b. Accepted papers 28.04 28 7 to 45 c. Published papers 27.56 29 1 to 46 Abstracts a. Rejected papers 18.43 18 0 to 47 b. Accepted papers 17.85 16 0 to 44 c. Published papers 17.93 17 0 to 50 TABLE3 MATRIX DEPICTING CORRELATIONS AND T-STATISTICS Flesch-Kincaid Flesch Reading Ease Gunning's Fog Index Grade Level Correlation t-statistics Correlation t-statistics Correlation t-statistics Five-paragraph text: . a. Accepted/ rejected papers .141 40.714 -.195 84.788 -.216 69.136 b. Accepted/ published papers -.027 32.842 -.048 66.402 -.064 51.774 c. Rejected/ published papers -.164 36.877 .159 91.711 .169 74.426 *The following Pearson's Product-Moment Correlations and t-statistics are all significant (p < .05). In table 2, the section on "low to high score" confirms that both the text and abstracts are "difficult" to "very diffi- cult" to read. However, beyond this simple statement, tables 1 and 2 are not comparable. The measures of grade level do not coincide with the categories rep- resented in the Flesch Reading Ease. The latter measure does not differentiate among precise years of college educa- tion. The data in table 2 do not appear to support either hypothesis. Hypothesis Testing Perusal of table 1 might suggest that the text of rejected papers has an appre- ciably lower reading level than does the text of accepted and published papers. Using analysis of variance (ANOVA) and the t-test, the investigator more closely examined the hypotheses con- cerning the text and abstracts of rejected, accepted, and published papers. The AN OVA for the five-paragraph text indi- cated statistical significance according to the Flesch Reading Ease (F = 3.4932, p < .05), Gunning's Fog Index (F = 4.7315, p < .05), and Flesch-Kincaid Grade Level formula (F = 5.7098, p < .05). However, no statistical significance emerged for ab- stracts (Flesch Reading Ease, F = .0539, p > .05; Gunning's Fog Index, F =.1772, p > .05; and Hesch-Kincaid, F = .1962, p > .05). The t-test indicates that regardless of the readability measure there is a statis- tically significant difference among the sample of five-paragraph texts for the three groups depicted in table 3. Al- though the texts of articles reflect a scholarly level of readability, there are significant differences. Because rejected papers are the most readable using the three measures, the first hypothesis is not supported. In the case of the abstracts, the ANOVA, as already discussed, did not Published, Accepted, and Rejected Papers 521 disclose statistically significant differ- ences for any readability measure. The t-test, as well, did not identify significant differences. Therefore, the second hy- pothesis is not supported. Abstracts for rejected, accepted, and published papers are all difficult to read. Tables 1 and 2 support this finding. TOPICS MERITING INVESTIGATION The readability of texts and abstracts merits further examination. An interest- ing question is: Why were rejected papers the most readable? Presumably, the copyediting of accepted papers re- sults in a more readable published paper. However, further analysis of this question is needed. It is important to understand the readability of abstracts, as well as their content and form. 12 If one function of an abstract is to entice read- ership of a paper, the level of difficulty might be decreased. This study might be duplicated using submitted and published papers for more than a two-year span. Both hy- potheses might be tested using other journals, scholarly and perhaps popular as well. Instead of limiting data collec- tion and analysis to a quantifiable tech- nique, researchers might explore focus group interviews and other methods of qualitative data collection to obtain a complementary understanding of readability and subscriber preferences. CONCLUSION As journals strive to better address the interests and needs of their readership and to expand the number of readers and subscribers, readability becomes an important variable. As the reading level of the general public and perhaps some specialized publics declines, and as librarians and others become busier and read a smaller percentage of their pro- fessional literature, readability might be The readability of texts and abstracts merits further examination. An interesting question is: Why were rejected papers the most readable? linked with ''browse-ability" and, there- fore, scholarly journals should strive for an easier level of reading difficulty and changes in presentation format. With in- creased interest in electronic publishing, two important questions become: "What is the readability of electronic journals?" and "Is there a difference in readability between electronic and nonelectronic journals?" More papers and scholarly journals, regardless of the medium in which they appear, might aim for the "fairly difficult'' or "difficult'' as op- posed to the "very difficult" level on the Flesch Reading Ease (see note 7). 13 REFERENCES AND NOTES 1. C. R. Bybee, "Fitting Information Presentation Formats to Decision-Making," Com- munication Research 8 (July 1981): 343-70. 2. Edgar Dale and Jeanne Chall, "The Concept of Readability," Elementary English 26 (Jan. 1949): 23. 3. Chaffai Tekfi, "Readability Formulas: An Overview," Journal of Documentation 43 (Sept. 1987): 262. 4. Marie J. Abram, "Readability: Its Use in Adult Education," Lifelong Learning: The Adult Years 4 (Jan. 1981): 8. 5. Ibid. 6. The Flesch Reading Ease formula considers the average number of words per sentence and the number of syllables per 100 words. Gunning's Fog Index includes the average number of words per sentence and the number of words of three or more syllables. The Flesch-Kincaid Grade Level formula examines the average number of words per sentence and the average number of syllables per word. 522 College & Research Libraries 7. The scoring categories for the Flesch Reading Ease are as follows: Score Reading Difficulty Very easy Easy Grade Level 4th grade 5th grade 6th grade 7th-8th grade Some high school November 1993 90-100 80-90 70-80 60-70 50-60 30-50 0-30 Fairly easy Standard Fairly difficult Difficult Very difficult High school and college Minimum of college 8. The instructions for sampling paragraphs were as follows: a. Mark paragraphs in the article which cannot be measured due to special characters or whatever. b. Number the remaining paragraphs from one to n. c. Using n as the total number of paragraphs in a paper, turn to the Appendix and find that number. Then, select those five paragraphs for word processing. For papers having five or fewer paragraphs, take all the paragraphs. d. Assign each paper a unique ID number and indicate whether the paper has been accepted, published, or rejected. 9. For reliability purposes, readability scores were computed using RightWriter (Que Software, Carmel, Ind.), a software package similar to Grammatik, and compared to the scores generated using Grammatik. There were no differences in the scores. 10. Abram, "Readability," p.9. 11. Robert Calfee and Priscilla Drum, "Research on Teaching Reading." in Handbook of Research on Teaching, ed. Merlin C. Wittrock (New York: Macmillan, 1986), 804-49. 12. See Tibor Koltay, "The Structure of Medical Papers and Their Author-Abstracts," Health Information and Libraries 1 (1990): 55-60; Milica Milas-Bracovic and Jasenka Zajec, "Author Abstracts of Research Articles Published in Scholarly Journals in Croatia (Yugoslavia): An Evaluation," Libri 39 (Dec. 1989): 303-18; Timothy C. Craven, "Sen- tence Dependency Structures in Abstracts," Library & Information Science Research 10 (Oct./Dec. 1988): 401-10; Elizabeth Liddy, Susan Bonzi, and Jeffrey Katzer, "A Study of Discourse Anaphora in Scientific Journals," Journal of the American Society for Infor- mation Science 38 (July 1987): 255-61; and Elizabeth Liddy, "The Discourse-Level Structure of Empirical Abstracts: An Exploratory Study," Information Processing & Management 27(1991): 55-81. 13. It is interesting to note that this paper scored the following on the three readability measures: Flesch Reading Ease: Gunning's Fog Index: Flesch-Kincaid Grade Level: Five- Paragraph Text 24 19 16 Abstract 5 23 20 Published, Accepted, and Rejected Papers 523 APPENDIX SELECTION OF FIVE PARAGRAPHS FROM A PAPER No. of Paragraphs First Second Third Fourth Fifth 6 1 2 3 5 6 7 1 2 4 5 7 8 1 2 4 6 8 9 1 2 5 7 9 10 1 3 5 8 10 11 1 3 6 8 11 12 1 3 6 9 12 13 1 3 7 10 13 14 1 4 7 11 14 15 1 4 8 11 15 16 1 4 8 12 16 17 1 4 9 13 17 18 1 5 9 14 18 19 1 5 10 14 19 20 1 5 10 15 20 21 1 5 11 16 21 22 1 6 11 17 22 23 6 12 17 23 24 6 12 18 24 25 1 6 13 19 25 26 1 7 13 20 26 27 1 7 14 20 27 28 1 7 14 21 28 29 1 7 15 22 29 30 1 8 15 23 30 31 1 8 16 23 31 32 1 8 16 24 32 33 1 8 17 25 33 34 1 9 17 26 34 35 1 9 18 26 35 36 1 9 18 27 36 37 1 9 19 28 37 38 1 10 19 29 38 39 1 10 20 29 39 40 1 10 20 30 40 41 1 10 21 31 41 42 1 11 21 32 42 43 1 11 22 32 43 44 1 11 22 33 44 45 1 11 23 34 45 46 1 12 23 35 46 47 1 12 24 35 47 48 1 12 24 36 48 49 1 12 25 37 49 50 1 13 25 38 50 51 1 13 26 38 51 52 1 13 26 39 52 53 1 13 27 40 53 (continued) 524 College & Research Libraries November 1993 APPENDIX (continued) No. of Paragraphs Flrst Second Third Fourth Flfth 54 1 14 27 41 54 55 1 14 28 41 55 56 1 14 28 42 56 57 1 14 29 43 57 58 1 15 29 44 58 59 1 15 30 44 59 60 1 15 30 45 60 61 1 15 31 46 61 62 1 16 31 47 62 63 1 16 32 47 63 64 1 16 32 48 64 65 1 16 33 49 65 66 1 17 33 50 66 67 1 17 34 50 67 68 1 17 34 51 68 69 1 17 35 52 69 70 1 18 . 35 53 70 71 1 18 36 53 71 72 1 18 36 54 72 73 1 18 37 55 73 74 1 19 37 56 74 75 1 19 38 56 75 76 1 19 38 57 76 77 1 19 39 58 77 78 1 20 39 59 78 79 1 20 40 59 79 80 1 20 40 60 80 81 1 20 41 61 81 82 1 21 41 62 82 83 1 21 42 62 83 84 1 21 42 63 84 85 1 21 43 64 85 86 1 22 43 65 86 87 1 22 44 65 87 88 1 22 44 66 88 89 1 22 45 67 89 90 1 23 45 68 90 91 1 23 46 68 91 92 1 23 46 69 92 93 1 23 47 70 93 94 1 24 47 71 94 95 1 24 48 71 95 96 1 24 48 72 96 97 1 24 49 73 97 98 1 25 49 74 98 99 1 25 50 74 99 100 1 25 50 75 100 101 1 25 51 76 101 102 1 26 51 77 102 103 1 26 52 77 103 104 1 26 52 78 104 105 1 26 53 79 105 106 1 27 53 80 106 107 1 27 54 80 107 Published, Accepted, and Rejected Papers 525 No. of Paragraphs First Second Third Fourth Fifth 108 1 27 54 81 108 109 1 27 55 82 109 110 1 28 55 83 110 111 1 28 56 83 111 112 1 28 56 84 112 113 1 28 57 85 113 114 1 29 57 86 114 115 1 29 58 86 115 116 1 29 58 87 116 117 1 29 59 88 117 118 1 30 59 89 118 119 1 30 60 89 119 120 1 30 60 90 120 121 1 30 61 91 121 122 1 31 61 92 122 123 1 31 62 92 123 124 1 31 62 93 124 125 1 31 63 94 125 126 1 32 63 95 126 127 1 32 64 95 127 128 1 32 64 96 128 129 1 32 65 97 129 130 1 33 65 98 130 131 1 33 66 98 131 132 1 33 66 99 132 133 1 33 67 100 133 134 1 34 67 101 134 135 1 34 68 101 135 136 1 34 68 102 136 137 1 34 69 103 137 138 1 35 69 104 138 139 1 35 70 104 139 140 1 . 35 70 105 140 141 1 35 71 106 141 142 1 36 71 107 142 143 1 36 72 107 143 144 1 36 72 108 144 145 1 36 73 109 145 146 1 37 73 110 146 147 1 37 74 110 147 148 1 37 74 111 148 149 1 37 75 112 149 150 1 38 75 113 150 151 1 38 76 113 151 152 1 38 76 114 152 153 1 38 77 115 153 154 1 39 77 116 154 155 1 39 78 116 155 156 1 39 78 117 156 157 1 39 79 118 157 158 1 40 79 119 158 159 1 40 80 119 159 160 1 40 80 120 160 161 1 40 81 121 161 (continued) 526 College &: Research Libraries November 1993 APPENDIX (continued) No. of Paragraphs First Second Third Fourth Fifth 162 1 41 81 122 162 163 1 41 82 122 163 164 1 41 82 123 164 165 1 41 83 124 165 166 1 42 83 125 166 167 1 42 84 125 167 168 1 42 84 126 168 169 1 42 85 127 169 170 1 43 85 128 170 171 1 43 86 128 171 172 1 43 86 129 172 173 1 43 87 130 173 174 1 44 87 131 174 175 1 44 88 131 175 176 1 44 88 132 176 177 1 44 89 133 177 178 1 45 89 134 178 179 1 45 90 134 179 180 1 45 90 135 180 181 1 45 91 136 181 182 1 46 91 137 182 183 1 46 92 137 183 184 1 46 92 138 184 185 1 46 93 139 185 186 1 47 93 140 186 187 1 47 94 140 187 188 1 47 94 141 188 189 1 47 95 142 189 190 1 48 95 143 190 191 1 48 96 143 191 192 1 48 96 144 192 193 1 48 97 145 193 194 1 49 97 146 194 195 1 49 98 146 195 196 1 49 98 147 196 197 1 49 99 148 197 198 1 50 99 149 198 199 1 50 100 149 199 200 1 50 100 150 200 Pick a system. Any system. At EBSCO we think one of the most important decisions you'll make for your library is that of choosing quality subscription service. Equally important is choosing the automated system that best fits your library's needs. When you choose EBSCO for subscription ser~ vice, we respect your library automation decision and provide an array of electronic services that complements your system's capabili~ ties. Our services are compatible with most major systems - so you can have superior subscription service and your system of choice, too. Call us today to learn more about our independence~oriented library automation services. ii~J.t•J SUBSCRIPTION SERVICES P.O. Box 1943 • Birmingham, AL 35201 (205) 991~6600 • Fax: (205) 995~1636 Where library automation is a liberating experience. TECHNOLOGY FOR THE 90's AND FOR THE LIBRARIES OF TOMORROW BLACKWELL'S PROUDLY INTRODUCES ... FOR COLLECTION DEVELOPMENT Blackwell's New Tides Online database now includes Tables of Contents and Descriptive Stnnmaries for new and forth-co~g scholarly monographs. • Browse the contents pages before placing the order • View the publisher's tide description • Order the tide electronically • Insure a tide is included on approval • Access through the INTERNET Blackwell's MARC With Books® service now offers LC- MARC records enriched with Tables ofContents. • Enhance subject access • Improved chapter-level author access • View contents & stnnmaries in your PAC before searching the shelves • Increase interlibrary loan efficiency FIND OUT HOW EASY AND COST EFFECTIVE IT CAN BE TO GET CONTENTS INFORMATION TO BOTH YOUR STAFF AND PATRONS TODAY! BLACKWELL. NORTH AMERICA, INC. TECHNICAL SERVICES DIVISION 6024 SW Jean Road, Building G, Lake Oswego, Oregon 97035. Telephone: 503-684-1140 Fax: 503-639-2481