Book Reviews

Editorial Peer Review: Its Strength and Weaknesses

Weller, Ann C. Editorial peer review: its strength and weaknesses. (ASIST monograph series) Medford NJ: American Society for Information Science and Technology, 2001. ISBN 1-57387-100-1. $44.50.

Editorial Peer Review: Its Strength and Weaknesses is an exhaustive analysis of published studies on peer review and its role in the scholarly publication process. This book might be called a meta-study, because the author does not offer any new research on peer review, but rather collates and analyzes what past studies have found. And there is plenty of raw material to work with: the author identified over 1,500 separate citations from literature searches.

Peer review can be defined simply as the process by which journal editors solicit evaluations of submitted articles from outside experts who remain anonymous to the authors. The role of journals as gatekeepers to the scientific record dates to the 17th century, when the Royal Society's council was instructed to review submissions to its Philosophical Transactions. But the modern process of "blind" peer review is much more recent. Until the mid-20th century many papers were approved solely by a journal's editors rather than independent reviewers, and for some major journals this is still the case. The explosion of scientific output after World War II strained the review process, significantly extending the lag times between submission and publication.

Peer review has always had to strike a balance between speed and quality. Each end of this continuum has its champions. How the question plays out depends largely on the "culture" of a particular scientific discipline. In fast-moving fields like physics, today's breakthrough is tomorrow's old news, and speed is preferred over editorial thoroughness. This led physicists to create the preprint system, whereby pre-publication copies of papers under review were traded among scientists, essentially relegating their journals to archival repositories rather than front-line communication mechanisms. This preprint culture was quickly and enthusiastically converted to an "e-print" system pioneered by Paul Ginsparg at Los Alamos in the early 90s. Yet in other fields, such as chemistry, preprints did not catch on and journals remain supreme, although rapid-publication "letters" journals stepped in to speed up the process.

In the medical sciences there is obviously a need for great care in publishing accurate research results, particularly from clinical trials. Yet speed is also essential when lives and health are at stake. The balance is delicate, and the general public -- thanks to an attentive media -- is much more attuned to the results than they are in the physical sciences. Major medical journals have become the harshest critics of any form of open prepublication or preprints, though their motives may have more to do with protecting their turf and prestige than with research integrity.

The main questions raised by these issues are straightforward. Is peer review worth the enormous amount of time, energy, and money it costs? Do the "best" journals really publish the "best" science? Unfortunately, the book never seems to get around to answering them. The author takes a limited approach to the topic, never wavering from her focus on "studies on peer review" rather than peer review itself. Though the book's thoroughness as a literature review can't be faulted, I found myself wishing for something more. The review itself is handicapped by the sheer number of studies cited, which employ different methodologies and cover different fields at different times. This makes it difficult to compare studies or draw generalizations. The author tends to end chapters by stating that conclusions are difficult and that more investigation is needed.

The book would have benefited from a more general discussion of the issues. Instead, it meticulously catalogs prior research and organizes it according to the many facets of the peer review process: rejection rates, the roles of editorial boards and reviewers, authorship trends, agreement and bias among reviewers, and the changes that electronic publication has brought to the process. It lists and summarizes reams of studies, editorials, and policies, arranging them in tables and applying statistical analysis where possible. Although not specific to one discipline, it concentrates on the sciences, particularly medicine. (The author is a medical librarian at the University of Illinois at Chicago.)

Peer review is an important topic that needs careful, objective analysis. Opinions can differ wildly between eager reformers and die-hard defenders of the status quo, and there is much at stake. The author's final conclusion is that peer review, while far from perfect, remains essential to the integrity of science -- a safe statement that few scientists would disagree with. But readers may be disappointed by the lack of connection with larger questions: publish-or-perish requirements, journal economics and proliferation, publisher attitudes, and the growing tensions between traditionalists and innovators in the emerging world of electronic scholarship. Nevertheless, it will be of some interest to journal editors and information scientists. Those doing research on the process of scholarly communication will find its comprehensive literature review an excellent starting point. But those expecting an engaging introduction to the topic will probably find Editorial Peer Review rather heavy going.