Guest Editorial Peer Review and Replication Data: Best Practice from Journal of Peace Research Nils Petter Gleditsch, Ragnhild Nordås, & Henrik Urdal Peace Research Institute Oslo (PRIO) Journal of Peace Research is an independent, interdisciplinary, and interna- tional journal devoted to the study of war and peace. It is owned by the Peace Research Institute Oslo (PRIO) and published on contract with Sage. Its articles range across all the social sciences, although a large majority of its authors now have their main training in political science. The international character of the journal is visible in the composition of the editorial committee and the authorship. The journal has long been a leader among the journals in political science and international relations in making research data publicly available, and is a pioneer in publishing dataset in the form of “special data features.” Introducing Peer Review When the journal was founded in 1964, editorial decisions were made by the editor in consultation with close colleagues, as was common in the social sciences at the time. A large fraction of the articles were also invited or written by the staff and associates of PRIO. Eventually, the editorial committee came to play a more active role in reviewing articles. Outside peer review was introduced in 1983. This may seem late, but in fact most social science journals at the time had either weakly developed peer review or none at all, or they had just introduced it. When a proposal to introduce peer review was first suggested to the editorial committee, it met with great skepticism. This was in part because the committee wanted to maintain control of the process, and in part because it was believed—erroneously as we now know—that outside peer review would require the prior recruitment of a large number of colleagues who were willing and able to undertake the reviewing. In fact, referees are never requested in advance, and the rate of rejection and non-response is tolerable. Despite increasing laments about referee fatigue, the overall rate for non-responses and declined requests for JPR has been stable at about 35% since 2010. All reviewing was initially single-blind—i.e. the name of the reviewer is secret to the author,1 but not vice versa—based on the principle that it was better for the editor to know that the reviewer knew the identity of the author, rather than suspecting that the s/he had guessed. Given the relatively small-sized community of peace and conflict scholars at the time, this was not an entirely unreasonable assumption. Furthermore, the editing needed to make the articles anonymous was not cost-free in the paper-based era. The increasing pool of potential reviewers and the advent of electronic word pro- cessing eliminated two of the major objections, and from 2002 double-blind has been the norm at JPR. Thus, possible biases based on gender, seniority, and nationality are reduced, although the most compelling argument for the change was probably that double-blind rapidly became the norm in most social science journals. Ironically, the 267doi:10.5860/crl.78.3.267 http://10.5860/crl 268 College & Research Libraries March 2017 increasing availability of earlier versions of articles on conference sites and personal websites, has facilitated the identification of authors for referees who are sufficiently curious. Data The foundation of peace research in the Nordic countries in the early 1960s was closely related to the behavioral revolution in the social sciences, and an empirical orienta- tion was central to the journal from its first volume in 1964. Most of the articles were empirical and many of them used original, quantitative data. There is probably a link here to the international orientation of the journal. While journals catering mainly to a regional or national audience are likely to be influenced by local cultural norms, topics, and terminology, the principles of systematic data collection are universal. Analyses published in the mid-1960s, before the use of computers became common in the social sciences, were largely limited to bivariate relationships, with occasional controls of a few other variables. They may seem simple, even simplistic by today’s standards. But for the time, they were state-of-the art, and represented a quantum leap from what one of the pioneers of peace research, J. David Singer, was fond to call “arm-chair specu- lation.” The introduction of computers in the social sciences, along with advances in mathematical modeling and multivariate statistics, has made the journal’s data-based articles more sophisticated and thus increasingly complex. Technical progress has also enabled new forms of data collection, such as data based on satellite imagery. And a stronger international community—notably the efforts of the UN specialized agencies and regional organizations such as the OECD—have facilitated international data collection over a range of topics. The ambitious goals set at the international level for improvements in health, environment, development, and peace through the Sustainable Development Goals formulated by the UN in 2015,2 require additional data collection in order to assess how the goals are fulfilled. Making Datasets Available Most empirical articles in JPR focus on the testing of theoretical frameworks, but the journal also has a long tradition of making new datasets available to the research com- munity without extensive analysis. The pioneering case was the publication in 1966 of the first version of the dataset on formal alliances generated by the Correlates of War Project and followed in 1969 by an update. Like many of the datasets generated by the Correlates of War Project, this has become one of the standard tools of empirical research on war and peace. The dataset has been regularly updated, most recently in 2013, and is available for download from the project’s website.3 Another early case was a global dataset on local wars for the period after World War II, developed by Istvan Kende, first published in JPR in 1971 and later updated for another ten-year period. These early datasets were of limited size and could be published in their entirety in the print journal. Later editions were much more extensive and were published in book form.4 From the early 1990s most articles presenting new data were presented under the general heading of Special Data Feature,5 and since then the journal has published dozens of such articles. They have played a crucial role in maintaining the prominence of the journal. The two most-cited articles in JPR ever are data articles, and there are four of them among the top ten.6 Indeed, for each of the ten years 2006–15, a data article is always found and among the top five-cited articles, usually at the top of the list. A special place of prominence is occupied by the annual article on global armed conflict from the Uppsala Conflict Data Program (UCDP). The joint UCDP/PRIO article in 2002 extending the time series back to 1946, is the article of record for a dataset that has become a standard tool in empirical conflict research, and the annual updates are also studied by many. “Good Guest Editorial 269 data beats better methods every time,”7 so it is not surprising that a lot of innovative work is invested in producing new data, and that such articles are frequently cited. Since the main interest in publishing such articles is to present new data, JPR does not “require the same level of theoretical sophistication and detailed empirical investigation as for a regular research article.” Authors are nevertheless asked to show “how the new data can make a genuine contribution to the study of conflict and peace, for instance by pointing to results that are significantly different from previously published work.”8 Peer Review of Data Given the prominence of data presentations and empirical analyses of data, the ques- tion of assessing their reliability and usefulness becomes crucial to the peer-review procedures of the journal. The referees have to be competent in empirical methods and have an understanding of what new data can contribute to the field. Since the journal’s empirical articles, including data presentations, cover a wide range of topics, a reviewer is not necessarily fully competent to judge the substance as well as the methods and data. Thus it may become necessary to select different referees for these tasks. In 1998, JPR introduced a replication requirement. Authors of all quantitative articles were required to post a copy of the data on a suitable website “or in some equivalent manner.”9 The idea was to promote greater transparency by permitting other scholars to rerun the analyses and see whether there were errors or peculiar coding decisions that had led to non-robust or even false results. This decision was inspired by a report from an economist colleague on some early initiatives in economics journals.10 Since then, a replication requirement has become increasingly common in journals in international relations and political science, and economics is hardly ahead of the curve anymore.11 Initially, authors would usually post the data on their own homepage or that of their institution. However, such websites often provide an unstable environment, and some of the early datasets can no longer be located. In some cases, authors have also posted new versions of their data. While the new data may well be an improvement over the old, it effectively prevents replication of the original results. Since 2002, JPR has required that the data be posted on the journal’s own website,12 although authors are welcome to post them on their own site or in other data repositories as well. Many authors now post their data on the Dataverse Project repository at Harvard University, which currently holds more than 61,000 datasets with a combined number of downloads extending two million times.13 This site also provides more stability. While International Studies Quarterly, the flagship journal of ISA, maintains its own site for replication data, other ISA journals such as International Studies Perspectives encourage their authors to post their data on the Dataverse. Also from 2002, authors have been asked to provide a codebook to the data, a com- mand file for the analysis, and other information relevant to the dataset. There are no space limitations on the replication material. The JPR website currently hosts 630 replication files.14 In addition to enhancing the transparency of empirical research, this website also serves as a resource for student projects. An additional advantage for the journal itself is that the availability of replication data increases citations to the articles and thus also measures of the prestige of the journal, such as the impact factor.15 Advance Submission of Data? In a symposium on replication in international relations in 2002, Bruce Bueno de Mesquita, then President of the International Studies Association (ISA), proposed that replication data be made available at the time of submission, which would enable a referee to check the analysis.16 The referee would not be obligated to make this addi- tional effort, but should be able to do so. Even the knowledge that a referee might go 270 College & Research Libraries March 2017 the extra step, would probably keep the authors on the alert to avoid sloppy coding and careless analysis. Many replication files look plausible at first glance, but when they are examined more closely then turn out to be incomplete or hard to unravel. The research community would not have to wait for errors or questionable interpretations to be discovered years later. Instead, errors would be weeded out prior to publication (with no embarrassment to the author) and non-recoverable articles rejected. While attractive, implementing such a policy cannot be expected to uncover all errors, and it also has potential disadvantages. A cynical referee might try to take the data and run with them, or the author might suspect that this would happen and not submit to a journal with such a policy. Referees might feel that an extra burden was added to their shoulders, adding to the problem of reviewer fatigue. The burden increases with increasing complexity of the analysis. Editors might feel that they were getting perilously close to crossing the boundary to becoming co-authors of the article. Also, if the referees ask for additional analysis, a new set of replication data would have to be submitted with the revised version. After extensive consideration, JPR decided not to go down this route. However, its “Notes for authors” specify that authors are welcome to submit the data with the article “and may find that the reviewers are able to provide better feedback if given access to the data on a privileged basis.” To our knowledge, the only journal in our field that has introduced a requirement to submit the data with the initial version of the article is International Interactions.17 Their “preplication” is aimed at making sure that the reported results can be duplicated with the data provided and to correct outright errors. Graduate student assistants are used to test the data. The editor reports that a significant fraction of the submitted articles do have some initial problems with the replication data. The process seems to have worked quite smoothly. As an alternative or supplement to the “preplication” of each article, Allan Dafoe has suggested that journals should set up an audit panel, which from time to time would test a set of randomly chosen articles for their replicability. In addition, journals should be willing to retract articles where later replication efforts demonstrate that they are not up to professional standards. Such measures are not without some costs either, and to our knowledge no journal has so far put them into practice. The Future of Data Review Surveys of the policies and practices of journals in international relations and political science indicate that many journals still lack a replication policy and that even when they do they fail to follow it in practice.18 Preplication, as practiced by International Interactions, draws attention to the replication requirement at the start of the review process. But JPR’s strict requirement that the data be supplied to the journal prior to the final acceptance of the article also prevents the commitment to replication from becoming lip service only. However, preplication, audit panels, and the withdrawal of articles may well become standard features of journal policy in the future. The integrity of science is threatened in many ways, by outright censorship, by secrecy (military, political, or commercial), by the reluctance of many journals to pub- lish non-surprising findings, by the excessive reliance of many scholars on statistical significance at the expense of substantive importance or predictive ability, as well as the cartel-like behavior of predatory publishers that drive up the cost of journals and thus limit the dissemination of academic work. At the moment, political attention seems to be focused mainly on the latter problem, with open access featured as the solution. While open access can no doubt play an important role in the wider dissemination of scholarly work, there is still considerable uncertainty about how peer review and other elements of quality control are to be funded. The role of academic journals is Guest Editorial 271 crucial, and the enthusiasm for open access should not be allowed to undermine the position of established, well-edited, and responsible journals. For JPR, peer review and openness about the data and the way they have been analyzed, remain key issues, and hopefully funding will be available for continuing to pursue these goals in the future. The authors Nils Petter Gleditsch is research professor at PRIO, professor emeritus of political science at the Norwegian University of Science and Technology, and former editor of Journal of Peace Research; Henrik Urdal is research professor at PRIO and editor of Journal of Peace Research; Ragnhild Nordås is assistant profes- sor of political science at University of Michigan, senior research associate at PRIO, and deputy editor of Journal of Peace Research. Notes 1. Although there has always been possible for a JPR reviewer to sign with his or her name, that option is almost never chosen. 2. www.un.org/sustainabledevelopment/. 3. http://www.correlatesofwar.org/data-sets. 4. Douglas M Gibler M (2009) International Military Alliances, 1648–2008. Washington, DC: CQ Press. Klaus Jürgen Gantzel (1995) Die Kriege nach dem Zweiten Weltkrieg, 1945 bis 1992. Daten und Tendenzen [The Wars after World War II, 1945–1992. Data and Trends]. Münster: Lit. 5. Originally just “Data Feature.” 6. Search on Web of Science, 13 December 2016. 7. Gary King (2001) Proper nouns and methodological propriety: Pooling dyads in interna- tional relations data. International Organization 55(2): 497–507, p. 505. 8. Notes for authors (p. 2), downloaded from http://file.prio.no/journals/JPR/JPR-Notes-for- Authors-140909.pdf, 13 December 2016. 9. Nils Petter Gleditsch (1998) Preface. JPR 35(1): 5–6. 10. Personal communication, Ron P. Smith, 3 January 2017. 11. Garret S Christensen & Edward Miguel (2016) Transparency, reproducibility, and the cred- ibility of economics research. NBER Working Paper (22989), www.nber.org/papers/w22989. 12. Nils Petter Gleditsch (2002) Double-blind but more transparent. JPR 39(3): 259–262. 13. http://dataverse.org/, statistics downloaded 13 December 2016. 14. www.prio.no/jpr/datasets, statistics as of 1 February 2017. 15. Nils Petter Gleditsch, Claire Metelits & Håvard Strand (2003) Posting your data: Will you be scooped or will you be famous? International Studies Perspectives 4(1): 89–97. 16. Bruce Bueno de Mesquita (2003) Getting firm on replication. International Studies Perspec- tives 4(1): 98–100. 17. Michael Colaresi (2016) Preplication, replication: A proposal to efficiently upgrade journal replication standards, International Studies Perspectives 17(4): 367–378. 18. Nils Petter Gleditsch & Claire Metelits (2003) The replication debate. International Studies Perspectives 4(1): 72–79; Nils Petter Gleditsch & Nicole Janz (2016) Replication in international rela- tions. International Studies Perspectives 17(4): 361–366; and Sergiu Gherghina & Alexia Katsanidou (2013) Data availability in political science journals. European Political Science 12(3): 333–349. http://www.un.org/sustainabledevelopment/ http://www.correlatesofwar.org/data-sets http://file.prio.no/journals/JPR/JPR-Notes-for-Authors-140909.pdf http://file.prio.no/journals/JPR/JPR-Notes-for-Authors-140909.pdf http://www.nber.org/papers/w22989 http://dataverse.org/ http://www.prio.no/jpr/datasets