key: cord-0776592-yp9pb6tf
authors: Garritty, Chantelle; Gartlehner, Gerald; Nussbaumer-Streit, Barbara; King, Valerie J.; Hamel, Candyce; Kamel, Chris; L, Affengruber; Stevens, Adrienne
title: Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews
date: 2020-10-15
journal: J Clin Epidemiol
DOI: 10.1016/j.jclinepi.2020.10.007
sha: 5192d888af71fb6453f8709317322e610d2a1593
doc_id: 776592
cord_uid: yp9pb6tf

Objective To develop methods guidance to support the conduct of rapid reviews (RRs) produced within Cochrane and beyond, in response to requests for timely evidence syntheses for decision-making purposes including urgent health issues of high priority. Study Design Interim recommendations were informed by a scoping review of the underlying evidence, primary methods studies conducted, and a survey sent to 119 representatives from 20 Cochrane entities, who were asked to rate and rank RR methods across stages of review conduct. Discussions among those with expertise in RR methods further informed the list of recommendations with accompanying rationales provided. Results Based on survey results from 63 respondents (53% response rate), 26 RR methods recommendations are presented for which there was a high or moderate level of agreement or scored highest in the absence of such agreement. Where possible, how recommendations align with Cochrane methods guidance for SRs is highlighted. Conclusion The Cochrane Rapid Reviews Methods Group offers new, interim guidance to support the conduct of RRs. Because best practice is limited by the lack of currently available evidence for some RR methods shortcuts taken, this guidance will need to be updated as additional abbreviated methods are evaluated.

In healthcare, systematic reviews (SRs) are highly valued evidence syntheses that inform decisions. However, the methodological rigor and process that makes SR evidence trustworthy often take one to two years to complete [1] [2] [3] [4] , and limit their utility to meet time-sensitive needs of stakeholders. For example, given the onset of coronavirus, decision-makers have an urgent need for evidence that cannot be met using traditional SR methods. Rapid reviews (RRs) have emerged as an efficient tool to get evidence to decision-makers more quickly and are now considered part of the knowledge synthesis family [5] . Although published descriptions of RRs date back nearly a decade [6, 7] , a standard or consensus definition does not exist. However, RRs have been described as a type of knowledge synthesis in which SR methods are streamlined and processes are accelerated to complete the review more quickly [1, 2, 7, 8] . Evidence suggests that policymakers are increasingly using RRs in their daily decision-making [9] [10] [11] [12] . Respected national and international health agencies [13] , 14] are also using RRs, including to inform guideline recommendations in urgent and emergent public health settings [15, 16] .

Cochrane, a global leader in the production of high-quality SRs and methodological guidance, has taken important steps to foster engagement in RRs. In 2015, the Cochrane Rapid Reviews Methods Group (RRMG) was established [17] and has been involved in related methods research including developing standards for the reporting of RRs [18] . In 2018, Cochrane's Content Strategy identified the need to explore and, if appropriate, produce RRs [19] . To inform this work, the RRMG conducted several research activities that culminated in the development of J o u r n a l P r e -p r o o f 6 interim RR methods recommendations. Although a formal implementation strategy for Cochrane RRs had been planned, due to the onset of the COVID-19 pandemic, Cochrane supported the early release of this guidance, so it was available to Cochrane's network of reviewers and others in the research community conducting RRs for urgent, highest priority topics.

The objectives of this manuscript are to present features that define a Cochrane RR and to provide interim methodological recommendations for the development of RRs. While there are examples of RR guidance [16, 20, 21] , to our knowledge, this is the first that provides clear, actionable recommendations and minimum standards based on up to date empirical evidence evaluating RR methods.

Recommendations were informed by a suite of related methodological work briefly described below, that included two scoping reviews, two primary methods studies, and a survey of Cochrane stakeholders.

We conducted a systematic scoping review and thematic analysis of definitions and defining characteristics of RRs, and proposed a definition of RR that covers the most common themes that were identified in approximately 50% or more of the 216 RRs and 90 methods papers included [22] . We also undertook a second scoping review that identified 14 empirical studies that evaluated RR methods, which we mapped to stages of review conduct [23] . The RRMG also led J o u r n a l P r e -p r o o f two methodological studies: one that assessed the impact of limiting inclusion criteria solely to English language publications [24] ; and one that assessed the accuracy of single-reviewer screening versus dual-reviewer screening as part of an online parallel-group randomized controlled trial (RCT) using the Cochrane Crowd platform [25] . Collectively, this work formed the evidentiary base for the subsequent RR methods options survey. Details of these publications are reported elsewhere [23] [24] [25] . 

The survey was intended to elicit input on the major identified streamlined RR methods across the stages of performing a RR and was developed following recommended survey methods [26] .

Where possible, for certain questions, we highlighted identified research supporting or cautioning the use of a particular shortcut or approach (Appendix A & Web Appendix 2).

Surveys were sent to 119 individuals from 20 specific Cochrane entities in September 2019.

Results were analyzed using descriptive statistics with details reported in the text and summarized in tabular form. Given this was an internal organizational survey, formal ethics approval was not sought but informed consent was required to participate. See Figure 1 for further survey methods details.

As a preliminary approach, the RRMG elected to adopt recommendations based on survey items for which there was a high or moderate level of agreement using the following as a guide: highlevel (endorsed by ≥70% of respondents); moderate-level (endorsed by ≥50-69% of respondents); low-level (endorsed by <50% of respondents). If, for a particular question, a J o u r n a l P r e -p r o o f response set only reached 'low-level agreement,' we still recommended the most endorsed response so that each stage of conduct had an accompanying recommendation.

The guidance also incorporated information as to how endorsed items mapped to Cochrane MECIR guidance for SRs. Further discussion among RRMG convenors informed the agreed list of recommendations presented along with the accompanying rationales provided to support and provide further explanation and context for each recommendation with supporting evidence cited if available. Although we did not follow a formal recommendation process, we used a deliberate and systematic approach to develop this interim guidance as per Web Appendix 3.

The overall survey response rate was 53% (n=63) with a completed response rate of 46% (n=53).

Respondents were from 19 of 20 entities approached. Of those that responded, a large proportion were either extremely/very (38%; n=24) or somewhat (38%; n=24) familiar with RRs. Further, 62% (n=39) had previously participated in a RR in various capacities (See Appendix B.

Participant Characteristics). Overall, there was general approval for Cochrane implementing RRs as a product (only one of 59 respondents indicated Cochrane should not undertake RRs).

The label Cochrane RR was strongly endorsed as a sufficient term (43/57; 75%) for use within Cochrane. Overall, we included twenty-six items that attained a moderate to a high level of agreement or that ranked first as an approach across the stages of conduct (See Appendix C.

Recommendations, Level of Agreement and Relation to MECIR Guidance). See Web Appendix 1 (https://osf.io/zeq9p/) for main survey results.

We recommend that Cochrane adopts the following definition of a RR: "A rapid review is a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting various methods to produce evidence for stakeholders in a resource-efficient manner" [22] . Further, RRs should be driven by the need for timely evidence for decision-making purposes including to address urgent and emergent health issues and questions deemed to be of high priority.

The Cochrane RRMG has put forth a list of twenty-six recommendations for Cochrane RRs as outlined in Table 1 , with rationales for the recommendations provided below.

Evidence from one study suggests that knowledge brokering of proposals improved the perceived clarity of information provided to policymakers; reasons for commissioning the RR;

and context for the questions, scope, methods, and report conclusions. Further, it improved the confidence of reviewers to meet the policymakers' needs [27] . It is also important to develop a protocol that includes the review questions using a question framework (e.g., PICOs) and that details the eligibility criteria.

The various limits to eligibility criteria are important to ensure RRs are manageable and timely.

Such restrictions need to be considered together with stakeholders. Limits include clearly defining the population, intervention, and comparator. If the findings of a RR are to influence practice and policy, then outcome selection needs to be relevant to clinicians, policymakers and J o u r n a l P r e -p r o o f patients. Although exploring the full range of outcomes at the outset of the project is good practice, the list of outcomes needs to be carefully condensed given time and resource limitations of RRs. Author teams, therefore, must judiciously select the outcomes to consider. Prioritizing outcomes will depend on the needs of stakeholders, who should be involved in the selection process.

If limiting the inclusion criteria by study date of publication, evidence suggests that limiting the search to a set number of years may lead to a loss of studies and a change in the results of metaanalyses [28] . In this study, most effect size changes were small but moderate, and significant changes were relatively common. Therefore, although setting a date restriction is a pragmatic shortcut, this needs to be carefully considered for each topic. Language restriction (i.e., English only studies) is also practical when conducting RRs for conventional interventions. A recent study assessed whether limiting inclusion criteria solely to English language publications affected the reviews' overall conclusions. Findings suggest that exclusion of non-English publications from SRs on clinical interventions had a minimal effect on overall conclusions, and can be a viable methodological shortcut for RRs [24] . However, we do not recommend restricting to English-only publications if the expectation is that relevant studies may be published in languages other than English. For example, if RRs are related to COVID-19, or involve complementary and alternative medicine (CAM) therapies, it is expected that studies would emerge in languages other than English.

Consider a stepwise approach for the inclusion of evidence, emphasizing synthesized research (e.g., SRs) first, where available, then on higher-quality designs for primary studies.

Emphasizing locating and summarizing evidence first from relevant, higher-quality study designs will assist in streamlining available evidence.

Searching for RRs needs to involve an experienced Information Specialist. Further, the selection of databases to search will depend on the topic under review and access to them. Two key studies helped inform the recommendation for searching major databases [28, 29] . In making our recommendation, we have adopted a conventional approach involving CENTRAL, MEDLINE and Embase (if available), which also conforms to Cochrane MECIR guidance.

Although Peer Review of Electronic Search Strategies (PRESS)] [30] is not part of current MECIR guidance, we recommend its use for Cochrane RRs. Evidence suggests that the absence of search strategy peer review often results in many missed studies not retrieved. Unless captured in an accompanying grey literature search, these records would not have appeared in the published RRs, thus reducing the integrity of reports [31] .

We also recommend limiting grey literature and supplemental searching. If it is warranted, consider limiting searches to clinical trial registries, the scanning of SR bibliographies and reference lists of included studies to identify potentially relevant studies. In most cases, grey literature searching should be done after the abstract and full-text screening is completed.

Screening reference lists can detect missed studies when searching electronic databases or eligible studies excluded in error during screening.

J o u r n a l P r e -p r o o f One reviewer to include and two to exclude title and abstract screening for Cochrane RRs was the highest-ranked among survey respondents. By comparison, currently, Cochrane allows for both dual and single title and abstract screening for Cochrane Reviews, although MECIR states that it is desirable to use two screeners working independently. For RRs, it is important that a standardized title and abstract form is used, and that before the start of screening, a pilot exercise is conducted using a minimum of the same 30-50 abstracts screened by the entire team to calibrate and test the review form. At the outset, two reviewers should be used to dual screen at least 20% of abstracts with conflicts resolved. This should be followed by one reviewer screening the remaining abstracts, and a second reviewer screening all excluded abstracts.

Although single reviewer screening may be a practical solution for certain RRs, we do not recommend this for RRs at this time. Findings from two recent studies indicate that single screening of the titles and abstracts is not equivalent to dual screening, as more studies are missed [25, 32] . Further, this approach ranked as least acceptable among Cochrane survey respondents. Nonetheless, forthcoming advances in automation and crowdsourcing have the potential to reduce the time spent screening when conducting RRs.

The recommendation (i.e., one reviewer to include, two reviewers to exclude) represents a methodological shortcut for RRs, when compared to full-text screening for Cochrane Reviews (i.e., two independent reviewers using full text to determine if the study meets eligibility criteria). As with title and abstract screening, a standardized full-text form should be used. A pilot exercise is recommended using the same 5-10 full-text articles for the entire screening team to calibrate and test the review form. Dual, independent screening of full-text articles ranked second as most acceptable, fully adheres to Cochrane guidance and may be used if time and J o u r n a l P r e -p r o o f resources are available. Review teams should also use SR software to make the screening, tracking, and documentation more efficient whenever possible.

The recommended approach for data extraction for RRs was the highest-ranked among survey respondents but deviates from Cochrane Reviews whereby data extraction may be separated into two parts: i) extracting study characteristics for which Cochrane allows for both dual and single extraction although in duplicate is highly desired; ii) extracting outcomes data, which Cochrane makes mandatory to extract in duplicate.

Data for RRs should be extracted using a pilot-tested form. It will be important that data is efficiently abstracted using concise descriptions of the participant, intervention and comparator characteristics and outcomes assessed. Although a recently published study that found the experience of data abstractors may matter less than initially thought; and that adjudication is what leads to reduced errors [33] , skilled extractors will be key to minimizing errors rates for RRs.

The recommended approach that ranked highest in the survey for RRs is less stringent than Cochrane Reviews, but still involves one reviewer to do RoB with another reviewer to verify all judgements. In order to effectively manage this section of the RR, it is important to limit the RoB ratings to the primary outcomes included in the summary of findings tables, and to use, if possible, a valid RoB assessment tool specific to the study design(s) included in the RR.

Because grading is an involved process and takes time, for RRs, we recommend limiting to the main intervention and comparator, and to those outcomes deemed most critical to decisionmaking set out in advance, inclusive of harms. If using this grading system, at minimum the reasons for uprating or downrating should be transparently described in conjunction with evidence in tabular or narrative form.

To minimize duplication, and to ensure relevancy and transparency, author teams should confer with Cochrane or other registries (e.g., PROSPER, Open Science Framework [OSF]) before starting a RR. Specific to Cochrane, authors will be required to submit a completed protocol that will undergo editorial and methodological checks. For transparency, authors will need to register the Cochrane RR protocols with PROSPERO or the Open Science Framework (OSF) prespecifying the methods to be used. Cochrane currently has a streamlined intake process for COVID-19 priority topics and provides access to several resources (e.g., protocol and review templates) to support the review process. In future, we expect Cochrane will expand this workflow to accommodate RRs across other priority topics.

Sometimes changes to the protocol are necessary once the RR has started. For example, search parameters may be expanded or limited, depending on the search yields; or eligibility criteria may need to be tweaked following preliminary screening. Therefore, the RR process should allow for post-hoc changes. Significant changes should be discussed with the stakeholders involved, and any amendments tracked and reported. Moreover, authors should seek stakeholder feedback throughout the process to ensure the RR meets their information needs.

We strongly encourage the use of software in the production of RRs. Online systematic review software enhances collaboration by allowing for real-time project management and multiuser participation across geographic boundaries. Importantly, it enables members of the RR team to work in parallel across all stages of the review and provides a fully transparent process. It also facilitates the incorporation of protocol amendments and other changes to questions and forms often common to RRs. It also improves the data quality and efficiency through the automated collation of the screening results (inclusions/exclusions). Those undertaking RRs and other types of syntheses should look to ways to harness innovation, using software and adopting automation tools that reliably assist in expediting stages of review conduct. Importantly, given the methodological modifications inherent to RRs, authors must be transparent in reporting their methods and results. Although an extension to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for RRs is underway [18] , until it is officially completed, we suggest authors use the general PRISMA Statement to the extent possible and adapt it accordingly.

This article presents 26 recommendations as part of interim guidance developed by the Cochrane RRMG to standardize the conduct of RRs. Having specific guidance will improve the utility and robustness of the results of Cochrane RRs and more broadly contributes to the ongoing progression of RR methodology as a synthesis tool to be used more widely for timely evidencebased decision-making in healthcare.

A strength of this guidance is that recommendations were informed by a review of related RR methods studies across stages of conduct, two meta-epidemiological studies conducted by the J o u r n a l P r e -p r o o f RRMG, through gathering the opinions and preferences from a wider group of researchers involved in the Cochrane community using survey methods, and the involvement of RR methodology experts. Notably, this guidance provides actionable recommendations and minimum standards yet promotes a flexible and iterative RR process that supports a tailored approach for each review. For example, although survey respondents agreed Cochrane RRs should take no longer than six months, some of the shortcuts could be modified further or additional shortcuts taken to reflect very compressed timelines (e.g., <2 weeks) in circumstances when there is more urgent need for evidence.

actively used in the development of early Cochrane RR products to address pressing clinical questions posed by international stakeholders in response to COVID-19 [35, 36] . As such, its use is encouraged though we recognize that further enhancements and fine-tuning are needed. We plan to collect feedback on the guidance's perceived utility as applied in urgent, real-time RR scenarios in terms of the next steps. We also intend to adapt the guidance beyond interventions of effectiveness to other review types (e.g., RRs of diagnostic test accuracy or screening). Certain recommendations (e.g. involving stakeholders; involving an information specialist) will be applicable across other types of RRs, while some may not be (e.g. English-only inclusion may not be appropriate in some cases; database selection will potentially differ). Further, specific RR types will pose unique methodological issues [37] . There are many challenges to the conduct of RRs that further merit discussion including how to establish a manageable set of rank-ordered outcomes of most importance [38] . We also need to develop criteria for the appropriateness of undertaking RRs versus traditional SRs or living SRs. We intend to refine specific recommendations as new evidence emerges, building on work completed to date.

What sets our RR methods apart from other RR guidance is that it was developed within the context of Cochrane and focuses on RRs of interventions. Other RR guides focus on health policy and systems research, public health, or WHO guideline development in the face of public health emergencies [16, 20, 21] . The focus of our guide is on the conduct of RRs, not on planning, packaging or dissemination. Recommendations for every step of the review are very specific, compared to other guides that rather provide an overview of common RR practices. Although other RR guides are evidence-informed too, our guide is based on up-to-date empirical evidence evaluating RR methods complemented by an expert survey. Cochrane RRs are recommended to address only urgent and high priority questions explicitly requested by decision-makers, while other guides do not necessarily limit RR conduct to urgent questions only [20] (Web Appendix 4).

In spite of the survey response rate being somewhat lower than those typically obtained from Internet surveys (58%) [26] , it did include participation from representatives across 19 of 20

Cochrane groups approached to take part in this survey. Although not all stages of a RR had corresponding evidence to inform our survey, we know this is similar for SR methods in that many established steps in the process are based on limited, outdated, or no available evidence.

Importantly, the body of work underlying this guidance is mutually beneficial to informing both RRs and SRs.

Given RRs serve an important purpose for certain stakeholders, ongoing interest in them is expected to grow. Beyond COVID, this guidance will also be applicable for future circumstances when decisions need to be made in a period of weeks to a few months, and for which the traditional SR timeline does not meet the timeframe for urgent decision-making. Although the rationale and the context for this guidance is heavily posited within the Cochrane landscape, the J o u r n a l P r e -p r o o f recommended methods are widely applicable for anyone conducting a RR. As a leader in methods guidance on SRs, Cochrane is well positioned as an organization to not only produce RRs but with input of the Cochrane RRMG, to also advise on strengths and limitations of abbreviated methods and their potential impact on decision-making to minimize compromising validity. Endorsing a RR approach alongside interim methods guidance, demonstrates Cochrane's ability to respond quickly as a world leader in knowledge synthesis.

This paper offers new, interim guidance comprised of 26 methods recommendations to support the conduct of RRs produced within Cochrane and beyond. We hope this guidance will encourage thoughtful use of abbreviated SR practices for RRs and fosters further development through deliberation and formal assessment. Application of the proposed methods in exemplar RRs must be closely studied to understand methodological choices made and the challenges experienced. 

None of the authors report any conflicts of interest with respect to this manuscript. 

Findings at a Glance

(Moore 2017) [1] Study assessed the involvement of stakeholders in setting the review question/protocol

Knowledge brokering of proposals significantly improved the perceived clarity of information provided to policymakers including clarity as to why the review was commissioned; clarity of the review questions; clarity of the scope; clarity of the method; and clarity of the report conclusions. Further, it improved the confidence of reviewers that they can meet the policymakers' needs.

Meta-epidemiological study simulated the effects of RR methods on 2,512 Cochrane Reviews (with a meta-analysis, and a binary variable primary outcome using a relative effect measure) to assess the impact of restricting search dates Meta-epidemiological study simulated the effects of RR methods on 2,512 Cochrane Reviews (with a meta-analysis, and a binary variable primary outcome using a relative effect measure)

Findings suggests that limiting the search strategy to PubMed-only resulted in 19% of results with a ≥5% change in the meta-analyses (MA's). Further, using PubMed-only lost all studies in 3.7% of MA's and resulted in no important change in 81% of MA's. Authors suggest that PubMed-only searching might be considered in situations where a 10% risk of ≥20% change in odds ratio for the primary outcome is tolerable. The majority of effect size changes were small but moderate and large changes were relatively common.

(Nussbaumer-Streit 2018) [3] Based on a sample of 60 Cochrane reviews (which included 1,335 primary studies), the noninferiority of abbreviated searches allowing for a maximum of 10% changed conclusion was assessed.

When the reduction of the certainty of a conclusion was of concern, all abbreviated searches were inferior. Searching MEDLINE-only led to changed conclusions in 20% of the cases. However, Embase-only rendered the greatest proportion of changed conclusions (27%, 95% confidence interval [CI]: 16%-40%); combining MEDLINE, Embase, CENTRAL with searches of references lists the lowest (8%, 95% CI 3%-18%). When falsely reaching an opposite conclusion was of concern, combining one database with another or with searches of reference lists was noninferior to comprehensive searches (2%, 95% CI: 0%-9%).

This study concluded that searches should be done in at least two databases or in one database plus reference searching. Searching only a single electronic database is never a reliable method for any evidence synthesis and should be avoided for RRs.

Searching -Peer review of the search strategy ) [4] This study investigated a sample of search strategies without peer-review compared to peer-review (n=200 RR reports).

Findings suggest that in the absence of peer review, 2,507 potentially relevant records would not have been retrieved by the PubMed search strategy. However, this led to including only 4% (n=99) of these records in the reports. Unless captured in the accompanying grey literature search, these records would not have appeared in the published RRsthus reducing the integrity of the reports. The authors did not assess impact of these missed studies on conclusions.

(Nussbaumer-Streit 2019) [5] Based on an analysis of 59 randomly selected Cochrane Intervention Reviews with no language restrictions (which included 1281 studies), one study assessed whether limiting inclusion criteria solely to English language publications affected the overall conclusions of the reviews.

Findings suggest that although exclusion of non-English publications led to the exclusion of 31 studies (40 outcomes), exclusion of non-English studies did not markedly alter the size or direction of effect estimates or statistical significance. Overall, the proportion of changed conclusions in this sample was 0.0% (95% CI 0.0 -0.6) which indicated non-inferiority of the approach. Therefore, exclusion of non-English publications from SRs on clinical interventions had a minimal effect on overall conclusions and could be a viable methodological shortcut, especially for RRs.

J o u r n a l P r e -p r o o f 3

Study selection-PICobased title only screening to reduce overall screening effort (Rathbone 2017 ) [6] One study evaluated the feasibility of PICo-based title only screening by measuring the reduction in screening effort and maintenance of recall of relevant records using a sample of 10 datasets (31,359 records) from across completed SRs related to a variety of clinical topics. Five reviewers independently performed title only screening.

Results indicated PICo-based title only screening reduced screening effort (11-78% with a median reduction of 53%) and expedited citation screening, which is useful for RRs. However, this approach requires a thorough workup of the potential synonyms and alterative terms. PICobased title-only screening may be able to expedite citation screening, however there is a chance for missed studies.

Study selectiontitle/abstract screening [7] This study was an online, parallel-group RCT that assessed the accuracy of singlereviewer screening compared with dualreviewer screening. Using the Cochrane Crowd platform, eligible participants were randomized to a pharmacological or a public health SR and asked to screen 100 abstracts each following a training exercise.

Overall, 280 reviewers started screening abstracts of whom 239 (85%) completed the review of all 100 assigned abstracts. In total, reviewers made 24,942 screening decisions and on average each abstract was screened 12 times. Overall, single-reviewer screening achieved a sensitivity of 88.4% (95% confidence interval [CI], 83.6% to 91.9%). Dual-reviewer screening reached a sensitivity of 97.8% (95% CI, 95.5% to 99.0%). In summary, findings suggest that single-reviewer abstract screening misses about 13% of relevant studies. Based on prior research, 10% is a generally accepted level of risk decision-makers are willing to take for missing evidence; therefore, this could be a viable approach for RRs. J o u r n a l P r e -p r o o f

Expediting systematic reviews: methods and implications of rapid reviews

Evidence summaries: the evolution of a rapid review approach

Measuring the performance of the Cochrane library

Systematic review automation technologies

All in the Family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more

A typology of reviews: an analysis of 14 review types and associated methodologies

A scoping review of rapid review methods

Same family, different species: methodological conduct and quality varies according to purpose for five types of knowledge synthesis

Do policy-makers find commissioned rapid reviews useful?

Policymaker experiences with rapid response briefs to address health-system and technology questions in Uganda

Fit for purpose: perspectives on rapid reviews from end-user interviews

User survey finds rapid evidence reviews increased uptake of evidence by Veterans Health Administration leadership to inform fast-paced health-system decision-making

Moving Knowledge into Action: Developing the Rapid Synthesis and Translation Process Within the Interactive Systems Framework

The Use of Rapid Review Methods for the U.S. Preventive Services Task Force

Effectiveness of Personal Protective Equipment for Healthcare Workers Caring for Patients with Filovirus Disease: A Rapid Review

Developing WHO rapid advice guidelines in the setting of a public health emergency

Cochrane Rapid Reviews Methods Group to play a leading role in guiding the production of informed high-quality, timely research evidence syntheses

Developing PRISMA-RR, a reporting guideline for rapid reviews of primary studies (Protocol

Rapid Review Guidebook

World Health Organization, Alliance for Health Policy and Systems Research

Defining rapid reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews

Few evaluative studies exist examining rapid review methodology across stages of conduct: a systematic scoping review

Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study

Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

Mail and internet surveys: The tailored design method

Does knowledge brokering improve the quality of rapid review proposals? A before and after study

Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study

Abbreviated literature searches were viable alternatives to comprehensive searches: a metaepidemiological study

PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement

The impact of the peer review of literature search strategies in support of rapid review reports

Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review

Adjudication rather than experience of data abstraction matters more in reducing errors in abstracting data in systematic reviews

GRADE handbook for grading quality of evidence and strength of recommendations. The GRADE Working Group

Quarantine alone or in combination with other public health measures to control COVID-19: a rapid review

Hand cleaning with ash for reducing the spread of viral and bacterial infections: a rapid review

Challenges of rapid reviews for diagnostic test accuracy questions: a protocol for an international survey and expert consultation

Rapid review methods more challenging during COVID-19: Commentary with a focus on 8 knowledge synthesis steps

Does knowledge brokering improve the quality of rapid review proposals? A before and after study

Rapid reviews may produce different results to systematic reviews: a metaepidemiological study

Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study

The impact of the peer review of literature search strategies in support of rapid review reports

Excluding non-English publications from evidencesyntheses did not change conclusions: a meta-epidemiological study

Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews

Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

The Cochrane RRMG would like to acknowledge the support of Ella Flemyng (Cochrane Methods Implementation Coordinator) for facilitating survey administration. A further thanks to 

Cochrane RRs should: § be driven primarily by requests for timely evidence for decisionmaking High (95%)* Not applicable § be conducted to be to address urgent/emergent (public) health issues § to determine if a new, full Cochrane Review is warranted; or § to identify gaps in existing evidence High (94%)* Moderate (56%)* Moderate (54%)* § should take no longer than 12-26 weeks to complete; § a completion time of 12-16 weeks was also supported by the majority of respondents as a reasonable timeline depending on the topic to be reviewed High (78%)* Moderate (55%)* § should first focus on developing RRs addressing effectiveness of interventions in terms of types of RRs High (88%)* § should follow a 'tailored approach' using selected abbreviated methods most appropriate to the topic Moderate (53%) § 'Cochrane Rapid Review' is an accurate and sufficient label [Yes] High (75%)

Setting the Research Question (Topic Refinement) § R1. Cochrane RRs should directly involve key stakeholders (e.g., review users such as consumers, health professionals, policymakers, decision-makers) in contributing to setting the review question, eligibility criteria and the outcomes of interest Study Selection -Full-text Screening § R15. Most acceptable approach is single reviewer; with a second reviewer to screen all studies excluded by the first reviewer § Note: dual, independent screening of full-text articles was also endorsed.

Ranked #2C39 states that reviews should use (at least) two people working independently to determine whether each study meets the eligibility criteria, and define in advance the process for resolving disagreements.[M]Data extraction § R16. Most acceptable approach is single reviewer with full verification by a second reviewer of the data § Note: single extraction; with verification by a second reviewer of a proportion of study characteristics and all outcome data was also endorsed.

Ranked #2We note that for Cochrane Reviews, data extraction may be separated into two parts: C45 and C46 state to use (at least) two people working independently to extract study characteristics [HD] and outcomes data [M] from reports of each study, and define in advance the process for resolving disagreements. § R17. Only a minimal data set should be extracted (i.e., streamlining how much information is extracted about the study characteristics, the interventions, and outcomes data) High (71%) Not applicable. § R18. Consider using data from existing SRs when possible to reduce time spent extracting Moderate (59%) Not applicable. § Only more experienced systematic reviewers should be involved in data extraction Moderate (69%) Not applicable.

§ R19. Most acceptable approach (ranked 1) -single reviewer with full verification of all judgements (and support statements) by a second reviewer § Note: dual independent review was also endorsed.

Ranked #2C53 states to use (at least) two people working independently to apply the risk-of-bias tool to each result in each included study, and define in advance the process for resolving disagreements. *Instances where two categories were collapsed (e.g., strongly agree/agree; extremely useful/very useful); † Indicates when a survey question only reached a 'low-level' of agreement for each response. We still recommended the response that scored highest (even in the absence of moderate or high-level endorsement) to ensure each stage of conduct had an accompanying recommendation.[HD] Highly desirable; [M] Mandatory J o u r n a l P r e -p r o o f Table 1 

Setting the Research Question -Topic Refinement • Involve key stakeholders (e.g., review users such as consumers, health professionals, policymakers, decisionmakers) to set and refine the review question, eligibility criteria, and the outcomes of interest. Consult with stakeholders throughout the process to ensure the research question is fit for purpose, and regarding any adhoc changes that may occur as the review progresses. (R1) • Develop a protocol that includes review questions, PICOS, and inclusion and exclusion criteria.Setting Eligibility Criteria • Together with key stakeholders:Clearly 

• Using a standardized title and abstract form, conduct a pilot exercise using the same 30-50 abstracts for the entire screening team to calibrate and test the review form. • Use two reviewers for dual screen of at least 20% (ideally more) of abstracts, with conflict resolution.• Use one reviewer to screen the remaining abstracts and a second reviewer to screen all excluded abstracts, and if needed resolve conflicts. (R14)

• Using a standardized full-text form, conduct a pilot exercise using the same 5-10 full-text articles for the entire screening team to calibrate and test the review form. • Use one reviewer to screen all included full-text articles and a second reviewer to screen all excluded full-text articles. (R15) Data Extraction • Use a single reviewer to extract data using a piloted form. Use a second reviewer to check for correctness and completeness of extracted data. (R16) • Limit data extraction to a minimal set of required data items. (R17) • Consider using data from existing SRs to reduce time spent on data extraction. (R18) J o u r n a l P r e -p r o o f

• Use a valid risk of bias tool, if available for the included study designs.• Use a single reviewer to rate risk of bias, with full verification of all judgements (and support statements) by a second reviewer. (R19) • Limit risk of bias ratings to the most important outcomes, with a focus on those most important for decisionmaking. (R20) Synthesis • Synthesize evidence narratively.• Consider a meta-analysis only if appropriate (i.e., studies are similar enough to pool). (R21) Standards for conducting a meta-analysis for a SR equally apply to a RR. • Use a single reviewer to grade the certainty of evidence, with verification of all judgements (and footnoted rationales) by a second reviewer. (R22) Other Considerations for Cochrane RRs RRs should be preceded by a protocol submitted to and approved by Cochrane (R23); the protocol should be published (e.g., PROSPERO or Open Science Framework) (R24); allow for post-hoc changes to the protocol (eligibility criteria etc.) as part of an efficient and iterative process (R25); document all post-hoc changes; and incorporate use of online SR software (e.g., Covidence, DistillerSR, EPPI-Reviewer) to streamline the process (R26).

None of the authors report any conflicts of interest with respect to this manuscript.