key: cord-0900764-ng1sv0ui authors: McDonald, Steve; Turner, Simon; Page, Matthew J; Turner, Tari title: Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency date: 2022-02-19 journal: J Clin Epidemiol DOI: 10.1016/j.jclinepi.2022.02.006 sha: 5de316af614ecbad693faa9ed2f861eab03df722 doc_id: 900764 cord_uid: ng1sv0ui OBJECTIVE: To investigate the completeness and currency of published systematic reviews of remdesivir for COVID-19 and to compare this with a living guidelines approach. STUDY DESIGN AND SETTING: In this cross-sectional study, we searched Europe PMC on May 20, 2021 for systematic reviews of remdesivir (including preprints, living review updates). Completeness and currency were based on the inclusion of four major randomized trials of remdesivir available at the time of publication of the review (including as preliminary results and preprints). RESULTS: We included 38 reviews (45 reports), equivalent to a new publication every nine days. Twenty-three (51%) reports were out of date at the time of publication. Eleven reviews that were current on publication had a median survival time of 10 days (range 4–57). A third of reviews cited other systematic reviews, but only four provided justification for why another review was necessary. Eight (21%) of the reviews were registered in PROSPERO. The Australian COVID-19 Clinical Evidence Taskforce living guidelines were updated within 14 days for three of the remdesivir trials, and within 28 days for the fourth. CONCLUSION: There was considerable duplication of systematic reviews of remdesivir, and half were already out of date at the time of publication. Key findings  We identified 38 systematic reviews of remdesivir for COVID-19 published to May 2021, equivalent to a new publication every nine days. Half were out of date at the time of publication and of those that were initially current, the median survival time was 10 days. Only a fifth of these reviews were registered in PROSPERO.  Evidence has shown duplication of systematic reviews to be a persistent problem, leading to wasted effort among reviewers and adding to the challenges faced by users in selecting reliable, up-to-date summaries of evidence.  This empirical study measured the extent of this problem in the context of COVID-19 by investigating the completeness and currency of published systematic reviews of remdesivir.  The susceptibility of systematic reviews to becoming out of date, especially when evidence is rapidly accumulating, should further accelerate the adoption of living evidence approaches. Despite the efforts of prospective registration to deter duplication, more efforts are needed to dissuade the conduct and publication of redundant reviews. Research proliferation continues to be a feature of the COVID-19 pandemic, stress-testing the systems by which evidence is produced [1] . By January 2022 there were over 225,000 publications in the National Institute of Health's iSearch COVID-19 portfolio (https://icite.od.nih.gov/covid19/search/) and 110,000 studies in the Cochrane COVID-19 Study Register (https://covid-19.cochrane.org/). This rapid production of evidence-the socalled infodemic-also extends to evidence synthesis [2] . The COVID-19 Evidence Reviews resource (https://www.covid19reviews.org/) lists over 4,500 published systematic reviews and other syntheses, with a similar number underway. The growth in research output stoked by the pandemic has raised concerns about duplication of research effort, scientific rigour, and the spread of misinformation [3] [4] [5] . Ideally, bringing together studies in systematic reviews should provide decision-makers with timely evidence to inform their decisions. However, the rapid accumulation of evidence during the pandemic has created challenges, not least that evidence syntheses are only useful for the period in which they include all the relevant, reliable evidence to underpin health decisions. In areas of clinical uncertainty where research is rapidly accruing, living methods (whereby evidence is continually identified and included) have been proposed as a strategy for ensuring evidence syntheses are up to date [6] . The lack of evidence to guide treatment decisions early in the pandemic led to several initiatives to establish rapid and living guidance [7] [8] [9] , as well as sites dedicated to living meta-analysis and evidence synthesis of COVID-19 trials, such as metaEvidence.org and COVID-NMA.com. In Australia, the National COVID-19 Clinical Evidence Taskforce produced GRADE-based living guidelines for the clinical care of people with COVID-19 that were updated weekly throughout 2020 [10, 11] . Daily evidence surveillance for these guidelines involved identifying randomized trials of treatments and monitoring relevant reviews and syntheses. Early in the pandemic, the antiviral drug remdesivir emerged as a potential treatment for COVID-19 [12] . Since publication of the first placebo-controlled randomized trial on April 29, 2020 [13] , seven randomized trials involving over 8700 participants have been published up to September 2021, with all but one comparing remdesivir with placebo or standard care (Table 1 ). In addition to identifying published reports of randomized trials and announcements of preliminary findings, the continual evidence surveillance for the Australian COVID-19 Taskforce guidelines also detected a flurry of systematic reviews of remdesivir, despite the limited availability of primary studies. It soon became apparent that published reviews of remdesivir far-outnumbered primary studies, suggesting substantial duplication in synthesis of evidence. Further, many of these reviews appeared out of date at the time of publication, limiting their value for supporting clinical decisions. This study was conceived in response to the high number of remdesivir SRs we observed while conducting our living guideline for COVID-19. We did not prepare a protocol for the current study. We searched Europe PMC (includes PubMed and all major preprint servers) on May 20, 2021 for systematic reviews of remdesivir for the treatment of COVID-19. We adopted inclusive eligibility criteria, such that any review with the aim of evaluating the clinical effects of remdesivir in humans was included, irrespective of whether authors explicitly used the term systematic review. We considered reviews eligible regardless of whether they used less than optimal methods, such as not conducting risk of bias assessments. Preprints and reviews published in languages other than English were eligible. Exclusion criteria were protocols, reviews of adverse effects only, and reviews of multiple antivirals for the treatment of COVID-19, which may or may not have included remdesivir. The exception was to include two major living systematic reviews of all treatments for COVID-19 [25, 26] . The rationale for including these was to assess the contribution published living reviews make in circumstances where evidence is accumulating rapidly. One author (SM) screened titles/abstracts and reviewed the full text of potentially eligible reviews. A second author (TT) independently checked assessments, with any discrepancies resolved through discussion. One author (SM) extracted data on the type of review (as described by the review authors), country of lead author, aims of the review, date of search, number of records screened, publication date, details of the randomized trials included in the review, and whether the review included a risk of bias assessment and/or had used GRADE to assess the certainty of evidence. We focused on extracting information on randomized trials, ignoring observational designs, such as cohort studies. Although it was beyond the scope of this study to appraise the conduct of the reviews or consider the results or conclusions, we were interested in the extent to which the systematic reviews were replications (i.e., authors intentionally redo a particular systematic review to test whether the same findings as the original review are found when using either similar or modified methods) or duplications (i.e., unacknowledged repetition of a review without a clearly defined purpose for the repetition) [27, 28] . To explore this issue, one author (MJP) looked at whether review authors cited other systematic reviews of remdesivir, provided justification for doing the review given others already existed (and what that justification was), or discussed how the results of their review compared with others. For each systematic review report included in the analysis (i.e., preprint, journal publication, living review update), we determined if the report was current or out of date at the time it was published. A systematic review report was deemed complete and current if it included the four major randomized trials of remdesivir versus placebo or standard care that were available at the time of publication (whether as preliminary results, preprint, or journal article) [13, 15, 19, 21] . Despite the ACTT-1 and SIMPLE-2 trials reporting their preliminary results as media releases [14, 18] , sufficient data about the effects of remdesivir were provided such that we deemed any systematic review missing these studies to be critically deficient and out of date for the purposes of informing clinical decisions. By limiting assessment of completeness and currency to the inclusion of the four placebo or standard-ofcare trials, we ignored the one trial that compared different durations of remdesivir treatment, as this study did not directly address whether remdesivir is an effective treatment [17] . We also ignored the smallest trial, since this contributed very little data and thus had minimal impact on the evidence available [22] . (The most recent randomized trial was published after the period covered by our analysis [23] .) These living guidelines were updated weekly from April to December 2020 and continued to be updated every 2-3 weeks during 2021 [11] . We reviewed previous versions of the guidelines to extract information on when each randomized trial of remdesivir was added to the guideline, noting when the trial was first listed in the guideline (as being under review) and when the results were fully incorporated into the guideline recommendation. We searched PROSPERO on June 1, 2021 for register entries with remdesivir in the title and checked in Europe PMC whether any had been published subsequently. We also checked PROSPERO to see which of the systematic reviews included in our cohort had been registered. We summarized data as frequency and percentages for categorical characteristics and medians and interquartile ranges for continuous characteristics. The timeline showing the publication of trials and reviews was recorded in Excel and plotted using Stata SE, version 16.1 (StataCorp). We included 36 reviews of remdesivir plus two major living reviews of all treatments for COVID-19 [25, 26] (collectively reported in 45 publications up to May 20, 2021). The first review was published on April 30, 2020 [42] , the day after the first randomized trial of remdesivir and the preliminary findings of two other trials were published. On average, across the period analysed, a new review was published every nine days. Reviews were in English (or had English versions) and had lead authors from 20 countries, with the U.S. and China accounting for six and five reviews, respectively. All reviews had the stated purpose of evaluating the effects of remdesivir as a treatment for COVID-19. Thirtyone (82%) reviews used the term systematic, meta-analysis, living or rapid in the title. Of the four living systematic reviews, one had published two updates, two had published one update, and one had not published any update in the period covered by our analysis. Nine were first published as preprints. Of these, three were subsequently published without any changes to the search date or studies included (shown as single reviews in Figure 1 ) [54, 55, 60] , and one was updated with a new search and additional studies (shown as separate reviews in Figure 1 ) [49, 50] . Excluding the two living reviews of all treatments for COVID-19, the median number of records screened per review (in the 24 reviews that reported this information) was 694 (IQR For the purposes of determining currency, we considered each report of a systematic review separately (i.e., preprint versions, living review updates). Of the 45 reports, 23 (51%) were out of date at the time of publication (i.e., they did not include the results of one or more of the four placebo-controlled or standard care trials that were already published or for which preliminary results were available) (Figure 1 ). Eleven reviews that were current at the time of publication subsequently became out of date as further trials were published. The median duration of currency for these 11 reviews was 10 days (range 4-57). As of May 20, 2021, 11 systematic reviews of remdesivir remained current. However, none of these 11 included the smallest trial that contributes <1% of the data across all the remdesivir studies [22] . The two living reviews of remdesivir by Verdugo-Paiva and Wilt were first published shortly before publication of the WHO Solidarity trial and remained current for 17 and 10 days, respectively [65, 66] . The first update to the Wilt review was published on February 9, 2021, over 16 weeks after the WHO Solidarity trial was published [39] . The living review by Verdugo-Paiva has not been updated as of September 2021. The search date was not reported in five (13%) reviews. For the 40 A third (12/36) of reviews cited other systematic reviews of remdesivir, either in the introduction or discussion, but only four provided justification in the introduction for why another review was necessary. Only one of these reviews [40] was a replication of a previous systematic review, which focused on alternative outcomes. In two instances the availability of additional trials was cited as the reason for updating existing reviews and a difference in the review's scope was cited in another as justification for doing the review [41, 48, 59] . Of the 20 reviews published since October 2020, only 11 commented in the discussion on how their review's findings compared with other reviews of remdesivir. One review optimistically stated that theirs was the first to provide evidence on the efficacy and safety of remdesivir in COVID-19, despite many reviews having already been published at the time it was received by the journal [32] . For the four placebo-controlled or standard care trials of remdesivir, the time from publication of the results to full inclusion in the published guidelines was 9, 13, 27 and 14 days, respectively (Table 2, Figure 2 ). Using the same criterion to determine currency as was applied to the systematic reviews, the Taskforce guidelines were current from May 8 to June 1, 2020, from September 17 to October 15, 2020, and from October 29, 2020 onwards. This study investigated the completeness and currency of systematic reviews of remdesivir published since the start of the COVID-19 pandemic. Despite there being only a handful of randomized trials, 38 systematic reviews of remdesivir had been published by May 2021, equivalent to a new publication every nine days. Half were out of date at the time of publication and of those that were initially current, the median survival time was 10 days. Only a fifth of these reviews were registered in PROSPERO. In situations where evidence is rapidly accumulating, readers of reviews need to easily determine a review's currency based on the search date. Yet despite the recommendations of well-established reporting guidelines such as PRISMA [67] , 43% of reviews failed to report the search date in the abstract and in 13% of reviews this information was missing entirely. Our data suggests that, in the absence of search dates, publication date is unlikely to be a reliable indicator of the most current review, at least for journal articles. Although the median time from search to publication was nine weeks, this ranged from a few days to 39 weeks. Studies have reported substantially shorter acceptance times by journals for COVID-19 publications [4, 68] , but this has not been sufficient to prevent half the remdesivir reviews being out of date when published. The prominence of preprints during the pandemic has accelerated scientific dissemination [69] , but although most of the reviews published as preprints in our analysis were posted within three weeks of the search date, this was no guarantee of currency, with 45% (4/9) still out of date at time of publication. The obvious duplication and high rate of redundancy we observed calls into question the value of traditional systematic reviews for informing decisions in situations where the evidence base is rapidly developing, such as pandemics of novel diseases. Even the presence of several living reviews appears to have done little to deter review teams from registering or conducting many duplicate reviews. Understanding the motivation of authors behind these reviews would be instructive. For some review teams, the opportunity to publish a relatively straightforward systematic review to derive academic credit may serve as the compelling motive, while for others, the impetus might be the need to inform clinical practice through the development of robust guidance. In this latter situation, even if review teams are aware that other reviews are underway, there are no guarantees these reviews will be completed, and if they are, when results will be available. As has been the case with remdesivir, review findings may be contentious and interpreted differently across jurisdictions and settings, legitimising the need to replicate an existing review. However, although around half the reviews published in the most recent six months of our analysis did acknowledge the existence of other reviews of remdesivir, very few (10%) provided a justification for why another systematic review was necessary. The speed at which published systematic reviews become out of date in areas of rapid research generation has underscored the importance of adopting living evidence approaches [7] . In this study, of the two living reviews of remdesivir, only one has published successive updates [39] . This review was current for 10 days following the original published version but was then out of date for over 16 weeks until it had incorporated the results of the largest trial. By contrast, living guidelines that use online publishing platforms have greater flexibility and can publish updated findings more quickly as evidence becomes available. Demonstrating the feasibility of this approach, the National COVID-19 Clinical Evidence Taskforce guidelines were updated within two weeks whenever studies were published that necessitated changes in the recommendation. Alongside living guidelines, several initiatives dedicated to living meta-analysis and evidence synthesis of COVID-19 trials have been established that also avoid the shortcomings of published systematic reviews, whether traditional or living. Sites such as metaEvidence.org and COVID-NMA.com provide decision-makers with timely access to recently published trials and rapidly updated syntheses. By our estimate, at least 50 systematic reviews of remdesivir had either been published or registered by May 2021. And given the low rate of registration observed, it is likely even more are underway, especially when reviews of antivirals more broadly are included. The waste of effort observed in this empirical study is a persistent problem with systematic reviews that is likely becoming more entrenched [70, 71] , despite the preventive measure of prospective registration [72] . Since March 2020, a message on the PROSPERO site urges authors to check the register for similar, existing COVID-19 reviews and to not duplicate without good reason. Although it is impossible to know how many review teams abandoned their plans to conduct a systematic review because of this advice, many went ahead regardless, contributing to avoidable research waste and adding to the noise decision makers must contend with. Further steps to reduce unintended duplication deserve consideration, perhaps including a requirement for authors to explain why a new review is warranted. Although several meta-research studies have explored different aspects of COVID-19 research, such as volume, characteristics, and quality [2, 4, 73, 74] , this is the first study to our knowledge that has focused on the extent of duplication and currency among published systematic reviews of one intervention. In addition, looking at which studies were included in the reviews allowed us to contrast the utility of systematic reviews with living guidelines for informing up-to-date decisions. By excluding systematic reviews of all antivirals or all treatments for COVID-19 (except for two living reviews), the true number of systematic reviews assessing the effects of remdesivir is likely much higher. We also limited our search to a single database (Europe PMC), which, although it includes the major preprint servers, may have led us to miss additional systematic reviews. Living guidelines are but one example of a living evidence approach and we acknowledge that online platforms dedicated to living meta-analysis of COVID-19 studies would also demonstrate the feasibility of a living evidence approach. There was considerable duplication of systematic reviews of remdesivir for the treatment of COVID-19 despite the efforts of prospective registration to deter this. The accumulation of data from randomized trials over the course of the pandemic and the lag between conducting and publishing reviews meant that most were already out of date at the time of publication, and those current at time of publication rapidly became out of date. The escalation in systematic reviews observed during the pandemic not only represents redundant effort but is also a hinderance to users who are faced with a barrage of reviews of questionable value. Living evidence approaches are a feasible alternative to the limited usefulness of systematic reviews in providing timely, robust evidence to support decision-making. How COVID broke the evidence pipeline Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study Waste in covid-19 research Methodological quality of COVID-19 clinical research The truth is out there, somewhere Living systematic review: 1. Introduction-the why, what, when, and how Rapid and Living Guidance for COVID-19 Developing Guidelines Before, During, and After the COVID-19 Pandemic Developing Treatment Guidelines During a Pandemic Health Crisis: Lessons Learned From COVID-19 National COVID-19 Clinical Evidence Taskforce. Australian guidelines for the clinical care of people with COVID-19. 2021. www.covid19evidence.net.au Weekly updates of national living evidence-based guidelines: methods for the Australian living guidelines for care of people with COVID-19 Hopes rise for coronavirus drug remdesivir Remdesivir in adults with severe COVID-19: a randomised, double-blind, placebo-controlled, multicentre trial NIH Clinical Trial Shows Remdesivir Accelerates Recovery from Advanced COVID-19. www.niaid.nih.gov/news-events/nih-clinical-trial-shows-remdesivir-acceleratesrecovery-advanced-covid-19 Remdesivir for the Treatment of Covid-19 -Final Report Gilead Announces Results From Phase 3 Trial of Investigational Antiviral Remdesivir in Patients With Severe COVID-19 Remdesivir for 5 or 10 Days in Patients with Severe Covid-19 Gilead Announces Results From Phase 3 Trial of Remdesivir in Patients With Moderate COVID-19 Effect of Remdesivir vs Standard Care on Clinical Status at 11 Days in Patients With Moderate COVID-19: A Randomized Clinical Trial Repurposed antiviral drugs for COVID-19-interim WHO SOLIDARITY trial results Repurposed Antiviral Drugs for Covid-19 -Interim WHO Solidarity Trial Results Clinical outcomes of using remdesivir in patients with moderate to severe COVID-19: A prospective randomised study Remdesivir for the Treatment of Hospitalised Patients with COVID-19 (DisCoVeRy): A Randomised, Controlled, Open-Label Trial Remdesivir plus standard of care versus standard of care alone for the treatment of patients admitted to hospital with COVID-19 (DisCoVeRy): a phase 3, randomised, controlled, open-label trial Interventions for treatment of COVID-19: A living systematic review with meta-analyses and trial sequential analyses (The LIVING Project) Drug treatments for covid-19: living systematic review and network meta-analysis One more time": why replicating some syntheses of evidence relevant to COVID-19 makes sense When to replicate systematic reviews of interventions: consensus checklist Remdesivir for the treatment of COVID-19: A systematic review and meta-analysis of randomized controlled trials Remdesivir use in patients with coronavirus COVID-19 disease: a systematic review and metaanalysis of the Chinese Lancet trial with the NIH trial Use of remdesivir for patients with Covid-19: a review article Mortality Benefit of Remdesivir in COVID-19: A Systematic Review and Meta-Analysis Remdesivir in Treatment of COVID-19: A Systematic Benefit-Risk Assessment Efficacy and safety of remdesivir in hospitalized Covid-19 patients: Systematic review and meta-analysis including network meta-analysis Remdesivir for the treatment of coronavirus COVID-19: A meta-analysis of randomised controlled trials Remdesivir and its antiviral activity against COVID-19: A systematic review A Review on Remdesivir: A Possible Promising Agent for the Treatment of COVID-19 Effectiveness of remdesivir for the treatment of hospitalized COVID-19 persons: A network meta-analysis Update: Remdesivir for Adults With COVID-19 : A Living Systematic Review and Meta-analysis for the American College of Physicians Practice Points Effect of remdesivir on mortality in patients with COVID-19: A meta-analysis of randomized control trials Clinical efficacy and safety of remdesivir in patients with COVID-19: a systematic review and network meta-analysis of randomized controlled trials Rapid review for the anti-coronavirus effect of remdesivir Remdesivir in Coronavirus Disease 2019 (COVID-19) treatment: a review of evidence The efficacy and safety of remdesivir in the treatment of patients with COVID-19: a systematic review and meta-analysis. medRxiv Remdesivir for the Treatment of COVID-19: A Systematic Review of the Literature Use of Remdesivir in the Management of COVID-19: A Systematic Review on Current Evidences Remdesivir for coronavirus disease 2019 (COVID-19): a systematic review with metaanalysis and trial sequential analysis of randomized controlled trials Clinical studies assessing the efficacy, effectiveness and safety of remdesivir in management of COVID-19: A scoping review Efficacy and harms of remdesivir for the treatment of COVID-19: A systematic review and meta-analysis Efficacy and harms of remdesivir for the treatment of COVID-19: a systematic review and meta-analysis Remdesivir therapy in patients with COVID-19: A systematic review and meta-analysis of randomized controlled trials Remdesivir for treatment of COVID-19; an updated systematic review and meta-analysis Impact of remdesivir on 28 day mortality in hospitalized patients with COVID-19 The efficacy of remdesivir in coronavirus disease 2019 (COVID-19): a systematic review Randomized controlled trials of remdesivir in hospitalized coronavirus disease 2019 patients: A meta-analysis A potential game-changer or just a myth? A systematic review and meta-analysis Remdesivir in COVID-19: A critical review of pharmacology, pre-clinical and clinical studies Is remdesivir important in clinical practice as a treatment of COVID-19? A study based on metaanalysis data Safety and Efficacy of Remdesivir for the Treatment of COVID-19: A Systematic Review and Meta-Analysis Remdesivir for the treatment of COVID-19: a living systematic review Remdesivir use in the coronavirus disease 2019 pandemic: A mini-review Effectiveness and Safety of Remdesivir in Patients with COVID-19: A Systematic Review and Meta-Analysis. Research Square Effect of remdesivir on patients with COVID-19: A network meta-analysis of randomized control trials Efficacy and Safety of Remdesivir for COVID-19 Treatment: An Analysis of Randomized, Double-Blind, Placebo-Controlled Trials Remdesivir for the treatment of COVID-19: A living systematic review Remdesivir for Adults With COVID-19 : A Living Systematic Review for American College of Physicians Practice Points The PRISMA 2020 statement: an updated guideline for reporting systematic reviews Methodological assessment of systematic reviews and meta-analyses on COVID-19: A meta-epidemiological study The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses Overlapping meta-analyses on the same topic: survey of published studies Why prospective registration of systematic reviews makes sense The quality of systematic reviews and other synthesis in the time of COVID-19 Critical Review of the Scientific Evidence and Recommendations in COVID-19 Management Guidelines We thank Rhiannon Tate for creating Figure 2 .