key: cord-0955961-ndsohxa1
authors: Chen, Yuehong; Li, Ling; Zhang, Qiuping; Liu, Huan; Huang, Yupeng; Lin, Sang; Yin, Geng; Xie, Qibing
title: Epidemiology, methodological quality, and reporting characteristics of systematic reviews and meta-analyses on coronavirus disease 2019: A cross-sectional study
date: 2021-11-24
journal: Medicine (Baltimore)
DOI: 10.1097/md.0000000000027950
sha: 764af3ccf3c789c1e50feffeab3a0cce75d007ed
doc_id: 955961
cord_uid: ndsohxa1

During the coronavirus disease 2019 (COVID-19) pandemic, convenient accessibility and rapid publication of studies related to the ongoing pandemic prompted shorter preparation time for studies. Whether the methodological quality and reporting characteristics of published systematic reviews (SRs)/meta-analyses are affected during the specific pandemic condition is yet to be clarified. This study aimed to evaluate the epidemiology, methodological quality, and reporting characteristics of published SRs/meta-analyses related to COVID-19. The Ovid Medline, Ovid Embase, Cochrane Library, and Web of Science electronic databases were searched to identify published SRs/meta-analyses related to the COVID-19 pandemic. Study screening, data extraction, and methodology quality assessment were performed independently by 2 authors. The methodology quality of included SRs/meta-analyses was evaluated using revised version of a measurement tool to assess SRs, and the reporting characteristics were assessed based on the preferred reporting items for SRs and meta-analyses guidelines. A total of 47 SRs/meta-analyses were included with a low to critically low methodological quality. The median number of days from the date of literature retrieval to the date that the study was first available online was 21 days; due to the limited time, only 7 studies had study protocols, and the studies focused on a wide range of COVID-19 topics. The rate of compliance to the preferred reporting items for SRs and meta-analyses checklists of reporting characteristics ranged from 14.9% to 100%. The rate of compliance to the items of protocol and registration, detailed search strategy, and assessment of publication bias was less than 50%. SRs/meta-analyses on COVID-19 were poorly conducted and reported, and thus, need to be substantially improved.

A systematic review (SR) is intended to integrate all currently available pieces of evidence that meet the predefined eligibility criteria in order to address a specific research question using specific and systematic methods that have been tested to minimize bias and provide more reliable findings, from which conclusions can be drawn and decisions can be made. Many SRs include meta-analysis, which is a statistical method used to synthesize the results of several independent studies. [1] SRs/meta-analyses provide the highest level of evidence because they examine all the available evidence instead of individual studies with reproducible and rigorous methods, and evaluate the entire body of evidence to ensure the reliability of the results. [2] With the dissemination of SRs/meta-analyses and training in methodology, the number of SRs/meta-analyses is increasing rapidly; more than 30,000 SRs/meta-analyses were published in 2019. For Cochrane reviews, the annual prevalence of SRs/meta-analyses increased by three-fold from 2004 to 2014. [3] Nevertheless, evidence from a SR/meta-analysis may be limited for use if the methodology or reporting is flawed. [4] To assess the methodological quality of SRs/meta-analyses, the revised instrument of a measurement tool to assess systematic reviews (AMSTAR-2) tool for assessing SRs, a critical appraisal tool, was published in 2017. [5] To improve the reporting of SRs/meta-analyses, preferred reporting items for systematic reviews and meta-analyses (PRISMA) statements [6] and Meta-Analysis Of Observational Studies in Epidemiology guidelines [7] were published. However, the reporting guidelines were only applied in 29% of SRs, and the rate of compliance to the items of reporting guidelines varied, ranging from 0% to 93%. [3] The coronavirus disease 2019 (COVID-19) is caused by severe acute respiratory syndrome coronavirus 2, which has a high case fatality rate (6.3%-15.23%), [8] and has attracted attention worldwide. The number of published studies on COVID-19 is increasing rapidly, and over 10,000 studies have been published since the pandemic. The number of SRs/meta-analyses is also accumulating. Whether the convenient accessibility, rapid publication of studies on COVID-19, and shorter preparation time affect the methodological quality and reporting characteristics of published SRs/meta-analyses, during the specific period of the COVID-19 pandemic, still needs to be clarified. To date, no study has assessed the methodological quality and reporting characteristics of SRs/meta-analyses related to COVID-19. Therefore, this cross-sectional study aimed to investigate the epidemiology, methodological quality, and reporting characteristics of published SRs/meta-analyses related to COVID-19.

This cross-sectional study aimed to investigate the epidemiology, methodological quality, and reporting characteristics of published SRs/meta-analyses related to COVID-19. Ethical approval was not required, as this study was cross-sectional in nature and only included SRs and meta-analyses.

According to the definition provided by the Cochrane book, the following criteria were employed: the study is a SR and/or a metaanalysis, or a systematic scoping review, a scoping review, a systematic rapid living review, or a rapid review; the study clearly describes the search strategy; the study explicitly illustrates the study selection criteria and study selection; the study reports the statistical methods to synthesize data including but not limited to heterogeneity assessment, sensitivity analysis, subgroup analysis, and assessment of publication bias if applicable, or the descriptive summary when meta-analysis cannot be performed [3] ; and the topic of published SRs/meta-analyses is related to COVID-19 regardless of the specific research theme. SRs/meta-analyses that were published in English or Chinese were included.

Meanwhile, duplicate studies, letters, and study protocols were excluded from the analysis.

To perform a systematic retrieval of literatures, the Ovid PubMed, Ovid Embase, Cochrane Library, and Web of Science were searched using both Mesh terms and keywords, without language limitations, and the literature search was conducted from January 1, 2019 to April 28, 2020. The search terms used were as follows: "systematic review," "meta-analysis," and "COVID-19"; the detailed search strategy can be found in Supplemental Digital Content, http://links.lww.com/MD/G491. The reference lists of included studies were manually checked to identify potential eligible studies.

The study screening was performed in 2 steps: title and abstract screening, and full-text screening. The references retrieved from electronic databases were introduced to Endnote; after removal of duplicates, the studies were exported to the Microsoft Access Database for preliminary screening of titles and abstracts. The full-texts of the remaining studies obtained from the first screening were downloaded and read for further screening based on the study eligibility criteria. The reference lists of all included SRs and meta-analyses were manually checked. Two authors independently screened the studies, and any disagreement was resolved via discussion or adjudication by a third reviewer, if necessary.

Two authors independently collected the following data: name of the first author; country of the first author; study question; main conclusion; number of authors; journal impact factor; literature retrieval date; date that the study was first available online; focus of review; number of included studies; study design and total number of participants included in the SRs/meta-analyses; risk of bias assessment tool; reporting guidelines mentioned; publication language; literature retrieved; whether the article is a Cochrane review or not; type of review; certainty of evidence based on the Grading of Recommendations Assessment, Development, and Evaluation; funding information; statistic information of the meta-analyses including the software used for data analysis; judgement standards for statistical significance; indicators of data pooling; model for data pooling; method for meta-analysis; statistical method for determining the heterogeneity; method used to assess for publication bias; and additional analyses, like subgroup analysis, sensitivity analysis, and meta-regression. Any disagreement on data extraction was resolved via discussion or adjudication by a third reviewer if necessary.

The methodological quality of all included SRs/meta-analyses was assessed using AMSTAR-2, [5] which has a total of 16 items and is used to assess the quality of SRs that enroll both randomized controlled trials (RCTs) and observational studies on Chen et al. 100: 47 Medicine healthcare interventions. Each item is answered by partially yes, yes, or no. The overall quality can be categorized as critically low, low, moderate, or high, based on whether the study has noncritical or critical weakness domains and on the number of noncritical or critical flaw domains. Two authors independently performed the quality assessment, and any disagreement was resolved via discussion or adjudication by a third reviewer, if necessary.

The reporting information of each study was evaluated based on the PRISMA guidelines; if the information was reported according to the PRISMA checklist, the answer related to this item was yes; otherwise, the answer was no. Two authors independently performed the PRISMA checklist assessment, and any disagreement was resolved via discussion or adjudication by a third reviewer, if necessary.

The number was counted, and the proportions were calculated. The statistical distribution difference of counts was assessed using a chi-square (x 2 ) test. Statistical significance was judged by a P value of <.05 by a two-tailed test. The information of each study was described and summarized in tables.

A total of 319 references were retrieved from the Ovid PubMed (n = 22), Ovid Embase (n = 176), Cochrane Library (n = 2), and Web of Science (n = 119). After removal of duplicates (n = 120), 199 studies underwent title and abstract screening, and 96 studies were excluded. A total of 103 studies underwent full-text reading, and 56 studies were excluded. No eligible studies were identified after manually checking the reference lists of all included studies. Overall, 47 SRs/meta-analyses were included in the final analysis ( Fig. 1 ).

Of the 47 included studies, only 13 were SRs, [10, 15, 18, 20, 22, 23, 26, 29, [38] [39] [40] 43, 52] while 34 were meta-analyses. [9, [11] [12] [13] [14] 16, 17, 19, 21, 24, 25, 27, 28, [30] [31] [32] [33] [34] [35] [36] [37] 41, 42, [44] [45] [46] [47] [48] [49] [50] [51] [53] [54] [55] A total of 17 studies were published by Chinese authors, 8 by Italian authors, and 4 by American authors. Nearly half of the SRs/meta-analyses were performed by <5 authors (n = 20, 42.6%), which was comparable between SRs only (n = 6, 46.2%) and meta-analysis (n = 14, 41.2%); nevertheless, Chinese-based studies (n = 11, 64.7%) were commonly performed by 6 to 10 authors compared with nonChinese-based studies (n = 7, 23.3%). SRs/meta-analyses were most commonly published by journals with an impact factor of no more than 5 (n = 34, 72.3%), and no difference was found between SRs only (n = 9, 69.2%) and meta-analyses (n = 25, 73.5%) or studies performed by Chinese authors (n = 12, 70.6%) or authors outside of China (n = 22, 73.3%) (Table S1 , Supplemental Digital Content, http://links.lww.com/MD/G491). The studies mainly focused on the epidemiology (n = 14, 29.8%), clinical manifestations (n = 8, 17.0%), therapeutic interventions (n = 7, 14.9%), prevention (n = 7, 14.9%), diagnosis (n = 6, 12.8%), and prognosis (n = 1, 2.1%); the detailed study aims and main conclusions are listed in Table S2 , Supplemental Digital Content, http://links.lww.com/MD/G491. The distribution of review focus was different between SRs only and meta-analysis (x 2 = 14.991, P = .015). For SRs, the study focused on the therapeutic interventions (n = 4, 30.8%) and prevention (n = 5, 38.5%); for meta-analysis, epidemiology (n = 13, 38.2%) was the most common topic. For studies published by Chinese authors, the top 2 study topics were epidemiology (n = 6, 35.3%) and clinical manifestations (n = 5, 29.4%); for studies published by authors from other countries, epidemiology (n = 8, 26.7%) the most common topic. The median number of days from the date of literature retrieval to the date that the study was first available online was comparable between SRs only and meta-analysis (x 2 = 2.54, P = .314) and most common ranged from 16 to 30 days. Nevertheless, the period from the date of literature retrieval to the date that the SR/meta-analysis performed by Chinese author was first available online was most common for more than 30 days (n = 8, 47.1%), while that conducted by nonChinese authors ranged from 16 to 30 days (n = 17, 56.7%) ( Table S1 , Supplemental Digital Content, http://links.lww.com/ MD/G491).

The number of included studies in the SR/meta-analysis was common for <10 (n = 18, 38.3%); the included studies were observational studies (n = 10, 21.3%), case series/case reports (n = 7, 14.9%), and RCTs (n = 3, 6.4%). The most common total number of participants ranged from 1001 to 5000 (n = 26, 55.3%). Several risk of bias assessment tools were used to evaluate the quality of studies included in the reviews, including the Newcastle-Ottawa Scale (NOS) ( Table 3 . 

This cross-sectional study investigated the methodological quality and reporting characteristics of published SRs and meta-analyses related to COVID-19. Overall, the study quality was low to critically low, and the rate of compliance to the PRISMA guidelines varied from 14.9% to 100%. As the first COVID-19 outbreak occurred in China, the number of studies performed by Chinese authors was relatively high. The median number of days from the date of literature retrieval to the date that the study was first available online was 21 days; due to the limited time, only 7 studies had study protocols. Various risk of bias assessment tools were used in different SRs/ meta-analyses; this was due to the fact that SRs/meta-analyses focused on several topics such as epidemiology, therapeutic interventions, prevention, diagnosis, and prognosis. The following methodological quality assessment tools were recommended, based on the study designs: [56] the Cochrane Collaboration's tool for RCTs, the Newcastle-Ottawa Scale for cohort and case-control studies, the methodological index for nonrandomized studies for nonrandomized interventional studies, the Agency for Healthcare Research and Quality methodology checklist for cross-sectional studies, the Quality Assessment of Diagnostic Accuracy Studies-2 for diagnostic accuracy test studies, the Systematic Review Centre for Laboratory animal Experimentation for animal studies, the revised version of the AMSTAR for SRs/meta-analyses, an 18-item tool for case series studies, and the Appraisal of Guidelines Research and Evaluation-II instrument for clinical practice guidelines.

In this study, none of the SRs/meta-analyses reported the sources of funding of the studies included in the review. Although this is only a noncritical weakness item and does not dramatically affect the methodological quality, it had associations with the published journal impact factor. [57] This may be partially Table 3 Methodological quality of included systematic reviews/meta-analyses assessed by AMSTAR-2. influenced by the shorter manuscript preparation time, although the rate of reporting of funding sources in meta-analyses of trials on pharmacological treatment in high-impact biomedical journals was low (7%). [58] For individual conflicts of interest, 49% and 33% of Cochrane and nonCochrane reviews, respectively, reported the type of conflict of interest for at least 1 author; for institutional conflicts of interest, the rate of reporting was lesser: 19% and 5% for Cochrane and non-Cochrane reviews, respectively. [8] With regard to the compliance to the PRISMA guidelines, the number of items related to the risk of bias within studies and the risk of bias across studies were higher for the Methods section than for the Results section; this finding indicated that some SRs/ meta-analyses did not report the risk of bias within studies and across studies in the Results section, although the methodology was reported in the Methods section. Other reporting items, like the detailed search strategy and performance of additional analysis, need to be improved. In our study, 72.3% of SRs/metaanalyses reported the funding information, which was similar to the percentage of previous studies that reported this information (64% [3] but higher than 41.4%). [59] This cross-sectional study was the first to assess the study quality of published SRs/meta-analyses related to COVID-19 regardless of the study topic. This study has several limitations. Firstly, we did not compare the study characteristics, methodological quality, and reporting characteristics of SRs/metaanalyses on COVID-19 with those of studies investigating other topics published during the same period. Nevertheless, when compared with the study performed by Page et al in 2014, [3] our study showed similar results and concluded that the performance and reporting should be improved. Secondly, due to the limited time to conduct the reviews, only 7 studies had a study protocol, which is one of the critical domains in the study quality assessment tool, AMSTAR-2; thus, the study quality was considered low if the reviews did not have a study protocol; only 3 (6.4%) studies were Cochrane reviews. In addition, none of the original studies included in the SRs/meta-analyses reported the funding information.

This study aimed to evaluate the methodological quality and reporting characteristics of SRs/meta-analyses on COVID-19. However, the conduct and reporting quality of SRs/metaanalyses on COVID-19 were poor, which should be improved. The funding information of studies included in the SRs/metaanalyses should be reported. 

Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated

Synthesizing evidence: shifting the focus from individual studies to the body of evidence

Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study

Reducing waste from incomplete or unusable reports of biomedical research

AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group

Epidemiology of COVID-19

Cerebrovascular disease is associated with an increased disease severity in patients with coronavirus disease 2019 (COVID-19): a pooled analysis of published literature

The possible of immunotherapy for COVID-19: a systematic review

Coronavirus disease 2019 (COVID-19) CT findings: a systematic review and meta-analysis

Medical masks vs N95 respirators for preventing COVID-19 in healthcare workers: a systematic review and meta-analysis of randomized trials

Novel coronavirus infection (COVID-19) in humans: a scoping review and meta-analysis

Imaging and clinical features of patients with 2019 novel coronavirus SARS-CoV-2: a systematic review and meta-analysis

Systematic rapid "living" review on rehabilitation needs due to covid-19: update to March 31st 2020

Clinical characteristics and diagnostic challenges of pediatric COVID-19: a systematic review and metaanalysis

Gastrointestinal manifestations of SARS-CoV-2 infection and virus load in fecal samples from the Hong Kong cohort and systematic review and meta-analysis

COVID-19 in cardiac arrest and infection risk to rescuers: a systematic review

Outcome of Coronavirus spectrum infections (SARS, MERS, COVID 1-19) during pregnancy: a systematic review and meta-analysis

A systematic scoping review of COVID-19 during pregnancy and childbirth

Prevalence of underlying diseases in hospitalized patients with COVID-19: a systematic review and meta-analysis

The role and response of palliative care and hospice services in epidemics and pandemics: a rapid review to inform practice during the COVID-19 pandemic

Systematic review of the efficacy and safety of antiretroviral drugs against SARS. MERS or COVID-19: initial assessment 2020

Clinical characteristics of coronavirus disease 2019 (COVID-19) in China: a systematic review and metaanalysis

Hematologic, biochemical and immune biomarker abnormalities associated with severe illness and mortality in coronavirus disease 2019 (COVID-19): a meta-analysis

Barriers and facilitators to healthcare workers' adherence with infection prevention and control (IPC) guidelines for respiratory infectious diseases: a rapid qualitative evidence synthesis

Prevalence and severity of corona virus disease 2019 (COVID-19): a systematic review and meta-analysis

Diabetes mellitus is associated with increased mortality and severity of disease in COVID-19 pneumonia -a systematic review, meta-analysis, and meta-regression

Enhancing immunity in viral infections, with special emphasis on COVID-19: a review

Diagnostic performance of CT and reverse transcriptase-polymerase chain reaction for coronavirus disease 2019: a meta-analysis

Prevalence and impact of cardiovascular metabolic diseases on COVID-19 in China

The impact of 2019 novel coronavirus on heart injury: a systematic review and meta-analysis

COVID-19 patients' clinical characteristics, discharge rate, and fatality rate of meta-analysis

Thrombocytopenia is associated with severe coronavirus disease 2019 (COVID-19) infections: a meta-analysis

Hypertension in patients with coronavirus disease 2019 (COVID-19): a pooled analysis

Conjunctivitis and COVID-19: a meta-analysis

Coronavirus disease 2019 and prevalence of chronic liver disease: a meta-analysis 2020

Quarantine alone or in combination with other public health measures to control COVID-19: a rapid review

Delivery in pregnant women infected with SARS-CoV-2: a fast review

A systematic review of COVID-19 epidemiology based on current evidence

Clinical, laboratory and imaging features of COVID-19: a systematic review and meta-analysis

Diabetic patients with COVID-19 infection are at higher risk of ICU admission and poor shortterm outcome

Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients

Cardiac injury is associated with mortality and critically ill pneumonia in COVID-19: a meta-analysis

Virological and clinical cure in COVID-19 patients treated with hydroxychloroquine: A systematic review and meta-analysis 2020

Clinical characteristics of hospitalized patients with SARS-CoV-2 infection: a single arm metaanalysis

Personal protective equipment for preventing highly infectious diseases due to exposure to contaminated body fluids in healthcare staff

Does comorbidity increase the risk of patients with COVID-19: evidence from meta-analysis

Clinical features and outcome of treatment for novel coronavirus pneumonia: a meta-analysis

Prevalence of comorbidities and its effects in patients infected with SARS-CoV-2: a systematic review and metaanalysis

The effect of corticosteroid treatment on patients with coronavirus infection: a systematic review and meta-analysis

Maternal and perinatal outcomes with COVID-19: a systematic review of 108 pregnancies

The impact of COPD and smoking history on the severity of COVID-19: a systemic review and meta-analysis

Clinical characteristics of 3,062 COVID-19 patients: a meta-analysis

CT imaging features of 4121 patients with COVID-19: a meta-analysis

The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review

Reporting of financial and non-financial conflicts of interest by authors of systematic reviews: a methodological survey

Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments

Epidemiology characteristics, reporting characteristics, and methodological quality of systematic reviews and meta-analyses on traditional Chinese medicine nursing interventions published in Chinese journals