key: cord-0829080-nhu91miw
authors: Wang, Zijun; Liu, Hui; Li, Yang; Luo, Xufei; Yang, Nan; Lv, Meng; Zhou, Qi; Li, Qinyuan; Wang, Ling; Zhao, Junxian; Liu, Yunlan; Su, Renfeng; Wu, Shouyuan; Liu, Xiao; Zhao, Siya; Shi, Qianling; Estill, Janne; Liu, Xingrong; Zhang, Wenhong; Chen, Yaolong
title: COVID-19 vaccine guidelines was numerous in quantity but many lack transparent reporting of methodological practices
date: 2021-12-14
journal: J Clin Epidemiol
DOI: 10.1016/j.jclinepi.2021.12.015
sha: cce623167efd256e4e9ddb18ffa60f0711d8fe14
doc_id: 829080
cord_uid: nhu91miw

AIM: To describe the current status of COVID-19 vaccine guidelines. STUDY DESIGN AND SETTING: We searched databases, Google and guideline platforms to retrieve COVID-19 vaccine guidelines published between January 1, 2020 and July 8, 2021. We worked in pairs to identify the eligible guidelines and extract data of whether the methodology, funding, and conflict of interests were assessed/reported, and so on. Results were presented descriptively. RESULTS: A total of 106 COVID-19 vaccine guidelines were included. In the first half of 2021, on average 15 guidelines were published every month. Fifty (47.2%) guidelines addressed the vaccination of people with specific medical conditions, and 18 (17.0%) guidelines focused on adverse effects after vaccination. Only 28 (26.4%) guidelines reported the methodology they used. Four (3.8%) of guidelines assessed both the quality of evidence and strength of recommendations; 42 (39.6%) and 65 (61.3%) guidelines reported their funding sources and conflict of interest, respectively. Most guidelines were published in English (n=92, 86.8%). CONCLUSION: A high number of guidelines on COVID-19 vaccines have been published in the recent months, but most of them lack clear and transparent reporting of methodology, funding, and conflicts of interest. Rigorous methodological and reporting quality evaluation of these guidelines is needed.

of methodology, funding, and conflicts of interest. Rigorous methodological and reporting quality evaluation of these guidelines is needed.

COVID-19, vaccine, guideline, methodology, transparency, conflict of interest 1 Background

Since the outbreak of COVID-19 at the end of 2019, over 200 million cumulative confirmed cases and 4.6 million deaths have been reported in nearly 200 countries and territories worldwide.

[1] Vaccination is the safest way to control the spread of the epidemic in the absence of effective antiviral drugs. [2] [3] [4] As of August 22, 2021, 22 vaccines have been approved, among which seven vaccines have been approved for emergency use by the World Health Organization (WHO). [5, 6] According to Our World in Data, 41.8% of the world's population has received at least one dose of a COVID-19 vaccine, and 29.7% have been fully vaccinated. [3] However, this number is far below the level of vaccination needed to produce herd immunity. Vaccination campaigns are hampered by vaccine accessibility, efficacy and hesitancy, for example, there are no approved vaccines for children, and difficulties in obtaining vaccines in low-income countries. [7] To address this problem, practice guidelines have been developed by many organizations and institutions aiming to translate the research findings into recommendations for practice and to provide a guidance for the appropriate use of COVID-19 vaccination. [8] [9] [10] [11] [12] [13] [14] [15] [16] There are already many different types of guidelines at present, some of the guidelines are brief, others lengthy, however, the overall picture of the vaccine guidelines is unclear. To our knowledge so far, no study has comprehensively summarized the characteristics of these guidelines. Therefore, we conducted a systematic survey to comprehensively analyze the methodological and reporting characteristics of guidelines on COVID-19 vaccines to help stakeholders understand the current situation and inform the development of future guidelines.

We systematically searched MEDLINE (via PubMed), Embase, Web of Science, WHO COVID-19 database, Wanfang Data, China Biomedical Literature database (CBM) and China National Knowledge Infrastructure (CNKI) to identify guidelines on COVID-19 vaccines published between January 1, 2020 and July 8, 2021. We used the search terms "vaccine", "COVID-19", "guideline" and their derivatives. The official websites (see Table 1 ) and Google were also searched to identify any guidelines missed in the literature database search. The details of the search strategy are presented in Supplementary File 1.

We included records that fulfilled the following three criteria: (1) the article was an evidence-based or consensus-based practice guideline with one of the words "guideline", "recommendation", "statement", "consensus" and "guidance" in the title;

(2) on the article addressed COVID-19 vaccines or the administration and management of complications of COVID-19 vaccination; and (3) the record was the latest available version. There was no language restriction. The following types of articles were excluded: (1) the number of authors was less than three (when the number of authors is less than three, differences of opinions cannot be reached effectively) and the guideline was not endorsed or supported by any academic or public health organization (e.g. association, society, international organizations or government agencies); (2) guidelines that addressed COVID-19 vaccination amongst other topics (for example general COVID guidelines with some items related to vaccines); (3) we failed to access the full text of the article; (4) the article was a summary of recommendations from different guidelines; and (5) duplicates.

Twelve investigators (Zijun Wang, Qianling Shi, Hui Liu, Junxian Zhao, Xufei Luo, Meng Lv, Yunlan Liu, Shouyuan Wu, Xiao Liu, Renfeng Su, Nanyang, and Ling Wang) were divided into six groups of two. The records were divided across the six groups, within which both investigators screened the literature independently. The formal selection process consisted of two stages: (1) screening all titles and abstracts to determine whether the record is potentially relevant to the research question; and (2) obtaining and screening full texts of potentially eligible articles to decide about the inclusion. Disagreements were resolved by discussion, or by consulting a third investigator.

Fourteen investigators (Zijun Wang, Siya Zhao, Hui Liu, Junxian Zhao, Xiao Liu, Ling Wang, Shouyuan Wu, Renfeng Su, Nanyang, Yunlan Liu, Xufei Luo, Meng Lv, Qi Zhou, and Qinyuan Li) were divided into seven groups of two. Each group was given a part of the records. All investigators in the group independently extracted the following data using a pre-defined extraction table: (1) basic information: publication year and month (first online publication), country/region of main developer institution, leading organization/institution, number of authors, and target population; (2) information related to guideline development: whether the methodology, strength of recommendations, quality of evidence, funding, conflict of interests and research gap were assessed/reported; and (3) information related to the publication: publication format (journal, website, or both), whether the guidelines were accessible free of charge or not, impact factor of the journal (Journal Citation Reports 2020) and language. Disagreements were solved by consensus or consultation with a third reviewer. In order to ensure the accuracy and consistency of the extracted data, we conducted two rounds of pilot tests before the formal extraction and calculate the intraclass correlation coefficient (ICC) value of all investigators respectively. (ICC value 1 = 0.934; ICC value 2= 0.984).

We used Microsoft Excel 2019 for data collection. Dichotomous and categorical variables were presented as absolute numbers and percentages, and continuous variables as means with standard deviation or medians with interquartile range [IQR].

We identified initially 2604 records. After de-duplicating and screening of titles and abstracts, we read the full texts of 179 potentially eligible records. Based on inclusion and exclusion criteria, 106 COVID-19 vaccine guidelines were finally included in our study ( Figure 1 ).

The 106 included COVID-19 vaccine guidelines were published between May 2020 and July 2021, of which 95 (89.6%) in 2021 ( Figure 2 ). On average 14.8±1.6 guidelines were thus published monthly between January 2021 and June 2021. The country or region of main developer institution that issued the highest number of guidelines was Canada (n=22, 20.8%), followed by international (n=18, 17.0%) and the United States (n=13, 12.3%) (see Supplementary File 2). About half (n=55, 51.9%) of the guidelines were developed by societies or associations; the highest number of guidelines developed by a single society was four (Advisory Committee on Immunization Practices). International organizations led 18 (17.0%) guidelines, of which seven by the WHO; and national organizations issued 13 (12.3%), nine of which were from the Public Health Agency of Canada National Advisory Committee on Immunization (NACI). Almost half (n=50, 47.2%) of the guidelines addressed people with specific medical conditions; the most commonly addressed condition was rheumatic and/or musculoskeletal diseases (n=8; Figure 3 ). Eighteen (17.0%) guidelines targeted adverse effects after vaccination, half of which addressed COVID-19 vaccine associated anaphylaxis and allergic reactions (n=9, Figure 4 ). Eighty (75.5%) guidelines reported their authors, ranging from one author to 62 authors across the studies (median 10.5, IQR [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] . There were 28 different terms in the titles that were used to identify the article as a guideline, of which the most common were "Recommendations" (n=17, 25.5%), "Position Statement" (n=16, 15.1%) and "Statement" (n= 13, 12.3%; see Supplementary File 2, Table 2 ). (Table 2 ).

Our study showed that more than a hundred COVID-19 vaccine guidelines were developed by a total of 74 institutions in 33 national and international organizations so far. The number is close to the number of clinical practice guidelines of COVID-19. [16] The majority of these guidelines were published in peer-reviewed journals in English by developers from high-income countries such as Canada and the United States. There is a lack of guidelines for COVID-19 vaccines from low-and middle-income countries and published in languages other than English and a lack of recommendations for these countries. This may be related to the limited ability to develop national guidelines and limited access to COVID-19 vaccine candidates. What's more, COVID-19 will not end anytime soon due to virus variants. The research and application of vaccination is still in high gear and still in an early stage, ditto for the guidelines.

Our results indicated that most guidelines focused on patients with specific medical conditions who may be at a high risk of severe vaccine-related adverse reactions due to their physiological state or to external interventions, and thus a more vaccine hesitancy population. In particular, people with conditions such as rheumatic and/or musculoskeletal diseases, cancer, and dermatological disorders, who are taking immunomodulatory or immunosuppressive drugs, are at higher risk of getting infection or worsening of the clinical course of COVID-19. [19, 20] The prevention and management of adverse effects maybe an issue in all vaccines that guidelines for this topic are necessary, especially in COVID-19 vaccination. This may be due to the successive occurrence of allergic reactions in vaccine recipients, which, although not very frequent, have been of wide concern to researchers and require recommendations to explain and guide the actions of clinicians and healthcare staff. [21] [22] [23] [24] However, there was insufficient clinical evidence in the guidelines to support recommendations for specific populations and adverse reactions to vaccines, e.g., evidence on COVID-19 vaccination in cancer patients was lacking, and the current evidence on allergies against mRNA vaccines still needs to be updated. [25, 26] However, a large number of clinical trials for specific populations and adverse reactions to vaccines are still ongoing and if these guidelines are updated promptly, the quality of evidence and strength of recommendations will continue to improve. [5] Our study also analyzed research gaps in vaccine guidelines, which provided clues and information for future research priorities.

While there is an urgent need for guidelines that are able to provide stakeholders with recommendations on COVID-19 vaccination and administration, a rigorous and reliable guideline development process is essential to avoid potential harm to patients and the healthcare system due to misinformation and misunderstanding. [27] We found that most guidelines did not report the methodology in detail and information on conflict of interests and funding disclosures was often lacking. The lack of clear recommendations, clear citations of evidence to support the recommendations, and evaluation of the strength of recommendations and evidence quality were the main methodological problems. This finding is similar to that of previous studies on COVID-19 guidelines. [27] [28] [29] According to the Institute of Medicine (IOM), the development processes and funding of a clinical practice guideline (CPG) should be transparent, i.e. described in detail and publicly accessible. [30] The conflict of interests and the funding received throughout the guideline development are both important concerns by stakeholders that will directly affect each step of the guideline development. [31] However, less than half of the COVID-19 vaccine guidelines reported the conflicts of interest and funding. Guidelines that are not evidence-based and have potential conflict of interests do not provide reliable guidance to healthcare professionals and patients and may undermine public trust in the vaccine. [32] Based on our results, we suggest that: 1) guideline developers should strictly follow the guideline handbook especially in disclosing conflict of interests, for vaccine guidelines; [31] also, vaccine guidelines should be published in multiple languages in open access journals and websites whenever possible to facilitate their dissemination; 2) guideline assessors should conduct methodological and reporting quality assessments of COVID-19 vaccine guidelines using tools such as the Appraisal of Guidelines for Research & Evaluation Instrument II (AGREE II) and Reporting Items for Practice Guidelines in Healthcare (RIGHT), in order to gain a more comprehensive understanding of these guidelines; [33, 34] 3) policymakers and funders should further increase their support on the areas of guideline now focusing on, like vaccine-related adverse reactions and research gaps, to provide higher quality clinical evidence for the development of future guidelines and updates.

To our knowledge, this is the first analysis that provides a full summary and presentation of the current overview of COVID-19 vaccine guidelines, identifies problems with COVID-19 vaccine guidelines, and proposes recommendations for future guidelines. The main limitation of this study was that we only included a limited number of local and national databases and websites. However, we expect that the number of such guidelines is small enough to not essentially affect our main conclusions.

Although a number of COVID-19 vaccine guidelines have been developed in the past year, most do not follow established methodological requirements. These limitations make it challenging for clinicians and other users to select and appropriately utilize the guidelines. To promote the quality of vaccine guidelines, a more critical evaluation and systematic analysis of these guidelines are needed. English and Bulgarian 1 (0.9) English, French, Arabic and Russian 1 (0.9) English, French, Spanish, Italian, Turkish and Arabic 1 (0.9) English and Spanish 1 (0.9) English, Chinese and Arabic 1 (0.9) §Not applicable: Guidelines without clear recommendations $General journal means a general, multidisciplinary journal; Specialty journal means a specialized, discipline-specific journal #Not applicable: Guidelines without abstracts *Others: respiratory health care professionals (1), staff of medical institutions (1) and not clear (1). †SCI: Science Citation Index.

Lack of Effectiveness of Repurposed Drugs for COVID-19 Treatment

Our World in Data. Coronavirus (COVID-19) Vaccinations

Benefits of Getting a COVID-19 Vaccine

WORLD HEALTH ORGANIZATION (WHO). 7 Vaccines Approved for Use by WHO

With Great Hopes Come Great Expectations: Access and Adoption Issues Associated With COVID-19 Vaccines

World Health Organization. Interim recommendations for use of the inactivated COVID-19 vaccine, CoronaVac, developed by Sinovac

World Health Organization. Interim recommendations for use of the inactivated COVID-19 vaccine BIBP developed by China National Biotec Group (CNBG)

World Health Organization. Interim recommendations for use of the ChAdOx1-S [recombinant] vaccine against COVID-19 (AstraZeneca COVID-19 vaccine AZD1222

National Health Commission Disease Control and Prevention Bureau. Guidelines of vaccination for COVD-19 vaccines in China

EAACI statement on the diagnosis, management and prevention of severe allergic reactions to COVID-19 vaccines

An Advisory Committee Statement (ACS) National Advisory Committee on Immunization (NACI) Recommendations on the use of COVID-19 Vaccines

Analysis of COVID-19 Guideline Quality and Change of Recommendations: A Systematic Review

OCEBM Levels of Evidence Working Group 'The Oxford

GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables

American College of Rheumatology Guidance for COVID-19 Vaccination in Patients With Rheumatic and Musculoskeletal Diseases: Version 1

Disease-modifying therapies and SARS-CoV-2 vaccination in multiple sclerosis: an expert consensus

UK probes whether COVID-19 vaccine caused allergic reactions

Reports of anaphylaxis after receipt of mRNA COVID-19 vaccines in the US

Acute allergic reactions to mRNA COVID-19 vaccines

COVID-19 Vaccination for Cancer Patients: Progress and Preliminary Recommendations

The Risk of Allergic Reaction to SARS-CoV-2 Vaccines and Recommended Evaluation and Management: A Systematic Review, Meta-Analysis, GRADE Assessment, and International Consensus Approach

The methodological quality is insufficient in clinical practice guidelines in the context of COVID-19: systematic review

Scope, quality, and inclusivity of clinical guidelines produced early in the covid-19 pandemic: rapid review

A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern

Clinical practice guidelines we can trust

In guidelines we cannot trust

Why we can't trust clinical guidelines

AGREE II: advancing guideline development, reporting and evaluation in health care

A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement