key: cord-0758838-j8lgwivt
authors: Pottie, Kevin; Smith, Maureen; Matthews, Micayla; Santesso, Nancy; Magwood, Olivia; Kredo, Tamara; Scott, Sarah; Bayliss, Kerin; Saad, Ammar; Haridas, Rinila; Detambel, Nicole; Motilall, Ashley; Tan, Yvonne; Steinberg, Sally; Litynska, Justyna; Dietl, Bart; Ioiri, Alfonso; Reveiz, Ludovic; Welch, Vivian; Klugar, Miloslav; Mbuagbaw, Lawrence; Rojas, Maria Ximena; Florez, Ivan D.; Lotfi, Tamara; Qaseem, Amir; Mathew, Joseph L.; Akl, Elie A.; Tugwell, Peter; Schünemann, Holger J.
title: A multistakeholder development process to prioritize and translate COVID-19 health recommendations for patients, caregivers and the public. A case study of the COVID-19 Recommendation Map
date: 2022-04-30
journal: J Clin Epidemiol
DOI: 10.1016/j.jclinepi.2022.04.012
sha: 377f4cc83a1a36f0658bcffffa845f788fcbdf6e
doc_id: 758838
cord_uid: j8lgwivt

Objective To develop a digital communication tool to improve the implementation of up-to-date COVID-19 recommendations. Specifically, to improve patient, caregiver and public understanding of healthcare recommendations on prevention, diagnoses and treatment. Study Design and Setting Multi-stakeholder engagement design. In conjunction with the COVID-19 Recommendations and Gateway to Contextualization RecMap, we co-developed a stakeholder prioritization, drafting and editing process to enhance guideline communication and understanding. Results This paper presents the multi-stakeholder development process with three distinct plain language recommendation formats: formal recommendation, good practice statement and additional guidance. Our case study of COVID-19 PLRs addresses both public health interventions (e.g., vaccination, face masks) and clinical interventions (e.g., home pulse oximetry) Conclusion This paper presents a novel approach to engaging stakeholders in improving the communication and understanding of published guidelines during the COVID-19 pandemic.

The COVID-19 pandemic has led to an unprecedented number of public health guidelines. 1, 2 Television, electronic press, news websites and social media interpret these guidelines and report this as health information. 3 Patients, caregivers, and members of the public have had to process thi s i nformati on and make informed decisions about their health, often without the traditional direct support of healthcare professionals. How can I protect myself against COVID-19? How can I prevent transmission? Should I get vaccinated? Can I safely send my children to school?

The WHO, CDC, PHAC and other guideline development organizations have published recommendations to guide healthcare professionals. The contextualization and plain language translation of these recommendations could enable the public to better understand their health and healthcare options as well as create a more meaningful patient-physician relationship. In fact, research has shown that patients and the public think that accessible recommendations could educate, advise, reassure, promote self-care and prepare patients for a visit to a healthcare professional. 4 A recommendation is an actionable statement about the choice between two or more management options (sometimes including no action) in a specific population or setting that results from methods and processes that are deemed trustworthy. [5] [6] [7] 'Trustworthy' guidelines provide recommendations that are supported with rigorous systematic reviews with an explicit development process, a deliberative evidence to decision approach and declaration and handling of competing interests. 6, 8 The eCOVID19 RecMap project has identified more than 4000 COVID-19-relevant actionable statements. 5, 9 These statements include recommendations, good practice statements, additional guidance, and implementation considerations, tools and tips. 5 Most statements have targeted health professionals as the end user. Of note, new guideline development methods promote the engagement of patients, the public and other stakeholders in the guideline development and implementation processes to include public perspectives. 10 However, J o u r n a l P r e -p r o o f Confidential: Technical/Scientific and plain language definitions of guideline statements 5 the end product of the guideline development process, i.e. the actual recommendations, need to be made accessible and understandable to a range of public stakeholders.

We created a digital communication tool to be accessed in web browsers without needing to be downloaded. Successful digital initiatives need to be co-developed with 'users,' and this means partnering and engaging citizens and addresses specific 'user' contexts, culture and behaviors within the ongoing tool development process. 11 Plain language summaries are defined as easy-to-read, balanced statements that are derived from scientific research or guidelines 5 . The Campbell and Cochrane Collaborations have developed and instituted plain language summaries to effectively summarize the results of systematic reviews. The Guidelines International Network (GIN) Public Toolkit has emerged as a leading guide for preparing patient summaries. 12 Our reported r e f o r m a t t e d a n d d i g i t a l i z e d plain language recommendations aim to give the public access to recent, reliable and educational healthcare information. 13 This paper reports on enhancements to the GRADE plain language recommendation template and case studies that use a multi-stakeholder plain language development process focusing on prioritized COVID-19 recommendations.

Our plain language recommendations (PLRs) are derived from selected, up-to-date, and critically appraised published guidelines from the COVID19 Recommendations and Gateway to Contextualization RecMap. 9 We developed an administration table (see Appendix 1) for each of our selected PLR topics including the original guideline citation, the guideline organization, a RecMap link to the underlying evidence, rationale and the original AGREE II appraised published guideline. Our methods built on the Guideline International Network public guideline development approach 14 and the UK DISCERN checklist for authors of consumer health information. 15 The eCOVID project partnered with the Cochrane Consumer Network. We refined our methods using a series of 32 COVID-19 related plain language recommendations addressing the following clinical and public health topics: for exampl e, vaccinations, usage of face masks, physical distancing in long term care, breastfeeding guidance, international travel and home pulse oximetry.

We developed a multi-stakeholder engagement method for the prioritisation and plain language drafting of our COVID-19 PLRs. Stakeholders involved in the prioritisation process included undergraduate science students, members of the public, clinicians, guideline methodologists, and academicians. Our team prioritized COVID-19 topics for plain language translation using three criteria: (1) availability of recommendations that included quality appraisal on the COVID19 RecMap (2) of interest to stakeholders as judged by the COVID19 RecMap executive team, and (3) interest to citizen stakeholders as judged through multistakeholder implementation team discussion. We enhanced our GRADE plain language recommendation template following the UK DISCERN Checklist to guide the development of our plain language summaries. 15 This checklist provided a valid and reliable way to improve the quality of written consumer information about treatment choices. Section 1 covers reliability and certainty, section 2 covers the quality of the information and section 3 covers an overall rating.

We assigned sel ected gui del i ne statements to trained PLR drafters (undergraduate health science university students and other volunteers with a health science background) who completed the first draft by applying the original GRADE PLR template 16 with instructions and following the plain language training manual. As part of the plain language translation process, drafters were instructed to utilize a machine learning programmed editor called "Hemingway Editor" 17 to detect complex sentences, unnecessary adverbs, and to obtain a grade level estimate for their J o u r n a l P r e -p r o o f Confidential: Technical/Scientific and plain language definitions of guideline statements 6 writing sample. We also instructed drafters to use plain language dictionaries and glossaries, including the J o u r n a l P r e -p r o o f Confidential: Technical/Scientific and plain language definitions of guideline statements 6 Michigan Plain Language Medical Dictionary and GET IT Glossary. During the drafting of the initial Plain Language Recommendations , we determined that writing at a grade 11 level provided easy-to-read accessible PLRs that did not lose their intended meaning. A PLR checker, graduate student level, then verified the work using the same process, and resolved any discrepancies with the drafter.

A clinical expert then verified the translation for nuance, balance, and medical accuracy. Finally, as part of our feedback loop, a trained public member (citizen editor) verified the translation for succinctness, readability and understandability for a public audience.

This process was facilitated using a centralized PLR drafting platform (Google Sheet at docs.google.com) that could be accessed by all drafters, reviewers and editors. Our final step, verification and linking the digital PLR to the RecMap, involved using a trained and standardized staff person and supporting medical editor. These team members checked the draft for version fidelity relative to original recommendations, content and plain language details before uploading the content into a customized GRADEpro module (COVID19 Recommendations and Gateway to Contextualization Platform, Evidence Prime). The PLR was then added to the eCOVID19 RecMap alongside the original recommendation statement, guideline details, AGREE II score, and supplementary information including evidence to decision and summary of findings tables. See Figure 

Our team (14 undergraduate students and volunteers, 6 clinicians, 4 guideline developers 4 members of the public, and 3 research assistants) completed 32 digital COVID-19 plain language recommendations for patients, caregivers and the public [(See Figure 2 and RecMap (https://covid19.recmap.org/recommendations?plainLanguageSummaryPresent=T)]. Using the UK DISCERN Checklist, we report on the reliability, certainty of the original recommendation, and the fidelity of the patient information for our case series GRADE PLRs. The original GRADE PLR is described in our companion paper . 16 During the ongoing development process, we received a CIHR grant to optimize and conduct 3 randomized controlled trials on our PLR tool. The team worked with the Cochrane Consumer Group, youth partners and advisors from Sick Kids and the pediatric parent advisory groups affiliated with the ARCHE and ECHO research groups to optimize the digital layout of the tool. 

Our multi-stakeholder process developed and assessed the GRADE plain language recommendations summaries. Within the RecMap we included a description of the aim, development process and relevance of our plain language recommendations: "Plain Language Recommendations (PLR) are easy-toread summaries of published and quality-checked recommendations. PLR are balanced statements that include an explanation of the recommendation, what it means for patients and the public, and the guideline source. These PLR are derived from leading guideline development organizations and include a link to the underlying evidence and their rationale."

Our multi-stakeholder development process identified a few language improvements (i.e. Talk with vs Speak to) for the original PLR template. Working originally with two citizen partners (MS, SS), we recognized the importance of adding the source of the recommendation and the publication date of the original recommendation in the GRADE PLR. For example, WHO Pfizer BioNTech recommendation published 2021. We linked each PLR to the parent eCOVID RecMap which provides background details on our guideline producing organizations (e.g., WHO, CDC, PHAC), a description of the evidence-based nature of the project and an overall research disclaimer for the eCOVID project. 9

Our multi-stakeholder team selected topics that included (formal) recommendations (for example, WHO Pfizer BioNTech vaccine and WHO Home Pulse Oximetry). But our team also selected two other types of actionable health statements: good practice statements (for example, PHAC mRNA vaccines) and additional guidance statements (for example, CDC Breastfeeding). These latter two 'types' of actionable statements occurred where guideline panels identified an important issue, described the population and intervention options, but the published guideline lacked information needed for formal ratings of certainty of evidence. J o u r n a l P r e -p r o o f Confidential: Technical/Scientific and plain language definitions of guideline statements 9 recommendation if they are not self-evident. These statements are the results of a formal deliberation process, and contain an explicit and direct link to the bodies of evidence resulting from a systematic literature search and appraisal process. 

We also distinguished the quality or certainty of the information. Each PLR began with a clear statement related to the recommendation 'type'; for example, Recommendation, or GPS of Additional Guidance and generic statement (see Table 2 ). Within each PLR, we also described the intervention and the benefits, harms or risks of each intervention. Although suggested by the UK DISCERN Checklist, we were not able to explicitly describe the impact of the intervention on overall quality of life because this information was not available in the original guideline publications. We clearly described intervention choices when available. Our final section focused on "What does it mean for you?" This section included explicit suggestions on what you may wish to talk with your health care professional about. Some of the guidelines included detailed discussion points for shared decision making; for example, how to assess home pulse oximetry readings. Other guidelines did not always provide explicit discussion point information; for example, WHO Moderna vaccine. 

Additional guidance are 'informal' actionable statements that do not fulfill the taxonomy criteria for formal recommendations or Good Practice Statements (GPS) in terms of methods and/or process.

They often lack a clear link to all the relevant evidence, or a systematic review, or a Health Technology Assessment (HTA) process. They also lack a clear description of the judicial process of moving from the evidence to the recommendation, including a lack of the process of how consensus was reached. These 'informal' actionable statements should be used with caution. A "drop down" interactive summary of findings table was added to the PLR template for formal recommendations at a later stage allowing users to have the opportunity to explore the benefits and harms using numbers. For four PLRs, our team created simple plain language statements to summarize each outcome within the table. We determined that it was important to communicate both the magnitude of the effect and the certainty of the evidence. This aligned with GRADE guideline 26 which provides specific terminology for communicating both elements in GRADE tables. 19 For example, to describe an outcome with a large effect size and high certainty of evidence, terms such as "large decrease" and "very certain" were used respectively. Emoticons and plus signs were also incorporated to enhance the understanding of the certainty in the evidence. Our CDC Pfizer BioNTech Vaccine example can be found using this web link: https://covid.gradepro.org/presentations/#/premium/premium_presentation:p_l_f_cruciani_deplazio_it_ 0_a2c6674c-5bd0-459c-b230-be5552544296_cd92e290-2465-4294-b017-0bcb96ed5a88 Table 2 Plain Language Generic Statement relating to the certainty of the actionable statement J o u r n a l P r e -p r o o f Discussion Plain language recommendations have been high demand during the COVID-19 pandemic. 1 This paper reports on a multi-stakeholder priority setting and translation process for digital COVID-19 plain language recommendations and the creation GRADE PLR templates for 'good practice statements' and 'additional guidance'. Our multi-stakeholder team recognized that public health and practice recommendations can create uncertainty and a strain on citizens. 20 Therefore, our plain language recommendations attempted to balance between education and confidence (certainty of information) for patients 21 ,caregivers and the public. As part of our research, we are now optimizing the digital PLR tool in collaboration with consumer groups. We will conduct 3 randomized controlled trials to test the effectiveness of the new digital tool to improve adolescent, parent and public understanding of COVID-19 recommendations. We have also begun translating the digital PLR into several languages, and research on this element is ongoing.

Our multi-stakeholder development process identified a few language improvements (i.e. 'Talk with' vs 'Speak to') for the original PLR template. DISCERN 21 and our citizens suggested adding in the publication year with the recommendation to enhance reliability of the information. Conceptually, the most important innovation is the development of generic statements that distinguished between three recommendation types: one for 'formal recommendations' that was in the original PLR 15 one for 'good practice statements' and one for 'additional guidance'. In the drafting process, we noted that the original guidelines often provided only limited, if any, information on what the recommendation would mean for a patient or the public. DISCERN 21 identifies this as a key element in consumer health information and so we thus identify this as an important area for improvement in guideline development standards.

Our administration table provided links to the most recent guideline and distinguished the recommendation 'type' for student drafters. To ensure accurate depictions of the existing guidelines, we found an annotated template was necessary to build consistency in our sections, provide direct links to extracted data and original publications, and to prevent the inclusion of new scientific evidence. An The eCOVID RecMap has identified over 4000 actionable statements for providers, policymakers, and the public. This open access RecMap classified the 'type' of recommendation 5 and provided quality AGREE II appraisals 8 . Our multi-stakeholder team used the map to identify priority recommendations for patients and the public, while recognizing that these plain language recs may also be useful for providers and policymakers. We initially focused our attention on 'formal' recommendations, however, citizens on our team identified topics that involved other recommendation 'types': good practice statements and additional guidance. In order to include these novel actionable statements, and distinguish the certainty of information, we created two other PLR formats with appropriate interpretive generic plain language translations.

During our development process, we recognized that our online PLRs benefited from staying closely linked to the COVID19 Recommendations and Gateway Contextualization RecMap. The online link provided the public with our PLR definition and process description, information in evidence to decision extractions, Appraisal of Guidelines for Research and Evaluation II (AGREE II) appraisals, plain language glossaries, and background details on the eCOVID19 RecMap. We observed that publishing the PLR as stand-alone products would require additional work to ensure there is a clear reference to the original source guideline, version and publication dates, and information on how the PLRs were developed as per the DISCERN reporting guidelines. 21 We encourage guidelines developers to include GRADE plain language recommendations (PLR) for their guidelines, including more information on "What does it mean for you?" We will also seek additional support for multilingual publication of our ongoing series of COVID-19 PLRs.

The strengths of our development process included having the tool link directly to a digital guideline appraisal platform with sustained engagement of various stakeholders. Our digital PLR were amendable to timely updates and optimization; for example, our consumer groups have optimized tool design. The limitations of our approach included the variable amount of training needed to support public members and citizens and the importance of including staff checkers to verify the consistency of the health information. Another consumer challenge that presents in the final component of our digital tool, is the high degree of health and digital literacy required to navigate and interpret the interactive summary of findings table data.

Our series of multi-stakeholder translated COVID-19 PLRs demonstrate easy-to-read summaries of high quality published guidelines. Priority selection criteria and grounding the PLR in original guidelines set the stage for GRADE evidence to decision concepts. These concepts include judging the benefits and harms of an intervention, certainty of the evidence, and values and preferences' relevance for the public. 22 Working directly with multi-stakeholders including students and members of the public may improve access along with our use of plain language medical glossaries. Using an annotated PLR template and an interdisciplinary team contributed to the consistency of our initial series of 32 COVID-19 PLRs.

The multi-stakeholder development has produced a series a digital PLR tool for the public. Our team has received CIHR funding to optimize and conduct 3 clinical trials with adolescents, J o u r n a l P r e -p r o o f Confidential: Technical/Scientific and plain language definitions of guideline statements 13 parents and the general public. These trials will evaluate the effectiveness of the digital tool to improve understanding of COVID-19 recommendations compared to a generic summary of the technical guidelines. Subsequent research will be needed to evaluate if the tool can improve patient and public behavior and important outcomes. Finally, research on the impact of translating the tool into other languages is needed.

We used and enhanced the GRADE PLR template, and developed a multi-stakeholder process to create digi tal plain language COVID-19 recommendations that were reproducible and transferable to other recommendations. Plain language, easy-to-read summaries have long played an important role in presenting systematic review results to the public. Here we introduce our digital pl ai n language COVID-19 recommendation tool poised to communicate COVID-19 guidelines to patients, caregivers, and the public. As guideline developers formalize their development processes, we anticipate more formal and good practice statements. Furthermore, we believe that our plain language translations could also enable rapid and consistent communication to providers, policymakers and other stakeholders.

Canada, 2. a) CT Lamont Centre for Primary Care

Pandemic publishing poses a new COVID-19 challenge

Infodemic" of journal publication associated with the novel coronavirus disease

COVID-19-related misinformation on social media: a systematic review

Patient and public attitudes to and awareness of clinical practice guidelines: a systematic review with thematic and narrative syntheses. BMC health services research

Recommendation mapping of the World Health Organization's guidelines on tuberculosis: A new approach to digitizing and presenting recommendations

Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise

Guidelines International Network: toward international standards for clinical practice guidelines

AGREE II: advancing guideline development, reporting and evaluation in health care

Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation. Systematic reviews

Agile research to complement agile development: A proposal for an mHealth research lifestyle

GIN Public Toolkit: Patient and Public Involvement in Guidelines

Plain language summaries in the British Journal of Dermatology: connecting with patients

SUPPORT Tools for evidence-informed health Policymaking (STP) 15: Engaging the public in evidence-informed policymaking. Health Research Policy and Systems

DISCERN: an instrument for judging the quality of written consumer health information on treatment choices

Focus groups and interviews with patients and the public led to the development of a template for a GRADE plain language recommendation

GRADE: an emerging consensus on rating quality of evidence and strength of recommendations

GRADE guidelines 26: informative statements to communicate the findings of systematic reviews of interventions