key: cord-257707-brrrg4fr
authors: Jayasighe, Ravindri; Ranasinghe, Sonali; Jayarajah, Umesh; Seneviratne, Sanjeewa
title: Quality of online information for the general public on COVID-19
date: 2020-08-07
journal: Patient Educ Couns
DOI: 10.1016/j.pec.2020.08.001
sha: 
doc_id: 257707
cord_uid: brrrg4fr

OBJECTIVES: To analyse the quality of information included in websites aimed at the public on COVID-19. METHODS: Yahoo!, Google and Bing search engines were browsed using selected keywords on COVID-19. The first 100 websites from each search engine for each keyword were evaluated. Validated tools were used to assess readability [Flesch Reading Ease Score (FRES)], usability and reliability (LIDA tool) and quality (DISCERN instrument). Non-parametric tests were used for statistical analyses. RESULTS: Eighty-four eligible sites were analysed. The median FRES score was 54.2 (range:23.2-73.5). The median LIDA usability and reliability scores were 46 (range:18-54) and 37(range:14-51), respectively. A low (<50%) overall LIDA score was recorded for 30.9% (n = 26) of the websites. The median DISCERN score was 49.5 (range: 21-77). The DISCERN score of ≤50% was found in 45 (53.6%) websites. The DISCERN score was significantly associated with LIDA usability and reliability scores (p < 0.001) and the FRES score (p = 0.024). CONCLUSION: The majority of websites on COVID-19 for the public had moderate to low scores with regards to readability, usability, reliability and quality. PRACTICE IMPLICATIONS: Prompt strategies should be implemented to standardize online health information on COVID-19 during this pandemic to ensure the general public has access to good quality reliable information.

The coronavirus COVID-19 pandemic has become the greatest global health crisis of the 21st century [1] .

During this pandemic, the demand for information on COVID-19 has skyrocketed. Information such as latest news updates on the pandemic, its symptoms, prevention and mechanism of transmission are highly sought by the public [2] . On the other hand, free access to information, especially through social media, which is accessed by the majority [3] has led to an increase in misinformation and panic associated with COVID-19 [4] . Although, high quality health information is known to be related to lower stress levels and better psychological health [5] , previous studies have shown that online information on many medical disorders to be of substandard quality [6, 7] .

A previous study done on websites related to COVID-19 has reported substandard quality information that could potentially mislead the public [8] . However, this study has used a limited search strategy and had not assessed some important areas including usability and reliability of the information. Therefore, this topic remains a knowledge gap in COVID-19 research [9] . Therefore, we conducted this study to analyse the current COVID-19 websites targeting the general public in terms of quality, usability, readability, and reliability using a wide search strategy and validated instruments.

Yahoo!, Google and Bing, were searched using the keywords "novel coronavirus","SARS-CoV-2", "severe acute respiratory syndrome coronavirus-2", "COVID-19" and "coronavirus". The search was performed during J o u r n a l P r e -p r o o f the first week of May 2020. The details of the search strategy and the piloting process are provided in the supplementary material (File S1) [10] .

Two independent investigators with previous experience of conducting similar studies assessed the selected websites [11, 12] . Prior to the assessment, a pilot run was conducted to ensure uniformity and accuracy. The information on symptoms, investigations, public health measures, and available treatment modalities were collected. The accuracy of the content was assessed using the national institute for health and care excellence (NICE) guidelines on COVID-19 [13] . A rating was given as all or none based on congruence with the guidelines.

Validated instruments were used to assess the quality of websites. Readability was assessed using the Flesch Reading Ease Score (FRES) [14] . The LIDA Instrument (2007)-Version:1.2 was used to analyse the content and the design of the websites using the usability and reliability domains [15, 16] . The quality of the content was assessed using the DISCERN questionnaire which has 16 questions in two separate groups [17] . The detailed assessment criteria and the scoring system is included in the supplementary material (File S1).

A website was classified as governmental if it was maintained by the country's public health authority. If managed by private institutions, non-governmental organizations, or voluntary institutions independent from the government, they were considered as non-governmental. The online health-related websites are standardized in terms of their credibility and reliability by online certification sites. We chose the Health on the Net code of conduct (HON-code) which is the oldest and widely used out of the quality evaluation tools available [18] .

Data analysis was performed using SPSS (Version-20) software and the associations were determined with nonparametric tests. A p-value of <0.05 was considered statistically significant.

Of the 1500 retrieved websites, 1416 were excluded and the remaining 84 websites were included in the analysis. The characteristics of the websites are mentioned in Table 1 .

Half (50%) were governmental websites and only 17.9% (n=15) were HON accredited websites. The median FRES was 54.1 (range:23.2-73.5, 10 th -12 th grade readability level) which is classified as fairly difficult to read.

Only three websites (3.6%) had a readability score of above 70 (equivalent to 7 th grade) which is the recommended standard.

The overall median LIDA score was 84 (range: 32-105), while the median LIDA usability and reliability scores were 46 (range: 18-54) and 37 (range:14-51), respectively. The median DISCERN score was 49.5 (range: 21-77) which classifies websites as being of "fair quality". (Excellent=63-80; Good=51-62; Fair=39-50; Poor=27-38; Very poor=16-26). However, the top 10 websites (Table A 2) were of excellent quality.

Significant correlations were observed between the DISCERN score and the overall LIDA score as well as LIDA usability and reliability scores ( Pertaining to the currency of information, only 34 (40.5%) publishers stated the date of the publication. Most websites (n=51, 60.7%) did not declare the sources of evidence. This was further established by the "low" median reliability score of 37. Nevertheless, the authors have included a disclosure statement in most (n=51, 60.7%) websites. More than half of the websites failed to discuss the treatment options available (n=46, 54.7%), benefits or risks (n =54, 64.2%), and effects of no treatment (n =51, 60.7%). Furthermore, potential complications and prognosis were stated only in 28 (33.3%) and 37 (44%) websites, respectively.

This study has shown that still most of the websites on online health information on COVID-19 are of suboptimal quality except for a few credible sources of good-quality health information. Nevertheless, the websites ranked among the top 10 according to the DISCERN score (Table A. 2) had high scores indicating the potential for publishing credible high-quality information online which would benefit the public.

Misinformation is a major concern during this pandemic as people fail to spend adequate time to critically analyse the online information. This, in turn, causes panic which ranges from hoarding medical supplies to panic shopping and using drugs without prescription with negative social and medical consequences [19] . Therefore, J o u r n a l P r e -p r o o f measures implemented to ensure the quality and accuracy of online information by the responsible authorities may help negate these adverse consequences.

Stating the methods of content production, with names of the contributing authors may help increase the credibility of online health information while displaying the date of the publication provides an idea of the currency of the information. Absence of such information in over half of the websites was a major drawback, especially for COVID-19 where new information is generated almost daily. Health authorities should therefore ensure that the patient information websites provide the above information and certify websites based on such details so that the public can get information from trusted sources [20] .

Most users of the worldwide web only have an average level of education and reading skills [21] . Guidance from the National Institute of Health (NIH) had shown that the readability should be below the level of seventh grade for the lay public to adequately understand the content [21] . However, the median readability level was found to be equivalent to 10 th -12 th grade readability. Such complexities with the readability of information may increase the risk of misunderstandings or misinterpretation. Using short sentences in writing, using the active voice, using 12-point or larger font size, using illustrations and non-textual media as appropriate and accompanying explanations with examples would be helpful to overcome this problem [22] .

So far only a limited number of studies have been done to assess the quality of health information websites related to COVID-19. The study by Cuan-Baltazar et al prior to February 2020, reported poor quality information with approximately 70% of included websites with low DISCERN scores [8] . Our study done three months later shows similar results with only a minimal improvement in the quality of information. Furthermore, the Cuan-Baltazar study had several limitations which includes the limited search strategy and non-inclusion of key quality parameters including readability. Furthermore, 61 out of the 110 (55.4%) sites they had included were online news sites that are not considered as patient information websites. In that study, the HON-code seal was present only in 1.8% (n=2) websites, whereas in our study, 17.9% of the sites were HON-code certified.

There were several limitations in this study. Although most popular search engines were used in this study under default settings, they may produce variable results depending on many factors including geographical location and popularity of websites at a given point of time. The algorithms unique to those search engines are subjected J o u r n a l P r e -p r o o f to constant change, and therefore the exact results of our study may not be reproducible. However, we believe the general patterns observed in our study are valid.

This study has shown the quality, readability, usability, and reliability of the information on COVID-19 on majority of websites providing health information to the general public are to be of substandard quality.

To improve the credibility of the content, the websites should state methods of content production and display the date of the publication to give an idea about the currency of the information. To improve the readability of the content, the websites should incorporate more non-textual media, write in short sentences, using the activevoice and use larger font sizes. The patient information websites should display scores of reliability, quality, and readability as a guidance for its users. Furthermore, it is vital for medical regulatory authorities and the government to impose regulations to ensure quality and to prevent the spread of misinformation. 

Availability of data and materials: On reasonable request from the corresponding author, the data used in the above study can be made available.

Ethics approval and consent to participate: Unnecessary in this type of study.

Informed consent and patient details: Not applicable in this type of study. Competing interests -All authors declare that there are no competing interests

Text-only 21 25% 

Novel Coronavirus Infection (COVID-19) in Humans: A Scoping Review and Meta-Analysis

Demand for health information on COVID-19 among Vietnamese

A longitudinal study on the mental health of general population during the COVID-19 epidemic in China

Mental health strategies to combat the psychological impact of COVID-19 beyond paranoia and panic

Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China

Quality and scientific accuracy of patient-oriented information on the internet on minimally invasive surgery for colorectal cancer

Quality of patient-oriented web-based information on oesophageal cancer

Misinformation of COVID-19 on the Internet: Infodemiology study

COVID-19) Pandemic: A Global Analysis of Literature

How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews

Assessment of the quality of patient-oriented information over internet on testicular cancer

Quality of the patientoriented information on thyroid cancer in the internet

Coronavirus (COVID-19)

The Flesch Reading Ease and Flesch-Kincaid Grade Level

The Minervation validation instrument for healthcare websites (LIDA tool)

Is the Lida website assessment tool valid?

The DISCERN Instrument

Quality of patient health information on the Internet: reviewing a complex and evolving landscape

As Disinfectant Use Soars to Fight Coronavirus, So Do Accidental Poisonings

Coverage of health information by different sources in communities: Implication for COVID-19 epidemic response

The readability of pediatric patient education materials on the World Wide Web

Acknowledgements-None declared.This research did not receive any specific grant from funding agencies in the public, commercial, or not-forprofit sectors.J o u r n a l P r e -p r o o f