key: cord-0802800-a5jnxamu
authors: Aboueid, Stephanie; Meyer, Samantha B; Wallace, James R; Mahajan, Shreya; Nur, Teeyaa; Chaurasia, Ashok
title: Use of symptom checkers for COVID-19-related symptoms among university students: a qualitative study
date: 2021-01-29
journal: BMJ Innov
DOI: 10.1136/bmjinnov-2020-000498
sha: a96a0243b1f216bae04cca9138ea472118dbe453
doc_id: 802800
cord_uid: a5jnxamu

OBJECTIVE: Symptom checkers are potentially beneficial tools during pandemics. To increase the use of the platform, perspectives of end users must be gathered. Our objectives were to understand the perspectives and experiences of young adults related to the use of symptom checkers for assessing COVID-19-related symptoms and to identify areas for improvement. METHODS: We conducted semistructured qualitative interviews with 22 young adults (18–34 years of age) at a university in Ontario, Canada. Interviews were audio-recorded, transcribed, and analysed using inductive thematic analysis. RESULTS: We identified six main themes related to the decision of using a symptom checker for COVID-19 symptoms: (1) presence of symptoms or a combination of symptoms, (2) knowledge about COVID-19 symptoms, (3) fear of seeking in-person healthcare services, (4) awareness about symptom checkers, (5) paranoia and (6) curiosity. Participants who used symptom checkers shared by governmental entities reported an overall positive experience. Individuals who used non-credible sources reported suboptimal experiences due to lack of perceived credibility. Five main areas for improvement were identified: (1) information about the creators of the platform, (2) explanation of symptoms, (3) personalised experience, (4) language options, and (5) option to get tested. CONCLUSIONS: This study suggests an increased acceptance of symptom checkers due to the perceived risks of infection associated with seeking in-person healthcare services. Symptom checkers have the potential to reduce the burden on healthcare systems and health professionals, especially during pandemics; however, these platforms could be improved to increase use.

COVID-19 caused by SARS-CoV-2 was declared a pandemic by the WHO in March 2020. As of 30 December 2020, there was a total of 82 237 082 confirmed COVID-19 cases globally. 1 The average incubation period is 5 days but can be highly variable Health Apps and mHealth lasting up to 2 weeks. 2 A common global trend across primary healthcare settings is the move towards virtual consultations to help reduce the risk of COVID-19 transmission. [3] [4] [5] Only medically necessary services that require in-person consultations may be permitted 6 7 ; however, individuals are encouraged to report whether or not they are experiencing any COVID-19 symptoms to mitigate risks of infection. The quick spread of the virus and increased pressure on healthcare systems have led to the development of symptom checkers specifically for COVID-19-related symptoms.

Symptom checkers were developed and shared by various governmental websites and corporations to provide the general public with a means of self-assessing their level of COVID-19 risk or help users identify the level of care required. [8] [9] [10] [11] Symptom checkers vary in sophistication and have differing objectives with some designed to provide the likelihood of having COVID-19 and others designed for triage. 12 13 Importantly, most symptom checkers are public-facing tools with the known exception of the University of California, San Francisco (UCSF), which is one of the first COVID-19 symptom checkers that is fully integrated with the patient's medical record enabling direct appointment scheduling. 13 A study by Judson et al suggests that the UCSF recommended emergency-level care with high sensitivity, decreased triage time for patients with less severe illness and may have prevented hundreds of unnecessary encounters. 13 Therefore, despite the limited knowledge available on the use of symptom checkers 14 they are showing potential in improving efficiency.

Given the unpredictability of pandemics and their potential manifestation in the future, it is imperative to gather perspectives of symptom checker users to allow for improvement of this digital platform and maximise use. The data presented are part of a larger study that seeks to understand the factors associated with the use of symptom checkers among young adults. While this age segment is at lower risk of severe complications due to COVID-19, triage is still important among this population to reduce the risk of infection and community transmission.

The overarching objective of this study is to understand the perspectives and experiences of young adults (individuals between the ages of 18 and 34) related to the use of symptom checkers for assessing COVID-19related symptoms. Given that participants may not have been aware of the existence of symptom checkers prior to the interview, they were asked to use the WebMD symptom checker or Babylon Health. Once they were familiar with the platform, they were asked questions related to the use of a symptom checkers for COVID-19 and identify potential areas for improvement.

The tool under study is any public-facing symptom checker, which was used (or not used) by participants to assess COVID-19-related symptoms. In this study, symptom checkers were defined as digital selfassessment tools that use a chatbot to assist users in identifying their level of COVID-19 risks or identifying the need to seek urgent care based on symptoms reported.

To understand young adults' perspectives on the use of COVID-19 symptom checkers, we conducted semistructured interviews with university students enrolled in a university in Ontario, Canada. The first author who conducted the interviews did not personally know the participants. The semistructured interview protocol was used because it offers flexibility to the interviewer in determining when it is appropriate to explore certain subjects in greater depth or pose new questions that were not originally anticipated when the interview protocol was developed. 15 This interview is part of a larger mixed-methods study that aims to investigate the use of symptom checkers more broadly among young adults. To familiarise participants with symptom checkers, they were each asked to use the WebMD symptom checker or Babylon Health (the platform used was determined based on a draw)-these platforms were chosen because of their popularity and use by credible health institutions, respectively.

University students between the ages of 18 and 34 were eligible to participate in this study. To allow for a broad range of perspectives to be gathered, students across faculties, in all levels of education, and year of study were eligible to participate. University students were notified of the study through emails from the administrative assistant of their faculty; as such the number, of students who received and opened the email is unknown. Interested individuals were asked to contact the principal investigator (SA) to schedule an interview. All participants were provided with an information letter prior to the interview outlining its objectives. Informed consent was obtained from all participants. One-on-one interviews were conducted virtually through a secured platform, whereby. All participants were provided with a CAD $10 electronic coffee shop gift card as a token of appreciation for taking the time to participate in the study.

Interviews were conducted in Winter and Spring 2020, a time during which a state of emergency was declared in Canada. A preinterview questionnaire and a semistructured interview were developed to address this study's main objectives and are found in online supplemental appendix 1. To provide contextual information on each participant, the preinterview questionnaire was comprised of questions related to demographics such as age and gender as well as a single-item question assessing general self-rated health-this question was shown to be valuable in the meta-analysis conducted by DeSalvo and colleagues and may have an influence on participants' responses. 16 The first author (SA) conducted all interviews. SA holds a Master of Science in health systems; is trained in qualitative research methods, and was a PhD candidate in Public Health and Health Systems at the time of this study. A total of 31 participants expressed interest and 22 were included in the study based on a first-come, first-served basis and time of data saturation. Interviews ranged from ~30 min to 90 min in length. Given that data collection and analysis were occurring concomitantly, it was possible to cease recruitment once data saturation was reached (ie, collecting more data would not reveal new information). 17 Various qualitative researchers have given their interpretations of when participant recruitment should cease with the four types of saturation being theoretical saturation, inductive thematic saturation, a priori thematic saturation and data saturation. 18 This was assessed by (1) the first author who conducted all the interviews and thus was able to identify whether the data collected are becoming redundant (ie, the interviewer heard the same responses over and over from participants) and (2) the analysts who read the transcripts and acknowledged that the raw data are redundant.

We used the thematic analysis approach outlined by Castleberry and Nolen, which consist of compiling, disassembling, reassembling and interpreting the data. 19 Data analysis was completed independently by three researchers (SA, SM and TN) who have been trained in qualitative research. The first step of compiling consisted of transcribing all the interviews and getting a sense of the whole data by reading the transcripts in their entirety. To disassemble the data, a line-by-line coding approach was used to reduce the superimposition of preconceived notions on the datathis step generated descriptive codes. These descriptive codes were then used as a tag to retrieve and categorise similar data. The coding process was highly inductive, and a codebook was developed throughout the coding process. The third and final steps consisted of grouping codes into main themes and interpreting the themes, respectively. Inter-coder reliability of 85% was achieved and discrepancies were resolved through discussions and consensus. Emerging themes address the two main areas of focus of this study: themes related to the use of symptom checkers and themes related to user experience.

The sample consisted of a total of 22 participants with an age range between 18 and 33 years of age (median age=25). Demographic information of study participants are found in table 1. A total of eight participants used symptom checkers to assess COVID-19 symptoms of whom two used the platform for the first time due to COVID-19 and four used one from a governmental source. Participants who used a symptom checker for COVID-19 symptoms learnt about the platform through a governmental website, social media such as Twitter or through the Google search engine.

The first factor identified in this study as influencing the use of a symptom checker is the presence (or lack) of symptoms. Exhibiting a symptom or a combination of symptoms that could indicate COVID-19 led to the use of the platform:

I just wanted to see if I should be more concerned… if there is anything else that I should be concerned about these two symptoms together. (P9) Some (4/22) who were already aware of COVID-19 symptoms and did not exhibit them forewent the use of a symptom checker: I wanted to be more knowledgeable about the symptoms associated with the virus, but I did not use a symptom checker because I did not have the symptoms that I researched. (P3) 

Nonetheless, those (10/22) who did not use a symptom checker simply due to absence of COVID-19 symptoms indicated that they would have used the platform if they experienced symptoms.

I think with the pandemic it definitely adds some perspective. I think when you asked me about whether or not I would go to a clinic, I would not go in this pandemic. It put symptom checkers in a more positive light for me." (P21) "Potentially I would [ use a symptom checker ] if I had confusing symptoms. If they were between the flu and COVID-19. I would definitely prefer the symptom checker to not expose myself in the clinic. (P15)

The second theme relates to the participant's level of knowledge about COVID-19 symptoms and ability to remain informed of the latest emerging symptoms. There seemed to be a perception that symptom checkers are platforms that are designed to include the latest information about the COVID-19.

The use of a symptom checker was justified by some (3/22) who believed that it is difficult to remain abreast with emerging COVID-19 symptoms: I would seek out symptom checkers because it is tough to remember all the symptoms that are coming up everyday like the rash on the feet or loss of smell. (P15)

In contrast, some participants (3/22) who did not use a symptom checker felt that there was no need to do so given the readily available information about the COVID-19: I think the coronavirus symptoms are pretty selfexplanatory, everyone knows the symptoms about now and it's always in the news and in articles. (P21)

Another theme that emerged and that seemed to influence the use of symptom checkers is trust. A few participants (6/22) perceived symptom checkers to be more trustworthy than the Google search engine:

There is research being conducted on symptom checkers so maybe more trustworthy than Google. The only thing I google are people experiences with a condition but not symptoms. (P5) This theme seemed to be influenced by the participant's age group:

The only barrier is the age groups but people of our age, millennials, are the ones who believe in saving time and energy and we can trust this kind of platform for sure. (P6)

Fear of seeking in-person healthcare services Level of fear associated with seeking in-person healthcare services has been mentioned as a factor in deciding whether or not to use a symptom checker. Many participants (13/22) mentioned that due to the pandemic, symptom checkers are becoming more accepted due to the perceived risks of an infection associated with seeking in-person care. Another reported reason (4/22) for accepting the use of a symptom checker is to reduce the burden on the healthcare system:

The results from the symptom checker would suffice because there is a risk with going to the clinic. I don't need to burden the healthcare system at all with something that I could potentially manage on my own. (P3) Before the coronavirus, I never thought I would be fearful of going to a hospital. Even if I am suffering from some issues not related to coronavirus, I personally would not go to the hospital because I feel that they are the most contaminated places right now. I know they are sterilizing the facilities, but I do not trust it. (P20)

In contrast, a few participants (2/22) mentioned that using a symptom checker does not suffice and that a physical examination is necessary to diagnose COVID-19. This was especially important for a participant who mentioned the seriousness of the condition and the importance for him to know whether he truly has the virus to ensure that he can protect his elderly family members. Taking precautionary measures such as wearing personal protective equipment and choosing a clinic with lower foot traffic were mentioned to be ways to reduce risks of infection:

That's a good question, is the coronavirus testing free in Ontario? If it is, I would check that first, and then I would probably get it checked properly because it's quite serious if I have it because I live with my grandparents so I would want to know if I have it. I would definitely prepare myself properly before going to the clinic by wearing gloves and a mask. I would also check how many clinics there are in my area and would try to go to one that is less crowded. (P19)

Another theme that was identified was the level of awareness of the existence of symptom checkers. Some participants (9/22) did not use a symptom checker simply because they did not know about their existence; however, participants seemed interested in using such a platform:

Lack of awareness, I did not know the platform existed. I would definitely use it if I knew about it." (P18) "It [ this study ] made me more aware about the different websites that offer symptom checkers. The only one I ever used to be the one on WebMD. (P15)

Health Apps and mHealth outlets has resulted in a feeling of paranoia regarding symptoms that would otherwise be perceived as minor in the absence of a pandemic:

When you listen to the news you become a bit paranoid. Oh, I am tired today, is it coronavirus? Oh, I have a headache, is it coronavirus? Oh, I have a runny nose, is it coronavirus? (P14) Using a symptom checker was comforting for some users (2/22) who reported feelings of paranoia; however, the output provided seemed trivial:

They did provide recommendations on what to do should someone have COVID. I think it was comforting because it said I had a mild flu, but anyone could have guessed that. (P10)

Curiosity was another theme that emerged from the data to justify the use of symptom checkers. Some participants (5/22) were curious to identify the cause of their symptoms while others were curious about the platform itself: I used it because of curiosity mainly and some paranoia. I always have headaches, I am usually tired, symptoms that are typical to any nutritional deficiencies-there are things that have other explanations but I'm like ok, I'll take a look at this." (P10) "I always wondered what it looked like… But I just wanted to see how it worked, how it looks like because it [ symptom checker ] was put together so quickly. (P2)

Codes identified regarding participants' user experience were grouped under positive or suboptimal experiences. The experience of those who used a symptom checker varied based on the platform used. While most used a symptom checker found on a governmental website, a few (4/22) used symptom checkers that seemed to be non-credible.

Those who used a governmental symptom checker (4/22) reported an overall positive experience with most reporting that the platform was well designed, straightforward, fast and simple:

It [ symptom checker ] was really good. It was welldesigned and the questions were straight forward. I didn't feel like I did not know how to answer a question so I felt confident with the answers I gave. I was pleased with the results it gave and how fast it was. I think it was a minute or two to use, it was very fast… I used the symptom checker on the Government of Ontario website. (P11) Some participants (2/22) mentioned that the symptom checker was useful as it reassured them that they did not have COVID-19, which reduced fear among users: I was impressed at how quickly it was mobilized and shared. I think the language was good, it didn't result in fear or hysteria… I was impressed by it and I was happy that it was online because I think it reduced the fear that people have. (P13)

While most believed that the governmental symptom checker to be useful (3/4), a few perceived the questions to be general and that the recommendations provided to most people are the same indicating a lack of a personalised approach to symptom checkers: I didn't find it super accurate; it gives the same recommendations to most people unless you're dying and experiencing a shortness of breath or chest pain. It will likely tell you to stay home…. I used the Government symptom checker. (P12) As for those who used a non-governmental symptom checker (4/22), credibility and legitimacy were two codes that emerged from the data and indicated suboptimal experiences with symptom checkers: I don't think the website I used was credible. It was a link on WhatsApp, and it was super short, and the questions were shallow, I don't think it was a legitimate website. I think it was fine, if I used a better website, I would have probably been happier with the outcome because it felt that it was just a media thing. (P10) Nonetheless, participants mentioned improvements that could be made to COVID-19 symptom checkers. These improvements were grouped into five main themes and are presented in table 2.

This study is part of a larger study that seeks to understand the factors associated with the acceptance and use of symptom checkers for self-triage. This area of inquiry relates to the technology acceptance model, which describes that the usefulness of a new technology is hindered if target segments are not willing to accept or use the technology. 20 We believe that the COVID-19 pandemic has played a key role in influencing acceptance of symptom checkers by the general public due to the perceived infection risks associated with in-person medical consultations. While a COVID-19 symptom checker has the potential to reduce hundreds of unnecessary encounters, its usefulness is limited if the volume of use decreases over time-a phenomenon that was observed in a recent seminal study. 13 While there are many factors that could contribute to this such as increased knowledge about COVID-19 symptoms and fewer individuals experiencing symptoms, other factors related to symptom checker use should be considered.

Enablers for using a COVID-19 symptom checker identified were experiencing COVID-19 related symptoms, not being able to remain up-to-date with the latest emerging COVID-19 symptoms, fear of seeking in-person healthcare services and exhibiting feelings of paranoia or curiosity. Barriers for using a COVID-19 symptom checker identified were lack of awareness about the existence of the platform and perception of the severity of the disease.

Despite the limited knowledge regarding symptom checkers 14 most participants who used a COVID-19 symptom checker reported a positive experience. This is in line with a study that reported the perceived usefulness of symptom checkers for assessing the health of oneself. 21 Participants who had suboptimal experiences with a COVID-19 symptom checker reported using one that was from a non-credible source. This highlights the need to ensure the credibility and validity of COVID-19 symptom checkers as failing to do so poses clinical risks to consumers. 22 23 As such, while symptom checkers have the potential to improve quality of care and health system performance worldwide, they must be well designed to prevent unintended consequences such as failing to advise users to seek testing when warranted.

In addition to involving consumers, regulators and healthcare professionals in developing and testing symptom checkers for quality improvement, 23 governmental institutions have a key role in raising awareness and disseminating information about COVID-19 symptom checkers that are known to be credible and regularly updated based on the latest emerging scientific findings. Participants also mentioned the importance of knowing who developed the platform indicating that transparency may be a factor influencing decision of platform use.

Given the similarities between the influenza and COVID-19, participants mentioned the need to explain symptoms further to enable users to identify the symptoms they are truly experiencing and in turn obtain accurate results. This suggests that the risk of misinterpreting symptoms is not unique to older adults accentuating the need to explain symptoms to symptom checker users. 24 While most COVID-19 symptom checkers were designed as a population health tool, participants believed that the platform was too generic and did not provide an individualised experience. Some individuals may believe that their case is unique, providing users with a personalised experience may influence their use of the platform. However, it is important to inform users about the use of clear clinical guidelines to develop self-triage tools that would justify the lack of personalisation.

Moreover, some participants mentioned that the platform would be more useful if they had the option to seek further care or testing. This speaks to a limitation of most public-facing symptom checkers as they do not provide an option for appointment scheduling. This further accentuates the need for Electronic Health Record-tethered portals in enhancing the overall benefits of self-triage tools by allowing patients to be triaged and scheduled in a matter of minutes. 13 Language options were also mentioned to be important by participants whose first language was neither English nor French. Given North America's cultural and lingual diversity, offering COVID-19 symptom checkers in various languages may further reduce unnecessary medical visits.

While this study, to our knowledge, is the first to provide some important insights into the use of symptom checkers for COVID-19-related symptoms among university students, there are several limitations that warrant mention. First, the use of symptom checkers during the interview was not focused on COVID-19 symptoms as the purpose was to familiarise participants with the platform prior to asking questions regarding its use. Second, given that existing symptom checkers vary widely in terms of their layout and user experience, these findings are limited to the ones used in this study; however, some areas of improvement may be relevant to another symptom checker (eg, language options). Third, with the exception of participants who used a governmental symptom checker, some participants were unable to recall the symptom checker they used limiting the research team's ability to understand which platforms were resulting in suboptimal experiences. In addition, as with any qualitative study, findings are customised to the group segment and settings included; as such, given that the sample is relatively highly educated, (15/22) When I typed in things related to my sore throat a lot of other things would pop up and I didn't know the difference between the symptoms. So, I wish it explained the differences between the sub-sections of having a sore throat. (P9) Personalised experience (12/22) Maybe a bit disappointed because I thought answers would be more straightforward. It was expected that it would tell me to self-isolate but it's normal given the level of information that is currently available on the symptoms. (P14) Language options (3/22) More languages would also be helpful other than English and French. (P11) Option to get tested (6/22) Or even having the option to do the test would have been helpful as well. (P14)

Health Apps and mHealth healthy and perceived to be eager adopters of technology, 25 the transferability of the results may be limited to similar group segments. Last but not least, while there is ongoing debate related to the topic, data saturation rather than inductive thematic saturation or a priori thematic saturation was used to guide when participant recruitment should cease-this approach typically requires a smaller sample size as the focus is on the redundancy of data rather than themes. Given these limitations and to expand our knowledge on the use of symptom checkers, future work should leverage and adapt the interview protocol used in this study to focus on gathering the perspectives of other age and group segments.

Symptom checkers have the potential to reduce unnecessary medical visits and provide useful information to users. The COVID-19 pandemic seems to have improved symptom checker acceptance due to the risks associated with seeking in-person healthcare services.

To improve the use of symptom checkers, the platform will need to provide information about its creators, explain symptoms to address any ambiguity, provide a personalised experience, include additional language options, provide options to get tested for COVID-19, and increase awareness about the existence of the platform. These areas could be addressed, in part, by striving to integrate symptom checkers with electronic health records.

COVID-19 Dashboard by the center for systems science and engineering

Early transmission dynamics in Wuhan, China, of novel coronavirus-infected pneumonia

COVID-19 transforms health care through telemedicine: evidence from the field

Virtual health care in the era of COVID-19

Digital technology and COVID-19

Specialty guides for patient management during the coronavirus pandemic, United Kingdom

COVID-19 pandemic guidance for the health care sector

Healthdirect coronavirus (COVID-19) symptom Checker. Australia

9 Government of New Brunswick. COVID-19 self-assessment

Covid-19 screener

COVID-19 care assistant

I asked eight chatbots whether I had Covid-19. The answers ranged from 'low' risk to 'start home isolation

Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19

The use of artificially intelligent Self-Diagnosing digital platforms by the general public: Scoping review

Qualitative research and evaluation methods

Mortality prediction with a single General self-rated health question. A metaanalysis

Constructing grounded theory

Saturation in qualitative research: exploring its conceptualization and operationalization

Thematic analysis of qualitative research data: is it as easy as it sounds?

Perceived usefulness, perceived ease of use, and user acceptance of information technology

Patient perspectives on the usefulness of an artificial Intelligence-Assisted symptom Checker: cross-sectional survey study

Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review

Safety of patient-facing digital symptom checkers

Older adult experience of online diagnosis: results from a scenario-based think-aloud protocol

Shaping the future of health and medicine

The authors would like to thank the participants who took the time to complete the interview.Contributors SA conceived and designed the study. SBM and AC assisted in the conception of the study. SA conducted and transcribed all qualitative interviews. SA, SM and TN analysed the data. SA drafted the manuscript. All authors reviewed and approved the final version of the manuscript.

The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Patient consent for publication Not required. Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

This article is made freely available for use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, noncommercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.