key: cord-1041266-4s4s5sd8 authors: Blacklow, Serena O; Lisker, Sarah; Ng, Madelena Y; Sarkar, Urmimala; Lyles, Courtney title: Usability, inclusivity, and content evaluation of COVID-19 contact tracing apps in the United States date: 2021-05-22 journal: J Am Med Inform Assoc DOI: 10.1093/jamia/ocab093 sha: c3e38a82259a046fc24640a56a4845e047eaebd4 doc_id: 1041266 cord_uid: 4s4s5sd8 OBJECTIVE: We evaluated the usability of mobile COVID-19 contact tracing apps, especially for individuals with barriers to communication and limited digital literacy skills. MATERIALS AND METHODS: We searched the Apple App Store, Google Play, peer-reviewed literature, and lay press to find contact tracing apps in the United States. We evaluated apps with a framework focused on user characteristics and user interface. RESULTS: Of the final 26 apps, 77% were on both iPhone and Android. 69% exceeded 9(th) grade readability, and 65% were available only in English. Only 12% had inclusive illustrations (different genders, skin tones, physical abilities). 92% alerted users of an exposure, 42% linked to a testing site, and 62% linked to a public health website within 3 clicks. CONCLUSION: Most apps alert users of COVID-19 exposure but require high English reading levels and are not fully inclusive of the U.S. population, which may limit their reach as public health tools. 59 60 In the United States (U.S.), contact tracing has been performed with localized, sometimes disjointed, approaches. Manual contact tracing, in which trained personnel conduct interviews with those who have tested positive for the virus ("cases"), requires a large workforce and cannot keep pace with the mounting number of COVID-19 infections.[1-3] Therefore, some states introduced contact tracing mobile applications (apps) to augment their contact tracing efforts. [4] [5] [6] It is estimated that these apps must be used by nearly 60% of the population to reduce the spread of the virus, though any use at all helps identify people with potential exposure ("contacts"). [7] Adoption of contact tracing apps is relatively high in Qatar and Iceland (91% and 40%, respectively). [5] In the U.S. and other countries, however, major issues with app adoption revolve around engagement, privacy concerns, cybersecurity, and accuracy. [6, [8] [9] [10] [11] [12] [13] [14] [15] [16] Many of these barriers are inextricably linked with the digital divide in the U.S., from structural to individual levels. [6, [8] [9] [10] [11] [12] [13] [14] [15] [16] At the structural level, there are clear disparities in smartphone ownership and broadband/high speed Internet access. [17] At the individual level, even among those with existing devices and sufficient Internet access, digital literacy skills and app use vary among consumers. Experts define digital engagement as "how well users can use a product to achieve their goals and how satisfied they are with that process." [18] Therefore, it is clear that contact tracing apps can only achieve widespread use if they are usable by diverse populations, especially considering the disproportionate number of COVID-19 infections in low-income, non-white populations. Several studies have assessed national contact tracing apps, along with COVID symptom-tracking or informational apps. [5, 9, [19] [20] [21] [22] [23] These studies evaluated apps with various frameworks in the following general categories: accessibility, functionality, engagement, aesthetics, and inclusion of resources and information. [5, 9, [19] [20] [21] [22] [23] They were not specific to common health communication barriers and diversity or inclusion concepts, which were our focus here. Other usability frameworks, such as Nielsen's or Kientz's, similarly encompass these broad, subjective categories. [24, 25] No study to 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 This study focused on usability within two domains of the FDA Applying Human Factors and Usability Engineering to Medical Devices guidance framework: 1) "user characteristics" with careful expansion within this domain to define categories related to health communication, such as readability and language; and 2) "user interface" with a focus on specific elements in this domain related to audiovisual information (including racial and gender diversity within visual elements) and functionality/logic of information presented (such as number of clicks and provided resources). [26] The "user environment" within this FDA framework was not evaluated specifically, as we limited our evaluation to upfront engagement in the apps rather than completion of the contact tracing process. [26] Focusing on "user characteristics" and "user interface and functionality" domains of the FDA framework for this analysis allowed the study team to narrow in on elements related to accessibility and inclusivity; we anticipate that other research may examine additional, broader usability aspects such as type of exposure notification. More specifically, we combined this framework with concepts from multiple published studies outlining health communication and diversity barriers related to technology use, resulting in a 14-item evaluation framework (see Table 2 for details). [5, 9, [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] Five of these items were more closely related to "user characteristics," such as readability, while the remaining items and our clickability evaluation were more related to "user interface" and functionality. One author (SB) primarily evaluated apps using an iPhone 7. A second coder (SL) used an iPhone 8 to independently assess 5 (20%) of the apps to ensure reproducibility of coding. We downloaded 50% of apps evaluated on the iPhone onto an Android (Samsung Galaxy J3) to verify the same functionalities and appearances on Android as on iPhone (with no differences to note, data not shown). We scored apps in each category and recorded the data in Microsoft Excel. For the majority of coding, we categorized apps either dichotomously as yes/no (e.g., app has audio or visual components) or into discrete groups (e.g., readability level at <6 th grade, 6 th -9 th grade, or >9 th grade). For six prespecified functions, we recorded the number of clicks to reach the target feature or content. Finally, we selected three apps that provided concrete examples of several user and user interface domains that could be considered in future app designs: 1) written at lower reading levels; 2) used racially and gender diverse visual features; 3) included audiovisual components. The initial search yielded 54 apps related to contact tracing in the U.S. After applying our exclusion criteria, we evaluated 26 apps (Figure 1) . The coding was highly consistent between coders, with a range of Cohen's Kappa between the categories of 0.95 to 1.0. State and county government-affiliated entities created 19/26 (73%) of the apps, but notably Apple and Google jointly created the Exposure Notification Express system, which they embedded in the operating systems without a corresponding app on iPhone (Table 1, Figure 2 ). Most apps (20/26, 77%) were available on both iPhone and Android ( Table 1, Table 2 ). Public health departments in some regions (CO, DC, CA, MD, CT, and WA) only provided Android apps, but all of those states were also available in the Exposure Notifications Express system in iPhone Settings (Table 1) . Overall, 18/26 (69%) were above 9 th grade readability and 17/26 (65%) were available only in English ( Table 2) . 5/26 (19%) were available in more than three languages, including Spanish ( Table 2) . 22/26 (85%) did not require users to input a phone number to sign up ( Table 2) . Regarding user interface and functionality, the vast majority of apps directly alerted users of an exposure and explained how their alert system worked, but only 8/26 (31%) included videos or illustrations to do so ( Table 2 ). Very few (3/26, 12%) included illustrations with diverse representations (different genders, skin tones, or physical abilities) ( Table 2) . Though less than half provided links to find physical testing locations, 16/26 (62%) linked to a local or state health department website within 3 clicks (Table 2, Figure 3 ). 22/26 (85%) verified a user-reported positive test with local health authorities (data not shown). None of the apps provided direct links to social support services or resources for quarantining ( Table 2 ). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 *Exposure Notifications Express refers to the iPhone-integrated system that can be accessed from iPhone Settings. The listed states enabled this method for contact tracing. The corresponding system on Android has generated apps, which were evaluated separately (see T-Y). Therefore, some apps on Android do not have corresponding apps on iPhone. The average reading level in the United States is 7 th -8 th grade, and 20% of adults cannot read above a 5 th grade level. [23] Though some of these apps had home screens with few sentences or words, their explanations of how the apps work and summaries of the privacy policy were well above 5 th grade level. English 4.9% of United States citizens over age 18 speak English less than "very well." [34] For these apps to have high usage rates and be efficacious, they must not exclude those who primarily speak a language other than English or cannot read English, especially given the high literacy levels already required to understand the apps. Each concept is linked to one word (as opposed to switching between terms, such as "coronavirus" and "COVID-19" or "alerts" and "notifications"). Consistent wording aids in understanding and remembering information. People should be able to participate in contact tracing efforts without having to share their personal information. iPhone and Android compatible 20 (77) A-S, Z Apps are available on both iPhone and Android. Apps will work best when they include as many users as possible. Limiting an app to a certain device and a certain operating system (not addressed here) limits the potential number of users and therefore its potential efficacy. Audio or video option 2 (8) K, S Audio or video is available to explain how the app works and/or to help navigate the app. Apps should include audio or video options to assist those with communication barriers (e.g. visual impairment, limited literacy) use the app. Providing a "How it Works" section accessible from the home page allows users to re-visit this information after first use. This feature may help build user trust. [15] Directly alerts user of an exposure 24 (92) A-L, S-Z The app alerts the user of a potential exposure using Exposure Alerts rather than solely collecting data. Part of contact tracing includes alerting contacts of their exposure. If the app only collects information and sends it to the associated Health Department, without alerting contacts automatically, there is a time lag between when a contact could be alerted of an exposure and when the contact is finally alerted. Links to a COVID testing site 11 (42) A, B, D, E, I, K, M, O-R The app helps the user find a testing site. In the event of an exposure, the user should be able to quickly find a testing site. Within app: J, R Externally linked: A-P, R-Z The Privacy Policy is accessible from the app without requiring the user to search for it online. The Privacy Policy should be clearly displayed within the app. Ideally, the Privacy Policy is described in simple writing within the app. The comprehensive policy could be an external link available from the app. The user can access further guidance and resources (for example, social support for quarantine). Links to social support for quarantine 0 (0) None The app links the user to social support services, if the user requests, for quarantine or isolation support. Users are equipped to help reduce the spread of COVID-19 if they have access to the tools they need to quarantine after an exposure alert. September 2020. It is free and does not prompt for user registration. The app runs in the background of a user's phone and, as explained within the app, does not collect any personally identifiable information because it uses Bluetooth rather than Location sensing. Using a specific, illustrated example of two people meeting in a grocery store, the app explains its utilization of Bluetooth token exchanges to remember interactions between users exceeding 10-20 minutes. If a user tests positive and reports it in the app, the Department confirms the case before the app alerts users of an exposure. All app illustrations represent a spectrum of skin tones and of genders. However, readability level of the "How it Works" explanation is 9.1, and the app is only available in English. AlohaSafe Alert was launched by the Hawaii State Department of Health in November 2020. It similarly does not require registration, uses Bluetooth sensing, and confirms cases before alerting contacts of exposures. When first opening the app, users choose between Spanish and English. The welcome illustration represents different skin tones and genders. A concise explanation of how the app works has a readability level of 6.0. Part of this explanation is: "In the event of an encounter, your data and information remain anonymous. The app doesn't store any personal data. Only random IDs are exchanged. These are deleted after 14 days." Within the app, however, there is some inconsistent wording, as "exposure," "detection," "alert," and "report" are used interchangeably. The app also links to the Hawaii Department of Health website, but it does not clearly present a link to access testing or provide user feedback. Exposure Notifications Express was launched by Apple and Google in September 2020. It is not an app on the iPhone, rather a system that can be enabled from Settings. (On Android, Google creates apps corresponding to the states that opt-in to their system, and those apps were evaluated separately). Since this system is integrated with the iPhone, the user can use any language available on the iPhone and enable VoiceOver for low vision. Unfortunately, there are no illustrations and the welcome message readability, depending on the state, is around 11. This system does not provide links to more information or testing sites. Overall, most contact tracing apps included basic functionalities such as alerting users of exposures with few clicks. The most room for improvement was in inclusivity for potential users. First, while the average reading level in the U.S. is 7 th -8 th grade and 20% of adults cannot read above a 5 th grade level, [23] the readability of contact tracing apps (even excluding the comprehensive privacy policies) were higher and potentially less accessible for the general population. High readability levels have also been reported for privacy policies of other contact tracing apps. [23] Second, 30.6% of U.S. citizens over age 18 who speak a language other than English speak English less than "very well," [34] yet only 35% of apps were accessible in languages other than English. Third, audiovisual features have the potential to enhance understanding, improve recall, and make users feel familiar, [35, 36] and these were infrequent within the contact tracing apps. Finally, none of the apps in this study provided direct links to social support services that individuals may need in the event of an exposure alert. Prior studies evaluated contact tracing apps; however, this was mostly done outside the U.S. using subjective rating scales, such as the Mobile App Rating Scale or the System Usability Score. [5, 9, [19] [20] [21] [22] [23] 28] Our more objective rating system with a specific emphasis on health communication and inclusivity 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 may therefore offer new insights, such as providing novel data on audiovisual features, readability, and inclusivity of graphics presented (e.g., people of different races, genders, and abilities). Our study has limitations. First, we acknowledge that these apps were designed within tight timeframes and developers might not have been able to implement all features to date. We also did not use broader usability metrics as identified in other research, [24] [25] [26] [27] [28] since our intention was to focus on health communication and diverse end users more explicitly. Future work is needed to replicate and/or expand our list of inclusivity and functionality criteria, as this paper is not intended to validate but rather to enumerate and document the various domains to consider in this space. In addition, we did not fully interact with the exposure alert systems because we did not come in contact with cases during the study (i.e. we were unable to evaluate further links or information that may become available to users in the event of an exposure alert). We also only evaluated the apps in English. Finally, it is possible that the apps have changed since our evaluation due to the iterative nature of development. Despite these limitations, our suggestions are still valid for future app development. For contact tracing apps to be maximally effective, they must be usable and accessible to the population they aim to serve, including those with low digital literacy and different backgrounds. Our findings present concrete features and categories for developers to consider in current and future apps (for contact tracing and beyond). Further, our work builds upon existing standards for accessibility of digital health to reach diverse end users, such as those developed by the Agency for Healthcare Research and Quality, [18] The Commonwealth Fund, [19] , the FDA, [26] Xcertia, [31] and the National Academy of Medicine. [32] Moving forward, developers should routinely reference these standards to increase usability of apps and implementation guidelines into real-world practice. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Journal of the American Medical Informatics Association 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 It's taking a week for contact tracing to reach COVID-19 patients Did someone with COVID-19 infect you? California is short on contact tracers to find out. The Sacramento Bee Digital technology applications for contact tracing: the new promise for COVID-19 and beyond? COVID-19 Fight enlists digital technology: Contact tracing apps. Engineering (Beijing, China) 2020. Advance online publication Integrating emerging technologies into COVID-19 contact tracing: Opportunities, challenges and pitfalls Acceptability of App-Based Contact Tracing for COVID-19: Cross-Country Survey Study Health Apps on COVID-19 Launched in the Early Days of the Pandemic: Content Analysis and Review Peer-to-Peer Contact Tracing: Development of a Privacy-Preserving Smartphone App Personalizing symptom monitoring and contact tracing efforts through a COVID-19 web-app More than 1.6 million Swiss have downloaded Covid tracing app Attitudes Toward Using COVID-19 mHealth Tools Among Adults With Chronic Health Conditions: Secondary Data Analysis of the COVID-19 Impact Survey Belief of having had unconfirmed Covid-19 infection reduces willingness to participate in app-based contact tracing Facilitators and barriers to engagement with contact tracing during infectious disease outbreaks: A rapid review of the evidence Adoption of a Contact Tracing App for Containing COVID-19: A Health Belief Model Approach Accessible Health Information Technology for Populations with Limited Literacy: A Guide for Developers and Purchasers of Health IT. (Prepared by the National Opinion Research Center for the National Resource Center for Health IT.) Agency for Healthcare Research and Quality Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications A Systematic Review of Smartphone Applications Available for Corona Virus Disease 2019 (COVID19) and the Assessment of their Quality Using the Mobile Application Rating Scale (MARS) Features and Functionalities of Smartphone Apps Related to COVID-19: Systematic Search in App Stores and Content Analysis Citizen-Centered Mobile Health Apps Collecting Individual-Level Spatial Data for Infectious Disease Management: Scoping Review COVID-19 Contact-Tracing Apps: Analysis of the Readability of Privacy Policies 10 Usability Heuristics for User Interface Design Heuristic Evaluation of Persuasive Health Technologies Applying Human Factors and Usability Engineering to Medical Devices: Guidance for Industry and Food and Drug Administration Staff SUS: A Retrospective Mobile app rating scale: a new tool for assessing the quality of health mobile apps Digital health and the COVID-19 epidemic: an assessment framework for apps from an epidemiological and legal perspective Improving the User Experience App Guidelines: 2019 Board Approved Xcertia Guidelines Health Literacy Insights for Health Crises Derivation of new readability formulas (automated readability index, fog count, and flesch reading ease formula) for Navy enlisted personnel United States Census Bureau. Language Spoken at Home. TableID: S1601 Why illustrations aid understanding 5322&rep=rep1&type=pdf#:~:text=But %20illustrations%20are%20also%20used,as%20in%20a%20paragraph%20of. Accessed Effective use of patient education illustrations