key: cord-0888765-1w6bdsbk authors: Ben-Zeev, Dror; Chander, Ayesha; Tauscher, Justin; Buck, Benjamin; Nepal, Subigya; Campbell, Andrew; Doron, Guy title: A Smartphone Intervention for People With Serious Mental Illness: Fully Remote Randomized Controlled Trial of CORE date: 2021-11-12 journal: J Med Internet Res DOI: 10.2196/29201 sha: 5f8eaf38219f3a2e5991dcd7dbdb9a439a5e76e2 doc_id: 888765 cord_uid: 1w6bdsbk BACKGROUND: People with serious mental illness (SMI) have significant unmet mental health needs. Development and testing of digital interventions that can alleviate the suffering of people with SMI is a public health priority. OBJECTIVE: The aim of this study is to conduct a fully remote randomized waitlist-controlled trial of CORE, a smartphone intervention that comprises daily exercises designed to promote reassessment of dysfunctional beliefs in multiple domains. METHODS: Individuals were recruited via the web using Google and Facebook advertisements. Enrolled participants were randomized into either active intervention or waitlist control groups. Participants completed the Beck Depression Inventory-Second Edition (BDI-II), Generalized Anxiety Disorder-7 (GAD-7), Hamilton Program for Schizophrenia Voices, Green Paranoid Thought Scale, Recovery Assessment Scale (RAS), Rosenberg Self-Esteem Scale (RSES), Friendship Scale, and Sheehan Disability Scale (SDS) at baseline (T1), 30-day (T2), and 60-day (T3) assessment points. Participants in the active group used CORE from T1 to T2, and participants in the waitlist group used CORE from T2 to T3. Both groups completed usability and accessibility measures after they concluded their intervention periods. RESULTS: Overall, 315 individuals from 45 states participated in this study. The sample comprised individuals with self-reported bipolar disorder (111/315, 35.2%), major depressive disorder (136/315, 43.2%), and schizophrenia or schizoaffective disorder (68/315, 21.6%) who displayed moderate to severe symptoms and disability levels at baseline. Participants rated CORE as highly usable and acceptable. Intent-to-treat analyses showed significant treatment×time interactions for the BDI-II (F(1,313)=13.38; P<.001), GAD-7 (F(1,313)=5.87; P=.01), RAS (F(1,313)=23.42; P<.001), RSES (F(1,313)=19.28; P<.001), and SDS (F(1,313)=10.73; P=.001). Large effects were observed for the BDI-II (d=0.58), RAS (d=0.61), and RSES (d=0.64); a moderate effect size was observed for the SDS (d=0.44), and a small effect size was observed for the GAD-7 (d=0.20). Similar changes in outcome measures were later observed in the waitlist control group participants following crossover after they received CORE (T2 to T3). Approximately 41.5% (64/154) of participants in the active group and 60.2% (97/161) of participants in the waitlist group were retained at T2, and 33.1% (51/154) of participants in the active group and 40.3% (65/161) of participants in the waitlist group were retained at T3. CONCLUSIONS: We successfully recruited, screened, randomized, treated, and assessed a geographically dispersed sample of participants with SMI entirely via the web, demonstrating that fully remote clinical trials are feasible in this population; however, study retention remains challenging. CORE showed promise as a usable, acceptable, and effective tool for reducing the severity of psychiatric symptoms and disability while improving recovery and self-esteem. Rapid adoption and real-world dissemination of evidence-based mobile health interventions such as CORE are needed if we are to shorten the science-to-service gap and address the significant unmet mental health needs of people with SMI during the COVID-19 pandemic and beyond. TRIAL REGISTRATION: ClinicalTrials.gov NCT04068467; https://clinicaltrials.gov/ct2/show/NCT04068467 yes: all primary outcomes were significantly better in intervention group vs control partly: SOME primary outcomes were significantly better in intervention group vs control no statistically significant difference between control and intervention potentially harmful: control was significantly better than intervention in one or more outcomes inconclusive: more research is needed Other: Approx. Percentage of Users (starters) still using the app as recommended after 3 months * Overall, was the app/intervention effective? * Pilot/feasibility Is this a full powered effectiveness trial or a pilot/feasibility trial? * Manuscript tracking number * If this is a JMIR submission, please provide the manuscript tracking number under "other" (The ms tracking number can be found in the submission acknowledgement email, or when you login as author in JMIR. If the paper is already published in JMIR, then the ms tracking number is the four-digit number at the end of the DOI, to be found at the bottom of each published article in JMIR) 1a) Does your paper address CONSORT item 1a? * I.e does the title contain the phrase "Randomized Controlled Trial"? (if not, explain the reason under "other") 1a-i) Identify the mode of delivery in the title Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if offline products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms. Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support"). Does your paper address subitem 1a-ii? Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 1b-ii) Level of human involvement in the METHODS section of the ABSTRACT Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). "for People with Serious Mental Illness" "Participants in the active condition used CORE from T1 to T2 and participants in the waitlist control condition used CORE from T2 to T3." subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-ii? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials). Note: In traditional offline trials, an open trial (open-label trial) is a type of clinical trial in which both the researchers and participants know which treatment is being administered. To avoid confusion, use "blinded" or "unblinded" to indicated the level of blinding instead of "open", as "open" in web-based trials usually refers to "open access" (i.e. participants can self-enrol). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-iii? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer INTRODUCTION 1b-iv) RESULTS section in abstract must contain use data Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. "Traditional participant recruitment strategies are dependent on collaborative partnerships with study sites and clinician referrals [21] . These create potential biases in clinical trial samples because the individuals enrolled already need to be known to healthcare providers and actively receiving care in order to be offered the opportunity to engage in research. The participant samples used in standard clinical trials may therefore be particularly unrepresentative of hard-to reach populations such as people with SMI, who are known to disengage from clinic-based services for extended periods of time [22, 23] ." "we report on the first fully remote randomized controlled clinical trial of an mHealth intervention for people with SMI. The first objective of the study was to evaluate whether individuals with SMI can be recruited, assessed, and engaged in a digital intervention successfully in a fully remote clinical trial. " subitem not at all important essential Does your paper address subitem 2a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 2b) In INTRODUCTION: Specific objectives or hypotheses Does your paper address CONSORT subitem 2b? * "Treatment of people with SMI typically takes place in publicly-funded clinics and community mental health centers that are chronically under-resourced, under-staffed, and over-extended [4, 5] . These public sector agencies are rarely able to meet the demand for services [6] . Barriers in capacity to provide high quality care on the provider-side may be compounded by hesitancy around treatment seeking on the patient-side [7] . People with SMI are often exposed to pervasive societal stigma about their conditions, and therefore can be reluctant to openly seek services at a clinic if it risks being labeled "mentally ill" [8, 9] . The consequence of these intersecting challenges is a vicious cycle in which those who are most impaired receive the least amount of support, thus deteriorating even more over time." "One area where the opportunities available to people with SMI are not grossly behind those afforded to the general population is access to mobile technologies [10, 11] . Thanks to subsidy programs that offer people with disabilities access to mobile phones and data plans (e.g., Federal Communications Commission Lifeline Program) and the dropping prices of mobile phones globally, people with SMI now have unprecedented opportunities to access information, stay connected to others, and potentially receive services remotely via personal mobile devices [12] ." "Recruitment of participants for clinical trials is the most salient cause for study delays and a major obstacle in expeditiously moving novel and potentially helpful interventions to real-world practice [20] . Traditional participant recruitment strategies are dependent on collaborative partnerships with study sites and clinician referrals [21] . These create potential biases in clinical trial samples because the individuals enrolled already need to be known to healthcare providers and actively receiving care in order to be offered the opportunity to engage in research. The participant samples used in standard clinical trials may therefore be particularly unrepresentative of hard-to reach populations such as people with SMI, who are known to disengage from clinic-based services for extended periods of time [22, 23] . To overcome these obstacles, a growing number of studies have employed online participant recruitment strategies including social media advertisements, virtual outreach through online interest groups, search engine advertisements, and various other website campaigns. Across clinical populations these efforts have yielded impressive results in terms of cost effectiveness, time-efficiency, and reach [24] . In clinical trials where the experimental intervention is digital and behavioral in nature (i.e., does not require direct contact between a patient and provider) and the comparator condition does not require in-person contact with the research team (e.g., no intervention control, waitlist control, or digital attention-control alternative), recruitment, treatment, and assessment of outcomes can all be conducted fully remotely and potentially on a single device [25] [26] [27] [28] [29] [30] . To our knowledge, fully remote randomized controlled trials have not been conducted with people with SMI [24]." METHODS 3a) Description of trial design (such as parallel, factorial) including allocation ratio 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 3a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 3b-i) Bug fixes, Downtimes, Content Changes "The first objective of the study was to evaluate whether individuals with SMI can be recruited, assessed, and engaged in a digital intervention successfully in a fully remote clinical trial. The second objective of the study was to evaluate the clinical effectiveness of the intervention, CORE; a smartphone app designed to challenge dysfunctional thoughts that underlie common symptoms of SMI, self-stigmatizing attitudes, and maladaptive beliefs that impede treatment seeking and recovery. " "This study involved a fully remote, randomized controlled trial with a crossover design. " Not relevant for this study. essential Bug fixes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug fixes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have influenced study design such as staff changes, system failures/downtimes, etc. [2]. Does your paper address subitem 3b-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 4a-i) Computer / Internet literacy Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clarified. "Participants were eligible if they could speak English; were over age 18; lived in the United States; self-reported a diagnosis of schizophrenia or schizoaffective disorder, bipolar disorder, or major depressive disorder; and owned a smartphone with a data plan. They were excluded if they had previously participated in the study or were unavailable for 60 days of participation. " subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 4a-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email confirmation, phone calls) were used to detect/prevent these. Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 4a-iii) Information giving during recruitment Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results. "Online recruitment was conducted through ads on Google and Facebook. " 4b) Settings and locations where the data were collected subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 4a-iii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise. "Online recruitment was conducted through ads on Google and Facebook." "Individuals who clicked on the ads were directed to the study website. The study website provided written and video descriptions of the project, a downloadable version of the study consent form, and an option to complete a screening questionnaire via a "see if I am eligible" button. If eligible, participants were again presented with the consent form, and required to answer questions demonstrating their understanding of study details. If they answered these questions correctly within three attempts, they could proceed to the baseline assessments. Participants were excluded if they were unable to complete this step successfully as they were deemed unable to provide informed consent. Participants completed a battery of self-report questionnaires at baseline online and were randomized to either an active intervention or waitlist control group. Those randomized to the active intervention arm were given instructions on how to download and install the CORE mobile app." subitem not at all important 1 2 3 4 5 essential 5) The interventions for each group with sufficient details to allow replication, including how and when they were actually administered subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 4b-ii) Report how institutional affiliations are displayed Report how institutional affiliations are displayed to potential participants [on ehealth media], as affiliations with prestigious hospitals or universities may affect volunteer rates, use, and reactions with regards to an intervention.(Not a required item -describe only if this may bias results) Does your paper address subitem 4b-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners Mention names, credential, affiliations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Conflict of interest" section or mentioned elsewhere in the manuscript). "Participants completed a battery of self-report questionnaires at baseline online " subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-ii) Describe the history/development process Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results. Does your paper address subitem 5-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b). subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-iii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-iv) Quality assurance methods Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable. Does your paper address subitem 5-iv? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scientific reporting. Does your paper address subitem 5-v? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login. Does your paper address subitem 5-vi? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of specific group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi). Does your paper address subitem 5-vii? * Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study After completing baseline questionnaire and being randomized participants "were given instructions on how to download and install the CORE mobile app. " "Participants were incentivized by being entered into lotteries to win $500 and $1,000 after completing the 30-day and 60-day assessments, respectively. " subitem not at all important 1 2 3 4 5 essential 5-ix) Describe use parameters Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum. Does your paper address subitem 5-ix? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the "The CORE intervention is a smartphone application that uses the GGtude platform, a system designed to increase the cognitive flexibility of individuals struggling with a range of mental health problems through brief daily trainings [44] [45] [46] [47] . CORE was specifically designed to help counteract dysfunctional thoughts in multiple domains that are relevant to the subjective experience of having SMI. The intervention consists of daily brief game-like exercises designed to produce changes in the relative activation of adaptive and maladaptive beliefs about the self, others, and the world such that adaptive beliefs would be more easily retrieved than maladaptive ones. CORE users are trained to respond to multiple statements in a sequence of modules that progress through the following domains: Self-Talk, Belief in Change, Self-Stigma, Self-Care, Self-Worth, Illness and Identity, Personal Strength, Social Avoidance, Feelings vs. Facts, Catastrophization, Thoughts of Reference, Paranoid Ideation (i.e. "people are against me"), Treatment Seeking, and Recovery. Modules begin with brief psychoeducation about the target domain and how maladaptive beliefs can hamper recovery. Thoughts appear as statements on the smartphone screen and users are required to either endorse (i.e., "drag" the statement down toward them on the touchscreen) or discard them (i.e., "push" the statement upwards away from them). Users learn to embrace self-statements reflecting more nuanced adaptive thoughts (e.g., belief in change, importance of self-care, alternative explanations to threat perceptions, value of treatment seeking)." subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability). Does your paper address subitem 5-x? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-xi) Report any prompts/reminders used Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21 -generalizability). Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Module content is organized into 53 levels and participants were recommended not to complete more than 3 levels a day, for a period of 30 days. Push notifications reminded users to complete their daily training. After each level, participants have the option of adding one of three positive statements that they most relate to, to their personal toolbox. Participants can access their toolbox at any time. " subitem not at all important 1 2 3 4 5 essential 6a) Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed 5-xii) Describe any co-interventions (incl. training/support) Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1] . It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21 -generalizability. Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study We did not offer any co-interventions. subitem not at all important 1 2 3 4 5 essential 6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed If outcomes were obtained through online questionnaires, describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed [9]. Does your paper address subitem 6a-i? Copy and paste relevant sections from manuscript text Your answer "Following enrollment, participants were randomized to either receive the intervention (i.e., CORE app) or waitlist control group in blocks of 4. In the active intervention group, participants completed baseline assessment and were then immediately given access to the CORE app for 30 days of use. After a month they concluded the intervention, uninstalled the app, and completed a second assessment. After an additional month (at 60 days), they completed a third assessment to measure stability of symptom change post-intervention. In the control arm, participants waited 30 days to receive the CORE app. After a month they completed a second assessment and were provided access to the CORE app. After an additional month (at 60 days), they completed a third assessment, to measure within-subjects changes. " "At all assessments, we administered measures of depression ( Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Describe whether and how expected attrition was taken into account when calculating the sample size. Does your paper address subitem 7a-i? Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study There were no changes to trial outcomes after the trial commenced. There were no interim analyses or stopping guidelines. 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 11a-i) Specify who was blinded, and who wasn't Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering co-interventions (if any). Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study One long randomization sequence was generated at the beginning of the study and participants were sequentially assigned based on that. Participants would not know what group they were assigned prior to completing the baseline survey. The random allocation sequence was generated by the software/development team using block randomization. Participants were enrolled via online ads and assigned to interventions as per the randomization obtained. subitem not at all important 1 2 3 4 5 essential 11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator". Does your paper address subitem 11a-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study No blinding was used in this study. Not relevant for this study. subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 12a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 12a-i) Imputation techniques to deal with attrition / missing values Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]). Does your paper address subitem 12a-i? * "Means, standard deviations, and frequencies were reported using descriptive statistics for all participant demographics, symptom measures, and acceptability/usability scale items. A series of t and χ2 tests were used to test for group differences in demographic variables and symptoms measures at baseline (T1) before examining intervention effects at 30-days (T2) and 60-days (T3). Preand post-scores for both groups were explored using a series of repeated measures analysis of variance (ANOVA). The intervention effect between groups at T1 and T2 was initially tested using a 2 x 2 mixed-design repeated measures ANOVA. Following, a 1 x 3 repeated measures ANOVA tested the performance of the active group across T1, T2, and T3 with post-hoc Bonferroni pairwise comparisons used to explore outcomes when significant differences occurred. Finally, a series of 1 x 3 repeated measures ANOVAs tested the intervention effect after crossover (T2 and T3) for the waitlist control participants to see if intervention effects were replicated. A series of equivalence tests were done using independent sample t-tests to determine if the magnitude of intervention effects differed for the active group participants between T1 and T2 as compared to the intervention effect for waitlist control participants between T2 and T3. Spaghetti plots are provided to visualize change over time across symptom measures in both groups. In order to minimize bias during analysis, we used an intent-to-treat approach and implemented a multiple imputation strategy to replace missing values [48] . Multiple imputation is considered the most appropriate method to handle missing data for a study of this size and has been used successfully in repeated measure designs [49, 50] . Sensitivity analyses using pattern-mixture models were used to evaluate whether missing data were missing at random and biased analysis prior to imputation [51]. " 12b) Methods for additional analyses, such as subgroup analyses and adjusted analyses X26) REB/IRB Approval and Ethical Considerations [recommended as subheading Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 12b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Sensitivity analyses using pattern-mixture models were used to evaluate whether missing data were missing at random and biased analysis prior to imputation [51] . " "Means, standard deviations, and frequencies were reported using descriptive statistics for all participant demographics, symptom measures, and acceptability/usability scale items. A series of t and χ2 tests were used to test for group differences in demographic variables and symptoms measures at baseline (T1) before examining intervention effects at 30-days (T2) and 60-days (T3). Preand post-scores for both groups were explored using a series of repeated measures analysis of variance (ANOVA). The intervention effect between groups at T1 and T2 was initially tested using a 2 x 2 mixed-design repeated measures ANOVA. Following, a 1 x 3 repeated measures ANOVA tested the performance of the active group across T1, T2, and T3 with post-hoc Bonferroni pairwise comparisons used to explore outcomes when significant differences occurred. Finally, a series of 1 x 3 repeated measures ANOVAs tested the intervention effect after crossover (T2 and T3) for the waitlist control participants to see if intervention effects were replicated. A series of equivalence tests were done using independent sample t-tests to determine if the magnitude of intervention effects differed for the active group participants between T1 and T2 as compared to the intervention effect for waitlist control participants between T2 and T3. Spaghetti plots are provided to visualize change over time across symptom measures in both groups. In order to minimize bias during analysis, we used an intent-to-treat approach and implemented a multiple imputation strategy to replace missing values [48] . Multiple imputation is considered the most appropriate method to handle missing data for a study of this size and has been used successfully in repeated measure designs [49, 50] . Sensitivity analyses using pattern-mixture models were used to evaluate whether missing data were missing at random and biased analysis prior to imputation Does your paper address subitem X26-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 13b) For each group, losses and exclusions after randomisation, together with reasons Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) Does your paper address subitem X26-iii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 13b-i) Attrition diagram Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other figures or tables demonstrating usage/dose/engagement. Does your paper address subitem 13b-i? Copy and paste relevant sections from the manuscript or cite the figure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study " Figure 3 . Consolidated Standards of Reporting Trials (CONSORT) diagram." outlines the number of participants in each grouping. "Online recruitment ads were placed between January 2020 to September 2020" In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known. Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 16-i) Report multiple "denominators" and provide definitions Report multiple "denominators" and provide definitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at specific pre-defined time points of interest (in absolute and relative numbers per group). Always clearly define "use" of the intervention. Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate " 17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval) direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 16-ii) Primary analysis should be intent-to-treat " Figure 3 . Consolidated Standards of Reporting Trials (CONSORT) diagram." has a full list of Ns who were exposed, consented and randomized. "We were able to collect CORE app use data from a total of 162 (51.43%) participants. Of these participants, 82 (49.1%) completed all 53 intervention levels while participants, on average, completed 35 levels." 17a-i) Presentation of process outcomes such as metrics of use and intensity of use In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational definitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "Between-Group Differences: Active Group Versus Waitlist Control at T2 Analyses of differences between the active group and the waitlist control group at T2 revealed a significant treatment x time interaction on the BDI-II (F1,313 = 13.38, p<.001), GAD-7 (F1,313 = 5.87, p<.05), RAS (F1,313 = 23.42, p<.001), RSES (F1,313 = 19.28, p<.001), and SDS (F1,313 = 10.73, p<.01). This indicates that participants engaging in the active condition showed improvements on these outcome measures after 30 days of using the application as compared to participants in the waitlist control who did not have access to the app over the same period ( Figure 4) . Table 3 displays means and standard deviations for all measures at each time point for the active group participants and the waitlist control participants. Within-Group Change: Active Group at T2 and Maintenance at T3 A series of 1 x 3 repeated measures ANOVA were used to examine intervention effects within the active group between T1 and T2 and whether these effects were maintained 30 days following discontinuation of app use ( . The lack of change on HPSVQ scores after 30 days of CORE use was also replicated (F2, 322 = 1.14 , p>.05). Post-hoc analysis reveal that significant improvements took place for the waitlist control group between T2 and T3 for all outcome measures except the HPSVQ. Improvements for the waitlist control group were also found between T1 and T2 for the GAD-7, GPTS, and SDS. These improvements as a result of time were sufficient to negate a between-subject effect for the GPTS but not for the BDI-II or the GAD-7. The magnitude of change observed following 30 days of CORE use between groups was equivalent across all outcome metrics except for the RAS (t105= 2.28, p<.05), with the active group showing a slightly larger improvement than the waitlist control group." subitem not at all important 1 2 3 4 5 essential 17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended 18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory "average session length". These must be accompanied by a technical description how a metric like a "session" is defined (e.g., timeout after idle time) [1] (report under item 6a). Does your paper address subitem 17a-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study We did not have any binary outcomes. A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii). "Between-Group Differences: Active Group Versus Waitlist Control at T2 Analyses of differences between the active group and the waitlist control group at T2 revealed a significant treatment x time interaction on the BDI-II (F1,313 = 13. . This indicates that participants engaging in the active condition showed improvements on these outcome measures after 30 days of using the application as compared to participants in the waitlist control who did not have access to the app over the same period (Figure 4) . Table 3 displays means and standard deviations for all measures at each time point for the active group participants and the waitlist control participants. Within-Group Change: Active Group at T2 and Maintenance at T3 A series of 1 x 3 repeated measures ANOVA were used to examine intervention effects within the active group between T1 and T2 and whether these effects were maintained 30 days following discontinuation of app use ( Does your paper address subitem 22-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "This paper reports on the first fully remote randomized controlled trial of a smartphone intervention involving people with more severe forms of psychopathology, including schizophrenia and bipolar disorder. Building on and expanding online enrollment strategies used in previous work [52,53], we were able to reach, recruit, randomize, treat, and assess a socio-demographically diverse sample of participants from 45 states, far exceeding the reach of traditional localized study recruitment approaches and at a fraction of the cost. " "The study findings suggest that the detected effects were a product of the mobile intervention and were not merely a byproduct of participation in research. The waitlist-controlled study design enabled evaluation of both between-group effects during the intervention period and within-group changes in both groups over time. The results demonstrated a link between the timing of participants' exposure to the treatment app and when they experienced significant changes on key clinical and functional measures. A combination of factors such as simplicity, brevity, daily use, and game-like interactions may have encouraged use of the CORE app. Daily practice comprising identification and categorization of self-statements, repeated exposure to adaptive self-statements, and psychoeducation may have facilitated retrieval of adaptive beliefs over maladaptive ones, thereby reducing the severity of symptoms [56]. " Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations Does your paper address subitem 21-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other co-interventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting. Does your paper address subitem 21-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study In addition to the usual declaration of interests (financial or otherwise), also state the relation of the study team towards the system being evaluated, i.e., state if the authors/evaluators are distinct from or identical with the developers/sponsors of the intervention. Does your paper address subitem X27-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study As a result of using this checklist, did you make changes in your manuscript? * "Dr. Ben-Zeev has an intervention content licensing agreement with Pear Therapeutics and has a financial interest in FOCUS technology. He has consulted to Trusst Health, eQuility, and Otsuka Pharmaceuticals Ltd. Dr. Doron is the cofounder of GGtude Ltd. and has a financial interest in the CORE app described in this paper. Dr. Doron did not have access to the usability, acceptability, or clinical outcomes data during the study or during the preparation of this paper; data analyses were conducted by members of the team who were unaffiliated with GGtude Ltd. All remaining authors have no conflicts to disclose." STOP -Save this form as PDF before you click submit To generate a record that you filled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it. When you submit your (revised) paper to JMIR, please upload the PDF as supplementary file. Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you! role of funders Does your paper address CONSORT subitem 23? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address CONSORT subitem 25? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded. Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to non-use of the intervention/usability issues, biases through informed consent procedures, unexpected events.Does your paper address subitem 20-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The study has several limitations. The opportunity to participate was available only to individuals who use the internet (i.e., in order to receive study recruitment ads) and who own smartphones (i.e., to download the CORE app) and may therefore not be representative of the full range of people with SMI. Similar to fully-remote digital mental health trials involving participants with less severe psychopathology [28] our study sample was over-represented by female participants (84%) and study dropout rates were relatively high. All screening and assessment questions relied on individuals' self-report but were not corroborated by a trained clinician or additional documentation (e.g., EHR data). Further, while our measure of recovery taps "willingness to ask for help" [35,36] we did not formally evaluate changes in treatment seeking behavior, an area of focus in the CORE training program and key variable of interest. "