key: cord-0426440-e4vwqa2l authors: Morris, A. C.; Ibrahim, Z.; Heslin, M.; Moghraby, O. S.; Stringaris, A.; Grant, I. M.; Zalewski, L.; Pritchard, M.; Stewart, R.; Hotopf, M.; Pickles, A.; Dobson, R. J. B.; Simonoff, E.; Downs, J. title: Assessing the Feasibility of a Web-based Outcome Measurement System in Child and Adolescent Mental Health Services: myHealthE (MHE) a Randomised Controlled Feasibility Pilot Study date: 2021-10-11 journal: nan DOI: 10.1101/2021.10.07.21264418 sha: cf4f8a021f020de3ab89ef77229557cf9da5d0cb doc_id: 426440 cord_uid: e4vwqa2l Background: Interest in internet-based patient reported outcome measure (PROM) collection is increasing. The NHS myHealthE (MHE) web-based monitoring system was developed to address the limitations of paper-based PROM completion. MHE provides a simple and secure way for families accessing Child and Adolescent Mental Health Services to report clinical information and track their childs progress. This study aimed to assess whether MHE improves the completion of the Strengths and Difficulties Questionnaire (SDQ) compared with paper collection. Secondary objectives were to explore caregiver satisfaction and application acceptability. Methods: A twelve-week single-blinded randomised controlled feasibility pilot trial of MHE was conducted with 196 families accessing neurodevelopmental services in south London to examine whether electronic questionnaires are completed more readily than paper-based questionnaires over a 3-month period. Follow up process evaluation phone calls with a subset (n=8) of caregivers explored system satisfaction and usability. Results: MHE group assignment was significantly associated with an increased probability of completing an SDQ-P in the study period (adjusted hazard ratio, (HR) 12.1, 95% CI 4.7-31.0; p= <0.001). Of those caregivers who received the MHE invitation (n=68) 69.1% completed an SDQ using the platform compared to 8.8% in the control group (n=68). The system was well received by caregivers, who cited numerous benefits of using MHE, for example, real time feedback and ease of completion. Conclusions: MHE holds promise for improving PROM completion rates. Research is needed to refine MHE, evaluate large scale MHE implementation, cost effectiveness and explore factors associated with differences in electronic questionnaire uptake. • Paperless monitoring systems (i.e., digital) as an alternative to traditional outcome measure delivery and collection are growing in healthcare settings. • Remote questionnaire completion using the myHealthE (MHE) system is feasible and acceptable to caregivers of children accessing CAMHS in South London, yielding a 60.3 % increase in Strengths and Difficulties questionnaire reporting compared to standard practice. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Introduction Patient-reported outcome measures (PROMs) enable standardised and direct collection of a patient's perceived health status (Devlin & Appleby, 2010; NHS England, 2019) . Used routinely, PROMs are recognised as a clinically valuable method to measure patient or caregiver rated symptoms, assess intervention success, and encourage shared patient and practitioner communication and decision making (Carlier, Meuldijk, Van Vliet et al., 2012; Lambert, Whipple, Hawkins et al., 2003; Soreide & Soreide, 2013) . Child and Adolescent Mental Health Services (CAMHS) in England are encouraged to collect information about young people's presenting problems at entry to CAMHS and again within six months of receiving treatment (Department of Health (DoH), 2004 & DoH, 2015 using patient PROMs. However, audit and survey studies demonstrate low guideline adherence, suggesting that CAMHS struggle to implement PROMs (Batty, Moldavsky, Foroushani et al., 2013; Hall, Moldavsky, Baldwin et al. 2013; Johnston & Gowers 2005) . Recent research investigating the electronic health records of 28,000 patients accessing . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 6 CAMH services across South London identified paired use of the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997), a readily available behavioural screening tool at baseline and follow up in only 8% of these (Morris, Macdonald, Moghraby et al., 2020) and as few as 1% within specific clinical groups, namely attention deficit hyperactivity disorder (ADHD) (Cruz, Simonoff, McGough et al., 2015) . Data collection using traditional paper questionnaires is associated with several timeand resource-intensive steps, including printing, postage and processing returned outcome measures. Although paper questionnaires are practical and easy to complete, already-burdened clinicians struggle with the administrative effort required to capture paper-based questionnaires (Boswell, Kraus, Miller et al., 2015; Johnston and Gowers, 2005; Hall, Taylor, Moldavsky et al, 2014) . Response data are also easily comprised, for example, users can omit questions, select multiple responses per item, and mark outside the questions tick box margins, leading to missing or unusable data (Ebert, Huibers, Christensen et al., 2018) . A rapid rise in internet use has paved the way for electronic questionnaires (Lyon, Lewis, Boyd et al., 2016) . Electronic PROMs (ePROMs) are reported to be less time consuming (Cella, Hanh, Jensen et al., 2015) , require fewer administrative duties (Black, 2013; Coons, Eremenco, Lundy, et al., 2015; Eremenco, Coons, Paty et al., 2014) , cost less (Zuidgeest, Hendriks, Koopman et al., 2011) and evoke more honest (Black & Ponirakis, 2000) and less erroneous responses by prompting patients to respond to all items within a questionnaire and only provide one response . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Evaluating the Feasibility of myHealthE in Child and Adolescent Mental Health Services 7 per question (Dillon, Pirie, Rice et al., 2014; Jamison, Raymond, Levine et al., 2001; Coons et al, 2015; Eremenco et al, 2014) . Feasibility trials of web-based monitoring systems report positive outcomes relating to patient engagement, satisfaction, and clinical value (Schepers, Sint Nicolaas, Maurice-Stam et al., 2017; Barthel, Fischer, Nolte et al., 2016; Ashley, Jones, Thomas et al., 2013 , Nordan, Blanchfield, Niazi et al., 2018 . However, less research is available on the development and application of ePROM systems in CAMHS. Interviews with mental health service users demonstrate positive attitudes toward the use of technology to assist traditional care (Borzekowski, Leith, Medoff et al., 2009 ). However, patients have highlighted barriers to web-based portal acceptability, including computer literacy, perceived usefulness, suitability, confidentiality, feedback and the effect application use has on their capacity to manage their condition and therapeutic relationships (Niazkhani, Toni, Cheshmekaboodi et al., 2020) . The myHealthE (MHE) system was built to enable remote PROM monitoring in CAMHS. This system aims to automate the communication, delivery and collection of ePROMs at predefined post-treatment periods, providing caregivers with a safe and engaging way to share clinically relevant information about their child with their allocated care team with minimal human input. MHE architecture, development and implementation methodology have been described previously (Morris, Ibrahim, Moghraby, 2021). Novel healthcare applications require feasibility and acceptability testing to ensure that the technology is understandable and can be used . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Evaluating the Feasibility of myHealthE in Child and Adolescent Mental Health Services 8 successfully by the target end-user in real-world clinical surroundings before conducting a large-scale system evaluation (Steele Gray, Gill, Khan et al., 2016a) . As described in our protocol [(ISRCTN) 22581393], the primary purpose of the trial was to understand whether MHE use should be assessed in CAMHS on a wider scale. Therefore, we conducted a feasibility pilot study of MHE to evaluate whether introducing MHE enhanced completion of PROM questionnaires over the course of CAMHS treatment compared to standard data collection procedures, as measured by a count of how many electronic questionnaires were completed relative to paper questionnaires over a three-month period. Secondly, we aimed to assess caregiver satisfaction with the system via individual caregiver phone consultations. Given resource constraints we were unable to assess the economic benefit of MHE compared to standard data acquisition as per our protocol. We hypothesised that MHE implementation would afford a substantial increase in completed standardised caregiver-reported follow up data and caregiver satisfaction with CAMHS services compared to routine data collection. The current study comprised a single-blinded parallel group feasibility pilot randomised control trial (RCT) of MHE. Outcome, sociodemographic and service level data were obtained from the Clinical Record Interactive Search (CRIS) system. CRIS contains de-identified medical record history from the South London and . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The trial was conducted at Kaleidoscope, a community paediatric and children's and Caregivers were recruited if they had available contact details (mobile phone number and/or email address) in their child's EHR. The MHE data collection process was directly comparable to current paper-based practice, except for its electronic basis as such, the system only collects data which is ordinarily requested from families by their treating clinical team and its use does not instigate change in patient care. Therefore, caregivers did not have to provide informed consent to participate in this . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Evaluating the Feasibility of myHealthE in Child and Adolescent Mental Health Services 10 trial. Instead, caregivers could choose to opt-out via email or phone call to the trial research assistant (ACM). Recruitment was achieved through SLaM EHR screening. A SQL script developed and implemented by a senior member of the SLaM Clinical Systems Team automatically generated an extract of eligible patients which was then provided to the research team. Subsequently, computerised condition allocation and simple randomisation was performed to assign eligible caregivers to either receive PROM outcome monitoring as usual (MAU; control group) or enrolment to the MHE platform (intervention group) on a 1:1 basis. Clinicians were blinded to condition allocation, whereby they were not informed which patients on their case load had been allocated to received MHE or MAU. The primary outcome variable was the number follow-up caregiver reported SDQ (SDQ-P; electronic vs paper SDQ-P) forms completed over the three-month observation period. The SDQ-P (Appendix 1) is a structured 25-item questionnaire screening for symptoms of childhood emotional and behavioural psychopathology (Goodman, 1997) . SLaM holds a sub-licence to use the SDQ to support clinical service via NHS Digital Copyright Licensing Service. It is current clinical practice to collect SDQ-P for young people, either by post before their first face-to-face meeting, or on site during a clinical appointment to inform their baseline assessment and again six months after starting treatment or upon discharge from CAMHS. Other variables extracted from CRIS are presented in online supplement 1. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint To evaluate usability, from a subset of caregivers randomly assigned to MHE a convenience sample of 6 caregivers who had engaged with MHE and 2 caregivers who had not were contacted by telephone. These caregivers were asked to complete the System Usability Scale (SUS; Brooke, 1996) to examine subjective usability. SUS comprises 10 statements reported on a 5-point Likert scale ranging from strongly disagree to strongly agree. The total score is presented as a figure from 0 to 100, with a greater score reflecting higher usability. Mean SUS score was computed and ranked using Bangor, Kortum and Miller's (2008) acceptability scale defined as 'not acceptable', 'marginal' and 'acceptable'. Following administration of the SUS, caregivers were invited to ask questions about the platform or provide any further comments about their experience of using MHE. The current trial aimed to inform the development of a larger, adequately powered RCT by providing precise estimates of acceptability and feasibility, in addition to outcome variability. A threshold of clinical significance was decided a priori to be 15% between MAU and MHE groups for SDQ-P completion within three-months, based on consensus from Kaleidoscope staff and previous research indicating an expected baseline completion rate of 8% SDQ-P in the control group (Morris et al, 2020). For a fixed sample size design, the sample size required to achieve a power of 1-β = 0.80 for the two-tailed chi-square test at level α = 0.05, under the prior . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint 12 assumptions, was 2 × 91 = 182 on a 1:1 allocation ratio. The power calculation was carried out using Gpower 3.1.7. To increase power and reduce the risk of chance imbalance between MHE and non-MHE groups, we followed recent guidance on covariate adjustment within RCTs of moderate sample size provided with an patient information sheet and MHE information leaflet (Appendix 7a and 7b). After group assignment, caregivers allocated to receive MHE were contacted with an invitation to set up a personalised web-portal and complete an SDQ-P (caregivers were enrolled to the trial irrespective of whether they registered their MHE account if baseline SDQ information was present in their child's EHRs). Caregivers who did not register were prompted every week to enrol and complete an SDQ-P. Once an online questionnaire was completed, they were contacted monthly to provide new SDQ data. In the control group caregivers were requested to complete paper SDQ-P face-to-face or by post according to clinician discretion. Apart from electronic SDQ-P completion for the intervention group, treatment . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10. 1101 /2021 Evaluating the Feasibility of myHealthE in Child and Adolescent Mental Health Services 13 remained the same for all participants. Information collected through MHE was stored in the child's EHR and managed in the same way as all other confidential information. SDQ-P data was checked daily by ACM and promptly entered to the patient's EHR. Post intervention, all participants received a letter thanking them for their participation. All analyses were conducted using STATA version 14 (StataCorp, 2015) . Analyses were conducted to determine differences in the primary outcome, namely paper relative to electronic follow-up SDQ-P completion. Analysis was performed subject to intention-to-treat like principles (intention-to-contact), whereby all participants were analysed according to their initially assigned intervention arm, irrespective of protocol adherence or deviations. Cox regression was used to examine the relationship between monitoring as usual vs MHE group assignment and SDQ-P completion rates. Using a Kaplan-Meier curve, we checked whether group assignment (as predictor) satisfied the proportional hazards assumption. Our first analysis examined the association between treatment group only and SDQ-P completion. The second model adjusted for demographic and clinical covariates captured in this trial. An inverse Kaplan-Meier curve was plotted to visualise the probability of SDQ-P completion, comparing caregivers who completed electronic and paper SDQ-P. For the intervention group the MHE website-SDQ-P completion conversion rate was . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint reported as a percentage by measuring the number of caregivers that register on MHE and subsequently completed a follow-up SDQ-P. Within study participant flow and data collection rates are provided in Figure 2 . A total of 342 caregivers were screened for eligibility of which (n=196) met the inclusion criteria. During eligibility screening caregiver contact information was often missing or located in an area of the patients' EHRs different from expected, therefore manual contact detail collection was carried out to enable digital communication via MHE. Caregivers were enrolled and randomly assigned to the intervention group (MHE n=98) and the control group (MAU n=98). Of caregivers assigned to MHE and MAU 30 (36.3%) did not receive notifications from MHE. The conversion rate from account registration to SDQ completion was 98% (47/48). Online supplement 2 outlines account registration issues and opt-out preferences reported by caregivers. Table 1 . presents sociodemographic and service characteristics for the whole sample. Participants were ethnically diverse, predominantly male and at the older end of the age range accepted by CAMHS. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint During the trial 47 caregivers [47.9% of intention-to-contact (total n=98), 69.1% of actually contacted (total n=68)] registered an account on the MHE platform and completed at least one follow-up SDQ-P. In the corresponding timeframe 6 (intention to contact = 6% (n=98) and actually contacted = 8.8% (n=68) caregivers assigned to receive MAU completed at least one follow-up SDQ-P. Second follow-up was due for 43 of the MHE cohort by the end of the study period (at least one month had elapsed since completing their first online SDQ-P) and of these 31 caregivers completed this (72%). Overall, 87 follow-up SDQ-Ps were completed via the MHE platform: Figure 3 . provides a breakdown of SDQ-P completion within each 7-day notification reminder period. The ITC Cox regression models are presented in Table 2 , and graphically depicted in Figure 4 . MHE group assignment was significantly associated with an increased probability of completing an SDQ-P in the study period (adjusted hazard ratio, (HR) 12.1, 95% CI 4.7-31.0; p= <0.001). This was observed after controlling for potentially confounding socio-demographic characteristics and clinical factors including, gender, age at the start if the trial, baseline CGAS and SDQ profiles, co-morbid ADHD, learning disability, and emotional disorders as well as number of days of active care . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Evaluating the Feasibility of myHealthE in Child and Adolescent Mental Health Services 16 and attended face-to-face events. No significant interaction was found between ethnic status (white and non-white ethnic groups) and SDQ-P completion by group. A total of eight SUS questionnaires and usability interviews were completed. The mean SUS score for users of the website was 78/100 indicating that the application was 'acceptable' to users. Figure 5 . provides a summary of caregiver's comments regarding MHE. This trial showed that the collection of electronic PROMs using web-based technology is feasible and beneficial to CAMHS practice. Caregiver's who received regular invitations to register for MHE had a SDQ-P completion rate of 69% compared to 12% paper-based SDQ-P completion. These finding represents a 61% absolute improvement, or 12 fold increase, of paired SDQ-P collection rates (Morris et al., 2020). By automating unassisted delivery of PROMs with completion prompts at specified time points, MHE addresses several fundamental challenges inherent to paper-based information gathering in busy clinical settings, such as processing . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. In post-trial interviews caregivers rated MHE as 'acceptable', suggesting good levels of usability. Many caregivers favoured the ease and speed of using MHE to complete outcome measures compared to paper-based methods, while barriers included how readily information provided through the platform was used by clinicians to identify children with worsening symptoms and data privacy concerns. However, only a small number of caregivers were contacted to provide their views on the system; therefore, it is possible that other undetected usability issues influenced the results of this trial, for example: language, literacy level, disability, and cultural sensitivity difficulties (Hargittai, 2002) . This was the case for several caregivers who reported that their limited information technology capabilities and knowledge making it hard to navigate MHE without assistance from family members. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10. 1101 /2021 This disparity may deepen as digital platforms are increasingly integrated into routine clinical practice (Van Dijk, 2005) and should be iteratively considered during the design and implementation of emerging digital health platforms, paying particular attention to the role of co-design (Andersen, 2019). This trial was conducted in a naturalistic manner independent of clinical practice to ensure that clinician's behaviour, for example, promoting MHE use did not inflate observed rates of engagement. Moreover, the research was conducted in a sociodemographically diverse geographical area, resulting in a broad range of caregivers testing the system. Finally, condition allocation was computerised meaning that all participants were instantly allocate to either receive MAU or MHE. Therefore, it was unlikely that allocation bias would have influenced the trial findings. Limitations include the fact that families only had the opportunity to enrol to the trial if they had a baseline SDQ present in their child's EHR, which relies on this being initiated by a clinician in the first instance. In the future, using MHE to capture baseline and follow up SDQ-P data may afford a more realistic assessment of ePROM feasibility. It is also possible that neurodevelopmental team service users perceived the SDQ-P as less useful than a disorder specific questionnaire, which may have resulted in lower rates of completion level. Lastly, owing to resource constraints phone interviews were conducted after the trial ended meaning that responses could be influenced by recall bias. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. Collecting data from a larger number of caregivers will enable us to explore the effects of various patient factors on ePROM engagement. Research investigating differential uptake in PROM collection suggests that several patient characteristics including ethnicity and social deprivation are associated with inequitable PROM use (Latulippe Hamel & Giroux, 2017 , Morris et al., 2020 . While this was not the case in the current small-scale trial, it is essential that further research is conducted to determine whether these systems sustain possible health inequalities with larger sample sizes. In-depth interviews are needed to explore how ePROM platforms can be adapted to meet different service user needs and provide more general insights into caregivers' reasons for deciding to complete or not complete electronic questionnaires, their views on the concept, design and delivery of MHE, the barriers and facilitators for MHE implementation and identify potential harms and study protocol refinement (e.g., platform design and frequency of questionnaire completion). System . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Evaluating the Feasibility of myHealthE in Child and Adolescent Mental Health Services 20 refinements are also required to enable alternative methods for acquiring and inputting caregiver contact information to circumvent the difficulties encountered with automatic data extraction in this study. Routine PROM collection is essential for delivering personalised health services that reflect clinical need from the perspective of young people and their families. This study supports the feasibility of a remote PROM monitoring platform within a realworld outpatient setting providing treatment to a demographically diverse population. Intimating that web-platforms may provide an acceptable and convenient method to maintain and scale up improved patient monitoring, service-user communication, and service evaluation. A future multisite trial of MHE is required to evaluate this esystem at scale. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint source electronic data capture system offered increased accuracy and costeffectiveness compared with paper methods in Africa. Journal of clinical epidemiology, 67 (12), 1358 -1363 . https://doi.org/10.1016 /j.jclinepi.2014 .06.012 Downs, J., Ford, T., Stewart, R., Epstein, S., Shetty, H., Little, R., et al. (2019 . An approach to linking education, social care and electronic health records for children and young people in South London: A linkage study of child and adolescent mental health service data. BMJ Open, 9, e024355. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 research practices task force. Value Health, 17, 501-16.doi:10.1016 /j.jval.2014 .06.005 Fernandes, A. C., Cloete, D., Broadbent, M. T., Hayes, R. D., Chang, C. K., Jackson, R. G., Roberts, A., Tsang, J., Soncul, M., Liebscher, J., Stewart, R., & Callard, F. (2013 . Development and evaluation of a de-identification procedure for a case register sourced from mental health electronic records. BMC Medical Informatics and Decision Making,13, 71. doi: 10.1186 /1472 -6947-13-71. Goodman, R. (1997 . The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581-586. doi:10.1111 /j.1469 -7610.1997 .tb01545.x Hall, C. L., Moldavsky, M., Baldwin, L., Marriott, M., Newell, K., Taylor, J., et al. (2013 . The use of routine outcome Measures in two child and adolescent mental health services: A completed audit cycle. BMC Psychiatry, 13, 270. doi: 10.1186 /1471 Hall, C. L., Taylor, J., Moldavsky, M., Marriott, M., Pass, S., Newell, K., et al. (2014) . A qualitative process evaluation of electronic session-by-session outcome measurement in child and adolescent mental health services. BMC Psychiatry, 14, 113. doi:10. 1186 /1471 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Hargittai CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint a. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint Mean attended F2F events (SD) 6.0 (10.2) 9.1 (21.0) . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint Table 2 . An Intention to contact Cox-regression analysis of the relationship between electronic compared to paper-based SDQ-P assignment and SDQ-P completion rates (n=195), adjusted model taking into account participant characteristics. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Smartphone Ownership Is Growing Rapidly Around the World, but Not Always Equally Psychosocial Assessment Tool in pediatric cancer care A children's global assessment scale Stata Statistical Software: Release 14. College Station, TX: StataCorp LP The electronic patient reported outcome tool: Testing usability and feasibility of a mobile app and portal to support care for patients with complex chronic disease and disability in primary care settings . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 Appendix 3a) MHE invitation email; 3b) MHE reminder email. a. . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint Appendix 4) MHE login page -contact the corresponding author to request access to this image.Appendix 5) MHE home page (when questionnaire is due to be completedcontact the corresponding author to request access to this image.. CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101/2021.10.07.21264418 doi: medRxiv preprint a Covariate dropped due to <5 cell size value . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021 49 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted October 11, 2021. ; https://doi.org/10.1101 https://doi.org/10. /2021