key: cord-0714344-cx5lu4el authors: Békés, Vera; Aafjes‐van Doorn, Katie; Zilcha‐Mano, Sigal; Prout, Tracy; Hoffman, Leon title: Psychotherapists' acceptance of telepsychotherapy during the COVID‐19 pandemic: A machine learning approach date: 2021-11-22 journal: Clin Psychol Psychother DOI: 10.1002/cpp.2682 sha: 29128515f4419b952fc3037c0bbf4e81b8b8a3d1 doc_id: 714344 cord_uid: cx5lu4el OBJECTIVE: This study aimed to develop predictive models of three aspects of psychotherapists' acceptance of telepsychotherapy (TPT) during the COVID‐19 pandemic, attitudes towards TPT technology, concerns about using TPT technology and intention to use TPT technology in the future. METHOD: Therapists (n = 795) responded to a survey about their TPT experiences during the pandemic, including quality of the therapeutic relationship, professional self‐doubt, vicarious trauma and TPT acceptance. Regression decision tree machine learning analyses were used to build prediction models for each of three aspects of TPT acceptance in a training subset of the data and subsequently tested in the remaining subset of the total sample. RESULTS: Attitudes towards TPT were most positive for therapists who reported a neutral or strong online working alliance with their patients, especially if they experienced little professional self‐doubt and were younger than 40 years old. Therapists who were most concerned about TPT were those who reported higher levels of professional self‐doubt, particularly if they also reported vicarious trauma experiences. Therapists who reported low working alliance with their patients were least likely to use TPT in the future. Performance metrics for the decision trees indicated that these three models held up well in an out‐of‐sample dataset. CONCLUSIONS: Therapists' professional self‐doubt and the quality of their working alliance with their online patients appear to be the most pertinent factors associated with therapists' acceptance of TPT technology during COVID‐19 and should be addressed in future training and research. lifted. It is therefore important to understand which therapist, patient and treatment factors contribute to therapists' acceptance of the TPT format and their intention to continue providing therapy online. Various terms have been used to refer to remote therapy (Rochlen et al., 2004) , and in this study, based on the most recent literature, we use the term 'telepsychotherapy' to refer to psychotherapy sessions provided synchronously online via videoconferencing (Markowitz et al., 2020; Poletti et al., 2020; Rosen et al., 2020; Van Daele et al., 2020) . Preliminary studies have shown that despite the involuntary transition to TPT during the pandemic, therapists have a somewhat positive view of TPT (Feijt et al., 2020 , McBeath et al., 2020 and are inclined to use it in the future . In earlier studies, several factors have been proposed that may impact therapists' views of TPT (for a review, see Connolly et al., 2020) . One important factor is the perceived ability to develop a strong therapeutic relationship remotely (Roesler, 2017; Sucala et al., 2013) . Prepandemic studies showed that in fact, the therapeutic relationship is generally strong in TPT (see review by Norwood et al., 2018) and similar to in-person therapies (e.g. Bouchard et al., 2020; Watts et al., 2020) . This was true not only pre-pandemic but during the pandemic as well (Aafjes-van Doorn, Aafjes-van Doorn, Bekés, Prout, & Hoffman, 2020) . Moreover, during the pandemic, many therapists also reported that they were as authentic and genuine in their online sessions as they were before in in-person Békés, Grondin, & Bouchard, 2020) . Before the start of the pandemic, few therapists had attended training in TPT, and most felt uncomfortable using technology to provide psychotherapy (e.g. Essig & Russell, 2017; Glueckauf et al., 2018) . Therapists' sense of uncertainty regarding their ability to provide effective therapy online likely further increased during this sudden transition to TPT during the pandemic. Especially among therapists with less clinical and online experience at the start of the pandemic, and those struggling to build strong therapy relationships, professional self-doubt has been found to be salient (Aafjes-van Doorn, . Moreover, another salient factor in therapists' attitudes towards TPT during the COVID-19 pandemic is their experience of vicarious trauma. Due to the global health crisis and its negative mental health impact, therapists may have been more exposed to more stressful and traumatic material during their sessions than before the pandemic. Indeed, previous studies showed that on average, therapists experienced a moderate level of vicarious trauma during the pandemic and about 15% of therapists experienced high levels (Aafjes-van Doorn, Bekés, . These preliminary findings suggest that therapists' ability to create a strong therapeutic relationship online, their professional selfdoubt and experienced vicarious trauma likely impact therapists' acceptance of the TPT format. However, little is known about which of these factors are most strongly associated with TPT acceptance and how they interact with one another when relating to acceptance of TPT technology during the pandemic. Although traditional regression models and machine learning models, such as decision trees, may both base their variable selection on pre-existing theory, a decision tree model has certain statistical benefits. Traditional statistical approaches test each independent variable as a separate hypothesis, which can lead to erroneous conclusions because of multiple comparisons (inflated Type I errors), model misspecification and multicollinearity. In contrast, machine learning models are not constrained by model assumptions and are particularly helpful for finding patterns in complex datasets (Bi et al., 2019) . A decision tree model, for example, is able to predict the mean level of the dependent variable based on exploring interactions between potential independent variables (Loh, 2008) . In an attempt to improve the ability of identified models to generalize to new data that will be collected in the future, a machine learning model is usually developed in the training set and subsequently tested in a separate test set . This process of cross-validation avoids model overfit on the original data and mimics a replication of the model on an independent dataset, and it thus increases the confidence in the model's generalizability. The aims of the current study were twofold: (1) Therapists with neutral or strong online working alliance had the most positive attitudes towards telepsychotherapy. • Therapists with higher levels of professional self-doubt and especially with higher vicarious trauma were most concerned about using telepsychotherapy. • Therapists with low working alliance were least likely to use telepsychotherapy in the future. T A B L E 1 Therapist demographics and descriptives (n = 795) The online survey included individual items that assessed for therapists' demographics and professional activities as well as several stan- The novel Unified Theory of Acceptance and Use of Technology Therapist Version (UTAUT-T; Békés et al., 2021) was used to assess attitudes towards acceptance and usage of TPT. The UTAUT framework (Venkatesh et al., 2003 (Venkatesh et al., , 2012 ) offers a comprehensive model of acceptance and subsequent utilization of technological innovations. It has been adapted for various contexts, including occupational and physical therapy (Liu et al., 2015) . The UTAUT theoretical framework was recently used in a systematic review of pre-pandemic empirical studies as an overarching conceptualization of different aspects of acceptance of TPT technology to help frame their inclusion criteria and description of their findings (Connolly et al., 2020; see Venkatesh et al., 2016) . In line with the focus of the present study, the phrasing of the items was adapted to reflect TPT. For example, the item 'People who are important to me think that I should use the system' in the original UTAUT measure was adapted to 'People who are important to me think that I should use online therapy', whereas other items needed more modifications, for example, the original 'Using the system enables me to accomplish tasks more quickly' was adapted to therapists as 'Using online therapy saves me time and/or money' (for more details on the development of UTAUT-T, see Békés et al., 2021) . The original and extended UTAUT models have been validated in a variety of settings, including academic research (Gruzd et al., 2012) , higher education (Pynoo et al., 2011) and healthcare settings (Alapetite et al., 2009; Chang et al., 2007; Liu et al., 2015) , for a review, see Venkatesh et al. (2016) , whereas the novel UTAUT-T has not been validated on therapist samples yet. The UTAUT for psychotherapists includes three scales: Attitudes, Anxiety and Behavioural Intention. The 21-item UTAUT-T Attitudes subscale includes items related to performance expectancy, effort expectancy, social influence and facilitating conditions regarding using TPT. The six-item UTAUT-T Anxiety subscale includes items regarding feelings of apprehension or concerns about using technology (Compeau et al., 1999) . The Anxiety subscale has been shown to have a direct negative effect on actual technology use and is thus seen as an important inhibitory variable (Cenfetelli & Schwarz, 2011) . Finally, the two-item UTAUT-T Behavioural Intention subscale represents explicitly declared intent and plan to use technology in the future. As with all UTAUT adaptations, items of the UTAUT-T scales are scored on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree), with higher scores indicating more positive attitudes towards technology (Attitudes Scale), more concerns (Anxiety Scale) and more intention for future use of TPN (Behavioural Intention Scale). In the present study, the internal consistency of the UTAUT-T subscales was α = .78 (Attitudes), α = .80 (Anxiety) and α = .94 (Behavioural Intention). Therapeutic alliance was measured with the Working Alliance Inventory A global working alliance rating of 4 (sometimes), the middle point of the scale, is interpreted as a neutral working alliance (Horvath & Greenberg, 1989) . The WAI-SRT has shown adequate reliability and validity (Hatcher & Gillaspy, 2006) . The mean score of the WAI-SRT was used, with higher scores indicating a stronger working alliance. Cronbach's α for the total scale in the current sample was .85. The Real Relationship Inventory Therapist Form (RRI-T; Gelso et al., 2005) assesses the genuine human relationship between patient and therapist from the therapist perspective. In Gelso's tripartite model , the RR is conceptualized as an ongoing quality of the relationship distinguished from transference and the working alliance. The RR has been found to be highly correlated with the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989) , especially with the WAI's Bond subscale (Kelley et al., 2010) ; however, ratings of real relationship predict treatment progress and outcome above and beyond the variance explained by the WAI (Lo Coco et al., 2011; Marmarosh et al., 2009) . The RRI has 24 items that use a 5-point Likert scale from strongly disagree (1) to strongly agree (5), with higher overall scores reflecting a more genuine and authentic relation- ship. Reliability has shown to be high, with coefficient alphas ranging from .80 to .90 in various samples (e.g. Fuertes et al., 2013 Fuertes et al., , 2019 Marmarosh et al., 2009) . In this study, Cronbach's α was .73. The PSD includes nine items about how confident and competent the therapist feels, and items are rated on a 6-point Likert scale from 0 (never) to 5 (very often) and summed for a total score, with higher scores indicating more self-doubt. In our study, Cronbach's α was .84. The Vicarious Trauma Survey (VTS; Vrklevski & Franklin, 2008 ) is a selfreport measure of distress associated with working with traumatized patients. The VTS includes eight items, from which the first two ask about vicarious trauma exposure, whereas the other six ask about distress due to the exposure. Because the first two items are screening questions and do not assess the subjective experience of vicarious traumatization (Aparicio et al., 2013) , in the present study, we used the calculated sum of the six distress items. Items are rated on a 7-point Likert scale from strongly disagree (1) to strongly agree (7), where higher scores indicate more VT. The VTS has strong psychometric properties (Aparicio et al., 2013; Benuto et al., 2018; Michalopoulos & Aparicio, 2012) , with alpha .77 for the six-item version (Aparicio et al., 2013) . Cronbach's α for the total scale in the current sample was .81, and for the six-item subjective VTS, it was .83. The primary aim of the analyses was to develop a prediction model of the therapists' acceptance of the TPT format and their attitudes, concerns and intention for future use of videoconferencing. The three dependent variables were UTAUT-T Attitudes subscale, UTAUT-T Concerns subscale and UTAUT-T Behavioural Intention subscale. Based on the aforementioned clinical theory, we entered the following potential independent variables in the models: therapists' age and gender, as well as reported mean ratings of the real relationship, working alliance, professional self-doubt and vicarious trauma scale. Based on general recommendation for machine learning models (Vrigazova, 2021) and common practice (e.g. Prout et al., 2020) , the dataset was randomly split into two sets using PROC SURVEYSELECT of SAS: a training set (70%, n = 556) and a test set (30%, n = 239). For this purpose, regression decision tree analyses were run with the R party package (Hothorn et al., 2006) using Monte Carlo simulation for multiple-testing adjustment (Strasser & Weber, 1999 the first selected independent variable had the strongest association to the dependent variable. This association is estimated by a p-value corresponding to a test for the partial null hypothesis of a single independent variable and the dependent variable. Subsequently, the algorithm implemented a binary split in the selected independent variable, based on the corresponding p-value of less than .01. These steps were repeated recursively for each of the independent variables that were significantly associated with the dependent variable, until no remaining variable was associated with the dependent variable. As is common and recommended in prediction models (Loh, 2008) , performance of decision trees was evaluated by the root mean square error (RMSE; Kenney & Keeping, 1962) and R 2 (Glantz & Slinker, 2001) The real relationship has not been previously assessed with regard to online therapies. The working alliance scores on the WAI-SRT in our sample were similar to therapist-rated alliance scores on this same measure in previous studies in blended (combined in-person and online) therapy that was conducted before the abrupt transition to online therapies due to the pandemic (Vernmark et al., 2019) . On average, therapists experienced professional self-doubt sometimes or frequently in TPN during the pandemic, which is higher than the level of self-doubt experienced by therapists in a prior naturalistic study of PSD (Nissen-Lie et al., 2013) but still on the lower end of the 5-point Likert scale. The sample in our study reported significantly higher subjective VT compared to previously found pre-pandemic inperson sessions (Michalopoulos & Aparicio, 2012) . The UTAUT Attitude Scale significantly correlated with all independent variables, except for age and gender. The UTAUT Anxiety Scale significantly correlated with all independent variables. The UTAUT Behavioural Intention Scale correlated with UTAUT Attitude and Anxiety Scales, working alliance and professional self-doubt, but correlations with age, gender, real relationship and vicarious trauma were non-significant. After applying a Bonferroni correction (i.e. p < .005), the correlation between UTAUT Anxiety and gender was no longer significant. All the other correlations remained significant (when applying p < .005; see Table 3 ). Predicting models were trained on the train set (n = 556) and were assessed for predictive performance on the test set (n = 239). Three separate decision tree models were developed to predict the three aspects of acceptance of TPT technology. The regression tree model showed that higher alliance ratings (WAI > 3.9) was associated with more positive attitudes. Those with the higher alliance ratings (WAI > 3.9) with relatively low professional self-doubt (PD < =2.1) and younger than 40 reported the most positive attitudes. Therapists who reported the lowest alliance rates (WAI = <3) also reported the most negative attitudes (Figure 1 ). The regression tree model showed that therapists with the highest levels of professional self-doubt (PSD > 2.89) reported the most concern on UTAUT Anxiety. Those with relatively low levels of professional self-doubt (PSD < 1.67) and low levels of vicarious trauma (VTS < 1.17) reported the least concern about using the online platform ( Figure 2 ). The regression tree model showed that therapists who reported higher working alliance (WAI > 3) also reported more intention to use TPT in the future (Figure 3 ). The performance of the model of prediction for UTAUT-T Attitudes was relatively good in that it had the second best R 2 in the models .60*** À.36*** --6. Real relationship .01 À.40** .22*** À.20*** .07 --7. Working alliance inventory .29** À.04 .28*** À.18*** .10** .55*** --8. Professional self-doubt À.35*** À.12*** À.23*** .53*** À.11*** À.24*** À.25*** --9. Vicarious trauma À.25*** À.12*** À.13*** .33*** À.01 À.08* À.09** .47*** -- When we tested three aspects of the acceptance of TPT technology (attitudes, concerns, future intention to use), the first model showed that therapist-reported working alliance was most pertinent to be associated with attitudes towards TPT, in that sufficiently high alliance scores were associated with more positive attitudes, and most positive attitudes were reported by those who also experienced little professional doubt and were 40 years old or younger. Therapists who reported the weakest quality of the working alliance also reported the F I G U R E 1 Regression tree for attitudes towards telepsychotherapy technology (UTAUT-T Attitudes), n = 556 F I G U R E 2 Regression tree for concerns about using telepsychotherapy technology (UTAUT-T Anxiety), n = 556 most negative attitudes towards TPT technology. Therapists' concerns about TPT technology was most pertinently associated with the therapist's professional self-doubt, in that therapists with very little professional doubt (i.e. the most professionally confident therapists) had the least concerns about TPT, and concerns were the lowest for those who also reported low levels of vicarious trauma during the start of the pandemic. The prediction model for the intention to use TPT technology showed that the only pertinent predictor was working alliance. The majority of therapists who reported at least a medium level of working alliance were inclined to use TPT in the future, whereas a small subgroup of therapists who reported weak alliance also reported the least likelihood to use TPT in the future. Unlike previous research on in-person therapies pre-pandemic that suggested that besides the working alliance, the quality of the real relationship is also important in predicting treatment efficacy (Bhatia & Gelso, 2018) , the real relationship did not appear to play an important role in therapists' acceptance of TPT technology. In none of the three models, their reported level of the real relationship (RR) with their online patients was associated with any of the three dependent variables, once the independent variables that were more strongly associated with each of the dependent variables were included in the models, and therefore did not show in the respective decision trees. Similarly, gender of the therapist was not relevant in predicting any of the aspects of the therapists' acceptance of TPT technology. Overall, these results indicate that therapists' professional selfdoubt and the working alliance that they perceive with their online patients are the most prominent factors in predicting their reported acceptance of TPT. Our study findings partly confirmed previous results that therapists are often reluctant to use TPT due to fears of being unable to build a strong therapy relationship with their patients online (Roesler, 2017) . In line with previous studies, based on patient self-reported alliance measures, that reported that the working alliance in online therapies may be just as strong as in in-person therapies (Norwood et al., 2018) , we found similar ratings of the online working alliance in our participating therapists. Our results also highlight that concerns about the use of TPT is not pertinent to the therapists' ability to build a strong working alliance online, but rather to their professional self-doubt and experience of vicarious traumatization in their online work. This might imply that therapists can benefit from their own professional support in order to reduce their professional anxiety during the pandemic and more generally. Moreover, although professional self-doubt was linked to better treatment outcomes in a previous study of trainees (Nissen-Lie et al., 2013) , our study showed that the experience of more profes- (Sexton, 1999) , and this impact is especially strong when therapists and patients experience a disaster simultaneously (Boscarino et al., 2004; Culver et al., 2011) . In our study, during the pandemic, providing treatment remotely to patients dealing with traumatic stress represents a serious challenge. Before the pandemic, therapists considered severe psychopathology and acute crisis situations as counter-indicators of online treatment (Connolly et al., 2020; Simms et al., 2011) ; however, during the pandemic, there was no alternative to TPT. Our findings about heightened concerns regarding TPT in those with more subjective vicarious trauma experiences possibly reflect the therapists' general sense of anxiety during the pandemic or the impact of therapists' subjective experiences of stress and overwhelm on their ability to be openminded to new technology/ways of working. Also, due to the pandemic, therapists were probably more frequently exposed to highly stressful material in their sessions, and given that the stress of the pandemic impacts the therapists as well, they might have less emotional capacity to deal with their patients' trauma presentations. The heightened vicarious trauma experience in itself is an alarming fact, and its impact on concerns regarding TPT need to also be taken seriously when considering ways of using TPT with patients under high stress. Our results show that despite concerns described in previous literature about relational aspects of TPT (Roesler, 2017; Sucala et al., 2013) , the actual experience of working alliance and real relationship in online sessions was relatively positive. In fact, the quality of the real relationship was higher than in previous reports in inperson therapies (Bhatia & Gelso, 2018; Gelso et al., 2012) , even though due to the different study designs, these comparisons should be taken with some caution. Given that the quality of the therapeutic relationship is known to predict positive treatment outcomes, this finding provides some optimism regarding the potential impact of ongoing TPT. However, our study also indicated that therapists' rela- Providing therapy remotely has its own unique challenges, including the lack of certain aspects of non-verbal communication, which may hinder understanding of the patients' emotional state or the therapist's ability to sufficiently transmit their own emotions and empathy (Wagnild et al., 2006) . Training courses could provide tools to compensate for these aspects of TPT and help therapists to improve their online communication skills to enhance the therapeutic relationship online, and maybe most importantly, their sense of professional competence and confidence. Professional training and peer support groups are also valuable resources for ameliorating the subjective sense of vicarious traumatization while working during a global pandemic that impacts both patient and therapist. In many countries, therapy is delivered as part of government funded services, so decisions about mode of delivery are likely to be out of clinicians' hands. However, the increased clinical and research interest in TPT might mean that TPT will be more readily accepted by patients, therapists and payers. Given that more positive attitudes towards a treatment process tend to relate to better treatment outcomes, this bodes well for patients' well-being. Although previously clinicians were required to provide in-person therapy to ensure governmental funding, in the near future, clinicians might be able to make a clinical decision to determine which therapeutic format might be most beneficial to a patient at a given time. Of course, the use of inperson therapy or TPT does not have to be a binary clinical decision. Therapists and patients might consider integrating TPT sessions within an in-person treatment or do the full treatment via TPT technology (Van Daele et al., 2021) . Patients may receive a combination of in-person and remote sessions based on changing needs over the course of therapy (Yellowlees & Nafiz, 2010) . This study had several limitations. First, even though in our study we built the prediction models based on the well-supported assumption that attitudes towards TPT may be predicted by various factors (Connolly et al., 2020; Venkatesh et al., 2016) , including the relationship, professional doubt and vicarious trauma variables, a reverse causation may also be argued for. For example, it is possible that preconceived attitudes towards TPT may prevent therapists from trying to build strong therapeutic relationships in their online sessions. Second, even though the comparison between our sample and samples reported in previously published studies may be informative, these comparisons need to be interpreted with caution due to various confounders (e.g. different therapist and patient and treatment characteristics in these studies). Third, we used a convenience sample of therapists willing to report about their experiences during the pandemic in an online survey. Although our sample was relatively diverse, most respondents were female (as is the case in many survey studies; Moore & Tarnai, 2002) . At the same time, this is representative of the larger population of psychologists (American Psychological Association, 2015) . Therapists who volunteered to participate might also have been especially interested in TPT, which possibly biased the results. Fourth, a well-known limitation, common to all survey research, is that all variables were self-reported responses, which means that the relationship between these variables might have been spuriously inflated. Future research could benefit from using complementary methods to gain information about therapists' acceptance of TPT technology, for example, by having observer codings of behaviour of therapists or by interviewing supervisors and patients to gain their perspective on the therapist's attitudes. Finally, although the present study reports on a rigorous crossvalidation method for producing results that are likely to be generalizable to the broader population, this must be viewed as internal validation, rather than external validation, as the test set is still a sub- This large survey study of therapists' acceptance of TPT technology during the first weeks of COVID-19 indicates that therapists had relatively positive attitudes towards TPT technology and some concerns about using TPT in the future. The most consistent factor influencing therapists' acceptance levels was their professional self-doubt as well as their perceived quality of the online working alliance. This implies that professional support and training might be helpful in increasing therapists' sense of professional and relational competence and confidence online, and therefore generating a more positive attitude, fewer concerns and a higher likelihood of continued use of TPT technology in the future, when it is no longer a health requirement. Grappling with our therapeutic relationship and professional self-doubt during COVID-19: Will we use video therapy again? Counselling Psychology Quarter-lyAdvance online publication Psychotherapists' vicarious traumatization during the COVID-19 pandemic A scoping review of machine learning in psychotherapy research Acceptance of speech recognition by physicians: A survey of expectations, experiences, and social influence 2015 APA Survey of Psychology Health Service Providers Demographics of the U.S. Psychology Workforce An examination of the psychometric properties of the vicarious trauma scale in a sample of licensed social workers Psychotherapists' attitudes toward online therapy during the COVID-19 pandemic The development of a self-report scale to assess therapists' acceptance of online psychotherapy via video conferencing Stretching the analytic frame: Analytic therapists' experiences with remote therapy during COVID-19 Barriers and facilitators to the integration of web-based interventions into routine care. Clinical Psychology: Science and Practice The vicarious trauma scale: Confirmatory factor analysis and psychometric properties with a sample of victim advocates. Health & Social Care in the Community Therapists' perspective on the therapeutic relationship: Examining a tripartite model What is machine learning? A primer for the epidemiologist The generalizability of the psychoanalytic concept of the working alliance Compassion fatigue following the September 11 terrorist attacks: A study of secondary trauma among New York City social workers Videoconferencing psychotherapy for panic disorder and agoraphobia: Outcome and treatment processes from a non-randomized non-inferiority trial Identifying and testing the inhibitors of technology usage intentions Physicians' acceptance of pharmacokinetics based clinical decision support systems Social cognitive theory and individual reactions to computing technology: A longitudinal study A systematic review of providers' attitudes toward telemental health via videoconferencing Mental health professionals' experiences of vicarious traumatization in post-hurricane Katrina New Orleans A note from the guest editors Mental health care goes online: Practitioners' experiences of providing mental health care during the COVID-19 pandemic Real relationship, working alliance, transference/countertransference and outcome in timelimited counseling and psychotherapy Therapists' and clients' ratings of real relationship, attachment, therapist self-disclosure, and treatment progress Measuring the real relationship in psychotherapy: Initial validation of the therapist form The unfolding of the real relationship and the outcome of brief psychotherapy The real relationship and its role in psychotherapy outcome: A meta-analysis Primer of applied regression & analysis of variance Survey of psychologists' telebehavioral health practices: Technology use, ethical issues, and training needs Connected scholars: Examining the role of social media in research practices of faculty using the UTAUT model Development and validation of a revised short version of the working alliance inventory Development and validation of the working alliance inventory Unbiased recursive partitioning: A conditional inference framework The real relationship inventory: Development and psychometric investigation of the client form Root mean square What factors determine therapists' acceptance of new technologies for rehabilitation -A study using the unified theory of acceptance and use of technology (UTAUT) Relation of the real relationship and the working alliance to the outcome of brief psychotherapy Classification and regression tree methods Psychotherapy at a distance The real relationship in psychotherapy: Relationships to adult attachments, working alliance, transference, and therapy outcome The challenges and experiences of psychotherapists working remotely during the coronavirus* pandemic Vicarious trauma in social workers: The role of trauma history, social support, and years of experience Evaluating nonresponse error in mail surveys Working alliance inventory-short revised (WAI-SR): Psychometric properties in outpatients and inpatients Psychotherapists' self-reports of their interpersonal functioning and difficulties in practice as predictors of patient outcome Love yourself as a person, doubt yourself as a therapist? Working alliance and outcome effectiveness in videoconferencing psychotherapy: A systematic review and noninferiority meta-analysis Psychotherapist trainees' professional self-doubt and negative personal reaction: Changes during cognitive behavioral therapy and association with patient progress Development of psychotherapists: Concepts, questions, and methods of a collaborative international study Telepsychotherapy: A leaflet for psychotherapists in the age of COVID-19. A review of the evidence Identifying predictors of psychological distress during COVID-19: A machine learning approach Predicting secondary school teachers' acceptance and use of a digital learning environment: A cross-sectional study Online therapy: Review of relevant definitions, debates, and current empirical support Tele-analysis: The use of media technology in psychotherapy and its impact on the therapeutic relationship Telepsychotherapy during a pandemic: A traumatic stress perspective Encyclopedia of machine learning and data mining Vicarious traumatisation of counsellors and effects on their workplaces To use or not to use: Clinicians' perceptions of telemental health Psychotherapy via videoconferencing: A review On the asymptotic theory of permutation statistics Clinicians' attitudes toward therapeutic alliance in E-therapy Dropping the E: The potential for integrating e-mental health in psychotherapy Recommendations for policy and practice of telepsychotherapy and e-mental health in Europe and beyond User acceptance of information technology: Toward a unified view Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology Unified theory of acceptance and use of technology: A synthesis and the road ahead Working alliance as an independent variable of change in depression during blended cognitive behaviour therapy The proportion for splitting data into training and test set for the bootstrap in classification problems Vicarious trauma: The impact on solicitors of exposure to traumatic material Psychiatrists' satisfaction with telepsychiatry Telepsychotherapy for generalized anxiety disorder: Impact on the working alliance The psychiatrist-patient relationship of the future: Anytime, anywhere? Harvard Review of Psychiatry Is the alliance really therapeutic? Revisiting this question in light of recent methodological advances The data collected in China were supported by a research grant by the China American Psychoanalytic Alliance (VB, KAVD). We thank Lauren Smith (Kenyon College) for her contributions to updating references and creating the figures in this manuscript. The authors have no conflicts of interest to disclose. The dataset described in this article is not readily available because the dataset is not approved by the IRB for use by other researchers. Requests to access the datasets should be directed to vera.bekes@yu.edu. https://orcid.org/0000-0003-3043-5155Katie Aafjes-van Doorn https://orcid.org/0000-0003-2584-5897Sigal Zilcha-Mano https://orcid.org/0000-0002-5645-4429Tracy Prout https://orcid.org/0000-0002-3650-5890Leon Hoffman https://orcid.org/0000-0001-7466-5513 ENDNOTE