key: cord-0908335-v23diov5
authors: Parrish, Emma M.; Lin, Jiayi; Scott, Vanessa; Pinkham, Amy E.; Harvey, Philip D.; Moore, Raeanne C.; Ackerman, Robert; Depp, Colin A.
title: Mobile facial affect recognition and real-time social experiences in serious mental illness
date: 2022-04-08
journal: Schizophr Res Cogn
DOI: 10.1016/j.scog.2022.100253
sha: 986124fa8cec3e2ea0fffd6d8edd277e096ae934
doc_id: 908335
cord_uid: v23diov5

BACKGROUND: Emotion recognition deficits are linked with social dysfunction in psychosis, as is inaccurate self-assessment of emotion recognition abilities. However, little is known about the link between ER and real-time social appraisals and behavior. METHODS: In 136 people with psychotic disorders or affective disorder with psychosis we administered a novel ecological momentary cognitive test of emotion recognition which both assesses emotion recognition ability and self-assessed performance in conjunction with ecological momentary assessment of social appraisals, motivation, and time spent alone. Hybrid mixed effects models evaluated emotion recognition's associations with social experiences. RESULTS: Better recognition ability was associated with greater pleasure and more positive appraisals of others during interactions, whereas accuracy of self-assessment of emotion recognition ability was associated with more positive appraisals of interactions and social motivation. Overestimation of emotion recognition was linked with concurrent higher social motivation yet greater desire to avoid others. Time alone was unrelated to emotion recognition ability or self-assessment of ability. DISCUSSION: Mobile emotion recognition performance was associated with appraisals of recent interactions but not behavior. Self-assessment of social cognitive performance was associated with more positive appraisals and social motivation, and may be a novel target for interventions aimed at social dysfunction.

Impairments in social cognition are associated with social dysfunction in psychotic disorders (Fett et al., 2011; Horan et al., 2012; Green et al., 2019) and bipolar disorder (Yalcin-Siedentopf et al., 2014) . There are several mechanisms by which facial emotion recognition impairments impair function, including through motivational impairments, negative symptoms, and avoidance behavior (Gard et al., 2009; Green et al., 2019; Lin et al., 2013) . Additionally, diminished introspective accuracy (IA), or the degree to which one accurately gauges one's performance, for facial emotion recognition is also aberrant in psychotic disorders (Badal et al., 2021) . Indeed, many people with psychotic disorders tend to overestimate their own functioning, neurocognition, and social cognition, and this overestimation of ability is related to impairments in community and social functioning (Silberstein and Harvey, 2019; Sabbag et al., 2012; Bowie et al., 2007) . IA is a strong and independent predictor of functioning, particularly in psychosis (Silberstein and Harvey, 2019) . Importantly, impaired IA is independent from performance on formal tests of social cognitive ability (Silberstein and Harvey, 2019) . The day-to-day influences of emotion recognition impairments and IA of that ability on social experiences have received less study. In this report, we detail the associations between repeated ecological momentary testing of facial emotion recognition ability, IA of that ability, and contemporaneous reports of social appraisals, motivation, and behavior.

Emotion recognition is typically measured by in-lab tasks (e.g., the ER-40 task), and social dysfunction is most often measured by clinician or self-report scales (e.g., the Birchwood index; Fett et al., 2011) . In establishing the link between emotion recognition and functioning, the literature to date has largely correlated total scores from emotion recognition measures derived from lab-based tasks with global observer and self-rated functional scores (Irani et al., 2012; Fett et al., 2011) . While meta-analyses have established a link between emotion perception and function (Irani et al., 2012; Fett et al., 2011) , they are limited in addressing how facial emotion recognition might alter day-to-day social processes. Notably, although facial emotion recognition performance may be stable (Horan et al., 2012) , diminished facial emotion recognition may relate to social processes which might vary considerably over time within people, such as motivation, social appraisals, and/or behavior. Additionally, social cognition is associated with positive symptoms (Pinkham et al., 2011) , so one may expect variability in this construct as positive symptoms vary over time within people. To our knowledge, only one study has evaluated the link between emotion recognition task performance and ecological momentary assessment (EMA) of social behavior and appraisals (Janssens et al., 2012) . Opposite what might be expected from findings from studies of scale-based measures of function, that study found no association between the labbased emotion task and any EMA derived indicators of time spent alone or interaction appraisals. This study aligns with other studies that have found inconsistent associations between lab-based and EMA measures of function (Granholm et al., 2020) and contributed to a call for greater focus on ecological study of social cognition in relation to day-today social experiences (Myin-Germeys, 2020). One approach to potentially increase the ecological validity of labbased tasks is to deliver them at the same time and within the same context as the EMA surveys of behavior (Moore et al., 2017; Parrish et al., 2021b) . We developed and validated a novel mobile facial emotion recognition measure, based on the ER-40, that can be integrated with EMA paradigms for repeated, in vivo assessment, the Mobile Ecological Test of Emotion Recognition (METER) . This measure presents facial emotion stimuli and asks participants to judge the emotion being presented across five categories (anger, sadness, happiness, fear, or no emotion). To assess IA, participants are asked to estimate the number of faces that correctly identified, and IA is operationalized as the difference between the number of faces thought correct and the number actually correct. We validated this task in a sample of people with psychotic or mood disorders with a history of psychosis and found convergent validity between lab-based tasks of emotion recognition as well as expected significant associations with other cognitive domains that parallel that of in-lab tasks.

In this study, we evaluated the associations between social cognitive ability and IA of ability as measured by the METER with real-time concurrent EMA reports of social experiences. We focused on three clusters of social experience that are related to capacity and global measures of dysfunction (Granholm et al., 2020; Fulford et al., 2018) : 1) appraisals of recent interactions, 2) near-term social motivation and desire to avoid others, and 3) the ratio of samples spent alone. We evaluated the METER and these social experiences in sample of 136 persons with schizophrenia, schizoaffective disorder, or bipolar disorder with psychotic features who participated in an EMA study wherein participants completed both EMA and METER every day for 10 days. We hypothesized that worse METER performance and diminished IA of METER performance would be associated with more momentary negative social appraisals, lower social motivation, greater avoidance, and greater aggregated time spent alone.

Data from this study included 136 participants from a longitudinal, multi-site (University of California San Diego (UCSD); University of Texas at Dallas (UTD); University of Miami (UM)) study evaluating the relationship between social cognition and suicide in psychotic disorders. Full methods are described elsewhere (Parrish et al., 2021a; Depp et al., 2021) . Participants were recruited from outpatient facilities across the three study sites: UCSD (n = 51), UTD (n = 52), and UM (n = 33). The study was approved by each site's Institutional Review Board, and participants completed an assessment of capacity to consent to research prior to written informed consent.

Inclusion criteria for the study were: 1) aged 18-65; 2) current diagnosis of schizophrenia, schizoaffective disorder, bipolar disorder with psychotic features, or major depressive disorder with psychotic features, confirmed by the Mini International Neuropsychiatric Inventory (MINI; Sheehan et al., 1998) and the Psychosis Module of the Structured Clinical Interview for the DSM-5 (SCID 5; First et al., 2015) ; 3) available informant with whom the participant was in regular contact; 4) outpatient, partial hospitalization, or residential mental health care status; 5) proficient in English; and 6) able to provide informed consent. Exclusion criteria were: 1) history of a head trauma with loss of consciousness >15 min; 2) diagnosis of a neurological or neurodegenerative disorder; 3) vision or hearing problems that would interfere with data collection; 4) estimated IQ < 70, as determined by the Wide Range Achievement Test-4 (WRAT-4; Wilkinson and Robertson, 2006) ; 5) DSM-5 diagnosis of a substance use disorder in the past three months, excluding cannabis and tobacco, and confirmed by the SCID-5.

In-lab diagnostic and symptom assessments were conducted by raters trained in administering interview-based measures. Raters achieved a 0.80 inter-rater reliability kappa with gold-standard raters. Since the focus of this study was on the associations within EMA measures, in-lab measures are reported for sample characterization purposes but are not included in analyses. Symptom severity was evaluated with the Positive and Negative Syndrome Scale (PANSS; Kay et al., 1987) , and depression was evaluated with the Montgomery-Asberg Depression Rating Scale (MADRS; Montgomery and Åsberg, 1979) . During the baseline visit, participants completed an abbreviated version of the MATRICS Consensus Cognitive Battery (MCCB; Nuechterlein et al., 2008) .

Participants were given the choice to use their personal smartphone or a lab-provided Android smartphone. Incentives for survey completion were provided, with the option to earn $1.66 for each survey completed, with a maximum of $50. Surveys were sent three times daily during the EMA sampling period, and participants chose their one-hour sampling windows in the morning, afternoon, and evening. Participants were instructed to carry their smartphones with them throughout the 10-day protocol to answer the EMA surveys within 1 h of receiving them. To encourage adherence, study staff contacted participants by telephone on the first day of the EMA period and contacted again if they missed more than three consecutive surveys.

Surveys occurred 3×/day for a total of 30 possible EMA samples. EMA surveys captured the participant's perception and quality of social interactions. All EMA questions used a 7-point Likert scale (see Table 1 for Likert scale values). Participants reported whether they were alone or with other people at the time of taking the survey, and also if they had interacted with someone else since their past survey. If participants reported that they had interacted with someone else since the past survey, surveys branched and participants were asked to rate their recent social interactions based on their experience of pleasure, trust, previous motivation to engage, and self-perception during their social interaction as likable or unlikeable (see Table 1 for full EMA questions). Participants also rated their levels of social motivation and avoidance for interacting with others throughout the rest of the day. To evaluate the combined influence of avoidance and motivation, we averaged motivation and avoidance within subjects and then categorized based a 2 × 2 matrix in which low and high avoidance and motivation were combined into a single variable (e.g. low motivation, high avoidance).

The Mobile Electronic Test of Emotion Recognition (METER) is a smartphone-based mobile cognitive test based on the Penn Emotion Recognition test (Kohler et al., 2000) . Development and validation of this task are described elsewhere . The METER is administered concurrently with the EMA surveys once per day, stratified by time of day (either morning, afternoon, or evening periods). This task was administered immediately following the EMA questions. Participants were presented with a total of 10 faces each session from a pool of 100 unique faces. Each face displayed one of five emotions: happiness, sadness, anger, fear, or no emotion, and two per category were presented each session. After each session of the METER, participants were asked to self-assess their performance, reporting how many faces they believed that they had correctly identified from 0 to 10. IA was calculated as the difference between actual and self-assessed performance as the indicator of IA.

All analyses were performed using SPSS Statistics v.26. One item on the PANSS, N1 Blunted Affect, was missing for 31 participants due to remote data collection during the COVID-19 pandemic. This variable was imputed using multiple imputation (100 imputations) and using all other PANSS N symptoms (N2-N7) to inform this replacement. We calculated percentage of surveys spent alone based on who participants reported they were with at the time of the survey. We ran a hybrid mixed effects linear models (Twisk, 2019) which incorporate both mean-level (between persons) and within-person effects. We included random effects for subject in each of these models and included both METER performance and IA in the models. Participants were identified as having high or low motivation or avoidance based on a median split to understand the interplay of the two constructs. Then, these participants were classified into four groups: those with high motivation and high avoidance (HM/HA; N = 46), low motivation and high avoidance (LM/ HA; N = 46), high motivation and low avoidance (HM/LA; N = 29), and low social motivation and low avoidance (LM/LA; N = 15). An ANOVA with a post-hoc Tukey test was used to compare these four groups on METER performance and IA. Mixed effects linear models incorporate all data and therefore are robust to missing data. The p value was set to .05 for all analyses.

See Table 1 for full characteristics, including clinical characteristics and descriptive statistics of EMA responses. The sample was 57.4% female, had a mean age of 43.4 (SD = 11.4), and 46.3% Black or African American. 42.6% of participants in this sample had a diagnosis of schizoaffective disorder, followed by 32.4% with a diagnosis of schizophrenia, 22.8% with a diagnosis of bipolar disorder with psychotic features, and 2.2% with a diagnosis of major depressive disorder with psychotic features. On the METER, participants got an average of 7.5 out of 10 (SD = 1.2) faces correct and reported an average of 7.5 out of 10 (SD = 1.6) as correct. Participants reported that they were alone for 47.7% of the surveys. The percentage of surveys spent alone did not differ between participants who participated in the EMA surveys before or during the COVID-19 pandemic, t(126) = − 0.424, p = .336.

See Table 2 for full statistics. The effect of METER performance and IA of METER performance on social interaction appraisals was only significant at the mean level, not the within-person (momentary) level, for all dependent variables. Better METER performance (b = 0.28, SE = 0.11, t = 2.48, p = .014) related to greater appraised pleasure of social interactions since the past alarm. Additionally, IA underestimation of METER performance (b = − 0.24, SE = 0.08, t = − 3.07, p = .003) was related to reduced appraised pleasure of social interactions since the Mean "How much pleasure did you get out of the interaction?" (1 = Not at All; 7 = Very Much) b 4.6 (1.9), 1-7

Mean "How motivated were you in interacting with others since the last alarm?" (1 = Not at All; 7 = Very Much) b 2.9 (2.1), 1-7

Mean "What did you think others were thinking about you?" (1 = Unlikeable or inferior; 7 = Likeable or capable) b 5.0 (1.8), 1-7

Mean "How did you feel toward others in the interactions?" (1 = On guard or threatened; 7 = Trusting or warm) b 5.0 (1.7), 1-7

Mean "How much interest or motivation do you have in interacting with others later today?" (1 = Not at All; 7 = Very Much) 4.0 (1.9), 1-7

Mean "How much do you want to avoid others later today?" (1 = Not at All; 7 = Very Much 

See Table 3 for full statistics. Like social interaction appraisals, the effect of METER performance and IA of METER performance on social motivation and avoidance was significant only at the mean level, not at the momentary level. Greater IA underestimation of the METER related to reduced motivation for interacting with others later that day (b = − 0.24, SE = 0.08, t = − 2.83, p = .005). However, METER performance and IA of meter performance did not relate to the desire to avoid others later in the day (p's > .125).

Neither METER performance (r = 0.006, p = .944) nor IA of the METER (r = 0.061, p = .491) were correlated with percentage of surveys spent alone.

Finally, we examined the combination of motivation and avoidance by groups according to level of motivation and avoidance. Mean levels of motivation and avoidance for the four subgroups are presented in Table 4 . Participants with overall HM/HA ratings had worse momentary METER performance than participants with LM/HA (p = .040), LM/LA (p = .040), and HM/LA (p < .001), F(3, 983) = 5.66, p < .001. Similarly, participants with overall HM/HA ratings had greater momentary IA overestimation than those with LM/HA (p = .003) and LM/LA (p = .001), F(3, 979) = 6.07, p < .001.

This study centered on the association between facial emotion recognition ability and social experiences among people with psychotic disorders or affective disorders with psychosis, examining both the effect of emotion recognition ability and IA in relation to appraisal of social interactions and anticipation of future interactions. Both emotion recognition and social experiences were measured in vivo via mobile 

Mean EMA values for four subgroups of social motivation and avoidance. devices. We found that diminished real time mobile facial recognition ability was associated with diminished pleasure derived from recent social interactions and more negative appraisal of others perception from recent interactions, but not with social motivation or desire to avoid others in the near term. Compared to task performance, underestimation of social cognitive ability was associated with a broader variety of recent interaction appraisals as well as to diminished motivation to interact with others later in the day. Analyses examined patterns of ability and IA in relationship to the combination of diminished motivation and avoidance desires, indicating that people with greater underestimation of facial emotion recognition ability were more likely to report diminished social motivation, whereas overestimation of facial emotion ability displayed a more disorganized profile with both elevated motivation and avoidance desires. In contrast to hypotheses, associations between social time use (e.g., surveys alone) were not apparent for either performance or IA. While requiring replication, these findings showcase how mobile cognitive testing may identify processes involved in important functional outcomes like social dysfunction, and demonstrate that IA of social cognitive ability is related to day-to-day social experiences that drive social dysfunction. Importantly, IA appears to exert a more consistent impact than that of social cognitive ability. These findings build on prior work using lab-based approaches to evaluate facial emotion ability (Fett et al., 2011) and the one study we found that evaluated the link between social cognitive ability with EMA of social experiences (Janssens et al., 2012) . Whereas Janssens et al. (2012) did not find any relationship between social cognitive ability and EMA parameters, we did find that recent appraisals of pleasure from interactions and perspectives of others during the interaction were related to better METER performance. There are several possible explanations for the differing findings in the present study. First, differences in the EMA queries on social appraisals or composition of the samples may have accounted for this. Additionally, Janssens et al. (2012) used a lab-based measure in their study, whereas the METER task is employed throughout the day and administered concurrently with EMA of social experiences. Therefore, the METER task may be sensitive to relationships with appraisals in the real-world setting because the measurement of social cognition and social appraisals occur closer in time to one another. We note that this does not signify that fluctuations in social cognitive ability relate to changes in experiences, as our models found a mean (between person effect) but not a within person effect consistent with the stability of social cognitive ability (Horan et al., 2012) , suggesting that these processes may have trait-like characteristics.

However, consistent with Janssens et al. (2012), we did not find significant associations with the METER and surveys spent alone. In our previous research with another sample, the combination of being home, and alone, and engaging in concurrent unproductive activities (e.g., pacing, resting) was associated with the worst global functional outcomes (Strassnig et al., 2021) . It may be that social cognitive abilities are not useful for prediction of social behavior as suggested by Myin-Germeys (2020). It may also be that variables like surveys spent alone may be too coarse to provide reliable indicators of social activity, and may not be an accurate reflection of social functioning as there are a number of reasons why one may spend time alone. Furthermore, variation in social contexts (e.g., living situation) may obscure any effects of facial emotion ability on social interaction behavior. Research identifying the most sensitive and reliable measures of social behavior, be it through EMA or passive sensing, could help connect future work evaluating the links among abilities, appraisals and motivation, and behavior.

More than ability, we found that IA of facial emotion recognition was related to social appraisals and to real-time future motivation. IA can be considered in terms of inaccuracy but does not itself index the direction of inaccuracy, i.e., bias. Our findings supported an interesting phenomenon whereby diminished motivation was associated with a bias toward underestimation of facial emotion recognition ability, independent of avoidance of others. Notably, overestimation was associated with an "ambivalent" combination of higher avoidance desires yet with higher social motivation. Thus, overconfidence in IA in social cognition may impact behavioral control, which is consistent with prior lab-based work indicating overconfidence as reducing performance and social function (Jones et al., 2020) . These findings suggest potential ways in which poor IA could interfere with social function, with potentially different mechanisms depending on over-or underestimation biases likely requiring different intervention approaches.

Notwithstanding these possibilities, there are several limitations to this study. The causal direction between emotion recognition ability and social processes cannot be determined from the analyses conducted here. Emotion recognition ability is assumed to precede motivation and behavior, but a bidirectional relationship is possible. Although EMA enables such directional analyses, we did not explore these since our focus was on establishing concurrent links. Future studies can explore these sequential effects. We also did not look at ability to evaluate specific emotions or biases due to the large number of analyses. In addition, our sample was comprised of outpatients who were largely treated, and therefore these results may not generalize to inpatient or more severely acutely ill samples. Importantly, future studies should examine the influence of depression and sadness on social appraisals and IA of social cognition, as high levels of depression may be related to greater underestimation of ability (Bowie et al., 2007) .

In terms of implications, these findings suggest that EMA integrated with assessment of facial emotion recognition could help to pinpoint how social cognition training interventions, of which there are several (Grant et al., 2017) , might lead to early changes in social processes that are linked to functioning. Since improvement in social function may require longer time scales, understanding the impact of emotion recognition deficits on subjective momentary experiences could support development of early-stage interventions aimed at improving social function through evaluating change in intermediate mechanisms such as social appraisals. In addition, since IA of facial emotion recognition correlated with social motivation, even adjustments for ability, interventions targeting IA of facial emotion recognition should be considered. There are a number of studies indicating that IA of various cognitive abilities may be malleable (Carpenter et al., 2019; Bhome et al., 2019) , paralleling interventions targeting metacognition to reduce delusional thinking (Moritz et al., 2019) . Finally, it may be that mobile platforms could be used for real-time delivery cognitive behavioral therapy (CBT) or even drill-and-practice social cognition interventions aimed at the causes of diminished facial emotion recognition ability and accurate self-assessments of ability.

This work was supported by the National Institute of Mental Health (grant number: NIMH R01 MH116902-01A1; and T32 MH019934 to E. M.P.).

Emma M. Parrish: Lead conception and analysis, co-led development of the manuscript Jiayi Lin: Contributed to conception, analysis and development of the manuscript Vanessa Scott: Contributed to conception, analysis and development of the manuscript Amy E. Pinkham: Contributed to conception, analysis, and development of the manuscript, co-led development of data source Philip D. Harvey: Contributed to conception, analysis, and development of the manuscript, co-led development of data source Raeanne C. Moore: Contributed to conception, analysis, and development of the manuscript, co-led development of data source Robert Ackerman: Contributed to conception, analysis, and development of the manuscript, co-led development of data source Colin A. Depp: Contributed to conception, analysis, and development of the manuscript, led development of data source.

None of the authors has any conflicts of interest to report.

Computational methods for integrative evaluation of confidence, accuracy, and reaction time in facial affect recognition in schizophrenia

Metacognition in functional cognitive disorder-a potential mechanism and treatment target

Self-assessment of functional status in schizophrenia

Domain-general enhancements of metacognitive ability through adaptive training

Ecological momentary facial emotion recognition in psychotic disorders

The relationship between neurocognition and social cognition with functional outcomes in schizophrenia: a meta-analysis

Social motivation in schizophrenia: how research on basic reward processes informs and limits our understanding

Motivation and its relationship to neurocognition, social cognition, and functional outcome in schizophrenia

What do people with schizophrenia do all day? Ecological momentary assessment of real-world functioning in schizophrenia

Social cognition interventions for people with schizophrenia: a systematic review focussing on methodological quality and intervention modality

Nonsocial and social cognition in schizophrenia: current evidence and future directions

Social cognition in schizophrenia, part 2: 12-month stability and prediction of functional outcome in first-episode patients

A meta-analysis of emotion perception and functional outcomes in schizophrenia

Emotion recognition in psychosis: no evidence for an association with real world social functioning

Confidence, performance, and accuracy of self-assessment of social cognition: a comparison of schizophrenia patients and healthy controls

The positive and negative syndrome scale (PANSS) for schizophrenia

Emotion recognition deficit in schizophrenia: association with symptomatology and cognition

Clinical symptoms, mainly negative symptoms, mediate the influence of neurocognition and social cognition on functional outcome of schizophrenia

A new depression scale designed to be sensitive to change

Applications for self-administered mobile cognitive assessments in clinical research: a systematic review

Metacognitive and cognitivebehavioral interventions for psychosis: new developments

Real-life social interactions in schizophrenia

The MATRICS Consensus Cognitive Battery, part 1: test selection, reliability, and validity

Ecological momentary assessment of interpersonal theory of suicide constructs in people experiencing psychotic symptoms

Remote ecological momentary testing of learning and memory in adults with serious mental illness

Actively paranoid patients with schizophrenia over attribute anger to neutral faces

Predictors of the accuracy of self assessment of everyday functioning in people with schizophrenia

Impaired introspective accuracy in schizophrenia: an independent predictor of functional outcomes

Evidence for avolition in bipolar disorder? A 30-day ecological momentary assessment comparison of daily activities in bipolar disorder and schizophrenia

Mixed Model Analysis in Longitudinal Studies

Facial affect recognition in symptomatically remitted patients with schizophrenia and bipolar disorder

We would like to thank Katelyn Barone, Bianca Tercero, Cassi Springfield, Linlin Fan, Ian Kilpatrick, Snigdha Kamarsu, Tess Filip, Avery Quynh, and Maxine Hernandez for their involvement in data collection and recruitment.