key: cord-0436404-j0pebt0l
authors: Andreoletti, Davide; Luceri, Luca; Leidi, Tiziano; Peternier, Achille; Giordano, Silvia
title: The Virtual Emotion Loop: Towards Emotion-Driven Services via Virtual Reality
date: 2021-02-26
journal: nan
DOI: nan
sha: 66cd9768a18d4651464afc0c39eafc41723a00c2
doc_id: 436404
cord_uid: j0pebt0l

The importance of emotions in service and product design is well known. Despite this, however, it is still not very well understood how users' emotions can be incorporated in a product or service lifecycle. In this paper, we argue that this gap is due to a lack of a methodological framework for an effective investigation of the emotional response of persons when using products and services. Indeed, the emotional response of users is generally investigated by means of methods (e.g., surveys) that are not effective for this purpose. In our view, Virtual Reality (VR) technologies represent the perfect medium to evoke and recognize users' emotional response, as well as to prototype products and services (and, for the latter, even deliver them). In this paper, we first provide our definition of emotion-driven services, and then we propose a novel methodological framework, referred to as the Virtual-Reality-Based Emotion-Elicitation-Emotion-Recognition loop (VEE-loop), that can be exploited to realize it. Specifically, the VEE-loop consists in a continuous monitoring of users' emotions, which are then provided to service designers as an implicit users' feedback. This information is used to dynamically change the content of the VR environment (VE), until the desired affective state is solicited. Finally, we discuss issues and opportunities of this VEE-loop, and we also present potential applications of the VEE-loop in research and in various application areas.

Traditionally, products and services are developed considering their functional requirements as the primary design objective. However, the success of a product is also (or, according to many, perhaps even mainly) determined by its ability to engage its users at the emotional level. In this respect, the authors of Ref. [1] claim that up to 95% of our buying decisions are unconscious. Despite this, however, the role of emotions is often underestimated, if not even totally disregarded, when designing products and services [2] , and this contradiction is generally interpreted as a phenomenon of cultural inertia [2] . While also supporting this idea, in this paper we argue that another crucial reason is the lack of a methodological framework that enables designers to effectively consider the emotional response of users of services and products. In this paper, we give our definition of emotion-driven service/product and we propose a methodological framework, enabled by Virtual Reality (VR) technologies, that can be used towards the realization of this idea.

We say that a service (or product) is emotion driven if the emotions of the persons that use it are taken into consideration in as many phases of its lifecycle as possible (e.g., from design to delivery). Commonly, services undergo a validation process aimed to assess the fulfillment of their technical and utility requirements. In our view, emotion-driven services should be produced by performing a similar validation of their emotional requirements as well. To this end, we propose a methodological framework that allows to develop services following an emotion-by-design approach, i.e., by integrating users' emotions from the very early stages of service development, and not, as commonly done, by leaving them as an afterthought.

To realize this idea, we envision the implementation of a loop in which users' emotions are monitored and the characteristics of the service are dynamically changed to induce the emotions intended by its designers. This dynamic change can be either implemented in the design phase only (e.g., to validate the hypothesis that some feature triggers a specific emotional reaction) or, whenever possible, in the delivery phase as well (e.g., by dynamically modifying the characteristic of a service in real-time). In this paper, we provide arguments supporting the claim that the VR is the most ideal instrument to realize this loop, that we refer to as Virtual-Reality-Based Emotion-Elicitation-Emotion-Recognition loop or, in short, VEE-loop. The reasons why we consider the VR the most suitable technology to realize this scheme are manifold. First, among all the existing digital technologies, the VR is the one that guarantees the most tangible experience across the most varied domains. Then, the VR allows to flexibly modifying the experience of its users. In addition, Head Mounted Displays (HMDs) allows to collect a significant number of valuable users' bio-feedbacks (e.g., movements and postures) that can be exploited to infer their emotional status [3] , as well as their involvement, fatigue, and stress [4] .

The VEE-loop has a potentially high number of application areas. For example, the VEE-loop can be used as a tool to perform a validation of the capability of a product to trigger specific emotions, before its actual production. Indeed, designers could obtain an emotional feedback from potential customers and tune the design accordingly. In this respect, the tangibility of the VR guarantees a higher fidelity of this emotional reaction with respect to other methods, while its flexibility allows to test a high number of products' characteristics. The VEE-loop can also improve services in which having the information on users' affective states is highly beneficial but is unavailable for some reasons (e.g., due to physical distancing measures imposed to handle the Covid-19 pandemic). In remote learning, for example, the emotional statuses of students can be monitored, and the virtual lecture dynamically changed (e.g., to induce calm in students or to draw their attention).

This paper is structured as follows. In Section 2 we elaborate on the impor- tance of designing products and services based on users' emotions, we give our definition of emotion-driven services and products and we motivate our proposal of the VEE-loop as an enabling framework to realize them. Then, in Section 3 we provide more details of the main components of the VEE-loop, and we show its novelty with respect to previous works. Section 4 discusses potential applicative areas of the proposed VEE-loop. Finally, Section 5 elaborates on the challenges and opportunities of our framework, also adding some conclusive remarks.

This Section starts by elaborating on the importance of users' emotions when using products and services. Our aim is to clarify the rational of designing emotion-driven products and services, that are as well defined in this Section. Finally, we motivate the use of VR as an enabling technology towards the realization of this vision.

The authors of Ref [2] make a series of claims on the importance of emotions of persons that use products and services. First, emotions convey valuable information on the perceptions that users have when using a product. Emotions are also related to users' desires, which are often latent and not verbalized. Therefore, traditional marketing strategies fail to effectively capture the emotions that a given product or service elicit on its users. Indeed, traditional marketing strategies, such as surveys, allow designers to understand new functional requirements that can help to do only slight and superficial product modifications only. Instead, by capturing also the emotional reaction of users, designers could understand their customers more deeply, and so create more radical innovations. In this respect, our proposed VEE-loop would enable designers to qualitatively measure their users' emotions, and to test a high number of combinations of a product's sensory qualities (e.g., form and color). Then, co-design (i.e., performed in conjunction by designers and customers) allows to create more personalized services and products, which are generally regarded as more attractive than those done by the designers only, or by the customers only [2] . As users are often incapable of verbalizing their emotions, the emotional feedback obtained through the use of the VEE-loop is indeed valuable, as it could help both designers and customer to understand what customers really like about a product.

Finally, emotions are extremely important to consolidate brand identification. Indeed, a clear and tight connection between a brand and a specific emotion ensures loyalty of customers in the long term. In this respect, it is crucial that all the phases of a product or service lifecycle can evoke the very same emotion. For instance, both the use and after-sales assistance should, ideally, evoke the same emotions on customers. The VEE-loop can be used to check if there is consistency between intended and perceived emotions, and, in case there is not, adapt the content to ensure this consistency.

The aforementioned facts justify the idea of implementing the proposed loop involving emotion recognition and elicitation. In the following subsection, we elaborate more on the characteristics of emotion-driven services and products.

Ideally, an emotion-driven service or product considers users' emotions in all the phases of its lifecycle. Our proposal to take users' emotions in consideration is the VEE-loop. As an example, the design phase is done by exploiting the VEE-loop to validate the fulfillment of users' emotional requirements before the realization of the actual tangible product. This implies, for instance, that a product undergoes large-scale tests aimed to validate the assumption that it can evoke the intended emotions. In this phase, users' emotions are tracked and used as input for the designers, who can better understand which specific characteristic has caused the detected emotions. Then, various combinations of features are tried (exploiting the flexibility of the VR), and the operation is repeated, in a loop. Similarly to the design phase, the VEE-loop can also be used during the delivery phase to dynamically change the service until it elicits the emotions intended by the designers.

We are aware that this paradigm might not be applicable to all the phases of a product/service lifecycle. Let us clarify this issue with a couple of examples.

In an idealistic emotion-driven schooling, for instance, a teacher would prepare her lecture (i.e., in the design phase) also trying to induce enthusiasm on her students, and she will eventually modify the style (or even the content) of the lecture to achieve this goal (i.e., in the delivery phase). On the other hand, the designer of a commercial product might take particular stylistic choices with the aim to elicit a specific emotion that reinforces brand identification [5] (e.g., sense of comfort), but these choices would be limited to the design phase only (in case the product can not change after it has been made).

In the following subsection, we provide arguments that support our choice of the VR as the most suitable candidate technology to implement our idea of loop of emotion recognition and elicitation. Indeed, the VR can be employed both during design (exploiting its flexibility and tangibility) and during delivery, as an increasing number of services are being provided also with VR.

VR is the most natural, direct, and ideal technology for translating into real products and services all the ideas explained so far, and for many reasons.

First, pure VR (i.e., a user interacting with an entirely synthetic, computergenerated virtual environment) allows creating completely modifiable, dynamic experiences. Unlike augmented and mixed reality, which is limited and linked to surrounding, physical elements, pure VR can be easily distributed online, experienced everywhere, replayed at will, and its content regularly updated.

The immersion provided by VR also amplifies emotional reactions [6, 7] , which help both in the emotion recognition and elicitation phases compared to other less effective means to put a user in a given simulated situation. In addition, the retention rate of learning and training dispensed via VR is increased when compared to more conventional media [8] .

The recent revival of VR also provides a much lower entry-point to the technology, which is now considered a commodity, off-the-shelf option no longer limited to research laboratories or professional contexts. Thanks to this evolution, which also significantly increased the quality of modern VR compared to the state-of-the-art of just few years ago, a significantly larger user-base can now be targeted by VR-enabled solutions.

Finally, modern VR equipment already embeds sensors that are critical for inferring the user's emotional state (and its evolution) during the virtual experience. Since body tracking is a central requirement of VR, most of the recent Head Mounted Displays (HMDs) are capable of tracking user's head and hands position in real-time and at high frequency, while some models started including eye tracking, too. These sensors can be used not only for the proper positioning of the user within the virtual environment (e.g., to update the viewpoint and stereoscopic rendering parameters) and to precisely determine what the user is looking at at a precise moment, but also to derive a series of additional metrics such as heartbeat and respiratory rate [9] . Next-generation HMDs will directly embed dedicated sensors for monitoring such states (like the HP Reverb G2 Omnicept).

This constant source of information can be used to acquire data that previously required to dress the user with a cumbersome set of devices and/or to prepare the environment for different levels of motion tracking (from a simple Microsoft Kinect to professional-grade systems such as the Vicon). Most of these capabilities are now integrated into one single device that provides all the ingredients for building an emotion recognition and elicitation system under wearable and affordable constraints. Nevertheless, HMDs can still be coupled with additional monitoring devices to increase the amount, kind, and accuracy of user-generated signals for this task (e.g., by combining the full-body tracking provided by the Microsoft Kinect with the head and hands positions returned by the headset).

The VEE loop consists in the continuous monitoring of the affective states of users (performed by analysing their bio-feedbacks) and in the adaptation of the content of the VR to induce a transition from the current to the desired affective state (e.g., from fear to calm), or to keep the current emotion stable. We refer to the content of the VR to as Virtual Environment (VE). To make this possible, the VEE-loop is composed of a module for Emotion Recognition (ER) and one for Emotion Elicitation (EE). An overall representation of the VEE-loop architecture is depicted in Fig. 2 . Specifically, this figure shows that user's generated bio-feedback are given in input to the ER module, which infers from them the emotion most likely perceived by the user. The detected emotion and the emotion that service designer aims to evoke are then passed into the EE module, which dynamically changes the content of the VR. We further articulate the ER and EE components in the next subsections.

The ER module is responsible of inferring, from a set of multi-modal data, the emotion that the user is most likely perceive. Note that data acquisition might be performed with the HMD, as well as with other supporting tools that do not prevent the VR experience (e.g., wearable devices). The ER module consists of the following layers:

• Feature Extraction: hand-crafted and learned features can be considered for each type of gathered data (e.g., acceleration of joints for body's movements, or spectrogram for voices);

• Fusion: this layer is meant to combine data, features and algorithms to maximally exploit the information contained in users' data, in order to increase the generalization of the ER module;

• Segmentation: this layer is meant to make algorithms designed to work on standalone signals also able to recognize emotions from a continuous • Emotion Classification: a supervised learning algorithm that is trained to perform a classification of users' emotions.

To summarize, the architecture of the ER module infers emotions from a continuous stream of users' generated data (and not, as commonly done in ER research, from standalone data, i.e., signals associated with a single emotion), and works both on single-mode (e.g., on users' movements only) and multi-mode manner (e.g., on a combination of users' movements and voice).

As for Emotion Elicitation (EE), this module is responsible to select the content of the VE based on i) the emotions detected by the ER module and ii) the emotions that designers aimed to evoke. An open research question is how this selection can be performed. Due to the complexity of the task, rule-based automatic adaptation of VEs should be firstly considered. However, modelbased adaptations might be considered as well (e.g., using machine learning).

Ref. [10] proposes an architecture to perform users' emotion-driven generation of a VE and it is, among previous works, the one closest to the aim of the proposed framework. The authors of [10] also validates the effectiveness of the architecture in the context of mental health treatment. Such architecture is designed to detect users' emotions from multi-mode data (similarly to our ER module) and, accordingly, to generate a VE to stabilize them, e.g., to induce calm (similarly to our EE module). However, this existing architecture makes use of a very simple algorithm (i.e., a linear regression), while our proposed framework aims at developing a richer module that includes the most efficient existing ER algorithms Then, in Ref. [10] the generated VE is a simple maze, while our goal is to develop several more complex VEs in various applicative scenarios.

The second relevant existing work that investigates the use of VR as a tool to perform ER and EE can be found in Ref. [11] , whose authors propose, as we do, an integrated system to perform both ER and EE in VR, and they make the following important claim: they are the first to apply machine learning strategies to perform ER in the context of VR. Given that Ref. [11] is a very recent PhD Thesis (dated April 2020), we argue that the proposed system can provide a significant contribution to the current state of the art. Indeed, Ref. [11] presents three main drawbacks that we aim to handle: i) the VE is static, while our framework aims to be dynamic and able to automatically adapt to users' current affective states; ii) only user's electroencephalogram is used as input of the ER module, while we envision the gathering of a larger set of users' bio-feedbacks; iii) a very simple machine learning algorithm is employed, while our framework is expected to consider a vast array of advanced machine learning algorithms. Most of the research on ER is done on single-mode and standalone data (see the recent survey [12] ), which carry acted and exaggerated emotions. Instead, the proposed framework allows to consider streams of multi-mode data (which introduce the challenge of identifying the onset and end of emotions) and to exploit the immersiveness of the VR experience to induce (and then, recognize) more spontaneous emotions. Ref. [13] shows evidence that the VR is more effective than traditional media to perform EE, and studies its influence on decision making processes.

In this Section, we describe the applicative areas where the VEE-loop can be employed, as well as the potential impact that it can have across several areas.

By enabling emotion-driven services and products, the VEE-loop opens the way to a wide spectrum of applications. We envision these potential applications to fall in three main areas, which we refer to as: 1) service delivery, such as education [14] and human-machine interaction and familiarization (https://v-machina.supsi.ch/), 2) customer experience, e.g., in marketing to understand what creates favourable and unique consumer-brand relationships [15, 5] , 3) research and development, both from academics and industries.

First, the VEE-loop can benefit the delivery of services where the information on users' emotional states is unavailable for some reasons. Example of applications in this category are remote schooling (e.g., due to Covid-19 restrictions) and virtual training and practice (e.g., due to expensive and dangerous machineries). In such scenarios, the emotional response of students (or trainers) can be monitored in order to dynamically adapt the virtual environment (or required tasks) and enhance users' learning experience.

Second, the VEE can be used to enhance customer experiences by providing VR scenarios where users can test products, while allowing companies to validate the capability of products to trigger specific emotions, even before their production. In such a way, designers could obtain emotional response from potential customers and tune the design accordingly in order to create better "tell brand stories". For instance, in interior design, the inferred emotions can be used to understand which factors reinforce people's well-being [16, 17] .

Finally, the VEE-loop has the objective of stimulating research to create innovative emotion-driven services capable of smoothing the transition towards an increasingly-digitalized society, as well as to advance the state of the art on the growing fields of emotion recognition and elicitation. We envision the development of research and applications in disparate areas, ranging from user experience (UX) design (e.g., to optimize users' spatial perceptions) to fine arts (e.g., in theatrical performances, to understand the relation between emotions and mechanisms of an embodied acting).

In light of the numerous potential applications described before, our pioneering solution can potentially impact various dimensions of our society. We distinguish five dimensions where the VEE loop can provide benefits, which we detail as follows.

Economical Impact The VEE loop finds applications in a countless number of industrial sectors, while providing potential economical advantages both in the production and in the marketing phases. For example, it can be used by experts in advertising to understand what reinforces unique brand association, or by designers to evaluate users' emotional response to the characteristics of a product before its tangible development. This allows designers to take more informed decisions, therefore reducing the risks (and associated costs) of creating unsuccessful products and services.

Social Impact The VEE loop can help to deliver more empathetic services using the VR, therefore bringing a high social impact across many different areas (e.g., remote schooling). Potential applications can also target the treatment of pathologies characterized by disorders on the emotional sphere (e.g., autism) and collective trainings in emergency situations.

Environmental Impact The VEE loop integrates emotional aspects into services delivered remotely, therefore increasing their adoption. This has the potential of enabling remote working and practices, thus, limiting unnecessary travels, and, in turn, reducing the emissions produced by means of transport.

Research Impact Our vision contributes to the research on ER and EE, and provides a tool that researchers can readily use in the studies relative to these fields. The VEE loop is a novel and timely solution that can be a potential cornerstone in many different projects (from the research-oriented to the more applicative ones), therefore enabling transversal collaborations between academy and industry.

Cultural Impact The VEE loop enables avant-garde cultural events delivered with the VR. For instance, stylistic choices of a cultural event (e.g., in theatrical representations) can be modified according to the emotional response of the audience (even if attending remotely) in real time and in an economicallysustainable manner. This asset can find application in several cultural scenarios, e.g., theatre, virtual city trip and museum virtual tours.

The VEE-loop inherits all the benefits of VR technologies. First and foremost, the flexibility of real-time contents modification. While this characteristic is typical of other digital media as well, the VR also guarantees a much more tangible experience to its users, giving them the impression of dealing with real products (experience that is not instead possible with other digital technologies). Moreover, the VR simulated environments significantly outperform other media in evoking emotions. In fact, the sense of presence experienced using the VR lead users to perceive more spontaneous emotions that, for this reason, are a more valuable feedback for the designers. These facts allow service and products designers to experiment a high number of stylistic (and also functional) choices, and to have reliable emotional feedback from their users. The fact that emotions are more spontaneous, however, also poses the challenge of correctly identify them. Indeed, most of the research on emotion recognition is based on the analysis of emotions that are voluntarily exaggerated and that, for this reason, are also easier to recognize. On the other hand, the VR allows to perform emotion recognition exploiting a high number of multi-mode data, either collected with the HMD itself (e.g., head and hands micro-movements and eye movements) or with other supporting acquisition devices (ranging from non-invasive motion capturing technologies, to simple wearable devices and microphones). This multitude of heterogeneous data can significantly benefit the emotion recognition task. Another aspect to consider is that emotions must be estimated from a stream of signals and not, as generally done in previous work, from stand-alone data. Therefore, a segmentation process is required to identify the instants of transitions between two different affective states, before their actual classification. To our knowledge, the problem of segmentation is extensively considered in the action recognition task, but quite unexplored in the emotion recognition one. Finally, the dynamic modification of the virtual content is currently done either manually or according to simple rule-based approaches. A significant research effort is required to explore more efficient automation strategies, such as those based on powerful machine learning algorithms.

The subconscious mind of the consumer (and how to reach it)

Affected: Emotionally engaging customers in the digital age

Enhance emotional and social adaptation skills for children with autism spectrum disorder: A virtual reality enabled approach

Estimating cognitive workload in an interactive virtual reality environment using eeg

Building customer-based brand equity: A blueprint for creating strong brands

The impact of perception and presence on emotional reactions: a review of research in virtual reality

Real vs. immersive-virtual emotional experience: Analysis of psychophysiological patterns in a free exploration of an art museum

Virtual reality learning environments for vocational education: A comparison study with conventional instructional media on knowledge retention

Feasibility of heart rate and respiratory rate estimation by inertial sensors embedded in a virtual reality headset

Toward emotionally adaptive virtual reality for mental health applications

Modelling human emotions using immersive virtual reality, physiological signals and behavioural responses

Emotion recognition and detection methods: A comprehensive survey

The feeling is real: Emotion elicitation in virtual reality

Music during lectures: Will students learn better?

Consumer engagement in a virtual brand community: An exploratory analysis

Emotions-based interactions: Design challenges for increasing well-being

Oltre l'interfaccia: emozioni e design dell'interazione per il benessere