key: cord-0435323-7j1fi5s9 authors: Caputo, Ariel; Giachetti, Andrea; Abkal, Salwa; Marchesini, Chiara; Zancanaro, Massimo title: Real vs Simulated Foveated Rendering to Reduce Visual Discomfort in Virtual Reality date: 2021-07-04 journal: nan DOI: nan sha: 59000e7c34016d2aab31b6f5edf04de39d355f8c doc_id: 435323 cord_uid: 7j1fi5s9 In this paper, a study aimed at investigating the effects of real (using eye tracking to determine the fixation) and simulated foveated blurring in immersive Virtual Reality is presented. Techniques to reduce the optical flow perceived at the visual field margins are often employed in immersive Virtual Reality environments to alleviate discomfort experienced when the visual motion perception does not correspond to the body's acceleration. Although still preliminary, our results suggest that for participants with higher self-declared sensitivity to sickness, there might be an improvement for nausea when using blurring. The (perceived) difficulty of the task seems to improve when the real foveated method is used. The perceptual mismatch in sensory cues between the vestibular and the visual systems is known to induce in human subjects the so-called motion sickness that may strongly affect users' experience in Virtual Reality (VR) where they navigate synthetic environments while being still [9] . The visual discomfort can harm the diffusion and acceptance of immersive VR applications [13] . As discomfort is often related to the optical flow perceived at the margins of the field of view, a potential solution to reduce this effect is to reduce the amount arXiv:2107.01669v1 [cs.HC] 4 Jul 2021 An approximation of foveated rendering can be obtained by assuming that the user's gaze is consistently directed along the virtual camera axis. Therefore, image quality is varied with the distance from the center of the viewpoint [12] . Nowadays, low-cost eye-tracking devices are integrated into several Head Mounted Displays for Virtual Reality, making it possible to implement a truly foveated rendering, modifying the image's appearance far from the actual user's gaze point. A few recent papers [11, 1] proposed the use of eye-tracking to implement foveated rendering for the reduction of motion sickness. There is some empirical evidence on the positive effect of optical flow reduction techniques (for example, [1] , [5] ). Yet, the advantage of using a foveated approach by tracking the user's gaze rather than a simulated one (assuming that the gaze point coincides with the image center) has not been thoroughly investigated, at least for the blurring technique. In this work, we explicitly investigated the differences between real and simulated foveated blurring rendering while moving on a roller coaster in an immersive VR environment. Figure 1 exemplifies the effect seen by users in the two foveated conditions with respect to a view when no effects have been applied. To engage participants in being actively scanning the environment, we asked them to perform a recognition task. We did not expect improvements in the recognition task since it was just designed to engage participants visually. The blurring effect was not so strong to prevent recognition of the surroundings. We expected that both real and simulated foveated blurring reduced in some respect visual discomfort or the task difficulty and real foveated being better than the simulated one. Specifically, we measured visual discomfort using a reduced version of the validated questionnaire proposed by Sevinc and colleagues [15] that consider three scales for nausea, oculomotor fatique and disorientation. A large amount of research work has been dedicated to the study of visuallyinduced motion sickness (VIMS) [9, 14] . Several authors agree on the necessity of finding good strategies to reduce this effect in virtual environments, and different approaches based on specifically designed rendering solutions have been proposed. Carnegie and Rhee [5] proposed a dynamic depth of field blurring to reduce the amount of perceived motion showing a reduction of nausea and disorientation discomfort in a user study. Fernandes and Feiner [7] proposed a dynamic reduction of the field of view that seems to help the participants of a user study to stay in the Virtual Environment longer and feel more comfortable than they did in the control condition. Buhler et al. [3] proposed two visual effects: the "circle effect" (blending the view of a different static camera outside a circle) and the "dot effect", (adding rendered dots in peripheral vision that counteracts the virtual motion of the environment when the user is moving), to alter the visualization in the peripheral area of the perceived visual image aiming at reducing effective optical flow. The study did not show significant effects for reducing VR sickness due to the large variability of the effects. Cao et al. [4] proposed the addition of static rest frames and dynamic rest frames (with velocity-dependent opacity) to limit sickness. Nie et al. [10] used adaptive blurring of the background. The method detects not salient regions and applies selective blurring only to them. FOV reduction, DOF blurring, and rest-frame have been compared in a racing game task in [16] . A comparison of some of those techniques is proposed in [6] . The study suggested a weaker effect of the rest frame with respect to other methods. A selection of the methods has also been implemented in a Unity toolkit and made available in an open-source repository [2] . As suggested in [11] , gaze estimation might be an effective approach to select the regions where applying optical flow reduction. The use of eye-tracking to control sickness reduction algorithms has been proposed first in [1] . In this work, the authors compare foveated vs non-foveated large FOV reductions without specific tasks performed by the users. In this work, we aimed to assess the effect on real vs. simulated foveated blurring empirically. Our study complement and extend the results provided by Carnegie and Rhee [5] on foveated blurring while offering empirical support to the approach of using "real" foveated as initially proposed by Pai and colleagues [11] . The VR environment used for the study realizes a roller coaster simulation with a heavily textured scene with a relevant optical flow at the margins of the visual field to foster sickness ( Figure 2 ). This setting has been chosen because of its natural form of constrained navigation, limiting possible biases related to user control. The simulation program was implemented in the Unity framework. The VR experience was created to use HTC Vive Pro Eye Head-Mounted Display (HMD) equipped with a 120Hz eye-tracking technology. In each ride, the user goes through a VR roller coaster experience in which they are sitting in a cart, moving automatically through the course that consisted of a set of rails organized in a closed loop. The course features a variety of turns, uphill and downhill sections, and speed variations introduced to induce motion sickness. A full lap around the course track takes about 160 seconds. The simulation can be executed with three different visualization modes: foveated blurring, in which only a circular area is in focus (centered on the actual user's gaze point as tracked by an eye-tracking device embedded in the HMD) while the rest of the image is blurred; simulated foveated blurring, similar to the above but the gaze point is assumed fixed in the middle of the rendered frame; and no effects, on which no blurring effects are applied and all the scene in focus. For both the blurring modes, the in-focus area is a circle with radius r while, outside of it, a Gaussian blur with standard deviation σ is applied. The effect is smoothed from the edge of the in-focus circle with a ramp up parameter that scales linearly with the distance from the center of the circle. The parameters r and σ have been tuned in a set of pilot studies with participants not involved in the main study. A set of 35 visual targets (similar to those visible in Figure 1 ) are placed at different heights and angles on the sides of the track. Each one displays a string of 4 letters. Five of the targets contain the "X" character, and they have to be identified by the users while riding the roller coaster. This recognition task is relatively easy, but it requires active visual search and attention. The study design was a full factorial experiment with three within-subject conditions based on the visualization modes described above: (i) (Real) Foveated, (ii) Simulated foveated, and (iii) No effects The study was carried out in the VIPS lab at the University of Verona and received ethical clearance by the Human Research Approval Committee of the same university. The hypotheses were the following: -H1 : in general, foveated blurring reduces visual discomfort and the (perceived) task difficulty, with respect to the no effects condition (as partially demonstrated by Carnegie & Rhee [5] ); -H2 : the real foveated blurring technique reduces visual discomfort and task difficulty with respect to simulated blurring (as proposed, without empirical evidence, by Pai et al. [11] ) Thirty-eight (38) participants have been recruited (20 females with age between 18 to 25, 18 males with age between 18 to 25, and 2 males with age between 26 to 30). All of them had normal or corrected to normal vision. Twenty (20) participants had no previous experience with VR, and 13 had only one previous experience. Each participant was tested individually. Upon arrival, s/he received the informed consent form, the privacy form, and the information about the COVID-19 procedure adopted. S/he was also informed that the experiment can be interrupted at any moment (nobody actually did) and was asked to fill the pre-task questionnaire. Then, the participant was invited to wear the HMD and perform the eye tracker's calibration. This procedure provided an initial exposition to VR for the participants without previous experience and assessed their willingness to continue the study. The participant was then assigned to a randomized order and repeated the roller coaster experience in the three conditions. The positions of the targets to recognize were also randomized within the conditions and the participants. Each condition involved one single ride in the roller coaster in one of the visualization modes described above, followed by a 3-minutes break before starting the next ride. For the recognition task, the participants had to verbally report to the experimenter when they located a target with an "X" inside. The participant was eventually thanked and dismissed. No compensation was provided for the participation. A pre-task questionnaire was used to collect demographic information (gender and age), previous experience with VR, self-reported vision problems, selfreported sensitivity to sickness (measured on a 4-item scale from never to very often). Measures of visual discomfort and task difficulty have been collected after each condition. For measuring visual discomfort, a reduced version of the Simulator Sickness Questionnaire (SSQ) [15] has been used. Specifically, we used the items General discomfort, Fatigue, Eyestrain, Difficulty focusing, Sweating, Nausea, Blurred vision (all 4-point Likert scales from none to severe) which correspond to the scales of Nausea, Oculomotor, and Disorientation. In order to measure the perceived task difficulty, a single item on a 4-point Likert scale (from very simple to very difficult) has been used. Although the variability in the data is very high (see Table 1 ), the three SSQ scales have a good level of reliability (their Cronbach's alpha values are 0.71 for the Nausea scale, 0.84 for the Oculomotor scale and 0.76 for the Disorientation scale). It is worth noting that at least part of the variability is due to the selfdeclared sensitivity to sickness: its Pearson correlation with the Nausea scale is r = 0.48, p < 0.01; with the Oculomotor scale is r = 0.56, p < 0.01 and with the Disorientation scale is r = 0.59, p < 0.01. A first attempt to fit a Linear Mixed Effect Model on each scale by using both condition and the pre-test variables of gender, sight, and sensitivity to sickness as fixed effects with the individual participants as random effects confirms a statistically significant effect of sickness on each scale and also an effect of condition on the Nausea and Disorientation scales (but not for Oculomotor scale). Neither gender nor sight have statistically significant effects. Friedman non-parametric tests (using the individuals as grouping variable) revealed a difference between the three conditions on the Nausea scale (Friedman, f = 10.84, p < 0.01), and Conover post-hoc tests confirm a significant difference between the no effects condition and the other two (respectively p = 0.0054 for foveated vs no effects and p = 0.0054 for simulated vs no effects) but no difference between foveated and simulated. No statistical differences were found for the other two scales (respectively, Friedman f = 1.54, p = 0.46 for Oculomotor and Friedman f = 3.36, p = 0.19 for Disorientation). A significant effect has been found for the difficulty of task (Friedman f = 12.12, p < 0.01). Conover post-hoc tests confirm a difference between foveated and simulated conditions (p < 0.01) and between no effects and simulated (p < 0.01) but not between foveated and no effects. To better discount the effect of self-declared sensitivity to sickness, we decided to categorize sickness as low when the value of self-declared sensitivity to sickness was equal to 1 and high for the values 2,3 and 4. This procedure partitioned the data into two groups of 21 participants with low sensitivity to sickness and 17 participants with high sensitivity to sickness. Among the participants with low sensitivity to sickness, a Friedman nonparametric tests (using the individuals as grouping variable) revealed a difference among the three conditions for the Nausea scale (Friedman, f = 9.29, p < 0.01), and Conover post-hoc tests confirm a significant difference between the no effects condition and the other two (respectively p < 0.05 for foveated vs no effect and p < 0.01 for simulated vs no effects) but no difference between foveated and simulated. No statistical differences were found for the other two scales (respectively, Friedman f = 0.12, p = 0.94 for Oculomotor and Friedman f = 0.047, p = 0.98 for Disorientation). A significant effect has been found for difficulty of task (Friedman f = 8.16, p = 0.01). Conover post-hoc tests confirm a difference between foveated and simulated conditions and between no effects and simulated but not between foveated and no effects. For the participants with (self-declared) high sensitivity on sickness, there is no difference for the Nausea scale (Friedman f = 2.63, p = 0.27) nor for the Oculomotor scale (Friedman f = 4.38, p = 0.11) and the task difficulty item (Friedman f = 4.8, p = 0.09). There is a statistically significant effect for the Disorientation scale (Friedman f = 7.66, p < 0.05). Conover post-hoc tests reveal a difference between foveated and simulated but not between those two and no effect. As expected, there are no differences in the recognition task: 94% of the participants recognized more than 4 targets and only 5 participants recognized 3 or less. Our results, although still preliminary, provide evidence that both techniques of foveated blurring reduce visual discomfort (hypothesis H1), specifically for nausea, while the real foveated reduces the perceived task difficulty (hypothesis H2), but only for users with low sensitivity to sickness. Participants with high sensitivity for sickness do not seem to benefit from the blurring techniques for reducing nausea or the perceived task difficulty. Still, we found a small effect on reducing disorientation. Although still preliminary, our results consider the moderating effect of individual sensitivity to sickness and provide some evidence that it plays a role in the foveated rendering effect. In this respect, it contributes with new evidence to the growing body of research. Indeed, the variations in the scores, although small, were, at least for the Nausea and Disorientation scales in line with those found in other studies [1] , [5] . Differently from previous works, we had noticeable effect only on the Nausea and Disorientation scale and not on the Oculomotor scale for the SSQ. That might be due to the constrained navigation or the short duration of the ride that could have limited the VR experience's sickness effect. Another explanation might be that our task required more cognitive attention than simple navigational tasks used in other studies, and that may have an impact on the perception of ocular strain [8] . Another novel result is that we also measured the effect on the perceived task difficulty and observed a small but significant effect, particularly in differentiating real from simulated foveated blurring. Our experiment had some weaknesses and limitations that we plan to overcome in future studies. The single session duration was perhaps too short to create a relevant sickness on many subjects. The number of subjects tested was higher than in other studies but still limited;. It would be better to have a larger number of participants would allow to usinge more powerful statistical tools. Both the sensitivity level of sickness and the experienced sickness were self-reported while in further studies they should be objectively measured. For active recognition, it would also be helpful to use a more challenging task. Although we still believe that an active task helps control the participants' engagement with the environment, our task was probably too easy. Finally, an important issue to be considered is related to the accuracy and the latency of the eye tracker. Recent studies [17] revealed that the device we used is less precise and accurate far from the center of the headset FOV. The latency of the tracking could, in principle, also limit the sickness reduction. In conclusion, our results suggest that for participants with higher self-declared sensitivity to sickness, we found some evidence that the foveated technique provides a small benefit on disorientation for simulated blurring but not enough to improve with respect to applying no effects. For participants with lower sensitivity to sickness, there might be an improvement for nausea when using blurring. Still, the specific technique does not matter, although the (perceived) difficulty of the task seems to improve when the real foveated method is used. The effects of the differences are minimal (and the sample is small); therefore, the present study results should be considered an indication for further research rather than robust evidence. Nevertheless, we believe that this study contributes to the ongoing discussion on the effects of foveated techniques and, in particular, on the opportunity to implement "real" foveated using eye-tracking. The difference found in our study between participants with different levels of sensitivity to sickness has never been reported in the literature and deserves further investigation. The effect of a foveated field-of-view restrictor on vr sickness Gingervr: An open source repository of cybersickness reduction techniques for unity Reducing vr sickness through peripheral visual effects Visually-induced motion sickness reduction via static and dynamic rest frames Reducing visual discomfort with hmds using dynamic depth of field Software techniques to reduce cybersickness among users of immersive virtual reality environments Combating vr sickness through subtle dynamic fieldof-view modification A review on ocular biomechanic models for assessing visual fatigue in virtual reality Research in visually induced motion sickness Analysis on mitigation of visually induced motion sickness by applying dynamical blurring on a user's retina Gazesim: simulating foveated rendering using depth in eye gaze for vr Towards foveated rendering for gaze-tracked virtual reality User acceptance of virtual reality: an extended technology acceptance model Factors associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis Psychometric evaluation of simulator sickness questionnaire and its variants as a measure of cybersickness in consumer virtual environments Virtual reality sickness mitigation methods: A comparative study in a racing game Accuracy and precision of the htc vive pro eye tracking in head-restrained and head-free conditions Acknowledging the limitations of the eye-tracking device [17] , the recognition task has been designed in such a way that visual search does not require frequent and fast head movements.