key: cord-032466-1nfp1hcs authors: Gong, Liang; Söderlund, Henrik; Bogojevic, Leonard; Chen, Xiaoxia; Berce, Anton; Fast-Berglund, Åsa; Johansson, Björn title: Interaction design for multi-user virtual reality systems: An automotive case study date: 2020-09-22 journal: Procedia CIRP DOI: 10.1016/j.procir.2020.04.036 sha: doc_id: 32466 cord_uid: 1nfp1hcs Virtual reality (VR) technology have become ever matured today. Various research and practice have demonstrated the potential benefits of using VR in different application area of manufacturing, such as in factory layout planning, product design, training, etc. However, along with the new possibilities brought by VR, comes with the new ways for users to communicate with the computer system. The human computer interaction design for these VR systems becomes pivotal to the smooth integration. In this paper, it reports the study that investigates interaction design strategies for the multi-user VR system used in manufacturing context though an automotive case study. Due to the fast development in the domain of communication and an ongoing trend of digitization and digitalization, manufacturing enterprises are facing important challenges in today's market environments: a continuing tendency towards reduction of product development times and shortened product lifecycles. In addition, there is an increasing demand of customization, being at the same time in a global competition with competitors all over the world. This trend, which is inducing the development from macro to micro markets, results in diminished lot sizes due to augmenting product varieties (high-volume to low-volume production) [1] . To cope with this augmenting variety as well as to be able to identify possible optimization potentials in the existing production system, it is important to have a precise knowledge of the product range and characteristics manufactured and/or assembled in this system. In this context, the main challenge in modelling and analysis is now not only to cope with single products, a limited product range or existing product families, but also to be able to analyze and to compare products to define new product families. It can be observed that classical existing product families are regrouped in function of clients or features. However, assembly oriented product families are hardly to find. On the product family level, products differ mainly in two main characteristics: (i) the number of components and (ii) the type of components (e.g. mechanical, electrical, electronical). Classical methodologies considering mainly single products or solitary, already existing product families analyze the product structure on a physical level (components level) which causes difficulties regarding an efficient definition and comparison of different product families. Addressing this Virtual reality (VR) technology is attracting increasing attention in various industries during the past decade. It is the result of the ever-matured VR technology as well as the great potential to improve existing practices in different industries. The manufacturing industry is among those that actively pushing to find out the applications of using VR. There are growing studies shows that much effort are spending on the VR applications in the manufacturing industry [1]- [4] . It has shown that VR technologies have great advantages to improve areas like factory layout planning, product design, training, etc., especially for globally distributed manufacturing companies that have different functional team located in different parts of the world [3] , [5] - [8] . However, while the studies have showed the promising future of VR usages in manufacturing, they have also pointed out the new challenges that are hinder the VR integration process [7] , [9] , [10] . The usability related issues of VR systems is one of the most important to be addressed, as it would affect end users acceptance, which would ultimately affect the process of VR integration in manufacturing. These issues are resulted from the new interaction mediums that comes with the VR systems. Different from the "window, icon, menu, and pointing device" (WIMP) or Direct Manipulation interaction style, VR systems use handheld controllers, haptic devices or even voice and gesture recognition for users to interact with the systems. Jacob et al. pointed out that there is a lack of established interaction design principles and frameworks for post-WIMP systems [11] . In this study, we have chosen a globally distributed manufacturing company as the case to study the different possibilities of interaction design approaches for the multi-user VR system used in the design review process. Two iterations of VR system development and evaluation have been conducted. Virtual reality (VR) technology is attracting increasing attention in various industries during the past decade. It is the result of the ever-matured VR technology as well as the great potential to improve existing practices in different industries. The manufacturing industry is among those that actively pushing to find out the applications of using VR. There are growing studies shows that much effort are spending on the VR applications in the manufacturing industry [1]- [4] . It has shown that VR technologies have great advantages to improve areas like factory layout planning, product design, training, etc., especially for globally distributed manufacturing companies that have different functional team located in different parts of the world [3] , [5] - [8] . However, while the studies have showed the promising future of VR usages in manufacturing, they have also pointed out the new challenges that are hinder the VR integration process [7] , [9] , [10] . The usability related issues of VR systems is one of the most important to be addressed, as it would affect end users acceptance, which would ultimately affect the process of VR integration in manufacturing. These issues are resulted from the new interaction mediums that comes with the VR systems. Different from the "window, icon, menu, and pointing device" (WIMP) or Direct Manipulation interaction style, VR systems use handheld controllers, haptic devices or even voice and gesture recognition for users to interact with the systems. Jacob et al. pointed out that there is a lack of established interaction design principles and frameworks for post-WIMP systems [11] . In this study, we have chosen a globally distributed manufacturing company as the case to study the different possibilities of interaction design approaches for the multi-user VR system used in the design review process. Two iterations of VR system development and evaluation have been conducted. Available online at www.sciencedirect.com Virtual reality (VR) technology is attracting increasing attention in various industries during the past decade. It is the result of the ever-matured VR technology as well as the great potential to improve existing practices in different industries. The manufacturing industry is among those that actively pushing to find out the applications of using VR. There are growing studies shows that much effort are spending on the VR applications in the manufacturing industry [1]- [4] . It has shown that VR technologies have great advantages to improve areas like factory layout planning, product design, training, etc., especially for globally distributed manufacturing companies that have different functional team located in different parts of the world [3] , [5] - [8] . However, while the studies have showed the promising future of VR usages in manufacturing, they have also pointed out the new challenges that are hinder the VR integration process [7] , [9] , [10] . The usability related issues of VR systems is one of the most important to be addressed, as it would affect end users acceptance, which would ultimately affect the process of VR integration in manufacturing. These issues are resulted from the new interaction mediums that comes with the VR systems. Different from the "window, icon, menu, and pointing device" (WIMP) or Direct Manipulation interaction style, VR systems use handheld controllers, haptic devices or even voice and gesture recognition for users to interact with the systems. Jacob et al. pointed out that there is a lack of established interaction design principles and frameworks for post-WIMP systems [11] . In this study, we have chosen a globally distributed manufacturing company as the case to study the different possibilities of interaction design approaches for the multi-user VR system used in the design review process. Two iterations of VR system development and evaluation have been conducted. The research process as well as the findings are presented and discussed in the rest of the paper. VR technology is not new; the first VR system was successfully implemented with a HMD that presents a user with stereoscopic 3D view slaved to a sensing device, which tracks the user's head movement [12] . Ever since then, research effort are continuing spending on this area. Korves and Loftus described VR systems based on the different setups and thus categorized them as:  Desktop system.  Wide-screen projection system.  Immersive CAVE system.  Immersive VR system using HMDs [13] . Thanks to the latest advancement in hardware and software of VR devices, immersive VR system with HMDs is becoming more and more viable in many industries, thus it again attracted much attention in academia and industry during the last 10 years. In this emerging domain of VR, there are many studies on the different issues of integrating it to the manufacturing process. Most of them have shown that VR can improve existing practice in the manufacturing industry. For example, Wiendahl and Harmschristian have shown that immersive VR is an important tool for collaborative factory planning, especially when multiple viewpoints of users are visualized [14] . Menck stated that the VR based collaborative planning tool can extend the communication and facilitate cooperation beyond existing organizational boundaries, which would reduce the complexity of work and increase the work efficiency [15] . At the same time, some studies also pointed out that the VR systems is also bringing new challenges for both developers and end users [10] , [16] , [17] . Those challenges can be categorized into two major groups. One is associated with creating the realistic virtual environment. The challenge lies on the complexity of generating realistic enough models that can represent the real world so that enough context is provided to perform the intended tasks in VR. This involves the data source integration and data compatibility issues when reuse data already existed for other purposes in VR [18] - [20] . Another is the usability related issues of VR systems. It is reported that some early users of VR systems have difficulty understanding the interaction logic and some even feel motion sick [21] , [22] . Instead of the already established style of human computer interaction for desktop or mobile application, the VR interaction design is new and need further studies to develop supporting guideline and framework. Another worth noting point is that, most reported VR systems are designed for single user. In the real world, multiuser engagement is the norm for most activities. Therefore, one argument is that if it is not multi-user, then it is not virtual reality. Based on the above description, this study has chosen the immersive VR system with HMDs as the basic setup to study the interaction design of multi-user VR systems for manufacturing. An automotive company was selected as the case to study how interaction design would affect manufacturing companies from adopting the multi-user VR system. The company's research and development department is located in Gothenburg of Sweden, while the manufacturing plants are situated across different regions of China. It makes the selected company a perfect case for the purpose of this study. The team that was actively involved in this study was the manufacturing engineering (ME) team. They are mainly responsible for developing all body-in-white (BIW) process. Therefore, the industrial base of the study was set as fixture design and review process using multi-user VR system. The first step to develop the multi-user VR system is to gather the data needed for the intended task. For the purpose of this study, we used two types of data sources: CAD models of the new fixture design and point cloud data of the factory environment. The CAD models are created by ME team using CATIA TM [23] in the format of jt. However, as the jt files contains many irrelevant information for the VR scenario such as the internal structure of the fixture, the provided jt files were further optimized using PIXYZ [24] by removing unnecessary surfaces and converting the files to the fbx format which can be directly imported to the VR development platform. The point cloud data was captured onsite of their Chinese manufacturing plants using 3D laser scanning technology. It is a realistic version of virtual representation of the real factory environment. Studies have shown the point cloud environment provides more accurate contextual information compare to the simplified version of the completely computer generated CAD models. The disadvantage of point cloud data is that there is no surfaces or triangle meshes, but millions of points, which requires excessive computing power to rendering them in real time for VR system. It is also difficult to interact with the point cloud data in VR due to this limitation. Therefore, the point cloud data was processed with adjusted level of density based on the area of interest. Simple surfaces such as floor and walls were also meshed using mathematic algorithm. Unity3D [25] was chosen as the development platform to integrate different data sources and program the needed functionality. The Unity plugin PUN2 [26] was used to handle the network and synchronization for different users. HTC Vive headsets were used as the VR hardware throughout the study. In this study, two versions of the multi-user VR system were developed in sequence. The pilot version aimed at testing the feasibility of the multi user concepts as well as getting some feedback of the desired functionalities for the second version. Therefore, minimal effort were spent on user interface design, but focus on making the connection reliable and synchronization with low latency. The VR view is shown in Fig. 1 . A second version was later developed based on the feedback gathered through the pilot workshops with additional features such as customized user avatars. Moreover, two sets of interaction design approaches were implemented to further understand user's preference in multi-user VR system. In addition to the common practice that uses 3D trackable controller to trigger different functionalities in VR (illustrated in Fig. 2 ), 2D graphic user interfaces (GUI) were added on top of the projected screen so that conventional mouse and keyboard can also be used to activate the same functionalities (illustrated in Fig. 3 ). The pilot version were tested in two occasions. It was first tested for the cross continent connection from Gothenburg in Sweden with fellow colleagues in National Institute of Standards and Technology (NIST) in United States. This test was to verify the stability of the connection and the quality of real time synchronization of both objects and audio. After the verification, it was later tested in a workshop hosted at university site in Sweden. Five engineers from the case company as well as two senior researchers from Fraunhofer-Chalmers Centre (FCC) participated in the workshop. All participants tested the multi-user VR system in pairs and performed the intended design review task in VR. After the hands-on experience with the system, a following-up discussion was held to get feedback and potential improvement for the second version. The second version was developed based on the feedback collected from the pilot workshops. Then it was tested in case company's facility in Gothenburg. There were 14 participants from different groups of the case company joined the test. The participants' role ranging from process developer to vice president of the case company. A tutorial session was held to introduce the study and the basic functions of the multi-user VR system. Then all participants tested it in pairs to perform the design review task (illustrated in Fig. 4) . After the test, each participants filled in a scale-rating questionnaire regarding their general experience of the interaction design and a semistructured interview was held to collect qualitative data for better analysis of the rating result. The two tests have shown that the multi-user VR system can provide enough context for relevant engineers from different sites to collaborative perform the fixture review process in VR. The connection is stable while objects of interest and audio synchronization are reliable. The scale-rating questionnaire are a series of statements for the participants to rate based on their experience. The results are analyzed with the qualitative feedback from the semistructure interview. A portion of the questionnaire results are illustrated in Fig. 5 . With the statements listed in the vertical axis. It shows that all participants believe this type of multi-user VR system would be great benefit for their daily work. The customized user avatars allow users to easily identify their colleagues in VR, therefore increased their feeling of being presence in the virtual environment. Most participants still prefer controller interaction than the 2D GUI. But when there is an admin role that supervise the VR review session from the projected screen, then it would be preferable to have the 2D GUI for the admin to control the review process. When it comes to the question whether all users should have the same functions, the view are diverse. While 10 out of 16 believe the point cloud data make the virtual model more realistic, some questioned the necessity, especially for design review tasks that are independent from the factory environment. In this study, a multi-user VR system was developed to support the design review process for globally distributed manufacturing companies. We focused on the interaction design of such system through two iterations of development and evaluation. It has affirmed interaction design of VR systems is of great importance for manufacturing companies to widely adopt and benefit from the latest advancements of VR technologies. It is generally believe that the quality of the VR environment is pivotal to the user experience, which would affect whether VR systems can be widely adopted and used in manufacturing companies [27] - [29] . To put it another way, previous findings emphasis the importance of creating realistic virtual environment to attract users to switch from conventional work procedure to VR. The feedback we collected partly affirm it. As features such as customized user avatars and point cloud factory environment, which would improve the realistic level of the VR environment, received positive ratings from the participants. However, the test result also hinted that not all the design review work would need those features. Another aspect that need to consider is that those features would take longer time and much more effort to implement. Therefore, it should be judged by the individual cases whether to add those qualityimprovement features in the VR systems. The new interaction medium brought by the VR system is a challenge for both developer and end users. Today, most people are used to the WIMME interaction from the personal computer as well as touch screen in smart phone and tablets. Due to the specific setup with HMD in VR systems, it is impossible to transfer the matured interaction design approaches from those platforms directly to the VR world. Instead, handheld controllers with multiple buttons and various haptic devices are created to support the interaction in VR. However, these devices and interaction logic are new to most of the users. Previous studies have shown that this might be one of the hinder for VR usage [11] , [30] , [31] . The result of this study shows that the new interaction devices did make it difficult for new users to engage in the tasks at the beginning, but it also shows that going backward to find ways of using those matured medium in VR is not a favorable option either. Most users expressed the idea that they can learn the new ways rather fast and the tests also proved that most of them managed to use the different functions smoothly at the end. However, there is no established knowledge or standards that they can be referred to when they were first presented with a VR system. This should be improved by joint effort from academia and VR industry to further study and establish uniform framework for interaction design in VR. Additionally, technologies such as voice and gesture recognition are becoming increasingly viable with the latest development in deep learning and artificial intelligence (AI) [32] . Many practitioners and researchers are starting to integrate those new technologies to explore new ways of user interaction in the VR [33] , [34] . Due to the complex nature of manufacturing environment, there might be some negative factors such as the noise level of factory are too high for voice recognition. It is still worth the effort to have future studies that can explore voice and gesture interaction for VR systems used in the manufacturing industry. This study has set out to investigate appropriate interaction design approaches for multi-user VR system used in the ME process. A multi-user VR system was developed and evaluated to in two iterations with academia and industrial partners. The results indicates that multi-user VR system can complement and improve the existing ME processes. New interaction medium comes with VR system is a challenge for the wider usage in manufacturing, but also an opportunity to further develop principles and standards of interaction design so that the manufacturing companies can also enjoy then benefit brought by the latest advancement of VR technologies. An immersive and collaborative visualization system for digital manufacturing Introducing quantitative analysis methods into virtual environments for real-time and continuous ergonomic evaluations A novel facility layout planning and optimization methodology Adaptation of High-Variant Automotive Production System Using a Collaborative Approach Virtual manufacturing as a way for the factory of the future The use of virtual reality techniques during the design process: From the functional definition of the product to the design of its structure Improvement of manufacturing processes with virtual reality-based CIP workshops Virtual and augmented reality technologies for product realization Virtual factory: An integrated framework for manufacturing systems design and analysis Virtual reality applications in manufacturing industries : Past research , present findings , and future directions Reality-Based Interaction: A Framework for Post-WIMP Interfaces A head-mounted three dimensional display The application of immersive virtual reality for layout planning of manufacturing cells Virtual factory design -a new tool for a co-operative planning approach Collaborative factory planning in virtual reality A Survey on Human-Computer Interaction in Mixed Reality State of the art of the virtual reality applied to design and manufacturing processes Rationalizing virtual reality based on manufacturing paradigms Virtual reality-based approach to manufacturing process planning Virtual reality approaches for immersive design Virtual Reality and 3D Imaging to Support Collaborative Decision Making for Adaptation of Long-Life Assets Real walking in virtual environments for factory planning and evaluation CATIA 3D Experience PiXYZ Software What is Unity? Photon Unity Networking framework for realtime multiplayer games and applications A Novel VR Tool for Collaborative Planning of Manufacturing Process Change using Point Cloud Data A fast parametric deformation mechanism for virtual reality applications Virtual plastic injection molding based on virtual reality technique Immersive authoring of Tangible Augmented Reality content : A user study Tangible User Interfaces: Past, Present, and Future Directions Deep learning-based human motion recognition for predictive context-aware human-robot collaboration MR : an interdisciplinary framework for mixed reality experience design and criticism Cognitive Load by Context-Sensitive Information Provision Using Binocular Smart Glasses in an Industrial Setting This work is funded by VINNOVA (Swedish Agency for Innovation Systems) through the SUMMIT project. This work is also within the Sustainable Production Initiative and the Production Area of Advance at Chalmers University of Technology. The support is gratefully acknowledged.I would also like to express my sincere thanks to all the medical staff that are fighting with the covid-19. Keep safe and victory is ours!