key: cord-0985798-ohd6yk4h authors: Bran, Emanuela; Bautu, Bautu; Popovici, Dorin Mircea title: Towards a Sustainable Future: Ubiquitous Knowledge Mixed Reality Museum date: 2020-12-31 journal: Procedia Computer Science DOI: 10.1016/j.procs.2020.09.266 sha: 1892d67b5e0671d4552cf237dec449329e522a7c doc_id: 985798 cord_uid: ohd6yk4h An Intelligent Museum built under the Ubiquitous Computing paradigm, where services follow the user seamlessly across different environments, emerges as a new type of digital institution present beyond its physical limits: the Ubiquitous Museum. Mixed Reality provides the means of harvesting knowledge from memory institutions and smart devices connected to the Internet of Things. At the same time, it enables embedding multimodal contents from the Museum across the Intelligent Environment. We research the user experience of knowledge dissemination through the Ubiquitous Museum. We exemplify how our work aims to help achieve this. We also discuss how using embedded Museum contents helps sustainable development. According to the new paradigm of Ubiquitous Computing [1] where services follow the user seamlessly across Intelligent Environments, the museum experience may undergo a paradigm shift: "Make the world a museum". Analyzing the history of museums, we may identify several phases. Museums started as a brick and mortar institution According to the new paradigm of Ubiquitous Computing [1] where services follow the user seamlessly across Intelligent Environments, the museum experience may undergo a paradigm shift: "Make the world a museum". Analyzing the history of museums, we may identify several phases. Museums started as a brick and mortar institution where objects where brought by the imperial powers in order to be guarded as a treasure for the future generations. The trend was then adopted worldwide in order to preserve and disseminate the national heritage. As technology advanced, Smart Museums [2] started to embed new ways of interaction inside the museum and provide interaction at a distance over the internet [3] . Today, in a modern museum, we may take a guide and use walking stations in order to communicate. Different exhibited items have markers attached that provide a link to additional multimedia information. Some central spots in a museum use human computer interaction technologies that provide an immersive experience for the user. The internet has also allowed the public to remotely get a glimpse inside the museum and take part in different edutainment activities [4] . In the digital age, institutions evolved from e-Institution (i.e., electronic) to m-Institution (i.e., mobile) and now to u-Institution (i.e., ubiquitous) [5] . In this paper we tackle the concept of Ubiquitous Museum as a new type of digital institution. The Covid-19 pandemic brought several changes to the life we used to live. Extended periods of quarantine and social distancing made physical interaction with people and exploration of different places impossible. On the other hand, this disruption of our comfort zone was an opportunity to question our way of living, and to give consideration for our impact on nature and for other people all around the world. As we have witnessed, an incident in China has an impact in Europe, and a medical problem has economical consequences. In order for society to move into the right direction, our good thoughts and emotions should be harvested in a fruitful manner, by acting on them consciously towards a sustainable future. In this subsection it is presented how the idea of a Ubiquitous Museum may be used in this direction. The word museum comes from Latin and is referring to the Greek Temple of the Muses, the inspirational goddesses of science and arts. If we would see a muse in everyday encounters and good causes, we could channel this inspiration towards a positive change. Museums are places of science and art, and ultimately of knowledge and experience. They create by excellence bridges between people, cultures, nature and beyond. There is no need for this experience to be taking place between walls. The world may become a museum, an inspiration for its own evolution. Enabling users to access interactive digital Museum resources through the Internet, and making knowledge dissemination available at multiple places, will provide several positive changes. In times of crisis people will be able to access the resources they need, and these contents will still be available when the situation goes back to normal, benefitting everybody in the way that the Internet made democratization of information possible. Less crowded museums may be only a health benefit for the present. Nevertheless, in the long term they will provide a more qualitative experience for each visitor. Places which were not visited before may attract more people by implementing innovative knowledge dissemination technologies. The balanced distribution of places of interest around the world will encourage sustainable development and protection of our heritage and the environment. Our interest in understanding the latest technologies in delivering an extended reality [6] museum experience motivated us to research the Cultural Heritage Digitalization and Virtual Multimodal Museum initiatives of the European Union [7] , along with another initiative of preserving Indigenous Knowledge by empowering its people to contribute to the process. These initiatives provide insight into the social impact of the Extended Reality Museum. We wish to contribute to accomplishing world initiatives using our proposed conceptual Mixed Reality (MR) platform. The platform's main features are highlighted in the "Our Work" section. The European Union considers Cultural Heritage digitalization a top priority for sustainable social and economic development. Under the Horizon 2020 funding the Virtual Multimodal Museum [7] was created as a major coordination and support action across the field of Virtual Museums, within the overall context of the European policy and practice on Digital Cultural Heritage. A group of world leading experts in the field designed a Manifesto and Roadmap for Action concerning high quality policies, strategic decision making and the utilization of breakthrough MR technology [7] . The project aims to nurture an evidence-based growth and development impacted by Virtual Museums on South European countries affected by recession. Traditional knowledge is mostly based on accumulations of empirical observation and on interaction with the environment. Traditional knowledge includes types of knowledge about traditional technologies of subsistence such as tools and techniques for hunting or agriculture, midwifery, ethno-botany and ecological knowledge, traditional medicine, celestial navigation, ethno-astronomy and climate. It has been either orally passed between generations or embedded in stories, legends, folklore, rituals, songs, and laws. This knowledge that was crucial for subsistence and survival of its people may benefit humanity in multiple areas of life as it promotes a unique holistic view on existence. Efforts are made in preserving, genuinely expressing and disseminating indigenous knowledge [8] . Our work that tackles building MR applications inspired us to design a conceptual architecture for a general MR platform. In [9] we proposed interaction with the Internet of Everything through a MR platform designed to be used by the general public in order to create and share MR experiences that fit the Ubiquitous Computing paradigm, where services follow the user seamlessly across physical locations. An emphasis was made on context awareness, which is central in designing a MR application, by enabling the anchoring of the right multimodal content to the right circumstance. We have divided awareness into four levels with respect to knowledge representation, in order to provide a tool for non-programmers to design and create applications. The Conceptual Architecture splits Internet of Things connected devices roles into three categories, namely the input sensors, the multimodal output devices and the server. Smart sensors acquire data about the environment and offer an output of meaningful information further used to identify a circumstance. Multimodal output devices play content that is relevant to the identified situation. The input and the output devices need to have interfaces designed by programmers in order to communicate through the server. Non-programmers developing a MR application access these interfaces through the server and visually build causal trigger links between the input and output interfaces in order to create the desired MR interaction. Users may choose to start building their application from one of the four complexity levels of environment awareness: • Data: The simplest type of interaction only provides sensing raw data through another channel (e.g., numbers mapped to colors). • Information: One step further, data may be attributed a meaning and the outcome will be identical every time for the same input (e.g., QR code provides a link). • Knowledge: If the system has internal states it may be modeled with the help of a finite state machine and outcomes may be different for the same input (e.g., indicating the Next command while following a tutorial). • Inference: Finally, the system may be so complex that these states become unpredictable (e.g., playing a song that fits the inferred mood). The features of the MR platform allow for building triggers in both the real and the virtual environments that take effect in any of these two environments. A real to real world interaction involves only the laws of physics governing nature (e.g., picking of a flower). A real to virtual world interaction is expressed in an AR experience (e.g., midair scrolling over a book projection and turning pages). A virtual to virtual interaction takes place in a VR experience (e.g., painting a virtual artifact with a virtual brush). Another type of interaction may be achieved with the help of actuators, where the virtual elements have effect in the real world (e.g., typing an access code on a virtual interface and opening a real safe). In order to exemplify our proposed platform for MR application design, we chose a scenario where the user explores an amphora by providing him interaction at four different levels. The hardware used for the system was composed of a laptop running a WebSocket server, a LeapMotion™ device that detects the user's hands and gestures, and a screen that displays the scene of interaction. In the data level, interaction feedback is given in the form of a colored disk indicating whether a hand is being detected by the LeapMotion™ device. Color red is used for no hand detected and color green is used to indicate that a hand, be it left of right, is currently detected. This helps the user learn the volume of the interaction space. In the information level, the LeapMotion™ device detects the gesture of the right hand. If the right hand is not currently detected, the question mark sign will be highlighted. If the right hand is open or closed, the palm sign and the fist sign will be highlighted, respectively. The user will then know the exact gesture detected by the system. For the knowledge level, a state machine with five states was implemented. The first state is "Inactive", meaning that the right hand is not detected. Every other state may lead to "Inactive" if the hand is held outside the interaction volume. The "Inactive" state only leads to "Hand detected" which refers to the right hand of the user. If a grab gesture is performed the systems goes into "Object grabbed state". For each state messages help the user know the current state, and the recommended gesture needed to be performed by the user for the system to move towards the next state. In Fig. 1 ., after the object is grabbed, the user needs to open their hand so that the orientation becomes accurately detectable. Then, the system moves into "Rotation enabled" state and the user may rotate their hand and, in consequence, rotate the virtual artifact. The virtual amphora's 3D rotation is mapped to the 3D orientation of the user's palm. Afterwards, the user will perform a "Release object" gesture by closing and opening their hand and the object will become static once again. Thus, the state machine uses information about gestures (i.e., "no hand", "open hand" and "closed hand") to move from one state to another. In the inference level, we have implemented a natural user interaction for translating the virtual amphora along the three axes. This is achieved by using both hands in order to simulate the grabbing and repositioning of the virtual object. For each hand, the LeapMotion™ device detects its position in space corresponding to the center of the palm. Then, the average distance between the two detected hands will become the point in space where the user would like to put the object. Each hand was attributed a 50% weight for determining this target point. Another parameter considered was the distance between the two hands. If the distance is close enough resembling a two-hand grab gesture, then the movement is mapped 100%. Otherwise, if the hands are quite apart, the movement is affected in a lesser amount by the target point. The distance parameter influences the displacement in an inversely proportionally manner along the 3D vector direction given by the target point. Thus, the transition from one gesture to another is smooth and natural. We implemented this simple scenario in order to exemplify the four levels of application blueprints through several types of interaction. More complex scenarios may be designed which involve a multitude of smart sensors (e.g., facial expression recognition sensor, speech recognition sensor, gaze direction detection sensor, physiological parameters sensor) and multimodal output devices (e.g., haptic feedback device, wind and aroma diffuser, smart colored lighting). In the digital age, museums enhance the visitor's experience using breakthrough technologies that provide new ways of interacting with the exhibited items. Following the Ubiquitous Computing paradigm, three instances of the Museum institution emerge: the intelligent physical museum, the virtual remotely accessible museum, and the embedded museum inside the Intelligent Environment. In this section we devise the MR Museum as a space for MR experience designed using the MR platform. The contents of the Museum may be experienced through MR in three different environments (see Fig. 2.) . First there is the physical museum with the exhibited items and relevant virtual content anchored across the space. Secondly, there is the virtual museum present online that is mapped onto the physical museum. Finally, bits of the virtual museum may be embedded at another physical location in order to provide a thematic walkthrough. In the following we classify museums based on the environment that they reside in. In section 4.1., MR enhancements to the physical museum are discussed. Next, we describe how MR is used in virtual museums (section 4.2.), leading to the idea of the embedded museum that gathers knowledge/artefacts from diverse environments, for the benefit of the final user (section 4.3.). Finally, in section 4.4. we tackle the impact of the digital museum on the society, presenting our view on how it impacts sustainable development in the large. The museums building present at a specific location, hosts objects of interest organized in different exhibits over a vast spatial are. An enhanced museum experience may be provided for the visitors by means of MR technologies. The interactions in MR may be a combination of one or more of the Real to Real (R2R), Real to Virtual (R2V), Virtual to Real (V2R) and Virtual to Virtual (V2V) forms of interaction. One of the latest technological advancements is 3D printed becoming accessible [10] . Objects may be scanned with the help of lasers, reconstructed virtually with the help of computer vision algorithms and 3D printed on site in a different location. Tangible interfaces [11] are suitable for exploring the exhibited items by touching 3D replicas [12] or moving through the environments and making natural gestures [13] . By wearing a Mixed Reality device such as the HoloLens, objects that are exhibited become Reality Based Interfaces by rendering the objects clickable. Immersive virtual experiences may be achieved by stereo video projection or wearable AR/VR devices [14] . Interesting effects are achieved when interaction is carried out from the virtual with effect in the real environment. An example would be asking a virtual agent to open a door inside a castle that will be physically moved or making a fire lighting gesture by friction and starting a real fire from a camping reconstructed scene. The more there is interaction between the user and the Intelligent Environment, the more the user will feel immersed during the MR experience. Dissemination of the knowledge comprised inside a museum is achievable solely through digital means [15] . A website, and a mobile application, may constitute modern approaches in interaction with the exhibited items. Museums already keep digital records, but not enough information is designed to be accessed by the general public or the academic personnel. Interfaces designed to be adaptable to a specific target user make information easily accessible. The general public prefers interfaces that are intuitive and easy to use at first sight. They are impressed by eye candy and wish to experience new ways of innovative presentation and interaction with the content. A fit example of advanced technology in human computer interaction would be stereoscopic 3D rendering of a scene that blends with the real environment and midair gesture recognition in order to manipulate the virtual items. A device that may achieve this kind of MR experience is the HoloLens from Microsoft [14] . Professional users on the other hand prefer interfaces that provide means of easily accessing quality information. Such interfaces need to provide complex functionality and should have a clean and neatly organized design [15] . In a Ubiquitous Museum [16] , virtual elements embedded at another location are used to provide a walkthrough as part of an educational or entertainment program. In [17] , the location is a physical space specially conceived to virtually host items from the museum such as a thematic exhibit at another place than the museum. The location may also come in the form of a virtual space designed to incorporate those items for a special immersive experience [18] . Another possibility is for interactive items along with relevant information to be embedded into websites, presentations and books. Several spectacular experiences are achieved through the means of embedding the museum contents in different locations [18] , [19] . The Embedded Museum provides a sustainable approach to knowledge dissemination through the Ubiquitous Museum. In this article [20] researchers have studied the connection between different stakeholders, forces of influence and policy making in cultural sustainability. A theoretical model of sustainable development was devised considering the cultural, environmental, economic and social factors. In [21] the researchers propose that sustainability of a museum is achieved by an audience-centric approach to cultural dissemination. Another study [22] on mining heritage tourism proposes that bringing together eclectic elements within a museum may generate sustainability for the community. We hypothesize that the Ubiquitous Museum [23] provides new means of combining forces of development. Thematic events and campaigns may promote culture through embedded museum elements and consequently generate revenue. Encouraging industries such as filmmaking and game development to use realistic scenes and objects may produce quality digital materials to be used inside museums. This may bring popularity to certain topics and make development sustainable. Amusement parks and musical festivals may also be the target of cultural dissemination. Classical artistic events such as ballet, opera and theater shows would have access to high quality content for scene enrichment. Innovative halls of experience may be assembled by artists or scientists to express, promote and disseminate knowledge. A smart design of leisure activities may involve informal means of education that help personal cultural development and furthermore channel economic gain towards sustainable development. Future work will be devoted to quantifying the appropriateness of the hypothesis we formulated with respect to the influence of the Ubiquitous Museum as a source of sustainable knowledge. The Ubiquitous Museum based on MR technologies is a new concept of Knowledge Dissemination. Three types of museum emerge under the Ubiquitous Museum paradigm: the physical Intelligent Museum with experiences enhanced by MR, the Virtual remotely accessible Museum mapped upon the original museum containing the multimodal collection and the Embedded Museum reconstructed by linking the virtual multimodal content inside the Intelligent Real or Virtual Environments. Innovative experiences that have a social impact are constructed through reassembling by using the division between content and location which enables design flexibility. Based on our proposal of a platform for affordable MR [9] , we presented an application that exemplifies the four levels of awareness in the platform, developed for a MR museum setting. Future work includes further development of the MR platform that we proposed in [9] and testing the platform in a scenario involving an archaeological museum. We will gather insight into what kind of multimodal content users prefer and the means of interaction that are most fit for the content. Intelligent environments: a manifesto Smart Museums with Augmented Reality Cultural Heritage Interactive Dissemination through Natural Interaction From Museum to Amusement Park: The Opportunities and Risks of Edutainment A pedagogical framework for mobile learning: Categorizing educational applications of mobile technologies into four types All reality: Virtual, augmented, mixed (x), mediated (x, y), and multimediated reality ViMM-Virtual Multimodal Museum: a manifesto and roadmap for Europe's digital cultural heritage Bridging indigenous and scientific knowledge Open Affordable Mixed Reality: A Manifesto Fabricated: The new world of 3D printing Tangible interfaces for digital museum applications 3D printing for mixed reality hands-on museum exhibit interaction Designing Public Digital Cultural Service Interactive System Based on Reality-Based Interaction Principles Designing Spatial UI as a Solution of the Narrow FOV of Microsoft HoloLens: Prototype of Virtual Museum Guide Predictive user modeling in user interface design A context-aware knowledge map to support ubiquitous learning activities for a u-Botanical museum IoT and Engagement in the Ubiquitous Museum Extending museum exhibits by embedded media content for an embodied interaction experience Interactive museum exhibits with embedded systems: A use-case scenario Museums and cultural sustainability: stakeholders, forces, and cultural policies An audience-centric approach for museums sustainability Exploring the sustainability of mining heritage tourism The ubiquitous museum service environment: Concept, design, implementation, and a case study This research was co-funded by the Romanian Ministry of Culture, project number RO-CULTURA-A1-4/2020/31.01.2020-"A future for the past -the classification and capitalization of the Callatian epigraphic and sculptural patrimony" (project code CALL01-9) and the Romanian Ministry of Research and Innovation, CCCDI-UEFISCDI, project number PN-III-P1-1.2-PCCDI-2017-0917, contract no. 21PCCDI/2018, within PNCDI III, project P2, "Efficient communications based on smart devices for in-car augmented reality interactive applications".