Dynamic Operator Instructions Based on Augmented Reality and Rule-based Expert Systems Available online at www.sciencedirect.com 2212-8271 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the scientific committee of 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015 doi: 10.1016/j.procir.2015.12.113 Procedia CIRP 41 ( 2016 ) 346 – 351 ScienceDirect 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015 Dynamic operator instructions based on augmented reality and rule-based expert systems Anna Syberfeldt*,a, Oscar Danielsson a, Magnus Holm a, Lihui Wang a b aUniversity of Skövde, PO-54148, Skövde, Sweden bRoyal Institute of Technology, PO-1148, Stockholm, Sweden. * Corresponding author. Tel.: +46(0)500448577. E-mail address: anna.syberfeldt@his.se Abstract Augmented reality is currently a hot research topic within manufacturing and a great potential of the technique is seen. This study aims to increase the knowledge of the adaptation and usability of augmented reality for the training of operators. A new approach of using dynamic information content is proposed that is automatically adjusted to the individual operator and his/her learning progress for increased efficiency and shorter learning times. The approach make use of the concept of expert systems from the field of artificial intelligence for determine the information content on-line. A framework called "Augmented Reality Expert System" (ARES) is developed that combines AR and expert systems. A proof-of-concept evaluation of the framework is presented in the paper and possible future extensions are discussed. © 2015 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the Scientific Committee of 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015. Keywords: Augmented reality; Expert systems; Industrial operator; Information content 1. Introduction Augmented reality (AR) is a hot topic in manufacturing research today. With AR, it is possible to give industrial operators access to information that their ordinary senses could not have gathered from reality, and to give this information in the context of where it is needed. The basic concept behind AR is to overlay digital information about the environment and its objects on the real world, and thereby enhance the perception of reality. Over the last few years, the technology enabling AR has advanced rapidly and a number of real-world applications of AR can be seen today; mainly within areas such as gaming, sports and tourism. Within the context of industrial applications, AR has been discussed for over 20 years. There exist plenty of studies on the topic (see for example [1-4]), but few practical applications. Tiefenbacher et al. [5] discuss that the limited success of AR in industrial applications is mainly due to the fact that the industrial setting is highly challenging and complex. For the big breakthrough of AR, additional research is needed on how to tackle these complexities and successfully develop useful applications. An industrial application area were the research on AR has been specifically limited is the adaptation and usability of AR for training of industrial operators within the manufacturing domain. Lee [6] discusses that there exist only a few general studies on the topic of industrial training, and none of these are in the manufacturing domain. This study aims to contribute to an increased knowledge of the adaptation and usability of AR for training of operators within manufacturing. The authors believe that increased knowledge is critical for utilizing the full potential of AR, and to achieve a widespread use of the technique within the manufacturing domain. The standpoint in the paper is that a new approach for information content handling is necessary for the effective use of AR for training purposes and the learning a new tasks. That is, a new approach for what information that is presented to an individual operator, at which point in time and in what form. Previous studies on AR in the industrial context have paid little attention to the information content, instead mainly focusing on evaluating the concept of AR as such. A common property of the majority of the existing AR solutions is that the information content displayed to the operator is © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the scientifi c committee of 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015 347 Anna Syberfeldt et al. / Procedia CIRP 41 ( 2016 ) 346 – 351 predetermined and static. This might work well when the purpose of the AR solution is simply to guide the operator through a number of steps, but from a learning perspective it can be argued that this approach is too inflexible. Since learning is a dynamic and individual process, the authors believe that the information content must be dynamic and individual as well. In this paper, an approach of using dynamic information content is proposed that is automatically adjusted to the individual operator and his/her learning progress. The approach make use of the concept of rule-based expert systems from the field of artificial intelligence for determine the information content on-line. An expert system is basically a computer program that emulates the knowledge and experience of a human expert based on rules. By using an expert system, an operator can learn how to perform a task based on expert knowledge – without the need to actually have human expert physically present. For realizing the new approach of combining AR and expert systems, a framework called "Augmented Reality Expert System" (ARES) is developed as part of the study. In the next section, the concepts of AR and experts systems are described in further detail for the unversed reader. In Chapter 3, the framework ARES is presented. Chapter 4 describes a proof-of-concept evaluation of ARES, while Chapter 5 presents conclusions from the study and outlines important topics to consider in the future. 2. Background In this chapter, AR and expert systems are presented in further detail. 2.1. Augmented reality The term “augmented reality” was first used by Caudel and Mizell in 1992 to refer to a heads-up, see-through display combined with head position sensing and workplace registration systems [11, p.660]: “This technology is used to “augment” the visual field of the user with information necessary in the performance of the current task, and therefore we refer to the technology as 'augmented reality' (AR).”. Later on, Krevelen and Poelman [12] formally defined an AR system as having the following features (based on [13]): (a) ability to combine real and virtual objects in a real environment, (b) ability to register (align) real and virtual objects with each other, and (c) ability to run interactively, in three dimensions, and in real time. It can be noticed that this definition does not limit AR to specific technologies, which is also supported by other definitions. For example, Azuma et.al. [14] and Kipper and Rampolla [15] state that AR can be applied to all senses, including smell, touch and hearing. In general, AR is implemented through some form of anchor in the real world for the AR system to be able navigate. The most common way to implement anchors is to use target images. By connecting virtual information and target images, the AR system can orientate the virtual objects correctly in relation to the real world objects. 2.2. Expert systems In the research field of artificial intelligence, an expert system is a computer program that emulates the decision- making ability of a human expert. Basically, an expert system is designed to solve complex problems by reasoning about knowledge that is represented as if–then rules. A recent definition of expert systems has been provided by Patel, Virparia and Patel [16, p. 11]: "An expert system can be defined as a set of programs that use the human expertise as knowledge which is stored in an encoded form and may manipulate it to solve problems in a specialized domain. An expert system’s knowledge must be coded and stored in the form which the system can use in its reasoning processes performed by the inference engine.”. The main motivation behind expert systems is the fact that human experts are a scarce resource and their time is highly valuable. If an expert system is able to take over some of the work of the expert it cannot only free time and save money, but also give unlimited access to expert knowledge 3. Framework This chapter describes the motivation behind the ARES framework, related work and the overall framework design. 3.1. Motivation The motivation behind ARES ("Augmented Reality Expert System") is the assumption that the guidance and instructions in an AR solution must depend on the previous experience and skills of the individual operator in order for him/her to complete and learn the task at hand in shortest time possible. The authors believe that with customized information content, the effectiveness of an AR solution can be significantly increased and thereby also the efficiency of the operator. Dynamically customizing instructions for individual operators is, however, no easy task. For each task and operator it must be decided in real-time what information that is important to show at the moment and in which form, as well as what information not to show. The latter is important since too much information is not only frustrating for the operator, but might even be detriment due to the high cognitive load imposed. For solving the problems of dynamic instructions and realizing customized information content, the authors propose to use the concept of expert systems. The idea is to use an expert system with the knowledge of human cognition to dynamically determine the information content in the AR solution. Through this approach, it is intended to provide the operator an enhanced view of reality optimized for his/her individual need at the moment. With such support, the hypothesis is that the operator can learn a new task more efficiently compared to using the common approach of predetermined, static information content. 348 Anna Syberfeldt et al. / Procedia CIRP 41 ( 2016 ) 346 – 351 3.2. Related work As far as the authors are aware, expert systems have not previously been used for enabling dynamic, individual information content in AR applications. However, there are previous studies on related thoughts and issues. For example, Hajovy and Christensen [7] discussed already back in 1991 the possibilities to use dynamic expert systems in teaching applications. The idea in [7] is to adjust the instructions presented to a user based on individual needs, instead of using static instructions. Ten years later, Pan [8] evolved the topic and explored the use of strategies that enable an expert system to adapt to, or learn from, interactions with users. The aim of the study was to develop a concept of an expert system that created and adjusted its rules based on the feedback received from the human user in real-time. Neither [7] nor [8] performed their studies within the context of AR, but in 2003 Wiedenmaier et al. [9] discussed expert knowledge in relation to AR applications. Wiedenmaier et al. did not use expert systems, but they compared AR with paper instructions and expert guidance for a typical industrial assembling task and discovered that the assembling was completed in shortest time when the user was guided by an expert, followed by the use of AR support and in last place paper instructions. This finding was important since it indicates the value of expert knowledge in AR applications. Radowski et al. [10] recently presented a study that investigated different visual features for AR-based assembly instructions. Although expert systems was not used in the study, it is relevant in the context since the authors found out through user tests that the visual features used to explain a particular assembly operation must be adjusted to its relative difficulty level. This is indicates the need of dynamic information content in AR applications. 3.3. Overall design ARES consist of two systems; a rule-based expert system and an AR system. From an overall perspective, the expert system is responsible for determine the information content (what information to show and when), while the AR system is responsible for displaying the information properly in the user interface. The overall design of ARES is shown in Figure 1 below. Fig. 1. ARES framework. The AR system in ARES is developed in C# in Unity 3D with the use of the Vuforia for realizing the AR functionality. Vuforia is a software development kit (SDK) for mobile devices that supports the creation of augmented reality application. The SDK implements vision technology to recognize and track image targets and simple 3D objects in real-time. With the image recognition feature it is possible to position and orient virtual objects in relation to real world objects when these are viewed through the camera of a mobile device. The virtual object tracks the position and orientation of the image in real-time so that the viewer’s perspective on the object corresponds with their perspective on the image target. In that way, it appears that the virtual object is a part of the real world. More information about Vuforia can be found at www.qualcomm.com. The rendering of AR objects is determined on-line by the expert system. The expert system is implemented based on rules in the OpenRules format (http://openrules.com/). OpenRules utilizes the concepts of workbooks, worksheets, and tables to build rule repositories. Basically, each OpenRules workbook is comprised of one or several worksheets that separate information into different categories. The logical and physical organization of OpenRules repositories is shown in Figure 3. In ARES, the rules are parsed dynamically and the AR content is updated in real-time according to the outcome of the rules. The rules are created by a human expert in an ordinary Excel file. Using an Excel file has the advantage that is makes it easy for the expert to create and change rules without the need to manipulate programming code. An example of three simple rules is given in Figure 2. These rules specify the rendering of a certain AR object (called “Object 1”) and is based on the time elapsed in combination with the status of the current subtask. Condition Condition Conclusion Seconds StepStatus Active Render Object1 < 5 is TRUE is FALSE >= 5 is TRUE is TRUE is FALSE is FALSE Fig. 2. Example of rules in expert system. Fig. 3. Architecture of OpenRules (from http://openrules.com). Expert system AR system User interface ARES framework Information content Grapichal AR objects User input Content request 349 Anna Syberfeldt et al. / Procedia CIRP 41 ( 2016 ) 346 – 351 The parsing of rules is done in an if-then manner; for instance, the second row in Table 1 is interpreted by the system in the following way: “If the operator has spent 5 seconds or more on the current subtask and if the currently checked subtask is active then the guidance provided to the operator should be in the form defined by Object1”. 4. Proof-of-concept evaluation For a proof-of-concept evaluation of ARES, a practical scenario in which ARES is used has been set up as part of the study. In this scenario, an operator has to perform battery maintenance of a solar- and wind powered street lamp. The street lamp, shown in Figure 4, has been developed as a student project at the university. The battery to be changed is placed inside the light grey box at the bottom on the green fixture shown in the pictures. The top cover of the box is locked with two screws that have to be removed before any work can be done on the battery. As can be seen in Figure 3, the operator uses a tablet which runs ARES. The tablet is a Nvidia SHIELD tablet with a 9” screen (an off-the-self product). For the scenario, six rules have been defined. These rules are based on three conditions: a) the number of seconds elapsed since starting ARES, b) the competence of the operator, and c) if the top cover of the battery box is removed or not. The rules are shown in Figure 5 below. The defined rules are relatively simple, but perfectly serve the purpose of the proof-of-concept evaluation. Fig. 4. Proof-of-concept evaluation. Condition Condition Condition Conclusion Seconds Competence Top cover Render < 15 is beginner is on Basic text instruction <= 30 is beginner is on Detailed text instruction > 30 is beginner is on Detailed text instruction + pointing arrows & tool to use > 0 is beginner is off Text instruction & graphical highlights > 60 is experienced is off Text instruction > 90 is experienced is off Graphical highlights Fig. 5. Implemented rules. In Figure 6-8 below, the outcome of the first three rules in Table 2 is shown as example (screenshots from user interface in the tablet). As can be seen in the pictures, the information content shown to the operator varies depending on which rule that is active. The image on the top of the cover showing the university’s logo is the target image for the AR functionality. Results from using ARES in the described scenario shows that the framework is fully functional and that AR and expert systems can be successfully integrated for enabling dynamic information content. In testing ARES in the scenario, four persons from the university’s staff were engaged. None of these had any prior knowledge of how to maintain the street lamp battery or any previous experience of using AR. The test persons were given the task of changing the battery without any additional instructions on how to do it or how to use the system. All four persons succeeded to perform the task immediately and without flaws, which indicates that implementation worked as intended. Fig. 6. Rule 1 - basic text instruction is shown. Fig. 7. Rule 2 - detailed text instructions is shown. 350 Anna Syberfeldt et al. / Procedia CIRP 41 ( 2016 ) 346 – 351 Fig. 8. Rule 3 - graphical objects in the form of pointing arrows and picture of the tool to use. In the next chapter, possible improvements of ARES are further discusses and conclusions from the study are outlined. 5. Conclusions and future work In this paper, it is argued that a new way of handling information content is necessary for the effective use of AR for training operators in learning new tasks. A new approach of using dynamic information content is proposed in which the information is automatically adjusted to the individual operator and his/her learning progress for increased efficiency and shorter learning times. To do this, a rule-based expert system with the knowledge of human cognition is implemented to dynamically determine the information content in the AR solution. Through this approach, it is intend to provide the operator an enhanced view of reality optimized for his/her individual need at the moment. With such support, the authors believe that the operator can learn a new task more efficiently compared to using the common approach of predetermined, static information content. For realizing the new approach, a framework called "Augmented Reality Expert System" (ARES) is developed that combines AR and expert systems. A proof-of-concept evaluation of the framework is performed using a scenario of battery maintenance. The evaluation shows that it is possible to successfully combine AR and expert systems, and that such approach has potential to be a value support for operators in the manufacturing industry. For proving the full potential of the approach, however, it must be evaluated on also on real, industrial tasks of high complexity. The authors intend to perform such evaluations in the near future in collaboration with the manufacturing company Volvo Cars in Sweden. Furthermore, the authors plan to perform extensive user studies involving industrial operators with different skills and experience in order to evaluate the actual efficiency gain as well as the operators’ experiences. Regarding future research, an important topic to investigate further is the possibility to automatically generate and modify the rules in the expert system. As it is now, the rules must be entered manually by a human expert. Although this is a one- time effort, it might be time consuming and the quality of the rules is dependent on the quality of the expert. A valuable improvement would be the possibility for the system automatically generate proper rules and to modify existing rules in case of changed circumstances. Significant benefits of such self-adaptive rules can be seen, but it is most probably non-trivial to implement. One possible way to realize it might be to use artificial neural networks (ANNs). In general terms, an ANN is a non-linear statistical data modelling method used to model complex relationships between inputs and outputs [17]. Originally, the inspiration for the technique was from the area of neuroscience and the study of neurons as information processing elements in the central nervous system. ANNs have universal approximation characteristics and the ability to adapt to changes through training. Instead of only following a set of rules, ANNs are able to learn underlying relationships between inputs and outputs from a collection of training examples, and to generalize these relationships to previously unseen data. Theoretically, these attributes make ANNs very suitable to be used for implementing self-adaptive rules in an expert system. Whether this is possible in practice will be studied in the future with the aim of implementing a solution in ARES. At bit of topic, but yet of high relevance, is finally a comment on the hardware technology enabling AR. Today, there exist mainly three approaches of implementing AR from a hardware perspective: hand-held devices (e.g. tablets), head- worn devices (e.g. “smart” glasses) and spatial devices (e.g. projectors projecting information on real object surfaces). The first approach, hand-held, has the obvious disadvantage that is requires the operator use his/her hands to hold a device, which prevents to perform the work task. The second approach, head-worn device, does not have this problem as it frees the operator’s hands. Currently, there are two implementations of head-worn solutions on the commercial market; video-based and optical. An optical see- through solution has the advantage of leaving the view of the real-world almost intact, attempting to merge a virtual image into the view of the real-world. A disadvantage of the solution, however, is that image updating at head moves often lags behind which affect the user experience negatively. The visual lag is caused by, for example, communication or rendering delays and means that the virtual objects do not stay in the correct real-world position when the user moves. Even normal head movements require an extremely fast and frequent image updating in order to avoid visual lag, which is not possible to ensure even with the newest hardware technology. In comparison, a video see-through solution does not have the problem of lag as the real view and the virtual view is merged into the same view, but on the other hand it has the disadvantage of blocking out the real world. From a security perspective this is severe as the operator’s sight relies completely on hardware, which is dangerous in an industrial setting. The third approach, spatial devices, has the advantages that it does not directly affect the operator’s sight, nor does it require the operator to hold a device. However, it has a major disadvantage in that it requires hardware equipment to be permanently installed in the working environment, which is both expensive and inflexible. In summary, it can establish that for industrial applications, the AR technology still has a way to go. Solving the various problems with AR enabling hardware and to adjust it for 351 Anna Syberfeldt et al. / Procedia CIRP 41 ( 2016 ) 346 – 351 industrial settings is certainly a key factor for the success of technology in the future. References [1] Zauner J, Haller M, Brandl A and Hartman W. Authoring of a Mixed Reality Assembly Instructor for Hierarchical Structures. In: Proc. of International Symposium on Mixed and Augmented Reality, 2003, pp. 237–246. [2] Nilsson S and Johansson B. Fun and Usable: Augmented Reality Instructions in a Hospital Setting. In: Proc. of 19th Australasian Conference on Computer-Human Interaction, 2007, pp. 123-130. [3] Sääski J, Salonen T, Hakkarainen M, Siltanen S, Woodward C and Lempiäinen J. Integration of design and assembly using augmented reality. Micro Assembly Technologies and Applications 2008; 260: 395- 404. [4] Henderson S, and Feiner S. Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair. IEEE Transactions on Visualization and Computer Graphics 2011; 17(10): 1355-1368. [5] Tiefenbacher P, Lehment N, Rigoll G. Augmented Reality Evaluation: A Concept Utilizing Virtual. In: Proc. of Virtual, Augmented and Mixed Reality, LNCS 8525, 2014; 226–236. [6] Lee K. Augmented Reality in Education and Training. TechTrends 2012;2:13-21.. [7] Hajovy H, Christensen D. Intelligent Computer-Assisted Instruction: The Next Generation. In: Expert systems and intelligent computer-aided instruction. NJ Educational Technology Publications Englewood Cliffs. 1991; 191-196. [8] Pan X. An Experimental Adaptive Expert System. In: Proc. of InterSymp-2000. Baden-Baden, Germany, July 31 – August 4, 2000. [9] Wiedenmaier S, Oehme O, Schmidt L and Luczak H. Augmented Reality (AR) for Assembly Processes Design and Experimental Evaluation, International Journal of Human-Computer Interaction 2003; 16(3): 497- 514. [10] Radkowski R, Herremaa J and Olivera J. Augmented reality based manual assembly support with visual features for different degrees-of- difficulty. International Journal of Human-Computer Interaction 2015. [11] Caudel T, Mizell D. Augmented Reality: An Application of Heads-Up Display Technology to Manual Manufacturing Processes. In: Proc. of the Twenty-Fifth Hawaii International Conference on System Sciences (HICSS). Volume II, 1992, pp. 659-669. [12] Krevelen D, Poelman R. A Survey of Augmented Reality Technologies, Applications and Limitations. International Journal of Virtual Reality 2010; 9(2): 1-20. [13] Azuma R. A Survey of Augmented Reality. Malibu, Hughes Research Laboratories, 1997. [14] Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B. Recent Advances in Augmented Reality. Computer Graphics and Applications 2001; 21(6): 34-47 [15] Kipper G, Rampolla J. Augmented Reality: An Emerging Technologies Guide to AR. 1st Edition, Syngress, 2012. [16] Patel M, Virparia P, Patel D. Web based Fuzzy Expert System and its Applications – a Survery. International Journal of Applied Information Systems 2012; 11-15. [17] Mehrota K, Mohan C, Ranka S. Elements of artificial neural networks. The MIT Press, 1996.