Motion Parameters Identification for the Authoring of Manual Tasks in Digital Human Simulations: An Approach Using Semantic Modelling Available online at www.sciencedirect.com 2212-8271 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the scientific committee of 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015 doi: 10.1016/j.procir.2015.12.077 Procedia CIRP 41 ( 2016 ) 752 – 758 ScienceDirect 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015 Motion parameters identification for the authoring of manual tasks in digital human simulations: an approach using semantic modelling George Pintzosa, Nikolaos Nikolakisa, Kosmas Alexopoulosa, George Chryssolourisa,* aLaboratory for Manufacturing Systems and Automation, Dept. of Mechanical Engineering and Aeronautics, University of Patras, Patras, 26500 Greece * Corresponding author. Tel.: +30-2610-997262; fax: +30-2610-997744. E-mail address: xrisol@lms.mech.upatras.gr Abstract The use of digital simulation tools for the planning and verification of manufacturing processes has been identified as a key enabler technology. Through these tools, the need for physical prototypes is reduced, thus enabling the early assessment of decisions, regarding the efficiency of processes. The same stands for manual assembly planning. However, in industrial current practices, the digital simulation tools are scarcely used since the times for the generation of human simulations are still high. Furthermore, the current tools do not support the generation of motions that correspond to real life worker behaviors. This paper presents a methodology for the recognition and reuse of motions and motion parameters during a manual assembly execution. The methodology is based on a motion recognition algorithm using low cost sensors. This algorithm employs a rule based approach in order to identify motions that are translated into semantic individuals. A semantic model is also presented, accompanied by the relevant semantic rules for the organization and reuse of recorded motion parameters, during the production planning and more specifically, during the Digital Human Simulation. The methodology is applied to an industrial case study around the assembly of a car differential. © 2015 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the Scientific Committee of 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015. Keywords: Motion; Assembly(ing); Worker simulation 1. Introduction In today’s market, manufacturing companies face increased pressure for quick responses to changes in the environment [1]. Therefore, research efforts are being carried out to provide flexible, robust and competitive solutions that enable the faster and more effective commissioning of manufacturing systems. Regarding human centered operations, such as manual assembly, the manufacturing companies use digital tools to simulate and optimize them [2, 3]. Based on this, the simulation of human-centered systems has been one of the vital topics of research [4-6]. A challenge with manual operations is that, during their simulation, a number of constraints must be defined and taken into account. The human related processes should respect ergonomic and time values that correspond to real life practices. This should be done in a manner that would allow alternative strategies to be assessed, in order to come up with a near optimum solution for a given set of criteria. Details and results from operations without interfering and affecting their performance [7, 8] are needed. Since many industries make an intense use of human resources, e.g. for assembly, repair and recycle operations, the inclusion of this information would improve the accuracy of simulations [9-11]. © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the scientifi c committee of 48th CIRP Conference on MANUFACTURING SYSTEMS - CIRP CMS 2015 753 George Pintzos et al. / Procedia CIRP 41 ( 2016 ) 752 – 758 The use of modern Digital Human Simulation systems enables the representation and inspection of a process in a simulated environment. A digital simulation may provide the required functionalities for the assessment and evaluation of alternative strategies during the design and reconfiguration of a manufacturing process [2]. For example, the use of Digital Human Simulation tools has enabled the Ford automotive industry to reduce its assembly-related worker injuries dramatically, when at the same time, the quality of new vehicles had been improved by 11% [12]. Furthermore, mapping the performed operations to digital structures by means of information technologies can provide a framework for advanced planning and control of the overall industrial operation [13]. Detailed reviews on manufacturing system simulation techniques can be found in [14, 15]. From these detailed reviews, the requirement for more efficient methods is pointed out as one of the more important ones. Finally, Digital Human Simulations tend to include miscalculated process durations, which can lead to non-effective planning of the actual operations [16]. The fact that when simulations may be incorrect in terms of duration, thus resulting in non-realistic conclusions is one of the main motivations for the study presented in the following sections. The problem with standardizing motions and their parameters, such as the ones performed during assembly tasks, is that of variations caused by the environmental parameters. As an example, a carrying motion performed for different parts with different volumes will have different durations. The existing approaches propose either functions or discrete values, expressing the duration of a certain process, which has to be manually calculated or identified in relevant indexes [17]. In order to provide realistic parameters, related to the execution of manual processes with specific environmental characteristics, a motion recognition and classification approach is proposed. Motion recognition is used in order to identify the actual motions carried out on the shop floor for their classification, based on their type and the environmental characteristics (including the actor’s characteristics, parts’ dimensions etc.) and make them available for the simulation of similar motions in future scenarios. For the storage and reuse of the identified motion parameters, a semantic model is proposed. In the next section, the overall methodology is described. 2. Methodology As stated in the previous section, the approach discussed, is based on a motion recognition algorithm in combination with a data structure, which is used to storing the semantic representations of motions. In order to record human motions similar to the ones performed in a shop floor environment, three types of Fig. 1. Sensors used and the data provided by each type. sensors are proposed, namely the optical, force and tool embedded sensors that can provide process related information such as fastening torque for electric screwdrivers. In the implementation presented in this paper, a second generation Kinect sensor for Windows has been selected as an optical sensor due to its low cost. It can capture depth images in a 3 to 4 meters limited field of view, under a maximum recorded resolution of 30 frames per second (fps). The optical sensor can track 25 joints of the human body and store the related information into a structured form. The force sensors provide information by expressing the force applied to them in different key frames. Sensors, integrated into tools, such as the electric screw drivers, provide process related information, namely torque, start and finish of a process. All the data produced by these sensors are then used to identifying and properly recognizing the captured motions. In the implementation presented in this paper, only optical and force sensor data are used. Through various motion captures (MoCaps), common patterns are sought after for the identification of specific motions. However, motion styles can vary depending on a number of secondary features, such as age, gender, body shape etc. In order for the issues caused by these features (i.e. variations in sensor data) to be overcome, the use of rules is proposed. Performed motions, such as picking, placing, walking, etc. can comprise smaller ones. For example, picking is the combination of a reaching action towards an object, a grasping action and a motion where the hand moves closer to the body. Therefore, in order for the more effective recognition of motions to be enabled, a two level approach has been used, where the motions are recognized firstly through their sub-components and then confirmed and completed, on the basis of their sequence or intersection with a common timeline. The component motions are termed Motion Elements (MEs) and are recognized by the Motions Recognition Algorithm (MRA). Afterwards, the MEs are semantically stored and there, through additional semantic rules, the high level motions are identified. These motions are termed Elementary Actions (EAs). The EAs are provided by clustering MEs. EAs are generated and constrained through the semantic 754 George Pintzos et al. / Procedia CIRP 41 ( 2016 ) 752 – 758 correlation of MEs. As an example, the fusion of a grasping motion, preceded and followed by motions reaching forward and backwards respectively, results in a Pick Elementary Action. As presented in the following sections, the combination of MRA and semantic processing can currently recognize the following EAs: Walk, Walk Inverse, Pick, Place, Carry, Carry Inverse, Sidestep and Sidestep-Carry. Apart from the generation of EAs, the semantic storage functionality enables the reuse of MEs and EAs as well as their related parameters and constraints for future simulations. As stated earlier, MEs are stored into a semantic repository together with parameters related to the environment. EAs are also stored into this database after their generation. The database created has a triple database (TDB) format. Semantic queries enable the retrieval of EAs’ parameters, which can be used in digital simulations as constraints. Fig. 2. Overall methodology steps. 3. Motion recognition algorithm In order to test and evaluate the proposed methodology, a software application was developed on the basis of requirements for low processing time and low storage requirements. The developed application can be divided into two parts; motion recognition and semantic storage and processing. In this section, the first part will be described. In the current implementation, two types of data are collected and used as input; the optical and force ones (Fig. 3). Data coming from optical sensors include joints’ displacement and rotation values for each of the recorded time frames. Force data are measurements on the force applied to a body part during a motion, e.g. the fingers’ force during a grasp motion. Fig. 3. MRA Steps. The MEs recognition is based on the use of rules related to the joint distances, their differentials and raw force data, while the EAs upon semantic rules which are further explained in the next section. The MEs’ rules use parameters exported from optical and force data and are divided into two categories: Rules related to the motion, e.g. distance, velocity, start and stop time frames, force etc. The calculation of these parameters is made by considering the joint values related to the motion. Rules related to the environment, e.g. human body dimensions, size of picked object etc. In the current implementation, the MEs that can be recognized are Walk, Reach and their inverses as well as Grasp and Sidestep, Sidestep-Carry. The relations between these MEs and the EAs can be seen below: Fig. 4. Hierarchical relations between MEs and EAs. The first action performed by the algorithm is the calculation of the distances between the joints using the optical data: Table 1. Joint distances calculated for recognition of specific motions. Distance Used in recognition of Ankle – Waist Walk, Walk Inv., Sidestep, Sidestep Inv. Ankle – Ankle Walk, Walk Inv., Sidestep, Sidestep Inv. Wrist – Waist Reach, Reach Inv. Ankle – Wrist Reach, Reach Inv. Wrist – Wrist Reach, Reach Inv. 755 George Pintzos et al. / Procedia CIRP 41 ( 2016 ) 752 – 758 The velocity is calculated through differentiation. An interesting characteristic of the motions is that the velocity peak values arise near the middle of an ME. Based on this, the MEs’ starting points are sought after within the times that precede and follow the peak values. A starting point is set at the time when the relative velocity has a zero or near-zero value. This is done by following the velocity curve backwards from peak to the first zero point. The relative velocity, here, expresses either the change in the distance between two specified joints or one joint and the global coordinate system. Following a similar approach, an ME ends at a timeframe after a peak value, when velocity derives from a peak value to near-zero. At this point, it should be noted that the velocity criterion enables recognition only to a certain extent. In order to be firmly resolved whether an ME has been performed or not, further processing is required. The additional rules used involve the “IF… THEN…” condition. These rules examine the values of different, secondary to the motion, joints. As an example, a rule for walking would consider, apart from the movement of the ankle joints, the concurrent hip joints’ motion in space. For instance, a Pick EA, consists of two MEs that have to be recognized; Reach, and Grasp. The following figure shows the speed between the examined joints. The wrist to hip distance is calculated and translated by differentiation to speed. Within the speed values, peaks are identified and compared to threshold values. When walking, the users’ hands are also moving in a similar to reaching motion. In order to avoid including this, the threshold is set accordingly. The positive values in the following figure correspond to the Reach MEs, while the negative correspond to the hands’ return or the Reach Inverse ME. Afterwards, it is the start and stop frames that are identified. Fig. 6. Right Hand - Waist Speed Curve. In Fig. 6, points 1.1 and 3.1 define the start of the Reach MEs and 1.2 and 3.2 the end of them. In addition, points 2.1 and 4.1 correspond to the starting frames of the Reach Inverse MEs, while 2.2 and 4.2 to the end of the MEs. The existence of a Grasp ME is expected from the identification of a Reach and Reach Inverse MEs, but it is not recognizable by the kinematic data. The recognition of a Grasp ME is carried out by force sensor data. The Grasp is separately recognized and when all the MEs are stored into the semantic repository, the semantic rules identify the sequence of the MEs, leading to the generation of the relative EAs. In this way, when a Reach ME is falsely identified, it will not lead to the storage of a Pick EA. Each ME recognition is accompanied by additional rules which confirm the identification. As an example, for Reach, the ankle joints are also checked; at least, one foot has to remain still during a reaching motion. In a similar way, all possible MEs are recognized and further parameters are calculated. For Walk, the distance covered by walking is calculated and stored. For Reach, the height of the reached object’s position is calculated and so on. Extra parameters include the person’s height, which is calculated at the beginning of the MoCap session, the object’s size, when it is carried with both hands (average distance of hands during carrying), etc. The algorithm was written in Matlab® and follows a linear processing of data. The specific sequence and results of the described steps can be seen in Fig. 6. Fig. 5. Motions indication and recognition steps of the MRA. 756 George Pintzos et al. / Procedia CIRP 41 ( 2016 ) 752 – 758 4. Human motions semantic model and application The second part of the applications developed, concerns the MEs’ semantic representation and storage, the generation of the EAs and the reuse of their parameters. These features are performed within a Motion Semantic Repository (MSR). The MSR was developed through the Jena framework. Within the repository, the MEs are stored as individuals. Data Properties are used for the storage of the MEs’ related parameters, whilst the Object Properties are used for the storage of the relations between the MEs and the EAs. The developed semantic model is composed of two main classes; the MotionElement class is used for the MEs, and the ElementaryAction class for EAs. The recognized MEs and their parameters, provided from the MRA, are stored into the MotionElement class, using specific sub-classes. The following figure shows the different sub-classes of each main class. One sub-class is provided for each type of Motion Element and Elementary Action. The data properties presented describe the instances (individuals) of each sub-class. Depending on the type of motion, different parameters are stored as Data properties. As an example, for Walk or Carry motions, the “Distance” property is stored, which is the distance that the tracked person walked or carried an object. For Carry, Pick and Place EAs (and the relevant MEs) the hands that the person uses are stored using Boolean values. Fig. 7. Semantic model classes and properties. Based on the start and stop frame values, the ME individuals are sequenced with the use of the Data Property Sequence (integer). Then, on the basis of additional rules, the different sets or discrete ME individuals, generate EAs. An ElementaryAction individual obtains the IncludesME property, if the sequence of the ME individuals, stated to generate an EA, fulfil the constraints for the generation of this specific individual. An ME individual obtains the isIncluded if it is part of a generated EA. The semantic rules ensure the generation of EAs from the correct MEs as well as the correlation of the relevant parameters to them. Following the previous example, the start frame of the Reach ME becomes the start frame of the Pick EA, while the stop frame of the Reach Inv. ME becomes its stop frame. In other cases, such as the Carry EA, its start and stop frames are identified as the start and stop frames of the Walk ME being a part of it (Fig. 8), since a Carry ME comprises the parallel execution of a Grasp and Walk MEs. Fig. 8. Rule for identification of the start and stop frames of a Carry EA. After all EAs have been generated and associated with the necessary Data Properties, all the MEs are deleted from the MSR. Then, the user can easily search in the MSR for the identification and reuse of motion parameters (Fig. 9). Via the interface, the user can access, search and retrieve information from the MSR. Moreover, new MEs can be imported. Delete and export functionalities are also available. 5. Case study In order to present and evaluate the proposed concept in a real environment, a case study has been performed with the following scenario: Fig. 9. Input section of the MSR GUI. 757 George Pintzos et al. / Procedia CIRP 41 ( 2016 ) 752 – 758 A person performs assembly operations on an automotive differential. The session is recorded with the use of sensors. The results of the MoCap session are imported to the MSR and processed via semantic rules. Another user searches the MSR in order to identify time parameters for a digital simulation. The user identifies the parameters and generates the simulation based on them. The first step was performed in an industrial environment with the use of an optical sensor and a set of force sensors. The force sensors were embedded within a glove that was used for assembly operations. The optical sensor was placed on the side of the user in a two meter distance from his original position. The activities planned and tracked were the following: Walk to table with parts. Pick left side mount of differential. Carry left side mount of differential to differential casing’s position. Place right side mount of differential on the casing. Repeat all tasks for right side mount. The session was recorded and two separate files were generated comprising of optical and force data. Force data were communicated through a wireless node and collected. The two generated files (containing optical and force data), were processed by the MRA algorithm, and another file was generated. The file format used was of a spreadsheet application and contained all the MEs recognised as well as the relevant constraints per type. Afterwards, the files were imported through the Graphical User Interface (GUI) of the MSR. Fig. 10. Scene from the (a) MoCap sessions and from the (b) digital simulation. Finally, a simulation of the entire session was performed in a Digital Human Simulation software. The duration of all motions was requested and successfully retrieved through the GUI of the MSR, providing a realistic, in terms of time, simulation of processes similar to the ones performed in the MoCap sessions. During the MoCap sessions it was identified that, in order for the motion capturing to be successful, all key joints should be “visible” by the optical sensors at all times. This places the requirement for additional optical sensors as well as for a data fusion mechanism. 6. Conclusions and discussion The motivation behind the study presented in this paper, was centred on the need for faster planning of manual assembly operations, and more precise Digital Human Simulations. The main idea proposed was the use of motion data, obtained from the performance of assembly operations by shop floor personnel, using various types of sensors. A methodology for the recognition of different motions as well as the storage and reuse of the generated parameters was also presented. Finally, the overall methodology was verified in a shop floor environment and reusing the parameters identified in a new Digital Human Simulation. The presented methodology has shown promise for future advancements that could bring it closer to industrial practices and help generate Digital Human Simulations in less time and with greater realism. As a first step towards the improvement of the methodology, the inclusion of additional sensors will be realised. As described in Section 2, sensors on tools will be used in future developments to enable the identification of additional processes which will further enhance the algorithm’s capabilities. This solution is already available in off-the-shelf products that use Bluetooth as a standard for wireless communication. Furthermore, the use of multiple optical sensors will be investigated. In order to realise this, an existing fusion algorithm will be used in order to merge data from different sensors [18]. Beyond these, the use of additional sensors, will be investigated, namely Inertial Measurement Units (IMUs) will be explored for tracking the motion of assembly equipment. Regarding MRA and MSR, two developments are foreseen: One will be the inclusion of motions for recognition such as kneeling and screwing (processing). Furthermore, key joint positions will be added to the MSR, to enable the storage of the actors’ detailed poses for specific motions. This will allow the more precise reuse of motion parameters, both concerning time and space. Finally, the most important step for the significant improvement of the presented methodology and for bringing it closer to industrial practice, will be the integration of all the developed components both 758 George Pintzos et al. / Procedia CIRP 41 ( 2016 ) 752 – 758 internally and externally. Primarily, the distinct components, i.e. the Motion Recognition application and the Motion Semantic Repository, will be integrated into one application. Subsequently, the application will be integrated with existing, commercial Digital Simulation Tools in order to automatically receive requirements for assembly operation parameters and deliver them directly into simulations. Acknowledgments This study was partially supported by the project INTERACT / ICT-2013-10 - 611007, funded by the European Commission. The authors of this paper would like to thank EMPHASIS Telematics AE for developing and providing their support regarding the force sensors. References [1] Chryssolouris G. Manufacturing Systems: Theory and Practice. 3rd ed. New York: Springer-Verlag; 2006. [2] Mourtzis D, Papakostas N, Mavrikios D, Makris S, Alexopoulos K. The role of simulation in digital manufacturing: applications and outlook. International Journal of Computer Integrated Manufacturing 2013; 28:3-24. [3] Whitney D. Mechanical Assemblies: Their Design, Manufacture, and Role in Product Development. New York: Oxford University Press; 2004. [4] Elkosantini S. Toward a new generic behavior model for human centered system simulation. Simulation Modelling Practice and Theory 2015; 52:108-122. [5] Kesseler E, Knapen EG. Towards human-centered design: Two case studies. Journal of Systems and Software 2006; 79:301-313. [6] Shahrokhi M, Bernard A. A framework to develop an analysis agent for evaluating human performance in manufacturing systems. CIRP Journal of Manufacturing Science and Technology 2009; 2:55-60. [7] Orellana DW, Madni AM. Human System Integration Ontology: Enhancing Model Based Systems Engineering to Evaluate Human-System Performance. Procedia Computer Science 2014; 28:19-25. [8] Asllani A, Lari A. The effect of human pattern-recognition abilities in improving DSS performance. Computers & Industrial Engineering 2009; 57:246-252. [9] Han KH, Park JW. Process-centred knowledge model and enterprise ontology for the development of knowledge management system. Expert Systems with Applications; 36:7441-7447. [10] Mason S, Baines T, Kay JM, Ladbrook J. Improving the Design Process for Factories: Modeling Human Performance Variation. Journal of Manufacturing Systems 2005; 24:47-54. [11] Baines T, Mason S, Siebers P, Ladbrook J. Humans: the missing link in manufacturing simulation?. Simulation Modelling Practice and Theory 2004; 12:515-526. [12] Thornton J. At Ford, Ergonomics Meets Immersive Engineering. EHS Today; 2009. http://ehstoday.com/health/ergonomics/ford- ergonomics-simulation-0409 [13] Maropoulos P. Digital enterprise technology - Defining perspectives and research priorities. International Journal of Computer Integrated Manufacturing 2003; 16:467-478. [14] Negahban A, Smith J. Simulation for manufacturing system design and operation: Literature review and analysis. Journal of Manufacturing Systems 2003; 33:241-261. [15] Mourtzis D, Doukas M, Bernidaki D. Simulation in Manufacturing: Review and Challenges. Procedia CIRP 2014; 25:213-229. [16] Lämkull D, Hanson L, Örtengren R. A comparative study of digital human modelling simulation results and their outcomes in reality: A case study within manual assembly of automobiles. International Journal of Industrial Ergonomics 2009; 39:428-441. [17] Whitney D. Mechanical Assemblies: Their Design, Manufacture, and Role in Product Development. 1st ed. Oxford: Oxford University Press; 2004. [18] Geiselhart F, Otto M, Rukzio E. On the use of Multi-Depth- Camera based Motion Tracking Systems in Production Planning Environments. Procedia CMS 2015.