Towards Implementing a Real-time Deformable Human Muscle Model in Digital Human Environments 2351-9789 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AHFE Conference doi: 10.1016/j.promfg.2015.07.889 Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 Available online at www.sciencedirect.com ScienceDirect 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE 2015 Towards implementing a real-time deformable human muscle model in digital human environments Abhinav Sharma*, Angela Dani, Anith J. Mathai, Timothy Marler, Karim Abdel-Malek U.S. Army Virtual Soldier Research Program, Center for Computer Aided Design, The University of Iowa, Iowa City, IA 52242, U.S.A Abstract The current state of the art digital human models are a visual proxy for humans in part due to the advances in computer graphics. They perform with biomechanical accuracy that mimics real human motion. Models such as Santos® have a biomechanically accurate skeleton driving the motion, which in turn controls the deformation of a flexible skin for added realism, all in real time. However, these models lack realistic musculoskeletal systems that respond in real time to biomechanical motion. They require varying levels of pre-processing before motion can be applied to them, thus preventing the real time effect; muscle models need to be flexible and deformable, which in computer simulations generally translates to higher computation requirements. By combining advances in computer graphics, especially the fast rendering game graphics capability, with known literature on musculoskeletal modeling, a preliminary full body musculoskeletal system that deforms in real time is presented along with the skeleton and skin. Given the biomechanical studies focus of digital humans, the model implemented centers on the mathematical articulation, and not the graphical volumetric representation, of actual musculoskeletal systems. As such, each muscle is defined as a line that starts at an origin position, determined from anatomy, and ends at the corresponding insertion position, while wrapping as needed around cylindrical obstacles that emulate the minimum bulges required for that line to be at the centroid of the actual equivalent muscle. For each muscle, the origin position, insertion position and obstacle parameters (position, rotation, and scale) have to be obtained relative to key joints for accurate articulation. This was done manually on a per-muscle basis for 126 muscles, and can be extended to any anthropometric avatar. With this initial real-time model, there is potential for a quicker assessment of the effect of muscles on human task performance, leading to a complete model that deforms in real time. © 2015 The Authors. Published by Elsevier B.V. Peer-review under responsibility of AHFE Conference. Keywords: Digital humans; Muscle model; Anatomy; Real time graphics; Musculoskeletal system * Corresponding author. Tel.: +1-319-333-5075. E-mail address: abhinav-sharma@uiowa.edu © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AHFE Conference http://crossmark.crossref.org/dialog/?doi=10.1016/j.promfg.2015.07.889&domain=pdf 3845 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 1. Introduction 1.1. Digital human Digital human models are representations of humans rendered within a 3D computer graphical environment. The digital human model is driven by a mathematical skeleton. Unlike real human beings that comprise of bones, muscles, nerves, and organ systems, digital humans such as Santos® consist mainly of kinematic skeleton and a skin. The digital humans or ‘avatars’, visually characterize a human body by using advanced graphical techniques to realistically represent human skin. The skin is connected to the skeleton by using a ‘skinning’ algorithm. The skin is essentially a mesh of polygons, almost always triangles, that interlock together to form a continuous and closed skin. The skinning algorithm allows the skin to deform and move along with the skeleton. The polygonal elements that make up the skin do not deform and are rigid, but the joints of the polygons are allowed to bend. The deformation of the skin is analogous to chain-link armor, where each joint cannot deform but the overall system is flexible and pliable. The skinning algorithm is essentially a mathematical formulation that assigns weights to certain polygons. These weights are usually ‘painted’ on by a 3D modeler. The process of skinning is an advanced and mature technique in the computer graphics world and has been utilized by digital human modeling. 1.2. Kinematic modeling of human mechanism The 55-degrees of freedom (DOF) Santos™ whole-body human model [1] includes the six global DOFs from a global origin as well as the lower/upper limbs, the torso, and the head (Figure 1). The pelvis is chosen as the base link for the global DOFs under rigid-body assumption. The configuration of the human open loop kinematic chains is described by the Denavit-Hartenberg (DH) notation [2]. The 55×1 joint variables uniquely determine the configuration of the system. The global motion (translations and rotations) of the body reference point located at the center of the human pelvis can also be described using the DH method with mass-free link-joint structure representing the global DOFs. The displacements of the first three prismatic joints describe the global translations of the pelvis in the 3-D xyz Cartesian coordinates (units in meters). The last three revolute joint variables represent the global orientations of the pelvis in terms of Euler angles (units in radians) in a 3-D space. The global DOF generalized coordinates are expressed as follows: where the superscripts (p) and (r) indicate the prismatic and the revolute joint, respectively, and α, β, and γ are the Euler angles of Z-X-Y type. Fig. 1. A whole-body human-like mechanism in zero joint variables and the global DOFs. 3846 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 Fig. 2. Biceps muscles in humans. Fig. 3. Simplified representation of the biceps muscles. 1.3. Background on musculoskeletal modeling Musculoskeletal modeling is a field that has been in development for decades now. Currently, several musculoskeletal simulation packages such as SIMM Biomechanics and AnyBody Technology are available commercially. These packages however, require a certain level of preprocessing before any results can be obtained and analyzed. Furthermore, their sole purpose is musculoskeletal simulation; human performance cannot be analyzed as comprehensively as using digital humans. Digital humans, that are developed based on ‘skinning’ algorithms, in contrast, lack musculoskeletal models that can be used for more accurate task simulations. This paper aims at introducing a combination of these two capabilities for the entire digital human body. Using the Santos® digital human developed by the Virtual Soldier Research (VSR) team, this paper introduces a preliminary full body musculoskeletal system that responds in real time, without any noticeable delay, on an avatar. 2. Methods 2.1. Transitioning from actual human muscles to computerized muscles Figure 2 presents what the bicep muscles look like in human beings. From the figure, three main parts can be identified; the origin tendon, the part that initially attaches the muscle to the skeleton; the insertion tendon, the other part that attaches the muscle to the skeleton; and the muscle bulk, the part that expands or contracts during muscle action (contraction, extension, adduction, abduction). Trying to model and simulate the volumetric muscles presented in the figure will prove to be highly computationally expensive. As such, a mathematical, less intensive model based on the formulation proposed by Charlton and Johnson [3] is used instead. More specifically, the modified Charlton and Johnson algorithm developed by Amos Patrick, from the VSR team, is used [4]. Figure 3 summarizes this formulation. The muscle is depicted as a single line, rather than a volume, that still starts at an origin position and ends at the corresponding insertion position, but wraps around an obstacle, usually a sphere or a cylinder, placed such that the muscle line runs through the centroid of the actual equivalent muscle. That is, the obstacles serve to mimic the minimum bulges that will keep the approximating muscle line passing through the centroid of the actual volumetric muscle. This is usually accomplished by running the line from the origin, through known points called via points on the obstacles, to the insertion. The line in this formulation is referred to as the muscle action line. Every muscle can be approximated with a number of muscle action lines. For most muscles such as the bicep muscles, just one line suffices. But for muscles such as the trapezius muscles whose insertion tendons cover a much larger area, more than one action line is needed. The model presented in this paper will be based on the approach Patrick took; the number of action lines per muscle will be kept to a minimum [4]. 2.2. Musculoskeletal modeling on Santos® 2.2.1. Inputs To extend the musculoskeletal modeling formulation onto an avatar such as Santos®, three parameters first have to be obtained for each muscle action line. These are: the origin position, the corresponding insertion position, and the obstacle(s) information (position, rotation, and scale). To accomplish this, 3D human skeletal, and volumetric 3847 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 muscle models are imported in the rendering environment and scaled to fit within the skin-based digital human, as it would in a real person. It is worth noting that these are just visual proxies; although all the models are stacked together, the Santos® avatar, and the imported models remain entirely independent from each other in function. Then, on a per-muscle basis, the parameters are obtained; 3D frames are manually placed where the origin and insertion tendons attach to the skeletal model, and their positions are retrieved relative to the Santos kinematic skeletal joints. Similarly, cylindrical obstacles are manually placed on the on the volumetric muscles such that the action line that would wrap around the obstacles will be at the centroid of the volumetric models. The obstacle parameters are then retrieved relative to the kinematic joints. The joints relative to which the parameters are obtained are determined based on biomedical intuition. Consider the bicep short muscle, for example. This muscle originates from the shoulder and inserts on the ulna with the main bulge associated with the muscle being around the elbow. On the Santos® joint based kinematic skeleton, this would translate into the origin position being obtained relative to the shoulder joint and the insertion and obstacle parameters being retrieved relative to the elbow joint, given the lack of an actual skeletal system. In this scenario, the origin is said to be ‘parented’ to the shoulder joint. Similarly, the insertion and obstacle are said to be parented to the elbow given that their information was retrieved relative to the elbow joint. Once these parameters are obtained, the obstacle wrapping algorithm can be run. 2.2.2. Wrapping algorithm The wrapping algorithm employed is based on the approach suggested by Charlton and Johnson [3], and Patrick [4], in which the muscle action line is assumed to be a frictionless elastic string. Two cases are to be considered: when wrapping around the obstacle is required and when it is not. For illustration, assume the muscle being considered only has one obstacle. The muscle action line runs through 7 points. The first and last points are the origin and insertion, respectively. The remaining 5 points are the via points which would be on the surface of the obstacle in case wrapping occurs, or linearly interpolated between points 1 and 7 in case wrapping does not occur. Using the obstacle position, rotation, and scale parameters, and the Santos® joints chain information, a transformation matrix is constructed to obtain the origin and insertion positions in terms of the cylinder coordinate system. If wrapping must occur, it will occur along the shortest path from the origin to insertion around the cylinder. This implies that the via points will follow a helical pattern the cylinder. On the onset, not enough information is available to determine whether wrapping occurs or not. The problem here is a 3D problem. Given two initial positions relative to a 3D cylinder with a longitudinal z-axis, 5 positions must be determined. To accomplish this, the wrapping algorithm suggests carrying out an analysis in 2 planes—the circular x-y plane and longitudinal x-z or y-z plane (since the base is circular, the x-z plane is similar to the y-z plane). The circular x-y plane is defined by the x and y coordinates of the origin, insertion and center of the cylinder, all three of which are purposely never collinear. From there, the x-y positions of via points 2 and 6 are determined. These positions are To and Ti in Figure 4a. With those, whether wrapping should occur or not can be assessed. Wrapping occurs if the z component of the cross product between and is negative, and does not occur if it is positive. Figure 4b highlights this difference. If wrapping does occur, the z-coordinate of To and Ti must be determined. From those two points, the remaining via points can be readily determined by radial interpolation. The next step is to convert the via points from the coordinate system of the cylinder to that of the environment so that a line may be rendered through them for a visual check. Finally, to create the real time effect, the origin, insertion, obstacle positions and consequently, all of the via points positions, are updated and recomputed every rendering frame to take into account any limb movement by the digital human. 2.2.3. Multiple obstacle wrapping For complex muscles, more than one obstacle is needed. For such muscles, the wrapping algorithm must be run in series for each obstacle. Figure 5 presents this approach. The wrapping algorithm is run on the first obstacle, and if wrapping occurs, the last point of intersection, Ti, is used as the origin position for the subsequent obstacle. 3848 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 Fig. 4. (a) Cylindrical wrapping in the x-y plane (b) Difference in wrapping depending on sign of z-component of cross product. [4] Fig. 5. Multiple obstacle wrapping work flow. Fig. 6. Key joint concept used to establish minimum and maximum muscle lengths. 2.2.4. Elongation feedback To accompany the real time musculoskeletal model, a novice scheme has been implemented for muscle elongation feedback. This scheme is based on the length of the muscle action line. First, a key joint for each muscle action line is determined. When the joint angle of this key joint is set to its maximum, the muscle action line attains its maximum length, and when the joint angle is set to its minimum, the muscle action line attains its minimum length. Once the minimum and maximum lengths of each muscle have been established, elongation feedback given any muscle length can be provided through linear interpolation. Figure 6 illustrates the concept. The key joints are determined on the Santos® avatar using a trial and error process. 3. Results and discussion 126 muscle action lines parameters were retrieved through manual placement of 3D frames and cylindrical obstacles. The parenting scheme for each of these muscles was tested and refined for proper biomedical motion. For illustrative purposes, consider the bicep short muscle again. Originally, the insertion position of the muscle was (a) (b) 3849 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 Fig. 7. Preliminary musculoskeletal system on Santos® with real time elongation feedback. retrieved relative to the Santos® kinematic wrist joint. However, parenting the insertion to the wrist joint resulted in the muscle action line of the bicep short muscle protruding from the digital human during limb manipulation. When the insertion position was retrieved relative to the elbow joint instead, the problem was eliminated and the muscle action line moved approximately as anatomically predicted. Furthermore, for each of these action lines, key joints that would set the minimum and maximum length limits were determined for elongation feedback. The result is a preliminary full body musculoskeletal system that deforms and responds in real time, while providing elongation feedback. Figure 7 presents a view of this capability. 3.1. Shortcomings of the musculoskeletal model Of the several hundreds of muscles in the human body, only 126 muscle action lines were able to be added to the Santos® digital human. This is primarily because of the fact that the Santos® kinematic skeleton is not representative of an actual human skeleton. For the muscles that were not added, changing the parenting scheme did not prove as successful as with the bicep short muscle; the action lines of those muscles still did not respond as anatomically expected. Furthermore, the elongation feedback currently implemented using the preliminary musculoskeletal system is based on minimum/maximum limits established by key joints. This dependency on key joints makes using this feedback mechanism as constraint in human task simulation redundant in the Santos® software which already uses joint angles as constraints in its simulations. A more scientifically sound feedback mechanism will involve having the ‘rest’ lengths of each muscle stored relative to a ‘neutral’ avatar position. Any deviation from those muscle ‘rest’ lengths would then be reported as positive or negative elongation as necessary. 3.1.1. Neutral posture A neutral posture is the body position at which muscle activation levels are at a minimum. In this position, the amount of electrical activity stimulating the muscles is at its minimum. A number of methods have to been used to accurately quantify this amount of electrical activity. Amongst them, the use of electromyography (EMG) signals has yielded the best results. In this method, either electrodes are attached to the skin, for superficial muscle activity detection, or needles are injected into the muscle, for more detailed analysis, and the electrical activity of the muscle recorded. 3850 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 In the Santos® model, neutral postures need to be examined and developed for standing, sitting and lying down. This is to ensure the proper calculation and execution of already existing tasks. EMG signals have been used to determine neutral postures for the three cases. 3.1.1.1. Standing While standing, the human body is under constant stress as a result of the body trying to balance itself. The ankle joint experiences the most stress as the center of mass falls in front of, and not in line with, it. Constant activity in the calves, and major superficial dorsal and ventral muscles of the torso is necessary for stability. This is similar to guy wires (tensioned cables designed to add stability) on a cell phone tower. Without this constant activity, an individual would not be able to stand properly [5]. The Checkley System suggested by Edwin Checkley [6] is recommended for Santos. Figure 8 presents Santos in this standing neutral posture. 3.1.1.2. Lying down Not much muscle activity is needed in lying down. The lying down posture is completely at rest due to the absence of energy expenditure from having to balance the body [7]. The Woodhull-McNeal posture is recommended for use as the lying down posture in Santos (See Figure 9). Note from Figure 9 that the current Santos skeleton does not allow for the neck joint to align with the spine as per the recommendation. 3.1.1.3. Sitting A muscle activation level study carried out by NASA in zero gravity is used as basis for the relaxed sitting position. According to it, the center of gravity of the head is situated slightly anterior to the atlanto-occipital joint (articulation between the atlas and the occipital bone). Body weight is supported by the ischial tuberosities (known as the sitting bones) and the adjacent soft tissues. The degree of the lumbar curve during the sitting posture depends on sacral angulation which is governed by pelvic posture and the degree of mobility/fixation of the involved segments [8]. Figure 10 presents Santos in the suggested posture. Fig. 10. Sitting position (a) Santos (b) NASA reference position [8]. a b a b a b Fig. 8. Neutral standing posture on Santos (a). Standing posture depicting natural and forcible carriage of the body (b) [6]. Fig. 9. Lying down neutral posture (a) Top view (b) Side view. 3851 Abhinav Sharma et al. / Procedia Manufacturing 3 ( 2015 ) 3844 – 3851 These postures, once properly implemented in Santos®, can be used as basis for better muscle elongation output. 4. Future work Given the described shortcomings of the current musculoskeletal system, the model is still incomplete. First, a new parenting scheme has to be devised to enable the addition of further muscles. Secondly, based on the performed study, human ‘neutral’ positions for standing, lying down, and sitting have to be implemented on the Santos® model. Once that is done and the Santos® digital human is set to those neutral postures, muscle action line ‘rest’ lengths can be computed and used for a more scientific elongation feedback. With a better elongation feedback scheme, the musculoskeletal model can be extended for use in task simulations. From simple tasks such as touching a point in space, to complex ones such as obstacle courses, the Santos® digital human will be able to perform these out while minimizing or maximizing muscle elongation as required. Once refined and proven adequate for biomedical studies, the musculoskeletal model will be extended onto the other Santos® software digital humans. 5. Conclusion This paper presents a breakthrough in full body musculoskeletal system modeling on digital humans. Musculoskeletal system modeling and simulation has long remained a field quite separate from digital human modeling. The former requires a certain level of preprocessing before results can be obtained and analyzed, whereas the latter bases its simulations and resulting feedback on kinematic joint and skin movement. By combining known literature on musculoskeletal modeling with gaming graphics architecture, this paper presents a preliminary full body mathematical musculoskeletal system on a digital human that deforms and responds in real time, along with an amateur muscle elongation feedback mechanism. Each muscle is approximated by one or more muscle action lines that start at the anatomically correct origin position, and travel through 5 via points to the corresponding insertion position. The via points are determined using a wrapping algorithm on cylindrical obstacles that emulate the minimum bulges required to keep the approximating muscle action line running through the centroid of the equivalent volumetric muscle. For each muscle, key joints that set the minimum and maximum muscle action line lengths are also determined for elongation feedback. This model currently consists of 126 muscle action lines. More muscles still need to be added. A more sound elongation feedback based on muscle action line length deviation from its ‘rest’ length still needs to be implemented. After this, human tasks can be simulated and analyzed using muscle elongation as a constraint. That is, a digital human such as Santos® can then be instructed to carry a simple task such as touch a point in space while minimizing or maximizing muscle action line elongation in the process. Acknowledgements The authors of this paper will like to acknowledge the Virtual Soldier Research program, the University of Iowa, Amos Patrick, John Looft, and Laura Freylaw. References [1] Xiang Y., Chung H.J., Mathai A., Rahmatalla S., Kim J.H, Marler T., Beck S., Yang J., Arora J., Abdel-Malek K. “Optimization-based Dynamic Human Walking Prediction,” Proceedings of SAE Digital Human Modeling for Design and Engineering, Seattle, WA, June 2007. [2] Denavit J., Hartenberg R.S., “A kinematic notation for lower-pair mechanisms based on matrices,” Journal of Applied Mechanics, Vol. 77, pp. 215-221, 1955. [3] Chalrton Ian W., Johnson Garth R., “Application of spherical and cylindrical wrapping algorithms in the musculoskeletal model of the upper limb,” Journal of Biomechanics 34, pp. 1209-1216, 2001. [4] Patrick A., Abdel-Malek K., “A Musculoskeletal Model of the Upper Limb for Real Time Interaction,” SAE Technical Paper 2007-01-2488, 2007 [5] Poppen R., Maurer J., “Electromyographic Analysis of Relaxed Postures,” Biofeedback and Self-Regulation, pp. 491-98, 1982. [6] Checkley E., A Natural Method of Physical Training, Making Muscle and Reducing Flesh without Dieting or Apparatus, Brooklyn, NY, 1892. [7] Woodhull-Mcneal A. P., “Activity in Torso Muscles during Relaxed Standing,” European Journal of Applied Physiology and Occupational Physiology, pp. 419-424, 1986. [8] Mount F. E., Whitmore M., Stealey S. L., “Evaluation of Neutral Body Posture on Shuttle Mission STS-57 (SPACEHUB-1),” pp. 1-10, 2003.