Modeling Visually Guided Hand Reach for Digital Human Models 2351-9789 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AHFE Conference doi: 10.1016/j.promfg.2015.07.885 Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 Available online at www.sciencedirect.com ScienceDirect 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE 2015 Modeling visually guided hand reach for Digital Human Models Nitesh Bhatiaa, Anand V. Pathakb, Dibakar Sena,* aCentre for Product Design and Manufacturing, Indian Institute of Science, Bangalore, India bSpace Applications Centre, Indian Space Research Organisation, Ahmedabad, India Abstract Digital Human Models (DHM) are used in digital mockups to identify human factor problems in design and assembly. Present applications are mostly limited to the posture, biomechanics, reach and simple visibility analysis and they are mostly developed for independent simulations that are mostly hand crafted by designers. Our aim is to develop natural simulations using vision as a feedback agent for performing any postural simulations similar to humans. In this paper, the work presented is limited to demonstrating active vision based feedback for a typical hand reach task without using inverse kinematics. The proposed concept takes into account of previously developed vision and hand modules and describes an integration methodology such that both modules can work in tandem providing feedback and feed forward mechanisms. The scheme primarily utilizes vision module that acts similar to human eyes by providing spatial information about hand and object in workspace. Similar to retinal projection, the workspace object and the model of DHM hand is geometrically projected over the grid and the relative positions are computed in terms of grid-cells. The computed relative positions are used to compute a vector direction that is provided as a feedback to hand for guiding it towards object. The hand module independently is capable of natural grasping and visual feedback is used for motion guidance. The implementation shown in this paper is limited to monocular vision and two-dimensional hand movement as a proof of concept. This scheme is used to demonstrate a scenario where DHM is successfully able guide the hand and point it to a given object. The presented model finally shows vision as a guiding agent for hand reach simulations. It can be used for planning and placement of workspace objects to enhance human task performance. © 2015 The Authors. Published by Elsevier B.V. Peer-review under responsibility of AHFE Conference. Keywords: Digital Human Models; Hand reach tasks; Digital mockups; Hand reach simulations * Corresponding author. Tel.: +91-80-2293-3230; fax: +91-80-2360-1975. E-mail address: dibakar@cpdm.iisc.ernet.in © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AHFE Conference http://crossmark.crossref.org/dialog/?doi=10.1016/j.promfg.2015.07.885&domain=pdf 3821 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 1. Introduction Vision is important for processing information about the spatial location of objects and object characteristics such as shape, size, weight and texture. Reaching for an object in visual space is a simple process that involves bringing hand closer to an object by using visual feedback as a guiding medium. For visual tasks, reach would include moving the head-eye complex so as to align the target location at the fovea of the eye and to bring the images to the focus in a binocular vision scenario [1,2,3]. Reach is the step when parts of the body move in the free space to assume some desired configuration with respect to the parts/objects in the scene. The mathematical framework of direct and inverse kinematics, path planning, collision avoidance etc. is the support available in the existing literature for their use in DHM. However the contexts of variable functional kinematic structure, purposeful grasping etc. are the challenges that are still unresolved in literature. Unlike robotics where manipulations are predominantly point referenced, human activities are object referenced. Hence the computational paradigm is inherently fuzzy and iterative. Moreover, real life man- machine interaction not only involves errors in decision making but also errors in execution which affects his performance. For visual tasks, reach would include moving the head-eye complex so as to align the target location at the fovea of the eye and to bring the images to the focus in a binocular vision scenario. Modelling and simulation of such tasks require a close integration of visual and manual feedback, for which the present state of art in human simulation technology - the digital human models (DHM), provides valuable but very limited information. It is not possible to design and simulate tasks and evaluate the human performance, which is highly dependent on operator’s visual capabilities. Thus, the possibilities of providing sensory assistance to DHM opens up new opportunities towards simulation of natural performance for automatic identification of issues in a given work scenario [4]. In this paper we have presented an “active” vision based scenario to demonstrate visually guided simple hand reach for simulation of reach movements. The model takes its inspiration from the way human performs reach using active visual feedback and hand feed forward mechanisms. Our scheme is based on three interconnected modules viz. vision, hand and cognition. The vision module takes account of central and peripheral vision and provides feedbacks in terms of spatial location of object and hand located in visual workspace. The hand module consists of a basic hand model capable of movement across a plane on the basis of input obtained in terms of direction vectors. The cognition module has the central role of planning and execution of hand movement on the basis of visual feedback. A primitive task is simulated to show that this presented model is successfully able guide the hand and point it to a given object. In section 2 we discuss the relevant literature about reach modelling in existing human simulation systems. In section 3 we describe the newly developed active vision based reach framework and data- structure of a variable resolution cube-grid acting as receptor surface for sampling of spatial information in DHM vision model. In section 4, we present simulations and results based on functional implementation of discussed framework. Conclusions to the presented work are presented in section 5. 2. Literature Being a part of complex social system, humans constantly interact with each other and surroundings [5]. Hence maintaining sustainability and cost effectiveness is a desirable goal while designing any such system. Understanding and modeling the human behavior can effectively help in responding and adapting to the uncertainties that may arise while dealing with humans and the materials. In order to maximize human performance the elements of purpose, people, structure, techniques and information must be coordinated and integrated appropriately [6,7]. Human behavior and performance in any given system is based on biomechanical, physiological, and psychological capabilities of the human [8]. For any kind of interaction human vision, is the primary channel for processing perceptual information. Under human visual system capabilities, biomechanical includes head and eye anthropometry that act as the primary sources for sensory spatial information; physiological includes the eyes and brain with their combined ability to collect visible spatial information; and psychological, involves processing, interpretation and response generation based on the knowledge and experience of humans. Different phases in human performance include spatial searching, object recognition and localization, reaching to grasp, manipulation of objects, evaluation of the task progress, and completion [9,10]. Traditional methods for design and evaluation of 3822 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 such systems include experimental strategies, ethnographic design, and behavioral observation; open ended interviews, etc [11]. However, generally these studies are regionally confined and it is not feasible every time to assess the interaction of a large human population with system for testing each and every aspect of design. In recent times, moving away from traditional design process and using Digital Human Models (DHM) for virtual simulations of human interaction with the product has helped the designers significantly. DHM technology comprises of computer generated human simulation medium that is primarily used for simulation and visualization and assessment of human tasks in a virtual workspace [12]. This technology offers human factors and ergonomics specialists the promise of an efficient means to simulate a large variety of ergonomics issues early in the design of products and manufacturing workstations. With this advanced technology, human factors issues are assessed in virtual digital prototype of workstation with digital human model. Most products and manufacturing work settings are specified and designed by using sophisticated computer-aided design (CAD) systems. By integrating a computer-rendered avatar (or humanoid) and the CAD-rendered graphics of a prospective workspace, one can simulate issues regarding who can fit, reach, see, manipulate, and so on [12,13]. The implementation of digital human model reduces and sometimes eliminates the requirement of dummy model, cardboard manikin, 2D drawings and even real human trial in expensive physical mock-ups [14]. This technology has reduced the design time, cycle time and cost of designing new products along with improvements in quality, production, operation and maintenance costs. In past few decades, lot of work had been done in the area of posture and hand modeling. Being a relatively new area of research and development current DHM applications are mostly dominant towards whole body posture and biomechanical analysis used for simulations of material handling tasks mostly dominated in the areas of automotive production, assembly line simulations and vehicle safety [15,16]. Hand modeling and its interaction with objects often requires a realistic simulation of the hand-object interaction and a reliable estimation of performance. For grasping and reach simulations hand model specific approaches have been developed [17,18]. In [19,20], authors have shown usage performance, usage durations, and the handling (grasp) qualities of hand held devices. In [21], authors have developed Grasp quality Index that can be used for estimation of finger reach for a given element. All these approaches concentrate in the area of reach and grasp modeling and most don’t take care of human performance related applications. Human performance in any given task-workspace is based on biomechanical, physiological, and psychological capabilities of an operator8 where vision plays an important role in by providing spatial characteristics of visual space and frame of body. While interacting with workspace objects, visual information is either received directly while interacting with tools, reading text, etc. or indirectly by seeing things through monitors, microscopes, etc. There are several factors that determine visual capabilities of the operator and the effectiveness of corresponding tasks. The most important one include operator’s FoV, visual acuity and accommodation capabilities. For modeling and simulation of vision dependent tasks, the present DHM tools are mostly limited to symmetric FoV cones and line of sight based passive visibility analysis. For instance, in [12] JACK has been used to evaluate vehicle dashboard visibility and driver’s visual field using the uniform Field of Vision (FoV) cones. In [22] similar FoV cones are used to assess direct exterior vision of a postal delivery vehicle driver. Similar uniform FoV based vision analysis tools are available in other DHMs like Humancad/SAMMIE, RAMSIS, etc. For simulation of traditional assembly tasks active and continuous visual feedback is required during alignment and fastening phases. Vision plays a key role in precision assembly tasks and requires a close integration of visual and manual feedback. For simulation of such tasks, the present state of art in DHM technology provides valuable but very limited information. Hence it is not possible to design and simulate tasks and evaluate the human performances that are highly dependent on operator’s visual capabilities. Our primary aim is to use and simulate DHMs, digital hand models, and vision models for natural human simulations such that usability tests for human performance can be developed in the early phases of system development in order to save cost and labor. For this purpose there is a need to combine these independent models into a single high level framework that can work in tandem to perform human tasks without needing any handmade tinkering. Literature related to human behavior modeling generally talks about the approaches to combine anthropometric models and cognitive modeling. Using such models, the natural human simulations can be achieved using a DHM in a virtual environment [23,24]. In [25,26], the authors have considered cognitive aspects of behavior 3823 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 in human simulations. The existing independent models have been taken up by industry and commercial products since they produce stable results that match with humans. But developing the interdependent modules on the basis of these independent models are not yet mature enough, hence the industry uptake is very limited. Our aim is to develop a human behavior model where vision model can be as a feedback agent for performing any postural simulations similar to humans. In this paper, the work presented is limited to demonstrating active vision based feedback for hand reach task that is discussed further. 3. Vision guided hand reach simulations There are several motion simulation approaches that have been followed to produce reach and grasp motions. In general these techniques can be divides into two types as described by Weeber [27]. First kind is artificial animation where the designer usually imagines the desired motion before running the simulation. These techniques include key posture interpolation and frame based interpolation. In a key posture based simulation, the designer provides key postures and for motion the successive frames are calculated by interpolation [28]. In frame based animation, the motion is animated by applying inverse kinematics to each frame of the motion given by designer [29]. Several more advanced techniques like Motion Blending and Functional Regression have also been developed on the similar grounds [30,31]. Second kind is automated simulations that generally produce natural and human like behaviors. The system per se generates the motion based on designer’s input in form of some basic human attributes. The motion generated is automated and user knows less about what motion should result. Some techniques involve manual and artificial inverse kinematics based methods [32,33]. For cohesive and modular development of simulations, Reed et al proposed an approach where motion simulation is achieved by interconnected small and individual modules like task-oriented head and eye movement, posture balancing, etc [34]. We have followed similar technique to demonstrate vision guided hand reach simulations. The scheme utilizes two modules viz. vision model and hand model that have been developed by the authors. We have shown a use case where vision model can provide feedback to the hand model for guided motion towards an object. Fig. 1 shows the data flow architecture of proposed scheme. The modules are described as follows. 3.1. Vision model Sen. et. al. [35,36] introduced a DHM vision modeling framework for performing gaze-dependent FoV based workspace visibility analysis and object legibility analysis. Based on a 3D scanned human head, this model computes realistic gaze-dependent field of view (FoV) with respect to location of pupil and the facial features. The FoV is further divided into central and peripheral vision that simulates human acuity dependent vision for DHM. The analysis done using this model is based on actual human visual parameters that can be personalized across population and workspace objects. The computational framework behind this model relies on a uniform cube-grid having 6 faces. It integrates a unit-cube representation of a 360 direction with each face covering 90. In [36], the uniform resolution cube-grid from35, was enhanced to a variable resolution cube-grid. The resolution was taken is Fig. 1. Data Flow architecture of Vision Guided Hand Reach. 3824 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 Fig. 2. Visual output of Cube-Grid for the sentence,“The quick brown fox jumps over the lazy dog.”, of height 1.8 cm, when shown at a distance of 1 m, where the point of fixation is at first letter “T” [36]. similar to human acuity resolution, such that information projected on the cube-grid can be sampled similar to human retinal sampling. To simplify the model, the FoV is quantified into three regions viz. central (2º FoV), medium (2º to 20º FoV), and peripheral (beyond 20º). As an standard for each region, an average human acuity of 1.0, 0.3 and 0.1 was taken respectively. Fig. 2 shows the visual output of Cube-Grid for the sentence, “The quick brown fox jumps over the lazy dog.”, of height 1.8 cm, when shown at a distance of 1 m, where the point of fixation is at first letter “T” located in the central vision and remaining sentence stretches till the peripheral vision. In this work we have divided FoV into two regions central FoV ranging within 2º and peripheral FoV beyond 2º. A 3D model of hand and an object located in DHM workspace are geometrically projected on the cube-grid that acts as the receptor surface. The object of interest remains located in central vision region and hand can be located in any part of FoV. For the purpose of computation of a direction vector ‘D’, the center of object is considered as origin and is denoted by ‘O’. To compute the location of projected hand on the retinal surface, mean of occupied cells is taken as position ’H’. Hence vector ‘D’ is denoted is difference of vector ‘O’ and vector ‘H’. The vector provides a guiding direction to the hand. 3.2. Hand model Sen. et. al.17 developed a systematic hand manipulation schemes using the concept of relational description which is abstract (non-numeric) and yet precise; it can be used for describing hand-object interaction pattern using any hand model. The hand model is developed using 3D scan of a real human hand. The hand model is capable of performing six degrees of freedom motion dominant reorientation for grasping and force dominant behavior for slip prevention grasp. For the current work, we have considered monocular vision model and hence limited the hand motion in two dimension perspective parallel to the eye plane. For motion, the hand takes an input feedback in terms of direction vector ‘D’ computed by vision model and produces a motion in the same direction. The step size of distance is provided as a user input. Upon completion of a step, the position of hand and object is recomputed by the vision model and vector ‘D’ is recomputed for feedback. If the complete hand or a part of hand falls under central FoV the reach algorithm stops and hand interaction related methods are called further. Fig. 3 shows the flowchart of steps followed in performing proposed reach simulation. Fig. 3. Flow chart of instructions followed in Vision Guided Hand Reach. 3825 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 4. Simulation results For the purpose of simple simulation we have considered a task that reaching towards an object located in visual field. The object is located in front of DHM head and the hand is located in the periphery as shown in Fig. 4. In first step the visibility of object and its position on the visual grid is computed. Fig. 4a shows the highlighted object whose position is computed as vector O. In second step the hand is projected on the vision model that computed its position vector H with respect to object O as shown in Fig. 4b. Vector ‘D’ is computed on the basis of location of hand and object (Fig. 4c). This vector is given as an input to the hand model. In the current case Vector ‘D’ represents a straight line along negative X axis and hence the direction coordinates given to hand model as input are (-1.0, 0.0, 0.0). The step size is chosen as 20 units. In next step the hand is shifted in direction of vector ‘D’ by 20 units (Fig. 4d). The reach procedure starts again at this step and position of object and new position of hand H’ is computed. In this case since a part hand is already inside central FoV, some of the cells occupied by projection of hand come under central FoV. At this stage the reach algorithm stops. 5. Conclusion and future work In this paper, we have shown a methodology to achieve natural human behavior by integrating a cognitive vision model with anthropometric hand model. The work however is limited to demonstrating active vision based feedback for a typical hand reach task without using inverse kinematics. The presented model takes into account of our previously developed vision and hand modules and describes an integration methodology such that both modules can work in tandem providing feedback and feed forward mechanisms. The scheme shown primarily utilizes vision module and hand module. Vision model gathers spatial information about hand and object in workspace and computes the position and a vector direction from pointing hand to object. The hand module independently is capable of natural grasping. For reach visual vector direction is used as a feedback for motion guidance. The implementation shown in this paper is limited to monocular vision and two-dimensional hand movement as a proof Fig. 4. Vision guided hand simulations. 3826 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 of concept. This scheme is used to demonstrate a scenario where DHM is successfully able guide the hand and point it to a given object. The presented concept demonstrates a natural behavior using vision as a guiding agent for hand reach simulations. It can be used for planning and placement of workspace objects to enhance human task performance. In future we plan to integrate binocular vision model along with three dimensional hand models to simulate reach related tasks in any given workspace. Acknowledgements The work presented in this paper was partially funded by the Indian Space Research Organization (ISRO) and Indian Institute of Science (IISc) - Space Technology Cell (STC), under a project scheme IISc/ISTC0288. References [1] Marc Jeannerod. Intersegmental coordination during reaching at natural visual objects. Attention and performance IX, 9:153–168, 1981. [2] Marc Jeannerod. The timing of natural prehension movements. Journal of motor behavior, 16(3):235–254, 1984. [3] D Pelisson, C Prablanc, MA Goodale, and M Jeannerod. Visual control of reaching movements without vision of the limb. Experimental Brain Research, 62(2):303–311, 1986. [4] Daniel W Carruth, Mark D Thomas, Bryan Robbins, and Alex Morais. Integrating Perception , Cognition and Action for Digital Human Modeling. pages 333–342, 2007. [5] C. Zastrow and K. Kirst-Ashman. Understanding Human Behavior and the Social Environment. Available Titles CengageNOW Series. Cengage Learning, 2009. [6] W.A. Randolph and R.S. Blackburn. Managing Organizational Behavior. Irwin Series in Marketing. Irwin, 1989. [7] L.A. Montuori. Organizational longevity integrating systems thinking, learning and conceptual complexity. Journal of Organizational Change Management, 13(1):61–73, 2000. [8] D.A. Attwood, P.D.C.P.E.M.E.S. Joseph M. Deeb, and M.E. Danz-Reece. Ergonomic Solutions for the Process Industries. Elsevier Science, 2004. [9] Mary Hayhoe. Vision using routines: A functional account of vision. Visual Cognition, 7(1-3):43–64, 2000. [10] Donald A Norman and Tim Shallice. Attention to action: Willed and automatic control of behavior. Technical report, DTIC Document, 1980. [11] J.W. Creswell. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications, 2009. [12] Don B Chaffn. Digital human modeling for vehicle and workplace design. Society of Automotive Engineers, Inc., Warrendale, USA, 2001. [13] Don B Chaffn. Improving digital human modelling for proactive ergonomics in design. Ergonomics, 48(5):478–491, 2005. [14] Tara M Cappelli and Vincent G Duffy. Motion capture for job risk classifications incorporating dynamic aspects of work. Technical report, SAE Technical Paper, 2006. [15] Ulrich Berger, Raffaello Lepratti, and Henning Otte. Application of digital human modellng concepts for automotive production. In Proceedings of TMCE 2004, pages 365–373. Millpress, Rotterdam, 2004. [16] Giorgio Colombo and Umberto Cugini. Virtual manikins and prototypes to evaluate ergonomics safety. In Proceedings of TMCE 2004, pages 375–382. Millpress, Rotterdam, 2004. [17] JS Vipin and Dibakar Sen. Simulating grasping behavior of people using the relational description scheme. 2011. [18] Masaaki Mochimaru, Makiko Kouchi, Natsuki Miyata, Mitsunori Tada, Yoshiaki Yoshida, Kei Aoki, Katsuaki Kawachi, and Yoichi Motomura. Dhaiba: Functional human models to represent variation of shape, motion and subjective assessment. Technical report, SAE Technical Paper, 2006. [19] Natsuki Miyata, Makiko Kouchi, and Masaaki Mochimaru. Posture estimation for screening design alternatives by dhaibahand-cell phone operation. Technical report, SAE Technical Paper, 2006. [20] Yui Endo, Satoshi Kanai, Takeshi Kishinami, Natsuki Miyata, Makiko Kouchi, and Masaaki Mochimaru. An application of a digital hand to ergonomic assessment of handheld information appliances. Technical report, SAE Technical Paper, 2006. [21] Jingzhou Yang, Esteban Pena Pitarch, Joo Kim, and Karim Abdel-Malek. Posture prediction and force/torque analysis for human hands. Technical report, SAE Technical Paper, 2006. [22] Matthew P Reed and Kristy Satchell. Application of Digital Human Modeling to the Design of a Postal Delivery Vehicle. (724), 2005. [23] Nico Delleman. About avatars and maneuvering in virtual environments. Technical report, SAE Technical Paper, 2006. [24] John Rasmussen, Søren Tørholm Christensen, Karl Siebertz, and Jessica Rausch. Posture and movement prediction by means of musculoskeletal optimization. In 2006 Digital Human Modeling for Design and Engineering Conference, 2006. [25] Jennifer McGovern Narkevicius, Timothy M Bagnall, Robert A Sargent, and John E Owen. Networking human performance models to further utility of process models: The seaprint approach. Technical report, SAE Technical Paper, 2006. [26] Brian F Gore and Paul Milgram. The conceptual development of a time estimation model to predict human performance in complex environments. Technical report, SAE Technical Paper, 2006. 3827 Nitesh Bhatia et al. / Procedia Manufacturing 3 ( 2015 ) 3820 – 3827 [27] Bonnie LWebber, Cary B Phillips, and Norman I Badler. Simulating humans: Computer graphics, animation, and control. Center for Human Modeling and Simulation, page 68, 1993. [28] Michael Gleicher. Comparing constraint-based motion editing methods. Graphical models, 63(2):107–134, 2001. [29] Yahya Aydin and Masayuki Nakajima. Database guided computer animation of human grasping using forward and inverse kinematics. Computers & Graphics, 23(1):145–154, 1999. [30] Franck Multon, Laure France, Marie-Paule Cani-Gascuel, and Gilles Debunne. Computer animation of human walking: a survey. The journal of visualization and computer animation, 10(1):39–54, 1999. [31] Munetoshi Unuma, Ken Anjyo, and Ryozo Takeuchi. Fourier principles for emotion-based human figure animation. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 91–96. ACM, 1995. [32] Daniel E Whitney. Resolved motion rate control of manipulators and human prostheses. IEEE Transactions on man-machine systems, 1969. [33] Alain Liegeois. Automatic supervisory control of the configuration and behavior of multibody mechanisms. IEEE transactions on systems, man, and cybernetics, 7(12):868–871, 1977. [34] Matthew P Reed, Julian Faraway, Don B Chaffn, and Bernard J Martin. The humosim ergonomics framework: A new approach to digital human simulation for ergonomic analysis. Technical report, SAE Technical Paper, 2006. [35] Dibakar Sen and V Vinayak. A vision modeling framework for dhm using geometrically estimated fov. Computer-Aided Design, 44(1):15– 28, 2012. [36] Nitesh Bhatia, Dibakar Sen, and Anand V. Pathak. Legibility assessment for functional vision of dhm using differential acuity. In Imre Horvth and Zoltan Rusk, editors, Proceedings of TMCE 2014, pages 1235–1244. Organizing Committee of the International TMCE 2014 Symposium, May 2014.