CCC 2018 Proceedings of the Creative Construction Conference (2018) Edited by: Miroslaw J. Skibniewski & Miklos Hajdu DOI 10.3311/CCC2018-137 Corresponding author: Author email: asadi@engr.psu.edu Creative Construction Conference 2018, CCC 2018, 30 June - 3 July 2018, Ljubljana, Slovenia Quantitative ways of measuring client’s preferences: a step toward creating an intelligent architectural design agent Mai Sherifa, Somayeh Asadia,*, Ebrahim Karanb aDepartment of Architectural Engineering, Pennsylvania State University, University Park, PA 16802 USA bDepartment of Applied Engineering, Safety, and Technology, Millersville University, Millersville, PA 17551, USA Abstract Technology-driven and digital modeling developments are undeniably changing design and construction professions across the architecture, engineering and construction (AEC) industry. Today, it is impossible to conceive of an architectural practice without using computer tools right from initial design conceptualization to the creation of construction drawings for a given project. The computer applications delivered new capabilities to automate the design process and offered various possibilities of assistant to the designer. However, they are not smart enough yet to understand clients or end users' (e.g. building occupants) needs or expectations and innovate a building design. Artificial Intelligence (AI) could be a solution to making architectural and building design process less time-demanding and more organized with minimal manual interference. The ultimate aim of this study is to make AI part of AEC’s digital journey, by making an intelligent agent able to both understand client needs and produce building designs. An intelligent design agent is defined as an autonomous entity that perceives client needs and makes design decisions towards achieving client satisfaction while balancing design and technical objectives. There are two challenges with respect to creating intelligent design agents; (1) finding quantitative ways of measuring client’s needs and preferences, and (2) providing a mathematical framework for making design decisions. This study focuses primarily on the first challenge and proposes a solution for creating measurable data about the client’s needs and preferences. First, an overview of technologies that support quantitative measurement of people's physical, emotional, and behavioral characteristics is presented. These technologies include eye tracking, facial expressions, electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), and galvanic skin response (GSR). This overview will enable us to understand potential applications of psychological measurement in the AI domain. This will be followed by a hypothetical design experiment to test the hypothesis that we can determine a subject’s degree of satisfaction by recording and analyzing his or her eye-movements and facial expressions when presented with a collection of visual data, like window design options on a screen. The participants are provided with window design options and asked to evaluate each design based on their preference by giving it a score. While study subjects complete evaluation tasks for each design option, we record their interactions using eye-tracking and an automated facial expression tool. The objective is to find a relationship between subjects’ preferences (as the independent variables) and the emotion expressed and the time spent on each design (as the dependent variables). This will allow us to provide a first look at how subjects interact with different attributes. © 2018 The Authors. Published by Diamond Congress Ltd., Budapest University of Technology and Economics Peer-review under responsibility of the scientific committee of the Creative Construction Conference 2018. Keywords: Eye-tracking; Facial expressions; End-user sstisfaction, Building design 1059 1. Introduction The first stage of the design process is evaluating the clients’ needs and requirements but the final design does not necessarily meet users’ satisfaction and expectations. Although the integration of computers and various revolutionary programs massively aid in the building design process and help avoid multiple issues that may arise, the design process is not entirely computerized yet. The integration of building end users in early design stages has been addressed and highlighted by various researchers [1]; however, the automation of that process has not yet been studied in detail. Building occupants are people who spend most of their times indoor and do not necessarily have the knowledge for designing the building; yet, they may have preferences about the way it has been designed and built [2]. The computer applications delivered new capabilities to automate the design process and offered various possibilities of assistant to the designer. However, they are not smart enough yet to understand clients or end users' (e.g. building occupants) needs or expectations and innovate a building design. Self-learning systems, known as artificial intelligence (AI) are changing the way architecture is practiced, as they do our daily lives, whether or not we realize it. In recent years, AI has been used in lots of projects since it is now able to show emotion but its application in design is still less explored but gradually gaining impetus. One of challenges is a design should be visually eye-catching to consider it as a good design. To overcome this challenge and integrate user’s perception in the design process, the use of eye tracking technology as well as facial expression has been suggested. Eye tracking investigates user's perception of real-time reaction to design, while conventional methods (i.e., interviews, focus group, questionnaires, etc.) have generally failed since they depend on users' willingness and competency to express their feeling about that particular design. Recent studies found that user's needs is not solely about functionality, and they have transformed to more empirical perspectives, not only encompass usability, but also socio-cognitive and emotional aspects of user experience such as joy, surprise, sadness, positive feelings interacting with design, etc. [3]. To this end, the ultimate aim of this study is to make AI part of architecture, engineering and construction (AEC)’s digital journey, by making an intelligent agent able to both understand client needs and produce building designs. AI has the potential to influence the architectural design process at a series of different construction stages, from site plan to building operation and could be a solution to making architectural and building design process less time-demanding and more organized with minimal manual interference. In the context of architectural design, AI can be designed as the use of computers to simulate architect’s intelligence and perform activities normally considered to require architectural knowledge. Although there are various different definitions of AIs, a common view of an intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success [4]. The architectural design environment is faced with a variety of design considerations and client’s expectations. A schematic of the intelligent agent model is provided in Fig.1. The perception of an intelligent architectural design agent is the capability to interpret client’s expectations in a manner that is similar to the way humans use their senses to relate to the world around them. Currently, there is no accurate and through definition of great design. However, if we can measure and quantify users’ emotional and biological data as they interact with buildings, it can be included in the design. The use of eye tracking technology to measure users’ satisfaction seems to be a suitable approach to achieve this objective. In particular, this study compares eye fixations as well as facial expressions to the scores provided by the participants in order to understand whether user’s satisfaction can be accurately predicted by the use of this technology. Fig. 1. Schematic of the intelligent architectural design agent CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1060 2. Background Many studies have explored the automation of building design; however, the complete digitalization of the building design process has not yet been achieved. Although technology driven developments are shaping the fundamental cores of the AEC industry, there still lays a gap in achieving total client satisfaction. Several of the technologies used in the Human-Computer Interaction (HCI) domain can facilitate the ability of machines to process client’s needs and understand them. These technologies include eye tracking, facial expressions, electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), and galvanic skin response (GSR). The use of eye tracking technology was previously explored in relevance to different applications, some of which examined the HCI as well as user’s satisfaction with web design. Sharma et al. (2014) highlighted the issues, scope, and limitations of the implementation of this technology to test a specific interface. They discussed the level of usability of eye tracking technology as a method HCI and summarized the different measurements of eye tracking techniques, such as fixations, saccades, pupil size, and blink rate. They found that eye tracking systems were proved to be resourceful in usability testing and an indication of satisfaction which make it possible for HCI aiding in decision making process [5]. Khalighy and Green (2015) developed a methodology for quantifying visual aesthetics in a product design by the use of eye-tracking technology. The obtained results were compared to declared preferences by the contributors. It was found that number, duration, and coordinated of eye fixations can certainly indicate accurate approximations to the product preference and satisfaction, solely based on aesthetics. This proves the plausibility to objectively evaluate designs before being taken into construction and manufacturing processes [6]. Ding et al. (2016) conducted a study to test the feasibility of using eye tracking technology to measure and assess the product design. T wo different tasks were designed for this study in order to analyze the captured attention and its relation to the user’s experience. Two factors were focused on throughout the tasks; first fixation and pupil dilations. It was concluded that time to first fixation, fixation time, and minimized pupil variations can be used to indicate end user preference and satisfaction [7]. The probable correlation between eye movements and end-user perception associated with modern web-based EUD (End-user development) environments were studied by Tzafilkou and Protogeros. The aim of this study was to assess whether end user perception and acceptance can be measured by eye behavior. Overall, the study confirmed the significant bond between fixation on eye position and end-user perception [8]. Furthermore, many studies highlighted the missing gaps of using eye tracking technologies to determine end-user satisfaction. The state of the art problems that were raised from previous researches of eye tracking technology are mentioned by Fengpei (2006). Some of the patterns that appeared in the usability studies were stated including a number of fixation points, time at fixation points, and average fixation time. Some of the concerning matters when dealing with eye tracking technologies that were highlighted are: eye movement data being lost, accurate extraction of eye data, and appropriate interpretation and explanation of the data collected. In the conclusion of this paper, it is advised to conduct future research answering to the previously mentioned issues of eye tracking technology [9]. In support of building end user incorporation in early design phases, the matter of whether an advanced level of user contribution can be used as an approach to develop sustainable energy technologies and buildings was raised by Ornetzeder and Rohracher (2006). Their study focused on user-led innovations managed by high levels of participation from different end-users. The results showed that the involvement of users in the design was extremely helpful and result in better-quality and more sustainable buildings [10]. Moreover, Ochoa and Capeluto (2008) conducted a study to examine the impact of integrating building intelligence to improve building energy performance and user comfort. They mainly highlighting the importance of the decisions made at early design stages rather than those added at the end. The result proved that truly intelligent buildings are a product of a design progression that integrates intelligence as well as end-user needs [11]. Though there are many efforts to participate building end-users in the design process, an efficient method of doing so has not yet been defined. A complete automation and computerization of this process at the conceptual phase of design would assist the achievement of ground-breaking means of efficient, sustainable, and intelligent building designs. A couple of studies discussed the use of eye-tracking and facial expression analysis as a method of evaluating the building end-user perception. Mohammadpour et al. (2015) concluded that higher visual attention was indeed assigned to parts that were more attractive to the [12]. CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1061 Moreover, Amiri et al. (2015) highlighted the fact that the combination of facial expression analysis along with eye- tracking data enhances the accuracy and evaluation of user satisfaction [13]. To conclude from previous studies, the use of eye tracking technologies seems like a suitable way to introduce AI into the current building design process. When mastered, this could aid reach better user-oriented sustainable designs with less effort and time wasted in communicating clients’ needs, requirements, and rework. The addition of facial expression recognition analysis along with the data collected by eye tracking provides the possibility of addressing some of the reliability gaps in the use of the eye tracking technology on its own. The ultimate goal of this study is to create an intelligent automated design process. 3. Methodology The use of eye tracking technologies along with facial expression analysis provides the means to quantify the building’s end-user satisfaction level. The eye-tracking experiment was conducted in a quiet and soft light lab using Tobii X2-30 compact eye-tracker to record eye movement data. The technical specification of the eye-tracker includes gaze accuracy of 0.4 to 0.5-degree range, precision of 0.32 to 0.45 degrees, data rate 30 Hz, freedom of head movement 50 (width) 36 (height) × 90 (depth) cm, calibration 9 points (~15 seconds to complete), monocular and binocular tracking capability. To set up this experiment, the eye tracking software (iMotions 6.1: human behavior research, simplified software) was set on screen recording mode and was linked to the slideshow. A simplified experiment was designed consisting of various options of window designs with minor adjustments in a form of the slideshow as shown in Fig.2. The first slide consisted of directions that explained the process requirements. Each slide compared between two different window designs; both their front view and angled view were shown (refer to Fig. 2). The participant was asked to look at the designs for a set period of time (10 seconds), write down a score from 1-6 (1 being the worst, 6 being the best) for each design (e.g. both preferred and undesirable design options), and then click on the preferred design. According to the participant’s preferred design option, a different set of designs was shown on the next slide. The participant was to repeat the process until the slideshow ends. This is a within-subjects experiment in which every single participant was subjected to every single window design. The facial expression recognition software (i.e., iMotions – Emotient) analyses the pattern of wrinkles and crevices based on a visual database of facial expressions and shows seven basic emotion indices including anger, contempt, disgust, fear, joy, sadness, and surprise and two aggregated emotion indices; positive and negative. Positive emotions are defined as emotions that are positively correlated with the evaluation scores and negative emotions are the ones with negative correlation. The Emotient engine (formerly known as FACET and previously CERT) automates Paul Ekman’s FACS coding system (Ekman and Rosenberg 1997) and uses a neural network to analyse the pattern of wrinkles and crevices created by different facial expressions. Main outputs are: 7 basic emotions, 2 advanced emotions including confusion and frustration, 33 facial landmarks such as corners of the eyes, the tip of the nose, and mouth, and operational distance of 70 cm. The emotion index scores ranged from 0 to 5, higher numbers on the emotion index indicate higher levels of feeling (e.g., happiness). a b Fig. 2. (a) experiment slideshow; (b) instructions shown after a set period of time After 10 seconds CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1062 To perform this experiment, first a pilot study was conducted to ensure accuracy. To start, the participant had to provide their name, age, and gender. Next, the eye tracker device was connected to the software and made sure to be working. Then, the calibration process was done to ensure the participant’s eye movements were accurately recorded. When the calibration process was successful, the slideshow started. Throughout the experiment, the tracker device collected data, and so did the camera for the facial expressions data. Twenty people of different ages and genders participated in this experiment as shown in Fig.2. Out of 20 participants, half of them were male and the other half was female. The average age of the participants was 24 with the standard deviation of 3.9. All participants had normal or corrected-to-normal vision. Since the data was recorded as a video, areas of interest (AOI) were defined before the experiment in order to analyse eye-movements. AOI(s) are relevant areas of the stimulus and are closely connected to the research question and the corresponding research design. For each AOI, several measures were calculated using iMotions: time to first fixation (time from the start of the design displayed until the participant fixated on the AOI for the first time), total fixation duration (duration of all fixations within an AOI), fixation count (number of times that a participant fixated on an AOI) and revisit (number of times that a participant returned to an AOI), and pupil diameter and dwell time (the sum time of fixation and saccade) for both views (i.e., front and side). More details on the eye movement measurements and experiment setup can be found in Yousefi et al. (2015). The measured eye-movement data was then compared to the scores given by the participants whilst conducting the experiment. The facial expression data was also recorded and compared to develop a trend. The correlations between subjective evaluation and eye- movement as well as the t-test analysis were conducted by SPSS 22.0 [14]. The setup of the slides presented to one of the participants based on the preferred designs along with the heat map is shown in Fig.3. Heat map shows the areas with a large amount of interest and the aggregated eye fixation of those areas which is shown in red. The absolute duration heat maps are created by combining data from all of the participants to provide a quick view of average existing viewing patterns. Heat maps clearly show which objects capture participants’ attention in an area. Although the absolute duration heat map demonstrates the average viewing pattern, the heat maps created per each participant for each image present individual visual attention. Fig.3 (a). Heat Map Fig.3 (b). A participant taking the experiment 4. Results and discussions Data on the target variable (i.e., evaluation score) is presented in Table 1. In the experiment, each participant was asked to complete 4 evaluation stages within about 1 minute. The participant was asked to evaluate two design options per stage and label a 6-point evaluation score to each design option, where 6 means the most satisfactory (or excellent) and 1 means the least (or poor). Then they would be guided to continue with the preferred design option (e.g., higher evaluation score) to the next evaluation stage. From Table 1, it can be observed that the average score for design options for all participants was 4.0 while preferred design options received 49% higher scores compared CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1063 to undesirable design options. Another observation is that evaluation scores for preferred design options gradually increased when moving from one stage to another. Table 1. Average values of evaluation scores in the window design experiments Undesirable Design Preferred Design Total Stage 1 3.1 4.7 3.9 Stage 2 3.1 4.7 3.9 Stage 3 3.8 4.9 4.3 Stage 4 2.9 4.9 3.9 Overall Evaluation 3.2 4.8 4.0 The results of two-sample t-test associated with a 95% significant level for the window design experiment is shown in Table 2. The null hypothesis states “there is no difference in means response between eye movement measurements (i.e. fixation and time spent) or emotion indices (i.e., fixation, time spent, joy, positivity, sadness, and surprise) and preferred and undesirable design”. The significance (2-tailed) value for fixation time periods is 0.005 which means that there is a statistically significant difference between the mean number of fixation time for preferred and undesirable designs. Fixations are those times when participants’ eyes essentially stopped scanning about the design option. Since the fixation time average for the preferred design is greater than undesirable design, we can conclude that participants stare more at preferred design options compared to undesirable options. Although the difference in spent time between preferred and undesirable design options was not statistically significant (p>0.05), but participants spent more time on designs with higher evaluation scores. In addition, all the studied emotion indices including joy, sadness, positivity, and surprise are statistically significant at the 95% confidence interval (Table 2). Similar to a picture which is worth a thousand words, our faces can reveal lots of information. Facial expressions may have advanced as efficient methods to transmit feelings and intentions and make it possible to communicate intelligent emotions with a simple raised eyebrow or curl of the lip. Table 2. Quantitative analysis of t-test for different window design experiments Variables Design Evaluation Mean Score Std. deviation F Sig. (2-tailed) Fixation Preferred 7.18 4.03 0.672 0.005 Undesirable 5.48 3.55 Time Spent Preferred 2.12 1.28 0.008 0.078 Undesirable 1.72 1.57 Joy Preferred 1.04 0.38 0.64 <0.001 Undesirable 0.49 0.31 Positivity Preferred 1.07 0.40 0.45 <0.001 Undesirable 0.53 0.32 Sadness Preferred 0.48 0.19 31.8 <0.001 Undesirable 0.77 0.40 Surprise Preferred 0.47 0.19 32.7 0 Undesirable 0.76 0.38 The Pearson correlation between the evolution scores and the eye tracking and emotion indices is shown in Table 3. The correlation between positivity index and participant’s evaluation is higher in compare with other indices (correlation coefficient = 0.633) and statistically significant at 95% confidence level. This means that there is a very close relationship between positivity index and participant’s evaluation scores. In another words, participants showed relatively higher positive emotions for the designs (or components) they liked. The correlation between joy and participant’s evaluation scores is moderately high which indicate a close relationship between them. The negative significant correlation at 95% confidence level exists between both sadness and surprise with participant’s evaluation scores (respectively, -0.432 and -0.467). This means that low evaluation scores were associated with high surprise and participants expressed higher sadness emotions for the designs (or components) they disliked. Both fixation and time spent have a positive correlation with participant’s evaluation scores and are significant at the 95% confidence level. CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1064 Table 3. Comparison of eye tracking and emotion indices with evolution scores Variables Pearson correlation Significance Fixation 0.372 <0.001 Time Spent 0.249 <0.001 Joy 0.578 <0.001 Positivity 0.633 <0.001 Sadness -0.432 <0.001 Surprise -0.467 <0.001 The correlation between eye movement measurements and emotion indices and evaluation score is illustrated in Fig. 4. Correlation plot is the visual presentation of a matrix of Pearson's rank correlation coefficients between each variable. The trend line showed in red indicates the direction between each variable and evaluation score and the black circles show the level of strength between them. A positive association exists between fixation, time spent, joy, and positive emotion with evaluation score while the sadness and surprise have a negative association. In addition, these figures illustrate a linear relationship which means that the points on the scatterplot closely resemble a straight line. A relationship is linear if one variable increases by approximately the same rate as the other variables changes by one unit. a b c d e f Figure 4. Correlation plot between eye movement measurements and emotion indices and evaluation score 5. Conclusion The ultimate aim of this study is to make AI part of AEC’s digital journey, by making an intelligent agent able to both understand client needs and produce building designs. As a first step toward achieving this goal, in this study a quantitative method was used to measure building end-user satisfaction at the early stages of design process. To achieve this objective, twenty people were asked to look at a set of two different window designs at each stage, evaluate both designs, and select a preferred one. Meanwhile, their eye movements were being tracked and recorded in addition to their facial expressions and reactions to the presented designs. A statistical analysis using SPSS software was performed to analyze the collected data. The results of two-sample t-test showed there is a statistically significant difference between the mean number of fixation time for preferred and undesirable designs at 95% confidence interval. This means that participants stare more at preferred design options compared to undesirable options. However, the difference in spent time between preferred and undesirable design options was not statistically CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1065 significant (p>0.05), but it was found that participants spent more time on designs with higher evaluation scores. In addition, all the studied emotion indices including joy, sadness, positivity, and surprise are statistically significant at the 95% confidence interval. The correlation between eye movement measurements and emotion indices and evaluation score is illustrated in Fig. 4. Besides, a Pearson’s correlation study was performed which showed a positive association between fixation, time spent, joy, and positive emotion with evaluation score while the sadness and surprise have a negative association. The combination of the data collected by eye tracking and that from facial expression analysis provides us with a better understanding of how people evaluate designs and rate their level of satisfaction. It provides us with the first step in creating a smarter and digitalized design process. References [1] Negendahl, K. (2015). Building performance simulation in the early design stage: An introduction to integrated dynamic models. Automation in Construction, 54, 39-53. [2] McDonnell, J. (2009). "Collaborative negotiation in design: A study of design conversations between architect and building users." CoDesign, 5(1), 35-50. [3] Leitch, K. A., Duncan, S. E., O'Keefe, S., Rudd, R., & Gallagher, D. L. (2015). Characterizing consumer emotional respons e to sweeteners using an emotion terminology questionnaire and facial expression analysis. Food Research International, 76, 283-292. [4] L. Padgham and M. Winikoff, Developing Intelligent Agent Systems, John Wiley & Sons Ltd., England, 2004, pp.1 - 6. [5] Sharma, K., Jermann, P., & Dillenbourg, P. (2014). How Students Learn using MOOCs: An Eye-tracking Insight. Proceedings of the European MOOC Stakeholder Summit 2014, (2003), 147–154. [6] Khalighy, S., Green, G., Scheepers, C., & Whittet, C. (2015). Quantifying the qualities of aesthetics in product design u sing eye-tracking technology. International Journal of Industrial Ergonomics, 49, 31–43. https://doi.org/10.1016/j.ergon.2015.05.011. [7] Guo, F., Ding, Y., Liu, W., Liu, C., & Zhang, X. (2016). Can eye-tracking data be measured to assess product design?: Visual attention mechanism should be considered. International Journal of Industrial Ergonomics, 53, 229–235. https://doi.org/10.1016/j.ergon.2015.12.001. [8] Tzafilkou, K., & Protogeros, N. (2017). Diagnosing user perception and acceptance using eye tracking in web-based end-user development. Computers in Human Behavior, 72, 23–37. https://doi.org/10.1016/j.chb.2017.02.035. [9] Fengpei, H. (2006). The studies of eye tracking and usability test. 10.1109/CAIDCD.2006.329343. [10] Ornetzeder, M., & Rohracher, H. (2006). User-led innovations and participation processes: Lessons from sustainable energy technologies. Energy Policy, 34(2 SPEC. ISS.), 138–150. https://doi.org/10.1016/j.enpol.2004.08.037. [11] Ochoa, C. E., & Capeluto, I. G. (2008). Strategic decision-making for intelligent buildings: Comparative impact of passive design strategies and active features in a hot climate. Building and Environment, 43(11), 1829–1839. https://doi.org/10.1016/j.buildenv.2007.10.018. [12] Mohammadpour, A., Karan, E., Asadi, S., & Rothrock, L. (2015). Measuring end-user satisfaction in the design of building projects using eye-tracking technology. Congress on Computing in Civil Engineering, Proceedings, 2015–Janua(January), 564–571. https://doi.org/doi:10.1061/9780784479247.070. [13] Amiri, S. S., Masoudi, M., Asadi, S., & Karan, E. (2015). A Quantitative Way for Measuring the Building User Design Feedback and Evaluation, (Thompson 2012), 1813–1819. [14] Yousefi, M. V. (2015). Implementing Eye Tracking Technology in the Construction Process. 51st ASC Annual International Conference Proceedings, 752–759. CCC 2018 Proceedings DOI 10.3311/CCC2018-137 1066 https://doi.org/10.1016/j.ergon.2015.05.011 https://doi.org/10.1016/j.ergon.2015.12.001 https://doi.org/10.1016/j.chb.2017.02.035 https://doi.org/10.1109/CAIDCD.2006.329343 https://doi.org/10.1016/j.buildenv.2007.10.018