key: cord-024698-jrnw1i07
authors: Hogan, Kevin; Shan, Fei; Ravichandran, Monikka; Bagmar, Aadesh; Wang, James; Sarsony, Adam; Purtilo, James
title: Decidio: A Pilot Implementation and User Study of a Novel Decision-Support System
date: 2020-04-25
journal: Group Decision and Negotiation: A Multidisciplinary Perspective
DOI: 10.1007/978-3-030-48641-9_14
sha: 
doc_id: 24698
cord_uid: jrnw1i07

In this work, we add to the rich history of decision-support system research by implementing and evaluating a pilot implementation of a novel system, which we call Decidio. Our tool was integrated into a pre-existing decision-making process regularly conducted by 9 teams of undergraduate students. We find an overall positive response to Decidio based on the results of a tool evaluation survey that we conducted after our experiment. Furthermore, we conduct a Big-Five Factor personality survey of participants and associate personalities with interactions recorded by our tool. We find that the students who demonstrate leadership behaviors through their interactions score higher in extraversion and lower in conscientiousness than other students. Our analysis also reveals that agreeableness is strongly correlated with dissimilarity between group ranking outcomes and initially indicated individual preferences.

Team dynamics and group decision-making are central to any organization. Researchers have reviewed and elucidated what years of research on small groups and teams can tell us about the processes that contribute to team effectiveness. Using that knowledge they have identified leverage points for making teams more effective such as team learning, cohesion, efficacy and group potency [8] . Furthermore, a study examining relationships among work teams with respect to team composition (ability and personality), team process (social cohesion), and team outcomes (team viability and team performance) found that interpersonally oriented personality characteristics, such as conscientiousness, agreeableness, extraversion, and emotional stability can also be important predictors of team effectiveness [1] .

In the 1970s, decision support systems (DSS) were described as "interactive computer-based systems, which help decision makers utilize data and models to solve unstructured problems" [6] . Group decision support systems (GDSS) eventually evolved to consider not only the role of the ultimate decision-maker, but also the communication between everyone in the decision-making group [3, 5, 14] . Turban et al. [14] have identified the major components that make up these systems. Some of them, such as database analysis and visualization, constitute independent fields of research within computer science. Others, such as support for Multi-Criteria Decision Making (MCDM), more appropriately fall under management science [13] . In recent years, researchers have explored Internet-based algorithms and workflows for decision-making at scale [9, 11] . Furthermore, inferring consensus among members of a group in a data-driven fashion has long been a goal in anthropology and social psychology. [12] demonstrates the use of Cultural Mixture Modeling (CMM) for inferring consensus among members of heterogeneous multidisciplinary professional teams in an example decisionmaking process. The findings of their study inform team decision-making based on shared beliefs and team composition.

In this work, we present a pilot DSS of our own design and construction, which we call Decidio. While we plan to extend Decidio to be a general-purpose DSS, our present version of the software is only able to support a very basic collaborative ranking process. Our pilot implementation was designed to support a real collaborative ranking carried out by University of Maryland undergraduate students. Our goals in this work are two-fold. First, we wanted conduct a user study of Decidio to identify its effectiveness and potential areas of improvement. Second, we wanted to use the opportunity to explore group dynamics in the particular decision process that we were supporting.

The remainder of the paper proceeds as follows. In Sect. 2, we describe the functionality of our current implementation of Decidio. In Sect. 3 we present the use case for the user study of Decidio and demonstrate how our tool was incorporated into an existing group decision-making process. In Sects. 4 and 5, we document and discuss the results of our experiment. We measure the reception of the tool by its users through a user survey, and we find the feedback to be positive overall, with some areas of improvement identified. We also use log events from the tool to create behavioral profiles of the users involved in decision-making. Finally, we explore the connection between these behavioral profiles and user personalities.

Decidio is a web-based collaborative ranking to support groups that need to collectively rank a set of options. The tool uses the Python-based Flask web server for its backend and the React Javascript Library for its frontend. This iteration of Decidio was designed to 1) make the ranking of options more efficient, and 2) ensure that the deliberation on team preference would fairly represent the opinions of all members on the team. The tool therefore supports the following features:

1. Linking users and teams: Decidio supports grouping users into teams.

Every user is assigned to a team. 2. Adding a list of options: An administrator can add a list of options which will be visible to all the users. 3. Individual ranking: Users can individually view the list of options added by the administrator and can submit their own preferred ranking order. Individual rankings are submitted for every user. 4. Group ranking: Groups can collectively choose a final set of rankings in Decidio's group ranking page. Updates made to the ranking by any member in a group are displayed in real-time to everyone in that group. 5. Displaying statistics about individual rankings: Decidio displays basic statistics for the group, generated from the individual ranking phase when the users are working on the group rankings. This feature helps users make decisions which encompass opinions of everyone in the group. 6. Analysis of user behavior through logs: Decidio logs highly granular interactions in the tool. Based on the logged activity, we can classify users into different categories like operators, leaders, followers, etc. We describe these terms in Sect. 3 in more detail.

The Quality Enhancement Systems and Teams (QUEST) Honors Program is a University of Maryland program for undergraduate students. The first course in the program involves a final project in which teams of students collaborate with a campus organization on process improvement. Each team is able to indicate their preference for which of the available projects they would like to work on. Historically, teams had indicated this preference by submitting a hand-written ranking of projects to the course instructor after a brief period of deliberation. The instructor would then allocate the projects to fulfill these preferences to the best of her ability. Our team identified this decision-making process as an opportunity to test the pilot of Decidio in a real-world scenario. The tool was used by 45 users divided into 9 teams. There were 9 project options which the teams had to collectively rank in order of preference.

There are three main phases in the workflow that we designed for QUEST. We describe each phase in detail.

QUEST students attended a special class session in which the available projects were presented by the instructor. Before arriving to this session, students were required to indicate their individual preferences for projects. The instructor for the course distributed a document describing each of the projects in detail during the days leading up to the session. Students used the Decidio web interface to order these projects from most-desirable to least-desirable. Figure 1 shows the interface and describes its interactivity. Users click on a project to select and then use the up and down arrows to re-order that project. Once they are satisfied with a ranking, they click a submission button to finalize (button not shown in the screenshot).

After the instructor presented each project available for selection, students had 30 min to meet with their groups and decide on a single ranking of projects to submit to the instructor. Each team sat at a separate circular table in the same classroom during the discussion. Students logged into Decidio via their personal laptops and navigated to the group project ranking dashboard. The important components of the dashboard are shown in Fig. 2 and Fig. 3 . This dashboard is distinguished from the individual ranking dashboard in that 1) all users in a team have equal control over modifying and submitting a unified project ranking and 2) users are presented with the results of individual rankings for all team members. Below, we describe the flow of group ranking:

1. Collaborative Ranking. Each user is able to update the team ranking with the same interaction used for the individual ranking process. When the ranking changes, this change is published to the dashboard of each team member, and the dashboard will automatically refresh to reflect the changes. 2 . Section of the dashboard for group project ranking that shows the individual rankings for each project (rows are aligned with project names not shown in the screenshot). Rankings for team members are anonymized, but the user's ranking is hightlighted in red as a reminder of their submitted preferences. The tool calls attention to cases where there are large differences in project preferences or outlier ranks.

Staging controls at the bottom of the group project ranking dashboard. In the configuration shown, the ranking has been staged (the leftmost button toggles between "Stage" and "Unstage" depending on the staging status). While staged, the ranking component shown on the dashboard (similar to the individual ranking component shown in Fig. 1 ) will be read-only. The user has approved the ranking, and no other member of the team has approved. Since the number of approvals falls short of the team size, Decidio displays a warning icon.

Staging. Users can choose to stage their team's ranking at any point during Collaborative Ranking by clicking a button on the web page. When a ranking is staged, further modifications to this ranking are frozen and the staged ranking is displayed for review by team members. Any user is able to unstage the ranking after it has been staged, which will return all users to Collaborative Ranking. 3. Approval and Submission. When a ranking has been staged, users can click to indicate whether they approve of the ranking for submission. The tool indicates to all users the number of team members who have approved the ranking, warning users when approval is not unanimous. However, the tool does not prevent submission based on the approval status. A user can submit at any time by clicking a button on the dashboard.

Projects were assigned to teams outside of Decidio. The rankings for each team were presented to the instructor on an administrator dashboard, which allowed for convenient export to Excel. The instructor was then able to apply a technique used in previous semesters for the fair allocation of projects.

Every server request in Decidio is recorded in a log file. A log normally contains the timestamp, user ID, group ID, and the action taken. Actions include information like:

-Has a user submitted their individual ranking? -What did a user change the group rankings to? -Which user approved the ranking? -Which user moved the group ranking to the staging phase?

From the user actions in the log file, we created an activity log that informs us of their interaction patterns with the tool. This profile was later used to classify users into groups based on their actions. Results from the analysis of activity logs are described in Sect. 4.2.

A survey questionnaire (See Appendix) was designed and sent to the students after the group decision making activity. The goal of this survey was to 1. Collect feedback on tool 2. Learn their process, methods, and strategies employed for decision-making when ranking projects as a group 3. Capture some elements of the unstructured discussion that happened in-class for group project ranking

We received 38 responses for the survey which were analyzed, and the findings from the survey are listed below.

More generally, the visualizations showing the rankings of group members appeared well accepted by the students. We learned that the visualizations helped them understand the group members' project preferences that led to constructive discussions. One user (U11) described that "The chart was very helpful, because it showed majority perspectives which allowed everyone's perspective to be objectively viewed. In this way, everyone's opinions were incorporated so that one person's opinion couldn't overshadow others."

From the survey, we learned that during the group decision-making process students took up roles such as note takers and facilitators. Every student team had at least one member acting as the facilitator. Collectively, this showed us the need and the importance of "roles" in group decision making.

In general, project presentations in class prior to group ranking helped students gain knowledge on projects. Group discussion that followed allowed them to communicate their understanding and validate assumptions with their group. This showed us that the presentations and discussions held in class influenced the group's final project rankings. This validated that a mental model which is an internal representation of knowledge changes with individual's understanding of the present state of a system and that the development of a "shared" understanding of the problem within groups is key in group decision making [4] . 4. Groups used various techniques for deciding on the final project rankings: As a standard discussion protocol was not established for QUEST, groups used a variety of techniques to arrive at the final project ranking. These include making a pros and cons list (8), voting on projects (6), brainstorming ideas, taking-turns or going around the table to discuss implications and levels of interest for each project (10). U39 described their group's process as "We first started by saying our top 3 and bottom 2. From there we discussed and found the projects the most people had in common."

An activity log file created for every user that captured their actions as a workflow diagram was used to understand how student groups used Decidio in their decision making process. We observed varied numbers of actions amongst users ranging from just 2 to 73 in their activity logs. From analyzing workflows in the activity logs of group members, we noticed that the workflows can be grouped into four categories based on who the users were and the actions they performed in the tool.

-Absentee Workflow: This workflow was exhibited by users that did not participate in the group ranking activity. It showed only actions related to the submission of individual project rankings. Fig. 4a demonstrates this workflow. -Regular Workflow: The regular workflow is the one where users submit individual rankings, approve and/or submit group project rankings. Their flow is mostly linear and does not involve updating the project rankings, staging the project rankings and unstaging the temporary final rankings. Figure 4b demonstrates this user's workflow. -Active Operator Workflow: Workflows containing repeated updates to project rankings indicate that the user is an "active operator" for the group. Each group typically had one or two members actively updating the project order in Decidio. By mapping active operators to their roles from the survey, we found that 11 out of 17 active operators considered themselves to be the facilitators of their groups. Figure 4c demonstrates an active operator workflow.

(a) Workflow for a user who was absent during the group rankings (b) Workflow for a normal user not acting as an operator (c) Workflow for a user acting as the core operator. The user updates the rankings repeatedly.

(d) Shows an advanced workflow in a team with the most conflicts. The user is doing multiple actions along with just being the operator in this case.

-Irregular Workflow: Irregular workflows contain cycles. i.e, the users have performed staging and unstaging operations. This workflow seems to indicate that the group didn't have the approval of all its group members (no consensus) and had to revise their project rankings by going back and forth between staging and unstaging before proceeding to submit. However, we note that one group exhibiting this workflow claimed via our survey that they had a consistent consensus throughout the ranking process. For this reason, we suspect that there are other explanations for the irregular workflow, such as user errors while navigating the software. Figure 4d demonstrates this type of workflow.

Bayram and Aydemir [2] examine the relations of decision styles and personality traits among groups of university students. We use a personality survey and the activity logs from our tool to carry out a similar thread of investigation. We asked students to take a test that measures their "Big Five" personality traits, also known as the Five-Factor Model [10] and collected their results. We then analyzed the 37 responses to study the relations between personality traits of group members and group decision making. From comparing the personality trait scores of "active operators" from the logs, we learned that the active operators tend to have higher median score on extraversion and lower median score on conscientiousness. (Figure 5b) We also explored how personality may have affected the decision outcome for each group. We came up with a metric to represent the dissimilarity between two rankings. This is calculated as the sum of squared differences between ranks for each project. For example, the dissimilarity between a ranking of 1-2-3 and 3-2-1 for projects A-B-C would be 8.

The dissimilarity metric cannot fairly be compared between teams without normalization. This is because the distribution of initial rankings varies between teams (some teams naturally had more or less disagreement before the Group Project Ranking phase began). Therefore, we compute a normalized dissimilarity for each user that is the difference between that users raw dissimilarity and the average dissimilarity for that team. We looked for correlations between normalized dissimilarity and percentile scores for the Big Five Factor personality survey. We calculated a Pearson correlation coefficient between normalized dissimilarity and each of the five personality factors. The results of this analysis are shown in Table 1 . The only strong correlation that we see is the strong positive correlation between Agreeableness and normalized dissimilarity. We interpret this to mean that agreeable students were less likely to see their individual preferences reflected in the rankings submitted for their group. 

The personality comparison between all students and active operators (see Sect. 4.3) is partially aligned with prior research correlating Big Five traits with leadership [7] . Judge et. al. see positive correlations with Extraversion, Openness to experience, and Conscientiousness; and a negative correlation with Natural reactions (referred to as Neuroticism in their work). The bias towards Extraversion among active operators is consistent with these results, however the bias against Conscientiousness in that group is not. Of course, the association we make between active operation of the tool and general leadership is based in assumption and may not hold in all cases (as described in Sect. 5.2). We also find a strong correlation between agreeableness and our dissimilarity metric. This leads us to the satisfying (and perhaps expected) observation that more agreeable participants were less likely to "get their way" when working towards a team consensus on preference rankings.

We're excited by the insights into decision dynamics that the Decidio pilot provided in this user study, and we plan to enhance Decidio's capability of recording software interactions that are relevant to these dynamics.

We recognize that, because we are documenting results from a single experiment with our software, we should be careful to draw strong conclusions from our work.

There are also several limitations to identify within this one experiment. First, the log events that we collect are not sufficient to capture a rich behavioral profile of all participants in the decision study. We did not record the discussion between each team of participants, and therefore we can not make definitive conclusions about the team dynamic. It's an extrapolation, albeit a reasonable one, to connect the "Operator" workflow described in Sect. 4.2 to the subjective notion of team leadership, for example. We recognize that the "Operator" may have been acting as a sort of team note-taker or secretary for the session rather than the leader of the discussion. Second, we did not test with a control group. This nuances the results from the user survey, as we're not able to draw a comparison to a team who collaborates on a ranking with a default method (e.g., pen and paper). Finally, it is difficult to generalize results about user behavior in the tool when each team consists of a different mix of personalities and individual ranking preferences. For example, a team with insignificant differences between individual preferences for projects may not have much to disagree about, even if their personalities would suggest a higher likelihood of conflict.

The initial version of Decidio that we present in this paper is only narrowly applicable to group decisions that involve ranking a discrete set of options. However, the ultimate vision for Decidio is to build a DSS that can be easily tailored by users to fit specific decision workflows. This flexibility is valuable to decisionmakers because it will allow them to incorporate automation in existing workflows without substantial modification. It is also valuable to researchers who want to experiment with a variety of workflows.

In this paper, we present Decidio, a software tool that supports collaborative decision making. We evaluated the tool by using it in real-time with 9 student teams for ranking projects for a class. The results indicate that the overall reception for such a tool is positive. In summary, we learned from our tool evaluation survey that students assume roles and use various techniques when making group decisions. The activity logs showed us the various workflows used by student groups for decision-making. It was interesting to see the relations between personality traits and group decision making. We believe the results from this study will help us in the design and development of a robust decision support system for collaborative decision making. 13 . What were the tools or techniques used by your team when making decisions or in resolving conflicts? 14. Which of the following would you say was your team's decision-making strategy (A list of strategies such as Consensus, Compromise, Competition, Accommodating, followed by a space for open comments).

Did you or your team face any technical difficulties when using the tool? (Yes/no question

The top two things I like about this interface are

The top two things I dislike about this interface are

In what ways would you modify this tool to increase its usability or functionality

How was the chart showing individual rankings of your team members used to stimulate discussion?

How were the notes from individual rankings used when ranking projects as a team?

What was your role in the team when discussing project rankings?

How did the in-class presentations influence your team's project rankings? 10. What aspects of the projects were important to your team when ranking them?

How did your team handle disagreements or objections when ranking projects?

Would you say everyone in the team had the chance to voice their opinions? If no, what according to you, were the challenges that kept your team from hearing out everyone's opinions?

Relating member ability and personality to work-team processes and team effectiveness

Decision-making styles and personality traits

Communications design for Co-Op: a group decision support system

Shared mental models in expert team decision making

A foundation for the study of group decision support systems

A framework for management information systems

Personality and leadership: a qualitative and quantitative review

Enhancing the effectiveness of work groups and teams

A web-based group decision support system for multicriteria ranking problems

An introduction to the five-factor model and its applications

A group decision making support system for the web: how to work in environments with a high number of participants and alternatives

Identifying consensus in heterogeneous multidisciplinary professional teams

Multi-Criteria Decision Making Methods: A Comparative Study

Decision Support and Business Intelligence Systems