Affective States in Novice Programmers: Automatically Detecting and Analyzing the Impact on Learning

Abstract

Programming is a complex cognitive activity that often poses a significant challenge for beginners. Beyond the cognitive dimension, understanding the role of affective states, such as emotions, attitudes, and motivation, in the learning process can provide valuable insights for educators to improve teaching methodologies and curriculum design. Despite this importance, few studies relate the cognitive aspects of beginning programmers to their affective states in the context of problem-solving. In this article, we explore the affective states reported by novice programmers in introductory programming courses, relating them to programming concepts and success rates during problem-solving activities. Furthermore, we apply machine learning algorithms to detect affective states in novice programmers, considering both student and task characteristics. Our initial results allow us to understand the affective states most prevalent in problems associated with each programming concept, as well as to reveal a significant association between affective states and the performance of novice programmers. Positive affective states, including pleasure, motivation, and challenge, were positively correlated with better performance outcomes. On the other hand, negative affective states such as frustration, anxiety, and boredom impaired performance. Additionally, our algorithms demonstrated good performance in detecting affective states.

1 Introduction

Programming problem-solving is widely recognized as a challenging endeavor for beginners, often leading to high rates of academic failure among students [28]. Sheard et al. [5] further underscored the complexities of introductory coding, describing it as a subject difficult to both learn and teach. Over the past four decades, research in computing education has extensively documented the hurdles students face when introduced to programming concepts [22, 23, 25]. These studies predominantly explore conceptual and cognitive barriers encountered during the program construction process, alongside challenges posed by specific programming constructs and control structures [21]. These learning challenges are not solely cognitive; they are often intertwined with affective states, particularly frustration. Learning to code is frequently accompanied by feelings of frustration and lack of motivation, as coding is perceived as complex and daunting [3]. As students engage in early programming experiences to grasp fundamental concepts, the influence of affective states, notably negative ones like frustration, boredom, and anxiety, on novice programmers becomes apparent [18].

Negative emotions, such as confusion and boredom, have been associated with poor performance in introductory programming courses [25]. These emotions have been shown to impede students’ working memory, consequently affecting their overall performance [29]. Conversely, positive emotions, like concentration and motivation, are pivotal in enhancing learning outcomes [30]. This interplay between affective states and learning outcomes highlights the need for a comprehensive understanding of their relationship in the context of programming education [9]. Recent research has focused on developing learning environments that recognize and respond to students’ affect [20, 24]. While these systems aim to improve students’ affective states during tutoring sessions, they may not foster long-term learning or develop students’ meta-affective capacity [7]. Therefore, it is imperative to integrate affective awareness and self-regulation into programming education to support students’ holistic development [6].

Despite this, few studies have explored the relationship between cognitive and affective aspects of beginning programmers during problem-solving activities [26]. Investigating students’ emotional behavior when learning challenging programming concepts provides valuable insights into their affective responses and learning outcomes. By understanding how affective states and programming concepts influence problem-solving activities, educators can better support novice programmers’ learning journeys.

In this article, we address the following research questions:

RQ 01: What is the relationship between programming concepts, success rate, and affective states reported by novice programmers during problem-solving?

RQ 02: Has an effective automatic detector been developed to identify instances of affective states among novice programmers?

Through these questions, we intend to identify and detect the most prevalent affective states considering programming concepts and some implications for problem-solving. To achieve this, we conducted a study in three main stages: (i) data collection through an experiment with novice programmers; (ii) analysis of the collected data; and (iii) development of the detection model. In summary, the main contributions of this work are as follows: (i) a deeper understanding of the relationship between affective states, programming concepts, and the performance of programming students; and (ii) the development of an effective automatic detection model to identify affective states, exploring three relevant machine learning algorithms: Decision Tree, K-Nearest Neighbors, and Multilayer Perceptron Neural Network, with the Decision Tree model showing the best performance.

2 Background and Relatedwork

The investigation of emotions within the learning process, particularly in tutoring systems, represents a continually evolving area of research. Extensive efforts have been made to understand the intricate relationship between cognitive aspects and emotional states [37]. In the domain of computer programming, novice programmers commonly encounter both cognitive and emotional challenges [9, 21]. On the cognitive front, beginners often struggle with problem-solving tasks, including the formulation, execution, and evaluation of solutions [13]. Novice programmers frequently have difficulties grasping basic programming structures such as specific constructs, control structures, variables, data structures, looping mechanisms, and logical flow with conditionals and Boolean logic [21]. Additionally, challenges arise when analyzing and designing mathematical and logical expressions, assigning variable names, and selecting appropriate data types and structures [33]. Misinterpretation of conditional structures, particularly the OR operator, further compounds these challenges [34].

However, the learning hurdles in programming extend beyond the cognitive realm and permeate the affective dimension. Emotions have garnered significant attention in academic contexts, with boredom, confusion, pleasure, and frustration being prominent focal points [35]. Consequently, a novice programmer’s problem-solving efforts can be significantly influenced by a spectrum of affective and mental states. For instance, unexpected feedback can induce confusion among students [36]. Moreover, shifts in affective states often manifest in cognitive performance, with studies linking affective states to novice programmers’ final performance [10, 17]. High-arousal states like pleasure or fear have been associated with enhanced programming performance [19]. Furthermore, a study conducted during the COVID-19 pandemic shed light on the affective state of motivation among programmers [3].

In a study by Bosch and D’Mello [27], the prevailing emotions among novice programming students were investigated, alongside their relationship with behavior and academic performance. Participants assessed various emotions during the programming process, ranging from basic emotions to learning-related ones. The results indicated that students commonly experienced a state of flow/engagement, followed by confusion, frustration, and boredom. Similarly, Chetty and Van Der Westhuizen [26] explored students’ emotional experiences when grappling with challenging concepts. Emotions such as anxiety, fear, confusion, and despair were identified as typical responses to difficult concepts. Recognizing these emotions is crucial for educators in providing appropriate support during challenging learning moments.

Against this backdrop, numerous researchers have endeavored to develop solutions for identifying novice programmers’ emotional states during programming activities. Vea et al. [2] developed a model to detect negative emotional states, particularly boredom, confusion, and frustration, among C++ learners. The detection relied on participants’ typing patterns and mouse movements, although detecting reduced frustration levels proved challenging. Another study employed machine learning algorithms to predict emotional states among university students engaged in programming tasks of varying difficulty levels [14]. Features derived from keyboard and mouse logs were utilized for emotional state classification, although some researchers caution against relying solely on log measures to accurately capture emotional processes [12]. Self-reports have also been explored as an alternative approach, with Harley et al. [20] reporting a 75% agreement rate between self-reports and video-based emotion detection. Therefore, this article considers the emotional states reported by students within the learning environment during its utilization.

3 Methodology

The research described in this article is part of a project conducted in four distinct stages: the development of the virtual learning environment, approval of the project by the ethics board, data collection, and data analysis. Section 3.1 will detail the developed learning environment. In this environment, programming students can record their affective states as they solve the proposed problems. These states include negative affective states (frustration, boredom, and anxiety), positive affective states (pleasure, motivation, confusion, and challenge), and neutral states. Notably, half of these emotional states were identified as significant during the learning process [31], influencing many other learning environments to adopt portions of this set in their approaches [32]. Additionally, it is important to highlight that, given the involvement of collecting and analyzing data from student interactions, the project was submitted to the local ethics committee and received approval before proceeding to the subsequent stages of data collection and analysis. The following subsections will detail the materials and methods used in this research, including information about the study participants, the data collection instruments, and the data processing techniques.

3.1 Materials

The material resources used in this study included a technological tool (Virtual Learning Environment¹) that captures problem-solving attempts and the affective states reported by students.

• Hints: The hints in ADA are organized into four levels, each providing increasingly specific guidance. The initial hints are abstract, while the final level almost provides the complete answer, requiring the student only to input the solution. Students can request hints at any time, receiving progressively detailed assistance as needed.

• Problem Level Assessment: After submitting a partial solution, the system asks the student to evaluate the difficulty level of the problem, with options ranging from very easy to very difficult. Additionally, all problems are associated with the programming concepts presented in Table 1.

• Affective State: While solving programming problems, students can self-report their affective states using the top bar of ADA. They can select from negative affective states (frustration, boredom, and anxiety), positive affective states (pleasure, motivation, confusion, and challenge), and neutral states.

Table 1. Programming Concepts

3.2 Data Collect

In this study, students actively participated as part of their regular programming curriculum at two technical high schools in northeastern Brazil. The research took place during the first semester of the 2023 academic year and involved four programming classes: two CS1 classes and two CS2 classes. The CS2 course builds upon the foundation laid by CS1, with both courses utilizing Java as the primary programming language.

The selection of CS2 students was deliberate, as they were in the early stages of their computer science program, making them novices in the field. However, they had already acquired a fundamental understanding of programming and were reviewing programming concepts. This timing allowed us to conduct our study as these students were about to learn object-oriented programming within the CS2 curriculum. Throughout the study, participants were provided with snacks during all problem-solving sessions. These sessions were organized into four blocks, totaling 16 problem-solving sessions, with four sessions dedicated to each class. The study successfully concluded with the active participation of 131 students who attended all problem-solving sessions.

The breakdown of student participation across the session blocks is as follows: 220 students participated in the first block (averaging 55 students per class), 201 in the second block (averaging 50.25 students per class), 182 in the third block (averaging 45.5 students per class), and a final group of 131 students in the last block (averaging 42.5 students per class). Among the participants, 74 were female, and 57 were male, with an average age of 17 (SD = 0.76). Regarding their living arrangements, all students resided with their families. Notably, 68% of the students did not live in the same town as the school, and 27% lived in rural areas. While all participants had access to the internet, only 27% owned a personal computer, 14% had a tablet, and the majority, 54%, possessed a smartphone. Each student engaged with an average of 41.3 problems (SD = 25.0) during the study, and they tackled each problem an average of 4.3 times (SD = 2.7) to arrive at solutions.

Each problem-solving session was conducted face-to-face at the school using the tool presented in Sect. 3.1. The first session lasted an average of 140 min, and the remaining sessions lasted 125 min each.

During the first session of each class, the initial 15 min were dedicated to explaining the study’s objectives to the students, distributing the consent forms for participation, and explaining the functionalities of the ADA environment, particularly the meaning of each affective state. The participants then engaged in problem-solving for 100 min. At the end of each session, interviews were conducted with the students for 25 min. Regardless of whether the students finished the programming portion, the last 25 min were reserved for the researcher to ask questions about the behaviors observed during the programming task. The other sessions were divided into 100 min for problem-solving and 25 min for interviews. Two researchers conducted all 143 studies. Due to space constraints, this article focuses only on the programming part of our study, excluding the interview portion. In each problem-solving session, students could make several attempts to solve the problems. For each attempt, students had to make two submissions: an alternative solution and the code corresponding to the problem being solved.

Dataset. By running the experiment, we acquired two sets of data: the first comprising 5,902 problem-solving attempts and the second comprising 6,789 self-declarations of affective states. After merging these datasets based on trial numbers, we created a consolidated dataset consisting of 13,534 instances of student behavior. The variables present in this dataset include: problem identifier, difficulty level, problem concept, answer to the problem (alternative), affective state identifier, student identifier, number of errors, grade per problem, overall grade, number of attempts to solve the problem, and number of declarations of each affective state.

3.3 Classification Models

We analyzed three stages: pre-processing, classification, and evaluation of the data. In the data pre-processing stage, we performed two crucial tasks: data cleaning and data transformation. During data cleaning, we removed missing data to avoid inconsistencies and errors during analysis and modeling. For data transformation, we applied normalization and standardization. In normalization, attribute values were adjusted to a common scale, generally between 0 and 1, allowing each attribute to contribute equally to the final result. In standardization, data were adjusted to have a mean of zero and a standard deviation of one, which is useful for standardizing deviations from the mean.

In the classification step, we used a technique called 15-fold cross-validation. This method involves dividing the data into 15 parts and, in each iteration, training the model on 14 parts and testing it on the remaining part. By repeating this process 15 times, we increased the algorithm’s generalization capacity, resulting in more accurate predictions [15]. We applied three algorithms for classification: Decision Tree (using the CART algorithm), K-Nearest Neighbors (KNN) with 3 neighbors, and Neural Networks using Multi-Layer Perceptron (MLP). This selection was based on previous studies, and the implementation was carried out with the scikit-learn² library.

In the evaluation stage, we analyzed the performance of each algorithm on the training and testing subsets using six metrics: Area Under the ROC Curve (AUC), Accuracy, Precision, Recall, F1 Score, and Kappa. AUC evaluates the model’s ability to perform accurate classifications. Precision, Recall, and F1 Score provide a comprehensive view of performance, while Kappa checks whether the model determines the correct sequences for reasons that are not based on chance. A significance level of $p-value < 0.05$ was used.

4 Results and Discussion

RQ 01: What is the relationship between programming concepts, success rate, and affective states reported by novice programmers during problem-solving?

To address this research question, we investigated the students’ behavior during problem-solving using two main observation criteria: 1) We calculated the average grades assigned to each affective state for each concept. 2) We determined the average number of attempts required to solve the problems for each affective state associated with each concept. Consequently, we conducted a detailed analysis of each concept and affective state. It is important to note that, to evaluate the results, we applied specific statistical tests, such as the Wilcoxon test for comparing two samples and the Friedman test for comparing more than two samples, to determine the significance level of the results.

Programming Concepts. We calculated the average grades of students based on affective state and programming concept, while Table 3 provides the average number of attempts to solve the same problems, categorized by affective state and programming concept. Next, we will analyze each concept or group of concepts individually.

In the context of concept $C_{01}$, students reported significantly lower average marks when they experienced positive affective states of motivation (0.67, $p=0.093$) and challenge (0.69, $p=0.159$), as well as a lower average number of attempts while feeling motivated (1.37, $p=0.097$). In contrast, students who reported feeling pleasure achieved higher average marks (0.81, $p=0.087$), although there was no significant increase in the number of attempts (1.42, $p=0.009$). The negative affective state of anxiety also correlated with lower scores (0.69, $p=0.097$). In concept $C_{02}$, there was no significant difference in mean scores related to different affective states (F=9.89, $p=0.071$). However, there were significantly more attempts when students felt challenged (4.34, $p=0.088$).

For concepts $C_{03}$, $C_{04}$, and $C_{05}$, related to the declaration and manipulation of variables, students achieved the highest mean scores in concept $C_{05}$ when they experienced positive affective states (0.87, $p=0.091$), while they underperformed in negative affective states (0.72, $p=0.077$). There was no significant difference in the mean scores for concepts $C_{03}$ and $C_{04}$ (F=7.09, $p=0.082$). In concept $C_{06}$, students who reported feeling confused and anxious achieved the highest average score of 0.85, although there was no significant difference between these averages ($p=0.001$). For concepts $C_{07}$ and $C_{08}$, which deal with increment and decrement of variables, students who felt pleasure achieved a significantly higher average score of 0.98 for concept $C_{08}$ ($p=0.061$). Additionally, students obtained significantly lower grades when they felt bored in the same concept.

Considering concepts $C_{09}$, $C_{10}$, and $C_{11}$, which involve operators (Boolean, relational, and logical), students performed significantly better ($p=0.078$) when they felt positive states. However, students made significantly more attempts when solving problems associated with $C_{10}$ ($p=0.072$). Concepts $C_{12}$ and $C_{13}$, which deal with conditional structures, were associated with higher average scores when students were motivated and lower scores when they felt frustrated. Consequently, students achieved higher average grades in positive affective states ($p=0.072$).

Finally, for concepts $C_{14}$ and $C_{15}$, related to repetition structures and vectors, students who felt pleasure, motivation, and frustration when solving problems related to concept $C_{15}$, and motivation in concept $C_{14}$, achieved higher average grades. Conversely, students who experienced frustration and boredom achieved higher averages in $C_{15}$ and anxiety in $C_{14}$.

Affective State. Looking first at the positive affective states, the average values for concept $C_{05}$ are significantly higher compared to the other concepts ($p=0.070$). This indicates that students experiencing pleasure in this concept had a higher average grade. In Table 3, the average number of attempts for concept $C_{05}$ is also lower compared to the other concepts, suggesting that students are more motivated and engaged with this concept. On the other hand, there are no significant differences in the average motivation values between the concepts in the two tables. The standard deviations are also similar ($p=0.001$). Concepts $C_{02}$ and $C_{08}$ show significantly higher mean scores for challenge in the table compared to the other concepts ($p=0.081$). This may suggest that these concepts are perceived as more challenging. However, in Table 3, the average number of attempts for these concepts is not the highest, indicating that students are not necessarily avoiding these challenging concepts (Table 2).

Table 2. Correlações de Spearman nas taxas de sucesso dos alunos para cada conceito

Table 3. Average number of attempts by students for each affective state

Table 4. Number of students declaring each affective state per programming concept

Regarding negative states such as anxiety, concepts $C_{08}$, $C_{10}$, and $C_{11}$ seem to have significantly higher average scores compared to the other concepts ($p=0.081$). This may suggest that these concepts are precedents for better scores among anxious programmers. However, in Table 3, the average number of attempts for these concepts is not the highest.

To conclude the analysis of this research question, Table 4 shows that the number of students declaring each affective state varies widely between affective states and concepts. For example, the number of students declaring affective states such as "Boredom" or "Anxiety" is lower compared to other affective states.

RQ 02: Has the automatic detector been effectively developed to identify instances of affective states among novice programmers?

Table 5. Results obtained when applying the KNN algorithm to affective states (Test)

Table 6. Results obtained when applying the MLP algorithm to affective states (Test)

Table 7. Results obtained when applying the decision tree algorithm to affective states (Test)

Table 8. Wilcoxon test p-value for classifiers

The results of the KNN algorithm, shown in Table 5, indicate solid overall performance in classifying affective states. The accuracy (ACU) varies from 0.85 to 0.91, with Confusion presenting the best accuracy (0.91 ± 0.05). Precision (PRE) ranges from 0.78 to 0.87, and recall (REC) ranges from 0.69 to 0.74, indicating that KNN is relatively effective in correctly identifying affective states. The F-Measure (FMe) and the Kappa coefficient (KAP) also present moderate to high values, with Confusion once again standing out (FMe = 0.79 ± 0.04, KAP = 0.65 ± 0.03). The area under the ROC curve (ROC) is consistently high, ranging from 0.81 to 0.89, reinforcing the effectiveness of KNN in distinguishing between different affective states (Table 7).

MLP, as shown in Table 6, performs slightly worse compared to KNN. The accuracy (ACU) varies from 0.79 to 0.85, with Confusion achieving the best performance (0.85 ± 0.04). Precision (PRE) ranges from 0.72 to 0.81, while recall (REC) ranges from 0.61 to 0.67, indicating that MLP may have difficulty capturing all positive examples. The F-Measure (FMe) and the Kappa coefficient (KAP) are at moderate levels, with lower values compared to KNN. The area under the ROC curve (ROC) varies from 0.75 to 0.83, highlighting that, despite overall lower performance, MLP is still effective in distinguishing affective states.

The decision tree algorithm, shown in Table 8, demonstrated the best overall results among the three algorithms. The accuracy (ACU) varies from 0.87 to 0.92, with Confusion again achieving the best accuracy (0.92 ± 0.05). Precision (PRE) ranges from 0.80 to 0.88, and recall (REC) ranges from 0.71 to 0.75, showing a solid balance between correct identification and capturing positive examples. The F-Measure (FMe) and the Kappa coefficient (KAP) also present the highest values, indicating the consistency of the decision tree model. The area under the ROC curve (ROC) ranges from 0.83 to 0.90, reinforcing the algorithm’s ability to effectively distinguish between different affective states.

The Wilcoxon test results reinforce these conclusions, indicating statistically significant differences between the algorithms in several metrics. For each affective state, p-values are less than 0.05 in most cases, showing that the observed differences are not due to chance. In particular:

For the pleasure state, all metrics showed significant p-values (<0.001), except FMe (0.001), indicating a substantial difference between the algorithms. For the motivation state, all metrics presented p-values <0.001, suggesting a clear difference between the algorithms’ performances. For the challenge state, most metrics showed p-values <0.001, with a few exceptions (PRE: 0.005, KAP: 0.003). In the case of the confusion state, most p-values are <0.001, with some metrics slightly above this value (ACU: 0.012, PRE: 0.007, FMe: 0.001). For the states of frustration, boredom, and anxiety, all metrics presented p-values <0.001, indicating significant differences between the algorithms.

These results indicate that the decision tree, followed by KNN and, finally, MLP, presents the best performance in detecting affective states. The statistical robustness provided by the Wilcoxon test further validates the decision tree as the most effective algorithm for this context.

5 Conclusion

This paper explores the connection between cognitive aspects and the affective dimension of novice programmers’ learning experiences, investigating the possible relationships between programming concepts, affective states, and success rates in problem-solving activities. Our research questions focused on the prevalence of specific affective states associated with various programming concepts, the interaction between these concepts and affective states, the success rates, and the influence of affective states on novice programmers’ performance.

The analysis of Research Question 2 (RQ02) revealed that the relationship between programming concepts, success rates, and affective states reported by novice programmers during problem-solving is complex and varies among different concepts and affective states. When examining individual programming concepts, we found that the relationship between affective states and success rates was inconsistent across all concepts. Some concepts exhibited significant differences in success rates based on affective states, while others did not. For example, in concept $C_{05}$ (related to variables), students who experienced pleasure achieved a significantly higher average score compared to those in negative affective states. Conversely, for concept $C_{02}$ (data types), there were no significant differences in mean scores based on affective states. These results suggest that the impact of affective states on programming performance may depend on the specific programming concept being addressed.

Positive affective states, such as pleasure, motivation, and challenge, were generally correlated with higher average scores, while negative affective states, such as frustration, boredom, and anxiety, were associated with lower average scores. However, these trends were inconsistent across all concepts, indicating that the relationship between affective states and programming performance is context-dependent. The analysis of the number of attempts made by students also varied based on affective states and concepts. In some cases, students in positive affective states made more attempts to solve problems, reflecting higher engagement and persistence. Consequently, the relationship between programming concepts, affective states, and success rates among novice programmers is multifaceted and influenced by several factors. Although positive affective states generally seem to impact programming performance positively, the specific effects vary depending on the context.