1 Introduction

Known as a field of study that has gained significant importance in recent years, machine learning (ML) has emerged as a way of deducing a hypothesis based on past or lived experience [1]. In this context, machines are programmed to learn from previously stored data derived from previous experiences in order to generate knowledge. In ML, the learning algorithms can be broadly divided into supervised, semi-supervised and unsupervised algorithms. In this paper, semi-supervised learning (SSL) algorithms are assessed and these algorithms use labeled instances to build their initial hypothesis and combine the information obtained by these examples to label the unlabeled instances. In other words, it is possible to use partially supervised information to guide the learning process and increase the evidence of the target labels [2].

Nevertheless, we have observed that the automatic assignment of labels in an SSL algorithm is a difficult task, mainly related to the correct selection of the unlabeled instances to be labeled. An inefficient selection of the unlabeled instances and the consequent labeling of these instances can degrade enormously the performance of the SSL algorithms. A solution for this problem is proposed in [3] using a semi-supervised learning model called Flexible Confidence Classifier (FlexCon-C), which automatically adjusts the confidence parameter to select and label the unlabeled instance. This confidence is based solely on the accuracy of a classifier built in the previous iterations.

In the ML context, it is well-known that the accuracy of a single classifier can not be efficient for complex tasks. For this reason, combining different classifiers to improve the classifier performance has been emerged as an efficient alternative, as observed in several studies in the field [4,5,6,7,8,9]. In these studies, it has been noticed that combining information from various sources has resulted in systems with more reliability and performance than individual classifiers. Systems combining classifiers are known as classifier ensembles and they are composed of a set of base classifiers organized in a parallel way. These classifiers receive the input data and they produce several outputs that are used, in turn, as inputs by a combination method, which combines these outputs and generates the final ensemble output [7].

In this paper, we seek to improve the FlexCon-C [3] performance by introducing the classifier ensembles strategy to select and to label unlabeled instances and to include them in the labeled data set. Therefore, this paper proposes the Flexcon-CE (Flexible COnfidence with Classifier Ensembles) model, which uses an ensemble-based confidence adaptive confidence equation. In fact, two different versions of the Flexcon-CE method are proposed in this paper. The first one, called FlexCon-CE(Hom), uses a homogeneous ensemble (base classifiers of the same type) while the second one, named FlexCon-CE(Het), uses an heterogeneous ensemble. This study assesses the efficiency of the proposed methods when building representative classifier ensembles by analyzing the classification effectiveness of these methods in 20 well-known datasets, with three different classification algorithms and five different configurations of initially unlabeled data.

This article is organized as follows: Sect. 2 presents the theoretical aspects of semi-supervised learning algorithms as well as the FlexCon-C method and the classifier ensembles. Section 3 highlights some related work to this research while Sect. 4 presents and details the SSL method proposed in this article. Section 5 describes some methodological aspects of the empirical analysis while Sect. 6 presents the obtained results. Finally, Sect. 7 highlights some final considerations and future possibilities of researches.

2 Theoretical Aspects

2.1 Semi-supervised Learning

According to the tasks to which they are applied, the machine learning algorithms can be broadly divided into three types: 1) supervised learning - predictive tasks; 2) unsupervised learning - descriptive tasks; 3) semi-supervised learning - combining predictive and descriptive tasks. In other words, supervised learning requires the training dataset to have input and output attributes (labels or classes) in order to find a model, based on the data used for training, that can be used to predict a label or class of a given instance [1]. In unsupervised learning, dataset attributes are not required to have an output or label to shape the instances’ behavior pattern. This is possible due to the existence of tasks that look for behavior patterns of the instances based on the input attributes. Unsupervised or descriptive tasks are more related to exploring the dataset, with the aim of obtaining cluster patterns or association rules present in the data [2].

The existence of datasets that have a part of the instances with labels and another part without assigned labels became a motivation for the creation of a new classification model with the objective of improving the performance of learning tasks benefiting from the strategy of using labeled and unlabeled data, known as semi-supervised learning [10]. Following this concept, different algorithms were developed that propose the classification of unlabeled instances based on the ones with a label, including self-training, which will be used in this work.

The self-training method is characterized by using its own prediction results, obtained by a single classifier, and applying a new training to the model until all instances that did not have labels are properly labeled [11]. The Algorithm 1 depicts the flow of instructions that are executed in the self-training process. First, the database is divided into two sets: \(D_{l}\) - labeled instances and \(D_{u}\) - unlabeled instances (line 1). Then, a classifier f is generated through \(D_{l}\) (line 3). After assigning the new labels to the unlabeled data, these instances are inserted into the subset S which will be removed from \(D_{u}\) and added into \(D_{l}\) (lines 4–6). Then the process will restart until there is no more unlabeled data. Note that, based on the results generated by the classifier, the model is retrained in order to learn according to its assertiveness rate.

Algorithm 1
figure a

Self-Training

Full size image

In [12], the authors addressed an exploratory analysis of the self-training algorithm, regarding its adaptation by inserting a new parameter, called confidence, which is used to select and to label unlabeled instances. The main aim is to select instances that have a prediction higher than a confidence threshold. However, the confidence threshold is initially defined and this value is used throughout the whole labeling process. In [3], the authors proposed the FlexCon-C method, which is an extension of the work proposed in [12], which applies an adaptive (or flexible) confidence threshold, which is calculated in every iteration of the semi-supervised process. The FlexCon-C algorithm is presented by algorithm 2.

Algorithm 2
figure b

Self-Training with Confidence Adjustment (FlexCon-C)

Full size image

In Algorithm 2, the lines marked in blue highlight the main differences between this algorithm and the original self-training algorithm. Initially, \(D_{l}\) represents the initially labeled set, while \(D_{u}\) stands for the set of unlabeled instances. At each iteration, a classifier f is trained using the set of labeled instances \(D_{l}\) as training set and classifies the \(D_{u}\) instances (lines 3 to 4). Then, a new confidence threshold value is calculated (line 5). Then, select all unlabeled instances whose prediction is higher than or equal to the confidence threshold. These instances are then labeled (line 6). These instances are removed from the \(D_{u}\) set and included to the \(D_{l}\) set. This process is then repeated until the \(D_{u}\) set is empty.

In [3], three SSL algorithms are proposed (FlexCon-G, FlexCon, and FlexCon-C). All three algorithms use adaptive confidence threshold and they are all based on the self- training method. In the cited article, the FlexCon-C algorithm adjusts the confidence parameter at each iteration based solely on the precision of a classifier built the previous iteration. In this method, the adjustment will be done by increasing or decreasing the confidence value based on a change rate cr. Equation 1 presents the confidence update equation for FlexCon-C.

$$\begin{aligned} \textit{conf}(t_{i+1})={\left\{ \begin{array}{ll}{} \textit{conf}(t_{i}) \textit{ - cr}, \textit{if } \textit{acc} \ge \textit{mp} \textit{ + e}\\ \textit{conf}(t_{i}), \textit{if } \textit{mp} - \textit{e} < \textit{acc} < \textit{mp} \textit{ + e}\\ \textit{conf}(t_{i}) \textit{ + cr}, \textit{if } \textit{acc} \le \textit{mp} - \textit{e} \end{array}\right. } \end{aligned}$$
(1)

where \(conf(t_{i+1})\) is the confidence value of the current iteration, mp is the minimum acceptable precision; cr is the change rate it is a user-given hyperparameter, acc is the accuracy of the classifier. Finally, e is an acceptable precision variation that is allowed in order to define an stabilization in precision.

In this equation, if the accuracy is higher than a minimum acceptable precision (considering the acceptable variation e), mp, the confidence threshold is decreased by a change rate cr. If the accuracy is lower than a minimum acceptable precision, the confidence threshold is increased by a change rate cr. Otherwise, the confidence threshold does not change.

This method was further divide into two sub-methods with respect to the label definition of the unlabeled instances, FlexCon-C1 and FlexCon-C2. FlexCon-C1 uses classifier ensembles to define the label of each pattern, while FlexCon-C2 uses the label predicted by the classifier generated in the first iteration, which is stored by the algorithm. In FlexCon-C1, the classifier ensemble is composed of classifiers built in all iteration performed until a current iteration. In addition, these individual classifiers are combined by sum and majority voting, leading to two versions of this method, FlexCon-C1(v) and FlexCon-C1(s).

2.2 Classifier Ensembles

As previously mentioned, the increased complexity and wide applicability of classification systems has led to exploring many approaches and methodologies. Nevertheless, there is a perception that no classifier is considered completely satisfactory for a particular task; therefore, the idea of combining different methods to improve performance has emerged as a very promising possibility [13]. This combination is called classifier ensembles, also known as multi-classifier systems.

In classification tasks, an ensemble includes several sub-models called base classifiers, which are usually obtained by training a basic learning algorithm (decision tree, neural network, k nearest neighbors, among others). The ensembles can be built based on the same learning algorithm, producing homogeneous ensembles, or using different algorithms and generating heterogeneous ensembles [14].

The proposition of classifier ensembles is to create and combine several inductive models for the same domain, obtaining better prediction quality [9]. After generating the base classifier set, the next step is the choice for the methods to combine their outputs. There is a vast number of methods for combining classifiers in the literature. In this work, we will be used voting. The choice for this method was made due to the use of information from all classifiers. The voting method, often used to combine classifiers, performs the combination by voting on the results of each classifier when a new instance is presented.

3 Related Work

Over the years, several works have been developed using a confidence parameter in semi-supervised ML algorithms [3, 12, 15,16,17,18]. The goal is to investigate the behavior of these algorithms, including this parameter as a threshold for selecting new instances for the labeled dataset. On the other hand, researchers have explored the use of classifier ensembles to enhance the performance of various machine learning algorithms [4,5,6, 8]. The present work aims to merge these two approaches, both using the confidence parameter as well as classifier ensembles, as a way to improve the performance of the semi-supervised learning algorithm FlexCon-C.

In [12, 15] new versions of the self-training algorithm were proposed that included a fixed confidence parameter with the objective of minimizing inconsistencies in the classification choices during the labeling process. In [3, 16,17,18,19] algorithms for the automatic adjustment of the confidence threshold were proposed as an extension of those researches. Therefore, the focus of research development was on building solutions that could adjust the confidence parameter during the data labeling process.

In [8] a study was conducted on the existing risks in railway safety inspections. These inspections are reported using text, which generates a large amount of textual data. As motivation for the authors, a predictive model was proposed that used classifier ensembles to predict risk situations that could be addressed proactively. The chosen classification model was the decision tree, which was applied to an ensemble of Bagging classifiers that combined the results of the classifiers by voting. Similarly, [5] used different models of classifiers in an ensemble to predict a credit score for customers of financial organizations and banks.

In [4] a new ensemble learning algorithm named E\(\_\)RHFS was developed and applied as a tool to predict software failure. The authors used the voting combination method on the base classifiers to obtain the final ensemble classifier for software defect prediction. In [6] the proposal focused on two primary factors in creating a classifier ensemble, namely: accuracy and diversity. The paper aimed to propose a new diversity measure called Misclassification Diversity (MD) and an Incremental Layered Classifier Selection approach to build the ensemble.

4 The Proposed Method: FlexCon-CE

As already mentioned, the purpose of this paper is to propose a new FlexCon-C variation, called FlexCon-CE (Flexible Confidence with Classifier Ensemble). This variation uses an ensemble-based confidence threshold. Therefore, in order to develop the proposed method, it was necessary to modify the original FlexCon-C Algorithm 2 with lines marked in blue highlight. Algorithm 3 presents the FlexCon-CE step-by-step operation.

Algorithm 3
figure c

Flexible Confidence with Ensemble (FlexCon-CE)

Full size image

Initially, in Algorithm 3, the database is divided into two sets: a set of labeled instances \(D_{l}\) and a set of unlabeled instances \(D_{u}\). Additionally, a pool of classifiers \(F_{n}\) is available. An ensemble is defined as empty (line 1). Then, for each classifier f in \(F_n\), they are trained using \(D_{l}\) as training set. After that, the trained classifier f is added to the ensemble e (lines 2 to 7). Finally, the ensemble e will be applied to \(D_{u}\) in order to calculate the effectiveness generated by e (line 8). Then, the new confidence threshold is calculated (line 9), the unlabeled instances are selected, labeled, removed from the unlabeled set \(D_{u}\) and included in the labeled set \(D_{l}\) (lines 9 to 12). It is important to emphatize that the threshold is used to select unlabeled instances that will be labeled by the voting process of the ensemble.

It is important to emphasize that the calculation of the new confidence threshold is made using Eq. 1. The main difference is that the mp value is related to the ensemble accuracy (FlexCon-CE) instead of the classifier accuracy (FlexCon-C).

As previously described, two FlexCon-CE versions are proposed. In the first one, the classifier ensemble is constructed heterogeneously, using more than one type of classification method. In the second version, the classifier ensemble is built homogeneously, when using only one type of method. In this work, both heterogeneous and homogeneous ensemble approaches will be assessed.

5 Experimental Methodology

In order to assess the feasibility of the proposed method, an empirical analysis will be carried out. The next subsections will describe the main aspects of the experimental methodology.

5.1 Datasets

In this empirical analysis, the FlexCon-CE method is applied to 20 different databases. These datasets are available in platforms and repositories that maintain various experimental data, such as GitHub [20], UCI Machine Learning [21] and Kaggle Datasets [22]. Table 1 describes the characteristics of all datasets, regarding the number of attributes, instances, number of classes, data types (Categorical - C, Real - R and Integer - I), and data distribution (Balanced - B and Unbalanced - U).

Table 1. Datasets characteristics
Full size table

5.2 Methods and Materials

All method and learning procedures of this paper are implemented using the scikit-learn package available in the Python language.

In order to create the algorithm proposed in this article, we decided to use the scikit-learn Python library [23], since it offers several learning methods for classification, regression, and clustering domains. It also integrates with other widely used Python libraries for machine learning experiments, such as Matplotlib [24], Numpy [25], Pandas [26], and others. In order to facilitate the comparative analysis of the results of the FlexCon-CE method with FlexCon-C, three classifiers are used, which are: Naive Bayes (NB), Decision Tree (Tree) – using a CART-based algorithm – and K-Nearest Neighbors (k-NN) – with k values from 4 to 8. Additionally, the percentage values of the initially labeled instances, n, are 5\(\%\), 10\(\%\), 15\(\%\), 20\(\%\) and 25\(\%\), in the same way as it was performed by [16, 19, 27, 28].

Each trained classifier was included to the classifier ensemble heterogeneously and homogeneously, for FlexCon-CE (Het) and FlexCon-CE (Hom), respectively. For the combination of all n classifiers, the voting criterion was applied to define the overall ensemble output. The voting method consists of a non-linear combination of the classifiers’ outputs, and its process consists of determining the winning class, from a data input pattern, by the total number of votes counted for each classifier [4, 8, 29]. Therefore, the confidence of the classifier ensemble is defined by the number of classifiers that selected the winning class divided by the number of classifiers of the used ensemble.

The main aim of this empirical analysis is to evaluate the performance of the FlexCon-CE algorithm and to compare them to the FlexCon-C (previously explained: FlexCon-C1(v), FlexCon-C1(s) and FlexCon-C2) results.

In order to assess the obtained results, the Friedman statistical test was achieved. According to [30], this test compares related sample data, which means that the same object can be analyzed more than once. As it is non-parametric test, numerical values are not used directly, but rather their positions. After the ranking are performed for the groups separately, the hypothesis of similarity of the sum of the position of each group is verified. In this way, it is possible to compare the performance of different data classification algorithms in the scenarios created for each different dataset.

6 Experimental Results

This section presents the analysis of the obtained results. As previously mentioned, the analysis of the results will be conducted in a comparative way, evaluating the performance of both FlexCon-CE versions and three FlexCon-C versions, FlexCon-C1(v), FlexCon-C1(s), and FlexCon-C2 [3]. The next sections will present the analysis of the experimental results. The first one presents an analysis of the accuracy results presented by all analyzed methods.while the next section conducts a statistical analysis of the obtained results.

6.1 Accuracy Analysis

Tables 2 and 3 present the results obtained by each method used in this analysis. The difference between these two tables is that the first table shows the results of three different classifiers and compares the performance of FlexCon-CE using a homogeneous classifier ensemble (FlexCon-CE(Hom)) with all three FlexCon-C versions (FlexCon-C1(v), FlexCon-C1(s), and FlexCon-C2). For the second table, the best results of Table 2 are selected, for each classification algorithm (Naive Bayes, Decision Tree and k-NN), as well as the FlexCon-CE result using a heterogeneous classifier ensemble (FlexCon-CE(Het)).

Table 2. Results of average accurary
Full size table
Table 3. Results of best average accuracy
Full size table

Both tables have the same configuration: the first column presents the name of the used classifier while the second column indicates the semi-supervised learning method. Columns 3–7 display the average accuracy and standard deviation obtained by each semi-supervised method over all 20 used datasets, according to the settings of 5\(\%\), 10\(\%\), 15\(\%\), 20\(\%\), and 25\(\%\) of initially labeled instances. Additionally, in both tables, the results that showed the most significant accuracy values are highlighted in bold.

When analyzing results obtained with the Naive Bayes classifier in Table 2, it is possible to observe that the FlexCon-CE method using the homogeneous ensemble - FlexCon-CE(Hom) obtained the best results in 80% of the analysed cases (4 out of the 5% of initially labeled instances). In this same table, considering the results obtained by the Tree classifier, FlexCon-CE(Hom) also achieved the highest accuracy in 100% of cases (5 out of the 5% of initially labeled data). Based on these results, it is possible to state that the FlexCon-CE(Hom) method obtained significantly better performance results in two out of three analyzed classifiers.

Still exploring Table 2, in general, it is possible to observe that the FlexCon-CE(Hom) method performed better than all FlexCon-C versions in 60% of the analyzed cases (9 out of the 15 best results). This is a promising result showing that the use of a more efficient selection and labeling technique leads to more robust SSL methods.

As previously explained, Table 3 presents the performance of the proposed method using the heterogeneous ensemble - FlexCon-CE(Het), and the best results obtained in Table 2, for comparison purposes. When observing the aforementioned table, it is possible to identify that the FlexCon-CE(Hom) method obtained the best performance in 80% of the cases (4 out of 5% of initially labeled instances) when the number of initially labeled instances was higher than or equal to 10%. On the other hand, the other proposed method of this paper, FlexCon-CE(Het), obtained the best performance when the number of initially labeled instances was limited to 5%.

In summary, it is possible to state that one proposed version, FlexCon-CE(Hom) or FlexCon-CE(Het), achieved better accuracy results than all FlexCon-C versions, regardless of the type of classifier used. The next subsection presents the statistical analysis of the obtained results in order to demonstrate that the proposed semi-supervised machine learning methods are promising.

6.2 Statistical Analysis

The statistical analysis performed in this article uses critical difference (CD) diagrams. Figures 1 and 2 display the results obtained using the aforementioned diagrams. The post-hoc Friedman test is used to analyze the results based on their rankings. Additionally, in these diagrams, the methods located further to the left exhibit better results, while the ones on the right presented the poorest performances. Finally, a method is considered statistically different from another when both are not covered by the critical difference line (bold horizontal line in the diagram). Otherwise, when this line covers two or more approaches, it means that the null hypothesis of the Friedman test cannot be rejected.

Figure 1 presents the critical difference diagram with the results of the statistical test of the results obtained in Table 2. The applied classification algorithm is indicated in front of the method name. When analyzing the aforementioned diagram and observing the ranking of methods delivered by each classifier, it can be seen that FlexCon-CE(Hom)-TREE was the method that reached the highest ranking, which confirms the analysis carried out in the previous section regarding the accuracy of these methods. Additionally, according to the statistical analysis, this method is similar to the following four methods, in which two of them applied a decision tree (FlexCon-C2-TREE and FlexCOn-C1(v)) and the remaining two methods used a k-NN method (FlexCon-C2 and FlexCOn-C1(v)). On the other hand, the winning method is statistically superior to the remaining seven methods.

Fig. 1.
figure 1

Critical Difference Diagram using all methods

Full size image

Figure 2 illustrates the critical difference diagram with the results of the statistical test of the results obtained in Table 3. When analyzing the rankings of each method, it can be seen that the FlexCon-CE(Hom)-TREE and FlexCon-CE(Het) methods are statistically similar and both achieved the best results. This observation confirms what was reported in the previous section, in which the proposed methods outperformed the methods that do not use classifier ensembles.

Fig. 2.
figure 2

Statistics performance of the best

Full size image

In summary, the results obtained in this paper showed significant improvements of the proposed method, in comparison to the FlexCon-C versions. These results corroborate with the results obtained in the previous section, demonstrating an enhancement in the performance of the semi-supervised methods of the proposed methods. Moreover, the proposed methods are validated from two different perspectives, both in terms of performance and statistical analysis.

7 Conclusion and Future Works

This paper proposed a new SSL method, called FlexCon-CE, which was designed based on introducing classifier ensembles in the FlexCon-C functioning. The main difference between FlexCon-C and FlexCon-CE is that the latter uses classifier ensembles to define the adaptive confidence while the predictions of a classifier is used by FlexCon-C. Additionally, this work explores the use of two versions of the proposed method: one that uses an homogeneous ensemble structure and the other one used an heterogeneous ensemble structure.

In order to validate the proposed method, an empirical analysis was conducted, in which a comparative analysis of the performance of FlexCon-CE with FlexCon-C was carried out. In general, the performance of the analyzed methods was assessed in terms of accuracy and standard deviation and they were validated statistically using the Friedman test. After analyzing the results, it was concluded that the FlexCon-CE method obtained better results in most cases, with emphasis on the version that used the homogeneous ensemble with the Tree classifier, followed by the heterogeneous FlexCon-CE version.

For future work, we suggest to explore other classification algorithms and use new databases to address experimental scenarios. Furthermore, it is recommended to make the size of the classifier ensembles more flexible in relation to the number of base classifiers.