A Novel Genetic Algorithm Approach for Discriminative Subspace Optimization

Abstract

Image set representation by subspace methods has shown to be effective for several image processing tasks, such as classifying multiple images and videos. A subspace exploits the geometrical structure in which images are distributed, representing the image set with a fixed dimension giving more statistical robustness to input noise and compactness to the images. The mutual subspace method (MSM) and its extensions, the Orthogonal Mutual Subspace method (OMSM), and the Generalized Difference Subspace (GDS) are the most prominent subspace methods employed. However, these methods require solving a nonlinear optimization which lacks a closed-form solution. In this paper, we present a metaheuristic-based approach for discriminative subspace optimization. We develop a Genetic Algorithm (GA) for integrating OMSM and GDS discriminative subspaces. The initialization strategy and the genetic operators of the GA provide quality of objective function value of solutions and preserve their feasibility without any extra repair step. We validated our approach on four object recognition datasets. Results show that our optimization method outperforms related methods in accuracy and highlights the use of evolutionary algorithms for subspace optimization. Code: https://github.com/bernardo-gatto/Evolving_manifold.

1 Introduction

The problem of classifying image sets has been widely investigated in computer vision and supports several applications by handling, learning, and classifying data from multiple view cameras. For instance, in robot vision, where a data stream is available [1, 12, 20, 22]. In this context, a pattern set is a collection of images (or feature vectors) of the same object or event. This set can be unordered where the time stamp of the collected images is not relevant. The images can also be ordered when the timestamp is semantically meaningful.

Applications of subspaces for pattern-set learning frequently use the mutual subspace method (MSM) [16, 23]. These solutions are employed to solve gesture and action recognition problems, where video clips are described as subspaces, in which each subspace is computed from one of the pattern sets. Despite its advantages, traditional subspace methods like MSM lack discriminative feature extraction, which is critical for efficient pattern set modeling. A useful pattern-set model requires robustness to corrupt data; some images may contain noise, occluded targets, or dropped patches. The model must also correctly handle a variable set size, maintaining its computational complexity.

The orthogonal mutual subspace method (OMSM) [21] was developed to extract discriminative features using the Fukunaga-Koontz transformation (FKT). Its formulation has been applied to image-set modeling for solving a range of problems, such as gesture and action classification.

Recently, a variant of the subspace method called generalized difference subspace (GDS) has been developed [4]. The relationship between the patterns is considered by employing the concept of the generalized difference between the subspaces. This formulation provides a novel discriminative transformation, where the projected subspaces produce higher recognition results than conventional subspace-based methods.

Despite their excellent performance, OMSM and GDS lack closed-form solutions and cannot fully achieve their capabilities. Our work introduces a significant advancement by using a genetic algorithm (GA) to optimize the discriminative subspaces. This approach not only enhances the discriminative power of OMSM and GDS but also offers a more rigorous solution to the optimization challenges in subspace methods.

We optimize OMSM and GDS spaces to tackle these challenges. We develop a GA that efficiently handles subspaces by modifying its initialization, evaluation, selection, and crossover operators, which is complex given GA’s typical use in simpler optimization problems. Our GA-generated discriminative space surpasses OMSM and GDS in accuracy, showcasing GA’s versatility in subspace optimization. This innovation applies to traditional subspace methods, creating a compact model that enhances their inherent benefits.

This paper continues as follows: Sect. 2 describes the related work on subspace-related methods. Section 3 introduces the proposed optimization solution, and Sect. 4 shows the experimental results. Finally, Sect. 5 presents the final remarks and future directions. Code developed for this research can be found in the following repository: https://github.com/bernardo-gatto/Evolving_manifold (Fig. 1).

Fig. 1.

These plots illustrate the improvement in class separability achieved by different subspace optimization methods. From left to right, the subplots transition from the MSM with a 53% overlap between classes to the GA after 200 iterations (GA-200) with a reduced overlap of 5%, indicating a progressive enhancement in class distinctness..

2 Related Work

2.1 Subspace Method

It has been verified that using multiple images can significantly improve the performance of representing complex shape objects. The MSM [16] is a recognition method that efficiently handles multiple images. MSM is a method for classifying multiple image sets where a set of images is represented by a linear subspace generated by applying non-centered principal component analysis (PCA). The similarity between different image sets is calculated using the canonical angles between the subspaces, and this similarity is used to classify sets of input images.

While MSM plays an essential function in subspace-based methods, it cannot produce good recognition results since it does not consider the relationship between the sets of distinct classes. The OMSM  [5] has been developed to improve the discriminative ability of subspace-based methods. In OMSM, class subspaces are orthogonalized using Fukunaga-Koontz’s framework before computing the canonical angles. This transformation emphasizes the differences between the classes and significantly improves the recognition performance of MSM. Some of the applications of OMSM include face recognition and multiple view object recognition [6, 7].

Another variant of the subspace method was introduced in [4] named GDS, where the pattern sets are also represented as subspaces. The relationship between the patterns is taken into consideration by employing the concept of the generalized difference between the subspaces. This algebraic formulation provides a novel discriminative transformation, where the projected subspaces produce higher recognition results than conventional subspace-based methods.

2.2 Evolutionary and Genetic Algorithms

Research on the theory and applications of evolutionary algorithms (EAs) has been consistent due to the increasing number of complex nonlinear problems unsolvable by conventional gradient-based techniques [2]. Accordingly, EAs present several advantages, such as handling limiters, explaining and repeating the search process, understanding input-output causality within a solution, and introducing corrective actions.

EAs excel in various problem domains, notably in optimization, where they can uncover solutions beyond human intuition. Characterized as iterative refinement algorithms, they evolve a set of potential solutions using replication, reproduction, and selection, all guided stochastically towards a specific goal.

Since its first relevant publication [13], GAs are considered a well-known population-based EA designed to solve optimization problems where exact methods fail [9]. Their mechanisms are grounded in Darwinian natural selection, where fitter individuals produce more offspring, while less fit individuals are eliminated if they don’t adapt or mutate to improve their capabilities [17].

A canonical GA comprises these key genetic operators: 1) replication, which creates new solutions either by generating entirely new potential solutions or by altering specific existing ones; 2) reproduction, a mechanism to enhance the diversity of the solution pool, is achieved through two methods: recombination (crossover) and mutation; 3) selection, a process that identifies and retains solutions meeting certain criteria for inclusion in the subsequent iteration.

It is important to acknowledge the diverse range of optimization strategies discussed in the literature [3, 24]. For example, Bayesian optimization typically utilizes a cost function reliant on simulations, which often demands substantial computational resources and lacks a surrogate model. In contrast, EA offers an alternative approach, focusing on the iterative refinement of candidate solutions. This method is advantageous as it requires fewer cost function evaluations compared to the Bayesian approach, which relies on extensive sampling to create a pool of potential solutions.

The development of the proposed method is encouraged by the application of GAs to the well-known classifier in a non-Euclidean space. Such a space presents no trivial operations, such as sum or gradient, which are easily achieved in Euclidean spaces.

3 Proposed Method

Fig. 2.

Framework of the proposed optimization solution. Given a collection of labeled pattern sets, the task lies in 1) learning a subspace representation for each set and then 2) estimating a discriminative space where discriminative information is attainable. Our contribution lies in optimizing the discriminative space using GA, where a population of subspace candidates is evaluated and updated until a user-specific criterion is met.

3.1 Problem Formulation

Subspaces usually represent pattern sets to reduce computational complexity, achieving the immediate advantage of allowing parallel processing. Besides, subspace representation permits the examination of correlations among the various factors inherent in each pattern set.

In a classification problem, $\boldsymbol{P}=\{P_{i}\}_{i=1}^{n}$ denotes the set of all subspaces spanned by $\boldsymbol{U} = \{U_{i}\}_{i=1}^{n}$. We can then develop a projection matrix D that acts on the elements of $\boldsymbol{P}$ to extract discriminative information. In traditional GDS, this procedure is performed by removing overlapping components that represent the intersection between subspaces. Differently, OMSM weights the discriminative space, encouraging the decorrelation of the overlapping patterns.

Figure 2 shows the pattern-set classification pipeline and the proposed optimization solution. Given a collection of labeled pattern sets, the task lies in estimating subspace representations for each set and then learning a discriminative space, where the subspaces are projected, and improved patterns are obtained. Our contribution lies in optimizing the discriminative space using our Genetic Algorithm. An initial population derived from OMSM and GDS subspaces is successively evaluated and updated until a specific criterion is met.

3.2 Brief Review on MSM

To represent a pattern-set by a subspace, we use orthonormal basis vectors to describe them compactly. Given a feature matrix $X=[~x_1{\,}|{\,}x_2{\,}|{\,}\ldots {\,}|{\,}x_{m-1}{\,}|{\,}x_{m}~]$, where $x_j$ is a feature vector, possibly obtained through the vectorization of an image. Then, we can conduct a decomposition to gather knowledge of the geometric structure of X. The singular value decomposition (SVD) [14] produces a set of eigenvectors U and a set of eigenvalues $\varLambda$, where each column vector in U represents an axis and each value in $\varLambda$ describes how important this axis is in terms of reconstruction. Also, $\varLambda$ describes how much the vectors in X are correlated. The SVD of X is

$$\begin{aligned} {X}{X}^{\top } = {U}{\varLambda }{\,}{U}^{\top }~. \end{aligned}$$

(1)

Each column of U is a singular vector of ${X}{X}^{\top }$, and the main diagonal of $\varLambda$ presents the singular values in descending order. The analysis of $\varLambda$ is helpful in various problems, such as dimensionality reduction, signal filtering, and feature extraction. By analyzing the influence of each eigenvector, it is possible to select a small set by removing all but the top k eigenvalues in the diagonal of ${\varLambda }$.

The following criteria can be used to obtain the compactness ratio:

$$\begin{aligned} \mu (k) \le {\sum _{j=1}^{k}(\mathbf {\lambda }_j)}/{\sum _{j=1}^{m^\prime }(\mathbf {\lambda }_j)}~. \end{aligned}$$

(2)

In Eq. (2), k is the number of the selected eigenvectors which spans a subspace, ${\lambda }_j$ corresponds to the j-th eigenvalue of ${X}{X}^{\top }$. Then, $m^\prime = \text {rank}\left( {X}{X}^{\top }\right)$. It is useful to set k as small as possible to achieve a minimum number of orthonormal basis vectors, maintaining low memory requirements.

We employ the average of the canonical angles to compare the subspaces. A practical technique for computing the canonical angles between two subspaces P and Q is calculating the eigenvalues of the product of their basis vectors. Given $U_{p}$ and $U_{q}$, which span the subspaces P and Q, Eq. (3) computes the canonical correlations between P and Q:

$$\begin{aligned} U\,_{p}^{\top } U\,_{q} = U \varSigma V^{\top }, \end{aligned}$$

(3)

where the eigenvalues matrix $\varSigma$ provides the canonical correlations between the principal angles of $U_{p}$ and $U_{q}$ and can be used to compute the canonical angles, since $\varSigma = \textrm{diag}(\lambda _1, \lambda _2, \ldots , \lambda _{k})$. The canonical angles $\{\theta _j\}_{j=1}^{k}$ can then be computed by using the inverse cosine of $\varSigma$, as $\{\theta _j=\cos ^{-1}(\lambda _j) \}_{j=1}^{k}$. Finally, the average canonical angle $\bar{\theta }$ between P and Q is defined as $\bar{\theta }= \frac{1}{k^\prime } \sum _{j=1}^{k^\prime } \theta _j$, where $k^\prime \le \textrm{min}(\textrm{rank}(P),~\textrm{rank}(Q))$. Once the average of the canonical angles is obtained between the training and the test subspaces, we can employ the nearest neighbour algorithm (1-NN) to classify the test subspaces.

3.3 Conventional Discriminative Subspaces

Once equipped with all the subspaces (using Eqs. (1) and (2)), OMSM employs the FKT to generate the matrix $D_o$ that can decorrelate the subspaces. Each set of basis vectors ${U}_i$ spans a reference subspace ${P}_i$, where its compactness ratio is empirically defined by employing Eq. (2). The method to generate the matrix $D_o$ that efficiently decorrelates the subspaces is explained. We compute the total projection matrix:

$$\begin{aligned} {G} = \sum _{i=1}^n {U}_i {U}_i^{\top }, \end{aligned}$$

(4)

then we decompose the total projection matrix G:

$$\begin{aligned} D_o = {\varLambda }^{-1/2} \ {B}^{\top }, \end{aligned}$$

(5)

where B is the set of orthonormal eigenvectors corresponding to the largest eigenvalues of G, and ${\varLambda }$ is a diagonal matrix with the k-th highest eigenvalue of the matrix G as the k-th diagonal component.

Differently, $D_g$ produced by GDS can be computed by discarding the first d eigenvectors of B as follows: $D_g = B \setminus A$, where $A =\{U_{k}\}_{k=1}^{d}$ and $\setminus$ is the relative complement (i.e., removes all the elements of A from B). In practice, OMSM and GDS are very similar. Their differences arise from the fact that the eigenvectors of G are either weighted or discarded, suggesting that an optimization strategy should be implemented for weighting or discarding eigenvectors.

Since this problem can be analyzed from a feature selection point of view and, according to [10], it is an NP-hard problem, we argue that using an EA for combining both $D_o$ and $D_g$ discriminative spaces is a viable solution. In this scenario, a suitable objective function $\boldsymbol{f}$ must be an indicator of how much a discriminative space can discriminate different groups. We discuss the chosen objective function in the following section.

3.4 Proposed Algorithm

We propose an EA in the likes of the GA of Holland [13] with all genetic operators but the mutation operator. Algorithm 1 has the following components: InitializePopulation, Evaluate, Selection and Crossover. The overall steps of the GA are shown in Algorithm 1. The encoding of solutions in our GA is in the form of a matrix, each solution encodes the basis vectors of the discriminative subspaces $D_o$ and $D_g$. This representation allows the GA to effectively combine and evolve them over iterations. It can be thought of as a concatenated or structured representation of these basis vectors, where each vector undergoes optimization through genetic operations. This approach ensures that the evolutionary process refines the subspaces for optimal discriminative power in classification tasks.

Algorithm 1

. Genetic Algorithm for the Subspace Optimization Problem

In summary, our proposed GA starts by initializing a population of solutions $\boldsymbol{D}_0$ of size n. In each iteration, the algorithm evaluates the fitness of the current population $\boldsymbol{D}_i$ using an objective function $\boldsymbol{f}$ based on the Fisher score. Based on these fitness values, a selection process $\textsf {Selection}$ is carried out to choose a subset of the current population. This subset then undergoes a crossover operation $\textsf {Crossover}$, producing a new set of solutions $\bar{\boldsymbol{D}}_i$. The new generation for the next iteration $\boldsymbol{D}_{i+1}$ is formed by combining the selected solutions and the offspring from the crossover. The process repeats until the maximum number of iterations max is reached. Finally, the algorithm returns the best solution from the final population as determined by the objective function $\boldsymbol{f}$.

Part of OMSM’s traditional optimization approach performs the weighting of eigenvectors associated with the lowest eigenvalues. Intriguingly, the optimization process of GDS is performed by eliminating eigenvectors associated with the highest eigenvalues, suggesting a complementary relationship exists between both optimization strategies.

This greedy strategy produces good results; however, it does not ensure that the optimal discriminative space is the one obtained by the method. Besides, some eigenvectors associated with the middle eigenvalues are never removed or evaluated, and thus, this greedy strategy may lead to weak discriminative spaces.

It is worth mentioning that the removal and concatenation strategy employed in this work may produce discriminative spaces of different dimensions (i.e., different subspace dimensions), which is not exactly a problem in subspace analysis. The number of basis vectors exhibited by each element is expected to be modified according to the dataset complexity. We now detail each step as follows.

InitializePopulation: The initialization step ensures that the initial population $\boldsymbol{D}_0$ contains $D_o$ and $D_g$ discriminative spaces, their concatenations, and random variants with randomly selected dimensions and their respective concatenations. More precisely, the basis vectors of $D_o$ and $D_g$ can be concatenated to, after an orthogonalization process, produce a novel discriminative space, preserving the capabilities of OMSM and GDS. We create a completely unexplored population by randomly truncating and concatenating $D_o$ and $D_g$.

This initialization strategy has two immediate advantages: 1) We ensure that the produced solutions will not be worse than the ones provided by $D_o$ and $D_g$ discriminative spaces and, 2) We start from a good enough population, which may guide solutions to converge to a local optimum faster than purely random initialization. Our motivation assumes that a random initialization would be challenging to achieve the same performance as the warm start provided by OMSM and GDS. Besides, the probability of discovering better discriminative spaces in the neighborhood space is higher once we start from promising ones.

Evaluate: As the objective function $\boldsymbol{f}$ of the GA, we selected a score reflective of each discriminative subspace’s efficacy in classification metrics. These metrics can encompass accuracy, F1 score, and mean squared error. For our implementation, the Fisher score was chosen due to its straightforward interpretability and effectiveness in quantifying the discriminative power of subspaces. The Fisher score measures the separation between different classes in the feature space, making it a suitable metric for optimizing subspaces in classification tasks [11]:

$$\begin{aligned} \boldsymbol{f} = \text {trace}\left( (S_b)(S_w + \epsilon I)^{-1}\right) , \end{aligned}$$

(6)

where $S_b$ and $S_w$ are the between-class scatter matrix and within-class scatter matrix, respectively, measuring the dispersion of the subspaces. Here, $\epsilon$ is a positive regularization parameter, and I is the identity matrix. The Fisher score aims to maximize the between-class and within-class variance ratio, ensuring that the evaluated subspace is a robust discriminative model.

Selection: The selection process employs a rank-based approach. Specifically, it involves arranging the discriminative spaces according to their Fisher scores in decreasing order. From this ranked list, the top $p\%$ of spaces, those with the highest scores, are selected for inclusion in the crossover pool. This method ensures that the most promising solutions are carried forward, encouraging the generation of increasingly effective discriminative spaces in subsequent iterations.

Crossover: In the crossover phase, two elements (discriminative spaces) are randomly selected from the pool created in the Selection step. These elements undergo crossover, and a one-point crossover operation is performed. A cut point $c\%$ is randomly determined, and each offspring solution is composed of eigenvectors from both parents, split at this cut point. For instance, let $D_p = \{\phi _{(i)}\}_{i=1}^{n_p}$ and $D_q = \{\psi _{(i)}\}_{i=1}^{n_q}$ be two selected parent solutions. Their crossover operation would result in two offspring solutions $D_{p}^{~\prime }$ and $D_{q}^{~\prime }$, formulated as follows:

• $D_{p}^{~\prime } = \{\phi _{(1)},~\ldots ,~\phi _{(c)},~ \psi _{(c+1)},~\ldots ,~\psi _{(n_q)}\}$,

• $D_{q}^{~\prime } = \{\psi _{(1)},~\ldots ,~\psi _{(c)},~ \phi _{(c +1)},~\ldots ,~\phi _{(n_p)}\}$.

Following the progress of the Algorithm 1, the selected population $\boldsymbol{S}_i$ is merged with $\bar{\boldsymbol{D}}_i$ to compose the next population set; this can be seen as an update process that follows a measure of the elitism of solutions. Evaluation, Selection, and Crossover are then performed until a predefined number of iterations is achieved.

The proposed algorithm presents the main elements of a GA; however, its application is in a non-Euclidean space. The provided mathematical formulations employed for orthogonalization, basis vector concatenation, and projections are the foundations for discriminative subspace optimization employing EA. The following section is focused on applying the presented concepts for pattern set classification. We employ subspace-based methods for representing pattern sets and optimize a discriminative subspace using strategies motivated by the GA presented in this section.

4 Experimental Results

We evaluate the proposed GA to demonstrate its advantages over other pattern-set representation and classification methods. First, we describe each dataset and our experimental protocol. Next, we investigate the approach for initial population estimation. We then compare the proposed approach with existing methods for the same task. Lastly, we provide concluding remarks on our results.

ETH-80 is a dataset designed for object recognition. It contains images of 8 object categories, each with 10 subcategories in 41 orientations, totaling 410 images per category and 3280 images overall. We resized the images to $64 \times 64$ pixels and extracted grayscale information. For our analysis, we used the images with backgrounds.

For a comprehensive classification task, we used the ALOI dataset. ALOI is a large dataset of general objects with about 110 images per object, considering different illumination angles, colors, and viewing angles. We used the first 500 object instances. All images were segmented from the background, and we classified them into one of the 500 objects using a 10-fold cross-validation scheme.

We also used the Coil-20/100 datasets. Coil-20 contains 20 objects, with 72 images captured for each using a turntable, resized to $128 \times 128$ pixels. Similarly, Coil-100 was obtained with more objects. We split the samples into five groups, randomly selecting one for training and using the rest for testing.

In the experiments, descriptive statistics are the mean accuracy, standard deviation, and best result obtained from the runs of each experiment using pre-established random seeds. Chosen parameters for the proposed GA for all experiments were: number of iterations as 200; population size of 200 solutions; crossover rate c as 0.5; and populational elitism p of the selection step as $10\%$. The experiment reported in this paper was run on a Unix-like PC equipped with a Core i9 5.00GHz with 16 GB RAM written in Python.

4.1 Evaluating the Initialization Strategy

This experiment analyzes the adopted initialization strategy on the ETH-80 dataset. We utilized an initialization strategy where the elements are assigned to OMSM and GDS discriminative spaces and some variants based on random subspace truncation. Two elements are assigned to the $D_o$ and $D_g$; then, random truncation is performed on either $D_o$ or $D_g$ before concatenation. This strategy not only creates new discriminative subspaces but also ensures that a vast space is explored and that the results obtained will never be worse than the ones provided by OMSM and GDS.

Fig. 3.

Accuracy from 1000 runs of the presented initialization strategy.

Figure 3 shows the accuracy results of 1000 runs of the presented initialization strategy. We see that the initialization alone presents attractive solutions. For instance, some elements deliver accuracy higher than OMSM and GDS. However, some display very low accuracy, which may be due to selecting unsuitable eigenvectors. If discriminative eigenvectors are discarded, the learning algorithm accuracy may sharply drop.

4.2 Analysing Distance Matrices

In this experiment, we analyze the distance matrices produced by OMSM, GDS, and our GA. The objective is to understand if the optimized discriminative subspace presents advantages over OMSM and GDS w.r.t. feature extraction capabilities. Also, the distance matrices may provide information concerning their discriminative ability, such as the intra-class and inter-class separability. For this experiment, each algorithm was run for 200 iterations.

Figure 4 displays the distance matrices generated by OMSM, GDS, and the GA on the ETH-80 dataset.

Fig. 4.

Distance matrices of OMSM, GDS, and the proposed GA, respectively.

The proximity matrix produced by OMSM shows moderately high similarity even when subspaces describe different classes, indicating that the discriminative subspace may reveal insufficient information for classification. In contrast, the proximity matrix produced by GDS shows lower similarity, revealing moderately good discriminative information. This demonstrates the effectiveness of the GDS discriminative space, as it relies on an analytical solution.

Differently, the similarity between different classes is very low when our optimization method is employed, represented by the dark structures on the matrix. This suggests that the presented optimization technique based on the GA produces a discriminative space capable of separating subspaces of distinct classes while keeping together the same category.

4.3 Comparison with Related Methods

Table 1. Study of performances for the proposed optimization method (left) and comparison with OMSM and GDS (right).

This experiment evaluates the proposed GA for discriminative subspace optimization using the ETH-80, ALOI, COIL-20, and COIL-100 datasets. We compare the proposed method with MSM, OMSM, and GDS. The objective here is to understand whether the optimization strategy developed in this work can produce better results on diverse datasets. It is worth mentioning that more sophisticated manifold learning algorithms exist in the literature. However, we narrowed down the scope of our evaluation to linear methods since our optimization strategy was developed to support linear subspaces.

A comprehensive summary of the obtained results is shown in Table 1. We see from the results that the GA achieved its highest accuracy on the COIL-20 and ETH-80 datasets, while the results produced on the ALOI dataset are the lowest. These results may indicate that our GA may benefit from a larger initial population when more complex datasets, such as ALOI, are being evaluated. In this experiment, we present the best results obtained from 100 runs with varying random seeds, where each run is executed for 500 iterations.

Table 1 also displays the results achieved by OMSM, GDS, and the GA on all the employed datasets. The presented method exhibited the lowest accuracy gain on the ALOI dataset compared to the gain obtained on other datasets. This suggests that the GA encountered difficulty when the classification problem’s number of classes was high.

The highest improvement gain was obtained on the ETH-80 dataset. These results may have been achieved because the ETH-80 is the smallest dataset among those evaluated, which can benefit all methods based on linear subspaces. Furthermore, since the GA does not implement a regularization technique, the provided results on more complex datasets may overfit. Besides, OMSM and GDS presented competitive results despite implementing no optimization approach, indicating their usefulness when optimization is not practical.

4.4 Visualizing the Discriminative Spaces

This section presents a visualization of the discriminative spaces derived from the ETH-80 dataset after training. The dimensions of the spaces are as follows: the GA subspace comprises 196 eigenvectors, the OMSM subspace contains 83 eigenvectors, and the GDS subspace encompasses 222 eigenvectors. Figures 5, 6 and 7 display the first 10 and last 10 eigenvectors of the OMSM, GDS, and GA discriminative spaces, reshaped into $15 \times 15$ matrices.

Upon analyzing the similarity of eigenvectors across these subspaces, we observe no overlap between GDS and OMSM, which implies that the two spaces are capturing entirely distinct features of the dataset, confirming our previous hypothesis regarding the complementary nature of these spaces. In contrast, there is a significant overlap between GDS and GA, with 163 eigenvectors being shared. This suggests that GA has a strong affinity with the GDS subspace, potentially indicating that GA is preserving a substantial amount of the detailed features captured by GDS. Furthermore, GA discriminative space shares 33 eigenvectors with OMSM, which, although fewer in number, is a significant share considering the OMSM’s dimension.

The OMSM subspace, being more compact, may focus on the most salient features of the dataset, which are sufficiently representative of the specific discriminative tasks it’s designed for. On the other hand, the GDS subspace, with its larger number of eigenvectors, is likely to preserve more intricate details of the data, which could be advantageous for tasks requiring a high level of granularity.

The GA subspace’s ability to share eigenvectors with both OMSM and GDS indicates its capability to exploit both the compact, representative nature of OMSM and the detailed, nuanced capture of GDS. In quantitative terms, GA incorporates approximately 83.16% of GDS and 39.75% of OMSM eigenvectors, reflecting its comprehensive nature in encapsulating the diversity of features within the dataset. This makes the GA subspace potentially versatile and suitable for a wide range of applications that may require either a broad overview or a detailed analysis of the data’s characteristics.

Fig. 5.

Eigenvector visualization of the OMSM subspace.

Fig. 6.

Eigenvector visualization of the GDS subspace.

Fig. 7.

Eigenvector visualization of the GA subspace.

In this section, we evaluated our proposed optimization approach regarding the initialization strategy, its distance matrix, accuracy on various datasets, and the visual characteristics of the eigenvectors. We can describe two main findings from the obtained results: 1) the employed initialization strategy frequently shows advantages over conventional methods, suggesting that the proper use of OMSM and GDS discriminative subspaces improves the classification accuracy; 2) the GA presents discriminative subspaces capable of achieving even higher classification results by iteratively selecting merged subspaces of random dimensions. These findings not only validate the robustness of our proposed approach but also highlight the potential of combining discriminative spaces to achieve superior performance in classification tasks.

5 Final Remarks and Future Directions

This paper presents a Genetic Algorithm (GA) for optimizing discriminative subspaces to enhance feature extraction in object classification, utilizing Evolution Strategy (ES) principles. The GA employs initial population generation, evaluation, and selection, using the Fisher score for fitness assessment. Crossovers among randomly selected elements create new populations, refining solutions with each iteration based on Fisher scores. Evaluated on object recognition tasks, the method demonstrates high classification performance across various datasets, effectively capturing high-dimensional data and uncovering subspace structures.

Future directions include advanced evaluation methods and exploring kernel approaches for nonlinear data distributions. Our solutions are applicable in gesture and action recognition, adapting to tasks using basis vectors for data representation, suitable for subspaces of any pattern-set nature. We also plan to develop mutation techniques for subspaces to further improve GA performance and explore additional metaheuristics for pattern set optimization problems.