1 Introduction

Symbolic Regression (SR) is one of the main applications and motivators to Genetic Programming (GP), a method for automatically generate computer programs from a high level definition of a problem [15]. Since early 90ś, many data-driven problems are modeled as SR problems [2], where no previous knowledge or pre-processing input is required. GP is well suited for resolving such problems since any algebraic function set can be effectively represented as trees and implemented as computer programs for the problem domain [9].

Bloat - the uncontrolled and excessive growth of individuals without a proportional gain of fitness - is a well-known issue and a field of study in GP. It is specially noted in SR. The large amount of inefficient code causes excessive consumption of computational resources, as well as many other practical issues [16], hiding the problems real complexity and domain.

There are many approaches for avoiding uncontrolled code growth in tree-based GP [5], some of which are presented as follows. The most simple and popular of them is implementing a Depth Limit [9], although is not a truly effective approach, as it can induce growth in some scenarios. Parsimony Pressure [9, 13] is another popular technique which adds a penalty term in fitness function to punish large trees. Pseudo-hill Climbing [6] attempts to guarantee the fitness to improve in population rejecting individuals least fit than its parents until one is finally accepted. Code Editing approach mutates or removes redundant sub-trees in individuals.

Regarding the code editing approach, Koza [9] proposed a simple method to simplify (to convert a tree into a smaller, equivalent, tree) using grammar rewrite rules, which further inspired the Rule Based Simplification (RBS) method [7, 11, 20]. RBS was extended with Equivalent Decision Simplification (EDS) [11], which recursively compares all sub-trees in an individual for equivalency with a small set of terminals.

This paper proposes the Dynamic Operator with RBS and EDS (DORE), which improves a code editing bloat control algorithm using both RBS and EDS to maximize its reduction potential without greater punishments in execution time. The main feature of DORE is to dynamically learn redundant expressions to be applied with RBS from EDS outputs. It also optimizes the access of RBS rules using the any keyword in a hash-table implementation, as well as introduces a warm-up stage to grow RBS rules before the evolutionary process begins.

The article is organized as follows: Sect. 2 provides a quick overview about code editing operators; In Sect. 3 an improved strategy using the previous operators is proposed; Sect. 4 shows how RBS and EDS were implemented and the technical resources utilized in benchmarks; Sect. 5 shows a performance comparison between the simplification operators, beside the empirical results and discussion over three SR benchmark problems; and finally in Sect. 6, the conclusion is presented and the possibilities for further research.

2 Background

Redundancy is one of the key contributors to inefficient code growth. EDS [11] was introduced to extend RBS as they complement each other. RBS removes redundant subtrees by replacing a tree for a smaller equivalent one by applying arithmetic rules such as \(X/1 \rightarrow X\) and \(0*X \rightarrow 0\). These rules must be known and provided before the evolutionary process starts, thus each rule must be explicitly specified. Another limitation by RBS is that it’s rules must be specified exactly like it would appear in an individual. For example, the rule \(X * 0 \rightarrow 0\) would not be sufficient to simplify a tree \(0 * X\) unless the rule \(0 * X \rightarrow 0\) is specified.

EDS extends RBS in a manner that it simplifies trees without previous knowledge of algebraic rules. It can also remove redundancies that are only true in the training domain. The simplification by EDS is made by evaluating each subtree in an individual and comparing these subtrees with a set of small trees or terminals which is usually the result of simplifications, such as X, 1 and 0. In a SR problem, EDS is evaluated as follows [11]:

  1. 1.

    Determine a suitable set of simple trees \(S_{simple}\).

  2. 2.

    Check all subtrees in the target for equivalence to a tree in \(S_{simple}\).

  3. 3.

    If some subtree is equivalent to a tree in \(S_{simple}\), and larger than it, replace that subtree with the simple tree.

  4. 4.

    Repeat this procedure recursively until it fails.

Finding a suitable set of trees \(S_{simple}\) for a problem is not an easy task. Usually the set of terminals is a natural fit for \(S_{simple}\), but if it is already known that some subtrees must appear in the final output, such as in trigonometrical problems, they can be inserted in this set. However, the main issue of using EDS as a single operator of bloat control is the computational performance. With problems hard enough and individuals big enough, reducing a single generation can take as long as multiple generational evaluations. This scenario is not uncommon in simple SR problems. If an evaluated individual with EDS contains more subtrees than the population size, the evaluation function will be called upon as much as in the generational evaluation.

2.1 Simplification with RBS and EDS

The flow of simplification using both RBS and EDS is as follows [11]:

  1. 1.

    Let the genotype tree of individual i be \(t_{i}\).

  2. 2.

    Apply RBS recursively to all nodes of \(t_{i}\), until there is no node to which RBS can be applied, obtaining \(t'_{i}\).

  3. 3.

    Apply EDS to all nodes of \(t'_{i}\). If any node is translated, let the translated tree be \(t_{i}\) and go to (2). If there is no node to which EDS can be applied, finish and let \(t'_{i}\) be the final result.

Applying RBS before applying EDS is a good idea since it prevents EDS to be used for simplifying already known rules. Also, RBS execution time is lower than EDS, which will be explored further in this paper.

3 Proposed Improvements

The main idea of the improvements to the simplification with RBS and EDS flow is to increase RBS rules table R dynamically as soon as new rules, or more efficient ones, are discovered. This allows RBS to execute independently with no large impacts in execution time. A more robust RBS operator also helps in simplification itself, since any simplification made by EDS triggers new RBS calls, as well as new EDS calls. If simplifications made by EDS occurred in the first step of the simplification flow, then it would eliminate the need to reevaluate subtrees that would only be reduced by EDS.

The first improvement proposed is to allow the simplification rules table R, used by RBS, to have rules inserted in execution time. Each time a subtree is simplified by RBS using the keyword \(\_ANY\_\), which denotes any subtree, a new rule with the original subtree as input is inserted in R. Afterwards, new calls to RBS with the same subtree as input have O(1) access guaranteed in a hash-table implementation.

Analogously, each simplification made with EDS creates a new rule in R table with the learned simplification. This way, there is no need to apply simplification with EDS to the same subtree in subsequent individuals. Algorithm 1 shows an example algorithm to simplify a generation using this improved simplification flow with RBS and EBS in a SR problem.

Another strategy adopted to grow even more the R table was the insertion of a warm-up step before the evolutionary process begins. This warm-up step consists of generating random trees in the training domain of a problem, with random sizes, and then simplifying them with the improved flow. Each successful reduction creates a new rule in the table, so the GP starts with a larger R table.

figure a

4 Methodology

All algorithms were implemented in Python 3.8.3 with the package DEAP [4] (Distributed Evolutionary Algorithm in Python) in version 1.3.1. Parallelism was used to evaluate fitness functions using the package Ray [12] in version 0.8.6. The benchmarks were conducted in a 2018 MacBook Pro with 2.2 GHz 6-Core Intel Core i7 and 16 GB 2400 MHz DDR4 memory.

4.1 Rule Based Simplification

The RBS simplification rules table R was implemented as a hash table. The keyword \(\_ANY\_\) denotes any subtree and allows rules such \(\_ANY\_ * 0 \rightarrow 0\) to be defined. In the original paper [11] this keyword was referenced as A and no further details were provided on how it was implemented. In this paper, every subtree evaluated by RBS consults the hash table directly and only if no rule with the subtree is described the \(\_ANY\_\) rules are consulted. The access to the table R has complexity O(1), except on the scenario where the \(\_ANY\_\) keyword is included in the rule body, which makes the complexity O(N) where N is the amount of rules in the table.

Reducing the amount of rules with \(\_ANY\_\) is a slight improvement to this implementation as it maximizes the rate of O(1) accesses. Another way to attain this goal is to define as many redundant rules as possible in R, for example prioritizing the definition of rules \(1 * 0 \rightarrow 0\) and \(X * 0 \rightarrow 0\) instead of \(\_ANY\_ * 0 \rightarrow 0\). This task can be hard when there is no previous knowledge of such simplifications in the domain of the problem.

4.2 Equivalent Decision Simplification

The subtrees set \(S_{simple}\) for comparison for equivalence in EDS was implemented as a simple list. Two subtrees is considered equivalent if all fitness values elapsed from this subtree in the domain has relative error \(\epsilon < 0.005\). This threshold was chosen after some tests and set to all benchmark problems in both compared operators in this paper.

5 Experimental Results

This section is organized as follows: Sect. 5.1 shows a preliminary benchmark, validating the performance improvement on each simplification method; It is presented in Sect. 5.2 the results for three artificial SR problems using the proposed operator, comparing it with the original operator and baseline GP.

5.1 Simplification Methods Performance

A simple benchmark - using \(cos(2\pi x)\) as objective function - was ran 50 times to compare the execution time of the five simplification strategies.

Experimental Setting. The experiment was the simplification of many independent individuals in each reduction strategy: traditional versions of RBS and EDS, RBS and EDS with dynamic rules allowed and RBS after the warm-up step. More details of each strategy were discussed in Sect. 3. The fitness function was defined as the RMSE (Root Mean Squared Error), of 20 uniform points in the interval \([-\pi , \pi ]\). The function set used was \(\{+. -. *, \div \}\), where \( \div \) is the protected division satisfying \(X / 0 \rightarrow 1\). The terminal set is \(\{X, 0, 1, \pi \}\). For each execution, 200 trees are generated and then cloned in each step to avoid any bias or propagation of simplifications. The reductions applied in one step are not carried on to the next. The procedure is described as:

  1. 1.

    Reduction with RBS is applied to the 200 cloned individuals.

  2. 2.

    Reduction with EDS is applied to the 200 cloned individuals.

  3. 3.

    With the dynamic insertions in the rules table R allowed, reduction with RBS is applied to the 200 cloned individuals.

  4. 4.

    With the dynamic insertions in the rules table R allowed, reduction with EDS is applied to the 200 cloned individuals.

  5. 5.

    With the dynamic insertions in the rules table R allowed, a warm-up step is done, and reduction with EDS is applied to the 200 cloned individuals.

The 200 individuals are generated in each of 50 executions with the method Ramped Half and Half and with depth limited to [5, 10]. The warm-up step applied RBS and EDS sequentially to 200 trees generated with Grow method and with depth limited to [1, 5]. The initial RBS rules table R is defined in Table 1.

Table 1. Initial simplification rules table used by RBS
Full size table

Results and Discussion. Table 2 shows the arithmetic mean of the 50 executions in each strategy. Dynamic RBS is 4.84% faster than the default RBS. The RBS after the warm-up is shown to be 16.83% faster than the traditional RBS and provided a 11.52% improvement to dynamic RBS before the warm-up.

Table 2. Execution time comparison of the simplification operators
Full size table

This benchmark indicates that not only a bigger R table can improve the simplifications performance but also how much slower EDS is compared to RBS as well. With no improvements, EDS is 2.09 times slower than RBS. Dynamic EDS is 1.84 times slower than dynamic RBS before warm-up. Dynamic RBS after the warm-up can be twice as fast as the dynamic EDS. The warm-up increased the rules of table R, in average, from 19 entries to 135, to an around 86% improvement.

5.2 Symbolic Regression Problems

To validate the proposed improvements, 3 simple artificial SR problems were tested. These benchmarks were extracted in [5], selected from [8, 10, 17,18,19]. Since this paper presents a preliminary analysis of the operator, the benchmark problems are rather simple and harder problems will be explored in future works. The problems are presented in Table 3.

Experimental Setting. For each problem, 50 independent executions were performed, in 3 different approaches: Base GP: with no code editing bloat control operator, Baseline Operator: with the original simplification flow of RBS and EDS as in [11], and DORE: using the proposed dynamic operator.

Table 3. Artificial symbolic regression problems
Full size table

The function set to all benchmarks is set to \(\mathcal {F}=\{+, -, *, \div \}\) where \(\div \) is the protected division satisfying \(X / 0 \rightarrow 1\). The terminal set to benchmarks 1 and 2 is \(\tau = \{0, 1\}\) and the decision variables. For benchmark 3, the terminal set is \(\tau = \{0, 1, \pi \}\) and the decision variable. To all benchmarks, the fitness function is the RMSE of the training domain points.

Besides the traditional GP parameters, two new parameters were introduced to fit the operator’s context in more computational resource expansive problems. The first one is the percentage of population \(\rho \) which passes through the full simplification flow. The algorithm is developed in such way that the best \(\rho \%\) individuals of the population are chosen each generation. The second additional parameter is the remaining best \(\gamma \) percentage of the population that was not previously chosen and passes through the reduction with RBS only. For example, if the population size is 100, and the values \(\rho = 40\%\) and \(\gamma = 20\%\), the 40 most fit individuals would pass through the complete reduction flow and the remaining best 20 trees would be reduced with RBS only.

These additional parameters are needed due the high computational cost of the reductions that would make unfeasible the operation of all individuals without a great impact in execution time. A large \(\gamma \) value does not have a high impact on the execution time, since RBS reduction is simpler. Nonetheless, the \(\rho \) value greatly impacts the total execution time. For the benchmarks these parameters were chosen empirically, testing values big enough that would not make the total GP execution time significantly slower than the Base GP.

A custom tournament selection, named as Partial Tournament, is used to reduce the number of repeated individuals in the crossover. Being P the population, the Partial Tournament does |P| traditional K size tournaments, but limits the repeated number of each individual to two. Then, if necessary, the population is completed with individuals chosen at random up to |P|.

Table 4. GP settings used in symbolic regression benchmarks
Full size table

The GP settings used is presented in Table 4. The simplification rules used in RBS are as described in Table 1. The simple subtrees set \(S_{simple}\) used in EDS is the terminal set \(\tau \) for each problem. The warm-up step used in DORE runs is defined as 200 full trees with depth limited in [1, 5] range.

Results and Discussion. The median was preferred over the mean in this section since it is less sensitive to outliers and the data is not guaranteed to follow a normal distribution. In all of the data in any graph or table the median of the 50 executions is shown.

It was first analyzed how the fitness behaved over generations. Figure 1 shows how Base GP is worse than the others in all benchmarks. It also presents how Base GP fitness curve tends to flatten in each benchmark. DORE has slightly better results than the baseline operator. Figure 2 shows a fitness boxplot of the best individuals in each generation.

Fig. 1.
figure 1

Best fitness versus generations, for all benchmark problems

Full size image
Fig. 2.
figure 2

Boxplots of the best of generation individuals, for all benchmark problems

Full size image

The size of individuals is a crucial concern to have bloat-controlled GP executions, beside the fitness stagnation. A tree depth limit was used, so it was not expected a completely uncontrolled growth on Base GP. As Fig. 3 shows, both operators had a much better size control on the population over generations than Base GP. The difference between DORE and baseline operator was tiny, with DORE having best results on the first two benchmarks.

Table 5 shows the validation and test domains used to analyze the best individuals from each generation for all benchmark problems. It is presented in Table 6 the median and median average deviation (MAD), as utilized in [1], of each set of points - training, validation, and test - using the RMSE as reference. Table 7 shows the same metrics for the tree size of these best individuals. As the data is not normally distributed, Table 6 and Table 7 also present the p-value of the Mann-Whitney U-test [3] considering DORE x Base GP and DORE x Baseline Operator datasets. The null hypothesis is that the distributions of both datasets are equal. DORE performed better regarding tree size in all benchmark problems, compared to the other two strategies. It also presented better results in training error in all benchmarks. The benchmark 3 test error was worse using an operator, which suggests that this kind of strategy is not well suited for more complex problems even though the generated trees were smaller.

Fig. 3.
figure 3

Tree median size versus generation

Full size image

Besides the best trees found for each generation, the best individuals of each run were also analyzed. Figure 4 shows the mean training fitness against the mean tree sizes. It is clear that DORE dominates the other strategies in all 3 benchmark problems regarding the fitness.

In benchmark 1, due to bloat, Base GP generates smaller but unfit individuals. These individuals are, in average, from earlier generations than the ones using code editing operators, see Fig. 5. The boxplot shows how Base GP converges earlier than the other two strategies. Thus, it is expected that trees generated in later generations have better fitness but also a larger size. DORE performed better than the other strategies in Benchmarks 2 and 3 and regarding both size and fitness, as shown in Table 6 and Table 7. In all benchmarks it is noted that the average size of the best individuals in DORE are better than the ones with the baseline operator, see Table 7.

Table 5. Validation and test domain for best of run individuals analysis
Full size table

Another point of interest analyzed was the reductions made between baseline operator and DORE. Each RBS and EDS simplification made was logged in each run in these strategies. The tables and graphs related to reductions used the arithmetic mean as an average value. Table 8 highlights how DORE increased the number of unique simplifications made by RBS. This result was expected since DORE tends to increase the reductions table used by RBS.

Table 6. Best individuals for generation performance, of all benchmark problems
Full size table
Table 7. Tree Size data of best individuals for generation, of all benchmark problems
Full size table
Fig. 4.
figure 4

Best of run average fitness against best of run average size

Full size image
Fig. 5.
figure 5

Best of run generation boxplot

Full size image
Table 8. Average value of unique RBS simplifications performed
Full size table
Table 9. Average total simplifications made by RBS and EDS
Full size table

The average value of total reductions made by RBS and EDS is shown in Table 9. These graphs present different scenarios. In benchmark 1, the total reductions made by both RBS and EDS in DORE was greater than in baseline operator. In benchmark 2, the reductions by RBS were greater in DORE but lesser by EDS. Finally, in benchmark 3, the total reductions made by both RBS and EDS was greater in the baseline operator. These different scenarios are directly related to the nature of each benchmark problem. Also, although the RBS reduction table in DORE had more entries than the baseline table, each simplification called by EDS triggered another recursive RBS simplification attempt on each subtree in the individual. Thus, in problems where the number of simplifications by EDS is high - and with an increased average size reduction - the number of reductions by RBS tends to be higher as well.

Even though each benchmark presented a different reduction scenario, a pattern in the ratio between reductions by RBS and reductions by EDS could be noted, as is shown in Fig. 6. In all benchmark problems, except the proportion, the total amount of RBS reductions relative to the total amount of EDS reductions is bigger. It can also be noted that this ratio is greater in DORE than in the baseline operator, which is exactly the main goal of the proposed improvements.

Fig. 6.
figure 6

Average ratio of simplifications made

Full size image

The impact of the dynamically learned rules was analyzed and shown in Fig. 7. The graphs are histograms of the total amount of reductions made by RBS in each subtree size reduction percentage interval. The values highlighted in purple are from rules that could only be learned with DORE improvements. The average size reduction of these dynamically learned rules in each benchmark were 86.01%, 88.01% and 85.40%, respectively, compared to the average size reduction of 59.38%, 55.98% and 52.74% from the predefined rules. DORE not only provided more reductions to be applied but also better ones.

Fig. 7.
figure 7

RBS simplification frequency against size reduction percentage

Full size image

Finally the execution time was analyzed, an important point of interest in this work as DORE execution time should be close to the baseline operator. Average execution time - using the arithmetic mean - for all benchmark problems is shown in Table 10. Since the values of parameters \(\rho \) and \(\gamma \) were defined empirically to make execution time between baseline operator and DORE close, the values presented are expected. However, the difference between Base GP and executions with operators is notable. Using RBS and EDS operators as a single bloat control method seems to be not adequate in hard or complex problems as the execution time tends to increase even more due to all the fitness function evaluation performed recursively in EDS step of the simplification flow.

Table 10. Average total execution time (in seconds) of each strategy, for all benchmark problems
Full size table

6 Conclusion and Future Work

This paper presented improvements to a simplification flow introduced in [11] to make it more reasonable to be applied in complex problems. Besides a meta-learning reduction, two parameters were introduced to optimize the number of trees to be reduced by RBS and EDS in harder problems than the one presented in the original work, without greater impact in execution time.

The resulting operator was able to improve the execution time performance of reduction by RBS, as well as the relative frequency of this reduction in the simplification flow, even with a limit of trees applied. A greater percentage of the population could pass through reduction by RBS with no significant penalty in execution time.

The results of the three artificial SR problems benchmark highlighted how DORE improved the fitness and slightly the tree size in the population during the evolutionary process. The best individuals in these experimental runs had significantly better results in both fitness and tree size using the dynamic operator.

It is noted that DORE is helpful to control bloat in GP but could be challenging to adopt in more complex problems due the computational burden. This issue should be addressed in future research. Probably tuning the parameters \(\rho \) and \(\gamma \) may be needed as well as limiting the size of the subtrees in \(S_{simple}\).

Future work could measure the limits of the list \(S_{simple}\) and a robust strategy to handle random ephemeral constants, which was not explored in this paper. Parallelization of the reductions could improve the operator performance, making it more viable, although is not a trivial problem as the rule table is constantly updated by each EDS reduction performed. It would also be interesting to apply this operator with other bloat control methods, such as a equalization operator [14], using it as a tool to optimize a small set of individuals in a population.