# Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications
and Open Issues

Nan Li  Northeastern UniversityNo.195, Chuangxin RoadShenyangLiaoning
ProvinceChina<EMAIL_ADDRESS>,  Lianbo Ma  Northeastern
UniversityNo.195, Chuangxin RoadShenyang CityLiaoning ProvinceChina
<EMAIL_ADDRESS>,  Guo Yu  East China University of Science and
TechnologyMeilong Road 130ShanghaiChina<EMAIL_ADDRESS>,  Bing Xue
Victoria University of WellingtonWellingtonNew Zealand<EMAIL_ADDRESS>,  Mengjie Zhang  Victoria University of WellingtonWellingtonNew Zealand
<EMAIL_ADDRESS>and  Yaochu Jin  Bielefeld
UniversityBielefeldGermany<EMAIL_ADDRESS>
###### Abstract.

Over recent years, there has been a rapid development of deep learning (DL) in
both industry and academia fields. However, finding the optimal
hyperparameters of a DL model often needs high computational cost and human
expertise. To mitigate the above issue, evolutionary computation (EC) as a
powerful heuristic search approach has shown significant merits in the
automated design of DL models, so-called evolutionary deep learning (EDL).
This paper aims to analyze EDL from the perspective of automated machine
learning (AutoML). Specifically, we firstly illuminate EDL from machine
learning and EC and regard EDL as an optimization problem. According to the DL
pipeline, we systematically introduce EDL methods ranging from feature
engineering, model generation, to model deployment with a new taxonomy (i.e.,
what and how to evolve/optimize), and focus on the discussions of solution
representation and search paradigm in handling the optimization problem by EC.
Finally, key applications, open issues and potentially promising lines of
future research are suggested. This survey has reviewed recent developments of
EDL and offers insightful guidelines for the development of EDL.

deep learning, evolutionary computation, feature engineering, model
generation, model deployment.

††ccs: General and reference Surveys and overviews††ccs: Computing
methodologies Machine learning algorithms††ccs: Theory of computation
Evolutionary algorithms

##  1\. Introduction

Deep learning (DL) as a promising technology has been widely used in a variety
of challenging tasks, such as image analysis (Krizhevsky et al., 2012) and
pattern recognition (LeCun et al., 2015). However, the practitioners of DL
struggle to manually design deep models and find appropriate configurations by
trial and error. An example is given in Fig. 1, where domain knowledge is fed
to DL in different stages like feature engineering (FE) (Xue et al., 2015),
model generation (Zhou et al., 2021a) and model deployment (Choudhary et al.,
2020; Cheng et al., 2017). Unfortunately, the difficulty in the acquisition of
expert knowledge makes DL undergo a great challenge in its development.

In contrast, the automatic design of deep neural networks (DNNs) tends to be
prevalent in recent decades (He et al., 2021; Zhou et al., 2021a). The main
reason lies in the flexibility and computation efficiency of automated machine
learning (AutoML) in FE (Xue et al., 2015), parameter optimization (PO) (Zhang
and Gouza, 2018), hyperparameter optimization (HPO) (Stanley and Miikkulainen,
2002), neural architecture search (NAS) (Yao et al., 2018; He et al., 2021;
Zhou et al., 2021a), and model compression (MC) (Hu et al., 2021b). In this
way, AutoML without manual intervention has attracted great attention and much
progress has been made.

Figure 1.  An overview of DL, driven by domain knowledge or evolutionary
computation, where the life of DL gets through problem, data collection,
feature engineering, model generation and model deployment.

Evolutionary computation (EC) has been widely applied to automatic DL, owing
to its flexibility and automatically evolving mechanism. In EC, a population
of individuals are driven by the environmental selection to evolve towards the
optimal solutions or front (Jin, 2006). Nowadays, there are many automatic DL
methods driven by EC, termed as evolutionary deep learning (EDL) (Telikani et
al., 2021; Evans et al., 2018a; Zhang and Cagnoni, 2020; Zhang, 2018). For
example, a number of studies on EC have been carried out to the feature
engineering (Xue et al., 2015), model generation (Yao et al., 2018; Zhou et
al., 2021a), and model deployments (Choudhary et al., 2020), as shown in Fig.
1. Therefore, the integration of EC and DL has become a hot research topic in
both academic and industrial communities. Moreover, in Fig. 2, the number of
publications and citations referring to EC & DL by years from Web of Science
gradually increases until around 2012, whereas it sharply rises in the
following decade. Hence, more and more researchers work on the area of EDL.

Figure 2. Total publications and citations referring to EC & DL by years from Web of Science until July 2022. Table 1. Comparison between existing surveys and our work, where FE, PO, HPO, NAS, and MC indicate feature engineering, parameter optimization, hyperparameter optimization, neural architecture search, and model compression, respectively. “✓” and “-” indicate the content is included or not in the paper, respectively. Survey |  Type |  FE |  PO |  HPO |  NAS |  MC
---|---|---|---|---|---|---
(Yao et al., 2018) |  AutoML |  ✓ |  - |  ✓ |  ✓ |  \- [t]
(He et al., 2021) |  AutoML |  ✓ |  - |  ✓ |  ✓ |  -
(Yao, 1993) |  NAS |  - |  ✓ |  ✓ |  ✓ |  -
(Ren et al., 2021) |  NAS |  - |  - |  ✓ |  ✓ |  -
(Jaâfra et al., 2019) |  NAS |  - |  - |  ✓ |  ✓ |  -
(Santra et al., 2021) |  NAS |  - |  - |  ✓ |  ✓ |  -
(Telikani et al., 2021) |  EDL |  - |  - |  - |  ✓ |  -
(Liu et al., 2021b) |  EDL |  - |  ✓ |  ✓ |  ✓ |  -
(Zhou et al., 2021a) |  EDL |  - |  ✓ |  ✓ |  ✓ |  -
(Xue et al., 2015) |  EDL |  ✓ |  - |  - |  - |  -
(Al-Sahaf et al., 2019) |  EDL |  ✓ |  ✓ |  - |  - |  -
(Zhang et al., 2011) |  EDL |  - |  ✓ |  ✓ |  ✓ |  -
(Alexandropoulos and Aridas, 2019) |  EDL |  ✓ |  - |  ✓ |  ✓ |  -
(Mirjalili et al., 2019) |  EDL |  ✓ |  ✓ |  ✓ |  ✓ |  -
(Darwish et al., 2020) |  EDL |  - |  ✓ |  ✓ |  ✓ |  -
(Freitas, 2003) |  EDL |  - |  ✓ |  ✓ |  ✓ |  -
Ours |  EDL |  ✓ |  ✓ |  ✓ |  ✓ |  ✓[b]

In Table 1, we have listed recent surveys on automatic DL. A large number of
surveies concentrate on the optimization of DL models (He et al., 2021;
Telikani et al., 2021; Zhou et al., 2021a; Xue et al., 2015), or NAS (Liu et
al., 2021b; Yao, 1993). Many others focus on specific optimization paradigms
such as reinforcement learning (RL) (Jaâfra et al., 2019), EC (Sun et al.,
2020) and gradient (Santra et al., 2021). However, very few of them have
systematically analysed EDL and runs the gamut of FE, PO, HPO, NAS, and MC. To
fill the gap, we aim to give a comprehensive review of EDL in detail. The main
contributions of this work are as follows.

  * •

Existing work on EDL is reviewed from the perspective of DL and EC to
facilitate the understanding of readers from the communities of both ML and
EC, and we also formulated EDL into an optimization problem from the
perspective of EC.

  * •

The survey describes and discusses on EDL in terms of feature engineering,
model generation, and model deployment from a novel taxonomy, where the
solution representation and the search paradigms are emphasized and
systematically discussed. To the best of our knowledge, few survey has
investigated the evolutionary model deployment.

  * •

On the basis of the comprehensive review of EDL approaches, a number of
applications, open issues and trends of EDL are discussed, which will guide
the development of EDL.

The rest of this paper is organized as follows. Section 2 presents an overview
of EDL. In Section 3, EC-driven feature engineering is presented. EC-driven
model generation is discussed in Section 4. Section 5 reviews EC-driven model
compressions. After that, relevant applications, open issues and the trends of
EDL are discussed in Section 6. Finally, a conclusion of the paper is drawn in
Section 7.

##  2\. An Overview of Evolutionary Deep Learning

###  2.1. Deep Learning

DL can be described as a triplet $M$ = ($D$, $T$, $P$) (Yao et al., 2018),
where $D$ is the dataset used for the training of a deep model ($M$), and $T$
is the targeted task. $P$ indicates the performance of $M$. The aim of DL is
to boost its performance over specific task $T$, which is measured by $P$ on
dataset $D$. In Fig. 1, we can see there are three fundamental processes of
DL, i.e., feature engineering, model generation and model deployment.

Feature engineering: It aims to find a high-quality $D$ to improve the
performance ($P$) of the deep model ($M$) on specific tasks ($T$). In
practice, the feature space of $D$ may include redundant and noisy
information, which harms the performance ($P$) of the model ($M$). On Prostate
dataset, the size of feature subset (65) selected in (Tran et al., 2018) is
only 1% of the total size of features (10509).

Model generation: It targets at optimizing/generating a model ($M$) with
desirable performance ($P$) for specific task ($T$) on the given datasets
($D$) (He et al., 2021). Model generation can be further divided into
parameter optimization, model architecture optimization, and joint
optimization (Zhou et al., 2021a). Parameter optimization is to search the
best parameters (e.g., weights) for a predefined model. Architecture
optimization is dedicated to finding the optimal network topology (e.g.,
number of layers and types of operations) of a deep model ($M$) (Luo et al.,
2018). Joint optimization involves in the above two optimization issues by
automatically searching for a powerful model ($M$) on the datasets ($D$)
(Miikkulainen et al., 2019).

Model deployment: This process aims to deploy a deep model ($M$) to solve a
deployment task $T$ with acceptable performance ($P$) on input data ($D$)
within limited computational budgets. The key issue of model deployment is how
to reduce the latency, storage, and energy consumption when the number of
parameters of a deep model is large, e.g., Transformer-XL Large has 257M
parameters (Dowdell and Zhang, 2020).

###  2.2. Evolutionary Computation

EC is a collection of stochastic population-based search methods inspired by
evolution mechanisms such as natural selection and genetics, which does not
need gradient information and is able to handle a black-box optimization
problem without explicit mathematical formulations (Ma et al., 2022; Vargas-
Hákim et al., 2022). Owing to the above characteristics, EC has been widely
employed to the automatic design of DL.

Figure 3. A general framework of EC.

In principle, we can broadly divide EC methods into two categories:
evolutionary algorithms (EA) and swarm intelligence (SI) (Zhou et al., 2021a).
Our work doesn’t make an explicit distinction between EAs and SI since they
comply with a general framework, as shown in Fig. 3, which consists of three
main components.

Initialization:

is performed to generate a population of individuals which are encoded
according to the decision space (or search space and variable space) of the
optimization problem, such as the feature set, model parameters and
topological structure.

Evaluation:

aims to calculate the fitness of individuals. In fact, the evaluation of the
individuals in EDL is a computationally expensive task (Miahi et al., 2022).
For example, the work (Real et al., 2017) used 3000 GPU days to find a
desirable architecture.

Updating:

aims to generates a number of offspring solutions through various reproduction
operations. For example, a new soultion is generated via velocity and position
formula in particle swarm optimization (PSO) (Tran et al., 2018). In terms of
genetic algorithm (GA), some reproduction operators (e.g., crossover and
mutation) are used to generate new individuals (Vafaie and De Jong, 1998).

###  2.3. Evolutionary Deep Learning

####  2.3.1. EDL from two perspectives

In contrast to traditional DL which heavily relies on expert or domain
knowledge to build deep model, EDL is to automatically design the deep model
through an evolutionary process (Sun et al., 2020; Yao, 1993; Ren et al.,
2021; Zhang, 2018).

From the perspective of DL: Traditional DL needs a lot of expert knowledge in
inventing and analysing a learning tool to a specific dataset or task. In
contrast, EDL can be seen as a human-friendly learning tool that can
automatically find appropriate deep models on given datasets or tasks (Yao et
al., 2018). In other words, EDL concentrates on how easy a learning tool can
be used.

From the perspective of EC: The configurations of a model is represented as an
individual, and the performance as the objective to be optimized. EC plays an
important role in the optimization driven by evolutionary mechanisms. Namely,
EDL can be seen as an evolutionary optimization process to find the optimal
configurations of the deep model with high performance.

From the above analysis, EDL not only aims to increase the adaptability of a
deep model towards learning tasks via the automatic construction approach
(from the perspective of DL), but also tries to achieve the optimal model
under the designed objectives or constraints (from the perspective of EC).

####  2.3.2. Definition and Framework of EDL

According to the above discussion in Subsection 2.3.1 and following (Yao et
al., 2018), we can define EDL as follows.

(1)
$\begin{array}[]{l}\begin{array}[]{*{20}{c}}{\mathop{{\rm{Max}}}\limits_{config.}}&{{\textrm{Learning
tools' performance,}}}\end{array}\\\
\begin{array}[]{*{20}{c}}{s.t.}&{\left\\{{\begin{array}[]{*{20}{c}}{{\textrm{No
assistance from humans}}}\\\ {{\textrm{Limited computational
budgets.}}}\end{array}}\right.}\end{array}\end{array}$

where $config.$ indicates the configurations which form the decision space of
an optimization problem. The problem is to maximize the objective (i.e.,
learning tools’ performance $P$) of tasks $T$ on datasets $D$ under the
constraints of no assistance from humans and limited computational resources.
Accordingly, three aspects are taken into account in the design of EDL.

Desirable generalization performance::

EDL should have desirable generalization performance across given datasets and
tasks.

High search efficiency::

EDL is able to find optimal or desirable configuration within a limited
computational budges (e.g., hardware, latency, energy consumption) under
different designed objectives (e.g., high accuracy, small model size).

Without human assistance::

EDL is able to automatically configure without human intervention.

Following the EC framework described in Fig. 3, we present a general framework
of EDL as follows.

Step 1 Initialization::

A population of individuals are initialized according to the designed encoding
scheme.

Step 2 Evaluation::

Each individual is evaluated according to the objectives (e.g., high accuracy,
small model size) or constraints (e.g., energy consumption).

Step 3 Updating::

A required number of new solutions are generated from previous generation via
various updating operations.

Step 4 Termination condition::

Go to Step 2 if the predefined termination condition is unsatisfied;
Otherwise, go to Step 5.

Step 5 Output::

Output the solution with the best performance.

####  2.3.3. Taxonomy of EDL Approaches

In this section, a novel taxonomy of EDL approaches is proposed according to
“what to evolve/optimize” and “how to evolve/optimize”, as shown in Fig. 4.

Figure 4. A taxonomy of EDL approaches.

“What to evolve/optimize”: We may be concerned about “what EDL can do” or
“what kinds of problems EDL can tackle”. In feature engineering, there are
three key issues to be resolved, including the feature selection, feature
construction and feature extraction (Yao et al., 2018). In model generation,
parameter optimization, architecture optimization, and joint optimization
become the critical issues (Zhou et al., 2021a), while model deployment is
involved with the issues of model pruning and other compression technologies.

“How to evolve/optimize”: The answer to the question is designing appropriate
solution representation and search paradigm for EC, and acceleration
strategies for NAS. The representation schemes are designed for the encoding
of individuals, search paradigms for the achievement of optimal
configurations, acceleration strategies for the reduction of time or resources
consumption.

According to the above taxonomy, we will elaborately introduce EDL in feature
engineering, model generation and model deployment in Sections 3, 4 and 5,
respectively.

##  3\. Feature Engineering

Feature engineering is adopted to pre-process given raw data by filtering out
the irrelevant features of the data or creating the new features based on
original features (Xue et al., 2015). Various EC-based techniques have been
proposed to reduce data dimensionality, speed up learning process, or improve
model performance (Xue et al., 2015). The common techniques can be categorized
into feature selection (Nguyen et al., 2014), feature construction (Bhanu and
Krawiec, 2002) and feature extraction (Peng et al., 2021).

###  3.1. Feature Selection

####  3.1.1. Problem Formulation

Feature selection aims to automatically select a representative subset of
features where there are no irrelevant or redundant features. However, the
search space grows exponentially with the increase of features. If a dataset
has $n$ features, then there are $2^{n}$ solutions in the search space. In
addition, the interactions between features may seriously impact the feature
selection performance (Xue et al., 2015). In the followings, we will review
existing work on solution representations and search paradigms in EC for
feature selection.

####  3.1.2. Solution Representations

Generally, there are three different categories of solution representations.

Linear encoding: This encoding uses vectors or strings to store feature
information. For example, in (Estévez and Caballero, 1998), a fixed-length
binary vector was used to express whether a feature is selected or not, where
“1” indicates a corresponding feature is selected, and “0” is the opposite. In
(Hong and Cho, 2006), a binary index was used to indicate the corresponding
feature.

Tree-based encoding: In canonical genetic programming (GP), all leaf
nodes/terminal nodes represent the selected features and non-terminal nodes
represent functions (e.g., arithmetic or logic operators) (Krawiec, 2002). For
automatic classification on high-dimensional data, Krawiec et al. (Krawiec,
2002) proposed a tree-based encoding to select a subset of highly
discriminative features, where each feature consisted of sibling leaf nodes
and their paternal function node. On the basis of the tree-based encoding,
Muni et al. (Muni et al., 2006) proposed a multi-tree GP mothed for online
feature selection.

Graph-based encoding: In (O’Boyle and Palmer, 2008), the feature space of the
high-dimensional data is represented by a graph and each node of the graph
represents a feature. A feature subset is composed of visited nodes of the
graph, i.e., the path of node composition or subgraph. Yu et al. (Yu et al.,
2009) converted feature selection to the optimal path problem in a directed
graph, where the value of the node was “1” or “0” to indicate whether the
feature was selected or not.

####  3.1.3. Search Paradigms

In feature selection, representative types of search paradigms are introduced
as follows.

Basic EC search paradigm: In feature selection, typical evolutionary search
methods have been widely used, such as GA (Da Silva and Neto, 2011; Estévez
and Caballero, 1998), GP (Krawiec, 2002; Nekrasov et al., 2020), PSO (Vieira
et al., 2013; Nguyen et al., 2014), ant colony optimization (ACO) (Khushaba et
al., 2008; Ma et al., 2021c), and artificial bee colony (ABC) (Wang et al.,
2020d). Besides, some other studies (Khushaba et al., 2008) combined ACO with
DE to seek optimal feature subsets, where the solutions searched by the ACO
were fed into the DE to further explore the optimal solution. In (Da Silva and
Neto, 2011), a family of feature selection methods based on different variants
of GA were developed to improve the accuracy of content-based image retrieval
systems.

Co-evolution search paradigm: In co-evolution search paradigm for feature
selection, at least two populations are simultaneously evolved and interacted
toward the optimal subset of features (Wen and Xu, 2011; Rashid et al., 2020).
For example, a divide-and-conquer strategy was developed in (Wen and Xu, 2011)
to manage two subpopulations. One subpopulation was to conduct an evolution
process of classifier design, while the other one was to search for an optimal
subset of features.

Multi-objective search paradigm: This type of search paradigms are driven by
two or more conflicting objectives (Xue et al., 2012; Hancer et al., 2015;
Cheng et al., 2021), such as the maximization of the accuracy of a classifier
and minimization of the size of a feature subset. On the basis of the above
two conflicting objectives, Xue et al. (Xue et al., 2012) designed a multi-
objective PSO algorithm for feature selection and obtained a set of Pareto
non-dominated candidate solutions for feature selection after the multi-
objective search.

####  3.1.4. Summary

GA and GP are widely applied to feature selection. GA early serves for low-
dimensional (i.e., $\leq$1000) datasets (Xue et al., 2015; Hosni et al.,
2020). Recently, many GA-based approaches have been proposed to solve high-
dimensional feature selection (Cheng et al., 2021). Nevertheless, GP is
commonly applied to large-scale/high-dimensional feature selection since it is
flexible in feature representation (Xue et al., 2015). Especially, GP
outperforms GA on some small but high-dimensional datasets, e.g., Brain
Tumor-2 (Chen et al., 2022) with 10367 features but only 50 samples. In
addition, PSO has been proved with faster convergence rate to an optimal
feature subset than GAs and GP (Zhang et al., 2017). The graph representation
of ACO outperforms GA and GP on flexibility, but the challenge of ACO is how
to design appropriate graph encoding for large-scale scenarios (Telikani et
al., 2021; Xue et al., 2015).

###  3.2. Feature Construction

####  3.2.1. Problem Formulation

Feature construction is to create new high-level features from the original
features (Tran et al., 2016) via appropriate function operators (e.g.,
conjunction and average) (Xue et al., 2013; Neshatian et al., 2012), so that
the high-level features are more easily discriminative than the original ones.
Feature construction is a complicated combinatorial optimization problem,
where search space increases exponentially along with the total number of
original features and the function operators. In the following subsections, we
will describe the EC-based feature construction methods in terms of both
solution representations and search paradigms.

####  3.2.2. Solution Representations

Existing EC-based approaches for feature construction can be categorized into
three groups.

Linear encoding: The study (Xue et al., 2013) used $n$-bit ($n$ is the total
number of original features) binary vector to represent each particle, where
“0” indicated the corresponding feature not applied to build the new high-
level feature while “1” was in the opposite. On the basis of the encoding, a
local search was performed to select candidate operators from a predefined
function set to construct a new high-level feature.

Tree-based encoding: Tree-based encoding is natural for feature construction,
where leaf nodes represent the feature information and internal nodes
represent operators. Many studies (Tran et al., 2016; Bhanu and Krawiec, 2002)
have demonstrated the effectiveness of tree encoding in feature construction.
For example, Bhanu et al. (Bhanu and Krawiec, 2002) designed a GP-based
coevolutionary feature construction procedure to improve the discriminative
ability of classifiers. In (Tran et al., 2016), an individual in EC was
represented by a multi-tree encoding with multiple high-level features.

Graph-based encoding: In this encoding, the nodes and edges represent features
and operators (e.g., “+”, “-”, “*”, “/”), respectively. Teller et al. (Teller
and Veloso, 1996) applied an arbitrary directed graph to represent all
features and operators, where each possible high-level feature can be
represented as a subgraph of this directed graph. For linear GP, features and
operations form a many-to-many directed acyclic graph, in which each feature
is loaded into predefined registers and register’s value can be used in
multiple operators (Fogelberg and Zhang, 2005). However, graph encoding
becomes inefficient on high-dimensional feature sets since the complexity of
graph traversal exacerbates the difficulty of feature construction.

####  3.2.3. Search Paradigms

There are four categories of search paradigms for feature construction in
existing work .

Basic EC search paradigm: Existing studies include but are not limited to GA
(Vafaie and De Jong, 1998), and GP (Tran et al., 2016; Neshatian et al., 2007;
Tariq et al., 2018). For example, the work (Neshatian et al., 2007) designed
GP-based feature construction to reduce the feature (input) dimensions of a
classifier. Especially, GP has been also widely used to construct new
features, where each individual following the form of a GP tree usually
represents a constructed high-level feature (García et al., 2011).

Co-evolution search paradigm: It can be decomposed to feature construction
subproblem and classifier design subproblem, and each subproblem is solved
with a standalone subpopulation by an EC-based method (Bhanu and Krawiec,
2002; Roberts and Claridge, 2005). For example, the study (Roberts and
Claridge, 2005) decomposed feature construction into two subproblems (i.e.,
feature construction, and object detection), where the feature construction
was solved by evolving a population of pixel (i.e., feature) and the object
detection was optimized using object detection algorithm (ODA) (Roberts and
Claridge, 2005).

Multi-features construction search paradigm: Unlike early methods (Tran et
al., 2016; Vafaie and De Jong, 1998; Shafti and Pérez, 2008; Tran et al.,
2019) only constructing one high-level feature in a single search process,
this sort of paradigms are able to create multiple high-level features. For
example, Ahmed et al. (Ahmed et al., 2014) employed Fisher criterion together
with $p$-value measure as the discriminant information between classes, based
on which multiple features were constructed through multiple GP trees.

Multi-objective evolutionary search paradigm: In this search paradigm, the
number of features and classification accuracy are commonly taken into account
as the objective functions for multi-objective evolutionary optimization
(Hammami et al., 2018; Castelli et al., 2011). Especially, Hammami et al.
(Hammami et al., 2018) constructed a set of high-level features by optimizing
a multi-objective optimization problem (MOP) with three objectives (i.e., the
number of features, the mutual information, and classification accuracy) with
Pareto dominance relationship.

####  3.2.4. Summary

GP-based approaches are popular in feature construction due to the flexible
representation of features and operations. In addition, the hybrid of
evolutionary algorithms also attracts much attention for feature construction.
However, there is still plenty of room for the improvement of efficiency in
constructing features in high-dimensional or large-scale scenarios, where a
large number of computational resources are needed (Tariq et al., 2018; Tran
et al., 2019). Notably, feature construction often requires more computational
overhead than feature selection, since feature construction commonly performs
after the feature selection and the quality of the selected features may
influence the performance of feature construction.

###  3.3. Feature Extraction

####  3.3.1. Problem Formulation

Feature extraction is to reduce the feature dimensions by altering the
original features/data via some transformation functions (He et al., 2021).
Traditional extractors include principal component analysis (PCA) (Abdi and
Williams, 2010) and linear discriminant analysis (LDA) (Izenman, 2013).
However, they cannot keep somewhat important information after the
transformation (Abdi and Williams, 2010) and it is tedious to tune their
hyperparameters (e.g., number of retained features) to find the best
extraction. Thus, automatically finding high-quality map functions by EC-based
approaches to achieve informative feature set tends to be popular.

####  3.3.2. Solution Representations

There are two typical ways for solution representation in EC-driven feature
extraction.

Linear encoding: In this encoding, map functions (Refahi et al., 2020;
Albukhanajer et al., 2015) or function parameters (Zhao et al., 2006) are
encoded as a linear format. For example, Wissam et al. (Albukhanajer et al.,
2015) predefined three sets of track functions (i.e., trace functions,
diametric functions, and circus functions) for feature extraction, and the
optimal combination between the functions were obtained by an EC-based method.
In (Zhao et al., 2006), the hyperparameters of map functions were encoded by
some linear vectors which were constructed by a number of optimal projection
basis vectors obtained via EC.

Tree-based encoding: In tree-based encoding, leaf nodes represent original
features or constants, while the non-leaf nodes are some operators for feature
extraction including common arithmetic, logical operators (i.e., “+”, “/”,
“$\cup$”) or other transformation operators (e.g., uLBP, and SobelY). In EC-
driven feature extraction, an individual represents a feature extractor or map
function (Peng et al., 2021; Zhang and Rockett, 2011). Especially, an EC-based
framework was developed in (Zhang and Rockett, 2011) to search for features
and sequences of operations by use of tree-based encoding.

####  3.3.3. Search Paradigms

In this section, some common search paradigms for feature extraction are
introduced.

Basic EC search paradigm: EC has been successfully utilized in various feature
extraction tasks (Z.-Flores et al., 2020; Bi et al., 2018). For example, Zhao
et al. (Zhao et al., 2007) introduced bagging concept to an evolutionary
algorithm for feature extraction. The work in (Zhao et al., 2009) developed an
evolutionary discriminant feature extraction (EDFE) algorithm by combining GA
with subspace analysis, which can reduce the complexity of the search space
and improve the classification performance.

Co-evolution search paradigm: In feature extraction, finding the optimal
extractor is an optimization problem, which can be decomposed into a series of
subproblems (Kotani and Kato, 2004; Hajati et al., 2010). For example, Hajati
et al. (Hajati et al., 2010) proposed a co-evolutionary method for feature
extraction. Specifically, a subpopulation was evolved to optimize the
classifier-related subproblem (i.e., classifier construction), and the other
subpopulation made use of genetic information from the first population for
the optimization of a feature-related subproblem (i.e., feature extraction).

Multi-objective search paradigm: In multi-objective feature extraction, the
model accuracy, computational time, complexity, and robustness are often taken
into account as the objectives (Zhang and Rockett, 2011; Cano et al., 2017).
Cano et al. (Cano et al., 2017) proposed a Pareto-based multi-objective GP
algorithm for feature extraction and data visualization, where the objectives
were to minimize the complexity of data transformation (i.e., tree size) and
maximize the recognition performance (i.e., accuracy).

####  3.3.4. Summary

In existing studies, many efficient searching and balancing strategies, driven
by EC approaches to achieve satisfactory solutions at significantly-reduced
computation overheads, have been developed in recent years (Zhang and Rockett,
2011; Cano et al., 2017; Mauceri et al., 2021; Shakya et al., 2021). However,
the performance of extractors may be limited with existing encoding methods
and predefined operation sets. Therefore, it is essential to develop efficient
algorithms, operation control strategies and representation for high-
dimensional feature extraction.

##  4\. Model Generation

Model generation is to search for optimal models with desirable learning
capability on given tasks (Yao et al., 2018; He et al., 2021). In this
section, we introduce corresponding evolutionary parameter optimization,
architecture optimization, and joint optimization from solution representation
to search paradigms. Readers interested in other model generation approaches
(e.g., RL-based and gradient-based approaches) can refer to the reviews (Ren
et al., 2021; Jaâfra et al., 2019).

###  4.1. Model Parameter Optimization

####  4.1.1. Problem Formulation

Model parameter optimization targets at searching for the best parameter set
(i.e., weights $W^{*}$) for a predefined architecture ($A$). The loss function
$L$ (e.g., the cross-entropy loss function) measures the performance of the
model with optimized parameters (i.e., $W$ in Eq. 2) on given datasets. The
general model parameter optimization can be formulated as

(2)
$\displaystyle\begin{matrix}{{W}^{*}}$=$\underset{W}{\arg\min}L\left(W,A\right)\end{matrix}$

where $W$ is usually large-scale (millions of model parameters) and highly
non-convex.

####  4.1.2. Solution Representations

There are two typical EC-based representation schemes for model parameter
optimization, including direct encoding and indirect encoding (He et al.,
2021).

Direct encoding: The model parameters are directly represented via a vector or
matrix, in which each element represents a specific parameter (Ijjina and
Chalavadi, 2016; Karegowda and Manjunath, 2011). For example, a chromosome
with 64 real numbers was used to directly represent the network corresponding
weights, where the first 63 real numbers were used to encode three convolution
masks of size 1 $\times$ 21\. The last real number was the random seed of a
generator for the initialization of a fully connected network (Ijjina and
Chalavadi, 2016). This encoding approach may require a huge computational
overhead to represent and optimize the large-scale weights.

Indirect encoding: This encoding approach represents only a subset of the
model parameters via a deterministic transformation (Li et al., 2019; Koutník
et al., 2010). In (Koutník et al., 2010), the weight information was encoded
as a set of Fourier coefficients in the frequency domain to reduce
dimensionality of representation by ignoring high-frequency coefficients.
Although this method is able to speed up the search process, the loss of
parameter the information may occur due to the incomplete information
representation, which maydegrades the model performance.

####  4.1.3. Search Paradigms

EC-based methods for model parameter optimization can be divided into two
categories according to whether or not method combines with the gradient
approach, i.e., pure EC and gradient-based EC.

Pure EC paradigms optimize model parameters only via evolutionary search,
including the basic EC search paradigm and co-evolution search paradigm.

  * •

Basic EC search paradigm: In addition to GA (Montana and Davis, 1989;
Karegowda and Manjunath, 2011), some heuristic algorithms like PSO (Al-kazemi
and Mohan, 2002), ABC (Karaboga et al., 2007) and ACO (Socha and Blum, 2007)
are also commonly utilized for model parameter optimization. For example,
Karaboga et al. (Karaboga et al., 2007) adopted ABC to find a set of weights
for a feed-forward neural network (FNN) on targeted tasks.

  * •

Co-evolution search paradigm: Co-evolution search is conducted on the
subproblems of the original optimization problem (e.g., synapse-based and
neuron-based problems (Chandra and Zhang, 2012; Chandra, 2015)). For example,
Chandra et al. (Chandra and Zhang, 2012) regarded a single hidden layer as a
subcomponent in the initialization phase, which will be merged with the
individuals with the best fitness from different sub-populations to constitute
new neural networks during the co-evolution optimization process.

(a)

(b)

(c)

Figure 5.  Three hybrid ways of gradient-based ECs.

Gradient-based EC combine basic EC with the gradient-based method to enhance
the exploitation ability in optimizing model parameters. According to the
execution order, there are three hybrid ways.

  * •

The first hybridization approach is shown in Fig. (5(a)), where the EC is used
to identify the optimal parameters for model, then the parameters are further
optimized using gradient-based method to find the final optimal solution
(Zhang and Smart, 2004; Chen et al., 2015). For example, a genetic adaptive
momentum estimation algorithm (GADAM) was proposed in (Zhang and Gouza, 2018)
by incorporating Adam and GA into a unified learning scheme, where Adam was an
adaptive moment estimation method with first-order gradient.

  * •

The second hybridization approach is given in Fig. (5(b)), where the gradient-
based method is used to produce a set of parameters for the initialization of
the population used in EC (Wu et al., 2021b). For example, the study (Khadka
and Tumer, 2018) firstly trained a RL agent through a gradient-based method,
then the parameters of the RL were used as the initial population to feed the
EC. As a result, the parameters will be further optimized by the EC.

  * •

The third approach is presented in Fig. (5(c)), which iteratively applies EC
and gradient-based method during the optimization to find the optimal
parameters. Following this framework, when the method is used and which method
is chosen are varying in different studies (Yang et al., 2021; Cui et al.,
2018).

####  4.1.4. Summary

In model parameter optimization, direct encoding is straightforward and able
to keep more information than the indirect encoding. Compared to gradient-
based methods easily trapped into local optima, EC shows more powerful ability
in global search. Here, several scenarios are introduced as follows, where EC
is applied to model parameter optimization.

Small-scale scenario: (Montana and Davis, 1989) shows that pure EC approaches
outperform gradient-based methods in search effectiveness on some small-scale
problems, where the models are with small numbers of parameters or simple
architectures (e.g., FNN).

Large-scale scenario: The performance of pure EC approaches might not be
promising in large-scale learning models, while a better way is to utilize the
hybridization of EC and gradient-based methods. Such hybrid methods can
alleviate the issue of getting trapped in local optima and increase the
effectiveness of subsequent exploitation (Yang et al., 2021).

In addition to above scenarios, EC-based method can be used to train the DNN,
when the exact gradient information of the loss function is difficult to be
acquired (Peng et al., 2018). For example, the rewards of policy network are
sparse or deceptive in deep reinforcement learning (DRL) so that the gradient
information is unattainable. The work (Conti et al., 2018) introduced the
novelty search (NS) and the quality diversity (QD) to the evolution strategies
(ES) for the policy network.

###  4.2. Model Architecture Optimization

####  4.2.1. Problem Formulation

Model architecture optimization, also termed as NAS, is to search promising
network architectures with good performance such as model accuracy on given
tasks. The model architecture optimization can be formulated as follows.

(3)
$\left\\{\begin{matrix}{{A}^{*}}$=$\underset{W,A}{\mathop{\arg\min}}\,L\left(W,A\right)\\\
\begin{matrix}s.t.&A\in\mathcal{A}\\\ \end{matrix}\\\ \end{matrix}\right.$

where $A^{*}$ indicates the architecture from the search space ($\mathcal{A}$)
with the best performance under the parameters $W$, and $L$ is used to measure
the performance of architectures on given tasks. Thereby, this optimization is
a bi-level optimization problem (Zhou et al., 2021a; Liu et al., 2021b), where
the model architecture optimization is subject to the model parameter
optimization (Lu et al., 2021). Since the current NAS works are mainly focused
on CNN, we will discuss the solution representations, the search paradigms,
and acceleration strategies of CNN. Due to the page limit, the design of the
search space of NAS are not introduced here, but interested readers can check
these surveys (Liu et al., 2021b; Ren et al., 2021) which have details about
search space design.

####  4.2.2. Solution Representations

According to varying lengths of encodings, we can classify the encoding
strategies into fixed-length and variable-length encodings.

Fixed-length encoding: The length of each individual is fixed during the
evolution. For example, a fixed-length vector is designed to represent the
model architecture of CNN (Xie and Yuille, 2017), where a subset of elements
in the vector represents an architectural units (e.g., convolutional, pooling
or fully-connected layer) of a CNN. Such encoding may be easily adapted to
evolutionary operations (e.g., crossover and mutation) of EC (Xie and Yuille,
2017), but it has to specify an appropriate maximal length, which is usually
unknown in advance and needs to predefine based on domain expertise.

Variable-length encoding: Different from the fixed-length approach, the
variable-length encoding strategy does not require a prior knowledge about the
optimal depth of model architecture and actually could be a way to reduce the
complexity of the search space. The flexible design of this encoding may
encode more detailed information about the architecture into a solution
vector, and the optimal length of the solution is automatically found during
the search process (Liu et al., 2021b). In (Chen et al., 2020), the entire
variational autoencoder (VAE) was divided into four blocks, including h-block,
$\mu$-block, $\sigma$-block and t-block, while the variable-length chromosomes
consisted of different quantities and types of layers. Notably, variable-
length encoding it is not straightforward to apply standard genetic operators
(e.g., crossover).

Since the neural network architectures are composed of basic units and
connections between them, so that both of them are to be encoded, as suggested
in (Liu et al., 2021b).

1) Encoding hyperparameters of basic units. In CNNs, there are many
hyperparameters to be specified for each unit (e.g., layer, block or cell),
such as feature map size, type of convolution layer, and filter size (Sun et
al., 2020). In (Sun et al., 2019b), DenseBlock only had to set two
hyperparameters (e.g., block type and specific parameter of internal unit) to
configure the block can be seen as a microcosm of a complete CNN model. The
parameterization of a cell is more flexible than that of a block since it can
be configured via a combination of different primitive layers (Sun et al.,
2019c).

2) Encoding connections between units. In general, there are two kinds of
model architectures according to the connection patterns of basic units:
linear topological architectures and non-linear topological architectures
(Yang et al., 2020). The linear pattern of architecture consists of sequential
basic units, and the non-linear pattern allows for skip or loop connections in
the architecture (Liu et al., 2021b).

  * •

Linear topological architecture: The linear topology widely appears in the
construction of layer-wise and block-wise search spaces. Due to the simplicity
of linear topology, basic units can be stacked one by one by a linear piecing
method. In this way, the skeleton of an architecture can be built up
effectively (Sun et al., 2019b; Chen et al., 2020) regardless of the
complexity of the internal of basic units.

  * •

Non-linear topological architecture: Compared to the linear architecture, the
non-linear topological architecture receives much more attention due to its
flexibility to construct well-performing architectures (Wang et al., 2021c,
2020b; Xie and Yuille, 2017), such as macro structures composed of basic
units, and micro structures within basic units. There are two typical encoding
approaches for non-linear topological architectures. The one is to use
adjacent matrix to represent the connections in non-linear architectures,
where “1” of the matrix denotes the existence of the connection between two
units and “0” goes the opposite. In (Lorenzo and Nalepa, 2018), skip
connections are represented by a matrix where constraints can be set in place
to guarantee valid encoding and avoid recurrent edges while performing skip
connections. Note that adjacent matrix has a limitation that the number of
basic units needs to be fixed in advance (Kitano, 1990). Another one is to
utilize an ordered pair to represent a directed acyclic graph, and then encode
the connections between unites. The ordered pair can be formulated as $G$ =
($V$, $E$) where $V$ is a set of vertices and $E$ is a directed edge in the
acyclic graph, and it has been applied in (Irwin-Harris et al., 2019) to
encode the connections.

####  4.2.3. Search Paradigms

In this section, the commonly used EC-based search paradigms for NAS are
introduced.

Basic EC search paradigm: Many basic EC algorithms have been widely applied in
existing NAS methods, such as GA (Kitano, 1990) and PSO (Sun et al., 2019d). A
general framework of EC is presented in Fig. 3.

Incremental search paradigm: A model architecture can be built in an
incremental way where model elements (e.g., layers and connections) are
gradually added to the model during the evolutionary process (Liu et al.,
2018c; Wang et al., 2020c; Shen et al., 2019). This way allows to find parts
of architecture at different optimization stages, which reduces the
computational burden on acquiring a complete model at once (Liu et al.,
2018c). For example, Wang et al. (Wang et al., 2020c) used an incremental
approach to stack blocks for building architectures, which improved the
capacity of the final architecture via a progressive process.

Co-evolution search paradigm: An architecture optimization problem is
decomposed into the optimizations of a blueprint and its components (O’Neill
et al., 2018; Zhang et al., 2022b). Specifically, the blueprint plays a role
in specifying the topological connection patterns of its components, and an
optimal architecture is acquired by cooperatively optimizing the blueprint and
its components. For example, O’Neill et al. (O’Neill et al., 2018) proposed a
co-evolution search paradigm for NAS, where the candidate blueprints and
components were sampled from two populations, and then combined to form new
architectures.

Multi-objective search paradigm: This paradigm targets at searching for a set
of Pareto optimal architectures based on multiple criteria, and finding the
final solutions according to some practical considerations, such as
computational environment (Neshat et al., 2020; Lu et al., 2019). This
paradigm becomes popular in practical applications, since many objectives are
required to be considered such as the accuracy, inference time, model size,
and energy consumption. In (Neshat et al., 2020), NSGA-II and RL were used to
explore model architectures with respect to the model accuracy, and model
complexity (e.g., the number of model parameters and multiply-adds operators).

####  4.2.4. Acceleration Strategies

NAS is a high computational overhead task, mainly due to the large search
space and highly time-consuming evaluation (Yao et al., 2018). To overcome
this challenge, various acceleration strategies (Sun et al., 2019d; Assunção
et al., 2019b) have been developed to accelerate the optimization. In this
section, we summarize the speed-up strategies from the aspects of algorithm
design to the hardware implementation, as shown in Fig. 6.

Figure 6. Overview of acceleration strategies.

From the algorithm design point of view, we summarized a number of
acceleration strategies from population initialization to evaluations.

  * •

Initialization:

    * –

Reduced population: The simplest way of acceleration during the initialization
stage is to set the population with a small size. In other words, less
evaluations are required with a smaller size of population since the
evaluation of a candidate architecture is time-consuming (Yao et al., 2018).
As a result, some studies (Assunção et al., 2019b, 2018) use small population
with fixed size to speed up their evolution, like CARS (size = 32) (Yang et
al., 2020). In contrast, some other studies use dynamic sizes of populations
during the optimization. In (Fan et al., 2020), the population size is
dynamically changed to reach a balance between algorithmic efficiency and
population diversity.

    * –

Efficient search space: Another way is to design efficient search space to
speed up the search process. For example, an architecture constructed on the
basis of cell-wise search space (Liu et al., 2021b) is composed of many
similar structures of cells and only representative cells need to be
optimized, which contributes to significant computational speed-up.

  * •

Evaluations:

    1. (1)

Early stopping policy: A relatively small number of training epochs are used
to reduce the training cost (i.e., early stopping policy) since the training
time is reduced (Ahmed et al., 2019; Tian et al., 2019; Sun et al., 2019d).

    2. (2)

Reduced training set: Some methods are designed to reduce the size of the
training set to improve training efficiency at the expense of a little
accuracy (Sapra and Pimentel, 2020; Liu et al., 2019a). Besides, low-
resolution data (e.g., ImageNet 32) (Chrabaszcz et al., 2017) is also commonly
used as the training set to accelerate the search process for the optimal
architecture.

    3. (3)

Weight inheritance 111In (Rumelhart et al., 1985, 1986; LeCun et al., 1989),
Rumelhart/Hinton/LeCun used the term ”weight sharing” to mean that different
network connections/links share the same set of weights, and pointed out that
”weight sharing” is the core of shared weight NNs/CNNs. More recent (Lu et
al., 2021; Xie et al., 2022a) use of this term refers to ”weight/parameter
replications” or ”weight inheritance”.:

       * –

Supernet-based inheritance: uses an over-parameterized and pre-trained
supernet to encode all candidate architectures (i.e., subnets). In other
words, the subnets share weights of the identical structures from the
supernet, and they are directly evaluated on the validation dataset to obtain
their model accuracy (Rapaport et al., 2019; Dahal and Zhan, 2020; Elsken et
al., 2019; Sapra and Pimentel, 2020).

       * –

Parent-based inheritance: inherits weights from previously-trained networks
(i.e., parental networks) instead of a supernet, since offspring individuals
retain some identical parts of their parental architectures (Real et al.,
2017; Elsken et al., 2017; Kwasigroch et al., 2019; Zhu et al., 2019) (Real et
al., 2017). As a result, offspring architectures can inherit the weights of
the identical parts and no longer need to be trained from scratch.

    4. (4)

Surrogate model: Since the evaluation of an architecture is time-consuming
(Yao et al., 2018; Liu et al., 2021b), cheap surrogate models have been
introduced in NAS as performance predictors to reduce the computational time
(Lu et al., 2021).

       * –

Online performance predictors: They are trained online on the datasets sampled
from past several epochs (Liu et al., 2021b), including a sequence of data
pairs with different training epochs and their corresponding performance of
these epochs (Lu et al., 2021). After that, they will be used for the
performance prediction on new architectures. To reduce the true evaluations of
architectures, some performance predictors directly predict whether a
candidate architecture can be survived into next iteration through a trained
ranking or classification method, such as classification-wise NAS (Ma et al.,
2021a).

       * –

Offline performance predictors: They are essentially a sort of regression
models mapping the architectures to specific performance. End-to-end
predictors can be trained in an offline manner, so that they are able to
predict the performance of architectures during the entire search process.
Consequently, they can significantly reduce the computational burden (Sun et
al., 2019a; Liu et al., 2018c; Sun et al., 2021).

    5. (5)

Population memory: Population memory is used to store elite individuals from
different generations during the optimization (Sun et al., 2020; Fujino et
al., 2017). When a new individual is generated, it does not need to be
evaluated again if it is the same as an individual in the memory. In other
words, the performance of individuals sharing the same architectures are the
same and can be acquired via the population memory instead of training from
scratch. This mechanism relies on the fact that similar or same individuals
may repeatedly appear in different generations.

According to the above introduction, we can conclude that many of them improve
the search efficiency at the expense of sub-optimiality. For example, a small
population cannot well cover a multi-objective optimal front. Parameter
sharing may lead to the biased search due to much similarities among the
individuals. Highly accurate surrogates need a large number of training data,
which are commonly time-consuming. Population memory heavily relies on the
random emergence of similar or same individuals.

Hardware implementation: Importantly, a powerful hardware platform can
significantly speed up the search process under the reasonable utilization of
computing resources (e.g., cloud computing (Chiba et al., 2019) and volunteer
computers (Atre et al., 2021)). Parallel computation is a powerful tool to
decompose large search problems into small sub-problems, which can be
simultaneously optimized by several cheaper hardware(Jin et al., 2019, 2018).
For example, Lorenzo et al. (Lorenzo et al., 2017) proposed a parallel PSO
algorithm to search for optimal architecture of CNN. The security of the
computing device also becomes an important consideration. For this reason, an
emerging decentralized privacy-preserving framework is applied to NAS, which
unites multiple local clients to collaboratively learn a shared global model
trained on the parameters or gradients of the local models, instead of the raw
data. For example, Zhu et al. (Zhu and Jin, 2022) firstly proposed a real-time
federated NAS that can not only optimize the model architecture but also
reduce the computational payload. Specifically, the decentralized system is
able to accelerate the algorithm efficiency of federated NAS. Besides, data
encryption is employed on the transmitted data (parameters or gradients of the
local models) between the clients and the server to ensure the privacy even
though all of the training are performed in local. Accordingly, federated NAS
is highly efficient and secure, which may become a new hot research topic.

Table 2. Different acceleration strategies Algorithm design | Initialization | Reduced population |  (Assunção et al., 2019b),(Fan et al., 2020),(Liu et al., 2019a),(Yang et al., 2020)
---|---|---|---
Evaluation | Early stopping policy |  (Sun et al., 2019d),(Ahmed et al., 2019),(Ortego et al., 2020),(Frachon et al., 2019),(Assunção et al., 2019a),(Assunção et al., 2019b),(Mo et al., 2021)
Reduced training set |  (Sapra and Pimentel, 2020),(Liu et al., 2019a),(Wang et al., 2020c)
Weight inheritance | Supernet-based sharing |  (Sapra and Pimentel, 2020),(Rapaport et al., 2019),(Dahal and Zhan, 2020),(Elsken et al., 2019)
Parent-based sharing |  (Elsken et al., 2017),(Kwasigroch et al., 2019),(Zhu et al., 2019),(Ahmed et al., 2019),(Frachon et al., 2019),(Schorn et al., 2020),(Chen et al., 2019b)
Surrogate model | Online performance predictors |  (Lu et al., 2021),(Liu et al., 2018c), (Ma et al., 2021a), (Lomurno et al., 2021)
Offline performance predictors |  (Sun et al., 2019a),(Liu et al., 2021a), (Rawal and Miikkulainen, 2018)
| Population memory |  (Sun et al., 2020),(Miahi et al., 2022),(Li et al., 2022),(Chu et al., 2020)
Hardware implementation |  (Chiba et al., 2019),(Jin et al., 2019),(Jin et al., 2018),(Zhu and Jin, 2022)

Table 2 lists the common acceleration strategies to improve the algorithm
efficiency. It is noted that multiple strategies can be utilized together to
improve computational speed-up. For example, Lu et al. (Lu et al., 2021)
employed supernet and learning curve performance predictor in NAS, while Liu
et al. (Liu et al., 2019a) leveraged a small populations size and a small
dataset to reduce the time overhead of evaluation.

####  4.2.5. Summary

Most NAS methods are based on basic EC search paradigms on an entire-
structured search space, which are introduced above. However, there are also
some other automatic search techniques such as RL-based (Jaâfra et al., 2019),
Bayesian-based (White et al., 2021), and gradient-based (Liu et al., 2018b)
methods, for architecture search.

RL-based methods can be regarded as an incremental search, where a policy
function is learned by using a reward-prediction error to drive the generation
of incremental architecture. Due to the large-scale of state space and action
space, RL-based methods require immense computational resources. In addition,
there are a large number of hyper-parameters (e.g., discount factor) in RL-
based NAS. Besides, they transform a multi-objective optimization problem into
a single-objective problem via a priori or expert knowledge, so they are
unable to find a Pareto optimal set to the target tasks.

Bayesian-based methods are a common tool for hyperparametric optimization
problems with low dimensions. In comparison to EC-based methods, they are much
more efficient on the condition that a proper distance function has to be
designed to evaluate the similarities between two subnets. However, the
computational cost of Gaussian process grows exponentially and its accuracy
decreases, when the dimensionality of the problem increases.

Gradient-based methods, taking a NAS problem as a continuous differentiable
problem instead of a discrete one, are able to efficiently search
architectures with proper weight parameters. Unfortunately, their GPU costs
are usually very high due to a large number of parameters to be updated in
gradient-based algorithms (Liu et al., 2018b).

In contrast, EC-based methods benefit from less hyperparameters to be
optimized and no distance functions to be designed. In addition, EC-based
methods can be applied to NAS with multiple objectives and constraints.
Although there many acceleration strategies in EC-based methods, they still
suffer from high computational overheads.

###  4.3. Joint Optimization

####  4.3.1. Problem Formulation

The independent optimization of architecture or parameters is difficult to
achieve the optimal model on give tasks. Hence, joint optimization methods
have been developed to search for the optimal configuration of architecture
($A^{*}$), and parameters ($W^{*}$, associated weights). The optimization
problem can be defined in Eq. 4.

(4)
$\displaystyle\begin{matrix}\left({{W}^{*}},{{A}^{*}}\right)$=$\underset{W,A}{\arg\min}L\left(W,A\right)\end{matrix}$

where $L$ is the loss function.

In the followings, we will introduce the joint optimization regarding the
solution representations and search paradigms, and then discuss the pros and
cons of EC-based methods in comparison to others.

####  4.3.2. Solution Representations

There are three typical classes of encoding schemes used for joint
optimization.

Linear encoding: This is a simple but effective encoding strategy, which has
been widely used in many studies to build architecture with high performance
(Aljarah et al., 2018; Maniezzo, 1994). In (Maniezzo, 1994), a variable-length
binary vector was used to represent weights and structure of neural networks,
where the weights utilize direct encoding.

Tree-based encoding: In this encoding, the topology and weights of an
architecture can be represented by a tree structure with a number of nodes and
edges (Zhang and Mühlenbein, 1995; Golubski and Feuring, 1999). In (Zhang et
al., 2020b), the mechanism of Reverse Encoding Tree (RET) was developed to
ensure the robustness of a deep model, where the topological information of an
architecture was represented by a combination of nodes and the weight
information was recorded on the edges.

Graph-based encoding: In this encoding, the nodes of a graph represent neurons
or other network units, and the edges are used to record the weight
information (Han and Cho, 2006; Chai et al., 2022). For example, a graph
incidence matrix was developed in (Oong and Isa, 2011) to encode a neural
network. The size of the matrix was set to ($N_{i}$ \+ $N_{h}$ \+ $N_{o}$)
$\times$ ($N_{i}$ \+ $N_{h}$ \+ $N_{o}$), where $N_{i}$, $N_{h}$ and $N_{o}$
indicate the numbers of input, hidden, and output nodes, respectively. In the
graph incidence matrix, real numbers represented the weight and biases, and
“0” meant that there was no connection between two nodes.

####  4.3.3. Search Paradigms

There are a number of effective search paradigms for joint optimization, and
the EC-based search paradigms are in the spotlight.

Basic EC search paradigm: Some basic EC search methods have been employed to
handle joint optimization problems (Fahrudin et al., 2016; Oong and Isa, 2011;
Stanley and Miikkulainen, 2002; Barman and Kwon, 2020; Behjat and Chidambaran,
2019). In (Oong and Isa, 2011), an architecture and its corresponding weights
were simultaneously optimized by an EC-based method using linear and graph
encodings. In neuro-evolution of augmenting topologies (NEAT) (Stanley and
Miikkulainen, 2002), the architecture of a small network is evolved by an
incremental mechanism, while the weights are optimized by an EC-based method.
NEAT is able to ensure the lowest dimensional search space over all
generations. Some representative studies on NEAT are presented in (Barman and
Kwon, 2020; Behjat and Chidambaran, 2019).

Multi-objective search paradigm: Multi-objective optimization on model design
has been developed in many studies (e.g., artificial neural network (Rostami
and Neri, 2016) and recurrent neural network (Smith and Jin, 2014)). For
example, Smith et al. (Smith and Jin, 2014) built a bi-objective optimization
(i.e., the minimizations of the mean squared error (MSE) on a training dataset
and the number of connections in the network) to search for optimal weights
and connections of network architectures. The chromosome of an individual was
composed of two parts, where the one with Boolean type represented the
structure of a network, and the other with real values represented the
weights.

####  4.3.4. Summary

Direct encoding is used to be prevalent in the joint optimization of small-
scale neural networks (Oong and Isa, 2011; Yao and Liu, 1996). However, with
the increase of the scale of neural networks, direct coding of high-
dimensional vector or matrix of weights is not realistic. Therefore, recent
studies are more on indirect encoding. For example, a complex mapping with
acceptable accuracy loss is designed in (Koutník et al., 2014; D’Ambrosio and
Stanley, 2007) to construct weight vectors with arbitrary size.

EC-based approaches with capability of searching the optimal solution have
been developed to configure a DL model for the specific task. However, they
often encounter a prohibitive computational cost, which is even higher than
that of model architecture optimization. Hence, designing efficient EC-based
approaches for architecture and parameter search deserves much investigation.

##  5\. Model Deployment

The large-scale DNNs are not straightforward to be deployed into devices
(e.g., smartphones) with limited computation and storage resources (e.g.,
battery capacity and memory size). To solve this issue, various model
compression approaches have been proposed to reduce the model size and
inference time, such as pruning, model distillation, and quantization
(Choudhary et al., 2020). However, they need much expert knowledge and a lot
of efforts on the manual compression of neural network models. In contrast,
EC-based approaches are automation approaches and has been recently introduced
to achieve automated model compression. We have observed that most of them
concentrate on the area of model pruning.

###  5.1. Model Pruning

####  5.1.1. Problem Formulation

DNN is commonly an over-parameterized model, which has redundant and non-
informative components (e.g., weights, channels and filters). To address this
issue, researchers have designed various pruning approaches (e.g., channel
pruning (He et al., 2017)) to obtain a lightweight deep network model with
high accuracy. Model pruning can be formulated as

(5) $\displaystyle Los{{s}_{A_{s}^{*}}}\approx Los{{s}_{A}}$
$\displaystyle\begin{matrix}\text{s}\text{.t}\text{.}&{A}_{{s}}^{*}$=$\underset{{C}}{\mathop{\text{pruning}}}\,\left({A,C}\right)\\\
\end{matrix}$

where $C$ represents redundant and non-informative units, $A$ and
$A{{}_{s}}^{*}$ represent original model and lightweight model, respectively,
$Los{{s}_{A_{s}^{*}}}$ and $Los{{s}_{A}}$ represent loss of $A{{}_{s}}^{*}$
and $A$.

This study aims to introduce EC-based methods for model pruning, and readers
interested in traditional pruning methods such as weight-based pruning,
neuron-based pruning, filter-based pruning, layer-based pruning, and channel-
based pruning may refer to the surveys (Choudhary et al., 2020) to get more
details.

####  5.1.2. Solution Representations

For model pruning, binary encoding is one of the most popular approaches among
these solution representations, where each element corresponds to the network
component (e.g., channel). In (Wang et al., 2018), the network pruning task
was formulated as a binary programming problem, where a binary variable was
directly associated with each convolution filter to determine whether or not
the filter took effect. Although binary representation is straightforward, the
length of the representation becomes large when the model complexity (i.e.,
the number of units) improves, and the overhead of exploration will also
increase.

To address the above issue, some efficient solution representations (i.e.,
indirect encoding) have been developed. For example, Liu et al. (Liu and Guo,
2021) used $N$ digits to record the number of compressed layers. The first
digit represented the number of compressed layers, and following digits
recorded the selected compression operator index of each layer. This way can
significantly improve the search efficiency. In (Liu et al., 2019b), encoding
vectors are used to represent the number of channels in each layer for
original networks. Then a meta-network is constructed to generate the weights
according to network encoding vectors. By stochastically fed with different
structure encoding, the meta-network gradually learns to generate weights for
various pruned structures.

####  5.1.3. Search Paradigms

The search paradigms in model pruning studies can be categorized into two main
groups.

Basic EC search paradigm: A number of studies introduce single-objective EC
search paradigm for model pruning (Wu et al., 2021a; Tang et al., 2019). For
example, Wu et al. (Wu et al., 2021a) first analysed the pruning sensitivity
on weights via differential evolution (DE), and then the model was compressed
by iteratively performing the weight pruning process according to the weight
sensitivity. In addition, this method adopted a recovery strategy to increase
the pruned model performance during the fine-tuning phase.

Multi-objective search paradigm: Recently, this sort of search paradigm has
been adopted to model pruning, which is able to provide users with a set of
Pareto lightweight models. For example, Zhou et al. (Zhou et al., 2020)
considered two objectives (i.e., minimizing convolutional filters and
maximizing the model performance) for biomedical image segmentation. During
the model pruning, a classical multi-objective optimization algorithm (NSGA-II
(Deb et al., 2002)) was used find the optimal set of non-dominated solutions,
where the optimization was based on a binary string encoding (each bit
represents a filter).

In Table 3, we have summarized these two categories of search paradigms as
well as their corresponding ways of encoding.

Table 3. Different search paradigms and solutions representations for model pruning | Direct encoding |  Indirect encoding
---|---|---
Basic EC search paradigm |  (Samala et al., 2018),(Junior and Yen, 2021a),(Tang et al., 2019),(Wu et al., 2021a),(Junior and Yen, 2021b),(Zhou et al., 2021b),(Gerum et al., 2020),(Zemouri et al., 2019),(Chen et al., 2019a),(Poyatos et al., 2022),(Fernandes and Yen, 2021),(Li et al., 2020),(Shang et al., 2022) |  (Liu et al., 2019b)
Multi-objective search paradigm |  (Zhou et al., 2020),(Zhou et al., 2021c),(Wu et al., 2019),(Hong et al., 2020),(Xu et al., 2021),(Yang et al., 2019),(Zhang et al., 2021a) |  (Wang et al., 2021a),(Zhang et al., 2021b),(Loni et al., 2020)

###  5.2. Other EC-based Model Deployment Methods

Different from model pruning, there are several other EC-based model
compression methods for model deployment. In the followings, some typical
methods are introduced, including knowledge distillation, low-rank
factorization, and EC for hybrid techniques.

####  5.2.1. Knowledge Distillation

Knowledge distillation (KD) (Gou et al., 2021) aims to get a small light
network but with good generalization capability. The basic idea is to transfer
the knowledge learned from a big cumbersome network (or teacher network) with
good generalization ability to a small but light network (or student network).

However, knowledge distillation may be seriously influenced when there is a
big gap in the learning capability between the teacher and student networks.
In other words, if the difference is large, the student network may not be
able to learn knowledge from the teacher network. Recently, several EC-based
approaches have been proposed to mitigate the above issue of knowledge
distillation. For example, Wu et al. (Wu et al., 2020) proposed an
evolutionary embedding learning (EEL) paradigm to learn a fast accurate
student network via massive knowledge distillation. Their experimental results
show that the EEL is able to narrow the performance between the teacher and
student networks on given tasks. Zhang et al. (Zhang et al., 2022a) developed
an evolutionary knowledge distillation method to improve the effectiveness of
knowledge transfer. In this method, an evolutionary teacher was learned online
and consistently transfers intermediate knowledge to the student network to
narrow the gap of the learning capability between them.

####  5.2.2. Low-rank Factorization

DNNs often involve in a huge number of weights, which may impact the inference
speed and seriously increase the storage overhead of the DNN. The weights can
be viewed as a matrix $W$ with $m$ $\times$ $n$ dimensions. The low-rank
approach is commonly applied to the weight matrix ($W$) after the DNN is fully
trained. For example, singular value decomposition (Fortuna and Frasca, 2021)
is a typical low-rank factorization method, where $W$ is decomposed as
follows.

(6) $\displaystyle\begin{matrix}W$=$USV^{T}\end{matrix}$

where $U$ $\in$ $R^{m\times m}$, $V^{T}$ $\in$ $R^{n\times n}$ are orthogonal
matrices and $S$ $\in$ $R^{m\times n}$ is a diagonal matrix.

Notably, most of the existing low-rank factorization methods rely on domain
expertise and experience for the selection of hyperparameters (e.g., the rank
and sparsity of weight matrix) to get an appropriate compression results
without serious performance degradation (Swaminathan et al., 2020; Winata et
al., 2019; Hsu et al., 2021).

Accordingly, EC-based methods have been introduced to solve the above
challenge (Wang et al., 2018; Huang et al., 2020). For example, Huang et al.
(Huang et al., 2020) presented a multi-objective evolution approach to
automatically optimize rank and sparsity for weight matrix without human
intervention, where two objectives were taken into account including the
minimization of the model classification error rate and maximization of the
model compression rate. They therefore generated a set of approximately
compressed models with different compression rates to mitigate the expensive
training process.

####  5.2.3. EC for Jointly Optimization

Many compression techniques (e.g., quantization) can be easily applied on top
of other techniques (e.g., pruning and low-rank factorization). For example,
pruning first and then quantification can obtain a lightweight model with
faster inference. Similarly, EC can optimize more than one model compression
method at the same time. In the followings, we will briefly review such works.

Phan et al. (Phan et al., 2020) designed an efficient 1-Bit CNNs, which
combined quantization with a compact model. Specifically, they firstly created
a number of strong baseline binary networks (BNNs), which had abundant random
group combinations at each convolutional layer. Then, they adopted
evolutionary search to seek an optimal group convolution combination with
accuracy above threshold. Finally, the obtained binary models werr trained
from scratch to achieve the final lightweight network. Different from (Phan et
al., 2020), Polino et al. (Polino et al., 2018) jointly utilized weight
quantization and distillation to compress large networks (i.e., teacher
network) into small networks (i.e., student network), where the latency and
model error were regarded as the objectives during the optimization.

Recently, Zhou et al. (Zhou et al., 2021b) developed an evolutionary
algorithm-based method for shallowing DNNs at block levels (ESNB). In ESNB, a
prior knowledge was extracted from the original model to guide the population
initialization. Then, an evolutionary multi-objective optimization mothed was
performed to minimize the number of blocks and the accuracy drop (i.e., loss).
After that, knowledge distillation was employed to compensate for the
performance degradation via matching output of the pruned model with the
softened and hardened output of the original model.

####  5.2.4. Summary

There is still a big room for the improvement on addressing the huge
computational overhead of evolutionary model deployment. Acceleration
strategies may be able to alleviate the issue. Besides, there is a high
coupling between model deployment and model generation since the performance
of the compressed network is strongly dependent on the performance of the
original network. The black-box nature of model also hampers deployment in
security-critical tasks (e.g., medicine and finance).Consequently, it is
promising and challenging to take the model compression, NAS, and
interpretability as a single optimization problem and handle it with
acceptable time consumption.

##  6\. Applications, Open Issues, and Trends

###  6.1. Applications

EDL algorithms have been widely used in various real-world applications. In
practical, great development has been achieved in computer vision (CV),
natural language processing (NLP) and other practical applications (e.g.,
crisis prediction and disease prediction).

####  6.1.1. Computer Vision

CV is an important domain of computer science, playing an important role in
identifying useful information (e.g., objects and classifications) for
specific tasks (e.g., image segmentation (Zhou et al., 2020) and object
detection (Zhang and Rockett, 2005)) on images or videos. In the early days,
manually designed models for computer vision achieved good performance on
public datasets at the expense of extensive time and labour. With the
development of EDL, many new structures have been developed by computer
programming and they show better performance than these manually designed
models, especially on the widely used benchmark datasets for image
classification, such as CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and
ImageNet (Deng et al., 2009). For example, the state-of-the-art NAT-M4 (Lu et
al., 2021) with a small model size achieves Top-1 accuracy of 80.5% on
ImageNet. Image-to-image processing (Liu et al., 2021b) (e.g., super-
resolution, image inpainting, and image restoration) also received extensive
attention from researchers (Rundo et al., 2019; Song et al., 2020; Zhan et
al., 2021). Ho et al. (Ho et al., 2021) employed NAS techniques to improve
image denoising, inpainting, and super-resolution on the foundation of deep
image prior (Ho et al., 2021). In addition to the above applications, EDL also
has great potential in other areas of CV, such as object detection (Demirkir
and Sankur, 2006), video/picture understanding (Maniezzo, 1994), and image
segmentation (Fan et al., 2020).

####  6.1.2. Natural Language Processing

Natural language processing (NLP) driven by computer science and computational
linguistics, aims to understand, analyze, and extract knowledge on text and
speech recoginition (Young et al., 2018). Many effective NLP models (e.g.,
GPT-2 (Radford et al., 2018) and BERT (Devlin et al., 2018)) narrow the chasm
between human communication and computer understanding using sophisticated
mechanisms. Recently, EC-inspired NLP models have been proposed such as
language model (Murray and Chiang, 2015), entity recognition (Sikdar et al.,
2012), text classification (Tanaka et al., 2016; Andersen et al., 2021; Londt
et al., 2021, 2020), and keyword spotting (Mazzawi et al., 2019). Satapathy et
al. (Song, 2021) introduced evolutionary multi-objective (i.e., inference time
and accuracy) optimization in an English translation system. Sikdar et al.
(Sikdar et al., 2012) employed DE in feature selection for named entity
recognition (NER).

####  6.1.3. Other Applications

In addition to CV and NLP, EDL also shows strong ability on handling other
practical applications, such as medical analysis (Liu et al., 2018a; Zhu et
al., 2007), financial prediction (Tang et al., 2019), signal processing
(Erguzel et al., 2014; Huang et al., 2012), and industrial prediction (Mei et
al., 2017; Ma et al., 2021b). In particular, Zhu et al. (Zhu et al., 2007)
presented a Markov blanket-embedded genetic algorithm for feature selection to
improve gene selection. In (Tang et al., 2019), financial bankruptcy analysis
was handled by an evolutionary pruning neural network. The work in (Erguzel et
al., 2014) designed a feature selection method based on ACO to classify
electromyography signals. For remote sensing imagery, a suitable model was
found by multi-objective neural evolution architecture search (Ma et al.,
2021b), where architecture complexity and performance error of searched
network were two conflicting objectives.

###  6.2. Open Issues

EDL is a hot research topic in both fields of machine learning and
evolutionary computation. There are a large number of publications on various
top conferences and journals, such as ICCV, CVPR, GECCO, TPAMI, TEVC, TCYB,
and TNNLS (see the reference list). Yet some challenges remain to be resolved.

Acceleration strategies: Many EDL approaches suffer from low efficiency due to
the expensive evaluations. So various acceleration strategies, such as
surrogate model (Sun et al., 2019a), supernet (Guo et al., 2020), and early
stop (Sun et al., 2019d) have been designed. However, the improvements of the
accuracy are at the expense of sacrifice a bit of model accuracy. Taking the
supernet-based inheritance as an example (Xie et al., 2022a), we cannot
guarantee that every subnet receives a reliable evaluation due to the
catastrophic forgetting (Zhang et al., 2020a) and weight coupling (Hu et al.,
2021a). Therefore, how to balance the efficiency and accuracy needs further
investigation.

Effectiveness: There is a debate on whether EDL has many advantages over other
search paradigms (e.g., random search and RL). Some studies argue that many
popular search paradigms (e.g., EC-based methods and RL-based methods) have no
big difference from the random search methods in their performance, and some
random search methods even outperform EC-based methods in some scenarios
(Sciuto et al., 2020). On the contrary, EC-based approaches have also been
proved to be more effective than random search methods in many studies (Real
et al., 2019; Guo et al., 2020; Jones et al., 2019). Thus, a unified platform
is essential to measure the effectiveness of different search models, under
the consistent search space and hyperparameters configuration (Xie et al.,
2022b). In addition, elaborate experiments are required to justify the effects
of different genetic operators (e.g., crossover operator) to the evolutionary
process of EDL.

Large-scale datasets: There is an issue for the studies of EDL on large-scale
datasets. It is noted that many studies of EDL are tested on small- and
medium-scale datasets such as CIFAR-10 and CIFAR-100 (including 60000
32$\times$32 images), and especially the accuracy on CIFAR-10 reaches up to
98% (Lu et al., 2021). Although large-scale datasets are ubiquitous and
essential in various domains like gene analysis (Yu et al., 2009) and ImageNet
(Deng et al., 2009), computational costs are unaffordable for many researchers
as pointed in some statistical reports (He et al., 2021; Liu et al., 2021b).
Therefore, the sensitivity of the EDL methods to different scales of datasets
is necessary (Zhou et al., 2021a) and how to economically and efficiently
verify EDL methods on large-scale datasets also deserves much investigations.

End-to-end EDL: Originally, AutoML aims to simultaneously optimize feature
engineering, model generation and model deployment as a whole. However, there
is a strong correlation between them where the performance of next phase
heavily relies on the results of the previous phase (Liu and Guo, 2021). As a
result, most studies only focus on parts of the EDL pipeline (Fig. 1). For
instance, TPOT (Olson and Moore, 2016) is designed on top of Pytorch for
building classification tasks, which however only supports a multi-layer
perception machine (i.e., model generation). There are many partially
accomplished end-to-end for EDL, such as ModelArts (model generation),
Google’s Cloud (NAS), and Feature Labs (feature engineering) (Yao et al.,
2018), to name but a few. The main reason is that the optimization of the
whole EDL pipeline may need huge computational cost not only on the
exploration of the large-scale search space but also on handling highly-
coupled relation between different parts of EDL. Consequently, finding an
optimal solution of the complete EDL pipeline is essential but challenging.

###  6.3. Challenges and Future Trends

Although remarkable progress has been made in EDL, there are still many
promising lines of research.

Fair comparisons: Unfair comparisons of different EDL methods are easily
encountered with the following reasons. Firstly, uniform benchmarks are
essential. In feature engineering, no uniform benchmark is for the fair
comparison of different algorithms due to different downstream prediction
models and feature sets. Secondly, there is no uniform criterion for different
methods in handling NAS and model compression by using different tricks (e.g.,
cutout (DeVries and Taylor, 2017) and ScheduledDropPath (Zoph et al., 2018)),
which may influence the performance of the final architecture. Thirdly, a fair
platform for EDL is essential. There are some fair benchmarks but only for
specific tasks, such as BenchENAS (Xie et al., 2022b), NAS-Bench-101 (Ying et
al., 2019), NAS-Bench-201 (Dong and Yang, 2020), NAS-Bench-301 (Siems et al.,
2020), and HW-NAS-Bench (Li et al., 2021).

Interpretability: EDL is known as a black-box optimization, and there is a
lack of theoretical analysis to explain its superiority (Wang et al., 2021b).
For example, it is difficult to explain why EC-based method tends to select
features contribute to the performance of the classification model in feature
engineering. As a result, the development of EDL in some sensitive domains
such as financial and medical fields is slow. To overcome this issue, Evans et
al. (Evans et al., 2018b) used visualization to expound how the evolved
convolution filter served and indirectly explained the search process of the
model. Nevertheless, some studies argue that the explanation for these
occurrences is usually post-hoc and lacks trustworthy mathematical deduction
(Liu et al., 2021b; Al-Sahaf et al., 2019). Thus, the interpretability of EDL
is an interesting and promising research direction.

Exploring more scenarios: There is still plenty of room for the improvement of
the performance of EDL on both benchmarks and real-world applications.
Although EDL methods outperform manually designed models on various image
benchmarks (CIFAR-10 and ImageNet), the state-of-the-art EDL methods (Wang et
al., 2020a) lost their advantages on NLP in comparison with human-designed
models (e.g., GPT-2 (Radford et al., 2018), Transformer-XL (Dai et al.,
2019)). In comparison with the benchmarks, it is more difficult to handle
real-world tasks, which inevitably contain noise (e.g., mislabeling and
inadequate or imbalance data) or may have small-scale datasets (leading to
overfitting). Hence, some techniques such as unsupervised and self-supervised
learning may be incorporated into EDL to mitigate these types of issues.

##  7\. Conclusions

With the development of machine leaning and evolutionary computation, many EDL
approaches have been proposed to automatically optimize the parameters or
architectures of deep models following the EC optimization framework. EDL
approaches show competitive performance in robust and search capability, in
comparison with the manually designed approaches. Therefore, EDL has become a
hot research topic.

In this survey, we first introduced EDL from the perspective of DL and EC to
facilitate the understanding of readers from the communities of ML and EC.
Then we formulated EDL as a complex optimization problem, and provided a
comprehensive survey of EC techniques in solving EDL optimization problems in
terms of feature engineering, model generation to model deployment to form a
new taxonomy (i.e., what, where and how to evolve/optimize in EDL).
Specifiically, we discussed the solution representations and search paradigms
of EDL at different stages of its pipeline in detail. Then the pros and cons
of EC-based approaches in comparison to non-EC based ones are discussed.
Subsequently, various applications are summarized to show the potential
ability of EDL in handling real-world problems.

Although EDL approaches have achieved great progress in AutoML, there are
still a number of challenging issues to be resolved. For example, effective
acceleration strategies are essential to reduce the expensive optimization
process. Another issue is to handle large-scale datasets and how to perform
fair comparisons between different EDL approaches or non-EC based methods.
More investigations are required to theoretically analyse or interpret the
search ability of EDL. In addition, a lot of efforts are required on the
improving the performance of EDL on both benchmarks (e.g., large-scale and
small-scale data) and real-world applications. Lastly, the development of end-
to-end EDL is challenging but deserves much efforts.

## References

  * (1)
  * Abdi and Williams (2010) Hervé Abdi and Lynne J Williams. 2010.  Principal Component Analysis.  _Comput. Stat._ 2, 4 (2010), 433–459.
  * Ahmed et al. (2019) Amr Ahmed, Saad Mohamed Darwish, and Mohamed M. El-Sherbiny. 2019\.  A Novel Automatic CNN Architecture Design Approach Based on Genetic Algorithm. In _Int. Conf. Adv. Intell. Syst. Inform._ 473–482.
  * Ahmed et al. (2014) Soha Ahmed, Mengjie Zhang, Lifeng Peng, and Bing Xue. 2014\.  Multiple Feature Construction for Effective Biomarker Identification and Classification Using Genetic Programming. In _Proc. Genetic Evol. Comput. Conf._ 249–256.
  * Al-kazemi and Mohan (2002) Buthainah Al-kazemi and Chilukuri Krishna Mohan. 2002.  Training Feedforward Neural Networks using Nulti-phase Particle Swarm Optimization. In _Proc. Int. Conf. Neural Inf. Process._ , Vol. 5. 2615–2619.
  * Al-Sahaf et al. (2019) Harith Al-Sahaf, Ying Bi, Qi Chen, Andrew Lensen, Yi Mei, Yanan Sun, Binh Tran, Bing Xue, and Mengjie Zhang. 2019\.  A Survey on Evolutionary Machine Learning.  _J. R. Soc. N. Z._ 49, 2 (2019), 205–228.
  * Albukhanajer et al. (2015) Wissam A. Albukhanajer, Johann A. Briffa, and Yaochu Jin. 2015\.  Evolutionary Multiobjective Image Feature Extraction in the Presence of Noise.  _IEEE Trans. Cybern._ 45, 9 (2015), 1757–1768.
  * Alexandropoulos and Aridas (2019) Stamatios-Aggelos N Alexandropoulos and Christos K Aridas. 2019.  Multi-objective Evolutionary Optimization Algorithms for Machine Learning: A Recent Survey.  _Approximation and Optimization_ (2019), 35–55.
  * Aljarah et al. (2018) Ibrahim Aljarah, Hossam Faris, and Seyed Mohammad Mirjalili. 2018\.  Optimizing Connection Weights in Neural Networks Using the Whale Optimization Algorithm.  _Soft Comput._ 22, 1 (2018), 1–15.
  * Andersen et al. (2021) Hayden Andersen, Sean Stevenson, Tuan Ha, Xiaoying Gao, and Bing Xue. 2021.  Evolving Neural Networks for Text Classification Using Genetic Algorithm-based Approaches. In _Proc. IEEE Congr. Evol. Comput._ 1241–1248.
  * Assunção et al. (2019a) Filipe Assunção, Joao Correia, and Rúben Conceição. 2019a.  Automatic Design of Artificial Neural Networks for Gamma-Ray Detection.  _IEEE Access_ 7 (2019), 110531–110540.
  * Assunção et al. (2018) Filipe Assunção, Nuno Lourenço, P. Machado, and Bernardete Ribeiro. 2018.  Evolving the Topology of Large Scale Deep Neural Networks. In _Proc. Eur. Conf. Genetic Program_. 19–34.
  * Assunção et al. (2019b) Filipe Assunção, Nuno Lourenço, Penousal Machado, and Bernardete Ribeiro. 2019b.  Fast denser: Efficient Deep Neuroevolution. In _Proc. Eur. Conf. Genetic Program_. 197–212.
  * Atre et al. (2021) Medha Atre, Birendra Jha, and Ashwini Rao. 2021.  Distributed Deep Learning Using Volunteer Computing-Like Paradigm. In _Proc. Int. Parallel and Distrib. Process. Symp._ 933–942.
  * Barman and Kwon (2020) Shohag Barman and Yung-Keun Kwon. 2020.  A Neuro-Evolution Approach to Infer A Boolean Network From Time-Series Gene Expressions.  _Bioinformatics_ 36, 2 (2020), i762–i769.
  * Behjat and Chidambaran (2019) Amir Behjat and Sharat Chidambaran. 2019.  Adaptive Genomic Evolution of Neural Network Topologies (AGENT) for State-to-Action Mapping in Autonomous Agents. In _Proc. Int. Conf. Robot. Autom._ 9638–9644.
  * Bhanu and Krawiec (2002) Bir Bhanu and Krzysztof Krawiec. 2002.  Coevolutionary Construction of Features for Transformation of Representation in Machine Learning. In _Proc. Genetic Evol. Comput. Conf._ 249–254.
  * Bi et al. (2018) Ying Bi, Bing Xue, and Mengjie Zhang. 2018.  An Automatic Feature Extraction Approach to Image Classification Using Genetic Programming. In _Proc. Int. Conf. Appl. Evol. Comput._ 421–438.
  * Cano et al. (2017) Alberto Cano, Sebastián Ventura, and Krzysztof J. Cios. 2017.  Multi-Objective Genetic Programming for Feature Extraction and Data Visualization.  _Soft Comput._ 21, 8 (2017), 2069–2089.
  * Castelli et al. (2011) Mauro Castelli, Luca Manzoni, and Leonardo Vanneschi. 2011\.  Multi Objective Genetic Programming for Feature Construction in Classification Problems. In _Proc. Int. Conf. Learn. Intell. Optim._ 503–506.
  * Chai et al. (2022) Zheng-Yi Chai, ChuanHua Yang, and Ya-Lun Li. 2022.  Communication Efficiency Optimization in Federated Learning Based on Multi-Objective Evolutionary Algorithm.  _Evol. Intell._ (2022), 1–12.
  * Chandra (2015) Rohitash Chandra. 2015\.  Competition and Collaboration in Cooperative Coevolution of Elman Recurrent Neural Networks for Time-Series Prediction.  _IEEE Trans. Neural Netw. Learn. Syst._ 26, 12 (2015), 3123–3136.
  * Chandra and Zhang (2012) Rohitash Chandra and Mengjie Zhang. 2012.  Cooperative Coevolution of Elman Recurrent Neural Networks for Chaotic Time Series Prediction.  _Neurocomputing_ 86 (2012), 116–123.
  * Chen et al. (2022) Ke Chen, Bing Xue, Mengjie Zhang, and Fengyu Zhou. 2022\.  Evolutionary Multitasking for Feature Selection in High-Dimensional Classification via Particle Swarm Optimization.  _IEEE Trans. Evol. Comput._ 26, 3 (2022), 446–460.
  * Chen et al. (2015) Qi Chen, Bing Xue, and Mengjie Zhang. 2015.  Generalisation and Domain Adaptation in GP with Gradient Descent for Symbolic Regression. In _Proc. IEEE Congr. Evol. Comput._ 1137–1144.
  * Chen et al. (2019a) Shuxin Chen, Lin Lin, Zixun Zhang, and Mitsuo Gen. 2019a.  Evolutionary NetArchitecture Search for Deep Neural Networks Pruning. In _Proc. Aust. Conf. Artif. Intell._ 189–196.
  * Chen et al. (2020) Xiangru Chen, Yanan Sun, Mengjie Zhang, and Dezhong Peng. 2020\.  Evolving Deep Convolutional Variational Autoencoders for Image Classification.  _IEEE Trans. Evol. Comput._ 25, 5 (2020), 815–829.
  * Chen et al. (2019b) Yukang Chen, Gaofeng Meng, Qian Zhang, Shiming Xiang, and Chang Huang. 2019b.  RENAS: Reinforced Evolutionary Neural Architecture Search. In _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ 4787–4796.
  * Cheng et al. (2021) Fan Cheng, Feixiang Chu, Yi Xu, and Lei Zhang. 2021\.  A Steering-Matrix-Based Multiobjective Evolutionary Algorithm for High-Dimensional Feature Selection.  _IEEE Trans. Cybern._ 52, 9 (2021), 9695–9708.
  * Cheng et al. (2017) Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017\.  A Survey of Model Compression and Acceleration for Deep Neural networks.  _arXiv preprint arXiv:1710.09282_ (2017).
  * Chiba et al. (2019) Zouhair Chiba, Noreddine Abghour, Khalid Moussaid, Amina El Omri, and Mohamed Rida. 2019.  Intelligent Approach to Build a Deep Neural Network Based IDS for Cloud Environment Using Combination of Machine Learning Algorithms.  _Comput. & Sec._ 86 (2019), 291–317.
  * Choudhary et al. (2020) Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. 2020.  A Comprehensive Survey on Model Compression and Acceleration.  _Artif. Intell. Rev._ 53, 7 (2020), 5113–5155.
  * Chrabaszcz et al. (2017) Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. 2017\.  A Downsampled variant of imageNet as an alternative to the Cifar datasets.  _arXiv preprint arXiv:1707.08819_ (2017).
  * Chu et al. (2020) Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Hailong Ma. 2020\.  Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search. In _Proc. Eur. Conf. Comput. Vis._ 99–113.
  * Conti et al. (2018) Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. 2018.  Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents. In _Proc. Adv. Neural Inf. Process. Syst._ , Vol. 31. 5032–5043.
  * Cui et al. (2018) Xiaodong Cui, Wei Zhang, Zoltán Tüske, and Michael Picheny. 2018.  Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks.  _Proc. Adv. Neural Inf. Process. Syst._ 31 (2018), 6051–6061.
  * Da Silva and Neto (2011) Sérgio Francisco Da Silva and João do ES Batista Neto. 2011\.  Improving The Ranking Quality of Medical Image Retrieval Using A Genetic Feature Selection Method.  _Decis. Support. Syst._ 51, 4 (2011), 810–820.
  * Dahal and Zhan (2020) Binay Dahal and Justin Zhijun Zhan. 2020.  Effective Mutation and Recombination for Evolving Convolutional Networks. In _Proc. Adv. Neural Inf. Process. Syst._ 1–6.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019\.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In _Proc. Assoc. Comput. Linguist._ 2978–2988.
  * D’Ambrosio and Stanley (2007) David B. D’Ambrosio and Kenneth O. Stanley. 2007.  A Novel Generative Encoding for Exploiting Neural Network Sensor and Output Geometry. In _Proc. Genetic Evol. Comput. Conf._ 974–981.
  * Darwish et al. (2020) Ashraf Darwish, Aboul Ella Hassanien, and Swagatam Das. 2020\.  A Survey of Swarm And Evolutionary Computing Approaches for Deep Learning.  _Artif. Intell. Rev._ 53, 3 (2020), 1767–1812.
  * Deb et al. (2002) Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. 2002.  A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II.  _IEEE Trans. Evol. Comput._ 6, 2 (2002), 182–197.
  * Demirkir and Sankur (2006) Cem Demirkir and Bülent Sankur. 2006.  Object Detection Using Haar Feature Selection Optimization. In _Proc. IEEE Signal Process. Commun. Appl._ 1–4.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. 2009\.  ImageNet: A Large-scale Hierarchical Image Database.  _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ (2009), 248–255.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * DeVries and Taylor (2017) Terrance DeVries and Graham W Taylor. 2017.  Improved Regularization of Convolutional Neural Networks with Cutout.  _arXiv preprint arXiv:1708.04552_ (2017).
  * Dong and Yang (2020) Xuanyi Dong and Yi Yang. 2020.  NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/2001.00326.
  * Dowdell and Zhang (2020) Thomas Dowdell and Hongyu Zhang. 2020.  Language Modelling for Source Code with Transformer-XL.  _arXiv preprint arXiv:2007.15813_ (2020).
  * Elsken et al. (2017) Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. 2017\.  Simple and Efficient Architecture Search for Convolutional Neural Networks.  _arXiv preprint arXiv:1711.04528_ (2017).
  * Elsken et al. (2019) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019\.  Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/1804.09081.
  * Erguzel et al. (2014) Turker Tekin Erguzel, Serhat Ozekes, Selahattin Gultekin, and Nevzat Tarhan. 2014.  Ant Colony optimization Based Feature Selection Method for QEEG data classification.  _Psychiatry Investig._ 11, 3 (2014), 243.
  * Estévez and Caballero (1998) Pablo A Estévez and Rodrigo E Caballero. 1998.  A Niching Genetic Algorithm for Selecting Features for Neural Network Classifiers. In _Proc. Int. Conf. Artif. Neural Netw._ 311–316.
  * Evans et al. (2018a) Benjamin Evans, Harith Al-Sahaf, Bing Xue, and Mengjie Zhang. 2018a.  Evolutionary Deep Learning: A Genetic Programming Approach to Image Classification. In _Proc. IEEE Congr. Evol. Comput._ 1538–1545.
  * Evans et al. (2018b) Benjamin Patrick Evans, Harith Al-Sahaf, Bing Xue, and Mengjie Zhang. 2018b.  Evolutionary Deep Learning: A Genetic Programming Approach to Image Classification. In _Proc. IEEE Congr. Evol. Comput._ 1–6.
  * Fahrudin et al. (2016) Tresna Maulana Fahrudin, Iwan Syarif, and Ali Ridho Barakbah. 2016\.  Ant Colony Algorithm for Feature Selection on Microarray Datasets. In _International Electronics Symposium_. 351–356.
  * Fan et al. (2020) Zhun Fan, Jiahong Wei, Guijie Zhu, Jiajie Mo, and Wenji Li. 2020.  Evolutionary Neural Architecture Search for Retinal Vessel Segmentation.  _arXiv preprint arXiv:2001.06678_ (2020).
  * Fernandes and Yen (2021) Francisco Erivaldo Fernandes and Gary G. Yen. 2021.  Automatic Searching and Pruning of Deep Neural Networks for Medical Imaging Diagnostic.  _IEEE Trans. Neural Netw. Learn. Syst._ 32, 12 (2021), 5664–5674.
  * Fogelberg and Zhang (2005) Christopher Fogelberg and Mengjie Zhang. 2005.  Linear Genetic Programming for Multi-class Object Classification. In _Proc. Aust. Joint Conf. Artif.l Intell._ 369–379.
  * Fortuna and Frasca (2021) Luigi Fortuna and Mattia Frasca. 2021.  Singular Value Decomposition.  _Optim. Rob. Control_ (2021), 51–58.
  * Frachon et al. (2019) Luc Frachon, Wei Pang, and George M Coghill. 2019.  Immunecs: Neural Committee Search by an Artificial Immune System.  _arXiv preprint arXiv:1911.07729_ (2019).
  * Freitas (2003) Alex A Freitas. 2003\.  A Survey of Evolutionary Algorithms for Data Mining and Knowledge Discovery.  In _Adv. Evol. Comput._ Springer, 819–845.
  * Fujino et al. (2017) Saya Fujino, Naoki Mori, and Keinosuke Matsumoto. 2017\.  Deep Convolutional Networks for Human Sketches By Means of The Evolutionary Deep Learning. In _Proc. Int. Conf. Soft Comput. Intell. Syst._ 1–5.
  * García et al. (2011) David García, Antonio González Muñoz, and Raúl Pérez. 2011.  A Two-Step Approach of Feature Construction for A Genetic Learning Algorithm.  _Proc. Int. Conf. Fuzzy Syst._ (2011), 1255–1262.
  * Gerum et al. (2020) Richard C Gerum, André Erpenbeck, Patrick Krauss, and Achim Schilling. 2020.  Sparsity Through Evolutionary Pruning Prevents Neuronal Networks From Overfitting.  _Neural Netw._ 128 (2020), 305–312.
  * Golubski and Feuring (1999) Wolfgang Golubski and Thomas Feuring. 1999.  Evolving Neural Network Structures by Means of Genetic Programming. In _Proc. Eur. Conf. Genetic Program_. 211–220.
  * Gou et al. (2021) Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.  Knowledge Distillation: A Survey.  _Int. J. Comput. Vis._ 129, 6 (2021), 1789–1819.
  * Guo et al. (2020) Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2020.  Single Path One-Shot Neural Architecture Search with Uniform Sampling. In _Proc. Eur. Conf. Comput. Vis._ 544–560.
  * Hajati et al. (2010) Farshid Hajati, Caro Lucas, and Yongsheng Gao. 2010\.  Face Localization Using an Effective Co-evolutionary Genetic Algorithm.  _Proc. Int. Conf. Digit. Image Comput.: Tech. and Appl._ (2010), 522–527.
  * Hammami et al. (2018) Marwa Hammami, Slim Bechikh, and Chih-Cheng Hung. 2018\.  A Multi-Objective Hybrid Filter-Wrapper Evolutionary Approach for Feature Construction on High-Dimensional Data. In _Proc. IEEE Congr. Evol. Comput._ 1–8.
  * Han and Cho (2006) Sang-Jun Han and Sung-Bae Cho. 2006.  Evolutionary Neural Networks for Anomaly Detection Based on the Behavior of a Program.  _IEEE Trans. Syst. Man Cybern._ 36, 3 (2006), 559–570.
  * Hancer et al. (2015) Emrah Hancer, Bing Xue, Mengjie Zhang, and Dervis Karaboga. 2015\.  A Multi-objective Artificial Bee Colony Approach to Feature Selection Using Fuzzy Mutual Information. In _Proc. IEEE Congr. Evol. Comput._ 2420–2427.
  * He et al. (2021) Xin He, Kaiyong Zhao, and Xiaowen Chu. 2021.  AutoML: A Survey of the State-of-the-Art.  _Knowl-Based Syst_ 212 (2021), 106622.
  * He et al. (2017) Yihui He, Xiangyu Zhang, and Jian Sun. 2017.  Channel Pruning for Accelerating Very Deep Neural Networks. In _Proc. IEEE Int. Conf. Comput. Vis._ 1398–1406.
  * Ho et al. (2021) Kary Ho, Andrew Gilbert, Hailin Jin, and John P. Collomosse. 2021\.  Neural Architecture Search for Deep Image Prior.  _Comput. & Graph._ 98 (2021), 188–196.
  * Hong and Cho (2006) Jin-Hyuk Hong and Sung-Bae Cho. 2006.  Efficient Huge-Scale Feature Selection With Speciated Genetic Algorithm.  _Pattern Recognit. Lett._ 27, 2 (2006), 143–150.
  * Hong et al. (2020) Wenjing Hong, Peng Yang, Yiwen Wang, and Ke Tang. 2020\.  Multi-objective Magnitude-Based Pruning for Latency-Aware Deep Neural Network Compression. In _Proc. Int. Conf. on Parallel Probl. Solving Nat._ 470–483.
  * Hosni et al. (2020) Mohamed Hosni, Ginés García-Mateos, and Juan Carrillo-de Gea. 2020\.  A Mapping Study of Ensemble Classification Methods in Lung Cancer Decision Support Systems.  _Med. & Biol. Eng. & Comput._ 58, 10 (2020), 2177–2193.
  * Hsu et al. (2021) Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2021\.  Language Model Compression with Weighted Low-rank Factorization. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/2207.00112.
  * Hu et al. (2021b) Bin Hu, Tianming Zhao, Yucheng Xie, Yan Wang, and Xiaonan Guo. 2021b.  MIXP: Efficient Deep Neural Networks Pruning for Further FLOPs Compression via Neuron Bond. In _Proc. Int. Joint Conf. Neural Netw._ 1–8.
  * Hu et al. (2021a) Yiming Hu, Xingang Wang, Lujun Li, and Qingyi Gu. 2021a.  Improving One-Shot NAS with Shrinking-and-Expanding Supernet.  _Pattern Recognit._ 118 (2021), 108025.
  * Huang et al. (2012) Hu Huang, Hong-Bo Xie, Jing-Yi Guo, and Hui-Juan Chen. 2012\.  Ant Colony Optimization-based Feature Selection Method for Surface Electromyography Signals Classification.  _Comput. Biol. Med._ 42, 1 (2012), 30–38.
  * Huang et al. (2020) Junhao Huang, Weize Sun, and Lei Huang. 2020.  Deep Neural Networks Compression Learning Based on Multiobjective Evolutionary Algorithms.  _Neurocomputing_ 378 (2020), 260–269.
  * Ijjina and Chalavadi (2016) Earnest Paul Ijjina and Krishna Mohan Chalavadi. 2016.  Human Action Recognition Using Genetic Algorithms and Convolutional Neural Networks.  _Pattern Recognit._ 59 (2016), 199–212.
  * Irwin-Harris et al. (2019) William Irwin-Harris, Yanan Sun, Bing Xue, and Mengjie Zhang. 2019.  A Graph-Based Encoding for Evolutionary Convolutional Neural Network Architecture Design. In _Proc. IEEE Congr. Evol. Comput._ 546–553.
  * Izenman (2013) Alan Julian Izenman. 2013\.  Linear Discriminant Analysis.  In _Modern Multivariate Statistical Techniques_. Springer, 237–280.
  * Jaâfra et al. (2019) Yesmina Jaâfra, Jean Luc Laurent, Aline Deruyver, and Mohamed Saber Naceur. 2019.  Reinforcement Learning for Neural Architecture Search: A Review.  _Image Vis. Comput._ 89 (2019), 57–66.
  * Jin et al. (2018) Haifeng Jin, Qingquan Song, and Xia Hu. 2018.  Auto-keras: Efficient Neural Architecture Search with Network Morphism.  _arXiv preprint arXiv:1806.10282_ (2018).
  * Jin et al. (2019) Haifeng Jin, Qingquan Song, and Xia Hu. 2019.  Auto-Keras: An Efficient Neural Architecture Search System. In _Proc. ACM SIGKDD Int. Conf. on Knowl. Discov. & Data Min._ 1946–1956.
  * Jin (2006) Yaochu Jin. 2006\.  _Multi-Objective Machine Learning_.  Springer Science.
  * Jones et al. (2019) David T Jones, Anja Schroeder, and Geoff S. Nitschke. 2019\.  Evolutionary Deep Learning to Identify Galaxies in the Zone of Avoidance.  _arXiv preprint arXiv:1903.07461_ (2019).
  * Junior and Yen (2021a) Francisco Erivaldo Fernandes Junior and Gary G. Yen. 2021a.  Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy.  _Inf. Sci._ 552 (2021), 29–47.
  * Junior and Yen (2021b) Francisco Erivaldo Fernandes Junior and Gary G. Yen. 2021b.  Pruning of Generative Adversarial Neural Networks for Medical Imaging Diagnostics with Evolution Strategy.  _Inf. Sci._ 558 (2021), 91–102.
  * Karaboga et al. (2007) Dervis Karaboga, Bahriye Akay, and Celal Öztürk. 2007\.  Artificial Bee Colony (ABC) Optimization Algorithm for Training Feed-Forward Neural Networks. In _Proc. Int. Conf. Modeling Decis. Artif. Intell._ 318–329.
  * Karegowda and Manjunath (2011) Asha Gowda Karegowda and A. S. Manjunath. 2011.  Application of Genetic Algorithm Optimized Neural Network Connection Weights for Medical Diagnosis of PIMA Indians Diabetes.  _Int. J. Soft Comput._ 2, 2 (2011), 15–23.
  * Khadka and Tumer (2018) Shauharda Khadka and Kagan Tumer. 2018.  Evolution-Guided Policy Gradient in Reinforcement Learning. In _Proc. Adv. Neural Inf. Process. Syst._ , Vol. 31. 1196–1208.
  * Khushaba et al. (2008) Rami N Khushaba, Ahmed Al-Ani, Akram AlSukker, and Adel Al-Jumaily. 2008.  A Combined Ant Colony and Differential Evolution Feature Selection Algorithm. In _Proc. Int. Conf. Ant Colony Optim. Swarm Intell._ 1–12.
  * Kitano (1990) Hiroaki Kitano. 1990\.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System.  _Complex Syst._ 4, 4 (1990), 225–238.
  * Kotani and Kato (2004) Manabu Kotani and Daisuke Kato. 2004.  Feature Extraction Using Coevolutionary Genetic Programming. In _Proc. IEEE Congr. Evol. Comput._ 614–619.
  * Koutník et al. (2010) Jan Koutník, Faustino J. Gomez, and Jürgen Schmidhuber. 2010.  Evolving Neural Networks in Compressed Weight Space. In _Proc. Genetic Evol. Comput. Conf._ 619–626.
  * Koutník et al. (2014) Jan Koutník, Jürgen Schmidhuber, and Faustino J. Gomez. 2014.  Evolving Deep Unsupervised Convolutional Networks for Vision-Based Reinforcement Learning. In _Proc. Genetic Evol. Comput. Conf._ 541–548.
  * Krawiec (2002) Krzysztof Krawiec. 2002\.  Genetic Programming-Based Construction of Features for Machine Learning and Knowledge Discovery Tasks.  _Genet. Program Evolvable Mach._ 3, 4 (2002), 329–343.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning Multiple Layers of Features From Tiny Images.  (2009).
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  ImageNet Classification with Deep Convolutional Neural Networks. In _Proc. Adv. Neural Inf. Process. Syst._ , Vol. 25. 1097–1105.
  * Kwasigroch et al. (2019) Arkadiusz Kwasigroch, Michał Grochowski, and Mateusz Mikolajczyk. 2019.  Deep Neural Network Architecture Search using Network Morphism. In _Proc. Int. Conf. Methods and Models in Autom. and Robot._ 30–35.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep Learning.  _Nature_ 521, 7553 (2015), 436–444.
  * LeCun et al. (1989) Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, and Wayne Hubbard. 1989\.  Backpropagation Applied to Handwritten Zip Code Recognition.  _Neural computation_ 1, 4 (1989), 541–551.
  * Li et al. (2020) Bailin Li, Bowen Wu, Jiang Su, Guangrun Wang, and Liang Lin. 2020.  EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning. In _Proc. Eur. Conf. Comput. Vis._ 639–654.
  * Li et al. (2021) Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, Cong Hao, and Yingyan Lin. 2021\.  HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/2103.10584.
  * Li et al. (2022) Qing Li, Wei Zhang, Lin Zhao, Xia Wu, and Tianming Liu. 2022.  Evolutional Neural Architecture Search for Optimization of Spatiotemporal Brain Network Decomposition.  _IEEE. Trans. Biomed. Eng._ 69, 2 (2022), 624–634.
  * Li et al. (2019) Youru Li, Zhenfeng Zhu, Deqiang Kong, Hua Han, and Yao Zhao. 2019.  EA-LSTM: Evolutionary Attention-Based LSTM for Time Series Prediction.  _Knowl.-Based Syst._ 181 (2019), 104785.
  * Liu et al. (2018c) Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. 2018c.  Progressive Neural Architecture Search. In _Proc. Eur. Conf. Comput. Vis._ 19–34.
  * Liu et al. (2021a) Chia-Hsiang Liu, Yu-Shin Han, Yuan-Yao Sung, Yi Lee, Hung-Yueh Chiang, and Kai-Chiang Wu. 2021a.  FOX-NAS: Fast, On-device and Explainable Neural Architecture Search. In _Proc. IEEE Int. Conf. Comput. Vis._ 789–797.
  * Liu et al. (2018b) Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018b.  DARTS: Differentiable Architecture Search. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/1806.09055.
  * Liu et al. (2019a) Peng Liu, Mohammad D. El Basha, Yangjunyi Li, Yao Xiao, Pina C. Sanelli, and Ruogu Fang. 2019a.  Deep Evolutionary Networks with Expedited Genetic Algorithms for Medical Image Denoising.  _Med. Image Anal._ 54 (2019), 306–315.
  * Liu and Guo (2021) Sicong Liu and Bin Guo. 2021.  AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications. In _Proc. ACM Interact., Mobile, Wearable Ubiquitous Tech._ , Vol. 5. ACM, 1–22.
  * Liu et al. (2018a) Xiao-Ying Liu, Yong Liang, Sai Wang, Zi-Yi Yang, and Han-Shuo Ye. 2018a.  A Hybrid Genetic Algorithm With Wrapper-Embedded Approaches for Feature Selection.  _IEEE Access_ 6 (2018), 22863–22874.
  * Liu et al. (2021b) Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen, and Kay Chen Tan. 2021b.  A Survey on Evolutionary Neural Architecture Search.  _IEEE Trans. Neural Netw. Learn. Syst._ (2021).  https://doi.org/10.1109/TNNLS.2021.3100554
  * Liu et al. (2019b) Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, K. Cheng, and Jian Sun. 2019b.  MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning. In _Proc. IEEE Int. Conf. Comput. Vis._ 3295–3304.
  * Lomurno et al. (2021) Eugenio Lomurno, Stefano Samele, Matteo Matteucci, and Danilo Ardagna. 2021.  Pareto-optimal Progressive Neural Architecture Search. In _Proc. Genetic Evol. Comput. Conf._ 1726–1734.
  * Londt et al. (2021) Trevor Londt, Xiaoying Gao, and Peter Andreae. 2021\.  Evolving Character-level DenseNet Architectures Using Genetic Programming. In _Proc. Int. Conf. Appl. Evol. Comput._ 665–680.
  * Londt et al. (2020) Trevor Londt, Xiaoying Gao, Bing Xue, and Peter Andreae. 2020\.  Evolving Character-level Convolutional Neural Networks for Text Classification.  _arXiv preprint arXiv:2012.02223_ (2020).
  * Loni et al. (2020) Mohammad Loni, Sima Sinaei, and Ali Zoljodi. 2020\.  DeepMaker: A Multi-Objective Optimization Framework for Deep Neural Networks in Embedded Systems.  _Microprocess. Microsyst._ 73 (2020), 102989.
  * Lorenzo and Nalepa (2018) Pablo Ribalta Lorenzo and Jakub Nalepa. 2018.  Memetic Evolution of Deep Neural Networks. In _Proc. Genetic Evol. Comput. Conf._ 505–512.
  * Lorenzo et al. (2017) Pablo Ribalta Lorenzo, Jakub Nalepa, Luciano Sánchez Ramos, and José Ranilla. 2017.  Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization. In _Proc. Genetic Evol. Comput. Conf._ 1864–1871.
  * Lu et al. (2021) Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishnu Naresh Boddeti. 2021.  Neural Architecture Transfer.  _IEEE IEEE Trans. Pattern Anal. Mach. Intell._ 43, 9 (2021), 2971–2989.
  * Lu et al. (2019) Zhichao Lu, Ian Whalen, Vishnu Naresh Boddeti, Yashesh D. Dhebar, and Kalyanmoy Deb. 2019.  NSGA-Net: Neural Architecture Search using Multi-objective Genetic Algorithm. In _Proc. Genetic Evol. Comput. Conf._ 419–427.
  * Luo et al. (2018) Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2018.  Neural architecture optimization. In _Proc. Adv. Neural Inf. Process. Syst._ , Vol. 31. 7827–7838.
  * Ma et al. (2021b) Ailong Ma, Yuting Wan, Yanfei Zhong, Junjue Wang, and Liang pei Zhang. 2021b.  SceneNet: Remote Sensing Scene Classification Deep Learning Network Using Multi-Objective Neural Evolution Architecture Search.  _ISPRS J. Photogramm. Remote Sens._ 172 (2021), 171–188.
  * Ma et al. (2022) Lianbo Ma, Min Huang, Shengxiang Yang, Rui Wang, and Xingwei Wang. 2022.  An Adaptive Localized Decision Variable Analysis Approach to Large-Scale Multiobjective and Many-Objective Optimization.  _IEEE Trans. Cybern._ 52, 7 (2022), 6684–6696.
  * Ma et al. (2021a) Lianbo Ma, Nan Li, Guo Yu, Xiaoyu Geng, Min Huang, and Xingwei Wang. 2021a.  How to Simplify Search: Classification-wise Pareto Evolution for One-shot Neural Architecture Search.  _arXiv preprint arXiv:2109.07582_ (2021).
  * Ma et al. (2021c) Wenping Ma, Xiaobo Zhou, Hao Zhu, Longwei Li, and Licheng Jiao. 2021c.  A Two-stage Hybrid Ant Colony Optimization for High-dimensional Feature Selection.  _Pattern Recognit._ 116 (2021), 107933.
  * Maniezzo (1994) V. Maniezzo. 1994\.  Genetic Evolution of the Topology and Weight Distribution of Neural Networks.  _IEEE Trans. Neural. Netw._ 5, 1 (1994), 39–53.
  * Mauceri et al. (2021) Stefano Mauceri, James Sweeney, Miguel Nicolau, and James McDermott. 2021.  Feature Extraction by Grammatical Evolution for One-class Time Series Classification.  _Genet. Program. Evolvable Mach._ 22, 3 (2021), 267–295.
  * Mazzawi et al. (2019) Hanna Mazzawi, Xavi Gonzalvo, Aleksandar Kracun, and Prashant Sridhar. 2019.  Improving Keyword Spotting and Language Identification via Neural Architecture Search at Scale. In _INTERSPEECH_. 1278–1282.
  * Mei et al. (2017) Yi Mei, Su Nguyen, Bing Xue, and Mengjie Zhang. 2017\.  An Efficient Feature Selection Algorithm for Evolving Job Shop Scheduling Rules With Genetic Programming.  _IEEE Trans. Emerg. Topics Comput. Intell._ 1, 5 (2017), 339–353.
  * Miahi et al. (2022) Erfan Miahi, Seyed Abolghasem Mirroshandel, and Alexis Nasr. 2022.  Genetic Neural Architecture Search for Automatic Assessment of Human Sperm Images.  _Expert Syst. Appl._ 188 (2022), 115937.
  * Miikkulainen et al. (2019) Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. 2019\.  Evolving Deep Neural Networks.  In _Artificial Intelligence in the Age Of Neural Networks and Brain Computing_. Elsevier, 293–312.
  * Mirjalili et al. (2019) Seyedali Mirjalili, Hossam Faris, and Ibrahim Aljarah. 2019\.  _Evolutionary Machine Learning Techniques_.  Springer.
  * Mo et al. (2021) Hyunho Mo, Leonardo Lucio Custode, and Giovanni Iacca. 2021\.  Evolutionary Neural Architecture Search for Remaining Useful Life Prediction.  _Appl. Soft Comput._ 108 (2021), 107474.
  * Montana and Davis (1989) David J Montana and Lawrence Davis. 1989.  Training Feedforward Neural Networks Using Genetic Algorithms. In _Proc. of the Int. Joint Conf. Artif. Intell._ , Vol. 4. 762–767.
  * Muni et al. (2006) Durga Prasad Muni, Nikhil R Pal, and Jyotirmay Das. 2006\.  Genetic Programming for Simultaneous Feature Selection and Classifier Design.  _IEEE Trans. Syst. Man. Cybern._ 36, 1 (2006), 106–117.
  * Murray and Chiang (2015) Kenton Murray and David Chiang. 2015.  Auto-Sizing Neural Networks: With Applications to N-Gram Language Models. In _Proc. Conf. Empir. Methods Nat. Lang. Proc._ 908–916.
  * Nekrasov et al. (2020) Vladimir Nekrasov, Chunhua Shen, and Ian Reid. 2020.  Template-Based Automatic Search of Compact Semantic Segmentation Architectures. In _Proc. Winter Conf. Appl. Comput. Vis._ 1980–1989.
  * Neshat et al. (2020) Mehdi Neshat, Meysam Majidi Nezhad, Ehsan Abbasnejad, Lina Bertling Tjernberg, Davide Astiaso Garcia, Bradley Alexander, and Markus Wagner. 2020.  An Evolutionary Deep Learning Method for Short-term Wind Speed Prediction: A Case Study of the Lillgrund Offshore Wind Farm.  _arXiv preprint arXiv:abs/2002.09106_ (2020).
  * Neshatian et al. (2012) Kourosh Neshatian, Mengjie Zhang, and Peter Andreae. 2012\.  A Filter Approach to Multiple Feature Construction for Symbolic Learning Classifiers Using Genetic Programming.  _IEEE Trans. Evol. Comput._ 16, 5 (2012), 645–661.
  * Neshatian et al. (2007) Kourosh Neshatian, Mengjie Zhang, and Mark Johnston. 2007\.  Feature Construction and Dimension Reduction Using Genetic Programming. In _Proc. Aust. Conf. Artif. Intell._ 242–253.
  * Nguyen et al. (2014) Hoai Bach Nguyen, Bing Xue, Ivy Liu, and Mengjie Zhang. 2014\.  PSO and Statistical Clustering for Feature Selection: A New Representation. In _Proc. Asia-Pacific Conf. Simulated Evol. Learn._ 569–581.
  * O’Boyle and Palmer (2008) Noel M O’Boyle and David S Palmer. 2008.  Simultaneous Feature Selection and Parameter Optimisation Using An Artificial Ant Colony: Case Study of Melting Point Prediction.  _Chem. Cent. J._ 2, 1 (2008), 1–15.
  * Olson and Moore (2016) Randal S. Olson and Jason H. Moore. 2016.  TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning. In _Proc. Int. Conf. Mach. Learn._ 151–160.
  * O’Neill et al. (2018) Damien O’Neill, Bing Xue, and Mengjie Zhang. 2018.  Co-evolution of Novel Tree-Like ANNs and Activation Functions: An Observational Study. In _Proc. Aust. Conf. Artif. Intell._ 616–629.
  * Oong and Isa (2011) Tatt Hee Oong and Nor Ashidi Mat Isa. 2011.  Adaptive Evolutionary Artificial Neural Networks for Pattern Classification.  _IEEE Trans. Neural Netw._ 22, 11 (2011), 1823–1836.
  * Ortego et al. (2020) Patxi Ortego, Alberto Diez-Olivan, Javier Del Ser, and Fernando Veiga. 2020.  Evolutionary LSTM-FCN Networks for Pattern Classification in Industrial Processes.  _Swarm Evol. Comput._ 54 (2020), 100650.
  * Peng et al. (2021) Bo Peng, Shuting Wan, Ying Bi, Bing Xue, and Mengjie Zhang. 2021.  Automatic Feature Extraction and Construction Using Genetic Programming for Rotating Machinery Fault Diagnosis.  _IEEE Trans. Cybern._ 51, 10 (2021), 4909–4923.
  * Peng et al. (2018) Yiming Peng, Gang Chen, Harman Singh, and Mengjie Zhang. 2018\.  NEAT for Large-scale Reinforcement Learning Through Evolutionary Feature Learning and Policy Gradient Search. In _Proc. Genetic Evol. Comput. Conf._ 490–497.
  * Phan et al. (2020) Hai T. Phan, Zechun Liu, Dang The Huynh, Marios Savvides, Kwang-Ting Cheng, and Zhiqiang Shen. 2020\.  Binarizing MobileNet via Evolution-Based Searching. In _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ 13417–13426.
  * Polino et al. (2018) Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018\.  Model Compression via Distillation and Quantization. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/1802.05668.
  * Poyatos et al. (2022) Javier Poyatos, Daniel Molina, Aritz Martinez, et al. 2022\.  EvoPruneDeepTL: An Evolutionary Pruning Model for Transfer Learning based Deep Neural Networks.  _arXiv preprint arXiv:2202.03844_ (2022).
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving Language Understanding by Generative Pre-training.  (2018), https://www.cs.ubc.ca/ amuham01/LING530/papers/radford2018improving.pdf.
  * Rapaport et al. (2019) Elad Rapaport, Oren Shriki, and Rami Puzis. 2019\.  EEGNAS: Neural Architecture Search for Electroencephalography Data Analysis and Decoding. In _Proc. Int. Joint Conf. Artif. Intell._ 3–20.
  * Rashid et al. (2020) ANM Bazlur Rashid, Mohiuddin Ahmed, Leslie F Sikos, and Paul Haskell-Dowland. 2020.  Cooperative Co-Evolution for Feature Selection in Big Data With Random Feature Grouping.  _J. Big Data_ 7, 1 (2020), 1–42.
  * Rawal and Miikkulainen (2018) Aditya Rawal and Risto Miikkulainen. 2018.  From Nodes to Networks: Evolving Recurrent Neural Networks.  _arXiv preprint arXiv:1803.04439_ (2018).
  * Real et al. (2019) Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2019.  Regularized Evolution for Image Classifier Architecture Search. In _Proc. AAAI Conf. Artif. Intell._ , Vol. 33. 4780–4789.
  * Real et al. (2017) Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. 2017\.  Large-Scale Evolution of Image Classifiers. In _Proc. Int. Conf. Mach. Learn._ 2902–2911.
  * Refahi et al. (2020) Mohammad Saleh Refahi, A Mir, and Jalal A Nasiri. 2020\.  A Novel Fusion Based on the Evolutionary Features for Protein Fold Recognition Using Support Vector Machines.  _Sci. Rep._ 10, 1 (2020), 1–13.
  * Ren et al. (2021) Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, and Zhihui Li. 2021.  A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions.  _ACM Comput. Surv._ 54, 4 (2021), 1–34.
  * Roberts and Claridge (2005) Mark E. Roberts and Ela Claridge. 2005.  A Multistage Approach to Cooperatively Coevolving Feature Construction and Object Detection. In _Proc. Appl. Evol. Comput._ 396–406.
  * Rostami and Neri (2016) Shahin Rostami and Ferrante Neri. 2016.  Covariance Matrix Adaptation Pareto Archived Evolution Strategy With Hypervolume-Sorted Adaptive Grid Algorithm.  _Integr. Comput. Aided. Eng._ 23, 4 (2016), 313–329.
  * Rumelhart et al. (1985) David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1985.  _Learning Internal Representations by Error Propagation_.  Technical Report. California Univ San Diego La Jolla Inst for Cognitive Science.
  * Rumelhart et al. (1986) David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986.  Learning Representations by Back-propagating Errors.  _Nature_ 323, 6088 (1986), 533–536.
  * Rundo et al. (2019) Leonardo Rundo, Andrea Tangherloni, Marco S Nobile, Carmelo Militello, Daniela Besozzi, Giancarlo Mauri, and Paolo Cazzaniga. 2019\.  MedGA: A Novel Evolutionary Method for Image Enhancement in Medical Imaging Systems.  _Expert Syst. Appl._ 119 (2019), 387–399.
  * Samala et al. (2018) Ravi K. Samala, Heang-Ping Chan, Lubomir M. Hadjiiski, Mark A. Helvie, Caleb D. Richter, and Kenny H. Cha. 2018.  Evolutionary Pruning of Transfer Learned Deep Convolutional Neural Network For Breast Cancer Diagnosis In Digital Breast Tomosynthesis.  _Phys. Med. Biol._ 63, 9 (2018), 095005.
  * Santra et al. (2021) Santanu Santra, Jun-Wei Hsieh, and Chi-Fang Lin. 2021\.  Gradient Descent Effects on Differential Neural Architecture Search: A Survey.  _IEEE Access_ 9 (2021), 89602–89618.
  * Sapra and Pimentel (2020) Dolly Sapra and Andy D Pimentel. 2020.  Constrained Evolutionary Piecemeal Training to Design Convolutional Neural Networks. In _Proc. Int. Conf. Industr., Eng. and Other Appl. of App. Intell. Syst._ 709–721.
  * Schorn et al. (2020) Christoph Schorn, Thomas Elsken, Sebastian Vogel, and Armin Runge. 2020.  Automated Design Of Error-Resilient and Hardware-Efficient Deep Neural Networks.  _Neural. Comput. Appl._ 32, 24 (2020), 18327–18345.
  * Sciuto et al. (2020) Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Cristian Musat, and Mathieu Salzmann. 2020\.  Evaluating the Search Phase of Neural Architecture Search. In _Proc. Int. Conf. Learn. Represent._ https://arxiv.org/abs/1902.08142.
  * Shafti and Pérez (2008) Leila Shila Shafti and E. Islas Pérez. 2008.  Data Reduction by Genetic Algorithms and Non-Algebraic Feature Construction: A Case Study.  _Proc. Int. Conf. Hybri. Intell. Syst._ (2008), 573–578.
  * Shakya et al. (2021) Pratistha Shakya, Eamonn Kennedy, Christopher Rose, and Jacob K. Rotein. 2021.  High-Dimensional Time Series Feature Extraction for Low-Cost Machine Olfaction.  _IEEE Sens. J._ 21, 3 (2021), 2495–2504.
  * Shang et al. (2022) Haopu Shang, Jia-Liang Wu, Wenjing Hong, and Chao Qian. 2022\.  Neural Network Pruning by Cooperative Coevolution.  _arXiv preprint arXiv:2204.05639_ (2022).
  * Shen et al. (2019) Mingzhu Shen, Kai Han, Chunjing Xu, and Yunhe Wang. 2019\.  Searching for Accurate Binary Neural Architectures. In _Proc. IEEE Int. Conf. Comput. Vis._ 2041–2044.
  * Siems et al. (2020) Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. 2020\.  NAS-bench-301 and The Case for Surrogate Benchmarks for Neural Architecture Search.  _arXiv preprint arXiv:2008.09777_ (2020).
  * Sikdar et al. (2012) Utpal Kumar Sikdar, Asif Ekbal, and Sriparna Saha. 2012\.  Differential Evolution Based Feature Selection and Classifier Ensemble for Named Entity Recognition. In _Proc. COLING_. 2475–2490.
  * Smith and Jin (2014) Christopher Smith and Yaochu Jin. 2014.  Evolutionary Multi-objective Generation of Recurrent Neural Network Ensembles for Time Series Prediction.  _Neurocomputing_ 143 (2014), 302–311.
  * Socha and Blum (2007) Krzysztof Socha and Christian Blum. 2007.  An Ant Colony Optimization Algorithm for Continuous Optimization: Application to Feed-forward Neural Network Training.  _Neural. Comput. Appl._ 16, 3 (2007), 235–247.
  * Song et al. (2020) Dehua Song, Chang Xu, Xu Jia, Yiyi Chen, Chunjing Xu, and Yunhe Wang. 2020\.  Efficient Residual Dense Block Search for Image Super-Resolution. In _Proc. AAAI Conf. Artif. Intell._ , Vol. 34. 12007–12014.