# Cognitive Evolutionary Learning to Select Feature Interactions for
Recommender Systems

Runlong Yu, , Qixiang Shao, Qi Liu, , Huan Liu, ,
and Enhong Chen

###### Abstract

Feature interaction selection is a fundamental problem in commercial
recommender systems. Most approaches equally enumerate all features and
interactions by the same pre-defined operation under expert guidance. Their
recommendation is unsatisfactory sometimes due to the following issues: (1)
They cannot ensure the learning abilities of models because their
architectures are poorly adaptable to tasks and data; (2) Useless features and
interactions can bring unnecessary noise and complicate the training process.
In this paper, we aim to adaptively evolve the model to select appropriate
operations, features, and interactions under task guidance. Inspired by the
evolution and functioning of natural organisms, we propose a novel Cognitive
EvoLutionary Learning (CELL) framework, where cognitive ability refers to a
property of organisms that allows them to react and survive in diverse
environments. It consists of three stages, i.e., DNA search, genome search,
and model functioning. Specifically, if we regard the relationship between
models and tasks as the relationship between organisms and natural
environments, interactions of feature pairs can be analogous to double-
stranded DNA, of which relevant features and interactions can be analogous to
genomes. Along this line, we diagnose the fitness of the model on operations,
features, and interactions to simulate the survival rates of organisms for
natural selection. We show that CELL can adaptively evolve into different
models for different tasks and data, which enables practitioners to access
off-the-shelf models. Extensive experiments on four real-world datasets
demonstrate that CELL significantly outperforms state-of-the-art baselines.
Also, we conduct synthetic experiments to ascertain that CELL can consistently
discover the pre-defined interaction patterns for feature pairs.

###### Index Terms:

Feature selection, CTR prediction, evolutionary learning, nature inspired
computing, model diagnosis.

##  I Introduction

Accurate targeting of commercial recommender systems is of great importance,
in which feature interaction selection plays a key role. Nowadays, feature
interaction selection facilitates a variety of applications. For example, with
multi-billion dollar business on display advertising, Click-Through Rate (CTR)
prediction has received growing interest from both academia and industry
communities [1, 2, 3, 4]. Another emerging application is client identifying
in FinTech, i.e., intelligent financial advisors of online bank apps attempt
to predict whether users are willing to purchase their recommended portfolios
[5]. To support such services, it is necessary to well organize the massive
high-dimensional sparse user features. Since research has shown that
interaction of feature pairs can provide predictive abilities beyond what
those features can provide individually [6], this brings out the fundamental
research problem of feature interaction selection.

Dash and Liu [7] proposed a general feature selection framework that consists
of four steps, that is: 1) generation strategy; 2) evaluation criteria; 3)
stopping condition; 4) result validation. Based on the framework, researchers
developed various feature selection methods which usually fall into three
categories: 1) filter; 2) wrapper; 3) embedded. The disadvantages of filters
and wrappers lie in poor robustness, besides wrappers are inefficient and
unsuitable for large-scale datasets or high-dimensional data. In contrast,
embedded methods have become more popular because of their reduced
computational demands and less overfitting problems [8]. Expert-designed
operations to model interactions of feature pairs have always dominated extant
embedded methods, e.g., factorization machines (FM) [9] embed each feature
into low-dimensional vector and model interactions by the inner product. These
shallow models have limited representation capabilities, which inspires
implicit deep learning models to improve over FM [10, 11, 12]. However, these
approaches present the same pre-designed modeling architecture for different
tasks or datasets, which leads to a limitation that their architectures are
poorly adaptable to tasks and data. Besides, previous methods simply enumerate
all features and interactions by equal relevance, among which useless features
and interactions may bring unnecessary noise and complicate the training
process.

To this end, we argue that how to adaptively model task-friendly interactions
and quantitatively select relevant features and interactions has become a
less-studied urgent problem. We expect an ideal feature interaction selection
approach should satisfy two desirable abilities: (1) It can evolve to find a
proper operation to use for modeling each interaction; (2) It should
distinguish which features and interactions to what extent are relevant to the
task.

One way to implement such an evolution of a model is evolutionary learning
[13, 14, 15]. Evolutionary learning refers to a class of evolutionary
algorithms that solve complicated search problems in machine learning [16],
which is naturally fit for the feature selection framework [17, 18]. Extant
evolution-based feature selection methods perform filters and wrappers [13],
which are not practical for massive high-dimensional commercial data. In
contrast, we propose an evolution-based embedded method. For the concerned
feature interaction selection problem, if we regard the relationship between
models and tasks as the relationship between organisms and natural
environments, interactions of feature pairs can be analogous to double-
stranded deoxyribonucleic acid (DNA), of which relevant features and
interactions can be analogous to genomes. It is easy to understand that,
different traits confer different rates of survival and fitness. The same is
true for the selection of features and interactions. A key challenge to
accomplishing such evolutionary learning is evaluating the fitness of the
model. To cope with the challenge, we propose a fitness diagnosis technique
that can reveal the learning abilities of models during training. Compared
with previous fitness evaluation techniques, which commonly adopt numerical
values to represent the fitness of models, fitness diagnosis can deeply
diagnose the abilities of inside components of the model. By doing so, an
evolution path can be planned and visualized, thereby enhancing the
interpretability of how the model selects operations, features, and
interactions that suit the task better.

Along this line, this paper presents a Cognitive EvoLutionary Learning (CELL)
framework for feature interaction selection, where cognitive ability refers to
a property of organisms that allows them to react and survive in diverse
environments. The CELL framework consists of three stages of learning, i.e.,
DNA search, genome search, and model functioning. We summarize the three
stages in the following:

  1. 1.

Stage I: DNA search. We regard the relationship between features and
operations as the relationship between nucleotides and linkages. To explore
the fittest operation that generates a task-friendly interaction of each
feature pair, we extend the operation set with several types of operations as
the search space, which is like linkage rules to bind nucleotides. We diagnose
the fitness of the model on operations to simulate the survival rates of
organisms for natural selection. To tackle the evaluation-consuming challenge
of discrete selection, we relax the search space to be continuous so that
operation fitness can be optimized by gradient descent. For each feature pair,
we retain the fittest operation to model the interaction.

  2. 2.

Stage II: genome search. Since the genome is a small fraction of DNA that
contains genetic information and can influence the phenotype of an organism,
the genome search discards or weakens some features and interactions
contributing little, to prevent extra noise. To discriminate the relevance of
features and interactions, we diagnose the fitness of the model on features
and interactions. The search space is the relevance fitness parameters
represented by real values, indicating the contributions of features and
interactions to the task. Mutation, as the source of genetic variations,
occurs when the relevance fitness of an interaction drops to a threshold,
resulting in the operation of the interaction mutating into other operations.
The mutation mechanism can avoid the search results based on DNA search
reaching a suboptimum and benefit genetic diversity.

  3. 3.

Stage III: model functioning. The model functioning stage utilizes selected
relevant features and interactions to capture non-linear interactions further,
which is like decoding genes. In this stage, we concatenate retained features
and interactions as vectors and then feed them into multilayer perceptrons
(MLP), while keeping the relevance fitness as attention units.

We perform extensive experiments on four datasets. Three of them are publicly
available advertising datasets for CTR prediction, and the last one is a
financial dataset collected from a high-tech bank for identifying clients. To
enhance the interpretability of how CELL adaptively selects operations,
features, and interactions under task guidance, we visualize the evolution
paths of DNA search and genome search. Furthermore, we also conduct synthetic
experiments to ascertain that CELL can consistently diagnose to discover the
relevant interactions and the pre-defined interaction patterns.

To summarize, we make the following contributions:

  1. 1.

We propose a nature-inspired feature interaction selection approach named
CELL. It can adaptively evolve into different models for different tasks and
data, so as to build the architecture under task guidance.

  2. 2.

We propose a fitness diagnosis technique that can quantitatively analyze and
reveal the learning abilities of models during training. It deeply diagnoses
the abilities of inside components of models and interprets the mechanism of
interaction modeling and selection.

  3. 3.

We use three publicly available advertising datasets and a new dataset
collected from a high-tech bank to perform experiments. The experimental
results show that CELL outperforms the state-of-the-art feature interaction
selection models on these tasks.

Overview. The remainder of this article is organized as follows. In Section
II, we will review some related work of our study. Section III will introduce
the notations, problem definition, and basic operations for generating
interactions of feature pairs. Then, the framework of our proposed CELL and
the three stages of learning processes will be detailed in Section IV.
Afterward, Section V will comprehensively evaluate CELL on real-world datasets
and synthetic datasets. Finally, conclusions will be drawn in Section VI.

##  II Related Work

The related work of our study can be grouped into three categories, namely
feature interaction selection, evolutionary learning, and diagnosis
techniques. In this section, we summarize the related work as follows.

###  II-A Feature Interaction Selection

Since research has shown that interaction of feature pairs can provide
predictive abilities beyond what those features can provide individually [6,
13], feature interaction selection based on embedded methods attracts much
participation from commercial recommendations, e.g., advertising, client
identifying [19, 20, 21, 5, 22]. In the earlier time, scholars try to design
operations to model interactions in an explicit learning way. For example,
factorization machines (FM) [9] project each feature into low-dimensional
vector and model interactions by the inner product. Field-aware FM (FFM) [23]
enables each feature to have multiple latent vectors to interact with features
from different fields. However, these models have limited representation
capabilities, which inspires implicit deep learning models to improve over FM.
For example, Attention FM (AFM) [11] and Neural FM (NFM) [10] stack deep
neural networks on top of the output of the FM to model higher-order
interactions. FNN uses FM to pre-train low-order interactions and then feeds
embeddings into an MLP [24]. IPNN (also known as PNN) also uses the
interaction results of the FM layer but does not rely on pre-training [25,
26]. Obviously, these models lack good interpretability. Lian et al. argue
that implicit deep learning models focus more on high-order cross features but
capture little low-order cross features [27]. Therefore, recent advances
propose a hybrid network structure, namely Wide&Deep, which combines a shallow
component and a deep component with the purpose of learning both memorization
and generalization [28, 12, 29]. Wide&Deep framework attracts industry
partners from the beginning. As for the first Wide&Deep model proposed by
Google, it combines a linear model and an MLP [28]. Later on, DeepFM uses an
FM layer to replace the shallow part [12]. Similarly, Deep&Cross [29] and
xDeepFM [27] take the outer product of features at the bit- and vector-wise
level respectively. Though achieving some success, most of them follow a
manner, which uses the pre-designed modeling architecture and equally
enumerates all features and interactions, which suffer from two main problems.
First, they cannot ensure the learning abilities of models because their
architectures are poorly adaptable to tasks and data. Second, useless features
and interactions can bring unnecessary noise and complicate the training
process.

###  II-B Evolutionary Learning

Evolutionary learning refers to a class of nature-inspired heuristic
algorithms that solve complicated search problems in machine learning [30, 14,
31], which commonly be used in all three parts: preprocessing (e.g., feature
selection and resampling), learning (e.g., parameter setting, membership
functions, and neural network topology), and postprocessing (e.g., rule
optimization, decision tree/support vectors pruning, and ensemble learning)
[17, 32, 33, 34]. Evolutionary learning also refers to evolutionary AutoML
concepts, in which different components of models are automatically determined
using evolutionary algorithms, such as architecture and hyperparameters [16].
Concepts of evolutionary learning are naturally fit for the feature selection
framework [13, 17, 18, 16]. Previous evolution-based feature selection methods
commonly perform filters and wrappers [13], which are not practical for
massive high-dimensional commercial data. Recent approaches have employed
AutoML to search neural architectures for CTR prediction [35, 36, 37].
However, they are almost all built on existing work as blocks with complex
functionality, such as MLP and FM, where each block is an architecture-level
algorithm [35, 36]. In general, for population-based NAS approaches, large
population size and massive generations are usually required to address the
huge search space issue, e.g., AutoCTR sets population size as 100, and every
individual search runs on a single GPU for days [36]. In contrast to searching
coarse-grained upper-level architectures, our work searches fine-grained
basic-level operations. To the best of our knowledge, CELL is the first work
that utilizes the meta-heuristic mutation mechanism for interaction search.

###  II-C Diagnosis Techniques

In educational psychology, cognitive diagnosis techniques have been developed
to access examinees’ skill proficiency and can be roughly divided into two
categories: continuous and discrete [38, 39]. These works follow the
Monotonicity Theorem that assumes that examinees’ proficiency increases
monotonously with probabilities of giving the right responses to test items
[40]. Furthermore, recent works leverage diagnosis techniques to reveal the
proficiency of trial lawyers in legal fields [41]. Also, research shows that
one can predict the outcome of e-sports matches through the diagnosis of
players [42]. However, previous diagnosis techniques are mostly oriented
toward human beings. Our work inherits the idea of both continuous and
discrete diagnosis techniques but changes whom to diagnose from examinees to
learning models. Noticeable improvement in machine learning benefits from the
inspiration of the way of human learning, whereas fitness evaluation of
machine learning models comes to a standstill. In contrast, our work provides
a fitness diagnosis targeting machine learning models to understand their
learning status during training. We diagnose the fitness of the model to
simulate the survival rates of organisms for natural selection, thereby an
evolution path can be planned and visualized. It interprets the mechanism of
interaction modeling and selection.

##  III Preliminaries

In this section, we state the formal problem definition of feature interaction
selection and introduce some basic operations for generating interactions of
feature pairs.

###  III-A Problem Statement.

We define a general form of feature interaction selection problem. If the
dataset consists of $n$ instances $(\boldsymbol{f},y)$, where
$\boldsymbol{f}=[f_{1},\dots,f_{m}]$ indicates instance features including $m$
fields, and $y\in\\{1,0\\}$ indicates an observed user behavior, the feature
interaction selection problem can be defined as how to precisely give the
predictive result through the learned model
$\hat{y}:\mathcal{M}(\boldsymbol{f},\boldsymbol{g}(\boldsymbol{f}))\mapsto[0,1]$,
where $\boldsymbol{g}$ denotes the set of operations between feature pairs,
and $\boldsymbol{g}(\boldsymbol{f})$ denotes the set of interactions. Usually,
the prediction model $\mathcal{M}$ suffers a Logloss (cross-entropy loss)
function [43], given as:

$\mathcal{L}(\mathcal{M})=-\frac{1}{|B|}\sum_{t\in
B}y_{t}\log(\hat{y_{t}})+(1-y_{t})\log(1-\hat{y_{t}}),$ (1)

where $B$ denotes the set of instance indices in a mini-batch, $\hat{y}$
denotes the predictive result given through the learned model.

Figure 1: An illustration of the CELL framework. Inputs include the feature
set and the operation set. Stage I: DNA search explores different operations
to model the interaction of feature pairs. Stage II: genome search discards or
weakens some features and interactions contributing little, and remodels some
interactions by mutating their operations. Stage III: the model functioning
stage utilizes selected relevant features and interactions to capture non-
linear interactions further.

###  III-B Operations.

As the fundamental components in feature interaction selection, operations are
regarded as functions where two individual features are converted into an
interaction. For the sake of simplicity, we adopt four representative
operations, i.e., $\boldsymbol{g}=\\{\oplus,\otimes,\boxplus,\boxtimes\\}$, to
present an instantiation of CELL, which are highly used in previous work [35,
36, 37]. As shown in the operation set part of Fig. 1, the following
operations are available for selection:

  1. 1.

Element-wise sum ($\oplus$): It takes two input vectors of dimension $|f|$ and
outputs a vector of dimension $|f|$ that contains their element-wise sum. It
has no parameters.

  2. 2.

Element-wise product ($\otimes$): It takes two input vectors of dimension
$|f|$ and outputs a vector of dimension $|f|$ that contains their element-wise
product. It has no parameters.

  3. 3.

Concatenation and a feed-forward layer ($\boxtimes$): It takes two input
vectors of dimension $|f|$, concatenates them, and passes them through a feed-
forward layer with $\rm{ReLU}$ activation functions to reduce the dimension of
the output vector to $|f|$.

  4. 4.

Element-wise product and a feed-forward layer ($\boxplus$): It takes two input
vectors of dimension $|f|$, passes their element-wise product through a feed-
forward layer with $\rm{ReLU}$ activation functions to output a vector of
dimension $|f|$.

The complexities of $\oplus$ and $\otimes$ are both $O(1)$. The complexities
of $\boxtimes$ and $\boxplus$ are both $O(|f|)$. In practice, we
simultaneously optimize operations $\boxtimes$, $\boxplus$ and feature
embedding $\boldsymbol{f}$. Thus, for an interaction of a feature pair, the
complexity is $O(|f|)$.

##  IV Cognitive Evolutionary Learning

In this section, we will introduce the framework of CELL, which consists of
three stages of learning, i.e., DNA search, genome search, and model
functioning. The overall framework is illustrated in Fig. 1. The core idea of
CELL is to adaptively model task-friendly interactions and quantitatively
select relevant features and interactions so as to build the architecture
under task guidance, which is inspired by the evolution and functioning of
natural organisms. We consider a feature interaction selection process as a
multi-stage iterative search process, which can be viewed as a natural
organism striving to evolve better traits for higher rates of fitness. The
traits of an organism can be inherited via two aspects, i.e., the structure of
DNA and the genome regions of DNA. We diagnose the fitness of the model to
simulate the survival rates of organisms for natural selection, which
interprets the mechanism of interaction modeling and selection. The three
stages are detailed successively as follows.

###  IV-A DNA Search

We regard the relationship between features and operations as the relationship
between nucleotides and linkages, so the evolution of DNA is expected to
explore the fittest operation that generates a task-friendly interaction of
each feature pair. Following various linkages of nucleotides within DNA, we
extend the operation set with four types of operations as the search space,
i.e., $\boldsymbol{g}=\\{\oplus,\otimes,\boxplus,\boxtimes\\}$. If $g_{k}$ is
a candidate operation from the operation set $\boldsymbol{g}$, an interaction
$g_{k}(f_{i},f_{j})$ is modeled by the operation $g_{k}$ applied to a feature
pair $(f_{i},f_{j})$. Along this line, the model can be analogous to an
organism, of which traits can be affected by the linkage of nucleotides. It
responds to instance predictions, just like the organism responds to natural
environments. Our intuitive goal here is to diagnose the fitness of the model
on operations through the difference between the linkage responses and true
labels, as elaborated below:

Let a set of continuous variables
$\boldsymbol{\theta}=\\{\boldsymbol{\theta}^{(i,j)}|1\leqslant i<j\leqslant
m\\}$ parameterize the fitness of the model on operations. Our diagnosis goal
is to learn
$\boldsymbol{\theta}^{(i,j)}=\\{\theta_{g_{k}}^{(i,j)}|g_{k}\in\boldsymbol{g}\\}$
for each feature pair $(f_{i},f_{j})$ of an instance.

For an organism, its fitness increase monotonously with probabilities of the
DNA decoding vital traits. Analogously, we assume that the fitness of the
model increase monotonously with probabilities of interacting feature pairs by
suitable operations. Motivated by this, we put forward that
$\boldsymbol{\theta}^{(i,j)}$ satisfies the following property:

######  Proposition IV.1.

$\forall g_{k},g_{k}^{\prime}\in\boldsymbol{g}$, $\boldsymbol{\theta}^{(i,j)}$
satisfies $\theta_{g_{k}}^{(i,j)}>\theta_{g_{k}^{\prime}}^{(i,j)}$ when
$\mathcal{L}(\mathcal{M}(g_{k}(f_{i},f_{j})))<\mathcal{L}(\mathcal{M}(g_{k}^{\prime}(f_{i},f_{j})))$.

For the sake of simplicity, we use $\mathcal{M}(g_{k}(f_{i},f_{j}))$ to denote
the learning model that selects the operation $g_{k}$ to interact the feature
pair $(f_{i},f_{j})$. $\mathcal{L}(\mathcal{M})$ is the suffered Logloss
function as we mentioned before. The above proposition provides us with a way
to optimize the relative value of the model fitness on operations. There are
obvious advantages to implementing DNA search with continuous fitness rather
than categorical choices. If we search over discrete sets of interactions
modeled by candidate operations, it will require much more evaluation.
Contrastively, we relax the categorical choices to be continuous so that the
set of operations can be optimized with respect to its performance on the
validation set by gradient descent. As opposed to inefficient black-box
search, the data efficiency of gradient-based optimization allows DNA search
to use much fewer computational resources.

Inspired by DARTS algorithm [44], for each feature pair, we adopt a mixed
interaction $\bar{g}(f_{i},f_{j})$ given by a softmax over all possible
operations as:

$\bar{g}(f_{i},f_{j})=\sum_{g_{k}\in\boldsymbol{g}}\frac{\exp\left(\theta_{g_{k}}^{(i,j)}\right)}{\sum_{g_{l}\in\boldsymbol{g}}\exp\left(\theta_{g_{l}}^{(i,j)}\right)}g_{k}(f_{i},f_{j}),$
(2)

where the coefficient of candidate operations is called mixed linkage. We
define the linkage response as the weighted sum of all feature pairs
interacted by mixed linkages:

$\hat{y}=\mathcal{M}(\boldsymbol{w}\cdot\boldsymbol{g}(\boldsymbol{f}))={\rm{Sigmoid}}\bigg{(}\sum_{1\leqslant
i<j\leqslant m}w_{i,j}\cdot\bar{g}(f_{i},f_{j})\bigg{)},$ (3)

where $w_{i,j}$ is the weight parameter of $\bar{g}(f_{i},f_{j})$. If we
consider an interaction given by the mixed linkage between a feature pair as a
DARTS process, the linkage response of all feature pairs is given by multiple
DARTS processes (practically hundreds and thousands of them) that run in
parallel.

According to Proposition IV.1, DNA search can optimize $\boldsymbol{\theta}$
via the difference between the linkage responses and true labels. We adopt to
jointly optimize $\boldsymbol{\theta}$ and $\boldsymbol{w},\boldsymbol{f}$ as
a bilevel optimization problem. Hence, our goal is to find
$\boldsymbol{\theta}^{*}$ that minimize the validation loss
$\mathcal{L}_{val}(\boldsymbol{w}^{*},\boldsymbol{f}^{*},\boldsymbol{\theta}^{*})$,
where $\boldsymbol{w}^{*},\boldsymbol{f}^{*}$ associated with the operations
are obtained by minimizing the training loss
$\boldsymbol{w}^{*},\boldsymbol{f}^{*}=\arg\min_{\boldsymbol{w},\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}^{*})$.
This implies that $\boldsymbol{\theta}$ is the upper-level variable and
$\boldsymbol{w},\boldsymbol{f}$ are the lower-level variables:

$ \begin{aligned}
&\min_{\boldsymbol{\theta}}\mathcal{L}_{val}\left(\boldsymbol{w}^{*}(\boldsymbol{\theta}),\boldsymbol{f}^{*},\boldsymbol{\theta}\right),\\\
{\rm{s.t.}}\quad&\boldsymbol{w}^{*}(\boldsymbol{\theta}),\boldsymbol{f}^{*}=\arg\min_{\boldsymbol{w},\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}).\end{aligned}$
(4)

To reduce the expensive inner optimization in Eq. (4), we can use a simple
approximation that assumes the current $\boldsymbol{w},\boldsymbol{f}$ are the
same as $\boldsymbol{w}^{*},\boldsymbol{f}^{*}$. Related techniques have been
used in DARTS [44], meta-learning for model transfer [45], gradient-based
hyperparameter tuning [46] and unrolled generative adversarial networks [47].

Through the approximation, we can replace
$\boldsymbol{w}^{*}(\boldsymbol{\theta}),\boldsymbol{f}^{*}$ by adapting
$\boldsymbol{w},\boldsymbol{f}$ using only a single training step, without
solving the inner optimization completely by training until convergence, given
as:

$\displaystyle\nabla_{\boldsymbol{\theta}}\mathcal{L}_{val}\left(\boldsymbol{w}^{*}(\boldsymbol{\theta}),\boldsymbol{f}^{*},\boldsymbol{\theta}\right)\approx$
$\displaystyle\nabla_{\boldsymbol{\theta}}\mathcal{L}_{val}\big{(}\boldsymbol{w}-\xi\nabla_{\boldsymbol{w}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}),$
(5)
$\displaystyle\boldsymbol{f}-\xi\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}),\boldsymbol{\theta}\big{)},$

where $\xi$ is the learning rate. Eq. (5) requires only a forward pass for
$\boldsymbol{w},\boldsymbol{f}$ and a backward pass for $\boldsymbol{\theta}$,
which is much more efficient. At the end of DNA search, we replace
$\bar{g}(f_{i},f_{j})$ with the interaction modeled by the fittest operation
and obtain a discrete choice of operation, i.e.,
$g=\arg\max_{g_{k}\in\boldsymbol{g}}\theta_{g_{k}}^{(i,j)}$.

The overall time complexity of DNA search is practicable owing to the real-
parameter model fitness mechanism and the approximation method. Compared to
searching over discrete sets of candidate operations, the complexity is
reduced from
$O(|\boldsymbol{w}|+4m\cdot|\boldsymbol{f}|\cdot|\boldsymbol{\theta}|)$ to
$O(|\boldsymbol{w}|+4m\cdot|\boldsymbol{f}|+|\boldsymbol{\theta}|)$, where
$|\boldsymbol{w}|=m\cdot(m-1)/2$, $|\boldsymbol{f}|=m\cdot|f|$, and
$|\boldsymbol{\theta}|=2m\cdot(m-1)$. Note that Eq. (5) will reduce to
$\nabla_{\boldsymbol{\theta}}\mathcal{L}_{val}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta})$
if $\boldsymbol{w},\boldsymbol{f}$ are already local optima, because
$\nabla_{\boldsymbol{w}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta})=0$
and
$\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta})=0$.

###  IV-B Genome Search

DNA search provides us a way to select appropriate operations from the
operation set to model interactions of feature pairs. Regarding the model as
an organism, in DNA search, the nucleotides evolve to be bound by linkages so
that nucleotide chains become a double helix structure. Since the genome is a
small fraction of DNA that contains genetic information and can influence the
phenotype of an organism, our next goal is to find genome regions in DNA.

For an organism, if the regions in DNA decodes a phenotype that benefits
survival, it will have better fitness; if the phenotype decoded by the regions
does not benefit survival, it will have worse fitness. The evolutionary
strategy should allow the former (i.e., the genome) to retain phenotype
genetic information, which inspires us to formulate the importance of features
and interactions with the relevance fitness parameters. Therefore, our
intuitive goal here is to discriminate the relevance of features and
interactions, so as to enhance the relevant features and interactions,
meanwhile, discard or weaken some features and interactions contributing
little.

Let $\boldsymbol{\alpha}=\\{\alpha_{i}|1\leqslant i\leqslant m\\}$ and
$\boldsymbol{\beta}=\\{\beta_{i,j}|1\leqslant i<j\leqslant m\\}$ denote the
relevance fitness parameters of features $\boldsymbol{f}$ and interactions
$\boldsymbol{g(f)}$. Our diagnosis goal is to learn $\alpha_{i}$ for each
feature $f_{i}$ and $\beta_{i,j}$ for each interaction $g(f_{i},f_{j})$ of an
instance. We put forward that $\boldsymbol{\alpha},\boldsymbol{\beta}$ satisfy
the following property:

######  Proposition IV.2.

Let
$\boldsymbol{\alpha},\boldsymbol{\beta}=\arg\min_{\boldsymbol{\alpha},\boldsymbol{\beta}}\mathcal{L}(\mathcal{M})$.
$\boldsymbol{\alpha}$ satisfies $|\alpha_{i}|>|\alpha_{j}|$ when
$\mathcal{L}(\mathcal{M}(f_{i},w/o(f_{j})))<\mathcal{L}(\mathcal{M}(w/o(f_{i}),f_{j}))$,
and $\boldsymbol{\beta}$ satisfies $|\beta_{i,j}|>|\beta_{k,l}|$ when
$\mathcal{L}(\mathcal{M}(g(f_{i},f_{j}),w/o(g(f_{k},f_{l}))))<\mathcal{L}(\mathcal{M}(w/o(g(f_{i},f_{j})),g(f_{k},f_{l})))$.

For the sake of simplicity, in the above proposition, we use
$\mathcal{L}(\mathcal{M}(f_{i},w/o(f_{j})))$ to denote a comparative form of
the learning model that selects feature $f_{i}$ and discards feature $f_{j}$.
Analogously, $\mathcal{L}(\mathcal{M}(g(f_{i},f_{j}),w/o(g(f_{k},f_{l}))))$
denotes the learning model that selects interaction $g(f_{i},f_{j})$ and
discards feature $g(f_{k},f_{l})$. Indeed, the core idea of fitness diagnosis
in genome search can be viewed as a comparative survival scheme, which
enhances the relevant features and interactions but discards or weakens others
through the difference between the genome responses and true labels.

Along this line, the genome response of the current learning model can be
given as follows:
$\hat{y}=\mathcal{M}(\boldsymbol{\alpha}\cdot\boldsymbol{f},\boldsymbol{\beta}\cdot\boldsymbol{g}(\boldsymbol{f}))={\rm{Sigmoid}}\left(\sum_{i=1}^{m}\alpha_{i}\cdot
f_{i}+\sum_{1\leqslant i<j\leqslant m}\beta_{i,j}\cdot g(f_{i},f_{j})\right)$,
where the $\alpha_{i}$ is the relevance fitness parameter of feature $f_{i}$,
and $\beta_{i,j}$ is the relevance fitness parameter of interaction
$g(f_{i},f_{j})$. We propose to optimize $\boldsymbol{\alpha}$,
$\boldsymbol{\beta}$ simultaneously with feature embeddings. Specifically,
feature embeddings are learned by Adam optimizer [48], while
$\boldsymbol{\alpha}$, $\boldsymbol{\beta}$ are learned by regularized dual
averaging (RDA) optimizer [49, 50]. RDA optimizer is an online optimization
method aiming at getting sparse solutions. The reason why we can guarantee
sparse solutions of $\boldsymbol{\alpha}$, $\boldsymbol{\beta}$ is because of
the truncation mechanism of the RDA optimizer. When the absolute value of the
cumulative gradient average value in a certain dimension is less than a
threshold, the weight of that dimension will be set to 0, resulting in the
sparsity of the relevance fitness parameters [49, 51]. We update
$\boldsymbol{\alpha}$, $\boldsymbol{\beta}$ at each gradient step $t$ with
data $B_{t}$ as:

$\boldsymbol{\alpha}_{t+1},\boldsymbol{\beta}_{t+1}=\mathcal{S}_{h(t,\gamma)}\bigg{\\{}(\boldsymbol{\alpha}_{0},\boldsymbol{\beta}_{0})-\gamma\sum_{i=0}^{t}\nabla\mathcal{L}(\boldsymbol{\alpha}_{i},\boldsymbol{\beta}_{i};B_{i})\bigg{\\}},$
(6)

where $\mathcal{S}_{h}:v\mapsto\text{sign}(v)\cdot\max\\{|v|-h,0\\}$ is the
soft-thresholding operator, $\boldsymbol{\alpha}_{0},\boldsymbol{\beta}_{0}$
are initializers chosen at random, $\gamma$ is the learning rate,
$h(t,\gamma)=c\gamma^{1/2}(t\gamma)^{\mu}$ is the tuning function, $c$ and
$\mu$ are adjustable hyper-parameters as a trade-off between accuracy and
sparsity. To avoid the expensive inner optimization of the gradient of
$\boldsymbol{\alpha}$, $\boldsymbol{\beta}$ and feature embeddings, the
parameters are updated together using one-level optimization with gradient
descent on the training set by descending on $\boldsymbol{\alpha}$,
$\boldsymbol{\beta}$ and $\boldsymbol{f}$ based on:

$\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{f}_{t-1},\boldsymbol{\alpha}_{t-1},\boldsymbol{\beta}_{t-1})$
(7)
$\text{and}\;\;\nabla_{\boldsymbol{\alpha,\beta}}\mathcal{L}_{train}(\boldsymbol{f}_{t-1},\boldsymbol{\alpha}_{t-1},\boldsymbol{\beta}_{t-1}).$
(8)

In this setting, $\boldsymbol{\alpha}$, $\boldsymbol{\beta}$ and
$\boldsymbol{f}$ can explore their search space freely until convergence. The
complexity is
$O(m\cdot|\boldsymbol{f}|+|\boldsymbol{\alpha}|+|\boldsymbol{\beta}|)$, where
$|\boldsymbol{f}|=m\cdot|f|$, $|\boldsymbol{\alpha}|=m$, and
$|\boldsymbol{\beta}|=m\cdot(m-1)/2$.

Algorithm 1 Cognitive Evolutionary Learning

Input: Training dataset of $n$ instances $(\boldsymbol{f},y)$, operation set
$\boldsymbol{g}$.

1:Randomly initialize fitness parameters
$\boldsymbol{\theta},\boldsymbol{\alpha},\boldsymbol{\beta}$.

2:procedure DNA Search($\boldsymbol{\theta}$)

3: while not converged do

4: Update model fitness $\boldsymbol{\theta}$ by descending
$\nabla_{\boldsymbol{\theta}}\mathcal{L}_{val}\big{(}\boldsymbol{w}-\xi\nabla_{\boldsymbol{w}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}),\boldsymbol{f}-\xi\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}),\boldsymbol{\theta}\big{)}$

5: Update weights $\boldsymbol{w}$ and feature embeddings $\boldsymbol{f}$ by
descending
$\nabla_{\boldsymbol{w}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta}),\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f},\boldsymbol{\theta})$

6: end while

7: return the selected operations
$g=\arg\max_{g_{k}\in\boldsymbol{g}}\theta_{g_{k}}^{(i,j)}$

8:end procedure

9:

10:procedure Genome Search($\boldsymbol{\alpha},\boldsymbol{\beta}$)

11: while not converged do

12: Update the relevance fitness $\boldsymbol{\alpha},\boldsymbol{\beta}$ by
descending
$\nabla_{\boldsymbol{\alpha,\beta}}\mathcal{L}_{train}(\boldsymbol{f}_{t-1},\boldsymbol{\alpha}_{t-1},\boldsymbol{\beta}_{t-1})$
$\triangleright$ Use RDA optimizer

13: Update feature embeddings $\boldsymbol{f}$ by descending
$\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{f}_{t-1},\boldsymbol{\alpha}_{t-1},\boldsymbol{\beta}_{t-1})$

14: if $\beta_{i,j}<\lambda$ for every $\tau$ steps then

15: $g_{k}$ of $g_{k}(f_{i},f_{j})$ probabilistically mutates

16: end if

17: end while

18: return the relevance fitness parameters
$\boldsymbol{\alpha},\boldsymbol{\beta}$

19:end procedure

20:

21:procedure Model Functioning($\boldsymbol{\alpha},\boldsymbol{\beta}$)

22: while not converged do

23: Update weights $\boldsymbol{w}$ of the MLP and $\boldsymbol{f}$ by
descending
$\nabla_{\boldsymbol{w}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f}),\nabla_{\boldsymbol{f}}\mathcal{L}_{train}(\boldsymbol{w},\boldsymbol{f})$

24: end while

25: return the final predictive model

26:end procedure

To further explore interactions modeled by operations between features, we
propose a mutation mechanism, inspired by the source of genetic variations.
Mutation probabilistically occurs when the relevance fitness drops to a
threshold, resulting in the operation of the interaction mutating into other
operations. In general, the mutation mechanism can avoid the search result
based on DNA search reaching a suboptimum and benefit the genetic diversity.

When $\beta_{i,j}$ drops to a threshold $\lambda$ for every $\tau$ steps, the
mutation is applied with probability $1/\sigma$, which means that, to
regenerate a new interaction, the operation $g_{k}$ of the interaction
$g_{k}(f_{i},f_{j})$ mutates into another operation $g_{l}$, given as:

$g_{k}=\left\\{\begin{aligned} g_{l}\;\text{with
probability}\,1/\sigma,\;\;\;\;&\textbf{if}\quad\beta_{i,j}<\lambda,\\\
g_{k},\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;&\textbf{otherwise}.\end{aligned}\right.$
(9)

where $g_{l}$ is randomly selected from the operation set as
$g_{l}=\\{g\,|\,g\in\boldsymbol{g},g\neq g_{k}\\}$. After the mutation, the
new interaction $g_{l}(f_{i},f_{j})$ replaces the old irrelevant interaction
$g_{k}(f_{i},f_{j})$ and participates in next $\tau$ steps of genome search.

In certain sense, genome search makes a good balance between exploitation and
exploration of interactions, because it enhances relevant features and
interactions, meanwhile it remodels irrelevant interactions by mutating their
operations.

###  IV-C Model Functioning

Through repeated replication and transcription, the genetic information of DNA
translates to make matching protein sequences. In this way, various possible
functions could be proposed for an organism. To simulate this natural process,
we retrain the model. Relevant features and interactions are selected
according to their relevance fitness parameters $\boldsymbol{\alpha}$,
$\boldsymbol{\beta}$. If $\alpha_{i}=0$ or $\beta_{i,j}=0$, the corresponding
individual features or interactions are fixed to be discarded permanently. To
further capture non-linear interactions with selected relevant features and
interactions, in the model functioning stage, we concatenate features and
interactions as vectors and feed them into an MLP:

$\displaystyle\hat{y}=$
$\displaystyle{\rm{Sigmoid}}({\rm{MLP}}([\alpha_{1}\cdot
f_{1},\dots,\alpha_{m}\cdot f_{m},\beta_{1,2}\cdot g(f_{1},f_{2}),$ (10)
$\displaystyle\dots,\beta_{m-1,m}\cdot g(f_{m-1},f_{m})])),$

where the relevance fitness parameters $\boldsymbol{\alpha}$,
$\boldsymbol{\beta}$ are fixed and serve as attention units. In this stage,
weights $\boldsymbol{w}$ of the MLP and $\boldsymbol{f}$ are learned by Adam
optimizer [48]. The time complexity depends on the number of neurons per layer
and the depth of layers of the MLP.

Algorithm 1 outlines the procedures of the CELL framework. DNA search
optimizes the model fitness on operations and returns the selected fittest
operations for interacting each feature pair. Genome search optimizes the
relevance fitness parameters and mutates the operation when the relevance
fitness of the interaction drops to a threshold. The model functioning
procedure retrains the model based on selected relevant features and
interactions to capture non-linear interactions.

Discussion on CELL. We view CELL as a novel nature-inspired framework for
feature interaction selection, which embodies three tribes of artificial
intelligence algorithms [52]. First, from the perspective of the analogizer
algorithm, all of the fine-grained operations in CELL follow the analogized
patterns, which embed the features into low-dimensional spaces to infer their
similarities. Second, from the perspective of the evolutionary algorithm, the
search procedures in CELL follow the evolution process, that is, the parent
model generates the offspring model based on fitness. Furthermore, CELL
combines the advantages of gradient descent search and stochastic search,
where the former is efficient and does not require massive processing units,
while the latter can explore the search space of interactions to avoid
reaching a suboptimum. Moreover, compared with previous fitness evaluation,
which commonly adopts numerical values to represent the fitness of models,
CELL deeply diagnoses the abilities of inside components of the model, which
can enhance interpretability and can help discover order through the diagnosis
results. Third, from the perspective of the connectionist algorithm, the model
functioning procedure calculates the nonlinear connections among features and
interactions to instance prediction. The border crossing, where different
basic algorithms meet, has always been one of the most promising parts of the
research for next-generation learning machines. CELL is more close to this
goal and is expected to facilitate feature interaction selection better.

In practice, one may concern about the scalability of CELL when the operation
set is expanded. Suppose the operation set has $|\boldsymbol{g}|$ operations,
because only DNA search is affected by $\boldsymbol{g}$, its time complexity
varies with the number of operations as $O(|\boldsymbol{g}|\cdot
m^{2}\cdot|f|)$. Therefore, the time complexity varies significantly with the
feature fields and not much with the size of the operation set. This shows
that our method is highly scalable and practical on a larger set of
operations. For industrial use cases where hundreds and thousands of feature
fields are available, common embedded methods are always incapable of
solution, and so is our proposed CELL. In these cases, we usually need to
preprocess the features with filter methods in advance, and remove the
evidently irrelevant features first. After that, the computational cost is
workable for CELL. In the experimental section, we will present an example
where we adopt this strategy to deploy CELL to an industrial FinTech scenario.

##  V Experiments

In this section, we conduct comprehensive experiments to demonstrate the
effectiveness of CELL. Specifically, we first describe the datasets and
experimental settings used in the experiments. Then, we compare the
recommendation performance of CELL with baseline approaches in terms of AUC
and Logloss. After that, we visualize the evolution paths of DNA search and
genome search to show the interpretability of CELL. Finally, we also conduct
synthetic experiments to ascertain that CELL can consistently diagnose to
discover the relevant interactions and the pre-defined interaction patterns.
All experiments are conducted on a Linux server with an NVIDIA Tesla K80 GPU.

TABLE I: The statistics of the four datasets. Dataset |  #Instances |  #Categories |  #Fields |  Positive ratio
---|---|---|---|---
Criteo |  45,840,617 |  34,290,882 |  39 |  0.34
Avazu |  40,428,967 |  9,449,445 |  23 |  0.20
Huawei |  41,907,133 |  1,096,074 |  35 |  0.04
FinTech |  31,182,310 |  7,037,326 |  36 |  0.05

###  V-A Datasets

We use four datasets in the experiments. The statistics are reported in Table
I. Three of them are publicly available advertising datasets for CTR
prediction, i.e., Criteo111https://www.kaggle.com/c/criteo-display-ad-
challenge/data, Avazu222https://www.kaggle.com/c/avazu-ctr-prediction/data,
Huawei333https://www.kaggle.com/louischen7/2020-digix-advertisement-ctr-
prediction. We describe the advertising datasets and the pre-processing steps
below.

  1. 1.

Criteo is a famous benchmark dataset for CTR prediction published by Criteo AI
Lab. Criteo contains one month of click logs with billion of data, and a small
subset of it was published in Criteo Display Advertising Challenge in 2013. We
select “day 6-12” for training and evaluation. 13 numerical fields are
converted into one-hot features through bucketing, where the features in a
certain field appearing less than 20 times are set as a dummy feature “other”.

  2. 2.

Avazu was published in the Avazu Click-Through Rate Prediction contest in
2014. This dataset contains users’ mobile behaviors including whether a
displayed mobile ad is clicked by a user or not. It has 23 feature fields
spanning from user/device features to ad attributes. We select 10 days of data
for training and evaluation.

  3. 3.

Huawei was published in Huawei DIGIX Advertisement CTR Prediction in 2020. The
datasets contain the advertising behavior data collected from seven
consecutive days. It has 35 feature fields spanning from user/device features
to ad attributes.

The last dataset is a financial dataset for client identifying, i.e.,
intelligent financial advisors of online bank apps predict whether their users
are willing to purchase recommended portfolios. As best as we know, there are
no such public datasets currently. To protect data privacy, all the user
information has been desensitized and the feature fields have been anonymized.
The details of it are given below.

  1. 4.

FinTech is collected from a high-tech bank in China and ranges from January 1,
2020, to December 1, 2021. It totally has 2257 feature fields on account of
the wide range of businesses involved. As many features are evidently
irrelevant to the task, we filter them by preliminary screening on a
predefined metric, i.e., Information Value (IV). The $IV$ of a feature field
can be calculated according to
$IV=\sum_{i=1}^{n}(Pos_{i}-Neg_{i})\times\ln(\frac{Pos_{i}}{Neg_{i}})$, where
$Pos_{i}$ and $Neg_{i}$ respectively represent of the proportion of positive
and negative instances under the corresponding features. The higher $IV$ of a
feature field is, the better predictive ability the feature has. After the
preliminary screening, we retain 600 feature fields. To further have a fine
selection, we use genome search to select relevant features and retain 36
feature fields to construct this FinTech dataset, which contains user basic
features, e.g., ages and genders, user financial features, e.g., asset
conditions and risk appetites, and user behavioral features, e.g., investments
and consumptions.

TABLE II:  Performance comparison. Impr is the relative AUC improvement. ⋆ and ◆ represent significance level $p$-value $<10^{-5}$ and $p$-value $<0.05$ of comparing CELL with the best baseline (indicated by underlined numbers). Model Name | Criteo | Avazu | Huawei | FinTech
---|---|---|---|---
|  AUC |  Logloss |  Impr(%) |  AUC |  Logloss |  Impr(%) |  AUC |  Logloss |  Impr(%) |  AUC |  Logloss |  Impr(%)
LR |  0.7784 |  0.4692 |  4.37 |  0.7627 |  0.3896 |  5.02 |  0.7658 |  0.1322 |  4.32 |  0.8477 |  0.1539 |  2.50
FM |  0.7940 |  0.4583 |  2.32 |  0.7895 |  0.3746 |  1.46 |  0.7860 |  0.1281 |  1.64 |  0.8511 |  0.1533 |  2.09
AFM |  0.7985 |  0.4520 |  1.74 |  0.7867 |  0.3759 |  1.82 |  0.7925 |  0.1259 |  0.81 |  0.8569 |  0.1505 |  1.40
FFM |  0.8070 |  0.4449 |  0.67 |  0.7904 |  0.3738 |  1.34 |  0.7945 |  0.1245 |  0.55 |  0.8609 |  0.1490 |  0.93
Wide&Deep |  0.7984 |  0.4523 |  1.75 |  0.7928 |  0.3723 |  1.03 |  0.7916 |  0.1258 |  0.92 |  0.8643 |  0.1488 |  0.53
Deep&Cross |  0.7987 |  0.4522 |  1.72 |  0.7935 |  0.3719 |  0.95 |  0.7947 |  0.1242 |  0.53 |  0.8657 |  0.1472 |  0.37
NFM |  0.8042 |  0.4469 |  1.02 |  0.7913 |  0.3730 |  1.23 |  0.7910 |  0.1256 |  1.00 |  0.8582 |  0.1506 |  1.25
DeepFM |  0.8036 |  0.4481 |  1.10 |  0.7939 |  0.3715 |  0.89 |  0.7917 |  0.1247 |  0.91 |  0.8626 |  0.1501 |  0.73
IPNN |  0.8092 |  0.4420 |  0.40 |  0.7970 |  0.3698 |  0.50 |  0.7939 |  0.1240 |  0.63 |  0.8656 |  0.1469 |  0.38
OPNN |  0.8102 |  0.4417 |  0.27 |  0.7943 |  0.3715 |  0.84 |  0.7937 |  0.1242 |  0.66 |  0.8652 |  0.1480 |  0.43
xDeepFM |  0.8094 |  0.4421 |  0.37 |  0.7963 |  0.3707 |  0.59 |  0.7950 |  0.1239 |  0.49 |  0.8645 |  0.1481 |  0.51
AutoInt |  0.8082 |  0.4433 |  0.52 |  0.7929 |  0.3725 |  1.02 |  0.7909 |  0.1262 |  1.01 |  0.8632 |  0.1482 |  0.66
AutoGroup |  0.8089 |  0.4426 |  0.43 |  0.7982 |  0.3691 |  0.35 |  0.7949 |  0.1244 |  0.97 |  0.8637 |  0.1478 |  0.60
AutoFIS-FM |  0.8062 |  0.4452 |  0.77 |  0.7945 |  0.3712 |  0.82 |  0.7887 |  0.1268 |  1.29 |  0.8526 |  0.1533 |  1.91
AutoFIS-IPNN |  0.8094 |  0.4422 |  0.37 |  0.7978 |  0.3695 |  0.40 |  0.7945 |  0.1245 |  0.55 |  0.8644 |  0.1482 |  0.52
CELL (-DNA, Geno) |  0.8081 |  0.4436 |  0.53 |  0.7941 |  0.3713 |  0.87 |  0.7920 |  0.1253 |  0.87 |  0.8633 |  0.1486 |  0.65
CELL (-DNA) |  0.8108 |  0.4408 |  0.20 |  0.7983 |  0.3689 |  0.34 |  0.7962 |  0.1234 |  0.34 |  0.8648 |  0.1483 |  0.47
CELL (-Geno) |  0.8100 |  0.4415 |  0.30 |  0.7993 |  0.3682 |  0.21 |  0.7953 |  0.1238 |  0.45 |  0.8662 |  0.1472 |  0.31
CELL |  0.8124⋆ |  0.4393⋆ |  - |  0.8010⋆ |  0.3671⋆ |  - |  0.7989◆ |  0.1229◆ |  - |  0.8689⋆ |  0.1460⋆ |  -

###  V-B Experimental Settings

####  V-B1 Evaluation Metrics

Two widely used metrics, AUC (Area Under Curve) and Logloss (cross-entropy
loss) are selected to evaluate the recommendation performance.

TABLE III: Training cost of CELL (GPU hours). Dataset |  Stage I |  Stage II |  Stage III
---|---|---|---
Criteo |  $\sim$8.4 |  $\sim$1.4 |  $\sim$1.5
Avazu |  $\sim$18.5 |  $\sim$0.8 |  $\sim$0.8
Huawei |  $\sim$6.9 |  $\sim$1.7 |  $\sim$2.1
FinTech |  $\sim$6.3 |  $\sim$1.1 |  $\sim$1.2

####  V-B2 Baselines

The standard and state-of-the-art approaches we use as our baselines are: LR
[53], FM [9], AFM [11], FFM [23], Wide&Deep [28], Deep&Cross [29], NFM [10],
DeepFM [12], IPNN, OPNN [25], xDeepFM [27], AutoInt [54], AutoGroup [37],
AutoFIS-FM, and AutoFIS-IPNN [51]. We calculate the $p$-values for CELL and
the best baseline by repeating the experiments 10 times by changing the random
seeds. The two-tailed pairwise t-test is performed to detect the significant
difference. We use ⋆ and ◆ to represent significance level $p$-value
$<10^{-5}$ and $p$-value $<0.05$.

####  V-B3 Implementation Details

Specifically, to implement our proposed CELL, in the first stage we use Adam
[48] as the optimizer for DNA search with initial learning rate $\xi=10^{-3}$,
momentum $(0.5,0.999)$, weight decay $10^{-3}$. In the second stage, we use
RDA [49, 50] as the optimizer for genome search with the learning rate
$\gamma=10^{-3}$, adjustable hyper-parameters $c=0.5,\mu=0.8$. We set the
mutation mechanism as $\lambda=0.1$, $\sigma=5$, $\tau=100$. In the model
functioning stage, we set the depth of MLP as $2$ with $400$ neurons per
layer. In order to verify the effectiveness of DNA search and genome search,
we design ablation studies by dropping one or two stages, i.e., CELL (-DNA)
denotes CELL with DNA search dropped, which means operations are randomly
assigned to each feature pair. CELL (-Geno) denotes CELL with genome search
dropped, which means interactions are retained with equal relevance fitness.
CELL (-DNA, Geno) combines the settings of CELL (-DNA) and CELL (-Geno).

Figure 2: Visualization of DNA search on Criteo dataset and the demonstration
of diagnosed fitness on operations. Figure 3: Visualization of genome search
on Criteo dataset and the demonstration of diagnosed fitness on features and
interactions.

###  V-C Performance Comparison

Table II reports the performance of compared baselines and CELL on the four
datasets. Impr is the relative AUC improvement. It is worth mentioning that,
unlike AUC, the smaller the Logloss value, the better the result. In practice,
an improvement of 0.001-level in AUC or Logloss is usually regarded as being
significant, because it will lead to a large increase in the company’s revenue
due to a large user base, which has been pointed out in many existing articles
[28, 12, 6, 5]. From the experimental results, we have the following key
observations:

First, most neural network models outperform shallow models, i.e., LR, FM,
which indicates that MLP can learn non-linear interactions and endow
representation capabilities. Furthermore, OPNN, AutoGroup, and IPNN, the
representatives of interactions modeled by element-wise outer and inner
products, are the best baselines on Criteo, Avazu, and FinTech. Deep&Cross and
xDeepFM, the representatives of interactions modeled by cross networks and
compressed interaction networks, are the best baselines on FinTech and Huawei.
The results are further consistent with our statement that we cannot ensure
which pre-designed operations are better because they are poorly adaptable to
tasks and datasets.

Second, AUC and Logloss of CELL have significant improvement over previous
baselines on all four datasets. This gain in performance can be attributed to
adaptively modeling interactions by evolving to find suitable operations.
Another benefit comes from that CELL can distinguish the relevance of features
and interactions. Instead of numerating interactions in a brute-force way,
CELL is able to select task-friendly features and interactions to avoid extra
noise from irrelevant features and interactions.

Third, from the results of ablation studies, we can observe that CELL
outperforms CELL (-DNA), and CELL (-Geno) outperforms CELL (-DNA, Geno), which
indicate that DNA search is effective because it can diagnose the fitness of
the model on operations to obtain appropriate operations. Meanwhile, the
superior performance of CELL over CELL (-Geno) and the superior performance of
CELL (-DNA) over CELL (-DNA, Geno) indicate that genome search can diagnose
the fitness of the model on features and interactions to find relevant ones
and promote the model.

Beyond accuracy, we also display the time complexity of three stages of CELL
in Table III. Note that every stage runs on a single GPU. The training cost of
each stage is consistent with our analysis. Moreover, the entire training cost
for each dataset is less than one day. CELL has excellent efficiency compared
to previous automated machine learning approaches that usually run on massive
GPUs for days [36]. On the Avazu dataset, the training cost of the first stage
is a little longer, mainly because the dimension of embeddings is set
relatively higher than the other datasets.

(a)

(b)

(c)

Figure 4:  Results of the synthetic experimental studies. (a) Diagnosed
relevance fitness of interactions. (b) Diagnosed fitness on operations to
model the interactions associated with the selected poly-2 terms. (c)
Diagnosed fitness on the switched dataset.

###  V-D Visualization of Evolution Path

To make it clear how the model evolves to select suitable operations,
features, and interactions under the task guidance, we visualize the evolution
path of DNA search and genome search on the Criteo dataset, which contains 13
integer feature fields “$I1\sim I13$” and 26 categorical feature fields
“$C1\sim C26$”. If we use the following encoding
$\oplus=0,\otimes=1,\boxplus=2,\boxtimes=3$, the fitness diagnosis results can
be represented as a matrix. Furthermore, we assign different colors for
operations to describe a gene map of the model, where each gene indicates an
interaction, i.e., red “$0$”, green “$1$”, yellow “$2$”, blue “$3$”. For
example, green “$1$” in the block “$I1\times C12$” means that element-wise
product $\otimes$ is diagnosed as the fittest operation for feature $I1$ to
interact with feature $C12$. In genome search, in order to express the
relevance of features and interactions, we highlight some genes according to
the relevance fitness parameters. The darker color represents the feature or
the interaction that is diagnosed as the more relevant one, and the lighter
color represents the less relevant one.

We can observe that, in Fig. 2, operations were randomly assigned to model all
interactions at the beginning. Then, the gene map evolved rapidly in the early
iterations, and was occupied by a large number of operations $\otimes$ and
$\boxtimes$, which could be regarded as the model falling into the first local
optimal. After more batches of data were learned, the gene map changed
drastically. In the 1000th iteration, the gene map was significantly occupied
by operations $\boxplus$, which could be regarded as the model falling into
the second local optimal. What interests us is that the subsequent evolution
direction has undergone a dramatic change. We can see more operations
$\boxplus$ have been replaced by operations $\otimes$ and $\boxtimes$ when the
model converges. In addition to these dynamic changes, there are some static
patterns. For example, operations had the effect of string-like clustering on
the gene map as we marked, which means that interactions containing the same
feature were more likely to be modeled by the same operation.

In Fig. 3, all features and interactions shared equal relevance fitness at the
beginning. Later, some were discovered as relevant features and interactions
(the color becomes darker), while most of the others became less important
(the color becomes lighter), and some mutated into new interactions. The
relevance fitness parameters of a part of features and interactions were
reduced and truncated to $0$. We discarded these irrelevant features and
interactions, so their genes became white “$-1$”. Another interesting
observation is that, in DNA search, the operations of some interactions hardly
changed with the increase of the learning batches. Combined with genome
search, we can find that these interactions are almost either highly relevant
or irrelevant interactions.

###  V-E Synthetic Experiments

To further validate the effectiveness of the fitness diagnosis technique of
CELL, we conduct a synthetic experiment. We artificially construct such a
synthetic dataset, where the input feature $\boldsymbol{f}$ of this dataset is
sampled from the Criteo dataset containing $N$ categories of $m$ fields, and
the output $y$ is binary labeled and generated from a poly-2 function. Based
on this synthetic dataset, we investigate whether the fitness diagnosis
technique can diagnose to find (1) the relevant interactions and (2) the
operations to model these interactions.

Here we choose $m=6,N=4481$ to test the effectiveness of our model. We use the
element-wise sum $\oplus$ and the element-wise product $\otimes$ to build such
sums of poly-2 terms. The output binary label $y$ is given as:

$\small y=\delta\left(\sum_{i,j\in C_{1}}f_{i}\oplus f_{j}+\sum_{i,j\in
C_{2}}f_{i}\otimes f_{j}+\epsilon\right),$ (11)
$\displaystyle\delta(q)=\begin{cases}1,&\textbf{if}\quad
q\geqslant\text{threshold},\\\ 0,\;\;\;&\textbf{otherwise}.\end{cases}$ (12)

The feature pairs are $i.i.d.$ sampled to build the train and test datasets.
We also add a small random noise $\epsilon$ to the sampled data. First, the
selected poly-2 term sets $C_{1}=\\{(f_{0},f_{2}),(f_{1},f_{3})\\}$ and
$C_{2}=\\{(f_{2},f_{4}),(f_{4},f_{5})\\}$ are initialized and fixed. After
effortless training, CELL achieved AUC$=0.9920$, Logloss$=0.1269$ on this
synthetic dataset, which means CELL can consistently discover the pre-defined
interaction patterns for interacting feature pairs and make the right
predictions. Figure 4b shows the Diagnosed fitness on operations to model the
interactions associated with the selected poly-2 terms. Red color indicates
element-wise sum $\oplus$ and green color indicates element-wise product
$\otimes$. In order to further validate the diagnosis results are not affected
by assigned features, we switch the operations for the selected poly-2 terms
to generate another new dataset. Thus, this time the selected poly-2 terms
$C_{1}=\\{(f_{2},f_{4}),(f_{4},f_{5})\\}$ and
$C_{2}=\\{(f_{0},f_{2}),(f_{1},f_{3})\\}$ are initialized and fixed. Figure 4c
shows the diagnosed fitness on this switched dataset.

As shown in Figure 4a, four selected poly-2 terms have been clearly extracted
through fitness diagnosis, indicating that CELL can precisely discover the
relevant interactions. Furthermore, in Figure 4b, the correct operations for
modeling the relevant interactions are diagnosed as better learning
proficiency. Moreover, in Figure 4c, as switching the operations, the
diagnosed fitness is concomitantly reversed, which confirms that our diagnosis
technique on operations is effective and independent of assigned features.

Figure 5: Impact of hyper-parameters on Criteo dataset. Figure 6: Impact of
hyper-parameters on Avazu dataset. Figure 7: Impact of hyper-parameters on
Huawei dataset. Figure 8: Impact of hyper-parameters on FinTech dataset.

###  V-F Hyper-parameter Studies

We investigate the impact of hyper-parameters of CELL, including (1) embedding
size; (2) depth of MLP; (3) neurons per layer; (4) mutation threshold
$\lambda$; (5) mutation probability parameter $\sigma$. The experimental
results on four datasets in terms of AUC and Logloss are shown from Fig. 5 to
Fig. 8.

From the experimental results, we can observe that embedding size has the most
significant impact on performance. Meanwhile, even with small embedding sizes,
CELL still has comparable performance to some popular baselines with large
embedding sizes, e.g., for the advertising dataset Criteo, CELL achieves AUC
$>0.8090$ and Logloss $<0.4420$ with the embedding size set as $5$, which is
better than DeepFM with an embedding size set as $20$. For the financial
dataset FinTech, we get a similar observation that CELL achieves AUC $>0.8660$
and Logloss $<0.1470$ with the embedding size set as $5$, which is better than
Deep&Cross with embedding size set as $15$. Note that Deep&Cross is regarded
as one of the best baselines on the FinTech dataset. Furthermore, the model
performance boosts with the depth of MLP at the beginning. However, it barely
boosts when the depth of MLP is set to greater than $2$. Also, the model
performance barely boosts as the number of neurons per layer increases from
$200$ to $500$. We consider $400$ is a more suitable setting to avoid the
model overfitting. Moreover, the mutation threshold $\lambda$ should not be
set too large, and the mutation probability parameter $\sigma$ should not be
set too small, both to avoid over-random mutation.

##  VI Conclusion

This paper presents a nature-inspired learning approach to select feature
interactions for recommender systems, namely Cognitive Evolutionary Learning
(CELL). CELL can adaptively evolve a model to find (1) the proper operation
for modeling each interaction and (2) relevant features and interactions to
the task. Attributed to the proposed fitness diagnosis technique, CELL can
diagnose the learning abilities of inside components of models during
training, thereby enhancing interpretability and interpreting the mechanism of
interaction modeling and selection. We conducted extensive experiments on four
datasets, including three publicly available advertising datasets and one
collected financial dataset. Experimental results have proved that CELL can
significantly outperform state-of-the-art approaches. By tracing the evolution
path, we can reveal the learning abilities of models during training and
interpret how to select operations, features, and interactions. The synthetic
experiments are further consistent with our statement because CELL can clearly
extract the pre-defined interaction patterns. The CELL framework of utilizing
evolutionary learning and diagnosis techniques is new thinking that helps us
build a model under task guidance and is not limited to the instantiation in
this paper. We encourage more task-oriented instantiations to be proposed
based on our work.

## References

  * [1] W. Guo _et al._ , “Dual graph enhanced embedding neural network for ctr prediction,” in _KDD_ , 2021, pp. 496–504.
  * [2] P. Li _et al._ , “Dual attentive sequential learning for cross-domain click-through rate prediction,” in _KDD_ , 2021, pp. 3172–3180.
  * [3] S.-T. Shi, W. Zheng, J. Tang, Q.-G. Chen, Y. Hu, J. Zhu, and M. Li, “Deep time-stream framework for click-through rate prediction by tracking interest evolution,” in _AAAI_ , 2020, pp. 5726–5733.
  * [4] Z. Lyu, Y. Dong, C. Huo, and W. Ren, “Deep match to rank model for personalized click-through rate prediction,” in _AAAI_ , vol. 34, no. 01, 2020, pp. 156–163.
  * [5] Q. Shao _et al._ , “Toward intelligent financial advisors for identifying potential clients: A multitask perspective,” _Big Data Mining and Analytics_ , vol. 5, no. 1, pp. 64–78, 2021.
  * [6] R. Yu _et al._ , “Xcrossnet: Feature structure-oriented learning for click-through rate prediction,” in _PAKDD_ , 2021, pp. 436–447.
  * [7] M. Dash and H. Liu, “Feature selection for classification,” _Intelligent data analysis_ , vol. 1, no. 1-4, pp. 131–156, 1997.
  * [8] X. Li, Y. Wang, and R. Ruiz, “A survey on sparse learning models for feature selection,” _IEEE transactions on Cybernetics_ , pp. 1642–1660, 2022.
  * [9] S. Rendle, “Factorization machines,” in _ICDM_. IEEE, 2010, pp. 995–1000.
  * [10] X. He and T.-S. Chua, “Neural factorization machines for sparse predictive analytics,” in _SIGIR_ , 2017, pp. 355–364.
  * [11] J. Xiao _et al._ , “Attentional factorization machines: learning the weight of feature interactions via attention networks,” in _IJCAI_ , 2017, pp. 3119–3125.
  * [12] H. Guo _et al._ , “Deepfm: A factorization-machine based neural network for CTR prediction,” in _IJCAI_ , 2017, pp. 1725–1731.
  * [13] B. Xue, M. Zhang, W. N. Browne, and X. Yao, “A survey on evolutionary computation approaches to feature selection,” _IEEE Transactions on Evolutionary Computation_ , vol. 20, no. 4, pp. 606–626, 2015.
  * [14] Z.-H. Zhou, Y. Yu, and C. Qian, _Evolutionary learning: Advances in theories and algorithms_. Springer, 2019\.
  * [15] R. Yu, X. Xu, Y. Ye, Q. Liu, and E. Chen, “Cognitive evolutionary search to select feature interactions for click-through rate prediction,” in _KDD_ , 2023, pp. 3151–3161.
  * [16] A. Telikani, A. Tahmassebi, W. Banzhaf, and A. H. Gandomi, “Evolutionary machine learning: A survey,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 8, pp. 1–35, 2021.
  * [17] B. Tran, B. Xue, and M. Zhang, “A new representation in pso for discretization-based feature selection,” _IEEE Transactions on Cybernetics_ , vol. 48, no. 6, pp. 1733–1746, 2017.
  * [18] F. Cheng, F. Chu, Y. Xu, and L. Zhang, “A steering-matrix-based multiobjective evolutionary algorithm for high-dimensional feature selection,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 9, pp. 9695–9708, 2022.
  * [19] H. Liu and H. Motoda, _Feature selection for knowledge discovery and data mining_. Springer Science & Business Media, 2012, vol. 454.
  * [20] G. Zhou, N. Mou, Y. Fan, Q. Pi, W. Bian, C. Zhou, X. Zhu, and K. Gai, “Deep interest evolution network for click-through rate prediction,” in _AAAI_ , 2019, pp. 5941–5948.
  * [21] Y. Xie _et al._ , “Fives: Feature interaction via edge search for large-scale tabular data,” in _KDD_ , 2021, pp. 3795–3805.
  * [22] H. Guo, B. Chen _et al._ , “An embedding learning framework for numerical features in ctr prediction,” in _KDD_ , 2021, pp. 2910–2918.
  * [23] Y. Juan, Y. Zhuang, W.-S. Chin, and C.-J. Lin, “Field-aware factorization machines for ctr prediction,” in _RecSys_ , 2016, pp. 43–50.
  * [24] W. Zhang, T. Du, and J. Wang, “Deep learning over multi-field categorical data,” in _European Conference on Information Retrieval_. Springer, 2016, pp. 45–57.
  * [25] Y. Qu _et al._ , “Product-based neural networks for user response prediction,” in _ICDM_. IEEE, 2016, pp. 1149–1154.
  * [26] Y. Qu, B. Fang _et al._ , “Product-based neural networks for user response prediction over multi-field categorical data,” _ACM Transactions on Information Systems (TOIS)_ , vol. 37, no. 1, pp. 1–35, 2018.
  * [27] J. Lian _et al._ , “xdeepfm: Combining explicit and implicit feature interactions for recommender systems,” in _KDD_ , 2018, pp. 1754–1763.
  * [28] H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked _et al._ , “Wide & deep learning for recommender systems,” in _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems_ , 2016, pp. 7–10.
  * [29] R. Wang, B. Fu, G. Fu, and M. Wang, “Deep & cross network for ad click predictions,” in _ADKDD_ , 2017, pp. 1–7.
  * [30] T. Back, _Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms_. Oxford University Press, 1996.
  * [31] C.-F. Juang, C.-Y. Chou, and C.-T. Lin, “Navigation of a fuzzy-controlled wheeled robot through the combination of expert knowledge and data-driven multiobjective evolutionary learning,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 8, pp. 7388–7401, 2022.
  * [32] X. Zheng, W. Wu, W. Deng, C. Yang, and K. Huang, “Reconstruction of tree network via evolutionary game data analysis,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 7, pp. 6083–6094, 2022.
  * [33] K. Chen, B. Xue, M. Zhang, and F. Zhou, “An evolutionary multitasking-based feature selection method for high-dimensional classification,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 7, pp. 7172–7186, 2022.
  * [34] X.-F. Song, Y. Zhang, D.-W. Gong, and X.-Z. Gao, “A fast hybrid feature selection based on correlation-guided clustering and particle swarm optimization for high-dimensional data,” _IEEE Transactions on Cybernetics_ , vol. 52, no. 9, pp. 9573–9586, 2022.
  * [35] F. Khawar _et al._ , “Autofeature: Searching for feature interactions and their architectures for click-through rate prediction,” in _CIKM_ , 2020, pp. 625–634.
  * [36] Q. Song, D. Cheng _et al._ , “Towards automated neural interaction discovery for click-through rate prediction,” in _KDD_ , 2020, pp. 945–955.
  * [37] B. Liu _et al._ , “Autogroup: Automatic feature grouping for modelling explicit high-order feature interactions in ctr prediction,” in _SIGIR_ , 2020, pp. 199–208.
  * [38] L. V. DiBello, L. A. Roussos, and W. Stout, “A review of cognitively diagnostic assessment and a summary of psychometric models,” _Handbook of statistics_ , vol. 26, pp. 979–1030, 2006.
  * [39] J. De La Torre, “The generalized dina model framework,” _Psychometrika_ , vol. 76, no. 2, pp. 179–199, 2011.
  * [40] S. Tong _et al._ , “Item response ranking for cognitive diagnosis,” in _IJCAI_ , 2021, pp. 1750–1756.
  * [41] L. Yue _et al._ , “Circumstances enhanced criminal court view generation,” in _SIGIR_ , 2021, pp. 1855–1859.
  * [42] Y. Gu _et al._ , “Neuralac: Learning cooperation and competition effects for match outcome prediction,” in _AAAI_ , vol. 35, no. 5, 2021, pp. 4072–4080.
  * [43] W. Tao, Y. Li, L. Li, Z. Chen, H. Wen, P. Chen, T. Liang, and Q. Lu, “Sminet: State-aware multi-aspect interests representation network for cold-start users recommendation,” in _AAAI_ , vol. 36, no. 8, 2022, pp. 8476–8484.
  * [44] H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” in _ICLR_ , 2019.
  * [45] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_. PMLR, 2017, pp. 1126–1135.
  * [46] J. Luketina, M. Berglund, K. Greff, and T. Raiko, “Scalable gradient-based tuning of continuous regularization hyperparameters,” in _ICML_. PMLR, 2016, pp. 2952–2960.
  * [47] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, “Unrolled generative adversarial networks,” _arXiv preprint arXiv:1611.02163_ , 2016.
  * [48] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [49] L. Xiao, “Dual averaging method for regularized stochastic learning and online optimization,” _NeurIPS_ , vol. 22, pp. 2116–2124, 2009.
  * [50] S.-K. Chao and G. Cheng, “A generalization of regularized dual averaging and its dynamics,” _arXiv preprint arXiv:1909.10072_ , 2019.
  * [51] B. Liu _et al._ , “Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction,” in _KDD_ , 2020, pp. 2636–2645.
  * [52] P. Domingos, _The master algorithm: How the quest for the ultimate learning machine will remake our world_. Basic Books, 2015.
  * [53] M. Richardson _et al._ , “Predicting clicks: estimating the click-through rate for new ads,” in _WWW_ , 2007, pp. 521–530.
  * [54] W. Song _et al._ , “Autoint: Automatic feature interaction learning via self-attentive neural networks,” in _CIKM_. ACM, 2019, pp. 1161–1170.

|  Runlong Yu received his Ph.D. degree in computer science and technology at
the University of Science and Technology of China (USTC) in Hefei, China, in
2023, following his B.Eng. degree in computer science and technology from
USTC, in 2017. He is currently a Postdoctoral Associate in the Department of
Computer Science at the University of Pittsburgh. His research interests
include data mining, machine learning, evolutionary computation, AI for
science, and recommender systems. He has published over twenty papers in
refereed journals and conference proceedings including IEEE TKDE, KAIS, ACM
SIGKDD, ACM CIKM, ACM SIGIR, PAKDD, IEEE ICDM, IJCAI, AAAI, and DASFAA. He was
the recipient of the KDD CUP 2019 PaddlePaddle Special Award and the 2020 CCF
BDCI Contest championship.
---|---
|  Qixiang Shao received his Master degree in computer science and technology
at the University of Science and Technology of China (USTC) in Hefei, China,
in 2023. Before that, he received his B.Eng. degree in software engineering
from Nanjing University of Information Science and Technology (NUIST),
Nanjing, China, in 2017. He is currently working as a software engineer at
iFlytek. His research interests include recommender systems, data mining, and
FinTech.
---|---
|  Qi Liu received the Ph.D. degree from University of Science and Technology
of China (USTC), Hefei, China, in 2013. He is currently a Professor with Anhui
Province Key Laboratory of Big Data Analysis and Application (BDAA), State Key
Laboratory of Cognitive Intelligence, School of Computer Science and
Technology, University of Science and Technology of China, Hefei, China. His
research interests include data mining, machine learning, and recommender
systems. He was the recipient of KDD’ 18 (Research Track) Best Student Paper
Award and ICDM’ 11 Best Research Paper Award. He was the winner of the
National Science Fund for Outstanding Youth Science Foundation in 2019.
---|---
|  Huan Liu (F’ 12) received the B.Eng. degree in computer science and
electrical engineering from Shanghai JiaoTong University (SJTU), Shanghai,
China, in 1983, and the Ph.D. degree in computer science from University of
Southern California (USC), Los Angeles, CA, USA, in 1989. He is currently an
Ira A. Fulton Professor of Computer Science and Engineering at Arizona State
University (ASU), Tempe, AZ, USA. He is a Fellow of IEEE, ACM, AAAI, and AAAS.
His well-cited publications include books, book chapters, encyclopedia
entries, and conference and journal papers. His research interests include
data mining, machine learning, social computing, and feature selection. He was
recognized for excellence in teaching and research in computer science and
engineering at Arizona State University. He serves on journal editorial boards
and numerous conference program committees. He is also a Founding Organizer of
the International Conference Series on Social Computing, Behavioral Cultural
Modeling, and Prediction.
---|---
|  Enhong Chen (F’ 24) received the Ph.D. degree from University of Science
and Technology of China (USTC), Hefei, China, in 1996. He is currently a
Director (Professor) of Anhui Province Key Laboratory of Big Data Analysis and
Application (BDAA), and a vice Director of State Key Laboratory of Cognitive
Intelligence, University of Science and Technology of China, Hefei, China. He
is an executive dean of School of Data Science of USTC. He is an IEEE Fellow,
ACM Distinguished Member, and CCF Fellow. His research interests include data
mining, machine learning, and recommender systems. He has published 200+
refereed international conference and journal papers. He was the recipient of
KDD’ 08 Best Application Paper Award and ICDM’ 11 Best Research Paper Award.
He was the winner of the National Science Fund for Distinguished Young
Scholars in 2013, scientific and technological innovation leading talent of
‘Ten Thousand Talent Program’ in 2017.
---|---