# Extreme Cross Network: Representing Dense and Sparse Feature Interactions
for Recommender Systems

Anonymous Author(s)

###### Abstract

Click-Through Rate (CTR) prediction is a core task in nowadays commercial
recommender systems. Feature crossing, as the mainline of research on CTR
prediction, has shown a promising way to enhance the predictive performance.
However, previous methods are either low performance or retain implicit
components for crossing features. Moreover, they mainly focus on the modeling
of cross sparse features and neglect to represent cross dense features.
Motivated by this, we propose a novel Extreme Cross Network (XCrossNet), which
aims at learning dense and sparse feature interactions in an explicit manner.
In detail, we separately design a cross layer for crossing dense features and
a product layer for crossing sparse features, then feed these cross features
into an attention mechanism and an MLP for combination and selection.
XCrossNet as a feature structure-oriented model leads to a more expressive
representation and a more precise CTR prediction, which is not only explicit
and interpretable, but also time-efficient and easy to implement. Extensive
experiments on three large-scale public datasets show significant improvement
of XCrossNet over state-of-the-art models on effectiveness and efficiency.

##  1 Introduction

Accurate targeting of commercial recommender systems is of great importance,
in which Click-Through Rate (CTR) prediction plays a key role. CTR prediction
aims to estimate the ratio of clicks to the impression of a recommended item
for a user (2019; 2020). We show a common commercial recommender system for
online display advertising in Figure 1. Advertisers expect lower costs to
achieve a higher Return On Investment (ROI). Generally, advertisers have two
bid strategies, one is based on Cost Per Click (CPC), and the other is based
on Cost Per Action (CPA). From the publishers’ perspective, traffic monetizing
hinges on Cost Per Mille Impressions (CPM). The AD Exchange platforms usually
trade with advertisers and publishers according to the Generalized Second
Price (GSP) of the maximum effective Cost Per Mille (eCPM). Demand Side
Platforms (DSP) help plenty of advertisers manage display ads campaigns
through AD Exchange platforms. Therefore, campaign performance directly
depends on predicted Click-Through Rate (pCTR) or predicted ConVersion Rate
(pCVR). If CTR is overestimated, advertisers could waste campaign budgets on
the useless impression; On the other hand, if CTR is underestimated,
advertisers would lose some valuable impressions and the campaigns may under
deliver. With multi-billion dollar business on commercial recommendation today
(2019; 2020), CTR prediction has received growing interest from communities of
both academia and industry.

ROI = Return On Investment, GMV = Gross Merchandise Volume, CPC = Cost Per
Click, CPA = Cost Per Action, CPM = Cost Per Mille Impressions, pCTR =
predicted Click-Through Rate, pCVR = predicted ConVersion Rate, eCPM =
effective Cost Per Mille, GSP = Generalized Second Price.

Figure 1: Recommender systems for display advertising.

In web-scale commercial recommender systems, the inputs of users’
characteristics are in two kinds of structures. The first kind of structure is
described by numerical or dense parameters, e.g., “Age_years=22,
Height_cm=165”. Each of such characteristics is formalized as a value
associated with a numerical field, while the values are named as dense
features. The second kind of structure is described by categorical or sparse
parameters, e.g.,“Gender=Female, Relationship=In love”. Each of such
characteristics is formalized as a vector of one-hot encoding associated with
a categorical field, while the vectors are named as sparse features. Data
scientists usually spend much time on interactions of raw features to generate
better predictive models (Lian et al. 2018; Ouyang et al. 2019). Among these
feature interactions, cross features, previously focused more on cross-product
of sparse features, show a promising way to enhance the performance of
prediction (2014; 2016; 2019; 2020). Owing to the fact that correct cross
features are mostly task-specific and difficult to identify a priori, the
crucial challenge is in automatically extracting sophisticated cross features
hidden in high-dimensional data.

Research on feature crossing as the mainline of CTR prediction has attracted
widespread attention in recent years. Shallow models like Factorization
Machine (Rendle 2010) are simple, interpretable, and easy to scale, but
limited in expressive ability. In contrast, deep learning has shown powerful
expressive capabilities, nevertheless, as shown in (Beutel et al. 2018), Deep
Neural Networks (DNNs) require many more parameters than tensor factorization
to approximate high-order cross features. Besides, almost all deep models
leverage multilayer perceptron (MLP) to learn high-order feature interactions
(2017; 2017; 2018), however, whether plain DNNs indeed effectively represent
right functions of cross features remains an open question (2017; 2018).
Moreover, most methods mainly focus on the representation of cross sparse
features and neglect to represent cross dense features or the combination of
dense and sparse features. Previous work always directly concatenates dense
features with the embeddings of sparse features, which could cause an
important feature dimensionality imbalance problem.

Based on all these observations, we propose a novel Extreme Cross Network
(XCrossNet), to represent both dense and sparse feature interactions. Modeling
with XCrossNet consists of three stages: the Feature Crossing, the Feature
Concatenation, and the Feature Selection. In the Feature Crossing stage, we
separately design a cross layer for crossing dense features and a product
layer for crossing sparse features. In the Feature Concatenation stage, cross
dense features and cross sparse features interact through an attention
mechanism for combination. Lastly, in the Feature Selection stage, we employ
an MLP for capturing non-linear interactions and their relative importance.
Experimental results on three large-scale public datasets demonstrate the
superior performance of XCrossNet over the state-of-the-art baselines. The
main contributions are highlighted as follows:

  * •

Our cross layer and product layer explicitly model cross features to specific
orders, which is different from the previous ideas of implicitly learning
feature interactions.

  * •

We propose XCrossNet, which efficiently learns dense and sparse feature
interactions. It is feasible to balance the dimension of dense features up to
the dimension of sparse features for solving the feature dimensionality
imbalance.

  * •

Extensive experiments on three large-scale public datasets show significant
improvement of XCrossNet over state-of-the-art models on effectiveness and
efficiency.

##  2 Related Work

Studies on CTR prediction can be categorized into five classes which will be
respectively introduced below.

(1) Generalized linear models. Logistic Regression (LR) models such as FTRL
are widely used in CTR prediction for their simplicity and efficiency (2007;
2012; 2013). Yan et al. argue that LR cannot capture nonlinear feature
interactions and propose Coupled Group Lasso (CGL) to solve it. Human efforts
are usually needed for LR models. Gradient boosting decision tree (GBDT) is a
method to automatically do feature engineering and search interactions
(Friedman 2001), then the transformed feature interactions can be fed into LR.
In practice, tree-based models are more suitable for dense features but not
for sparse features.

(2) Quadratic polynomial mappings and Factorization Machines. Poly2 enumerates
all pairwise feature interactions to avoid feature engineering which works
well on dense features (2010). For sparse features, Factorization Machine (FM)
and its variants project each feature into a low-dimensional vector and models
cross features by inner product (Rendle 2010). FFM enables each feature to
have multiple latent vectors to interact with features from different fields
(2016). As both FM and FFM can only model order-2nd cross features. An
efficient algorithm Higher-Order FM (HOFM) for training arbitrary-order cross
features was proposed by introducing the ANOVA kernel (2016). As shown in
(2017), HOFM achieves marginal improvement over FM whereas using many more
parameters and only its low-order (usually less than 5) form can be
practically used.

(3) Implicit deep learning models. As deep learning has shown promising
representation capabilities in recommender systems, several models use MLP to
improve FM. Attention FM (AFM) considered the importance of different
order-2nd cross features (2017). Neural FM (NFM) stacked deep neural networks
on top of the output of the order-2nd cross features to model higher-order
cross features (2017). FNN uses FM to pre-train feature embeddings and then
feeds them into an MLP (2016). Moreover, CCPM uses convolutional layers to
explore local-global dependencies of cross features (2015). IPNN (also known
as PNN) feeds the interaction result of the FM layer and feature embeddings
into an MLP (2016). PIN introduces a micro-network for each pair of fields to
model pairwise cross features (2018). FGCNN combines a CNN and MLP to generate
new features for feature augmentation (2019). However, all these approaches
learn the high-order cross features in an implicit manner, therefore lack good
model explainability.

(4) Wide&Deep based models. Lian et al. argue that implicit deep learning
models focus more on high-order cross features but capture little low-order
cross features. The Wide&Deep model overcomes this problem by introducing a
hybrid architecture, which contains a shallow component and a deep component
with the purpose of learning both memorization and generalization (2016).
DeepFM uses an FM layer to replace the wide component in Wide&Deep (2017).
DeepFM and Wide&Deep cannot explicitly model higher-order feature
interactions, which could further improve model performance. To learn cross
features in an explicit fashion, Deep&Cross (2017) and xDeepFM (2018) take
outer product of features at the bit- and vector-wise level respectively.
However, it uses so many parameters that great challenges are posed to
identify important cross features in the huge combination space.

(5) AutoML based models. There exist some approaches using AutoML techniques
to deal with cross features in recommender systems. AutoCross is proposed to
search over subsets of candidate features to identify effective interactions
(2019). This requires training the whole model to evaluate the selected
feature interactions, but the candidate sets are incredibly many. AutoInt
leverages attention and residual networks to form an interacting layer to
determine the relevance of cross features (2019). AutoGroup treats the
selection process of high-order feature interactions as a structural
optimization problem, and solves it with Neural Architecture Search (2020). It
achieves state-of-the-art performance on various datasets, but is too complex
to be applied in industrial applications.

##  3 Extreme Cross Network

In this section, we describe the details of Extreme Cross Network (XCrossNet)
in the following three steps: Feature Crossing, Feature Concatenation, and
Feature Selection.

We firstly formalize CTR prediction problem. Suppose the dataset for training
consists of $n$ instances $([\bm{D};\bm{S}],y)$, where
$\bm{D}=[D_{1},D_{2},\cdots,D_{M}]$ indicates dense features including $M$
numerical fields, and $\bm{S}=[S_{1},S_{2},\cdots,S_{N}]$ indicates sparse
features including $N$ categorical fields, and $y\in\\{0,1\\}$ indicates the
user’s click behaviors ($y=1$ means the user clicked the item, and $y=0$
otherwise). The task of CTR prediction is to build a prediction model
$\hat{y}=pCTR\\_Model([\bm{D};\bm{S}])$ to estimate the ratio of clicks to
impressions of a given feature context.

Figure 2: The structure of XCrossNet.

###  3.1 Feature Crossing

#### Definition.

In CTR predictions, a feature pair $(x_{1},x_{2})$ of two dense features
$x_{1}$ and $x_{2}$ is defined as a cross feature if and only if
$\frac{\partial\hat{y}}{\partial(x_{1},x_{2})}\nrightarrow 0$, but
$\frac{\partial\hat{y}}{\partial x_{2}}\rightarrow 0$ as $x_{1}\rightarrow 0$,
and $\frac{\partial\hat{y}}{\partial x_{1}}\rightarrow 0$ as $x_{2}\rightarrow
0$; If $x_{1}$ or (and) $x_{2}$ is a sparse feature, the case is
$\frac{\partial\hat{y}}{\partial x_{2}}\rightarrow 0$ as
$x_{1}\rightarrow\vec{0}$, or (and) $\frac{\partial\hat{y}}{\partial
x_{1}}\rightarrow 0$ as $x_{2}\rightarrow\vec{0}$.

Some dense features and sparse features are more or less related to the
prediction results. Thus, we can quantify the impacts of features through the
weights of individual features. However, as for cross features, there are
individual features less related to the prediction, but their combinations
lead to a key impact. From a dynamic perspective, inspired by manual feature
engineering on cross features, we observe a cross feature like that, its value
increases significantly as the values of the individual features increase
jointly, but changes gently as one of the individual features approaches to
zero or zero vector.

Based on the definition, cross features can be generalized to high-order
cases. If we consider individual features as order-$1$st features, an
order-$k$th cross feature combines $k$ individual features, where some of them
could be less related to the prediction, but their feature combination has a
significant impact. In this way, we make the following assumption.

#### Assumption.

For any individual features, their order-$k$th cross feature can be expressed
by a multivariate polynomial of degree $k$.

When cross dense features are expressed by polynomials, we can simply treat as
successive multiplications of the associated feature values. As for cross
sparse features, we need the inner product of two feature vectors, since
sparse features are represented as vectors of fields. However, it is
unrealistic to enumerate all cross features to high orders, because it not
only requires huge storage and computation, but also causes the model to be
unable to generalize to unseen feature combinations. Therefore, the explicit
structures of modeling cross features based on our assumption will be
introduced, and is expressive, while possessing both generalization and high
efficiency.

#### Cross layers on dense features.

Here we introduce a novel cross layer for crossing dense features. Cross
layers have the following formula:

$\displaystyle\bm{C_{1}}$
$\displaystyle=\bm{D}\cdot\bm{D^{\mathsf{T}}}\cdot\bm{W_{C,0}}+\bm{b_{C,0}},\quad\bm{O^{C}_{1}}=[\bm{D};\bm{C_{1}}],$
(1) $\displaystyle\bm{C_{l+1}}$
$\displaystyle=\bm{D}\cdot\bm{C_{l}^{\mathsf{T}}}\cdot\bm{W_{C,l}}+\bm{b_{C,l}},\quad\bm{O^{C}_{l+1}}=[\bm{O^{C}_{l}};\bm{C_{l+1}}],$

where $\bm{D}\in\mathbb{R}^{M}$ indicates the input dense features, and
$\bm{C_{l}}\in\mathbb{R}^{M}$ is a column vector denoting the order-$(l+1)$th
cross features. Later we prove how $\bm{C_{l}}$ expresses multivariate
polynomials of degree $(l+1)$ after weighted mapping.
$\bm{W_{C,l}},\bm{b_{C,l}}\in\mathbb{R}^{M}$ are the weight and bias
parameters respectively, and $\bm{O^{C}_{l}},\bm{O^{C}_{l+1}}$ denote the
outputs from the $l$-th and the $(l+1)$-th cross layers.

We denote $\bm{\alpha}=[\alpha_{1},\cdots,\alpha_{M}]$. If the proposed cross
layer expresses any cross features of order-$(l+1)$th, it could approximate to
any multivariate polynomials of degree $(l+1)$, denoted as $P_{l+1}(\bm{D})$:

$P_{l+1}(\bm{D})=\bigg{\\{}\sum_{\bm{\alpha}}W_{\bm{\alpha}}D_{1}^{\alpha_{1}}D_{2}^{\alpha_{2}}\cdots
D_{M}^{\alpha_{M}}\,\bigg{|}\,|\bm{\alpha}|=l+1\bigg{\\}},$ (2)

where $|\bm{\alpha}|=\sum_{i=1}^{M}\alpha_{i}$. For simplicity, here we use
$\bm{W^{i}}=[W^{i}_{1},W^{i}_{2},\cdots,W^{i}_{M}]$ to denote the original
subscript of $\bm{W_{C,i}}$. We study the coefficient $\hat{W_{\bm{\alpha}}}$
given by $\bm{C_{l}^{\mathsf{T}}}\cdot\bm{W^{l}}$ from cross layers, since it
constitutes the output $\bm{O_{l+1}^{C}}$ from the $(l+1)$-th cross layer.
Moreover, the following derivations do not include bias terms. Then:

$\begin{array}[]{l}\bm{C_{l}^{\mathsf{T}}}\cdot\bm{W^{l}}=\Big{(}\bm{C_{l-1}^{\mathsf{T}}}\cdot\bm{W^{l-1}}\Big{)}\cdot\Big{(}\bm{D^{\mathsf{T}}}\cdot\bm{W^{l}}\Big{)}=\prod_{i=0}^{l}\bm{D^{\mathsf{T}}}\cdot\bm{W^{i}}\vspace{0.1cm}\\\
=\prod_{i=0}^{l}[D_{1},D_{2},\cdots,D_{M}]^{\mathsf{T}}\cdot[W^{i}_{1},W^{i}_{2},\cdots,W^{i}_{M}].\end{array}$
(3)

Afterwards, let $\bm{I}$ denotes the multi-index vectors of orders
$[0,1,\cdots,l]$, and $I_{j}$ denotes the order of field $j$. Clearly
$\bm{C_{l}^{\mathsf{T}}}\cdot\bm{W^{l}}$ from cross layers approaches the
coefficient $\hat{W_{\bm{\alpha}}}$ as:

$\hat{W_{\bm{\alpha}}}=\sum_{k=1}^{M}\sum_{|\bm{I}|=\alpha_{k}}\prod_{j=1}^{M}W_{j}^{I_{j}}.\vspace{-0.2cm}$
(4)

With $\bm{C_{l}^{\mathsf{T}}}\cdot\bm{W^{l}}$ approximate to multivariate
polynomials of degree $(l+1)$, the output $\bm{O^{C}_{l+1}}$ from the
$(l+1)$-th cross layer that includes all cross features to order-$(l+1)$th
could approximate polynomials in the following class:

$P_{l+1}(\bm{D})=\bigg{\\{}\sum_{\bm{\alpha}}W_{\bm{\alpha}}D_{1}^{\alpha_{1}}D_{2}^{\alpha_{2}}\cdots
D_{M}^{\alpha_{M}}\,\bigg{|}\,0\leq|\bm{\alpha}|\leq
l+1\bigg{\\}}.\vspace{-0.3cm}$ (5)

Therefore, cross layer is extremely expressive to cross dense features up to
explicit orders. Besides, by polynomial approximation, vectors of orders cross
different fields, which results in cross features relate to interdependent
weighted parameters. In other words, cross layers have each cross feature
learned some independent parameters from other features, while the weights of
a cross term could share corresponding parameters, which enables the model to
generalize to rare or never co-occurred feature combinations.

#### Efficiency.

The number of parameters involved in $l$ cross layers is $(M\cdot l\cdot 2)$.
The time complexity of cross layers is $O(M\cdot l)$, increasing linearly with
the dimensions of the dense features. Therefore, cross layers introduce
negligible complexity compared to plain neural networks. Noted that, the
efficiency benefits from the rank-one property of
$\bm{D}\cdot\bm{C_{l}^{\mathsf{T}}}$, which allows the generation of cross
features without storing the entire matrix.

#### Embedding and Product Layer on Sparse Features.

As sparse features $\bm{S}$ are represented as vectors of one-hot encoding of
high-dimensional spaces, we employ an embedding layer to transform these one-
hot encoding vectors into dense vectors $\bm{E}$ as:

$\displaystyle\bm{E}$
$\displaystyle=[\bm{E_{1}},\cdots,\bm{E_{i}},\cdots,\bm{E_{N}}],$ (6)
$\displaystyle\bm{E_{i}}$
$\displaystyle=\bm{W_{E,i}}\cdot\bm{S_{i}},\,\big{(}\bm{E_{i}}\in\mathbb{R}^{K},i=1,\cdots,N\big{)}$

where $\bm{S_{i}}$ indicates the input sparse feature of field $i$,
$\bm{W_{E,i}}$ is the embedding weights, $K$ denotes the embedding size, and
$\bm{E_{i}}$ denotes the feature embedding of field $i$.

Afterwards, we can propose a product layer for cross sparse features. First,
we donate order-2nd cross sparse features as $\bm{P_{2}}$, and order-1st
sparse features as $\bm{P_{1}}$, thus the output of product layer is
$\bm{O^{P}}=[\bm{P_{1}};\bm{P_{2}}]$.

The cross feature of two sparse features of field $i$ and field $j$ equals the
inner product of two embedding vectors as
$\langle\bm{E_{i}},\bm{E_{j}}\rangle$. As for represention of vectors, we
concatenate the weighted sums of inner products to formulate order-2nd cross
features as:

$\bm{P_{2}}=[P_{2}^{1},\cdots,P_{2}^{t},\cdots,P_{2}^{T}],$ (7)

where $T$ is the size of the product layer, and $\bm{P_{2}}$ is a $T$
dimensional vector, of each dimension $P_{2}^{t}$ denotes a weighted sum of
inner products of two sparse features. Thus, we have
$P_{2}^{t}=\sum_{i=1}^{N}\sum_{j=1}^{N}W^{2,t}_{i,j}\langle\bm{E_{i}},\bm{E_{j}}\rangle$.
We assume that the weighted parameter
$W^{2,t}_{i,j}=\Theta_{i}^{t}\cdot\Theta_{j}^{t}$ for reduction, so
$P_{2}^{t}$ can be given as:

$\displaystyle
P_{2}^{t}=\sum_{i=1}^{N}\sum_{j=1}^{N}\Theta_{i}^{t}\cdot\Theta_{j}^{t}\langle\bm{E_{i}},\bm{E_{j}}\rangle=\bigg{\langle}\sum_{i=1}^{N}\Delta_{i}^{t}\cdot\bm{E_{i}},\sum_{j=1}^{N}\Delta_{j}^{t}\cdot\bm{E_{j}}\bigg{\rangle}.$
(8)

The feature vector of order-1st features has a similar formula as follows:

$\bm{P_{1}}=[P_{1}^{1},\cdots,P_{1}^{t},\cdots,P_{1}^{T}],$ (9)

where $\bm{P_{1}}$ is a $T$ dimensional vector, of each dimension $P_{1}^{t}$
denotes a weighted sum of sparse features. The weighted feature can be
expressed as inner product $\langle\bm{W^{1,t}_{i}},\bm{E_{i}}\rangle$. Thus,
we have $P_{1}^{t}=\sum_{i=1}^{N}\langle\bm{W^{1,t}_{i}},\bm{E_{i}}\rangle$.

#### Efficiency.

The training time complexity of embedding layer is $O(N\cdot K)$. As for
product layer, the time complexity of training order-1st features is $O(N\cdot
K\cdot T)$, through Eq. 8 we reduce the time of training order-2nd cross
features from $O(N^{2}\cdot K\cdot T)$ to $O(N\cdot K\cdot T)$. So the overall
time complexity of product layer becomes $O(N\cdot K\cdot T)$. Moreover, inner
product operations are easily accelerated through parallelization by GPUs.
Therefore, the total time complexity of embedding layer and product layer on
sparse features can be accelerated as $O(N\cdot K)$ in practice.

###  3.2 Feature Concatenation

For the Feature Concatenation stage, in order to learn feature interactions of
different structures, cross dense features $\bm{O^{C}}$ and cross sparse
features $\bm{O^{P}}$ are fed into an attention mechanism to adaptively learn
the weights for feature combinations. Formally, the attention network is
defined as:

$\displaystyle a_{i,j}$
$\displaystyle=\bm{q^{\mathsf{T}}}\tanh(\bm{W_{C}}\cdot\bm{C_{i}}+\bm{W_{P}}\cdot\bm{P_{j}}+\bm{b}),$
(10) $\displaystyle\beta_{i,j}$
$\displaystyle=\frac{\exp(a_{i,j})}{\sum_{i,j}\exp(a_{i,j})},$

where $\bm{C_{i}}\in\mathbb{R}^{M}$ is a feature vector of cross dense
features $\bm{O^{C}}$ (Noted $\bm{C_{0}}=\bm{D}$),
$\bm{P_{j}}\in\mathbb{R}^{T}$ is a feature vector of cross sparse features
$\bm{O^{P}}$, and $\bm{W_{C}}\in\mathbb{R}^{Z\times M}$,
$\bm{W_{P}}\in\mathbb{R}^{Z\times T}$, $\bm{b}\in\mathbb{R}^{Z}$,
$\bm{q}\in\mathbb{R}^{Z}$ are learning model parameters. $Z$ is the hidden
layer size of the attention network, denoted as attention factor.
$\beta_{i,j}$ is the normalized weight. We use the $\tanh(\cdot)$ as the
activation function, which empirically shows good performance. The output of
the attention-based weighted sum pooling is formulated as follows:

$\bm{H^{0}}=\sum_{i,j}\beta_{i,j}[\bm{C_{i}};\bm{P_{j}}].$ (11)

where $\bm{H^{0}}$ is an $(M+Z)$ dimensional vector, which compresses all
hybrid structure feature interactions by distinguishing their importance.

#### Efficiency.

The training time complexity of the Feature Concatenation stage is mainly
determined by Eq. 10 as $O(Z^{2}\cdot(M+T))$, which can be accelerated through
parallelization as $O(Z\cdot(M+T))$ in practice.

###  3.3 Feature Selection

In the Feature Selection stage, we employ an MLP to capture non-linear
interactions and the relative importance of cross features. The deep layers
and the output layer respectively have the following formula:

$\displaystyle\bm{H^{i}}$
$\displaystyle={\rm{ReLU}}(\bm{W_{H,i-1}}\cdot\bm{H^{i-1}}+\bm{b_{H,i-1}}),$
(12) $\displaystyle O^{G}$
$\displaystyle={\rm{Sigmoid}}(\bm{W_{H,i}}\cdot\bm{H^{i}}+\bm{b_{H,i}}),$

where $\bm{H^{i}},\bm{H^{i-1}}$ are hidden layers, ${\rm{ReLU}}(\cdot)$ and
${\rm{Sigmoid}}(\cdot)$ are activation functions,
$\bm{W_{H,i}},\bm{W_{H,i-1}}$ are weights, and $\bm{b_{H,i}},\bm{b_{H,i-1}}$
are biases, and $O^{G}$ is the output result. For CTR prediction, the loss
function is the log loss as follows:

$\bm{\mathcal{L}}=-\frac{1}{n}\sum^{n}_{i=1}\,y_{i}\log(O^{G})+(1-y_{i})\log(1-O^{G}),$
(13)

where $n$ is the total number of training instances. The optimization process
is to minimize the following objective function:

$\bm{\mathcal{J}}=\bm{\mathcal{L}}+\lambda||\bm{\Theta}||,$ (14)

where $\lambda$ denotes the regularization term and $\bm{\Theta}$ denotes the
set of parameters, including cross layers, embedding layer, product layer,
attention layer, deep layers and output layer.

##  4 Experiments

In this section, extensive experiments are conducted to answer the following
research questions:

RQ1:

How does XCrossNet perform compared with the state-of-the-art CTR prediction
models?

RQ2:

How does the feature dimensionality imbalance impact CTR prediction?

RQ3:

How do different hyper-parameter settings impact the performance of XCrossNet?

RQ4:

How effective is each component of XCrossNet?

Dataset |  #instances |  #dimension |  #fields |  positive ratio
---|---|---|---|---
Criteo |  $1\times 10^{8}$ |  $1\times 10^{6}$ |  $39$ |  $0.50$
Avazu |  $4\times 10^{7}$ |  $6\times 10^{5}$ |  $24$ |  $0.17$
iPinYou |  $2\times 10^{7}$ |  $9\times 10^{5}$ |  $16$ |  $0.07$
Table 1: Statistics of Experimental Datasets.

###  4.1 Experimental Setup

#### Datasets.

Experiments are conducted on the following three public datasets. Criteo
dataset contains one month of ad click logs. We select 7 consecutive days of
samples as the training set while the next one day for evaluation. To counter
label imbalance, negative down-sampling is applied to keep the positive ratio
roughly at 50%. Avazu dataset was released in the CTR prediction contest on
Kaggle in 2014. 80% of randomly shuffled data is allotted to training and
validation with 20% for testing. iPinYou dataset was published in the iPinYou
RTB Bidding Algorithm Competition in 2013. We utilize seasons 2 and 3 as our
dataset. To make a fair comparison, we process the data in the three datasets
exactly the same as in (2018; 2019; 2020). Table 1 summarizes the
characteristics of the three datasets.

Params |  Criteo |  Avazu |  iPinYou
---|---|---|---
Genaral |  bs=2000; lr=1e-3 opt=Adam |  bs=2000; lr=1e-3 opt=Adam |  bs=2000; lr=1e-3 opt=Adam l2_e=1e-6
LR |  – |  – |  –
GBDT |  depth=25 |  depth=18 |  depth=6
|  #tree=1300 |  #tree=1000 |  #tree=600
FM, AFM |  K=20; t=0.01; R=32 |  K=40; t=1;R=256 |  K=20; t=1;R=256
|  l2_a=0.1 |  l2_a=0.1 |  l2_a=0.1
|  sub-net=[40,1] |  sub-net=[80,1] |  sub-net=[40,1]
FFM |  K=4 |  K=4 |  K=4
CCPM |  K=20 |  K=40 |  K=20
|  kernel=[7$\times$256] |  kernel=[7$\times$128] |  kernel=[7$\times$128]
|  net=[256$\times$3,1] |  net=[128$\times$3,1] |  net=[128$\times$3,1]
Wide&Deep |  K=20 |  K=40 |  K=20
|  net=[400$\times$3,1] |  net=[700$\times$5,1] |  net=[300$\times$3,1]
Deep&Cross |  K=20 |  K=40 |  K=20
|  net=[400$\times$3,1] |  net=[700$\times$5,1] |  net=[300$\times$3,1]
|  Cross=[200$\times$3] |  Cross=[200$\times$2] |  Cross=[200$\times$3]
FNN, DeepFM IPNN |  K=20 |  K=40 |  K=20
|  net=[700$\times$5,1] |  net=[500$\times$5,1] |  net=[300$\times$3,1]
|  LN=true |  LN=true |  LN=true
PIN |  K=20 |  K=40 |  K=20
|  net=[700$\times$5,1] |  net=[500$\times$5,1] |  net=[300$\times$3,1]
|  sub-net=[40,5] |  sub-net=[40,5] |  sub-net=[40,5]
|  LN=true |  LN=true |  LN=true
xDeepFM |  K=20 net=[400$\times$3,1] CIN=[100$\times$4] |  K=40 net=[700$\times$5,1] CIN=[100$\times$2] |  K=20 net=[300$\times$3,1] CIN=[100$\times$4] LN=true
FGCNN |  K=20 conv=9*1 kernel=[38,40,42,44] new=[3,3,3,3] BN=true net=[4096,2048,1] |  K=40 conv=7*1 kernel=[14,16,18,20] new=[3,3,3,3] BN=true net=[4096,2048, 1024,512,1] |  K=20 conv=3*1 kernel=[4,6,8] new=[1,1,1] BN=true net=[600$\times$3,1]
AutoGroup |  K=20 lr_h=10 $\tau$=0.1 net=[1024,512,256,1] n_p=[35,390,300, 500,450,150] q=6 |  K=40 lr_h=1e4 $\tau$=0.01 net=[1024,512,256,1] n_p=[15,130,170, 210,250,290] q=6 |  K=20 lr_h=1e3 $\tau$=0.01 net=[300$\times$3,1] n_p=[14,120,160, 200,240] q=5
XCrossNet |  K=20; T=350 |  K=40; T=250 |  K=20; T=150
|  R=256 |  R=256 |  R=256
|  n_c=6 |  n_c=4 |  n_c=4
|  net=[400$\times$3] |  net=[700$\times$3] |  net=[300$\times$3]

Note: bs=batch size, opt=optimizer, lr=learning rate, l2_e=$l_{2}$
regularisation on embedding layer, t=softmax temperature, l2_a=$l_{2}$
regularisation on attention network, R=size of attention factor, K=embedding
size, net=MLP structure, sub-net=micro network, LN=layer normalization,
BN=batch normalization, lr_h=learning rate for structural parameters,
$\tau$=Gumbel-softmax temperature, q=maximum order for explicit interactions,
n_p=number of feature sets in each order, T=size of product layers, n_c=number
of cross layers on the depth.

Table 2: Hyper-parameter settings.

#### Evaluation Metrics.

AUC (Area Under ROC) and Logloss (cross entropy) are selected as our
evaluation metrics. Note that an improvement of 0.001-level in AUC or Logloss
is usually regarded as being significant for CTR prediction, because it will
lead to a large increase in company’s revenue as a large user base, which has
been pointed in many existing work (2016; 2017; 2017; 2020).

#### Baselines.

As aforementioned, we use following highly related state-of-the-art models as
baselines: LR (2012), GBDT (2001), FM (2010), AFM (2017), FFM (2016), CCPM
(2015), Wide&Deep (2016), Deep&Cross (2017) and its shallow part Cross
Network, FNN (2016), DeepFM (2017), IPNN (2016), PIN (2018), xDeepFM (2018)
and its shallow part CIN, FGCNN (2019), and AutoGroup (2020).

#### Hyper-parameter settings.

Table 2 summarizes the hyper-parameters for each model. To be fair and achieve
the best performance, hyper-parameters of each model are tuned on validation
set by grid-seaching carefully 111All the source code will be released after
acceptance..

###  4.2 Overall Performance (RQ1)

The experiments for XCrossNet and the best baseline model are repeated 10
times by changing the random seeds. Table 3 summarizes the performance of all
compared methods on three large-scale public datasets, while the training time
on Tesla K80 GPUs is shown in Figure 3 for comparison of efficiency. We have
the following key observations:

Model | Criteo | Avazu | iPinYou
---|---|---|---
|  AUC(%) |  Logloss |  AUC(%) |  Logloss |  AUC(%) |  Logloss
LR |  78.00 |  0.5631 |  76.76 |  0.3868 |  76.38 |  0.005691
GBDT |  78.62 |  0.5560 |  77.53 |  0.3824 |  76.90 |  0.005578
FM |  79.09 |  0.5500 |  77.93 |  0.3805 |  77.17 |  0.005595
AFM |  79.13 |  0.5517 |  78.06 |  0.3794 |  77.71 |  0.005562
FFM |  79.80 |  0.5438 |  78.31 |  0.3781 |  76.18 |  0.005695
CCPM |  79.55 |  0.5469 |  78.12 |  0.3800 |  77.65 |  0.005593
Wide&Deep |  79.77 |  0.5446 |  78.10 |  0.3803 |  77.86 |  0.005571
Cross |  78.70 |  0.5550 |  77.62 |  0.3820 |  76.70 |  0.005603
Deep&Cross |  79.76 |  0.5445 |  78.11 |  0.3801 |  77.88 |  0.005569
FNN |  79.87 |  0.5428 |  78.30 |  0.3778 |  77.82 |  0.005573
DeepFM |  79.91 |  0.5423 |  78.36 |  0.3777 |  77.92 |  0.005588
IPNN |  80.13 |  0.5399 |  78.68 |  0.3757 |  78.17 |  0.005549
PIN |  80.18 |  0.5394 |  78.72 |  0.3755 |  78.22 |  0.005547
CIN |  78.81 |  0.5538 |  78.02 |  0.3797 |  77.30 |  0.005592
xDeepFM |  80.06 |  0.5408 |  78.55 |  0.3766 |  78.04 |  0.005555
FGCNN |  80.22 |  0.5389 |  78.82 |  0.3747 |  77.85 |  0.005612
AutoGroup |  80.28 |  0.5384 |  79.15 |  0.3729 |  78.59 |  0.005528
XCrossNet |  80.67⋆ |  0.5339⋆ |  79.54⋆ |  0.3698⋆ |  78.72∗ |  0.005507∗
|  $\pm$0.01 |  $\pm$0.0001 |  $\pm$0.03 |  $\pm$0.0002 |  $\pm$0.03 |  $\pm$ 4E-6

Note: ⋆ and ∗ repsent siginicant level $p$-value $<10^{-7}$ and $p$-value
$<0.05$
of comparing XCrossNet with the best baseline.

Table 3: Performance comparison of different models.

Firstly, most neural network models outperform linear models (i.e., LR), tree-
based models (i.e., GBDT), and FM variants (i.e., FM, FFM, AFM), which
indicates neural network can learn non-linear feature interactions and endow
better expressive ability. Meanwhile, comparing IPNN and PIN with FNN and
Wide&Deep based models, we find that explicitly modeling low-order feature
interactions can simplify the training of neural networks and boost the
performance. AutoGroup and FGCNN are overall the best baselines on three
datasets, mainly because some high-order feature interactions are captured via
CNN or feature grouping, and contribute in final predictions.

Secondly, XCrossNet consistently achieves the best performance on all
datasets. In detail, XCrossNet significantly outperforms the best baseline in
terms of AUC and Logloss on $p$-value $<10^{-7}$ level on Criteo and Avazu
datasets, while $p$-value $<0.05$ on iPinYou dataset. From training time
comparison, we can observe XCrossNet is more efficient than field-aware
models, mainly because they further allow each feature to learn several
vectors where each vector is associated with a field, which leads to huge
parameter consuming and time consuming.

Figure 3: Training time comparison of different models. Figure 4: Impact of
feature dimensionality imbalance.

###  4.3  Feature Dimensionality Imbalance Study (RQ2)

In XCrossNet, we denote
$\left.{\frac{\dim(O^{C})}{\dim(O^{P})}}\middle/{\frac{M}{N}}\right.$ as the
balance index of dimensions of dense and sparse features. Noted that, the
dimension of cross dense features $O^{C}$ equals $M\cdot l$, increasing with
the depth of cross layers. As for Criteo dataset, $M=13$ and $N=26$, we set
the depth of cross layers from 1 to 8, while the corresponding dimension of
cross dense features is from 13 to 104. Experimental results are shown in
Figure 4 in terms of AUC. We can observe that increasing the depth of cross
layers benefit XCrossNet to achieve stable improvements on AUC performance,
mainly because the higher dimensions of cross dense features are able to boost
the balance index, which results in relatively balanced impacts of dense and
sparse features on prediction.

(a) Embedding size

(b) Size of attention factor

(c) Activation functions

(d) Number of deep layers

Figure 5: Impact of network hyper-parameters on AUC performance.

(a) Embedding size

(b) Size of attention factor

(c) Activation functions

(d) Number of deep layers

Figure 6:  Impact of network hyper-parameters on Logloss performance.

###  4.4 Hyper-parameter Study (RQ3)

We study the impact of hyper-parameters of XCrossNet, including (1) embedding
size; (2) size of attention factor; (3) activation functions; (4) number of
deep layers.

#### Embedding size.

Figures 5a and 6a demonstrate the impact of embedding size. We can observe
that model performance on Criteo and Avazu datasets boosts steadily when the
embedding size increase from 4 to 20. Even with very low embedding sizes,
XCrossNet still has comparable performance to some popular Wide&Deep based
models with high embedding size. Specifically, on Criteo dataset, XCrossNet
achieves AUC$>$0.800 and Logloss$<$0.541 with embedding size set as 10, which
is better than DeepFM with embedding size set as 20.

Model | Criteo | Avazu
---|---|---
|  AUC(%) |  Logloss |  AUC(%) |  Logloss
XCrossNet-CL |  79.55 |  0.5471 |  79.16 |  0.3722
|  $\downarrow$ 1.4% |  $\uparrow$ 2.5% |  $\downarrow$ 0.49% |  $\uparrow$ 0.68%
XCrossNet-PL |  79.20 |  0.5524 |  79.01 |  0.3734
|  $\downarrow$ 1.8% |  $\uparrow$ 3.5% |  $\downarrow$ 0.68% |  $\uparrow$ 1.0%
XCrossNet-AL |  80.16 |  0.5402 |  79.15 |  0.3724
|  $\downarrow$ 0.64% |  $\uparrow$ 1.2% |  $\downarrow$ 0.50% |  $\uparrow$ 0.73%
XCrossNet-DL |  79.90 |  0.5426 |  78.83 |  0.3742
|  $\downarrow$ 0.97% |  $\uparrow$ 1.6% |  $\downarrow$ 0.91% |  $\uparrow$ 1.2%
Table 4: Performance of each component of XCrossNet.

#### Size of attention factor.

As shown in Figures 5b and 6b, the model performance on Criteo dataset boosts
while size of the attention factor increasing from 64 to 320, while on Avazu
dataset, 256 is a more suitable setting for the size of attention factor to
avoid the model being to over-complicated.

#### Activation functions.

As shown in Figures 5c and 6c, ReLU is indeed more appropiate for hidden
neurons of deep layers compared with different activation functions.

#### Number of deep layers.

Figures 5d and 6d demonstrate the impact of the number of deep layers. The
model performance boosts with depth of MLP at the beginning on Criteo dataset.
However, it starts to degrade when the depth of MLP is set to gteater than 3,
mainly because of overfitting evidenced by the observation that the training
error still keeps decreasing. For Avazu dataset, the model performance also
degrades when the depth of MLP is set to greater than 3.

###  4.5 Ablation Study (RQ4)

We conduct an ablation study to isolate the relative importance of each
component of XCrossNet. We downgrade the current model as 4 models: XCrossNet-
CL removes the cross layers on dense features; XCrossNet-PL removes the
product layer on sparse features; XCrossNet-AL removes the attention layer in
the Feature Concatenation stage; XCrossNet-DL removes the deep layers in the
Feature Selection stage. From Table 4, we can observe that cross layers and
product layer are beneficial to the model, which implies XCrossNet can
represent cross dense feature and cross sparse features effectively.
Meanwhile, it is necessary and effective to interact dense and sparse features
by the attention layer for feature combination. Lastly, the model performance
degrades apparently when we remove deep layers, because deep layers help to
capture non-linear feature interactions and their relative importance from
extensive cross features for prediction.

##  5 Conclusion

In summary, this paper presents a novel feature representation model, namely
Extreme Cross Network (XCrossNet), for improving CTR prediction in commercial
recommender systems. We separately design a cross layer for crossing dense
features and a product layer for crossing sparse features, then the cross
features interact through an attention mechanism and an MLP for combination
and selection. The main contribution of our approach is to represent both
dense and sparse feature interactions in an explicit and efficient way.
Moreover, it is feasible to balance the dimension of dense features up to the
dimension of sparse features for solving the feature dimensionality imbalance
problem. Empirical studies verified the effectiveness of our model on three
large-scale public datasets. The XCrossNet framework of three stages is a new
hybrid feature structure-oriented model. We encourage more feature crossing
network layers to be proposed with our framework.

## References

  * Beutel et al. (2018) Beutel, A.; Covington, P.; Jain, S.; Xu, C.; Li, J.; Gatto, V.; and Chi, E. H. 2018\.  Latent cross: making use of context in recurrent recommender systems.  In _WSDM_ , 46–54. ACM.
  * Blondel et al. (2016) Blondel, M.; Fujino, A.; Ueda, N.; and Ishihata, M. 2016.  Higher-order factorization machines.  In _NeurIPS_ , 3351–3359.
  * Chang et al. (2010) Chang, Y.; Hsieh, C.; Chang, K.; Ringgaard, M.; and Lin, C. 2010.  Training and testing low-degree polynomial data mappings via linear SVM.  _Journal of Machine Learning Research (JMLR)_ 11: 1471–1490.
  * Chapelle, Manavoglu, and Rosales (2014) Chapelle, O.; Manavoglu, E.; and Rosales, R. 2014.  Simple and scalable response prediction for display advertising.  _ACM Transaction on Intelligent Systems and Technology (ACM TIST)_ 5(4): 1–34.
  * Cheng et al. (2016) Cheng, H.-T.; Koc, L.; Harmsen, J.; Shaked, T.; Chandra, T.; Aradhye, H.; Anderson, G.; Corrado, G.; Chai, W.; Ispir, M.; et al. 2016.  Wide & deep learning for recommender systems.  In _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems_ , 7–10.
  * Feng et al. (2019) Feng, Y.; Lv, F.; Shen, W.; Wang, M.; Sun, F.; Zhu, Y.; and Yang, K. 2019.  Deep session interest network for click-through rate prediction.  In _IJCAI_ , 2301–2307.
  * Friedman (2001) Friedman, J. H. 2001.  Greedy function approximation: a gradient boosting machine.  _Annals of Statistics_ 1189–1232.
  * Guo et al. (2017) Guo, H.; Tang, R.; Ye, Y.; Li, Z.; and He, X. 2017.  DeepFM: A factorization-machine based neural network for CTR prediction.  In _IJCAI_ , 1725–1731.
  * He and Chua (2017) He, X.; and Chua, T.-S. 2017.  Neural factorization machines for sparse predictive analytics.  In _SIGIR_ , 355–364.
  * Juan et al. (2016) Juan, Y.; Zhuang, Y.; Chin, W.-S.; and Lin, C.-J. 2016.  Field-aware factorization machines for CTR prediction.  In _RecSys_ , 43–50.
  * Lee et al. (2012) Lee, K.; Orten, B.; Dasdan, A.; and Li, W. 2012.  Estimating conversion rate in display advertising from past performance data.  In _SIGKDD_ , 768–776. ACM.
  * Lian et al. (2018) Lian, J.; Zhou, X.; Zhang, F.; Chen, Z.; Xie, X.; and Sun, G. 2018.  xdeepfm: Combining explicit and implicit feature interactions for recommender systems.  In _SIGKDD_ , 1754–1763.
  * Liu et al. (2019) Liu, B.; Tang, R.; Chen, Y.; Yu, J.; Guo, H.; and Zhang, Y. 2019.  Feature generation by convolutional neural network for click-through rate prediction.  In _WWW_ , 1119–1129.
  * Liu et al. (2020) Liu, B.; Xue, N.; Guo, H.; Tang, R.; Zafeiriou, S.; He, X.; and Li, Z. 2020.  AutoGroup: Automatic feature grouping for modelling explicit high-order feature interactions in CTR prediction.  In _SIGIR_ , 199–208. ACM.
  * Liu et al. (2015) Liu, Q.; Yu, F.; Wu, S.; and Wang, L. 2015.  A convolutional click prediction model.  In _CIKM_ , 1743–1746.
  * Lu et al. (2020) Lu, W.; Yu, Y.; Chang, Y.; Wang, Z.; Li, C.; and Yuan, B. 2020.  A dual input-aware factorization machine for CTR prediction.  In _IJCAI_ , 3139–3145.
  * Luo et al. (2019) Luo, Y.; Wang, M.; Zhou, H.; Yao, Q.; Tu, W.-W.; Chen, Y.; Dai, W.; and Yang, Q. 2019.  Autocross: Automatic feature crossing for tabular data in real-world applications.  In _SIGKDD_ , 1936–1945.
  * Lyu et al. (2020) Lyu, Z.; Dong, Y.; Huo, C.; and Ren, W. 2020.  Deep match to rank model for personalized click-through rate prediction.  In _AAAI_ , volume 34, 156–163.
  * McMahan et al. (2013) McMahan, H. B.; Holt, G.; Sculley, D.; Young, M.; Ebner, D.; Grady, J.; Nie, L.; Phillips, T.; Davydov, E.; Golovin, D.; Chikkerur, S.; Liu, D.; Wattenberg, M.; Hrafnkelsson, A. M.; Boulos, T.; and Kubica, J. 2013.  Ad click prediction: a view from the trenches.  In _SIGKDD_ , 1222–1230. ACM.
  * Ouyang et al. (2019) Ouyang, W.; Zhang, X.; Ren, S.; Qi, C.; Liu, Z.; and Du, Y. 2019.  Representation learning-assisted click-through rate prediction.  In _IJCAI_ , 4561–4567.
  * Qu et al. (2016) Qu, Y.; Cai, H.; Ren, K.; Zhang, W.; Yu, Y.; Wen, Y.; and Wang, J. 2016.  Product-based neural networks for user response prediction.  In _ICDM_ , 1149–1154. IEEE.
  * Qu et al. (2018) Qu, Y.; Fang, B.; Zhang, W.; Tang, R.; Niu, M.; Guo, H.; Yu, Y.; and He, X. 2018\.  Product-based neural networks for user response prediction over multi-field categorical data.  _ACM Transactions on Information Systems (ACM TOIS)_ 37(1): 1–35.
  * Rendle (2010) Rendle, S. 2010.  Factorization machines.  In _ICDM_ , 995–1000. IEEE.
  * Richardson, Dominowska, and Ragno (2007) Richardson, M.; Dominowska, E.; and Ragno, R. 2007.  Predicting clicks: estimating the click-through rate for new ads.  In _WWW_ , 521–530.
  * Shi et al. (2020) Shi, S.-T.; Zheng, W.; Tang, J.; Chen, Q.-G.; Hu, Y.; Zhu, J.; and Li, M. 2020.  Deep time-stream framework for click-through rate prediction by tracking interest evolution.  In _AAAI_ , 5726–5733.
  * Song et al. (2019) Song, W.; Shi, C.; Xiao, Z.; Duan, Z.; Xu, Y.; Zhang, M.; and Tang, J. 2019.  Autoint: Automatic feature interaction learning via self-attentive neural networks.  In _CIKM_ , 1161–1170. ACM.
  * Wang et al. (2017) Wang, R.; Fu, B.; Fu, G.; and Wang, M. 2017.  Deep & cross network for ad click predictions.  In _ADKDD_ , 1–7.
  * Xiao et al. (2017) Xiao, J.; Ye, H.; He, X.; Zhang, H.; Wu, F.; and Chua, T. 2017.  Attentional factorization machines: learning the weight of feature interactions via attention networks.  In _IJCAI_ , 3119–3125.
  * Yan et al. (2014) Yan, L.; Li, W.; Xue, G.; and Han, D. 2014.  Coupled group lasso for web-scale CTR prediction in display advertising.  In _ICML_ , volume 32, 802–810.
  * Zhang, Du, and Wang (2016) Zhang, W.; Du, T.; and Wang, J. 2016.  Deep learning over multi-field categorical data.  In _European Conference on Information Retrieval_ , 45–57. Springer.
  * Zhou et al. (2019) Zhou, G.; Mou, N.; Fan, Y.; Pi, Q.; Bian, W.; Zhou, C.; Zhu, X.; and Gai, K. 2019\.  Deep interest evolution network for click-through rate prediction.  In _AAAI_ , volume 33, 5941–5948.