# On the validity of classical partition function.

Sushil K. Singh
Department of Physics
SGTB Khalsa College
University of Delhi &Savinder Kaur
Department of Physics
SGTB Khalsa College
University of Delhi<EMAIL_ADDRESS>
###### Abstract

The partition function for a system of non-interacting $N-$particles can be
found by summing over all the states of the system. The classical partition
function for an ideal gas differs from Bosonic or Fermionic partition function
in the classical regime. Students find it difficult to follow the differences
arising out of incorrect counting by the classical partition function by
missing out on the indistinguishability of particles and Fermi-Bose
statistics. We present a pedagogical computer-based experiment to probe and
demonstrate the key differences in the partition functions (i) $Q$, for the
system of distinguishable and indistinguishable particles (ii) $B$, for Bosons
and (ii) $F$, for Fermions without formally using the single-particle
partition function.

_K_ eywords Partition Function $\cdot$ Gibbs Gas $\cdot$ Fermions and Bosons

##  1 Introduction

The quintessential role that partition function (PF) plays in finding
properties of a system is well established. The indistinguishability of
identical particles has been a central point in the understanding of
thermodynamic properties of the system. The thermodynamics function for a
classical system can be obtained by defining the appropriate partition
function

$Q^{*}_{N}\equiv\frac{1}{A}Q_{N}$

where $A$ is a normalization factor and $Q_{N}=\int e^{\beta E_{s}}$ is the PF
for indistinguishable $N$-particles. The conclusion that entropy should be an
additive function for macroscopic objects, led Gibbs to propose that the
constant $A$ should be proportional to $N!$ in order not to overcount
configurations of identical particles which are identical except for how the
particles are labeled [1, 2]. Though, the idea is totally unintelligible
within classical mechanics, because interchanging particles does lead to
classically different states, much has been deliberated on the appropriate
normalization factor $A$ [3, 4, 5, 6]. For most purposes, classical physics
suffices to describe a system of $N$-particles under laboratory conditions and
the indistinguishability is often incorporated by a judicious division by $N!$
[8, 9]. The factor $N!$, although correctly accounts for all states in which
no orbital is occupied by more than one atom, fails for mutiple occupancy [5].
We are tempted to summarize this discussion by saying that the number of
quantum states for identical Fermions $n_{F}$ and that for identical Bosons
$n_{B}$ are given by $n_{F}=n_{B}=n_{C}/N!$ where $n_{C}$ is the number of
states for identical classical atoms. The division by $N!$ overcounts Fermi
states and undercounts Bose states with the consequence that the PFs [5, 6]

$B_{N}>Q^{*}_{N}>F_{N}$

where $B_{N}$ and $F_{N}$ represent the PF for $N$ non-interacting Bosons and
Fermions respectively. Though the topic has been explored in detail [5, 4, 6],
the subtlety of the subject demands exploration and implementation in the
computational lab. Despite the significance of this result, a pedagogical
solution is not explicitly present in the literature. Our goal is to perform a
computer based experiment (written in open source software Python) to analyse
the differences between PFs, $Q_{N}$, $B_{N}$, $Q^{*}_{N}$, $F_{N}$ and the
state functions which are usually overlooked in textbooks.

##  2 Canonical Ensemble

Consider a system of $N$-particles occupying a volume $V$ maintained at a
constant temperature $T$, through thermal contact with a heat bath. A
macrostate for the system can be realized by a large number of microstates
with equal a priori probabilities. The microstate is defined as any possible
individual configuration in the phase space. The system resides in any of
these microstates at an instant $t$ while continually switching from one
microstate to another. The more the number of available microstates the more
the system spends time in the corresponding macrostate. The state functions
are then expected to be "averaged" over these microstates through which the
system passes [7]. An entirely different approach would be to imagine all
these microstates to be occupied by copies of the system at the same instant.
The average behavior of the system in this collection or ensemble would be
identical to the time-averaged behavior of the given system. Statistically,
the system evolves towards a macrostate with the largest number of microstates
and spends an "overwhelmingly" large fraction of its time in this macrostate
[1, 7, 8].

###  2.1 The “Canonical” Distribution Function

For the system in equilibrium with a thermal reservoir at temperature $T$, the
Boltzmann factor $exp(-\beta E_{s})$ determines the “canonical” probability
distribution $P_{s}$ for state $s$

$P_{s}=\frac{e^{-\beta E_{s}}}{\sum_{r}^{allstates}e^{-\beta E_{r}}}$ (1)

where $s,r$ refer to complete orthonormal set of energy eigenstates of the
system. The properties of the reservoir appear only in the scalar factor
$\beta=1/k_{B}T$ where $k_{B}$ is the Boltzmann’s constant.

###  2.2 The Partition Function

The general recipe to extract information about the various macroscopic
properties of the given system is to find the quantity $Q_{N}(V,T)$ called the
“sum-over-states” (German: Zustandssumme) or the partition function

$Q_{N}(V,T)=\sum_{s}^{\textrm{all states}}e^{-\beta E_{s}}$ (2)

The dependence on $N$ and $V$ comes through the energy eigenvalues $E_{s}$. In
most physical cases the energy levels accessible to a system are degenerate,
that is, one has a group of states, $g_{s}$ in number, all belonging to the
same energy value $E_{s}$. In such cases it is more useful to write the
partition function as

$Q_{N}(V,T)=\sum_{s}^{\textrm{energy states}}g_{s}e^{-\beta E_{s}}$ (3)

The partition function counts the number of states available to the system.

The connection with thermodynamics is made by the Helmholtz Free energy
$(H=E-TS)$

$H_{N}(V,T)=-\frac{1}{\beta}\ln Q_{N}$ (4)

which is used to determine all other state functions of the system.

##  3 Computer Based Experiment (CBE)

Initially, we consider a system of distinguishable $N$-particles in which each
particle can access $n$ number of single-particle energy levels
$\epsilon_{r}$. There are $M=n^{N}$ possible combination of energy states
available to the system. Each state of the system is characterized by energy
$E_{s}$ ($s=1,...,M$)

$E_{s}=\epsilon_{r_{1}}^{s}+\epsilon_{r_{2}}^{s}+\dots+\epsilon_{r_{N}}^{s}$
(5)

where $\epsilon_{r_{m}}^{s}$ ($m=1,...,N$ and $r=1,...,n$) denotes energy of
single-particle state when the $m^{th}$ particle is in its $r^{th}$ level
while the system is in the $s^{th}$ state. The PF $Q_{N}$ can be written as

$Q_{N}=\sum_{s=1}^{M}e^{-\beta\sum_{m=1}^{N}\epsilon_{r_{m}}^{s}}=\sum_{s=1}^{M}\left(\prod_{m=1}^{N}e^{-\beta\epsilon_{r_{m}}^{s}}\right)$
(6)

For distinguishable particles it is often useful to define a single-particle
PF $q=\sum_{r}e^{-\beta\epsilon_{r}}$ where $\epsilon_{r}$ belongs to the
complete orthonormal set of single-particle observable energies. The
$N$-particle PF can then be found as $Q_{N}=q^{N}$. However, this single-
particle PF $q$ is not a relevant quantity when we are dealing with quantum
statistics, that is, Fermi-Dirac or Bose-Einstein statistics. The Fermionic
and Bosonic particles are indistinguishable and are not independent because of
the symmetry requirements of the wave functions [4, 8]. We thus approach the
problem without using the single-particle PF. We numerically calculate PFs for
an $N$-particle system with each particle being able to access $n$ energy
levels. All the three cases (a) the distinguishable system (b) the Bosonic
system and (c) the Fermionic system will be considered to find the differences
arising in the number of available states.

To illustrate the CBE, consider $2$ identical distinguishable particles each
of which can access the $3$ energy levels labelled $0$ ($\epsilon_{0}$), $1$
($\epsilon_{1}$) and $2$ ($\epsilon_{2}$). There are $3^{2}=9$ available
states $S_{p}(i,j)$ to the system where $p$ refers to a state in which the
particle $1$ is in the $i^{th}$ energy level and particle $2$ is in the
$j^{th}$ energy level.

(i) $(\epsilon_{0},\epsilon_{1},\epsilon_{2})$ $S_{p}(i,j)$ $E_{s}$ $g_{s}$
$S_{1}(0,0)$ $E_{0}=2\epsilon_{0}$ $1$ $S_{2}(0,1),S_{3}(1,0)$
$E_{1}=\epsilon_{0}+\epsilon_{1}$ $2$ $S_{4}(0,2),S_{5}(2,0)$
$E_{2}=\epsilon_{0}+\epsilon_{2}$ $2$ $S_{6}(1,1)$ $E_{3}=2\epsilon_{1}$ $1$
$S_{7}(1,2),S_{8}(2,1)$ $E_{4}=\epsilon_{1}+\epsilon_{2}$ $2$ $S_{9}(2,2)$
$E_{5}=2\epsilon_{2}$ $1$ (ii) $(0,1,2)$ $E_{s}$ $g_{s}$ $0$ $1$ $1$ $2$ $2$
$3$ $3$ $2$ $4$ $1$

Table 1: Distinguishable Particles : $S_{p}(i,j)$ observable states for
2-particle 3-level system, $E_{s}$ energy and $g_{s}$ degeneracy for levels
(i) ($\epsilon_{0}$, $\epsilon_{1}$, $\epsilon_{2}$) & (ii) in particular
($\epsilon_{0}=0$, $\epsilon_{1}=1$, $\epsilon_{2}=2$)

The Table 1 shows $9$ possible states of the system ($p=1,2...9$). States with
label $2$ and $3$ i.e. $S_{2}(0,1)$ and $S_{3}(1,0)$ in which the particles
just exchange their energy levels will be degenerate. Thus energy with label
$1$ i.e. $E_{1}$ has degeneracy $g_{1}=2$. States with same energy are listed
together and thus there are only $6$ distinct energy $E_{s}$ of the system
(label $s$ is $0$ to $5$) with degeneracy $g_{s}$. Table 1 also lists the
allowed energy states when single-particle energy levels $\epsilon_{0}=0$,
$\epsilon_{1}=1$ and $\epsilon_{2}=2$. Apart from the exchange degeneracy, the
choice of $\epsilon$’s also leads to additional degeneracy in $E_{2}$ and
$E_{3}$.

(i) $(\epsilon_{0},\epsilon_{1},\epsilon_{2})$ $S_{p}(i,j)$ $E_{s}$ $g_{s}$
$S_{1}(0,0)$ $E_{0}=2\epsilon_{0}$ $1$ $S_{2}(0,1)=S_{3}(1,0)$
$E_{1}=\epsilon_{0}+\epsilon_{1}$ $1$ $S_{4}(0,2)=S_{5}(2,0)$
$E_{2}=\epsilon_{0}+\epsilon_{2}$ $1$ $S_{6}(1,1)$ $E_{3}=2\epsilon_{1}$ $1$
$S_{7}(1,2)=S_{8}(2,1)$ $E_{4}=\epsilon_{1}+\epsilon_{2}$ $1$ $S_{9}(2,2)$
$E_{5}=2\epsilon_{2}$ $1$ (ii) $(0,1,2)$ $E_{s}$ $g_{s}$ $0$ $1$ $1$ $1$ $2$
$2$ $3$ $1$ $4$ $1$

Table 2:  Same as Table 1 but for Bosons

(i) $(\epsilon_{0},\epsilon_{1},\epsilon_{2})$ FD
$(\epsilon_{1},\epsilon_{2})$ $E_{s}$ $g_{s}$ $S_{1}(0,0)$ $\times$ $\times$
$S_{2}(0,1)=S_{3}(1,0)$ $E_{1}=\epsilon_{0}+\epsilon_{1}$ $1$
$S_{4}(0,2)=S_{5}(2,0)$ $E_{2}=\epsilon_{0}+\epsilon_{2}$ $1$ $S_{6}(1,1)$
$\times$ $\times$ $S_{7}(1,2)=S_{8}(2,1)$ $E_{4}=\epsilon_{1}+\epsilon_{2}$
$1$ $S_{9}(2,2)$ $\times$ $\times$ (ii) $(0,1,2)$ $E_{s}$ $g_{s}$ $\times$
$\times$ $1$ $1$ $2$ $1$ $\times$ $\times$ $3$ $1$ $\times$ $\times$

Table 3:  Same as Table 1 but for Fermions

The Table 3 lists states available for the same system but with
indistinguishable Bosonic particles along with energy eigen-states $E_{s}$ and
their degeneracies $g_{s}$. The symbols have the same meaning as in Table 1.
Since the Bosons are indistinguishable, states $S_{2}(0,1)$ and $S_{3}(1,0)$
are now the same state with just two different labels. This has to be counted
only once as exchange of particle for indistinguishable particles is
meaningless. Thus the degeneracy $g_{s}$ for corresponding energy $E_{1}$ is
now counted as $g_{1}=1$ and not $2$. Thus all the exchange degeneracy are
removed. However, the choice of $\epsilon_{0}=0$, $\epsilon_{1}=1$ and
$\epsilon_{2}=2$ leads to a degeneracy in energy states $E_{2}$ and $E_{3}$.

The Table 3 lists states available for the same system but with
indistinguishable Fermions along with energy eigen-states $E_{s}$ and their
degeneracies $g_{s}$. Fermions are indistinguishable particles and have
additional restriction that no two particles occupy the same single-particle
state (Pauli Exclusion principle). Thus the exchange degeneracy are again
removed and the Pauli Exclusion principle disqualifies some states ($S_{1}$,
$S_{6}$ & $S_{9}$) as being observable. Degeneracy may arise due to choice of
$\epsilon$ for a larger system.

The PF $Q$, $B$ and $F$ for the three systems are then evaluated. The PF for
the Gibbs gas is also evaluated as $Q^{*}=Q/N!$. The program is provided in
the Appendix A.

##  4 Results

###  4.1 Partition Function

The PFs for $2$-particle $3$-level distinguishable system $Q_{N}$, Gibbs gas
$Q^{*}_{N}$, Boson $B_{N}$ and Fermion $F_{N}$ are plotted with respect to
dimensionless temperature $T/T_{0}$ where $T_{0}=\epsilon_{0}/k_{B}$ in Figure
2.

Figure 1: $2$-particle $3$-level Partition Function

Figure 2: $3$-particle $10$-level Partition Function

At high $T/T_{0}$ values (i) $Q\rightarrow 9$ for distinguishable particles,
(ii) $B\rightarrow 6$ for Bosonic particles, (iii) $F\rightarrow 3$ for
Fermionic system and (iv) $Q^{*}\rightarrow 4.5$ for Gibbs gas. They reflect
the available energy states as can be verified from the respective Tables 1, 3
and 3. Further, for all temperatures the partition function for Gibbs gas
follows the inequality $B_{N}>Q^{*}_{N}>F_{N}$ [5, 6].

In figures $2-6$ we consider a modest $3$-particles and $10$-level system
which has $10^{3}$ accessible energy states. Computationally, we need finite
particles and levels to study the behaviour as we scale up the system. The
$1000$ available states are adequate to show that the $1/N!$ term is justified
as we choose larger number of particles.

The PF for $3$-particle $10$-level distinguishable system, Gibbs gas, Boson
and Fermion are shown in Figure 2. Though the system is still small, the
factor of $1/N!$ used to scale down the distinguishable system PF $Q$ to that
for a Gibbs gas $Q^{*}$ seems to predict values in-between those of $B$ and
$F$.

###  4.2 State Functions

For the $3$-particle $10$-level system (with $\epsilon_{0}=1$,
$\epsilon_{1}=2$, $\dots$ and $\epsilon_{9}=10$) the non-dimensional state
functions $H/\epsilon_{0}$, $S$, $E/\epsilon_{0}$ and $C_{v}/\epsilon_{0}$ per
particle are presented with $T/T_{0}$ in Figures 4, 4, 6 and 6 respectively.

Figure 3: The Helmholtz Free Energy

Figure 4: The Entropy

Figure 5: The Energy

Figure 6: The Specific heat

Though the system is still small, the factor of $1/N!$ used to scale down the
distinguishable system partition function $Q$ to that for a Gibbs gas $Q^{*}$
seems to predict values of the state functions close to those for $B$ and $F$.
We see that the logarithmic dependence of the state function on the partition
function bring the Gibbs gas closer to Boson and Femion gas [2].

##  5 Conclusions

We find that (a) the PFs $Q_{N}^{*}$, $B_{N}$ and $F_{N}$ are different and
(b) by taking large number of particles with even larger number of energy
states, the state functions for Bosons and Fermions approach those for the
classical case because the state functions have logarithmic dependence on the
PF.

The advantage of this approach is that (i) students by ignoring the single-
particle PF understand states available to the system itself (ii) the PF
correctly counts the available states and provides an interpretation to the
PF. (iii) Even though the PFs $Q_{N}$, $B_{N}$ and $F_{N}$ differ the state
functions can be described by using the $Q^{*}_{N}=1/N!\times Q_{N}$

## References

  * [1] A J Berlinsky and A B Harris, Statistical Mechanics: An Introductory Graduate Course, Springer Nature, Switzerland, 2019
  * [2] L D Landau and E M Lifshitz, Statistical Physics, Addison-Wesley, Reading, MA, 1958
  * [3] H A Buchdahl, Remark on the Factor $1/N!$ in the Partition Function, Am. J. Phys. $42$ : 51-53, 1974
  * [4] D I Ford, A note on the partition function for systems of Independent Particles, Am. J. Phys. $39$ : 215-220, 1971
  * [5] H Kroemer, How incorrect is the classical partition function for the ideal gas?, Am. J. Phys. $48$ : 962, 1980
  * [6] R. Baierlein, The fraction of "all different" combinations: Justyfying the semi-classical partition function, Am. J. Phys. $65(4)$ : 314-316, 1997
  * [7] R K Pathria and P D Beale, Statistical Mechanics, Elsevier, United States, 2011
  * [8] D A McQuarrie, Statistical Mechanics, Harper and Row, New York, 1976
  * [9] C Kittle and H Kroemer, Thermal Physics, Freeman, San Francisco, 1980


# EulerNet: Adaptive Feature Interaction Learning via
Euler’s Formula for CTR Prediction

Zhen Tian† Gaoling School of Artificial Intelligence, Renmin University of
ChinaBeijingChina<EMAIL_ADDRESS>,  Ting Bai∗‡ Beijing University of
Posts and TelecommunicationsBeijingChina<EMAIL_ADDRESS>,  Wayne Xin
Zhao∗† Gaoling School of Artificial Intelligence, Renmin University of
ChinaBeijingChina<EMAIL_ADDRESS>,  Ji-Rong Wen† Gaoling School of
Artificial Intelligence, Renmin University of ChinaBeijingChina
<EMAIL_ADDRESS>and  Zhao Cao  Poisson Lab, HuaweiBeijingChina
<EMAIL_ADDRESS>
(2023)

###### Abstract.

Learning effective high-order feature interactions is very crucial in the CTR
prediction task. However, it is very time-consuming to calculate high-order
feature interactions with massive features in online e-commerce platforms.
Most existing methods manually design a maximal order and further filter out
the useless interactions from them. Although they reduce the high
computational costs caused by the exponential growth of high-order feature
combinations, they still suffer from the degradation of model capability due
to the suboptimal learning of the restricted feature orders. The solution to
maintain the model capability and meanwhile keep it efficient is a technical
challenge, which has not been adequately addressed. To address this issue, we
propose an adaptive feature interaction learning model, named as EulerNet, in
which the feature interactions are learned in a complex vector space by
conducting space mapping according to Euler’s formula. EulerNet converts the
exponential powers of feature interactions into simple linear combinations of
the modulus and phase of the complex features, making it possible to
adaptively learn the high-order feature interactions in an efficient way.
Furthermore, EulerNet incorporates the implicit and explicit feature
interactions into a unified architecture, which achieves the mutual
enhancement and largely boosts the model capabilities. Such a network can be
fully learned from data, with no need of pre-designed form or order for
feature interactions. Extensive experiments conducted on three public datasets
have demonstrated the effectiveness and efficiency of our approach. Our code
is available at: https://github.com/RUCAIBox/EulerNet.

Feature Interaction, CTR Prediction, Recommender Systems, Neural Networks

∗ Ting Bai<EMAIL_ADDRESS>and Wayne Xin Zhao<EMAIL_ADDRESS>are
the corresponding authors.

† Also with Beijing Key Laboratory of Big Data Management and Analysis Methods

‡ Also with Beijing Key Laboratory of Intelligent Telecommunications Software
and Multimedia

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
46th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 23–27, 2023; Taipei, Taiwan††booktitle:
Proceedings of the 46th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’23), July 23–27, 2023, Taipei,
Taiwan††ccs: Information systems Recommender systems††ccs: Computing
methodologies Neural networks

##  1\. Introduction

Click-Through Rate (CTR) prediction, which aims to predict the probability of
a user clicking on an item, is a very critical task in online e-commerce
platforms. In the literature, various approaches have been proposed for
effective CTR prediction (Cheng et al., 2016; He and Chua, 2017; Li et al.,
2020; Shan et al., 2016). The key of CTR prediction is to accurately model the
complicated context data by capturing underlying feature relationships.
Typically, these methods either learn _explicit feature interaction_ by
manually setting the interaction form/order via factorization based models
(Pan et al., 2018; Sun et al., 2021), or _implicit feature interaction_ by
directly modeling the fusion of all the features via deep neural networks
(Cheng et al., 2016; Zhang et al., 2016).

Despite the progress, these methods still have limitations in learning
complicated feature relationships (_e.g.,_ high-dimensional varied contexts).
Firstly, due to an exponential growth of combinational complexity, explicit
learning methods usually set a small interaction order, which cannot scale to
the cases requiring high-order feature interaction modeling. Further, they
only model the integer-order interactions, thus leading to an inaccurate
modeling of real-world scenarios. Secondly, due to the lack of effective
design in interaction mechanisms, implicit learning methods are shown to be
less effective than explicit learning methods (Rendle et al., 2020).

A major challenge in modeling high-order interactions among raw features is
the incurred high computational cost due to the exponential feature
combinations as the number of raw features increases. In a practical scenario,
raw features tend to be very sparse and have hundreds of fields with millions
of dimensions. For example, identifier features like user ID or item ID become
very sparse when encoded as one-hot vectors, so are the multi-field features
extracted from the user behavior logs. Calculating high-order interactions on
such sparse features with hundreds of fields is computationally intensive and
time-consuming.

Considering the above limitations, several studies (Lian et al., 2018; Tian et
al., 2023; Wang et al., 2021) manually assign a maximal order, and further
remove useless interactions from them. However, they still suffer from the
degradation of model capability due to the restricted feature orders. As a
promising approach, a recent study AFN (Cheng et al., 2020) leverages
logarithmic neural network (LNN (Hines, 1996)) to adaptively learn the order
of feature interactions. It can automatically learn the orders of feature
interactions, but at the expense of limited feature representation space,
_i.e.,_ only positive feature embeddings can be learned in logarithmic space
transformation, which requires a large consumption of logarithmic neurons for
retaining the performance.

To address these issues, in this paper, we propose an adaptive feature
interaction learning model, named as EulerNet, for automatically learning
arbitrary-order feature interactions. Unlike prior work, the core idea of
EulerNet is to model the feature interaction in a _complex vector space_ by
conducting space mapping according to Euler’s formula. Specially, EulerNet
converts the exponential powers of feature interactions into simple linear
combinations of the modulus and phase of the complex features, making it
feasible to capture complicated feature interactions in an _efficient,
flexible_ way. Based on such an idea, we develop an Euler interaction layer
that performs the above transformation, which can be stacked to form a capable
interaction learning network. Such a network can be fully learned from data,
with no need of pre-designed form or order for feature interactions.
Furthermore, Euler interaction layer can be extended to integrate the implicit
feature interactions. Different from previous explicit-implicit hybrid
approaches, our model can fuse the feature representations from the two ways
in the Euler interaction layer, instead of simply keeping two separate feature
interaction models.

The contributions are summarized as follows:

$\bullet$ We propose an adaptive feature interaction learning model EulerNet.
It can automatically learn the arbitrary-order feature interactions from data.
Meanwhile, our model can jointly capture the explicit and implicit feature
interactions in a unified model architecture.

$\bullet$ We propose to model the feature interaction in the complex vector
space, by conducting space mapping according to Euler’s formula. It enables
EulerNet to convert the complicated exponential powers into simple linear
computation.

$\bullet$ We conduct extensive experiments on three widely used datasets.
EulerNet consistently outperforms a number of competitive baselines with much
fewer parameters, showing the effectiveness and efficiency of our model.

##  2\. Preliminary

We first introduce the CTR prediction task, then present the formulations for
explicit and implicit feature interactions in existing work, and finally
introduce the Euler’s formula used in our model.

CTR Prediction. The task of the click-through rate (CTR) prediction aims to
estimate the probability that a user will click on an item. It takes as input
a vector of context features (_e.g.,_ user and item features), denoted as
$\bm{x}=\\{x_{1},x_{2},...,x_{m}\\}$, where $m$ is the number of feature
fields and $x_{j}$ is the $j$-th feature, the label $y\in\\{0,1\\}$ represents
whether the item is clicked or not and it is predicted from the input feature
$\bm{x}$. We further apply a look-up operation to each feature ${x}_{j}$ by
mapping it into a $d$-dimensional embedding $\bm{e}_{j}\in\mathbb{R}^{d}$. In
this way, the original feature vector can be represented as a list of feature
embeddings $\\{\bm{e}_{1},\bm{e}_{2},..,\bm{e}_{m}\\}$.

Explicit Feature Interactions. The key of CTR prediction is to learn the
effective feature interactions, which is a fundamental problem for this task
(Zhang et al., 2021). According to the interaction forms, existing methods can
be roughly divided into _explicit_ and _implicit_ feature interactions.
Explicit feature interactions is usually modeled by a pre-designed interaction
formula with a controllable order, such as FM (Rendle, 2010), HOFM (Blondel et
al., 2016) and IM (Yu et al., 2020). We introduce a special symbol
$\Delta_{ex}$ to denote the explicit feature interaction, generally defined
as:

(1)
$\Delta_{ex}=\sum_{\alpha\in\mathcal{A}}\bm{e}_{1}^{\alpha_{1}}\odot\bm{e}_{2}^{\alpha_{2}}\odot\cdots\odot\bm{e}_{m}^{\alpha_{m}},$

where $\bm{\alpha}=[\alpha_{1},\alpha_{2},...,\alpha_{m}]$ consists of the
orders for each feature in $\bm{x}$, $\odot$ is the element-wise product.
Based on $\Delta_{ex}$, another prediction function $f(\cdot)$ (_i.e.,_
sigmoid function) can be employed to generate the predicted label $\hat{y}$ in
$[0,1]$. Here, $\mathcal{A}$ is the set of all planned interactions by a CTR
model. Most CTR models require the interaction orders to be non-negative
integers, _i.e.,_ $\alpha_{j}\in\mathbb{N}^{0}$. For example, FM (Rendle,
2010) only considers second-order interaction, which specify
$\mathcal{A}=\\{\bm{\alpha}|\sum_{j=1}^{m}\alpha_{j}=2,\forall\alpha_{j}\in\\{0,1\\}\\}$.
Different from most existing methods (Sun et al., 2021; Rendle, 2010; Pan et
al., 2018), we aim to learn the arbitrary-order feature interactions in an
adaptive learning way, _i.e.,_ $\bm{\alpha}$ could be arbitrary real values
that are automatically learned from data.

Implicit Feature Interactions. As another form of feature interaction,
implicit feature interactions are commonly modeled by feed-forward neural
networks, _e.g.,_ the multi-layer perceptron (MLP) used in xDeepFM (Lian et
al., 2018), DCNV2 (Wang et al., 2021) and DeepIM (Yu et al., 2020). Different
from explicit feature interactions, it does not specify the concrete
interaction forms in the model. Formally, given the concatenation of all
feature embeddings, _i.e.,_
${\bm{z}}^{(0)}=[\bm{e}_{1};\bm{e}_{2};...;\bm{e}_{m}]$, the implicit feature
interaction process $\Delta_{im}$ can be formulated as:

(2) $\displaystyle\Delta_{im}$ $\displaystyle=\bm{z}^{(L)},$ (3)
$\displaystyle\bm{z}^{(l)}$
$\displaystyle=\sigma(\bm{W}^{(l)}\bm{z}^{(l-1)}+\bm{b}^{(l)}),$

where $l\in[1,L]$, $L$ is the layer depth and $\sigma$ is the activation
function.

Euler’s Formula. Euler’s formula is a mathematical formula that establishes
the relationships between different expressions of complex vectors, and can be
formulated as:

(4)
$\displaystyle\bm{\lambda}e^{i\bm{\theta}}=\bm{\lambda}\cos{\bm{\theta}}+i(\bm{\lambda}\sin{\bm{\theta}}),$

where $\bm{\lambda}e^{i\bm{\theta}}$ and
$\bm{\lambda}\cos{\bm{\theta}}+i(\bm{\lambda}\sin{\bm{\theta}})$ are the
representations of a complex vector in the polar form and the rectangular form
respectively. Here, $i$ is the imaginary unit, $\bm{\lambda}$ and
$\bm{\theta}$ are the modulus and phase of a complex vector. For a complex
vector $\bm{r}+i\bm{p}$, we set the real part
$\bm{r}=\bm{\lambda}\cos{\bm{\theta}}$ and imaginary part
$\bm{p}=\bm{\lambda}\sin{\bm{\theta}}$. The modulus $\bm{\lambda}$ and phase
$\bm{\theta}$ can be represented as:

(5) $\displaystyle\begin{split}\bm{\lambda}&=\sqrt{\bm{r}^{2}+\bm{p}^{2}},\\\
\bm{\theta}&=\mathrm{atan2}(\bm{p},\bm{r}),\end{split}$

where $\mathrm{atan2}(\bm{y},\bm{x})$ is the two-argument arctangent function.
The transformation via Euler’s formula makes it feasible to convert the
complex vectors from the rectangular form to the polar form, providing a way
to encode the features in the polar space.

##  3\. Methodology

To adaptively learn the arbitrary-order feature interactions, we propose a
feature interaction learning model via Euler’s formula, named as EulerNet. We
first present a general introduction of our model, and then introduce the
technical details in each part.

Figure 1. The overall architecture of EulerNet.

###  3.1. Overview of EulerNet

The overview architecture of EulerNet is shown in Figure 1. EulerNet is
designed by stacking the key structure of _Euler interaction layer_. The core
idea of Euler interaction layer is to transform explicit interaction of
feature embeddings (Eq. (1)) in a _complex vector space_ according to Euler’s
formula (Eq. (4)). As such, we can model complicated feature relationships in
a flexible way, without the constraints in existing work (_e.g.,_ non-
negativity or integer). Further, exponential computation can be simplified as
linear computation, making it possible to adaptively learn the high-order
feature interactions in an efficient way. Further, Euler interaction layers
can be extended to incorporate implicit feature interaction learning, which
can naturally integrate the two kinds of feature interaction.

In what follows, we introduce the details of explicit feature interaction
(Section 3.2) and implicit feature interaction (Section 3.3).

###  3.2. Explicit Feature Interaction Learning

Previous works (Rendle, 2010; Guo et al., 2017; Wang et al., 2021; Lian et
al., 2018) mainly learn the feature interactions in the real vector space,
which limits the expressiveness of features, lacking the ability to adaptively
capture the arbitrary-order feature interactions. To address this issue, we
first map the input features from the real vector space to the complex vector
space, and then learn the explicit feature interactions in the complex vector
space.

####  3.2.1. Complex Vector Representation of Features

As discussed in Section 2, the original input vector $\bm{x}$ can be mapped
into a list of feature embeddings $\\{\bm{e}_{1},\bm{e}_{2},..,\bm{e}_{m}\\}$
via an embedding layer. Based on the embedding representations, we next
discuss how to map them into complex space and further conduct Euler
interaction.

Complex Space Mapping. To improve the expressiveness of features, we map the
feature embeddings from the _real vector space_ to the _complex vector space_.
Given a feature embedding, $\bm{e}_{j}$, in the complex vector space, we
utilize two real vectors $\bm{r}$ and $\bm{p}$ to represent the real and
imaginary parts of the complex vector respectively (_i.e.,_
$\tilde{\bm{e}}_{j}=\bm{r}_{j}+i\bm{p}_{j}$). To transform a feature embedding
into a complex vector, the key idea is to consider it as the phase and
incorporate a learnable parameter (or parameter vector) $\mu_{j}$ as the
modulus following Euler’s formula in Eq. (4):

(6)
$\displaystyle\bm{\lambda}\rightarrow\mu_{j},\text{~{}~{}~{}~{}~{}~{}~{}}\bm{\theta}\rightarrow\bm{e}_{j}.$

According to Eq. (4), we can obtain the corresponding complex representation
of $\bm{e}_{j}$ by introducing the modulus parameter $\mu_{j}$:

(7)
$\displaystyle\tilde{\bm{e}}_{j}=\underbrace{\mu_{j}\cos({\bm{e}}_{j})}_{\text{real}}+i\underbrace{\mu_{j}\sin({\bm{e}}_{j})}_{\text{imaginary}},$

where we have $\bm{r}_{j}=\mu_{j}\cos({\bm{e}}_{j})$ and
$\bm{p}_{j}=\mu_{j}\sin({\bm{e}}_{j})$. To enhance the field-specific
semantics, we let the feature embeddings corresponding to the same field share
the same modulus parameter. After complex space mapping, each feature is
represented by a complex vector $\tilde{\bm{e}}_{j}$. We utilize the complex
feature representations
$\\{\tilde{\bm{e}}_{j}\\}_{j=1}^{m}=\\{\bm{r}_{j}+i\bm{p}_{j}\\}_{j=1}^{m}$
for subsequent interaction modeling.

####  3.2.2. Euler Interaction Layer

Euler interaction layer is the core component of our proposed EulerNet, which
enables the adaptive learning of explicit feature interactions. An Euler
interaction layer performs the feature interaction under the complex space one
time, taking as input a complex representation and outputting a transformed
complex representation. In this way, we can stack multiple Euler interaction
layers for enhancing the model capacity. Next, we describe the transformation
process with an Euler interaction layer.

Euler Transformation. In order to adaptively learn the explicit feature
interactions, we utilize Euler Transformation to transform the complex feature
representations from the rectangular form to the polar form. This step can
convert exponential multiplications into simplified linear computation, making
it feasible to adaptive capture complicated feature interactions. Given the
input complex representation $\bm{r}_{j}+i\bm{p}_{j}$ of feature embedding
${\bm{e}}_{j}$, we use Euler’s formula in Eq. (4) to obtain the polar-form
representations:

(8) $\bm{r}_{j}+i\bm{p}_{j}\rightarrow\bm{\lambda}_{j}e^{i\bm{\theta}_{j}}.$

In this form, the explicit feature interaction can be formulated as:

(9)
$\displaystyle\begin{split}\Delta_{ex}&=\tilde{\bm{e}}_{1}^{\alpha_{1}}\odot\tilde{\bm{e}}_{2}^{\alpha_{2}}\odot..\odot\tilde{\bm{e}}_{m}^{\alpha_{m}}\\\
&=\prod_{j=1}^{m}\bigg{(}\bm{\lambda}_{j}^{\alpha_{j}}\exp{(i\alpha_{j}\bm{\theta}_{j})}\bigg{)}\\\
&=\exp\bigg{(}\sum_{j=1}^{m}\alpha_{j}\log(\bm{\lambda}_{j})\bigg{)}\exp{(i\sum_{j=1}^{m}\alpha_{j}\bm{\theta}_{j})},\end{split}$

where $\bm{\lambda}_{j}=\sqrt{\bm{r}_{j}^{2}+\bm{p}_{j}^{2}}$ (always non-
negative) and $\bm{\theta}_{j}=\mathrm{atan2}(\bm{p}_{j},\bm{r}_{j})$ are the
modulus and phase vectors of the complex features in the polar form. In this
way, explicit feature interaction has been cast into a linear weighted
combination of modulus and phase values in the polar space, and the original
interaction order (_i.e.,_ $\bm{\alpha}$) becomes the combination
coefficients.

Note that, to achieve the similar formulation, we can also perform the log
operation on the original feature interaction (Eq. (1)), while it requires the
feature embeddings to be _non-negative_ , which do not always hold for all the
cases. Such a transform provides a possibility to model complicated feature
interaction in a more simplified way.

Generalized Multi-order Transformation. In the above, we have discussed the
case with an order vector $\bm{\alpha}$. In this part, we generalize such a
transformation into a group of $n$ order vectors
$\\{\bm{\alpha}_{k}\\}_{k=1}^{n}$, where $\alpha_{k,j}$ denotes the $j$-th
order of the $k$-th vector $\bm{\alpha}_{k}$. Formally, we introduce the
$\bm{\psi}_{k}$ and $\bm{l}_{k}$ to generalize Eq. (9) as follows:

(10)
$\displaystyle\begin{split}\bm{\psi}_{k}&=\sum_{j=1}^{m}\alpha_{k,j}\bm{\theta}_{j}+\bm{\delta}_{k},\\\
\bm{l}_{k}&=\exp\bigg{(}\sum_{j=1}^{m}\alpha_{k,j}\log(\bm{\lambda}_{j})+\bm{\delta}^{\prime}_{k}\bigg{)},\end{split}$

where $\bm{\delta}_{k}$ and $\bm{\delta}^{\prime}_{k}$ are learnable bias
vectors that are incorporated for enhancing the representations. With this
generalized extension, we can obtain the explicit interaction with
$\bm{\alpha}_{k}$ in the polar form:

(11)
$\displaystyle\textstyle\begin{split}\Delta_{ex}=&\exp\bigg{(}\sum_{j=1}^{m}\alpha_{k,j}\log(\bm{\lambda}_{j})+\bm{\delta}^{\prime}_{k}\bigg{)}\exp{\bigg{(}i(\sum_{j=1}^{m}\alpha_{k,j}\bm{\theta}_{j}+\bm{\delta}_{k})\bigg{)}}\\\
=&\bm{l}_{k}e^{i\bm{\psi}_{k}}.\end{split}$

Inverse Euler Transformation. Since the above feature interactions are in the
polar form, we do not directly perform the corresponding interactions with a
group of multi-order coefficients $\\{\bm{\alpha}_{k}\\}_{k=1}^{n}$. We
further utilize _inverse Euler transformation_ to convert them into the
original complex vectors in the rectangular form as:

(12)
$\displaystyle\begin{split}\hat{\bm{r}}_{k}&=\bm{l}_{k}\cos(\bm{\psi}_{k}),\\\
\hat{\bm{p}}_{k}&=\bm{l}_{k}\sin(\bm{\psi}_{k}),\end{split}$

where $\hat{\bm{r}}_{k}$ and $\hat{\bm{p}}_{k}$ are the real and imaginary
vectors. In this way, the Euler interaction layer can model $n$ explicit
feature interactions. The generalized explicit feature interactions with a
group of multi-order coefficients $\\{\bm{\alpha}_{k}\\}_{k=1}^{n}$ learned in
Euler interaction layer can be described as:

(13)
$\displaystyle\begin{split}\Delta_{ex}&=\sum_{k=1}^{n}\bm{l}_{k}\cos(\bm{\psi}_{k})+i(\bm{l}_{k}\sin(\bm{\psi}_{k}))\\\
&=\sum_{k=1}^{n}(\hat{\bm{r}}_{k}+i\hat{\bm{p}}_{k}).\end{split}$

This formula is the core of the proposed EulerNet model for explicit feature
interactions. Unlike prior work, the order of the interactions (_i.e.,_
$\alpha_{k,j}$) can be set to arbitrary real value, without additional limits
such as _non-negativity_. Instead of manually setting the order coefficients,
we adaptively learn them from data, and use the number of order vectors $n$ to
control the model complexity. Furthermore, we can also set varying $n$ at
different layers to increase the model flexibility.

###  3.3. Integrating Implicit Interactions

Considering that feature relationship in the real scenarios is very
complicated, we further incorporate implicit feature interactions into our
model. Different from previous studies (Guo et al., 2017; Wang et al., 2021;
Lian et al., 2018), which model the explicit and implicit feature interactions
in different architectures, we integrate them in each Euler interaction layer
to enhanced the representation capacity.

####  3.3.1. Fusing Explicit and Implicit Interactions

To model more complicated feature relationship, we construct a neural network
component for capturing implicit feature interactions. Given the input complex
features
$\\{\tilde{\bm{e}}_{j}\\}_{j=1}^{m}=\\{\bm{r}_{j}+i\bm{p}_{j}\\}_{j=1}^{m}$,
we can obtain the input of the implicit interaction by concatenating the these
vectors as: $\bm{r}=[\bm{r}_{1};\bm{r}_{2},...;\bm{r}_{m}]$ (_real part_) and
$\bm{p}=[\bm{p}_{1};\bm{p}_{2},...;\bm{p}_{m}]$ (_imaginary part_). Then, we
feed the real and imaginary parts of feature representations into the same
linear layer with a subsequent non-linear activation function:

(14)
$\displaystyle\begin{split}\bm{r}_{k}^{\prime}&=\mathrm{ReLU}(\bm{W}_{k}\bm{r}+\bm{b}_{k}),\\\
\bm{p}_{k}^{\prime}&=\mathrm{ReLU}(\bm{W}_{k}\bm{p}+\bm{b}_{k}),\end{split}$

where $k\in\\{1,\cdots n\\}$, and $\bm{W}_{k}\in\mathbb{R}^{d\times md}$ is
the weight matrix and $\bm{b}_{k}\in\mathbb{R}^{d}$ is the bias. Finally, in
order to integrate the two kinds of feature interaction, we add the explicit
and implicit representations (See Eq. (12) and Eq. (14)) by real and imaginary
parts accordingly as:

(15)
$\displaystyle\\{{\bm{o}}_{k}\\}_{k=1}^{n}=\\{(\hat{\bm{r}}_{k}+\bm{r}_{k}^{\prime})+i(\hat{\bm{p}}_{k}+\bm{p}_{k}^{\prime})\\}_{k=1}^{n}.$

We can stack multiple Euler interaction layers by taking the output features
of the previous layer as the input for the next layer and optionally applying
normalization methods such as BatchNorm (Ioffe and Szegedy, 2015) or LayerNorm
(Ba et al., 2016) to adjust the distribution as it passes through each layer.

####  3.3.2. Output for CTR Predictions

In order to predict the CTR value, we further perform linear regression on the
output representations
$\tilde{\bm{o}}=\\{\tilde{\bm{o}}_{k}\\}_{k=1}^{n}=\\{\tilde{\bm{r}}_{k}+i\tilde{\bm{p}}_{k}\\}_{k=1}^{n}$.
Specially, we concatenate the real and imaginary vectors accordingly, and
introduce a regression weight vector $\bm{w}\in\mathbb{R}^{nd}$, so as to
obtain a scalar value for both the real and imaginary parts:

(16)
$z=\bm{w}^{\top}\tilde{\bm{r}}+i(\bm{w}^{\top}\tilde{\bm{p}})=z_{re}+iz_{im},$

where $z_{re}$ and $z_{im}$ are the real and imaginary part of $z$
respectively. The prediction for CTR by integrating both explicit and implicit
interactions can be given as:

(17) $\hat{y}=\sigma(z_{re}+z_{im}).$

For training, we utilize the binary cross entropy loss with a regularization
term to train our model, which is formulated as:

(18)
$\mathcal{L}(\Theta)=-\frac{1}{N}\sum_{j=1}^{N}\bigg{(}y_{i}\log(\hat{y}_{j})+(1-y_{j})\log(1-\hat{y}_{j})\bigg{)}+\gamma||\Theta||^{2}_{2},$

where $y_{j}$ and $\hat{y}_{j}$ are the ground-truth label and predicted
result of $j$-th training sample respectively, and $\Theta$ denotes the set of
the parameters and $\gamma$ is the $L_{2}$-norm penalty.

###  3.4. Discussion

####  3.4.1. Intuitive Explanation of Feature Interaction

To have an intuitive understanding of our approach, we consider a simple case
when the embedding dimension $d=1$. Further, since we apply normalization at
the Euler interaction layer, the modulus is around 1, so that we can omit the
corresponding $\lambda$ from Eq. (9). The forms of explicit and implicit
interaction can be simplified as:

(19) $\displaystyle g_{ex}(\tilde{\bm{e}}_{j},\tilde{\bm{e}}_{k})$
$\displaystyle=\tilde{\bm{e}}_{j}^{\alpha_{j}}\odot\tilde{\bm{e}}_{k}^{\alpha_{k}}\approx\exp{\big{(}i(\alpha_{j}\bm{\theta}_{j}+\alpha_{k}\bm{\theta}_{k})}\big{)},$
(20) $\displaystyle g_{im}(\tilde{\bm{e}}_{j},\tilde{\bm{e}}_{k})$
$\displaystyle=\mathrm{ReLU}(W_{j}\bm{r}_{j}+W_{k}\bm{r}_{k})+i\mathrm{ReLU}(W_{j}\bm{p}_{j}+W_{k}\bm{p}_{k}).$

As we can see, explicit interaction $g_{ex}(\cdot)$ mainly affects the phase
of features (_i.e.,_ $\theta_{j}$ and $\theta_{k}$), which can be
approximately considered as the rotations in the complex vector space, while
implicit interaction $g_{im}(\cdot)$ performs a parallelogram-like
transformation in the complex vector space, which mainly affects the modulus
instead of the phase (due to the limits in first quadrant). By integrating
both implicit and explicit feature interactions, our approach can model the
effect in both _phase_ and _modulus_ , thus leading to an improved capacity
due to mutual enhancement. Figure 2 presents a geometric interpretation of
explicit and implicit feature interactions.

Figure 2. The visual understanding of the interactions in EulerNet.

To further understand the explicit interaction, we present an illustrative
example with a simple interaction
$\tilde{\bm{e}}_{1}^{0.33}\odot\tilde{\bm{e}}_{2}^{0.25}$ with two feature
vectors: $\tilde{\bm{e}}_{1}=[-8,1]^{\top}$ and
$\tilde{\bm{e}}_{2}=[-16,4]^{\top}$. With some mathematical computations, we
can get the following representation:
$\tilde{\bm{e}}_{1}^{0.33}\odot\tilde{\bm{e}}_{2}^{0.25}=\hat{\bm{r}}+i\hat{\bm{p}}=[-0.99,1.41]^{\top}+i[3.85,0]^{\top}$.

Table 1. Comparison of different CTR methods. “Unified” denotes the integrated
learning of explicit and implicit interactions, “UR” means “Unrestricted”
indicating the elements of feature embeddings could be arbitrary real value,
and “SV” means “Single-vector” indicating that each feature is represented by
only one embedding vector.

Methods | Feature Interaction | Embedding | Complexity
---|---|---|---
High-order | Adaptive | Unified | UR | SV
FFM (Juan et al., 2016) | ✗ | ✗ | ✗ | ✔ | ✗ | $O(m^{2}d)$
FmFM (Sun et al., 2021) | ✗ | ✗ | ✗ | ✔ | ✔ | $O(m^{2}d^{2})$
DCNV2 (Wang et al., 2021) | ✔ | ✗ | ✗ | ✔ | ✔ | $O(m^{2}d^{2}L+mHd+H^{2}T)$
AFN+ (Cheng et al., 2020) | ✔ | ✔ | ✗ | ✗ | ✗ | $O(mdK+KHd+mHd+H^{2}T)$
EulerNet (ours) | ✔ | ✔ | ✔ | ✔ | ✔ | $O(mnd^{2}+n^{2}d^{2}L)$

####  3.4.2. Novelty and differences.

In Table 1, we compare our approach with existing feature interaction methods.
To the best of our knowledge, it is the first attempt that adaptively captures
arbitrary-order feature interactions in the complex vector space. Although
AFN+ (Cheng et al., 2020) leverages the LNN (Hines, 1996) to learn arbitrary-
order feature interactions adaptively, it constrains the feature
representations to positive real vectors. This approach not only degrades the
model performance, but also requires additional feature embeddings for
implicit interactions. Furthermore, most studies (Wang et al., 2021; Lian et
al., 2018; Guo et al., 2017; Yu et al., 2020) model the explicit and implicit
interactions in different architectures and seldom integrate them in a joint
approach. As a comparison, EulerNet is more _general, unified_ in integrating
the modeling of implicit and explicit feature interactions, via the enhanced
Euler interaction layer in Section 3.3. In general, our approach provides a
more capable solution to model complicated feature interactions.

####  3.4.3. Complexity Analysis

We also compare the time complexities of different CTR methods in Table 1. For
ease of analysis, we assume that the hidden size of different components is
set to the same number. Specially, $m$ is the number of feature fields, $d$ is
the embedding dimension, $L$ and $T$ are the layer depth of the explicit and
implicit component respectively, $H$ is the hidden size of MLP, $K$ is the
number of logarithmic neurons of AFN+ (Cheng et al., 2020), and $n$ is the
number of order vectors of EulerNet. Note that $K$ is much larger than $m\cdot
d$, leading to a very high complexity of AFN+ (Cheng et al., 2020). In
contrast, $n$ is very small, which can be set to $m$ in practice. The
complexity of EulerNet for a training instance can be estimated as
$O(m^{2}d^{2}L)$, which is comparable to mainstream efficient methods such as
FmFM (Sun et al., 2021) and DCNV2 (Wang et al., 2021) (See Table 3 for
experimental analysis).

##  4\. EXPERIMENTS

We conduct extensive experiments to show the effectiveness of EulerNet, and
analyze the effect of each learning component in it.

###  4.1. Experimental Settings

We introduce the experimental settings, including the datasets, baseline
approaches, and the details of hyper-parameters.

####  4.1.1. Datasets

We utilize three real world datasets in our experiments:
Criteo111http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-
dataset, Avazu222http://www.kaggle.com/c/avazu-ctr-prediction,
MovieLens-1M333https://grouplens.org/datasets/movielens. Table 2 summarizes
the dataset statistics information.

$\bullet$ Criteo. The most popular CTR prediction benchmark dataset contains
user logs over a period of 7 days.

$\bullet$ Avazu. It contains user logs over a period of 7 days, which was used
in the Avazu CTR prediction competition.

$\bullet$ MovieLens-1M. The most popular dataset for recommendation systems
research.

Table 2. The statistics of datasets. Dataset |  $\\#$ Features |  $\\#$ Fields |  $\\#$ Instances
---|---|---|---
Criteo | 1.3M | 39 | 45M
Avazu | 1.5M | 23 | 40M
MovieLens-1M | 13k | 7 | 740K

####  4.1.2. Compared Models.

We compare EulerNet with state-of-the-art methods in CTR prediction task,
including:

$\bullet$ FwFM (Pan et al., 2018) improves FM by considering field information
and uses field-specific weights to capture the field-wise relationship.

$\bullet$ FmFM (Sun et al., 2021) replaces the field scalar weight in FwFM
with a kernel matrix, allowing for modeling more informative interactions.

$\bullet$ DeepFM (Sun et al., 2021) uses FM to model the second-order
interactions, and incorporates DNNs to model the high-order interactions.

$\bullet$ DeepIM (Yu et al., 2020) utilizes Newton’s identity to implement
high-order FM, and incorporate implicit interactions via an MLP.

$\bullet$ xDeepFM (Lian et al., 2018) encodes high-order interactions into
multiple feature maps and combine an MLP to model implicit interactions.

$\bullet$ DCNV2 (Wang et al., 2021) takes the kernel product of concatenated
feature vectors to model high-order interactions and combine an MLP to model
implicit interactions.

$\bullet$ FiBiNet (Huang et al., 2019) uses the bilinear operation to model
pair-wise interactions and uses SENet (Hu et al., 2018) to capture the feature
importance.

$\bullet$ AutoInt (Song et al., 2019) uses the self-attention mechanism to
learn high-order interactions. AutoInt+ improves it by combining an MLP.

$\bullet$ FiGNN (Li et al., 2019) represents the features into a full-
connected graph, and uses gated GNNs to model the high-order feature
interactions.

$\bullet$ AFN (Li et al., 2021) encodes features into a logarithmic space to
adaptively learn the arbitrary-order feature interactions. AFN+ improves the
base model by using an MLP to model implicit interactions.

The above models we compared in our experiments have covered different types
of feature interaction methods. FwFM and FmFM are shallow models that only
model the second-order explicit interactions. DeepFM, DeepIM, xDeepFM and
DCNV2 are ensemble methods that learn both the explicit interactions by an
empirically designed component and implicit interactions by an MLP. FiBiNet,
AutoInt, and FiGNN have the ability to learn the importance of feature
interactions. AFN encodes features into a logarithmic space to adaptively
learn the arbitrary-order feature interactions. Different from them, our
proposed EulerNet represents the features in a complex vector space, in which
the exponential computation can be simplified as linear computation, making it
possible to adaptively learn the arbitrary-order feature interactions in an
efficient way.

####  4.1.3. Implementation Details

All methods are implemented in Pytorch (Paszke et al., 2019). The size of
feature embedding is 16. The learning rate is in {1e-3, 1e-4, 1e-5}. The
$L_{2}$ penalty weight is in {1e-3, 1e-5, 1e-7}. The batch size is 1024. The
training optimizer is Adam (Kingma and Ba, 2014). The hidden layer of MLP
component is $400\times 400\times 400$ and the dropout rate is 0.1. For
DeepIM, the interaction order is in {2, 3, 4}. For xDeepFM, the depth of CIN
is in {1, 2, 3, 4, 5} and the hidden size is in {100, 200, 400}. For DCNV2,
the depth of CrossNet is in {1, 2, 3, 4, 5}. For FiGNN, the graph interaction
step is in {1, 2, 3, 4, 5}. For AutoInt+, the depth, number of head and
attention size is 2, 2, 40 respectively. For AFN, the number of logarithmic
neurons is in {40, 400, 800, 1000}. For EulerNet, the number of Euler
interaction layer is in {1, 2, 3, 4, 5}, and the number of order vectors is
set as {7, 23, 39} for MovieLens-1M, Avazu and Criteo datasets respectively.
Our implementation is also available at RecBole (Zhao et al., 2021, 2022).

###  4.2. Overall Performance

Table 3. Performance comparisons. A higher AUC or lower Logloss at 0.001-level is regarded significant, as stated in previous studies (Song et al., 2019; Cheng et al., 2016; Guo et al., 2017; Wang et al., 2017). Model | Criteo | Avazu | MovieLens-1M
---|---|---|---
AUC | LogLoss | Params | Latency | AUC | LogLoss | Params | Latency | AUC | LogLoss | Params | Latency
FwFM | 0.8104 | 0.4414 | 0.74 K | 6.71 ms | 0.7741 | 0.3835 | 0.25 K | 3.96 ms | 0.8815 | 0.3351 | 0.02 K | 1.12 ms
FmFM | 0.8112 | 0.4408 | 0.39 M | 8.06 ms | 0.7744 | 0.3831 | 0.13 M | 4.32 ms | 0.8864 | 0.3295 | 0.01 M | 1.26 ms
DeepFM | 0.8121 | 0.4401 | 0.57 M | 10.31 ms | 0.7830 | 0.3790 | 0.47 M | 6.94 ms | 0.8935 | 0.3230 | 0.36 M | 3.78 ms
DeepIM | 0.8124 | 0.4397 | 0.57 M | 10.86 ms | 0.7838 | 0.3779 | 0.47 M | 7.19 ms | 0.8927 | 0.3230 | 0.36 M | 4.20 ms
xDeepFM | 0.8122 | 0.4407 | 2.44 M | 188.68 ms | 0.7821 | 0.3799 | 2.29 M | 78.01 ms | 0.8944 | 0.3235 | 0.52 M | 44.68 ms
DCNV2 | 0.8127 | 0.4394 | 1.74 M | 16.39 ms | 0.7838 | 0.3782 | 0.87 M | 11.63 ms | 0.8946 | 0.3229 | 0.39 M | 4.03 ms
FiBiNet | 0.8126 | 0.4415 | 9.82 M | 136.79 ms | 0.7837 | 0.3783 | 3.57 M | 32.08 ms | 0.8860 | 0.3291 | 0.87 M | 8.13 ms
FiGNN | 0.8109 | 0.4412 | 0.08 M | 121.51 ms | 0.7830 | 0.3799 | 0.05 M | 46.31 ms | 0.8939 | 0.3232 | 0.01 M | 10.75 ms
AutoInt+ | 0.8126 | 0.4396 | 3.80 M | 41.67 ms | 0.7838 | 0.3785 | 1.43 M | 18.52 ms | 0.8937 | 0.3288 | 1.17 M | 9.61 ms
AFN+ | 0.8123 | 0.4396 | 19.46 M | 132.45 ms | 0.7843 | 0.3785 | 14.19 M | 93.72 ms | 0.8950 | 0.3212 | 5.65 M | 84.18 ms
EulerNet | 0.8137 | 0.4389 | 0.79 M | 13.51 ms | 0.7863 | 0.3769 | 0.27 M | 9.09 ms | 0.9008 | 0.3114 | 0.02 M | 2.69 ms

We present the experimental results of different methods for CTR prediction in
Table 3, and have the following observations:

(1) Compared to the DNN-based methods, FwFM and FmFM perform worst due to the
limited ability to only capture the second-order explicit feature
interactions.

(2) Ensemble methods (_i.e.,_ DeepFM (Guo et al., 2017), DeepIM (Zhu et al.,
2020), DCNV2 (Wang et al., 2021) and xDeepFM (Lian et al., 2018)) achieve
competitive performance across on all three datasets, which shows the
effectiveness of integrating implicit feature interactions.

(3) For the feature importance learning methods (_i.e.,_ FiBiNet (Huang et
al., 2019), FiGNN (Li et al., 2019) and AutoInt+ (Song et al., 2019)), their
performance largely varies across different datasets. AutoInt+ performs very
well on all three datasets, while FiGNN and FiBiNet lose the advantage on the
Criteo and MovieLens-1M datasets respectively. This indicates that the self-
attention mechanism is more capable in modeling high-order feature
interactions.

(4) AFN+ outperforms all the other baseline methods on the Avazu and
MovieLens-1M datasets, demonstrating the effectiveness of adaptively learning
the arbitrary-order feature interactions in the CTR prediction task. However,
its advantage on the Criteo dataset is small. This may be caused by the
restriction of positive values assigned in the feature embeddings, which
hinders the representation capability of the model.

(5) Our proposed EulerNet consistently performs better than all of the
compared methods. It shows the effectiveness of encoding the features into the
complex vectors via Euler’s formula and conducting transformations in the
polar space.

As for the model efficiency, we can see that the latency of FwFM, FmFM,
DeepFM, DeepIM and DCNV2 are relatively small. They are more efficient due to
the simple architecture and fewer parameters learned in the model. For
AutoInt, FiGNN, FiBiNet and xDeepFM, the latency of them is much larger due to
the complex model architecture or the complicated training strategy. For AFN+,
due to the limited feature representation space, _i.e.,_ only positive feature
embeddings can be learned in logarithmic space transformation, it requires a
large amount of parameters for retaining the performance. This makes it
impractical in the industrial scenarios. In contrast, the latency of EulerNet
is much less than AFN+ (_i.e.,_ under $10.2\%$) and it is comparable to many
efficient methods such as DeepFM and DeepIM. With the highest accuracy and
lower complexity, EulerNet has a great potential to be applied into large-
scale industrial recommender systems.

###  4.3. Experimental Analysis

We conduct experiments to investigate the interaction orders learned in
EulerNet, and then visualize the learned feature representations to show their
correlation with the feature importance.

####  4.3.1. Arbitrary-Order Learning Analysis

Learning effective high-order feature interactions is very crucial in the CTR
prediction task. To verify the orders learned in EulerNet, we visualize the
learned feature interaction orders (_i.e.,_ the total order of each learnable
coefficient vector $\bm{\alpha}_{k}$ in Eq. (10)) in the explicit feature
interaction component. Note that the orders adaptively learned in our model
can be arbitrary values in $[-\infty,+\infty]$, we cluster them by setting the
interval to 0.5 for better presentation. As shown in Figure 3, we can see that
our model not only learns the integer-order feature interactions, but also can
adaptively learn the fractional-order feature interactions in a fine-grained
way. Specially, the feature interaction orders learned in our model varies
from $[0,3.5]$ on MovieLens-1M dataset and $[0.5,3.5]$ on Avazu dataset. Fine-
grained feature interaction learning can improve the capability of our model
and enable it to capture more effective information for CTR prediction.

(a) MovieLens-1M (b) Avazu

Figure 3. The statistics of the interaction orders learned in EulerNet.

####  4.3.2. Verification on Synthetic Dataset

Since it can not identify the ground-truth of meaningful feature interactions
in real-world public datasets, we further conduct an experiment using
synthetic data to verify the degree of coincidence with the learned orders in
EulerNet. The synthetic dataset consists of 1 million synthesized click-
through records with 7 fields ($F=[f_{1},f_{2},...,f_{7}]$) that simulate real
click-through records. Each field is independently created and contains one
thousand features, and each feature is assigned a probability that affects the
likelihood of a click-through event occurring. For a given record
$x_{i}=[p_{1},p_{2},...,p_{7}]$, its label $y_{i}$ is generated by sampling
from a probability distribution that is pre-defined by one of the patterns,
_i.e.,_ $R\in[R_{1},R_{2},R_{3}]$ (See Table 4). We compare the interaction
orders learned in the explicit feature interaction component of AFN+ and
EulerNet, and utilize fitting deviation to evaluate the difference between the
orders in different learning algorithms and the ground-truth pattern $R$. From
Table 4, we can see that the deviation in EulerNet is much smaller than AFN+,
demonstrating that EulerNet has the ability to adaptively learn more
meaningful feature interactions.

Specifically, we present the order vectors of EulerNet after training on the
synthetic dataset defined by the pattern $R_{3}$ in Figure 4. Different rows
represent the explicit feature interactions learned by different order vectors
(See Eq. (10)). For example, the most important feature interactions learned
by the order vector $\bm{\alpha}_{1}$ is
$p_{1}^{1.12}p_{3}^{1.90}p_{5}^{1.48}$. We can see that the combinational
feature interactions learned by a group of multi-order vectors
$\\{\bm{\alpha}_{1},\bm{\alpha}_{2},\bm{\alpha}_{7}\\}$ (_i.e.,_
$p_{1}^{1.12}p_{3}^{1.90}p_{5}^{1.48}+p_{2}^{0.36}p_{4}^{0.37}p_{6}^{0.38}+p_{7}^{0.67}$)
are quite similar (average order deviation is 0.47) to the ground truth
interaction pattern (_i.e.,_
$R_{3}=\frac{1}{3}(p_{1}^{1.3}p_{3}^{2.2}p_{5}^{1.7}+p_{2}^{0.5}p_{4}^{0.5}p_{6}^{0.5}+p_{7})$)
in the data, showing the ability of EulerNet to learn the effective feature
interactions.

Table 4. The pattern for creating the synthetic dataset and the deviation comparisons between different models. Pattern | Formula | Deviation
---|---|---
AFN+ | EulerNet
$R_{1}$ | $p_{1}^{0.3}p_{3}^{1.7}p_{4}^{2.5}$ | 0.6296 | 0.1141
$R_{2}$ | $\frac{1}{2}(p_{1}^{1.7}p_{2}^{1.7}+p_{3}^{0.5}p_{4}^{2.5})$ | 2.4021 | 0.7481
$R_{3}$ | $\frac{1}{3}(p_{1}^{1.3}p_{3}^{2.2}p_{5}^{1.7}+p_{2}^{0.5}p_{4}^{0.5}p_{6}^{0.5}+p_{7})$ | 1.4732 | 0.4779
Figure 4. Visualization of the order vectors in EulerNet under the
distribution pattern $R_{3}$, which is defined in Table 4.

####  4.3.3. Visualization of Feature Embeddings

EulerNet not only can adaptively learn the arbitrary-order feature
interactions, but also have the ability to capture the importance of features.
We visualize the learned feature embeddings in EulerNet and show its ability
in learning the importance of features. The heat map in Figure 5(a)
illustrates the mutual information scores between feature fields and labels on
the MovieLens-1M dataset, which represents the strength of each field on the
prediction results. We can observe that the fields _item_id_ , _user_id_ and
_zip_code_ have the strongest effect on the click results. The distributions
of feature embeddings are plotted with Gaussian kernel density estimation
(KDE) in two-dimensional space in Figure 5(b). The more dispersive the
distribution of feature embeddings, the less influence of it has on the
prediction results due to the low information quantity in the random varies.
It can be seen that for the important fields (_i.e.,_ _item_id_ , _user_id_
and _zip_code_) in Figure 5(a), the distribution of feature embeddings in
Figure 5(b) is more concentrated and has a smaller variance. While for the
fields with less importance (_i.e.,_ _age_ , _occupation_ and _release_year_),
they are chaotically distributed, and their variance is relatively large. This
indicates that the feature embeddings, which also represent the phase of the
complex features as defined in Eq. (6), can reflect the feature importance to
a certain extent. In EulerNet, the phase of the complex features is
effectively controlled by explicit feature interactions (See Section 3.4.1),
which enables it to capture the meaningful feature relationship and improves
the model capabilities.

(a) Mutual information score between fields and label in MovieLens-1M.

(b) user_id

(c) age

(d) occupation

(e) zip_code

(f) item_id

(g) release_year

(h) The distribution of feature embeddings in different feature fields. Figure
5. Visualization of the relationships between feature importance and the
distribution of feature embeddings in EulerNet.

###  4.4. Ablation Study

We conduct ablation studies to explore the impact of each component or hyper-
parameter on the model performance.

####  4.4.1. Effect of Implicit and Explicit Feature Interactions

EulerNet contains both the explicit and implicit interaction learning
components. In order to investigate the impact of each interaction type, we
conduct experiments on the two variants of EulerNet, termed as EulerNetE and
EulerNetI, in which the implicit and explicit learning parts are removed
respectively. As shown in Table 5, we can see that the model performance has a
decrease for both EulerNetI and EulerNetE, showing the mutual complementary
effects of them, which is consistent with the observations in Section 3.4.1.
Besides, EulerNetI shows a larger decrease in performance than EulerNetE on
the Avazu and MovieLens-1M datasets, but the decrease is smaller on the Criteo
dataset, showing both the implicit and explicit interactions are important for
CTR prediction.

Table 5. Performance comparison between different interactions. Dataset | Metric | EulerNet |  EulerNetE |  EulerNetI
---|---|---|---|---
Criteo | AUC | 0.8137 | 0.8117 | 0.8124
Decrease | - | $-0.25\%$ | $-0.16\%$
Avazu | AUC | 0.7863 | 0.7847 | 0.7840
Decrease | - | $-0.20\%$ | $-0.29\%$
MovieLens-1M | AUC | 0.9008 | 0.8988 | 0.8966
Decrease | - | $-0.22\%$ | $-0.47\%$

####  4.4.2. Impact of the Interaction Layer Number

EulerNet is designed by stacking the key structure of the Euler interaction
layer. We study the impact of the Euler interaction layer number, which
reflects the intricacy of feature interactions, on the model performance. As
shown in Figure 6, we can observe that the performance of EulerNet increases
as the number of layers increases. EulerNet achieves the best model
performance with 5 interaction layers. When the number of layers exceeds 5,
the model performance decreases due to the overfitting issue caused by
incorporating more parameters.

####  4.4.3. Impact of the Number of Order Vectors.

As introduced in Section 3.2.2, we use multiple order vectors to adaptively
learn the arbitrary-order feature interactions. The number of order vectors is
denoted as $n$ (See Eq. (13)), which controls the number of explicit feature
interactions in each Euler interaction layer. As illustrated in Figure 7, the
performance of EulerNet on the Avazu dataset increases as the number of order
vectors increases from 20 to 60. Whereas on the Criteo dataset, EulerNet
achieves the best performance as the number of order vectors increases to 40.
However, the model performance decreases when adding more order vectors. This
indicates that including too many feature combinations in the multi-order
transformation may incorporate the useless feature interactions that hurt the
model performance.

(a) AUC (b) LogLoss

Figure 6. Impact of the interaction layer number.

(a) AUC (b) LogLoss

Figure 7. Impact of the number of order vectors.

##  5\. Related work

Explicit Feature Interaction Learning. This line of research explicitly
enumerates feature combinations and uses vector operations such as inner
product to capture their relationships. Early CTR models (Richardson et al.,
2007; Chang et al., 2010; He et al., 2014; Chen and Guestrin, 2016) mainly
relied on manually designing feature combinations with simple architectures.
For example, FM (Rendle, 2010) assigns an embedding vector to each feature
that mainly captures second-order interactions. Inspired by FM, many variants
of factorization machines have been proposed (Cheng et al., 2014; Juan et al.,
2016; Hong et al., 2019; Xu and Wu, 2020; Lu et al., 2021). Among them, FFM
(Juan et al., 2016) assigns multiple embeddings to explicitly model field-wise
feature interactions. Besides, FwFM (Pan et al., 2018) and FmFM (Sun et al.,
2021) are proposed to model the field information to improve FM in a
parameter-efficient way. These factorization based methods mainly model
second-order interactions, which severely limits their performance. To capture
more effective feature interactions, xDeepFM (Lian et al., 2018) proposes the
CIN to model the high-order feature interactions by incorporating lots of
learnable parameters. Besides, DCNV2 (Wang et al., 2021) proposes the CrossNet
to capture the high-order feature interactions in an efficient way. Although
these methods leverage high-order feature interactions to achieve great
performance, their interaction components are empirically predefined, which
may lead to the suboptimal learning of restricted feature interactions. As a
promising approach, AFN (Cheng et al., 2020) uses logarithmic neural networks
(LNN) (Hines, 1996) to adaptively model the arbitrary-order interactions, but
at the expense of restricting feature embeddings to positive real vectors,
which may degrade the expressiveness of feature representations and require
much more parameters to retain the performance. Different from them, our
proposed EulerNet models the feature interactions in a complex vector space by
conducting the space mapping via Euler’s formula. The feature interactions in
our model are adaptively learned from data without additional restrictions,
which could largely improve its capacity and better balance the effectiveness
and efficiency.

Implicit Feature Interaction Learning. In recent years, many deep learning
based models (Huang et al., 2013; Zhang et al., 2016; Guo et al., 2017; Cheng
et al., 2016; Naumov et al., 2019; Chen et al., 2019) have been proposed to
model the high-order feature interactions via a deep neural network (DNN)
component. Among them, the Wide & Deep (Cheng et al., 2016) network combines
the logit value of a linear regression model with the output of a DNN.
Besides, PNN (Qu et al., 2016) introduces an MLP to improve the output of its
explicit interaction component, and NFM (He and Chua, 2017) stacks deep neural
networks after FMs to model the high-order feature interactions. Different
from the explicit feature interactions, the implicit feature interactions
modeled by deep neural networks lack good interpretability. Additionally, some
recent study (Rendle et al., 2020) has found that it is more challenging for
an MLP to effectively learn the high-order feature interactions compared to
using an inner product in FM. Most deep learning based methods (Guo et al.,
2017; Lian et al., 2018; Pande, 2020; Yu et al., 2020) leverage the implicit
feature interactions as the supplemental signal of the explicit feature
interaction component. Different from them, in EulerNet, the explicit and
implicit feature interactions are learned in a unified architecture: both of
them perform the linear transformations on the features in different forms
(_i.e.,_ the polar form for the explicit feature interactions and the
rectangular form for the implicit feature interactions). Euler’s formula
establishes the relationship between different representation forms and also
builds a bridge between the explicit and implicit feature interactions. It is
observed that there exists a complementary effect between the explicit and
implicit interactions in EulerNet, which enables them to promote each other
and further improve the model capabilities.

##  6\. Conclusion

In this paper, we proposed an adaptive feature interaction learning neural
network EulerNet. Different from prior work, EulerNet modeled the arbitrary-
order feature interactions in a complex vector space by conducting space
mapping according to Euler’s formula. In EulerNet, the exponential powers of
feature interactions were converted into simple linear combinations of the
modulus and phase of the complex features, enabling it to adaptively learn the
arbitrary-order feature interactions in an efficient way. Furthermore,
EulerNet integrated the implicit and explicit feature interactions into a
unified architecture, which can achieve the mutual enhancement and largely
boost the model capabilities. As the major contribution, we proposed to
conduct feature interaction learning in the complex vector space, which
provides a way to enhance the representation capability of models and promote
the feature interaction learning in this area.

As future work, we consider incorporating the user behavior features into our
method, and further explore the use of attention mechanism in the complex
vector space to capture more informative correlations for various
recommendation tasks.

###### Acknowledgements.

This work was partially supported by National Natural Science Foundation of
China under Grant No. 62222215 and 62102038, Beijing Natural Science
Foundation under Grant No. 4222027, and Beijing Outstanding Young Scientist
Program under Grant No. BJJWZYJH012019100020098.

## References

  * (1)
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016\.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ (2016).
  * Blondel et al. (2016) Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. 2016.  Higher-order factorization machines.  _Advances in Neural Information Processing Systems_ 29 (2016).
  * Chang et al. (2010) Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. 2010.  Training and testing low-degree polynomial data mappings via linear SVM.  _Journal of Machine Learning Research_ 11, 4 (2010).
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_. 785–794.
  * Chen et al. (2019) Wenqiang Chen, Lizhang Zhan, Yuanlong Ci, Minghua Yang, Chen Lin, and Dugang Liu. 2019\.  FLEN: leveraging field for scalable CTR prediction.  _arXiv preprint arXiv:1911.04690_ (2019).
  * Cheng et al. (2014) Chen Cheng, Fen Xia, Tong Zhang, Irwin King, and Michael R Lyu. 2014.  Gradient boosting factorization machines. In _Proceedings of the 8th ACM Conference on Recommender systems_. 265–272.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Cheng et al. (2020) Weiyu Cheng, Yanyan Shen, and Linpeng Huang. 2020.  Adaptive factorization network: Learning adaptive-order feature interactions. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 3609–3616.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_ (2017).
  * He and Chua (2017) Xiangnan He and Tat-Seng Chua. 2017.  Neural factorization machines for sparse predictive analytics. In _Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval_. 355–364.
  * He et al. (2014) Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014\.  Practical lessons from predicting clicks on ads at facebook. In _Proceedings of the eighth international workshop on data mining for online advertising_. 1–9.
  * Hines (1996) J Wesley Hines. 1996\.  A logarithmic neural network architecture for unbounded non-linear function approximation. In _Proceedings of International Conference on Neural Networks (ICNN’96)_ , Vol. 2. IEEE, 1245–1250.
  * Hong et al. (2019) Fuxing Hong, Dongbo Huang, and Ge Chen. 2019.  Interaction-aware factorization machines for recommender systems. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 3804–3811.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7132–7141.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 2333–2338.
  * Huang et al. (2019) Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. 2019\.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 169–177.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _International conference on machine learning_. PMLR, 448–456.
  * Juan et al. (2016) Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. 2016\.  Field-aware factorization machines for CTR prediction. In _Proceedings of the 10th ACM conference on recommender systems_. 43–50.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Li et al. (2020) Zeyu Li, Wei Cheng, Yang Chen, Haifeng Chen, and Wei Wang. 2020.  Interpretable click-through rate prediction through hierarchical attention. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 313–321.
  * Li et al. (2019) Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. 2019.  Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 539–548.
  * Li et al. (2021) Zekun Li, Shu Wu, Zeyu Cui, and Xiaoyu Zhang. 2021\.  GraphFM: Graph Factorization Machines for Feature Interaction Modeling.  _arXiv preprint arXiv:2105.11866_ (2021).
  * Lian et al. (2018) Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018\.  xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1754–1763.
  * Lu et al. (2021) Wantong Lu, Yantao Yu, Yongzhe Chang, Zhen Wang, Chenhui Li, and Bo Yuan. 2021\.  A dual input-aware factorization machine for CTR prediction. In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_. 3139–3145.
  * Naumov et al. (2019) Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al. 2019\.  Deep learning recommendation model for personalization and recommendation systems.  _arXiv preprint arXiv:1906.00091_ (2019).
  * Pan et al. (2018) Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, and Quan Lu. 2018.  Field-weighted factorization machines for click-through rate prediction in display advertising. In _Proceedings of the 2018 World Wide Web Conference_. 1349–1357.
  * Pande (2020) Harshit Pande. 2020\.  Field-embedded factorization machines for click-through rate prediction.  _arXiv preprint arXiv:2009.09931_ (2020).
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Qu et al. (2016) Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. 2016.  Product-based neural networks for user response prediction. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 1149–1154.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization machines. In _2010 IEEE International conference on data mining_. IEEE, 995–1000.
  * Rendle et al. (2020) Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020.  Neural collaborative filtering vs. matrix factorization revisited. In _Fourteenth ACM conference on recommender systems_. 240–248.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007\.  Predicting clicks: estimating the click-through rate for new ads. In _Proceedings of the 16th international conference on World Wide Web_. 521–530.
  * Shan et al. (2016) Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016\.  Deep crossing: Web-scale modeling without manually crafted combinatorial features. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 255–262.
  * Song et al. (2019) Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.  Autoint: Automatic feature interaction learning via self-attentive neural networks. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1161–1170.
  * Sun et al. (2021) Yang Sun, Junwei Pan, Alex Zhang, and Aaron Flores. 2021\.  Fm2: Field-matrixed factorization machines for recommender systems. In _Proceedings of the Web Conference 2021_. 2828–2837.
  * Tian et al. (2023) Zhen Tian, Ting Bai, Zibin Zhang, Zhiyuan Xu, Kangyi Lin, Ji-Rong Wen, and Wayne Xin Zhao. 2023.  Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation. In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_. 715–723.
  * Wang et al. (2017) Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017\.  Deep & cross network for ad click predictions.  In _Proceedings of the ADKDD’17_. 1–7.
  * Wang et al. (2021) Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021.  DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In _Proceedings of the Web Conference 2021_. 1785–1797.
  * Xu and Wu (2020) Canran Xu and Ming Wu. 2020\.  Learning feature interactions with lorentzian factorization machine. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 6470–6477.
  * Yu et al. (2020) Feng Yu, Zhaocheng Liu, Qiang Liu, Haoli Zhang, Shu Wu, and Liang Wang. 2020\.  Deep interaction machine: A simple but effective model for high-order feature interactions. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2285–2288.
  * Zhang et al. (2016) Weinan Zhang, Tianming Du, and Jun Wang. 2016.  Deep learning over multi-field categorical data. In _European conference on information retrieval_. Springer, 45–57.
  * Zhang et al. (2021) Weinan Zhang, Jiarui Qin, Wei Guo, Ruiming Tang, and Xiuqiang He. 2021.  Deep learning for click-through rate estimation.  _arXiv preprint arXiv:2104.10584_ (2021).
  * Zhao et al. (2022) Wayne Xin Zhao, Yupeng Hou, Xingyu Pan, Chen Yang, Zeyu Zhang, Zihan Lin, Jingsen Zhang, Shuqing Bian, Jiakai Tang, Wenqi Sun, et al. 2022\.  RecBole 2.0: Towards a More Up-to-Date Recommendation Library. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_. 4722–4726.
  * Zhao et al. (2021) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021\.  Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 4653–4664.
  * Zhu et al. (2020) Jieming Zhu, Jinyang Liu, Weiqi Li, Jincai Lai, Xiuqiang He, Liang Chen, and Zibin Zheng. 2020.  Ensembled CTR prediction via knowledge distillation. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2941–2958.