010005 2009 J. A. Bertolotto 010005

Using Monte Carlo simulations and finite-size scaling analysis, the critical
behavior of attractive rigid rods of length $k$ ($k$-mers) on square lattices
has been studied. An ordered state, with the majority of $k$-mers being
horizontally or vertically aligned, was found. This ordered phase is separated
from the disordered state by a continuous transition occurring at a critical
density $\theta_{c}$, which increases linearly with the magnitude of the
lateral interactions.

# The effect of the lateral interactions on the critical behavior of long
straight rigid rods on two-dimensional lattices

P. Longone  [inst1] D. H. Linares  [inst1] A. J. Ramirez-Pastor[inst1] E-mail:
<EMAIL_ADDRESS>
(9 July 2009; 6 November 2009)

††volume: 1

99 inst1 Departamento de Física, Instituto de Física Aplicada, Universidad
Nacional de San Luis-CONICET, Chacabuco 917, D5700BWS San Luis, Argentina.

##  1 Introduction

The study of systems of hard non-spherical colloidal particles has, for many
years, been attracting a great deal of interest and the activity in this field
is still growing [1–14]. An early seminal contribution to this subject was
made by Onsager [1] with his paper on the isotropic-nematic (I-N) phase
transition in liquid crystals. The Onsager’s theory predicted that very long
and thin rods interacting with only excluded volume interaction can lead to
long-range orientational (nematic) order. Thus, at low densities, the
molecules are typically far from each other and the resulting state is an
isotropic gas. However, at large densities, it is more favorable for the
molecules to align spontaneously (there are many more ways of placing nearly
aligned rods than randomly oriented ones), and a nematic phase is present at
equilibrium.

Interestingly, a number of papers have appeared recently, in which the I-N
transition was studied in two dimensions [10–14]. In Ref. [10], the authors
gathered strong numerical evidence to suggest that a system of square
geometry, with two allowed orientations, shows nematic order at intermediate
densities for $k\geq 7$ and provided a qualitative description of a second
phase transition (from a nematic order to a non-nematic state) occurring at a
density close to $1$. However, the authors were not able to determine the
critical quantities (critical point and critical exponents) characterizing the
I-N phase transition occurring in the system. This problem was resolved in
Refs. [11, 12], where an accurate determination of the critical exponents,
along with the behavior of Binder cumulants, showed that the transition from
the low-density disordered phase to the intermediate-density ordered phase
belongs to the 2D Ising universality class for square lattices and the three-
state Potts universality class for honeycomb and triangular lattices. Later,
the I-N phase transition was analyzed by combining Monte Carlo (MC)
simulations and theoretical analysis [13, 14]. The study in Refs. [13, 14]
allowed (1) to obtain $\theta_{c}$ as a function of $k$ for square, triangular
and honeycomb lattices, being $\theta_{c}(k)\propto k^{-1}$ (this dependence
was already noted in Ref. [10]); and (2) to determine the minimum value of $k$
($k_{min}$), which allows the formation of a nematic phase on triangular
($k_{min}=7$) and honeycomb ($k_{min}=11$) lattices.

In a recent paper, Fischer and Vink [15] indicated that the transition studied
in Refs. [10–14] corresponds to a liquid-gas transition, rather than I-N. This
interpretation is consistent with the 2D-Ising critical behavior observed for
monodisperse rigid rods on square lattices [11]. This point will be discussed
in more detail in Sec. III.

In contrast to the systems studied in Refs. [10–14], many rod-like biological
polymers are formed by monomers reversibly self-assembling into chains of
arbitrary length. Consequently, these systems exhibit a broad equilibrium
distribution of filament lengths. A model of self-assembled rigid rods has
been recently considered by Tavares et al. [16]. The authors focused on a
system composed of monomers with two attractive (sticky) poles that polymerize
reversibly into polydisperse chains and, at the same time, undergo a
continuous I-N phase transition. The obtained results revealed that nematic
ordering enhances bonding. In addition, the average rod length was described
quantitatively in both phases, while the location of the ordering transition,
which was found to be continuous, was predicted semiquantitatively by the
theory.

Beyond the differences between lattice geometry and the characteristics of the
rods (self-assembled or not), one fundamental feature is preserved in all the
studies mentioned above. This is the assumption that only excluded volume
interactions between the rods are considered (except in Ref. [16], where
monomers with two attractive bonding sites polymerize into polydisperse rods).
Moreover, one often encounters phrases in the literature, such as “This theory
[Onsager’s theory] shows that repulsive interactions [excluding volume
interactions] alone can lead to long-range orientational nematic order,
disproving the notion that attractive interactions are a prerequisite” [17],
which could be ambiguous with respect to the role that attractive lateral
interactions between the rods should play in reinforcing (or not) the nematic
order.

In this context, it is of interest and of value to inquire how the existence
of lateral interactions between the rods influences the phase transition
occurring in the system. The objective of this paper is to provide a thorough
analysis in this direction. For this purpose, an exhaustive study of the phase
transition occurring in a system of attractive rigid rods deposited on square
lattices was performed. The results revealed that $(i)$ the orientational
order survives in the presence of attractive lateral interactions; $(ii)$ the
critical density shifts to higher values as the magnitude of the lateral
interactions is increased; and $(iii)$ the continuous transition becomes first
order for interaction strength $w>w_{c}$ (in absolute values).

The outline of the paper is as follows. In Sec. II we describe the lattice-gas
model and the simulation scheme. In Sec. III we present the MC results.
Finally, the general conclusions are given in Sec. IV.

##  2 Lattice-gas model and Monte Carlo simulation scheme

We address the general case of adsorbates assumed to be linear rigid particles
containing $k$ identical units ($k$-mers), with each one occupying a lattice
site. Small adsorbates would correspond to the monomer limit ($k=1$). The
distance between $k$-mer units is assumed to be equal to the lattice constant;
hence exactly $k$ sites are occupied by a $k$-mer when adsorbed (see Fig. 1).
The surface is represented as an array of $M=L\times L$ adsorptive sites in a
square lattice arrangement, where $L$ denotes the linear size of the array. In
order to describe the system of $N$ $k$-mers adsorbed on $M$ sites at a given
temperature $T$, let us introduce the occupation variable $c_{i}$ which can
take the values $c_{i}=0$ if the corresponding site is empty and $c_{i}=1$ if
the site is occupied. On the other hand, molecules adsorb or desorb as one
unit, neglecting any possible dissociation. Under these considerations, the
Hamiltonian of the system is given by

$H=w\sum_{\langle i,j\rangle}c_{i}c_{j}-N(k-1)w+\epsilon_{o}\sum_{i}c_{i}$ (1)

where $w$ is the nearest-neighbor (NN) interaction constant which is assumed
to be attractive (negative), $\langle i,j\rangle$ represents pairs of NN sites
and $\epsilon_{o}$ is the energy of adsorption of one given surface site. The
term $N(k-1)w$ is subtracted in eq. (1) since the summation over all the pairs
of NN sites overestimates the total energy by including $N(k-1)$ bonds
belonging to the $N$ adsorbed $k$-mers. Because the surface was assumed to be
homogeneous, the interaction energy between the adsorbed dimer and the atoms
of the substrate $\epsilon_{o}$ was neglected for the sake of simplicity.

Figure 1: Linear tetramers adsorbed on square lattices. Full and empty circles
represent tetramer units and empty sites, respectively.

In order to characterize the phase transition, we use the order parameter
defined in Ref. [11], which in this case can be written as

$\delta=\frac{\left|{n}_{h}-{n}_{v}\right|}{{n}_{h}+{n}_{v}}$ (2)

where $n_{h}(n_{v})$ is the number of rods aligned along the horizontal
(vertical) direction. When the system is disordered $(\theta<\theta_{c})$, all
orientations are equivalents and $\delta$ is zero. As the density is increased
above $\theta_{c}$, the $k$-mers align along one direction and $\delta$ is
different from zero. Thus, $\delta$ appears as a proper order parameter to
elucidate the phase transition.

The problem has been studied by grand canonical MC simulations using a typical
adsorption-desorption algorithm. The procedure is as follows. Once the value
of the chemical potential $\mu$ is set, a linear $k$-uple of nearest-neighbor
sites is chosen at random and an attempt is made to change its occupancy state
with probability $W={\rm min}\left\\{1,\exp\left(-\Delta
H/k_{B}T\right)\right\\}$, where $\Delta H=H_{f}-H_{i}$ is the difference
between the Hamiltonians of the final and initial states and $k_{B}$ is the
Boltzmann constant. In addition, displacement (diffusional relaxation) of
adparticles to nearest-neighbor positions, by either jumps along the $k$-mer
axis or reptation by rotation around the $k$-mer end, must be allowed in order
to reach equilibrium in a reasonable time. A MC step (MCs) is achieved when
$M$ $k$-uples of sites have been tested to change its occupancy state.
Typically, the equilibrium state can be well reproduced after discarding the
first $r^{\prime}=10^{7}$ MCs. Then, the next $r=2\times 10^{7}$ MCs are used
to compute averages.

Figure 2: Adsorption isotherms (coverage versus chemical potential) for
$k=10$, $L=100$ different $w/k_{B}T$’s as indicated. Inset: Adsorption phase
diagram of attractive $10$-mers on square lattices.

In our MC simulations, we varied the chemical potential $\mu$ and monitored
the density $\theta$ and the order parameter $\delta$, which can be calculated
as simple averages. The reduced fourth-order cumulant $U_{L}$ introduced by
Binder [18] was calculated as:

$U_{L}=1-\frac{\langle\delta^{4}\rangle}{3\langle\delta^{2}\rangle^{2}},$ (3)

where $\langle\cdots\rangle$ means the average over the MC simulation runs.
All calculations were carried out using the BACO parallel cluster (composed by
60 PCs each with a 3.0 GHz Pentium-4 processor and 90 PCs each with a 2.4 GHz
Core 2 Quad processor) located at Instituto de Física Aplicada, Universidad
Nacional de San Luis-CONICET, San Luis, Argentina.

##  3 Results

The calculations were developed for linear $10$-mers ($k=10$). With this value
of $k$ and for non-interacting rods, it is expected the existence of a nematic
phase at intermediate densities [10]. The surface was represented as an array
of adsorptive sites in a square lattice arrangement with conventional periodic
boundary conditions. The effect of finite size was investigated by examining
lattices with $L=50,100,150,200$.

In order to understand the basic phenomenology, we consider, in the first
place, the behavior of the adsorption isotherms in presence of attractive
lateral interactions between the $k$-mers.

Fig. 2 shows typical adsorption isotherms (coverage versus $\mu/k_{B}T$) for
linear $10$-mers with different values of the lateral interaction (the solid
circles represent the Langmuir case, $w/k_{B}T=0$).

The isotherms shift to lower values of chemical potential, and their slopes
increase as the ratio $w/k_{B}T$ increases (in absolute value). For
interaction strength above the critical value ($w>w_{c}$, in absolute values)
the system undergoes a first-order phase transition, which is observed in the
clear discontinuity in the adsorption isotherms111In this situation, which has
been observed experimentally in numerous systems, the only phase which one
expects is a lattice-gas phase at low coverage, separated by a two-phase
coexistence region from a “lattice-fluid” phase at higher coverage. This
condensation of a two-dimensional gas to a two-dimensional liquid is similar
to that of a lattice-gas of attractive monomers. However, the symmetry
particle-vacancy (valid for monoatomic particles) is broken for $k$-mers and
the isotherms are asymmetric with respect to $\theta=0.5$.. In the case
studied, this critical value is approximately $w_{c}/k_{B}T\approx-0.80$ (or
$k_{B}T_{c}/w\approx-1.25$). The behavior of the adsorption isotherms also
allows us to calculate the phase diagram of the adsorbed monolayer in
“temperature-coverage” coordinates. In fact, once obtained the real value of
the chemical potential (or critical chemical potential $\mu_{c}$) in the two-
phase region, the corresponding critical densities can be easily calculated.
By repeating this procedure for different temperatures ranging between $0$ and
$T_{c}$, the coexistence curve can be built [20]. A typical phase diagram,
obtained in this case for attractive $10$-mers, is shown in the inset of Fig.
2.

On the basis of the study in Fig. 2, our next objective is to obtain evidence
for the existence of nematic order in the range $-0.80\leq w/k_{B}T<0$ of
attractive interactions. For this purpose, the behavior of the order parameter
$\delta$ as a function of coverage was analyzed for $k=10$, $L=100$ and
different values of the lateral interaction. The results are shown in Fig. 3,
revealing that $(i)$ the orientational order survives in the presence of
attractive lateral interactions and $(ii)$ the critical density shifts to
higher values as the magnitude of the lateral interactions is increased.

Figure 3: Surface coverage dependence of the nematic order parameter for
$k=10$, $L=100$ different $w/k_{B}T$’s as indicated.

In order to corroborate the results obtained in the last figure, we now study
the dependence of $\theta_{c}$ on $w/k_{B}T$. In the case of the standard
theory of FSS [18, 19], when the phase transition is temperature driven, the
technique allows for various efficient routes to estimate $T_{c}$ from MC
data. One of these methods, which will be used in this case, is from the
temperature dependence of $U_{L}(T)$, which is independent of the system size
for $T=T_{c}$. In other words, $T_{c}$ is found from the intersection of the
curve $U_{L}(T)$ for different values of $L$, since $U_{L}(T_{c})=$const. In
our study, we modified the conventional FSS analysis by replacing temperature
by density [11]. Under this condition, the critical density has been estimated
from the plots of the reduced four-order cumulants $U_{L}(\theta)$ plotted
versus $\theta$ for several lattice sizes. As an example, Fig. 4 shows the
results for $w/k_{B}T=-0.125$. In this case, the value obtained was
$\theta_{c}=0.542(2)$. In the inset, the data are plotted over a wider range
of temperatures, exhibiting the typical behavior of the cumulants in the
presence of a continuous phase transition.

Figure 4: Curves of $U_{L}(\theta)$ vs $\theta$ for $k=10$, $w/k_{B}T=-0.125$
and square lattices of different sizes. From their intersections one obtained
$\theta_{c}$. In the inset, the data are plotted over a wider range of
densities. Figure 5: Temperature-coverage phase diagram corresponding to
attractive $k$-mers with $k=10$. The inset in the upper-left (lower-right)
corner shows a typical configuration in the nematic (isotropic) phase.

The procedure of Fig. 4 was repeated for $-0.80\leq w/k_{B}T<0$, showing that
the values of $\theta_{c}$ increase linearly with the magnitude of the lateral
couplings (see solid squares in Fig. 5). The critical line (dotted line in the
figure) was obtained from the linear fit of the numerical data. As it is
possible to observe, the range of coverage at which the transition occurs
diminishes as $w/k_{B}T$ is increased (in absolute value). This finding
indicates that the presence of attractive lateral interactions between the
rods does not favor the formation of nematic order in the adlayer. The
phenomenon can be understood from the behavior of the second virial
coefficient, which will initially decrease on introducing attractive $w$. This
decrease implies that the isotherms shift to lower values of chemical
potential, and consequently, the critical point shifts to higher densities.

We did not assume any particular universality class for the transitions
analyzed here in order to calculate their critical densities, since the
analysis relied on the order parameter cumulant’s properties. However, the
fixed value of the cumulants, $U^{*}=0.617(9)$, is consistent with the
extremely precise transfer matrix calculation of $U^{*}=0.6106901(5)$ [21] for
the 2D Ising model. This finding may be taken as an indication that the phase
transition belongs to the 2D Ising universality class.

With respect to the behavior of the system for $w/k_{B}T<-0.80$, the adsorbed
layer “jumps” from a low-coverage phase to a high-coverage phase. This effect,
which has been discussed in Fig. 2, is represented in Fig. 5 by the dashed
coexistence line. The low-coverage phase is an isotropic state, similar to
that observed for $w/k_{B}T>-0.80$ and low density (see inset in the lower-
right corner of Fig. 5). On the other hand, the high-coverage phase is also an
isotropic state, but characterized by the presence of local orientational
order (domains of parallel $k$-mers). A typical configuration in this regime
is shown in Fig. 6).

Figure 6: Typical configuration of the adlayer in the high-coverage phase and
$w/k_{B}T<-0.80$.

Finally, it is worth pointing out that: $(1)$ the behavior of the order
parameter in Fig. 3 clearly indicates that the transition from the low-density
disordered phase to the intermediate-density ordered phase is an isotropic to
nematic phase transition (when all the words have the usual meaning). In this
case, the transition under study belongs to the 2D Ising universality class.
It can also be thought of as an unmixing or liquid-gas transition [15]. For
this reason we have called gas and liquid to the phases reported in Fig. 5;
and $(2)$ even though it has not been rigorously proved yet, a second phase
transition for non-interacting rods at high densities has been theoretically
predicted [10] and numerically confirmed [13]. This result has not been
confirmed for the case of attractive rods. An exhaustive study on this subject
will be the object of future work.

##  4 Conclusions

We have addressed the critical properties of attractive rigid rods on square
lattices with two allowed orientations, and shown the dependence of the
critical density on the magnitude of the lateral interactions $w/k_{B}T$. The
results were obtained by using MC simulations and FSS theory.

Several conclusions can be drawn from the present work. On the one hand, we
found that even though the presence of attractive lateral interactions between
the rods does not favor the formation of nematic order in the adlayer, the
orientational order survives in a range that goes from $w/k_{B}T=0$ up to
$w_{c}/k_{B}T\approx-0.80$ ($w_{c}/k_{B}T$ represents the critical value at
which occurs a typical transition of condensation in the adlayer). In this
region of $w/k_{B}T$, the critical density increases linearly with the
magnitude of the lateral couplings. On the other hand, the evaluation of the
fixed point value of the cumulants $U^{*}=0.617(9)$ indicates that, as in the
case of non-interacting rods, the observed phase transition belongs to the
universality class of the two-dimensional Ising model.

With respect to the behavior of the system for $w/k_{B}T<-0.80$, the
continuous transition becomes first order. Thus, the adsorbed layer jumps from
a low-coverage phase, similar to that observed for $w/k_{B}T>-0.80$ and low
density, to an isotropic phase at high coverage, characterized by the presence
of local orientational order (domains of parallel $k$-mers)

Future efforts will be directed to $(1)$ extend the study to repulsive lateral
interaction between the $k$-mers; $(2)$ obtain the whole phase diagram in the
space (temperature-coverage-rod’s size); $(3)$ develop an exhaustive study on
critical exponents and universality and $(4)$ characterize the second phase
transition from a nematic order to a non-nematic state occurring at high
density.

###### Acknowledgements.

This work was supported in part by CONICET (Argentina) under project number
PIP 112-200801-01332; Universidad Nacional de San Luis (Argentina) under
project 322000 and the National Agency of Scientific and Technological
Promotion (Argentina) under project 33328 PICT 2005.

## References

  * [1] L Onsager, The effects of shape on the interaction of colloidal particles, Ann. N. Y. Acad. Sci. 51, 627 (1949).
  * [2] P J Flory, Thermodynamics of high polymer solutions, J. Chem. Phys. 10, 51 (1942); P J Flory, Principles of Polymers Chemistry, Cornell University Press, Ithaca, NY (1953).
  * [3] M L Huggins, Some properties of solutions of long-chain compounds, J. Phys. Chem. 46, 151 (1942); M L Huggins, Thermodynamic properties of solutions of long-chain compounds, Ann. N. Y. Acad. Sci. 43, 1 (1942); M L Huggins, Theory of solutions of high polymers, J. Am. Chem. Soc. 64, 1712 (1942).
  * [4] J P Straley, Liquid crystals in two dimensions, Phys. Rev. A 4, 675 (1971).
  * [5] J Vieillard-Baron, Phase transitions of the classical hard-ellipse system, J. Chem. Phys. 56, 4729 (1972).
  * [6] D Frenkel, R Eppenga, Evidence for algebraic orientational order in a two-dimensional hard-core nematic, Phys. Rev. A 31, 1776 (1985).
  * [7] K J Strandburg, Two-dimensional melting, Rev. Mod. Phys. 60, 161 (1988).
  * [8] A J Phares, F J Wunderlich, Thermodynamics and molecular freedom of dimers on plane triangular lattices, J. Math. Phys. 27, 1099 (1986).
  * [9] A J Phares, F J Wunderlich, J D Curley, D W Grumbine Jr, Structural ordering of interacting dimers on a square lattice, J. Phys. A: Math. Gen. 26, 6847 (1993).
  * [10] A Ghosh, D Dhar, On the orientational ordering of long rods on a lattice, Eur. Phys. Lett. 78, 20003 (2007).
  * [11] D A Matoz-Fernandez, D H Linares, A J Ramirez-Pastor, Determination of the critical exponents for the isotropic-nematic phase transition in a system of long rods on two-dimensional lattices: Universality of the transition, Europhys. Lett. 82, 50007 (2008).
  * [12] D A Matoz-Fernandez, D H Linares, A J Ramirez-Pastor, Critical behavior of long linear $k$-mers on honeycomb lattices, Physica A 387, 6513 (2008).
  * [13] D H Linares, F Romá, A J Ramirez-Pastor, Entropy-driven phase transition in a system of long rods on a square lattice, J. Stat. Mech. P03013 (2008).

  * [14] D A Matoz-Fernandez, D H Linares, A J Ramirez-Pastor, Critical behavior of long straight rigid rods on two-dimensional lattices: Theory and Monte Carlo simulations, J. Chem. Phys. 128, 214902 (2008).
  * [15] T Fischer, R L C Vink, Restricted orientation “liquid crystal” in two dimensions: Isotropic-nematic transition or liquid-gas one(?), Europhys. Lett. 85, 56003 (2009).
  * [16] J M Tavares, B Holder, M M Telo da Gama, Structure and phase diagram of self-assembled rigid rods: Equilibrium polydispersity and nematic ordering in two dimensions, Phys. Rev. E 79, 021505 (2009).
  * [17] H H Wensink, Columnar versus smectic order in systems of charged colloidal rods, J. Chem. Phys. 126, 194901 (2007).
  * [18] K Binder, Applications of the Monte Carlo Method in Statistical Physics. Topics in current Physics, Springer, Berlin (1984).
  * [19] V Privman, Finite Size Scaling and Numerical Simulation of Statistical Systems, World Scientific, Singapore (1990).
  * [20] T L Hill, An Introduction to Statistical Thermodynamics, Addison Wesley Publishing Company, Reading, MA (1960).
  * [21] G Kamieniarz, H W J Blöte, Universal ratio of magnetization moments in two-dimensional Ising models, J. Phys. A: Math. Gen. 26, 201 (1993).


# An Embedding Learning Framework for Numerical Features in CTR Prediction

Huifeng Guo1∗, Bo Chen1∗, Ruiming Tang1, Weinan Zhang2, Zhenguo Li1, Xiuqiang
He1 huifeng.guo, chenbo116, tangruiming, li.zhenguo<EMAIL_ADDRESS><EMAIL_ADDRESS>1Huawei Noah’s Ark Lab 2Shanghai Jiao Tong University

(2021)

###### Abstract.

Click-Through Rate (CTR) prediction is critical for industrial recommender
systems, where most deep CTR models follow an Embedding & Feature Interaction
paradigm. However, the majority of methods focus on designing network
architectures to better capture feature interactions while the feature
embedding, especially for numerical features, has been overlooked. Existing
approaches for numerical features are difficult to capture informative
knowledge because of the low capacity or hard discretization based on the
offline expertise feature engineering. In this paper, we propose a novel
embedding learning framework for numerical features in CTR prediction
(AutoDis) with high model capacity, end-to-end training and unique
representation properties preserved. AutoDis consists of three core
components: _meta-embeddings_ , _automatic discretization_ and _aggregation_.
Specifically, we propose meta-embeddings for each numerical field to learn
global knowledge from the perspective of field with a manageable number of
parameters. Then the differentiable automatic discretization performs soft
discretization and captures the correlations between the numerical features
and meta-embeddings. Finally, distinctive and informative embeddings are
learned via an aggregation function. Comprehensive experiments on two public
and one industrial datasets are conducted to validate the effectiveness of
AutoDis. Moreover, AutoDis has been deployed onto a mainstream advertising
platform, where online A/B test demonstrates the improvement over the base
model by 2.1% and 2.7% in terms of CTR and eCPM, respectively. In addition,
the code of our framework is publicly available in
MindSpore111https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/recommend/autodis.

Numerical Features, Embedding Learning, Click-Through Rate Prediction, Neural
Network

*Co-first authors with equal contributions. Huifeng Guo and Ruiming Tang are the corresponding authors.

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2021; Virtual Event, Singapore††booktitle: Proceedings of the 27th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), August
14–18, 2021, Virtual Event, Singapore††price: 15.00††doi:
10.1145/3447548.3467077††isbn: 978-1-4503-8332-5/21/08††ccs: Information
systems Information retrieval††ccs: Information systems Recommender
systems††ccs: Computing methodologies Machine learning

##  1\. Introduction

To alleviate the problem of information explosion, recommender systems are
widely deployed to provide personalized information filtering in online
information services, such as web search, commodity recommendation and online
advertising. CTR prediction is crucial in recommender systems, which is to
estimate the probability that a user will click on a recommended item under a
specific context, so that recommendation decisions can be made based on the
predicted CTR values (McMahan et al., 2013; Zhou et al., 2018). Because of the
superior performance of representation learning in computer vision and natural
language processing, deep learning techniques attract the attention of
recommendation community. Therefore, plenty of industrial companies propose
various deep CTR models and deploy them in their commercial systems, such as
Wide & Deep (Cheng et al., 2016) in Google Play, DeepFM (Guo et al., 2017) in
Huawei AppGallery and DIN (Zhou et al., 2018) in Taobao.

Figure 1. Embedding & Feature Interaction Architecture for CTR prediction.

As shown in Figure 1, most of the existing deep CTR models follow an Embedding
& Feature Interaction (FI) paradigm. Due to the significant importance for
feature interaction in CTR prediction, the majority works focus on designing
network architectures in FI module to better capture explicit or implicit
feature interactions, e.g., wide component in Wide & Deep (Cheng et al.,
2016), CrossNet in DCN (Wang et al., 2017), and attention mechanism in DIN
(Zhou et al., 2018). Although not well studied in the literature, Embedding
module is also a key factor to deep CTR models for two reasons: (1) Embedding
module is the cornerstone of the subsequent FI module and directly affects the
effectiveness of FI module (Zhang et al., 2016); (2) The number of parameters
in deep CTR models is heavily concentrated in Embedding module, which
naturally has high impact on the prediction performance (Joglekar et al.,
2019). However, the Embedding module is overlooked by the research community,
which motivates us to conduct in-depth research.

The Embedding module works in a look-up table manner generally and maps each
feature of the input data in categorical fields222For clarify, we use “field”
to represent a class of features following (Juan et al., 2016; Qu et al.,
2019) and “feature to represent a certain value in a specific field., to the
latent embedding space with learnable parameters (Guo et al., 2017).
Considering an example instance (Gender=Male, Day=Tuesday, Height=175.6,
Age=18), for categorical field Gender, features are mapped to embeddings
trivially by assigning a unique embedding to each individual feature, which is
widely used in existing works (Guo et al., 2017; Qu et al., 2019).
Unfortunately, such categorization strategy cannot be utilized to handle
numerical features, as there may be infinitely many feature values in a
numerical field (e.g., Height).

In practical, existing representation approaches for numerical feature can be
summarized into three categories (red dashed box in Figure 1): (1) No
Embedding: using the original feature values or its transformation directly
without learning embeddings (Cheng et al., 2016; Covington et al., 2016;
Naumov et al., 2019); (2) Field Embedding: learning a single field embedding
for each numerical field (Guo et al., 2017; Song et al., 2019); (3)
Discretization: transforming numerical features to categorical features
through various heuristic discretization strategies and assigning embedding
like categorization strategy (Qu et al., 2019). However, the former two
categories may lead to poor performance due to the low capacity of
representations. The last category is also sub-optimal because such heuristic-
based discretization rules cannot be optimized with the ultimate goal of CTR
models. Besides, hard discretization-based methods suffer from _SBD_ (Similar
value But Dis-similar embedding) and _DBS_ (Dis-similar value But Same
embedding) problems whose details will be discussed in Session 2.

To address the limitations of existing approaches, we propose an automatic
end-to-end embedding learning framework for numerical features based on soft
discretization (AutoDis). AutoDis consists of three core modules: _meta-
embeddings_ , _automatic discretization_ and _aggregation_ to enable high
model capacity, end-to-end training and unique representation properties.
Specifically, we elaborately design a set of meta-embeddings for each
numerical field, which are shared among all the intra-field feature values and
learn global knowledge from the perspective of field with a manageable number
of embedding parameters. Then a differentiable automatic discretization module
is leveraged to perform soft discretization and capture correlations between
each numerical feature and field-specific meta-embeddings. Finally, an
aggregation function is performed to learn unique Continuous-But-Different
representations. To the best of our knowledge, AutoDis is the first end-to-end
_soft discretization_ framework for numerical feature embedding that can be
optimized jointly with the ultimate goal of deep CTR models. We summarize the
main contributions as follows.

  * •

We propose AutoDis, a pluggable embedding learning framework for numerical
features, which has high model capacity and is able to generate unique
representations with a manageable number of parameters in an end-to-end
manner.

  * •

In AutoDis, we design meta-embeddings for each numerical field to learn global
shared knowledge. Besides, a differentiable automatic discretization is used
to capture the correlations between the numerical features and meta-
embeddings, while an aggregation process is performed to learn a unique
Continuous-But-Different representation for each feature.

  * •

Comprehensive experiments are conducted on two public and an industrial
datasets to demonstrate the superiority of AutoDis over the existing
representation methods for numerical features. Moreover, AutoDis works
compatibly with various popular deep CTR models, by improving their
recommendation performance significantly. Online A/B test in a mainstream
advertising platform shows that AutoDis improves the commercial baseline by
2.1% and 2.7% in terms of CTR and eCPM.

##  2\. Preliminary

In this section, we delve into the feature embedding learning procedure in CTR
prediction. Suppose a dataset for training CTR models consists of $Q$
instances $(\mathbf{x},y)$, where $y$ is label and $\mathbf{x}$ is a multi-
field data record including $M$ categorical fields and $N$ numerical fields:

(1)
$\small\mathbf{x}=[\underbrace{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{M}}_{\text{one-
hot vectors}};\underbrace{x_{1},x_{2},\dots,x_{N}}_{\text{scalars}}],$

where $\mathbf{x}_{i}$ is the one-hot vector of the feature value in the
$i$-th categorical field and $x_{j}$ is the scalar value of the $j$-th
numerical field.

For the $i$-th categorical field, the feature embedding can be obtained by
embedding look-up operation:

(2) $\small\mathbf{e}_{i}=\mathbf{E}_{i}\cdot\mathbf{x}_{{i}},$

where $\mathbf{E}_{i}\in\mathbb{R}^{v_{i}\times d}$ is the embedding matrix
for $i$-th field, $v_{i}$ and $d$ is the vocabulary size and embedding size.
Therefore, the representations for categorical fields are constructed as
$[\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e}_{M}]\in\mathbb{R}^{M\times
d}$. For numerical fields, existing representation approaches can be
summarized into three categories: No Embedding, Field Embedding and
Discretization.

###  2.1. Category 1: No Embedding

This category uses the original values or the transformations directly without
learning embedding. For instance, Wide & Deep (Cheng et al., 2016) of Google
Play and DMT (Gu et al., 2020) of JD.com use the original and normalized
numerical features respectively. Besides, YouTube DNN (Covington et al., 2016)
utilizes several transformations (i.e., square and square root) of normalized
feature value $\tilde{x}_{j}$:

(3)
$\mathbf{e}_{{YouTube}}=[\tilde{x}_{1}^{2},\tilde{x}_{1},\sqrt{\tilde{x}_{1}},\tilde{x}_{2}^{2},\tilde{x}_{2},\sqrt{\tilde{x}_{2}},\dots,\tilde{x}_{N}^{2},\tilde{x}_{N},\sqrt{\tilde{x}_{N}}],$

where $\mathbf{e}_{{YouTube}}\in\mathbb{R}^{3N}$. In DLRM (Naumov et al.,
2019) of Facebook, they use a multi-layer perception to model all numerical
features:

(4) $\mathbf{e}_{{DLRM}}=[DNN([x_{1},x_{2},\dots,x_{N}])],$

where the structure of $DNN$ is 512-256-$d$ and
$\mathbf{e}_{{DLRM}}\in\mathbb{R}^{d}$. Intuitively, these No Embedding
methods are difficult to capture informative knowledge of numerical fields due
to their low capacity.

###  2.2. Category 2: Field Embedding

The commonly used method in academia is Field Embedding (Guo et al., 2017;
Song et al., 2019), where all the numerical features in the same field share a
uniform field embedding and then multiply this field embedding with their
feature values:

(5)
$\mathbf{e}_{FE}=[x_{1}\cdot\mathbf{e}_{1},x_{2}\cdot\mathbf{e}_{2},\dots,x_{N}\cdot\mathbf{e}_{N}],$

where $\mathbf{e}_{j}\in\mathbb{R}^{d}$ is the uniform embedding vector for
the $j$-th numerical feature field and
$\mathbf{e}_{{FE}}\in\mathbb{R}^{N\times d}$. However, the representation
capacity of Field Embedding is limited due to the single shared field-specific
embedding as well as the linear scaling relationship between different intra-
field features.

###  2.3. Category 3: Discretization

A popular method in industrial recommender systems to handle numerical
features is Discretization, i.e., transforming numerical features to
categorical features. For a feature $x_{j}$ in the $j$-th numerical field, the
feature embedding $\mathbf{e}_{j}$ can be obtained by a two-stage operation:
discretization and embedding look-up.

(6) $\mathbf{e}_{j}=\mathbf{E}_{j}\cdot d_{j}(x_{j}),$

where $\mathbf{E}_{j}\in\mathbb{R}^{H_{j}\times d}$ is the embedding matrix
for $j$-th field, $H_{j}$ is the number of buckets after discretization.
Function $d_{j}(\cdot)$ is the manually-designed discretization method for the
$j$-th numerical field, which project each feature value into one of the
$H_{j}$ buckets. Specifically, there are three widely-used discretization
approaches.

(1) EDD/EFD (Equal Distance/Frequency Discretization). EDD partitions the
feature values into $H_{j}$ equal-width buckets. Suppose the range of features
of the $j$-th field is $[x_{j}^{min},x_{j}^{max}]$, the interval width is
defined as $w_{j}=(x_{j}^{max}-x_{j}^{min})/H_{j}$. Accordingly, the
discretized result $\widehat{x}_{j}$ is obtained via EDD discretization
function $d_{j}^{EDD}(\cdot)$:

(7) $\widehat{x}_{j}=d_{j}^{EDD}(x_{j})=floor((x_{j}-x_{j}^{min})/w_{j}).$

Similarly, EFD divides the range $[x_{j}^{min},x_{j}^{max}]$ to several
buckets such that each bucket contains equal number of features.

(2) LD (Logarithm Discretization). The champion (Juan et al., 2016) of Criteo
advertiser prediction in Kaggle
Competition333http://https//www.kaggle.com/c/criteo-display-ad-
challenge/discussion/10555 utilizes the logarithm and floor operation to
transform the numerical features to categorical form. The bucket
$\widehat{x}_{j}$ after discretized is obtained by $d_{j}^{LD}(\cdot)$:

(8) $\widehat{x}_{j}=d_{j}^{LD}(x_{j})=floor(log(x_{j})^{2}).$

(3) TD (Tree-based Discretization). Apart from deep learning models, tree-
based models (such as GBDT) are widely-used in recommendation because of their
capability of handling numerical features effectively. Therefore, many tree-
based approaches are utilized to discretize the numerical features (He et al.,
2014; Ke et al., 2019; Grabczewski and Jankowski, 2005).

Figure 2. Limitations of existing discretization approaches: (1) _TPP_ (Two-
Phase Problem); (2) _SBD_ (Similar value But Dis-similar embedding); _DBS_
(Dis-similar value But Same embedding).

Although Discretization is widely-used in industry (Qu et al., 2019; He et
al., 2014), they still suffer from three limitations (shown in Figure 2).

  * •

_TPP_ (Two-Phase Problem): The discretization process is determined by either
a heuristic rule or another model, so that it cannot be optimized together
with the ultimate goal of the CTR prediction task, leading to sub-optimal
performance.

  * •

_SBD_ (Similar value But Dis-similar embedding): These discretization
strategies may separate similar features (boundary values) into two different
buckets hence their afterwards embeddings are significantly different. For
instance, a commonly used discretization for Age field is to determine [18,40]
as teenage and [41,65] as mid-age, which results in significantly different
embeddings for numerical value 40 and 41.

  * •

_DBS_ (Dis-similar value But Same embedding): Existing discretization
strategies may group significantly different elements into the same bucket,
leading to indistinguishable embedding. Using the same example, numerical
values between 18 and 40 are in the same bucket and therefore assigned with
the same embedding. However, persons with 18-year-old and 40-year-old are
likely with very different characteristics. Discretization-based strategies
cannot describe the continuity of numerical feature changes effectively.

###  2.4. Distinction and Relationship

To summarize, a three-aspect comparison between AutoDis with existing
representation approaches is presented in Table 1. We can observe that these
methods are either difficult to capture informative knowledge because of low
capacity or require offline expertise feature engineering, which might degrade
the overall performance. Therefore, we propose AutoDis framework. To the best
of our knowledge, it is the first numerical features embedding learning
framework with high model capacity, end-to-end training and unique
representation properties preserved.

Table 1. Comparison of approaches for numerical features representation
learning.

| Model | End-to-End | Unique Representation
---|---|---|---
| Capacity | Training | for Each feature
No Embedding | Limited | $\surd$ | $\times$
Field Embedding | Limited | $\surd$ | $\surd$
Discretization | High | $\times$ | $\times$
AutoDis | High | $\surd$ | $\surd$

##  3\. Methodology

To address the limitations of the existing approaches, we propose AutoDis,
which automatically learns distinctive embeddings for numerical features in an
end-to-end manner. As shown in Figure 3, AutoDis serves as a pluggable
embedding framework for numerical features and is compatible with the existing
deep CTR models.

Figure 3. AutoDis works as a numerical feature embedding learning framework
compatible to the existing deep CTR models.

###  3.1. Framework Overview

To preserve high model capacity, end-to-end training and unique representation
properties, we design three core modules: _meta-embeddings_ , _automatic
discretization_ and _aggregation_ in AutoDis. For the $j$-th numerical field,
AutoDis can learn a unique representation for each numerical feature $x_{j}$
by:

(9) $\small\mathbf{e}_{j}=f(d^{Auto}_{j}(x_{j}),\mathbf{ME}_{j}),$

where $\mathbf{ME}_{j}$ is the meta-embedding matrix for the $j$-th field,
$d^{Auto}_{j}(\cdot)$ is the automatic discretization function and $f(\cdot)$
is the aggregation function. The architecture of AutoDis is presented in
Figure 4. Finally the embeddings of both categorical and numerical features
are concatenated and fed into a deep CTR model for prediction:

(10)
$\small\hat{y}=\textit{CTR}(\underbrace{\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e}_{M}}_{\text{categorical
embeddings}};\underbrace{\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e}_{N}}_{\text{numerical
embeddings}}).$

###  3.2. Meta-Embeddings

A naive solution is to treat numerical features as categorical features and
assign an independent embedding to each numerical feature. Obviously, this
method has serious defects, such as explosively huge parameters and
insufficient training for low-frequency features, which is unaffordable and
inadvisable in practice. On the contrary, Field Embedding method shares a
single embedding among all the feature values within a field for saving
parameters, degrading the model performance due to the low capacity. To
balance the model capacity and complexity, for the $j$-th field, we design a
set of meta-embeddings $\mathbf{ME}_{j}\in\mathbb{R}^{H_{j}\times d}$ shared
by features within this field, where $H_{j}$ is the number of meta-embeddings.
Meta-embeddings are conductive to learn the intra-field shared global
knowledge. Each meta-embedding can be viewed as a sub-space in the latent
space for boosting expressiveness and capacity. By combining these meta-
embeddings, the learned embedding is more informative than that of the Field
Embedding method, so that the high model capacity can be well preserved.
Moreover, the parameters needed are determined by $H_{j}$. Consequently, the
model complexity is highly controllable, making our method scalable.

Figure 4. The architecture of AutoDis framework.

###  3.3. Automatic Discretization

To capture complicated correlations between a numerical feature value and the
designed meta-embeddings, we meticulously propose a differentiable automatic
discretization module $d^{Auto}_{j}(\cdot)$. Thanks to $d^{Auto}_{j}(\cdot)$,
each numerical feature in the $j$-th field is discretized with $H_{j}$ buckets
where each bucket embedding corresponds to a meta-embedding mentioned-above.

Specifically, a two-layer neural network with skip-connection is leveraged to
discretize the feature value $x_{j}$ to $H_{j}$ buckets by:

(11)
$\begin{split}&\mathbf{h}_{j}=\texttt{Leaky\\_ReLU}(\mathbf{w}_{j}x_{j}),\\\
&\mathbf{\widetilde{x}}_{j}=\mathbf{W}_{j}\mathbf{h}_{j}+\alpha\mathbf{h}_{j},\\\
\end{split}$

where $\mathbf{w}_{j}\in\mathbb{R}^{1\times H_{j}}$ and
$\mathbf{W}_{j}\in\mathbb{R}^{H_{j}\times H_{j}}$ are the learnable parameters
of the automatic discretization network for the $j$-th numerical feature
field, activation function is Leaky_ReLU (He et al., 2015) and $\alpha$ is the
control factor of skip-connection. The projecting result is
$\mathbf{\widetilde{x}}_{j}=[\widetilde{x}_{j}^{1},\widetilde{x}_{j}^{2},\dots,\widetilde{x}_{j}^{H_{j}}]$,
where $\widetilde{x}_{j}^{h}$ is the projection output of feature value
$x_{j}$ on the $h$-th bucket. Finally, the correlations between feature value
$x_{j}$ and each of the meta-embeddings $\mathbf{ME}_{j}$ is normalized via
Softmax function, denoted as:

(12)
$\small\widehat{x}_{j}^{h}=\frac{e^{\frac{1}{\tau}\widetilde{x}_{j}^{h}}}{\sum_{l=1}^{H_{j}}e^{\frac{1}{\tau}\widetilde{x}_{j}^{l}}},$

where $\tau$ is the temperature coefficient to control discretization
distribution. Therefore, the discretized result
$\mathbf{\widehat{x}}_{j}\in\mathbb{R}^{H_{j}}$ is obtained through the
automatic discretization function $d^{Auto}_{j}(\cdot)$:

(13)
$\small\mathbf{\widehat{x}}_{j}=d^{Auto}_{j}(x_{j})=[\widehat{x}_{j}^{1},...,\widehat{x}_{j}^{h},...,\widehat{x}_{j}^{H_{j}}].$

The discretized result $\mathbf{\widehat{x}}_{j}$ is a vector where each
element $\widehat{x}_{j}^{h}$ denotes the probability that feature $x_{j}$ is
discretized to the $h$-th bucket, which represents the correlation between
feature value $x_{j}$ and the $h$-th meta-embedding (i.e., bucket embedding).
This discretization method can be understood as soft discretization. Compared
with the hard discretization mentioned in Section 2.3, soft discretization
does not discretize feature values into a single determinate bucket, so that
the _SBD_ and _DBS_ issues can be well solved. Moreover, the differentiable
soft discretization enables our AutoDis to achieve end-to-end training and
optimize with the ultimate goal.

It is worth noting that when temperature coefficient in Eq.(12)
$\tau\rightarrow\infty$, the discretization distribution in Eq.(13) tends to
be uniform and when $\tau\rightarrow 0$, the distribution tends to be one-hot.
Therefore, temperature coefficient $\tau$ plays a significant role in the
automatic discretization distribution. Besides, the feature distribution is
diverse over different fields, so it is of great necessity to learn different
$\tau$ for different features. Specifically, we propose a temperature
coefficient adaptive network, which takes both global field statistics feature
and local input feature into consideration, shown as:

(14)
$\tau_{x_{j}}=\texttt{Sigmoid}(\mathbf{W}_{j}^{2}\texttt{Leaky\\_ReLU}(\mathbf{W}_{j}^{1}[\bar{\mathbf{n}}_{j}\|x_{j}])),$

where $\bar{\mathbf{n}}_{j}$ is the global statistics feature vector of the
$j$-th field including sampled Cumulative Distribution Function (CDF) and mean
value. $x_{j}$ is the local input feature, $\mathbf{W}_{j}^{1}$ and
$\mathbf{W}_{j}^{2}$ are the weight parameters. To guide model training, the
output range of $\tau_{x_{j}}$ is re-scale from (0, 1) to
$(\tau-\epsilon,\tau+\epsilon)$, where $\tau$ is a global hyperparameter.

###  3.4. Aggregation Function

After getting the correlations between a feature value and the meta-
embeddings, the embeddings can be generated by applying aggregation operation
$f(\cdot)$ over the meta-embeddings with the correlations as the selection
probabilities. The following aggregation functions $f(\cdot)$ are considered:

  * •

Max-Pooling selects the most relevant meta-embedding with highest probability
$\widehat{x}_{j}^{h}$:

(15) $\small\mathbf{e}_{j}=\mathbf{ME}_{j}^{k},\text{where }k=\arg
max_{h\in\\{1,2,\dots,H_{j}\\}}\widehat{x}_{j}^{h},$

where $k$ is the index of the most relevant meta-embedding with highest
probability $\widehat{x}_{j}^{h}$ and $\mathbf{ME}_{j}^{k}$ is the $k$-th
meta-embedding in $\mathbf{ME}_{j}$. However, this hard selection strategy
degenerates AutoDis to a hard discretization method, resulting in
aforementioned _SBD_ and _DBS_ problems.

  * •

Top-K-Sum sums up top-K meta-embeddings with highest correlation
$\widehat{x}_{j}^{h}$:

(16) $\small\mathbf{e}_{j}=\sum_{l=1}^{K}\mathbf{ME}_{j}^{k_{l}},\text{where
}k_{l}=\mathop{\arg top}_{l}\
{}_{h\in\\{1,2,\dots,H_{j}\\}}\widehat{x}_{j}^{h},$

where $k_{l}$ is the index of meta-embedding with the $l$-th ($l\in[1,K]$)
largest $\widehat{x}_{j}^{h}$, $\mathbf{ME}_{j}^{k_{l}}$ is the $k_{l}$-th
meta-embedding in $\mathbf{ME}_{j}$. However, Top-K-Sum approach suffers from
two limitations: (1) It can not fundamentally solve _DBS_ problem although the
number of possible generated embeddings increases from $H_{j}$ to
$C_{H_{j}}^{K}$ in comparison with Max-Pooling; (2) The learned embedding
$\mathbf{e}_{j}$ does not consider the correlation values.

  * •

To take advantage of the entire set of meta-embeddings and their correlations
to the feature value sufficiently, a Weighted-Average aggregation function is
proposed as
$\mathbf{e}_{j}=\sum_{h=1}^{H_{j}}\widehat{x}_{j}^{h}\cdot\mathbf{ME}_{j}^{h}$.
With this weighted aggregation strategy, relevant meta-embeddings contribute
more to providing an informative embedding while irrelevant meta-embeddings
are largely neglected. Besides, this strategy ensures that each feature learn
unique representation, and meanwhile, the learned embeddings are _Continuous-
But-Different_ , meaning that the closer the feature values, the more similar
the embeddings are.

###  3.5. Training

AutoDis is trained jointly with the ultimate goal of a concrete deep CTR model
in an end-to-end manner. The loss function is the widely-used LogLoss with a
regularization term:

(17)
$\small\mathcal{L}=-\frac{1}{Q}\sum_{i=1}^{Q}{y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i})}+\lambda\|\Theta\|_{2},$

where $y_{i}$ and $\hat{y}_{i}$ are the ground truth label and estimated value
of the $i$-th instance, respectively. $Q$ is the total number of training
instances and $\lambda$ is the $L_{2}$ regularization weight.
$\Theta=\\{\Theta_{Cat\\_Emb},\Theta_{AutoDis},\\\ \Theta_{CTR}\\}$ are the
parameters of feature embeddings in categorical fields, parameters of meta-
embeddings and automatic discretization (i.e., Eq.(11-14)) in AutoDis, as well
as deep CTR model parameters.

To stabilize the training process, we adopt the feature normalization
technique to scale numerical feature values into $[0,1]$ in the data pre-
processing stage. A numerical feature value $x_{j}$ of the $j$-th numerical
field is normalized as
$x_{j}\leftarrow\frac{x_{j}-x_{j}^{min}}{x_{j}^{max}-x_{j}^{min}}$.

##  4\. Experiments

###  4.1. Experimental Setting

####  4.1.1. Dataset and Evaluation Protocols

To evaluate the effectiveness of our proposed numerical feature embedding
learning framework AutoDis, we conduct extensive experiments on two popular
benchmarks:  _Criteo_ ,  _AutoML_ and one industrial dataset. Table 2
summarizes the statistics of all the three datasets.

  * •

Criteo dataset is published in Criteo Display Advertising Challenge
2013444https://www.kaggle.com/c/criteo-display-ad-challenge and is widely used
in evaluating CTR models, which contains 13 numerical feature fields.

  * •

AutoML dataset is published from AutoML for Lifelong Machine Learning
Challenge in NeurIPS 2018555https://www.4paradigm.com/competition/nips2018,
which contains 23 numerical feature fields.

  * •

Industrial dataset is sampled and collected from a mainstream online
advertising platform, with 41 numerical feature fields.

Table 2. Statistics of evaluation datasets.

Dataset | #Num field | #Cats field | #Instances | Positive Ratio
---|---|---|---|---
Criteo | 13 | 26 | 45.8M | 25.6%
AutoML | 23 | 51 | 4.69M | 5.8%
Industrial | 41 | 44 | 8.75M | 3.3%

To fully evaluate the model performance, we leverage the most commonly-used
evaluation metrics in CTR prediction, namely AUC and LogLoss. All the
experiments are repeated 5 times by changing the random seeds. The two-tailed
unpaired $t$-test is performed to detect significant differences between
AutoDis and the best baseline.

####  4.1.2. Baselines and Hyper-parameters Settings

To demonstrate the effectiveness of the proposed AutoDis, we compare AutoDis
with three categories representation learning methods for numerical features:
(1) No Embedding (YouTube (Covington et al., 2016) DLRM (Naumov et al.,
2019)); (2) Field Embedding (FE (Guo et al., 2017)); (3) Discretization (EDD
(Qu et al., 2019), LD as well as TD, namely DeepGBM (Ke et al., 2019)).

Moreover, to validate the compatibility of AutoDis framework with various
embedding-based deep CTR models, we apply AutoDis to six representative
models: FNN (Zhang et al., 2016), Wide & Deep (Cheng et al., 2016), DeepFM
(Guo et al., 2017), DCN (Wang et al., 2017), IPNN (Qu et al., 2019) and
xDeepFM (Lian et al., 2018).

Our implementation is publicly available in MindSpore. We optimize all the
models with mini-batch Adam (Kingma and Ba, 2014), where the learning rate is
searched from $\\{$10e-6, 5e-5, $\dots$, 10e-2$\\}$. Besides, the embedding
size is set to 80 and 70 in Criteo and AutoML dataset. Specifically, the
hidden layers in deep CTR models are fix to 1024-512-256-128 by default and
the explicit feature interactions in DCN and xDeepFM (namely, CrossNet and
CIN) are set to 3 layers. The weight of $L_{2}$ regularization is tuned in
$\\{$10e-6, 5e-5, $\dots$, 10e-3$\\}$. For AutoDis, the number of meta-
embeddings in each numerical field is 20 and 40 in Criteo and AutoML. The
skip-connection control factor is searched in $\\{0,0.1,\dots,1\\}$ and the
number of neurons in temperature coefficient adaptive network is set to 64.

###  4.2. Overall Performance

In this section, we first compare the performance of different representation
learning methods for numerical features. Then, we evaluate the compatibility
of AutoDis by showing the improvement over six popular deep CTR models.

####  4.2.1. Compared with Other Representation Learning Methods

We perform different numerical feature representation learning methods on
three datasets and choose DeepFM as the deep CTR model. The performance
comparison is presented in Table 3, from which, we have the following
observations:

  * •

The baselines perform inconsistently in different datasets due to discrepant
characteristics of numerical features. Comparatively speaking, AutoDis
outperforms all the baselines on all the datasets consistently by a
significant margin, demonstrating the superiority and robustness to different
numerical features.

  * •

No Embedding and Field Embedding methods perform worse than Discretization and
AutoDis. No Embedding methods achieve better results on industrial dataset
than on public datasets, while Field Embedding method performs the opposite.
These two category methods suffer from low capacity and limited
expressiveness. On the one hand, No Embedding in YouTube is parameter-free and
DLRM leverages a DNN to obtain a mixed representation of all the numerical
fields, ignoring the inter-field distinction, which limits the performance
significantly. On the other hand, for Field Embedding, all the feature values
in the same field share a single embedding and the learned embeddings are the
linear scaling of the shared embedding with the feature values. Such sharing
and linear scaling strategy limits the performance fatally.

  * •

In comparison with the three existing representative Discretization methods
that transform the numerical features into categorical form, AutoDis improves
AUC over the best baseline by 0.17%, 0.23%, 0.22%, respectively. Although
Discretization methods achieve better performance compared with the previous
two categories, they cannot be optimized with the CTR models, and also suffer
from SBD and DBS problems. The heuristic rule-based hard discretization
strategies result in step-wise “unsmooth” embeddings. On the contrary, AutoDis
overcomes these disadvantages by sharing a set of meta-embeddings for each
numerical field and leveraging differentiable soft discretization and
aggregation strategy to learn informative Continuous-But-Different
representations.

Table 3. The overall performance comparison. Boldface denotes the highest
score and underline indicates the best result of the baselines. $\star$
represents significance level $p$-value $<0.05$ of comparing AutoDis with the
best baselines.

Category | Method | Criteo | AutoML | Industrial
---|---|---|---|---
AUC | LogLoss | AUC | LogLoss | AUC | LogLoss
No Embedding | YouTube | 0.8104 | 0.4437 | 0.7382 | 0.1928 | 0.7259 | 0.1369
DLRM | 0.8114 | 0.4404 | 0.7525 | 0.1900 | 0.7253 | 0.1369
Field Embedding | FE | 0.8107 | 0.4412 | 0.7523 | 0.1899 | 0.7248 | 0.1369
Discretization | EDD | 0.8125 | 0.4399 | 0.7545 | 0.1898 | 0.7251 | 0.1371
LD | 0.8138 | 0.4388 | 0.7527 | 0.1899 | 0.7265 | 0.1368
TD | 0.8130 | 0.4392 | 0.7531 | 0.1899 | 0.7262 | 0.1369
AutoDis | 0.8152⋆ | 0.4370⋆ | 0.7562⋆ | 0.1892⋆ | 0.7281⋆ | 0.1366⋆
% Improv. | 0.17% | 0.41% | 0.23% | 0.32% | 0.22% | 0.15%

####  4.2.2. Compatibility with Different CTR Models

AutoDis is a universal framework and can be regarded as plug-in component for
improving various deep CTR models performance. To demonstrate its
compatibility, in this section, we conduct extensive experiments by applying
AutoDis on a series of popular models. From Table 4 we can observe that,
compared with the Field Embedding representation method, AutoDis framework
significantly improves the prediction performance of these models. The
numerical feature discretization and embedding learning procedure is optimized
with the ultimate goal of these CTR models, so that the informative
representations can be obtained and the performance can be improved. In fact,
a small improvement in AUC is likely to yield a significant online CTR
increase and huge profits (Cheng et al., 2016; Qu et al., 2019).

Table 4. Compatibility Study of AutoDis. Field Embedding method is performed
as the baseline to compare with AutoDis.

| Criteo | AutoML | Industrial
---|---|---|---
| AUC | LogLoss | AUC | LogLoss | AUC | LogLoss
FNN | 0.8059 | 0.4456 | 0.7383 | 0.1926 | 0.7271 | 0.1367
FNN-AutoDis |  0.8092⋆ |  0.4426⋆ |  0.7453⋆ |  0.1907⋆ |  0.7289⋆ |  0.1365⋆
Wide&Deep | 0.8097 | 0.4419 | 0.7407 | 0.1923 | 0.7275 | 0.1366
Wide&Deep-AutoDis |  0.8121⋆ |  0.4390⋆ |  0.7445⋆ |  0.1914⋆ |  0.7288⋆ |  0.1365⋆
DeepFM | 0.8107 | 0.4412 | 0.7523 | 0.1899 | 0.7248 | 0.1369
DeepFM-AutoDis |  0.8152⋆ |  0.4370⋆ |  0.7562⋆ |  0.1892⋆ |  0.7281⋆ |  0.1366⋆
DCN | 0.8091 | 0.4425 | 0.7489 | 0.1909 | 0.7262 | 0.1369
DCN-AutoDis |  0.8132⋆ |  0.4395⋆ |  0.7511⋆ |  0.1898⋆ |  0.7293⋆ |  0.1364⋆
IPNN | 0.8101 | 0.4415 | 0.7519 | 0.1896 | 0.7269 | 0.1366
IPNN-AutoDis |  0.8136⋆ |  0.4383⋆ |  0.7541⋆ |  0.1894⋆ |  0.7286⋆ |  0.1365⋆
xDeepFM | 0.8103 | 0.4414 | 0.7508 | 0.1903 | 0.7275 | 0.1367
xDeepFM-AutoDis |  0.8141⋆ |  0.4381⋆ |  0.7529⋆ |  0.1897⋆ |  0.7289⋆ |  0.1365⋆

###  4.3. Deployment & Online A/B Testing

Online experiments were conducted in a mainstream advertising platform for a
weak to verify the superior performance of AutoDis, where millions of daily
active users interact with ads and generate tens of millions of user log
events. In the online serving system, thousands of candidate ads that are most
popular or relevant to users are retrieved from the universal ads pool and
then ranked by a highly-optimized deep CTR model before presenting to users.
To guarantee user experience, the overall latency of the above-mentioned
candidate recall and ranking is required to be within a few milliseconds. The
commercial model is deployed in a cluster, where each node is with 48 core
Intel Xeon CPU E5-2670 (2.30GHZ), 400GB RAM as well as 2 NVIDIA TESLA V100 GPU
cards. For the control group, the numerical features are discretized via a
variety of hybrid manually-designed rules (e.g., EDD, TD, etc.). The
experience group chooses AutoDis to discretize and learn embeddings
automatically for all the numerical features. Deploying AutoDis to an existing
CTR model is convenient and requires nearly no engineering work in the online
serving system.

AutoDis achieves an offline AUC 0.2% improvement, and upgrades the online CTR
and eCPM by 2.1% and 2.7% relative to the control group666The base model in
control group serves the major proportion of users. (statistically
significant), which brings enormous commercial profits. Moreover, with AutoDis
integrated, the existing numerical features no longer require any
discretization rules. Furthermore, introducing more numerical features in the
future will be more efficient, without exploring any handcrafted rules.

###  4.4. Embedding Analysis

To understand the Continuous-But-Different embeddings learned via AutoDis more
deeply with insights, we do further investigation in macro-analysis of
embeddings and micro-analysis of soft discretization, respectively.

####  4.4.1. Macro-Analysis of Embeddings

Figure 5 provides a visualization of the embeddings derived from DeepFM-
AutoDis and DeepFM-EDD in the $3$-rd numerical field of the Criteo dataset. We
random select 250 embeddings and project them onto a two dimensional space
using t-SNE (Van der Maaten and Hinton, 2008). Nodes with similar colors have
similar values. We can observe that AutoDis learns a unique embedding for each
feature. Moreover, similar numerical features (with similar colors) are
represented by close-correlated embeddings (with similar positions in two-
dimensional space), which elaborates the Continuous-But-Different property of
the embeddings. However, EDD learns identical embedding for all the features
within a bucket and completely different embeddings across different buckets,
which leads to step-wise “unsmooth” embeddings and consequently results in
inferior task performance.

(a) AutoDis embeddings

(b) EDD embeddings

Figure 5. Visualization of the t-SNE transformed embeddings derived from
AutoDis and EDD. The points in different colors reflect different feature
values after normalization.

####  4.4.2. Micro-Analysis of Soft Discretization

In this section, we perform some micro-level analysis by investigating the
Softmax distribution during the soft discretization process from DeepFM-
AutoDis. We select an adjacent feature-pair (feature value “1” and “2”) and a
distant feature-pair (feature value “1” and “10”) from the $8$-th numerical
field of the Criteo dataset, and then visualize their discretization
distributions in Figure 6. We can observe that, the adjacent feature-pair have
similar Softmax distributions while distant feature-pair have diverse
distributions. This characteristic is conducive to ensure that similar feature
values can learn similar embeddings by AutoDis, so that the embedding
continuity can be preserved.

(a) Feature value “1” v.s. Feature value “2”

(b) Feature value “1” v.s. Feature value “10”

Figure 6. Soft discretization distribution in the $8$-th field of Criteo.

###  4.5. Numerical Fields Analysis

To evaluate the effect of DeepFM-AutoDis on each numerical field, we select
all the 26 categorical fields as the basic features and add one of the 13
numerical fields at a time cumulatively in the Criteo dataset. Figure 7 plots
the prediction performance w.r.t adding numerical fields in the original order
of the dataset and in a random order. From Figure 7, we have the following
observations: (1) AutoDis improves the performance even if there is only a
single numerical field; (2) AutoDis has a cumulative improvement on multiple
numerical fields; (3) Compared with the existing methods, the performance
improvement achieved by AutoDis is more significant and stable.

(a) The original order

(b) A random order

Figure 7. Improvements by introducing numerical fields one at a time by
different orders.

###  4.6. Model Complexity

To quantitatively analyze the space and time complexity of our proposed
AutoDis, the model parameters and batch inference time with the EDD
discretization method on DeepFM are compared. Table 5 reports the results on
Criteo dataset. We can observe that, compared with EDD, the increased amount
of model parameters by AutoDis are negligible. As online recommendation
services usually have high requirements on latency, the computational cost
during inference is important. We can find that AutoDis achieves acceptable
inference time compared to EDD, demonstrating its computational efficiency.
Furthermore, AutoDis is an end-to-end framework, indicating that there is no
extra efficiency overhead. However, other Discretization methods need
additional overhead, e.g., manual design and data preprocessing in EDD;
another GBDT model training in TD.

Table 5. Model complexity comparison.

Model | Parameters | Batch inference time
---|---|---
DeepFM-EDD |  $\sim$2.009M |  $\sim$3.48ms
DeepFM-AutoDis |  $\sim$2.012M |  $\sim$3.63ms
Relative Ratio | +0.15% | +4.31%

###  4.7. Ablation Study and Hyper-Parameters Sensitivity

####  4.7.1. Aggregation Strategy

To verify the effect of different aggregation strategies in Section 3.4, we
perform ablation study. Figure 8 summarizes the prediction performance of
DeepFM-AutoDis with Max-Pooling, Top-K-Sum and Weighted-Average aggregation
strategies. It can be concluded that Weighted-Average strategy achieves the
best performance. The reason is that, Weighted-Average makes full use of the
meta-embeddings and their corresponding correlations, and overcomes the DBS
and SBD problems completely compared with other strategies.

(a) Criteo

(b) AutoML

Figure 8. The effect of aggregation strategy.

####  4.7.2. Temperature Coefficient

To demonstrate the effectiveness of temperature coefficient adaptive network
mentioned in Section 3.3 Automatic Discretization for generating feature-
specific $\tau_{x_{j}}$, we compare the prediction results with global
temperature coefficient $\tau$. The global temperature coefficients are
searched and summarized in Figure 9. We find that the optimal global
temperature is around 1e-5 and 4e-3 on Criteo and AutoML datasets,
respectively. However, our temperature coefficient adaptive network achieves
best results (red dashed line) because it can adjust temperature coefficient
adaptively for different feature based on the global field statistics feature
and local input feature, obtaining more appropriate discretization
distribution.

(a) Criteo

(b) AutoML

Figure 9. Impact of the temperature coefficient.

####  4.7.3. Number of Meta-Embeddings

We study the number of meta-embedding in this section and vary it in the range
of $\\{10,20,\dots,100\\}$. The performance comparison is presented in Figure
10. It can be observed that increasing the number of meta-embeddings
contributes to improving the performance substantially at the beginning. The
reason lies in involving richer information in meta-embeddings. However, using
too many meta-embeddings not only increases the computational burden, but also
frustrates the performance. Considering both prediction effect and training
efficiency, we set the number to 20 and 40 in Criteo and AutoML datasets,
respectively.

(a) Criteo

(b) AutoML

Figure 10. Impact of the number of meta-embeddings.

##  5\. Related Work

As presented in Figure 1, most existing deep CTR models (Qin et al., 2020;
Cheng et al., 2016; Guo et al., 2017; Zhou et al., 2018; Wang et al., 2020)
follow an Embedding & Feature Interaction (FI) paradigm. Embedding module maps
input features to low-dimensional embeddings. Based on such learnable
embeddings, FI module models implicit/explicit low/high-order feature
interactions by learning nonlinear relationship among features. Below, we will
give a brief introduction of these modules in deep CTR models.

###  5.1. Embedding

As the cornerstone of the deep CTR models, Embedding module has significant
impact on model performance because it takes up majority of the model
parameters. Since the approaches for numerical features have been discussed in
detail in Section 2, we here give a brief introduction about research based on
embedding look-up in recommendation domain. Existing researches mainly focus
on designing adaptable embedding algorithms by allocating variable-length
embeddings or multi-embeddings to different features, such as Mixed-dimension
(Ginart et al., 2019), NIS (Joglekar et al., 2019) and AutoEmb (Zhao et al.,
2020). Another research line delves into the embedding compression (Kang et
al., 2020) to reduce the memory footprint on the web-scale dataset. However,
these methods can only be applied in the look-up table manner for categorical
features or numerical features after disretization. Few studies pay attention
to the embedding learning of numerical features, which is vital in deep CTR
models of the industry.

###  5.2. Feature Interaction

To capture the feature interactions effectively, different architectures are
proposed in the research community of CTR prediction. According to different
combinations of explicit and implicit feature interaction, existing CTR models
can be classified into two categories: stacked structure and parallel
structure. Models with stacked structure first perform explicit feature
interaction modeling over the embeddings and then stack a DNN to extract high-
level implicit feature interactions. Representative models include FNN (Zhang
et al., 2016), IPNN (Qu et al., 2016), PIN (Qu et al., 2019), FiBiNET (Huang
et al., 2019), FGCNN (Liu et al., 2019), AutoGroup (Liu et al., 2020a), DIN
(Zhou et al., 2018) and DIEN (Zhou et al., 2019). On the contrary, models with
parallel structure leverage two parallel networks to capture explicit and
implicit feature interactive signals respectively, and fuse the information in
the output layer. Classic members in this category are Wide & Deep (Cheng et
al., 2016), DeepFM (Guo et al., 2017), AutoFIS (Liu et al., 2020b), DCN (Wang
et al., 2017), xDeepFM (Lian et al., 2018) and AutoInt (Song et al., 2019). In
these models, implicit feature interactions are extracted via a DNN model;
while for explicit feature interaction modeling, Wide & Deep (Cheng et al.,
2016) uses handcrafted cross features, DeepFM and AutoFIS adopt a FM (Rendle,
2010) structure, DCN uses a cross network, xDeepFM employs a Compressed
Interaction Network (CIN) and AutoInt leverages a multi-head self-attention
network.

##  6\. Conclusion

In this paper, we propose AutoDis, a pluggable embedding learning framework
for numerical features in CTR prediction. AutoDis overcomes the shortcomings
of the state-of-the-art processing methods and achieves better performance. It
gains the improvement from these advantages: (1) High model capacity: The
meta-embeddings shared by intra-field features capture global informative
knowledge; (2) End-to-end training: The differentiable automatic
discretization performs soft discretization, so that AutoDis can be optimized
jointly with the ultimate goal of deep CTR models; (3) Unique representation:
The weighted-average aggregation ensures that each feature learns a unique
Continuous-But-Different representation. Extensive experiments on three real-
world datasets and online A/B test demonstrate the effectiveness and
compatibility of AutoDis.

## Acknowledgement

Weinan Zhang is supported by “New Generation of AI 2030” Major Project
(2018AAA0100900) and National Natural Science Foundation of China (62076161,
61632017). This work is also sponsored by Huawei Innovation Research Program.
We thank MindSpore (min, 2020) for the partial support of this work.

## References

  * (1)
  * min (2020) 2020\.  MindSpore.  https://www.mindspore.cn/
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide & Deep Learning for Recommender Systems. In _Proc. Workshop Deep Learning for Recommender Systems_.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep Neural Networks for YouTube Recommendations. In _RecSys_.
  * Ginart et al. (2019) Antonio Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, and James Zou. 2019.  Mixed dimension embeddings with application to memory-efficient recommendation systems.  _arXiv preprint arXiv:1909.11810_ (2019).
  * Grabczewski and Jankowski (2005) Krzysztof Grabczewski and Norbert Jankowski. 2005.  Feature Selection with Decision Tree Criterion. In _HIS_. IEEE Computer Society, 212–217.
  * Gu et al. (2020) Yulong Gu, Zhuoye Ding, Shuaiqiang Wang, Lixin Zou, Yiding Liu, and Dawei Yin. 2020\.  Deep Multifaceted Transformers for Multi-objective Ranking in Large-Scale E-commerce Recommender Systems. In _CIKM_. ACM, 2493–2500.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In _IJCAI_.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In _ICCV_. 1026–1034.
  * He et al. (2014) Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, and Joaquin Quiñonero Candela. 2014\.  Practical Lessons from Predicting Clicks on Ads at Facebook. In _ADKDD_. ACM.
  * Huang et al. (2019) Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. 2019\.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In _RecSys_. ACM, 169–177.
  * Joglekar et al. (2019) Manas R. Joglekar, Cong Li, Jay K. Adams, Pranav Khaitan, and Quoc V. Le. 2019.  Neural Input Search for Large Scale Recommendation Models.  _CoRR_ abs/1907.04471 (2019).  arXiv:1907.04471 http://arxiv.org/abs/1907.04471
  * Juan et al. (2016) Yu-Chin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. 2016.  Field-aware Factorization Machines for CTR Prediction. In _RecSys_.
  * Kang et al. (2020) Wang-Cheng Kang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Ting Chen, Lichan Hong, and Ed H. Chi. 2020\.  Deep Hash Embedding for Large-Vocab Categorical Feature Representations.  _CoRR_ abs/2010.10784 (2020).
  * Ke et al. (2019) Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie-Yan Liu. 2019.  DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks. In _SIGKDD_. ACM, 384–394.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Lian et al. (2018) Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018\.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In _SIGKDD_.
  * Liu et al. (2019) Bin Liu, Ruiming Tang, Yingzhi Chen, Jinkai Yu, Huifeng Guo, and Yuzhou Zhang. 2019\.  Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction. In _WWW_.
  * Liu et al. (2020a) Bin Liu, Niannan Xue, Huifeng Guo, Ruiming Tang, Stefanos Zafeiriou, Xiuqiang He, and Zhenguo Li. 2020a.  AutoGroup: Automatic Feature Grouping for Modelling Explicit High-Order Feature Interactions in CTR Prediction. In _SIGIR_. ACM.
  * Liu et al. (2020b) Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. 2020b.  AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. In _SIGKDD_.
  * McMahan et al. (2013) H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. 2013\.  Ad click prediction: a view from the trenches. In _SIGKDD_. ACM, 1222–1230.
  * Naumov et al. (2019) Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. 2019\.  Deep Learning Recommendation Model for Personalization and Recommendation Systems.  _CoRR_ abs/1906.00091 (2019).
  * Qin et al. (2020) Jiarui Qin, Weinan Zhang, Xin Wu, Jiarui Jin, Yuchen Fang, and Yong Yu. 2020\.  User Behavior Retrieval for Click-Through Rate Prediction. In _SIGIR_. ACM.
  * Qu et al. (2016) Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. 2016.  Product-based neural networks for user response prediction. In _ICDM_. IEEE, 1149–1154.
  * Qu et al. (2019) Yanru Qu, Bohui Fang, Weinan Zhang, Ruiming Tang, Minzhe Niu, Huifeng Guo, Yong Yu, and Xiuqiang He. 2019\.  Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data.  _ACM Trans. Inf. Syst._ 37, 1 (2019), 5:1–5:35.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization Machines. In _Proc. IEEE Int. Conf. Data Mining_.
  * Song et al. (2019) Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.  Autoint: Automatic feature interaction learning via self-attentive neural networks. In _CIKM_. 1161–1170.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2017) Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017\.  Deep & Cross Network for Ad Click Predictions. In _ADKDD_. ACM, 12:1–12:7.
  * Wang et al. (2020) Yichao Wang, Huifeng Guo, Ruiming Tang, Zhirong Liu, and Xiuqiang He. 2020.  A Practical Incremental Method to Train Deep CTR Models.  _CoRR_ abs/2009.02147 (2020).
  * Zhang et al. (2016) Weinan Zhang, Tianming Du, and Jun Wang. 2016.  Deep learning over multi-field categorical data. In _ECIR_. Springer, 45–57.
  * Zhao et al. (2020) Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Jiliang Tang. 2020\.  AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations.  _arXiv preprint arXiv:2002.11252_ (2020).
  * Zhou et al. (2019) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Deep Interest Evolution Network for Click-Through Rate Prediction. In _AAAI_. 5941–5948.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep Interest Network for Click-Through Rate Prediction. In _SIGKDD_. 1059–1068.