# Optimal Superpositions for Particle Detection via Quantum Phase

Eva Kilian  Department of Physics & Astronomy, University College London,
London WC1E 6BT, UK  Marko Toroš  School of Physics & Astronomy, University of
Glasgow, Glasgow, G12 8QQ, UK  P.F. Barker  Sougato Bose  Department of
Physics & Astronomy, University College London, London WC1E 6BT, UK

###### Abstract

Exploiting quantum mechanics for sensing offers unprecedented possibilities.
State of the art proposals for novel quantum sensors often rely on the
creation of large superpositions and generally detect a field. However, what
is the optimal superposition size for detecting an incident particle (or an
incident stream of particles) from a specific direction? This question is
nontrivial as, in general, this incident particle will scatter off with varied
momenta, imparting varied recoils to the sensor, resulting in decoherence
rather than a well defined measurable phase. By considering scattering
interactions of directional particulate environments with a system in a
quantum superposition, we find that there is an “optimal superposition” size
for measuring incoming particles via a relative phase. As a consequence of the
anisotropy of the environment, we observe a novel feature in the limiting
behaviour of the real and imaginary parts of the system’s density matrix,
linking the optimality of the superposition size to the wavelength of the
scatterer.

Quantum sensing with matter-wave interferometers has prompted the development
of a variety of commercial technologies and experiments, offering some of the
most precise sensors. State of the art experiments and proposals encompass
research in the areas of metrology [1], gravimetry [2, 3, 4, 5, 6, 7, 8],
geophysics [9, 10], quantum foundational principles [11, 12, 13, 14, 15, 16],
and sensing for fundamental physics [17, 18, 6, 19, 20, 21, 22, 23]. While
larger quantum objects as a sensor, such as a nanoparticle, to date have
typically been prepared in a near-classical or Gaussian (which can be somewhat
quantum in the sense of being squeezed) initial state [24, 25, 26], the full
potential of quantum mechanics becomes apparent when non-Gaussian states, such
as a quantum superposition of two distinct states, or a state during
interferometry, are utilized. Optimization of the experimental setup and
parameters in such cases in order to extract exquisitely weak signals is of
utmost importance.
The sensing of potentials, such as a linear potential generated by
electrostatic fields, or the gravitational potential near earth, often
necessitates the realisation of quantum superposition states with large
spatial separation $\delta x$ between the superposed components, since the
accumulated phase $\Delta\phi$ increases with increasing separation. For
example, an object of mass $m$ held in a quantum superposition of localized
states separated vertically by $\delta x$ for a time interval $\tau$ near
earth’s surface acquires the celebrated Collela Overhauser Werner (COW) phase
of $\Delta\phi\sim mg\delta x\tau/\hbar$ [27, 3, 4], while the phase due to
the curvature of a proximal mass is $\propto(\delta x)^{2}$ [7]. For a
dynamical monochromatic classical field of wavenumber magnitude $k$, again,
while phase $k\delta x$ is defined modulo $2\pi$, it surely does not harm the
coherence of the superposition to have $\delta x>1/k$[19]. At the other
extreme is the detection of particulate matter interacting with the sensing
system via a coupling term. This is, however, phenomenologically different as
it cannot be correctly approximated by a classical field, unless in the
macroscopic limit of a very large number of particles in a coherent state. In
this limit, the focus has been on detection of the particulate source via the
decoherence of a quantum superposition, manifested as loss of interference or
fringe visibility, as the particles scatter off the sensor mass, typically
imparting random momenta [20, 21, 28]. Thus, the measurement of a coherent
phase is typically not associated with the detection of particles. As far as
current understanding goes, the intuition is that if fields are concerned,
$\delta x$ higher is typically better, and surely not harmful, while for
particles, decoherence would be the prime signature.
In this work, we show that neither of the above intuitions are correct for
particulate matter incident from a given direction: We find that there exists
an optimal superposition size for quantum sensing in scattering experiments,
depending on the characteristics of the environmental source. This arises due
to a competition between a coherent phase contribution and a decoherence
contribution. To illustrate this effect, we consider the basic blueprint
consisting of an incoming particle (the signal) that scatters from a massive
quantum system placed in a spatial superposition (the sensor) as is
schematically represented in Fig. 1 (a). Working within the framework of open
quantum systems, we compute the effects arising from the interaction of the
system with a directional particulate matter environment and discuss in how
far the superposition size impacts the accumulation of the phase. Contrary to
expectations, reading the phase imparted due to scattering in the presence of
decoherence induced by the same scattering may not be optimized at a trivial
point. Aside the above fundamental point, we also present an application in
single photon and atom detection. We conclude by discussing the implications
of our finding for present-day sensing experiments aimed toward capturing
signatures of other/general scattering particles.

##  I Scattering Master Equation

Figure 1: (a) Basic schematics of a particle (pink) scattering from a quantum
object (purple spheres) in a Stern-Gerlach type interferometric experiment,
where the sensor’s spin-state dependent evolution is exploited to split and
recombine the motional wavepacket (purple line). The incoming particle is
sensed through its momentum transfer to the quantum object, which manifests in
the appearence of a relative phase between superposed components. (b) Short-
and long-wavelength regimes of quantum state evolution for Thompson
scattering. Regions shaded indicate the long wavelength regime in blue, the
intermediate region in white and a green shaded region that includes the
short-wavelength regime. The quantity $\lambda$ is the incoming particle’s
wavelength, $\delta x$ is the size of the superposition, Re and Im refer to
the behaviour of the real and imaginary parts of the localization rate’s
contribution to the off-diagonal elements of the system’s density matrix. The
real part (blue line) is typically used to quantifiy decoherence. The
imaginary part (orange) vanishes in an isotropic situation, where the
particles are coming randomly from all directions. When we have a directed
(non-isotropic) source such as a stream of particles however, then the
imaginary part can be used for sensing particles through relative phases,
particularly in regimes where the decoherence contribution (blue line) is
suppressed. The Goldilocks region for the superposition size for optimal
sensing is in the intermediate white-shaded region.

The appearance of the classical from the quantum is the cause of long-standing
debate and a key element of all interpretations of quantum mechanics. The
process of environmental decoherence offers one mechanism for this quantum-to-
classical transition, through which the suppression of quantumness following
measurements performed on a sensing system initially in superposition of
several eigenstates may be described. In this context, the interaction of a
superposed quantum object with gaseous particles and photons will constitute a
measurement, since it involves the transfer of positional information into the
environment. Following the work of Joos and Zeh [29], the authors of [30] have
extended and generalised the mathematical model reflecting the localizing
mechanism under the critical assumption that the scattering of the (weakly
coupled) environment does not significantly disturb the sensor.
Concretely, in a non-relativistic theory, the reduced density matrix of a
system interacting with a particulate environment through scattering [30] is
governed by the following master equation, which includes the corrective
factor derived in [31, 32],

$\displaystyle\frac{d\hat{\rho}_{S}(\mathbb{x},\mathbb{x}^{\prime})}{dt}$
$\displaystyle=\frac{1}{i\hbar}\langle\mathbb{x}|[\hat{H}_{S},\rho_{S}]|\mathbb{x}^{\prime}\rangle$
(1)
$\displaystyle-F(\mathbb{x}-\mathbb{x}^{\prime})\hat{\rho}_{S}(\mathbb{x},\mathbb{x}^{\prime}),$

where $\hat{H}_{S}$ is the Hamiltonian that describes the unitary evolution of
the sensor, $\hat{\rho}_{S}$ the density matrix of the sensing system and

$\displaystyle F(\mathbb{x}-\mathbb{x}^{\prime})$ $\displaystyle=\int
dqn(q)v(q)\int\frac{d\Omega d\Omega^{\prime}}{4\pi}p(\Omega,\Omega^{\prime})$
(2)
$\displaystyle\times(1-e^{i(\mathbb{q}-\mathbb{q}^{\prime})(\mathbb{x}-\mathbb{x}^{\prime})})|f(\mathbb{q},\mathbb{q}^{\prime})|^{2}.$

This result is derived through a careful perturbative treatment of the
scattering interaction. The function $p(\Omega,\Omega^{\prime})$ is a
normalized probability density with incoming (outgoing) scattering angle
$\Omega$ ($\Omega^{\prime}$), and where $p(\Omega,\Omega^{\prime})=1$ is the
typical scenario where scatterers impart momentum from all directions. The
quantities $n(q)$ and $v(q)$ refer to the number density and speed of
particles with wavenumber $q$, the latter of which is closely related to their
wavelength $\lambda=\frac{2\pi}{q}$ and momentum $\mathbb{p}=\hbar\mathbb{q}$,
with $\mathbb{q}$ being the wave vector. The scattering amplitude of the
relevant interaction process is denoted as
$f(\mathbb{q},\mathbb{q}^{\prime})$, with $\mathbb{q}$ and
$\mathbb{q}^{\prime}$ labelling incoming and outgoing wavevectors
respectively. $F(\mathbb{x}-\mathbb{x}^{\prime})$ is the localization rate. If
it is real-valued, the sensing system exhibits a loss of coherence over time,
while an imaginary contribution manifests in the appearance of a phase
$e^{i(\mathbb{q}-\mathbb{q}^{\prime})(\mathbb{x}-\mathbb{x}^{\prime})}$.
Differences in the phases arising at $\mathbb{x}$ and $\mathbb{x}^{\prime}$
can be measured and exploited in sensing with quantum systems.
For an incoming particle of wavelength $\lambda$, it is useful to divide the
system to two regimes of interest to investigate in order to describe the
behaviour of the sensing system.
In the long-wavelength regime, where $\lambda\gg\delta x$, with $\delta
x=|\mathbb{x}-\mathbb{x}^{\prime}|$, the phase term in Eq. (2) becomes small
enough to warrant an approximate treatment of the exponential function by
means of a Taylor-expansion of the argument. Calculating the expansion up to
second order, a quadratic dependency of the localization rate on the
superposition size
$F(\mathbb{x}-\mathbb{x}^{\prime})\propto\frac{1}{2}q^{2}(\mathbb{\hat{n}}-\mathbb{\hat{n}}^{\prime})\cdot(\mathbb{x}-\mathbb{x}^{\prime})^{2}$
is revealed. Assuming isotropy of the environmental source, the linear term in
the expansion averages to zero due to the integration over terms involving the
product of an even and odd function in directions $\mathbb{\hat{n}}$,
$\mathbb{\hat{n}}^{\prime}$.
Master equations of the form (1) can be mapped to equations of the Lindblad-
type in position representation and for $\hat{L}_{k}$ corresponding to a
physical observable, namely that of the position operator
$\hat{L}_{k}=\mathbb{\hat{x}}$, the equation that governs the evolution of the
system can be expressed as

$\displaystyle\frac{d\hat{\rho}_{S}(\mathbb{x},\mathbb{x^{\prime}},t)}{dt}=-\frac{\kappa}{2}(\mathbb{x}-\mathbb{x}^{\prime})^{2}\hat{\rho}_{S}(\mathbb{x},\mathbb{x^{\prime}},t),$
(3)

assuming the individual evolution of the sensing system’s density matrix to be
negligible. The quantity $\kappa$ incorporates information contained in the
localization rate $F(\mathbb{x}-\mathbb{x}^{\prime})$ as written in Eq. (2).
In the short-wavelength regime, the exponential function in Eq. (2) oscillates
rapidly and hence, quickly averages out when performing the integrals. Eq.
(2), expressed in the form of Eq. (3), tends to

$\displaystyle\frac{d\hat{\rho}_{S}(\mathbb{x},\mathbb{x^{\prime}},t)}{dt}=-\frac{\Gamma}{2}(1-\delta_{\mathbb{x},\mathbb{x^{\prime}}})\hat{\rho}_{S}(\mathbb{x},\mathbb{x^{\prime}},t).$
(4)

where, just for a qualitative understanding of the behaviour, we have taken a
discrete set of $\mathbb{x}$ values and
$\delta_{\mathbb{x},\mathbb{x^{\prime}}}$ is a Kronecker delta. The quantity
$\Gamma$ incorporates information contained in the localization rate
$F(\mathbb{x}-\mathbb{x}^{\prime})$. If a given environment is not isotropic
and the scattering particles are instead impinging from a specific direction,
the limiting behaviours reveal the emergence of an optimal superposition size
where the detection of the particle is also from the phase imparted. In order
to demonstrate this effect, we now resort to a combination of analytic
approximations and exact numerics of the reduced system density matrix for
explicit examples.

##  II On the Emergence of an Optimal Superposition Size for Particle
Detection

To illustrate the emergence of an optimal superposition size, let us focus on
two cases of a well-known example from the literature [30], where a special
form of the differential cross-section is taken to be

$|f(\mathbb{q},\mathbb{q}^{\prime})|^{2}=gq^{j}\frac{1}{2}\Bigl{(}1+\left|\frac{\mathbb{q}\mathbb{q}^{\prime}}{q^{2}}\right|^{2}\Bigr{)}.$
(5)

For the expansion order $j=0$ and $g=r_{e}^{2}$ with $r_{e}$ the square of the
classical electron radius, one recovers a differential cross-section for
Thompson scattering, while values of $j=4$ and
$g=a^{6}|\frac{\epsilon-1}{\epsilon+1}|^{2}$ with $a$ as the scatterer’s
radius and $\epsilon$ being the dielectric constant lead to a description of
Rayleigh scattering.
For all practical purposes, let us assume the directional source of particles
to travel along the z-Axis and the superposition to be oriented equally along
z. In the long wavelength-limit, we are again able to expand the small
exponent in Eq. (2) in orders of $\delta x$. This assumption is reflected in
the choice of
$p(\Omega,\Omega^{\prime})=\delta(\theta)\delta(\phi)/\sin{\theta}$ for a
spherical coordinate system. Further, we select the coordinates of our
incoming and outgoing particle wave vectors to be $\mathbb{q}=q(0,0,1)$,
$\mathbb{q}^{\prime}=q(\cos\varphi^{\prime}\sin\theta^{\prime},\sin\varphi^{\prime}\sin\theta^{\prime},\cos\theta^{\prime})$,
where we notably keep the magnitude of the particle’s wave vector and hence
momentum unchanged, which is a valid approximation for negligible momentum
transfers.
If $j=0$, calculating the Taylor expansion up to second order in $\delta x$
and performing the subsequent angular integration results in the following
terms of the localization rate

$\displaystyle F(\mathbb{x}-\mathbb{x}^{\prime})$ $\displaystyle=\int
dqn(q)v(q)g\Bigl{(}-\frac{2}{3}iq\delta x$ (6)
$\displaystyle+\frac{7}{15}q^{2}\delta x^{2}+\mathcal{O}(\delta
x^{3})\Bigr{)}.$

Similarly, we obtain a barely modified equation for $j=4$

$\displaystyle F(\mathbb{x}-\mathbb{x}^{\prime})$ $\displaystyle=\int
dqn(q)v(q)gq^{4}\Bigl{(}-\frac{2}{3}iq\delta x$ (7)
$\displaystyle+\frac{7}{15}q^{2}\delta x^{2}+\mathcal{O}(\delta
x^{3})\Bigr{)}.$

For both the above cases, we observe the emergence of an imaginary linear
(Hamiltonian) term in the master equation, sometimes known as the effective
index of refraction[33, 34], describing the evolution of the reduced density
matrix, reported in Tab. 1, showing the possibility of the detection of the
particles by a phase. Importantly, this behavior appears to be independent of
the exact form of the differential scattering cross section (i.e., it can work
in more generality, even beyond the above two cases), surfacing merely due to
the directional momentum impartment of the incoming scatterers.

| real part | imaginary part
---|---|---
long-wavelength limit | $\propto\delta x^{2}$ | $\propto\delta x$
short-wavelength limit | $\Gamma=\text{constant}$ |  $\rightarrow$ 0
Table 1: Limiting behaviour of the off-diagonal elements of the sensing
system’s density matrix. The behaviour of the real part follows the
theoretically predicted trends as observed already in random uniform
scattering, see Eq. (3) and Eq. (4). However, for a non-uniform environment,
the imaginary part ($\mathcal{O}(2)$) exhibits a drastically different
behaviour and is non-vanishing.

The limit of small wavelengths is more difficult to treat analytically due to
the rapid oscillatory behaviour of the integrand. We can however significantly
reduce the complexity of the problem by assuming the specific geometry
described at the beginning of this section. We then subsequently employ the
Jacobi-Anger expansion, noting the cosine appearing in the exponent. Through
this type of expansion, the trigonometric function in our exponential is
expressed in the basis of its cylindrical harmonics via the relation
$e^{iz\cos{\theta}}=J_{0}+2\sum_{n=1}^{\infty}i^{n}J_{n}(z)\cos{n\theta}$.
This expansion enables us to numerically evaluate the real and the imaginary
parts of the localization rate, as well as the signals in various phase
measurements.
In what follows, in order to simplify our presentation when making plots, we
assume a very narrow distribution of incoming momenta and hence wavenumber
magnitude, namely given by $\delta(q-q_{0})$. Figure 1 (b) displays the trend
of the real and imaginary parts for multiple values of the ratio $\frac{\delta
x}{\lambda}$, where the factor $g=r_{e}^{2}$, effectively scaling the plot’s
y-axis, has been neglected and the momentum integration has been performed
under the assumption of a delta-function in incident momentum, which implies a
specific $q$ scattering. In the small $\delta x/\lambda$ limit, the full
numerics clearly match with the polynomially increasing behaviour of the
imaginary (linear) and real (quadratic) parts as given by Eq.(6). Moreover,
the visible decay of the imaginary phase contribution to zero and the
saturation of the real part also confirm the expected limiting behaviour for
large values of $\frac{\delta x}{\lambda}$ in the short wavelength-limit.
However, the way in which these limits are reached differs drastically from
the case of random uniform scattering, where the exponential contribution
habitually averages out due to rapid oscillations (that is the standard
setting of decoherence due to scatterers [29, 30])
The second example for $j=4$ reveals a similar trend, though the resulting
localization function is again scaled by an additional factor of $q^{4}$ and
the coupling $g$ is dependent on the dielectric constant of the sensing
material. Employing previous calculational methods will therefore lead to the
same qualitative observation, namely the emergence of an optimal "Goldilocks
zone" in relation to the accumulation of phase as depicted in Fig. 1, where
the zone is shaded in white and the phase imparted is optimal for $\delta
x\sim 0.2\lambda$.

##  III Experimental Signature

The field of matter-wave interferometry offers a catalogue of schemes that
enable the extraction of the phase contribution arising due to a scattering
interaction. A popular approach is founded in the Stern-Gerlach interferometry
of spin-mechanical systems [11, 35], where the magnetic manipulation of a
mesoscopic test-mass with an embedded spin, such as levitated diamond with a
nitrogen-vacancy-center, is used. After the initialization of the sensing
system in a center-of-mass (COM) motional state $|c\rangle$ and a
superposition of (its embedded) spin states
$\frac{1}{\sqrt{2}}\bigl{(}|+1\rangle+|-1\rangle\bigr{)}$, the system is
allowed to evolve. This evolution is spin-state dependent and leads to a
spatial splitting of the COM, resulting in a combined quantum state of the
form
$|\psi(t)\rangle=\frac{1}{\sqrt{2}}\bigl{(}|c_{s=+1}(t)\rangle|+1\rangle+|c_{s=-1}(t)\rangle|-1\rangle\bigr{)}$.
The difference $\phi$ in distinct phases arising between the components of the
superposition due to a scattering interaction can be measured upon completion
of the interferometer purely from the spin states. In the ideal case of no
decoherence, it becomes
$\frac{1}{\sqrt{2}}\bigl{(}|+1\rangle+e^{i\phi}|-1\rangle\bigr{)}$. For an
illustration of this scenario, see Fig. 1 (a). In presence of a decoherence
term $A\sim e^{-\Gamma t}$ the density matrix of the spin at the end of the
interferometry becomes

$\displaystyle\rho=\frac{1}{2}\begin{pmatrix}a&Ae^{i\phi}\\\
Ae^{-i\phi}&b\end{pmatrix}.$ (8)

As the elements of the density matrix are going to be estimated from
probabilities of various measurements, it is clear that the exponentially
decaying decoherence term $A$ is less good as an estimator than $Ae^{\pm
i\phi}$ when the term $A\sim 1$ (varying $\phi$, say by controlling $\delta x$
can give oscillations in probabilities raging from 0 to 1). Thus the phase
effect found by us here from a directional source of particles presents an
important method to detect them in comparison to the decoherence they produce.

A method of extracting phase-differences arising between the off-diagonal
components is to apply $\pi/2$-phase and Hadamard gate transformations $S$ and
$H$ to the quantum state, which effectively results in a projection of the
phases onto the diagonal elements of the density matrix

$\displaystyle\rho_{f}$ $\displaystyle=HS\rho S^{\dagger}H$ (9)
$\displaystyle=\frac{1}{4}\begin{pmatrix}a+b+2A\sin{\phi}&a-b+2iA\cos{\phi}\\\
a-b-2iA\cos{\phi}&a+b-2A\sin{\phi}\end{pmatrix}.$

Figure 2: (a) Quantum efficiency $\eta$, as defined in Eq. (11), for Rayleigh
scattering of single photons with $\lambda=1064$ nm on a 0.1 micron-sized
sphere, assuming different spatial photon profiles $A_{p}$ denoting the
transverse area of the incoming photon assuming an estimated flux of $10^{6}$
photons per area $A_{p}$ per second. (b) Measurable signal (depth coloration)
for varying ratios of $\delta x/\lambda$ over a time interval $t=[0,5]$ s,
assuming the illustrative example of Thompson-type scattering with the
exponentiation index for $q^{j}$ in Eq. (5) as $j=0$ and taking the momentum
distribution to be a delta function (definite momentum, say, around
$q=q_{0}=2\pi/\lambda$) and setting the incident flux of the particles to be
such that $\int gn(q)v(q)dq\approx 1$. The orange colored region in the plot
indicates the maximization of the phase signature in the long-wavelength limit
for the lower island, which is also indicated by the non-vanishing first order
contribution in Eq. (6). Depending on the narrowness of the momentum
distribution, the depicted behaviour of the signal should exhibit similar
features in the case of atom scattering.

Subtracting the (diagonals) probabilities

$\displaystyle\rho_{f,11}-\rho_{f,22}=A\sin{\phi},$ (10)

hence relates to the sine of the accumulated phase $\phi$. This experimental
signature is plotted in Fig. 2 (b) for the normalized initial state where
$\rho_{11}=\rho_{22}=\rho_{12}=\rho_{21}=1/2$ and evolved final states with
varying values $\phi=2\pi\delta x/\lambda$, at times $t=[0,5]$s. For the sake
of simplicity and to concentrate on qualitative aspects of the core issue,
i.e., the results of angular averaging over outgoing momenta, we have not
chosen a specific distribution for the number density and speed of the
particles and taken the momentum distribution to be a delta function (definite
wavenumber around $q=q_{0}$) and set the incident flux of the particles to be
such that $\int gn(q)v(q)dq\approx 1$, effectively plotting the contributions
from $F(x-x^{\prime})$ resulting from the angular integrations. Specific
values for these quantities and subsequent integration over the wavenumbers
will lead to a shifted optimal range for the relative superposition size
$\delta x/\lambda$. Although we do not strictly define the notion of
optimality, it should be implicitly clear that values of the relative
superposition size leading to a phase contribution of $\mathcal{O}(1)$ are
considered as such. Benchmarking the range of suitable values for $\delta
x/\lambda$ to, say, $\sin(\phi)=0.95$ is one possible way of defining the
window of optimality. For larger or smaller fluxes, the time needed for
$\mathcal{O}(1)$ will be appropriately scaled.

###  III.1 Single Photon Detection

To give a concrete example of an experimental application, we analyse the
potential benefit of this novel effect for the detection of spatially shaped
single-photons, such as those which may be emitted from a quantum dot source
[36, 37]. With the aim of operating our sensing system as a "click" detector,
we introduce what we term the detection efficiency as follows

$\displaystyle\eta$ $\displaystyle=\langle-|\rho_{in}(t=1\text{s})|-\rangle$
(11)
$\displaystyle=\frac{1}{2}(1-(\rho_{12}(t=1\text{s})+\rho_{21}(t=1\text{s}))).$

The efficiency $\eta$ quantifies the distinguishability of a system initially
prepared in a superposition state $\rho_{in}=|+\rangle\langle+|$ from its
final state. If the scattering of a photon results in a $\pi$ phase shift,
projecting the initial quantum state into the orthogonal state $|-\rangle$,
the efficiency approaches its maximum. Figure 2 (a) shows the crucial
dependency of $\eta$ on the choice of $\delta x/\lambda$ for different
magnitudes of spatial photon profiles $A_{p}$. The single photon transverse
area corresponds to the inverse of $n(q)v(q)\tau$ with $\tau$ set to be 1s and
$n(q)v(q)\tau\sim\frac{10^{6}}{A_{p}}$. Whereas certain choices of the
superposition size will be suitable for operating our sensing system as a
single photon detector, others will result in phase shifts that render the
system insensitive to the signal, implying that the superposition size can be
used for wavelength selection. It can be seen in Fig. 2(a) that sensing will
be possible within the peaked regions of relatively broad bands.

###  III.2 Detection of Single Atomic Ions

Rutherford scattering of an atomic ion on a nanoparticle of radius $1\mu$m is
described via the differential cross section

$\displaystyle|f(\mathbb{q},\mathbb{q}^{\prime})|^{2}=\frac{m^{2}}{\hbar^{4}q^{4}}\frac{(ZZ^{\prime}e^{2})^{2}}{(4\pi\epsilon_{0})^{2}}\Bigl{(}1+\left|\frac{\mathbb{q}\mathbb{q}^{\prime}}{q^{2}}\right|^{2}\Bigr{)},$
(12)

where $Ze$ and $Z^{\prime}e$ are the charges of the atom and nanoparticle,
$m=10^{-25}$kg is the mass of a heavy atom and $\epsilon_{0}$ the vacuum
permittivity. We assume an atomic momentum determined via the relation
$k_{B}T/2=\hbar^{2}\mathbb{q}^{2}/2m$, at a temperature $T=100$K. Expressing
the momentum and inserting into Eq. (12) yields

$\displaystyle|f(\mathbb{q},\mathbb{q}^{\prime})|^{2}=10^{-14}Z^{\prime
2}\Bigl{(}1+\left|\frac{\mathbb{q}\mathbb{q}^{\prime}}{q^{2}}\right|^{2}\Bigr{)},$
(13)

where the atom’s charge number $Z=1$ has been assumed. We hence propose that a
micron sized nanoparticle is able to detect about one atomic ion interacting
with it per second if the atom has a wavefunction cross section on the order
of $10^{4}$nm2 for small values of $Z^{\prime}$ and the atomic flux
(corresponding to $n(q)v(q)$) is on the order of $\sim
10^{-4}\text{s}^{-1}\text{nm}^{-2}$).

##  IV Summary

Our observations are of critical relevance to experiments where a stream of
incoming particles scattering from a superposition has a unique or preferred
direction and the environment cannot be treated in the fashion of an isotropic
bath. Conversely, we expect that a similar observation can be made for a
superposed object that is not held in place, but instead propagating with a
given velocity with respect to the environment, such as a crystal with
horizontal velocity in a motional superposition state moving through a gas of
particles in the lab frame.
Considering two limiting regimes for the wavelength of an incoming scatterer
interacting with a quantum sensor, we have numerically shown that the
imaginary contribution arising due to the interaction is, in specific
scenarios, non-vanishing and we have provided a strong argument that the
behaviour is likely universal. The described relative phase may be used for
the detection of weak environmental signatures.
Moreover, we have observed the emergence of an optimal parameter-choice for
the superposition size $\delta x$ when it comes to measuring special types of
particulate environments and identified a Goldilocks-zone. We have also shown
that the superposition sensor is capable of detecting single photons and
single charged atoms with appropriate settings.
Our findings will doubtlessly result in improvements of state-of-the-art
quantum sensors and may be utilized to enhance signals which are typically
difficult to capture. Several emergent experiments and technologies [3, 6,
21], especially in the context of quantum gravimetry [38], rely on the
acceleration of the quantum mechanical sensor. Any such setup will be
influenced by non-isotropic sources, such as the directional scattering of a
stream of particles, interacting with the sensing apparatus. We therefore want
to emphasize the importance of the choice of the superposition size in
relation to phase contributions arising through these directional effects.

###### Acknowledgements.

EK would like to acknowledge support from the Engineering and Physical
Sciences Research Council (grant number EP/L015242/1) and an EPSRC Doctoral
Prize Fellowship Grant (EP/W524335/1). MT would like to acknowledge funding by
the Leverhulme Trust (RPG-2020-197). SB would like to acknowledge EPSRC grants
(EP/N031105/1 and EP/S000267/1) and grant ST/W006227/1. PFB acknowledges
funding from the EPSRC Grant No. EP/W029626/1.

## References

  * Pezzè  _et al._ [2018] L. Pezzè, A. Smerzi, M. K. Oberthaler, R. Schmied, and P. Treutlein, Rev. Mod. Phys. 90, 035005 (2018), publisher: American Physical Society.
  * Asenbaum  _et al._ [2017] P. Asenbaum, C. Overstreet, T. Kovachy, D. D. Brown, J. M. Hogan, and M. A. Kasevich, Physical review letters 118, 183602 (2017).
  * Scala  _et al._ [2013] M. Scala, M. S. Kim, G. W. Morley, P. F. Barker, and S. Bose, Physical Review Letters 111, 180403 (2013).
  * Wan  _et al._ [2016] C. Wan, M. Scala, G. W. Morley, A. A. Rahman, H. Ulbricht, J. Bateman, P. F. Barker, S. Bose, and M. S. Kim, Phys. Rev. Lett. 117, 143003 (2016), publisher: American Physical Society.
  * Qvarfort  _et al._ [2018] S. Qvarfort, A. Serafini, P. F. Barker, and S. Bose, Nature Communications 9, 3690 (2018).
  * Marshman  _et al._ [2020] R. J. Marshman, A. Mazumdar, G. W. Morley, P. F. Barker, S. Hoekstra, and S. Bose, New Journal of Physics 22, 083012 (2020), publisher: IOP Publishing.
  * Toroš  _et al._ [2021] M. Toroš, T. W. Van De Kamp, R. J. Marshman, M. Kim, A. Mazumdar, and S. Bose, Physical Review Research 3, 023178 (2021).
  * Wu  _et al._ [2023] M.-Z. Wu, M. Toroš, S. Bose, and A. Mazumdar, Physical Review D 107, 104053 (2023).
  * Bidel  _et al._ [2018] Y. Bidel, N. Zahzam, C. Blanchard, A. Bonnin, M. Cadoret, A. Bresson, D. Rouxel, and M. F. Lequentrec-Lalancette, Nature Communications 9, 627 (2018).
  * Stray  _et al._ [2022] B. Stray, A. Lamb, A. Kaushik, J. Vovrosh, A. Rodgers, J. Winch, F. Hayati, D. Boddice, A. Stabrawa, A. Niggebaum, M. Langlois, Y.-H. Lien, S. Lellouch, S. Roshanmanesh, K. Ridley, G. de Villiers, G. Brown, T. Cross, G. Tuckwell, A. Faramarzi, N. Metje, K. Bongs, and M. Holynski, Nature 602, 590 (2022).
  * Bose  _et al._ [2017] S. Bose, A. Mazumdar, G. W. Morley, H. Ulbricht, M. Toroš, M. Paternostro, A. A. Geraci, P. F. Barker, M. S. Kim, and G. Milburn, Phys. Rev. Lett. 119, 240401 (2017).
  * Marletto and Vedral [2017] C. Marletto and V. Vedral, Phys. Rev. Lett. 119, 240402 (2017).
  * Bose  _et al._ [2023] S. Bose, A. Mazumdar, M. Schut, and M. Toroš, Entropy 25, 448 (2023).
  * Biswas  _et al._ [2022] D. Biswas, S. Bose, A. Mazumdar, and M. Toroš, arXiv preprint arXiv:2209.09273 (2022).
  * Margalit  _et al._ [2021] Y. Margalit, O. Dobkowski, Z. Zhou, O. Amit, Y. Japha, S. Moukouri, D. Rohrlich, A. Mazumdar, S. Bose, C. Henkel, _et al._ , Science advances 7, eabg2879 (2021).
  * Yi  _et al._ [2021] B. Yi, U. Sinha, D. Home, A. Mazumdar, and S. Bose, arXiv preprint arXiv:2106.11906 (2021).
  * Dimopoulos  _et al._ [2009] S. Dimopoulos, P. W. Graham, J. M. Hogan, M. A. Kasevich, and S. Rajendran, Physics Letters B 678, 37 (2009).
  * Chaibi  _et al._ [2016] W. Chaibi, R. Geiger, B. Canuel, A. Bertoldi, A. Landragin, and P. Bouyer, Physical Review D 93, 021101 (2016).
  * Geraci and Derevianko [2016] A. A. Geraci and A. Derevianko, Physical review letters 117, 261301 (2016).
  * Riedel [2013] C. J. Riedel, Phys. Rev. D 88, 116005 (2013), publisher: American Physical Society.
  * Bateman  _et al._ [2015] J. Bateman, I. McHardy, A. Merle, T. R. Morris, and H. Ulbricht, Scientific reports 5, 1 (2015).
  * Riedel and Yavin [2017] C. J. Riedel and I. Yavin, Physical Review D 96, 023007 (2017).
  * Kilian  _et al._ [2023] E. Kilian, M. Toroš, F. F. Deppisch, R. Saakyan, and S. Bose, Phys. Rev. Res. 5, 023012 (2023).
  * Afek  _et al._ [2022] G. Afek, D. Carney, and D. C. Moore, Phys. Rev. Lett. 128, 101301 (2022).
  * Burd  _et al._ [2019] S. C. Burd, R. Srinivas, J. J. Bollinger, A. C. Wilson, D. J. Wineland, D. Leibfried, D. H. Slichter, and D. T. C. Allcock, Science 364, 1163 (2019).
  * and others [2021] and others, Quantum Science and Technology 6, 024002 (2021).
  * Colella  _et al._ [1975] R. Colella, A. W. Overhauser, and S. A. Werner, Physical Review Letters 34, 1472 (1975).
  * Fragolino  _et al._ [2023] P. Fragolino, M. Schut, M. Toroš, S. Bose, and A. Mazumdar, (2023), arXiv:2307.07001 [quant-ph] .
  * Joos and Zeh [1985] E. Joos and H. D. Zeh, Zeitschrift für Physik B Condensed Matter 59, 223 (1985).
  * Gallis and Fleming [1990] M. R. Gallis and G. N. Fleming, Physical Review A 42, 38 (1990).
  * Hornberger and Sipe [2003] K. Hornberger and J. E. Sipe, Phys. Rev. A 68, 012105 (2003).
  * Hornberger  _et al._ [2003] K. Hornberger, S. Uttenthaler, B. Brezger, L. Hackermüller, M. Arndt, and A. Zeilinger, Phys. Rev. Lett. 90, 160401 (2003).
  * Schmiedmayer  _et al._ [1995] J. Schmiedmayer, M. S. Chapman, C. R. Ekstrom, T. D. Hammond, S. Wehinger, and D. E. Pritchard, Phys. Rev. Lett. 74, 1043 (1995).
  * Vacchini and Hornberger [2009] B. Vacchini and K. Hornberger, Physics Reports 478, 71 (2009).
  * Marshman  _et al._ [2022] R. J. Marshman, A. Mazumdar, R. Folman, and S. Bose, Phys. Rev. Research 4, 023087 (2022), publisher: American Physical Society.
  * Rakher  _et al._ [2011] M. T. Rakher, L. Ma, M. Davanço, O. Slattery, X. Tang, and K. Srinivasan, Physical Review Letters 107, 083602 (2011).
  * Hanschke  _et al._ [2018] L. Hanschke, K. A. Fischer, S. Appel, D. Lukin, J. Wierzbowski, S. Sun, R. Trivedi, J. Vučković, J. J. Finley, and K. Müller, npj Quantum Information 4, 43 (2018).
  * Rademacher  _et al._ [2020] M. Rademacher, J. Millen, and Y. L. Li, Advanced Optical Technologies 9, 227 (2020).


# MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by
Instance-Guided Mask

Zhiqiang Wang, Qingyun She, Junlin Zhang  Sina Weibo CorpBeijingChina
roky2813@sina.com,qingyun˙she<EMAIL_ADDRESS>
###### Abstract.

Click-Through Rate(CTR) estimation has become one of the most fundamental
tasks in many real-world applications and it’s important for ranking models to
effectively capture complex high-order features. Shallow feed-forward network
is widely used in many state-of-the-art DNN models such as FNN, DeepFM and
xDeepFM to implicitly capture high-order feature interactions. However, some
research has proved that addictive feature interaction, particular feed-
forward neural networks, is inefficient in capturing common feature
interaction. To resolve this problem, we introduce specific multiplicative
operation into DNN ranking system by proposing instance-guided mask which
performs element-wise product both on the feature embedding and feed-forward
layers guided by input instance. We also turn the feed-forward layer in DNN
model into a mixture of addictive and multiplicative feature interactions by
proposing MaskBlock in this paper. MaskBlock combines the layer normalization,
instance-guided mask, and feed-forward layer and it is a basic building block
to be used to design new ranking model under various configurations. The model
consisting of MaskBlock is called MaskNet in this paper and two new MaskNet
models are proposed to show the effectiveness of MaskBlock as basic building
block for composing high performance ranking systems. The experiment results
on three real-world datasets demonstrate that our proposed MaskNet models
outperform state-of-the-art models such as DeepFM and xDeepFM significantly,
which implies MaskBlock is an effective basic building unit for composing new
high performance ranking systems.

††copyright: acmcopyright††conference: DLP-KDD 2021; August 15, 2021;
Singapore

##  1\. Introduction

Click-through rate (CTR) prediction is to predict the probability of a user
clicking on the recommended items. It plays important role in personalized
advertising and recommender systems. Many models have been proposed to resolve
this problem such as Logistic Regression (LR) (McMahan et al., 2013),
Polynomial-2 (Poly2) (Rendle, 2010), tree-based models (He et al., 2014),
tensor-based models (Koren et al., 2009), Bayesian models (Graepel et al.,
2010), and Field-aware Factorization Machines (FFMs) (Juan et al., 2016). In
recent years, employing DNNs for CTR estimation has also been a research trend
in this field and some deep learning based models have been introduced such as
Factorization-Machine Supported Neural Networks(FNN)(Zhang et al., 2016),
Attentional Factorization Machine (AFM)(Cheng et al., 2016), wide &
deep(W&D)(Xiao et al., 2017), DeepFM(Guo et al., 2017), xDeepFM(Lian et al.,
2018) etc.

Feature interaction is critical for CTR tasks and it’s important for ranking
model to effectively capture these complex features. Most DNN ranking models
such as FNN , W&D, DeepFM and xDeepFM use the shallow MLP layers to model
high-order interactions in an implicit way and it’s an important component in
current state-of-the-art ranking systems.

However, Alex Beutel et.al (Beutel et al., 2018) have proved that addictive
feature interaction, particular feed-forward neural networks, is inefficient
in capturing common feature crosses. They proposed a simple but effective
approach named ”latent cross” which is a kind of multiplicative interactions
between the context embedding and the neural network hidden states in RNN
model. Recently, Rendle et.al’s work (Rendle et al., 2020) also shows that a
carefully configured dot product baseline largely outperforms the MLP layer in
collaborative filtering. While a MLP can in theory approximate any function,
they show that it is non-trivial to learn a dot product with an MLP and
learning a dot product with high accuracy for a decently large embedding
dimension requires a large model capacity as well as many training data. Their
work also proves the inefficiency of MLP layer’s ability to model complex
feature interactions.

Inspired by ”latent cross”(Beutel et al., 2018) and Rendle’s work (Rendle et
al., 2020), we care about the following question: Can we improve the DNN
ranking systems by introducing specific multiplicative operation into it to
make it efficiently capture complex feature interactions?

In order to overcome the problem of inefficiency of feed-forward layer to
capture complex feature cross, we introduce a special kind of multiplicative
operation into DNN ranking system in this paper. First, we propose an
instance-guided mask performing element-wise production on the feature
embedding and feed-forward layer. The instance-guided mask utilizes the global
information collected from input instance to dynamically highlight the
informative elements in feature embedding and hidden layer in a unified
manner. There are two main advantages for adopting the instance-guided mask:
firstly, the element-wise product between the mask and hidden layer or feature
embedding layer brings multiplicative operation into DNN ranking system in
unified way to more efficiently capture complex feature interaction. Secondly,
it’s a kind of fine-gained bit-wise attention guided by input instance which
can both weaken the influence of noise in feature embedding and MLP layers
while highlight the informative signals in DNN ranking systems.

By combining instance-guided mask, a following feed-forward layer and layer
normalization, MaskBlock is proposed by us to turn the commonly used feed-
forward layer into a mixture of addictive and multiplicative feature
interactions. The instance-guided mask introduces multiplicative interactions
and the following feed-forward hidden layer aggregates the masked information
in order to better capture the important feature interactions. While the layer
normalization can ease optimization of the network.

MaskBlock can be regarded as a basic building block to design new ranking
models under some kinds of configuration. The model consisting of MaskBlock is
called MaskNet in this paper and two new MaskNet models are proposed to show
the effectiveness of MaskBlock as basic building block for composing high
performance ranking systems.

The contributions of our work are summarized as follows:

  1. (1)

In this work, we propose an instance-guided mask performing element-wise
product both on the feature embedding and feed-forward layers in DNN models.
The global context information contained in the instance-guided mask is
dynamically incorporated into the feature embedding and feed-forward layer to
highlight the important elements.

  2. (2)

We propose a basic building block named MaskBlock which consists of three key
components: instance-guided mask, a following feed-forward hidden layer and
layer normalization module. In this way, we turn the widely used feed-forward
layer of a standard DNN model into a mixture of addictive and multiplicative
feature interactions.

  3. (3)

We also propose a new ranking framework named MaskNet to compose new ranking
system by utilizing MaskBlock as basic building unit. To be more specific, the
serial MaskNet model and parallel MaskNet model are designed based on the
MaskBlock in this paper. The serial rank model stacks MaskBlock block by block
while the parallel rank model puts many MaskBlocks in parallel on a sharing
feature embedding layer.

  4. (4)

Extensive experiments are conduct on three real-world datasets and the
experiment results demonstrate that our proposed two MaskNet models outperform
state-of-the-art models significantly. The results imply MaskBlock indeed
enhance DNN model’s ability of capturing complex feature interactions through
introducing multiplicative operation into DNN models by instance-guided mask.

The rest of this paper is organized as follows. Section 2 introduces some
related works which are relevant with our proposed model. We introduce our
proposed models in detail in Section 3. The experimental results on three real
world datasets are presented and discussed in Section 4. Section 5 concludes
our work in this paper.

##  2\. Related Work

###  2.1. State-Of-The-Art CTR Models

Many deep learning based CTR models have been proposed in recent years and it
is the key factor for most of these neural network based models to effectively
model the feature interactions.

Factorization-Machine Supported Neural Networks (FNN)(Zhang et al., 2016) is a
feed-forward neural network using FM to pre-train the embedding layer. Wide &
Deep Learning(Xiao et al., 2017) jointly trains wide linear models and deep
neural networks to combine the benefits of memorization and generalization for
recommender systems. However, expertise feature engineering is still needed on
the input to the wide part of Wide & Deep model. To alleviate manual efforts
in feature engineering, DeepFM(Guo et al., 2017) replaces the wide part of
Wide & Deep model with FM and shares the feature embedding between the FM and
deep component.

While most DNN ranking models process high-order feature interactions by MLP
layers in implicit way, some works explicitly introduce high-order feature
interactions by sub-network. Deep & Cross Network (DCN)(Wang et al., 2017)
efficiently captures feature interactions of bounded degrees in an explicit
fashion. Similarly, eXtreme Deep Factorization Machine (xDeepFM) (Lian et al.,
2018) also models the low-order and high-order feature interactions in an
explicit way by proposing a novel Compressed Interaction Network (CIN) part.
AutoInt(Song et al., 2019) uses a multi-head self-attentive neural network to
explicitly model the feature interactions in the low-dimensional space.
FiBiNET(Huang et al., 2019) can dynamically learn feature importance via the
Squeeze-Excitation network (SENET) mechanism and feature interactions via
bilinear function.

###  2.2. Feature-Wise Mask Or Gating

Feature-wise mask or gating has been explored widely in vision (Hu et al.,
2018; Srivastava et al., 2015), natural language processing (Dauphin et al.,
2017) and recommendation system(Ma et al., 2018; Ma et al., 2019). For
examples, Highway Networks (Srivastava et al., 2015) utilize feature gating to
ease gradient-based training of very deep networks. Squeeze-and-Excitation
Networks(Hu et al., 2018) recalibrate feature responses by explicitly
multiplying each channel with learned sigmoidal mask values. Dauphin et
al.(Dauphin et al., 2017) proposed gated linear unit (GLU) to utilize it to
control what information should be propagated for predicting the next word in
the language modeling task. Gating or mask mechanisms are also adopted in
recommendation systems. Ma et al. (Ma et al., 2018) propose a novel multi-task
learning approach, Multi-gate Mixture-of-Experts (MMoE), which explicitly
learns to model task relationships from data. Ma et al.(Ma et al., 2019)
propose a hierarchical gating network (HGN) to capture both the long-term and
short-term user interests. The feature gating and instance gating modules in
HGN select what item features can be passed to the downstream layers from the
feature and instance levels, respectively.

###  2.3. Normalization

Normalization techniques have been recognized as very effective components in
deep learning. Many normalization approaches have been proposed with the two
most popular ones being BatchNorm (Ioffe and Szegedy, 2015) and LayerNorm (Ba
et al., 2016) . Batch Normalization (Batch Norm or BN)(Ioffe and Szegedy,
2015) normalizes the features by the mean and variance computed within a mini-
batch. Another example is layer normalization (Layer Norm or LN)(Ba et al.,
2016) which was proposed to ease optimization of recurrent neural networks.
Statistics of layer normalization are not computed across the $N$ samples in a
mini-batch but are estimated in a layer-wise manner for each sample
independently. Normalization methods have shown success in accelerating the
training of deep networks.

##  3\. Our Proposed Model

In this section, we first describe the feature embedding layer. Then the
details of the instance-guided mask, MaskBlock and MaskNet structure we
proposed will be introduced. Finally the log loss as a loss function is
introduced.

###  3.1. Embedding Layer

The input data of CTR tasks usually consists of sparse and dense features and
the sparse features are mostly categorical type. Such features are encoded as
one-hot vectors which often lead to excessively high-dimensional feature
spaces for large vocabularies. The common solution to this problem is to
introduce the embedding layer. Generally, the sparse input can be formulated
as:

(1) $x=[x_{1},x_{2},...,x_{f}]$

where $f$ denotes the number of fields, and $x_{i}\in\mathbb{R}^{n}$ denotes a
one-hot vector for a categorical field with $n$ features and
$x_{i}\in\mathbb{R}^{n}$ is vector with only one value for a numerical field.
We can obtain feature embedding $e_{i}$ for one-hot vector $x_{i}$ via:

(2) $e_{i}=W_{e}x_{i}$

where $W_{e}\in\mathbb{R}^{k\times n}$ is the embedding matrix of $n$ features
and $k$ is the dimension of field embedding. The numerical feature $x_{j}$ can
also be converted into the same low-dimensional space by:

(3) $e_{j}=V_{j}x_{j}$

where $V_{j}\in\mathbb{R}^{k}$ is the corresponding field embedding with size
$k$.

Through the aforementioned method, an embedding layer is applied upon the raw
feature input to compress it to a low dimensional, dense real-value vector.
The result of embedding layer is a wide concatenated vector:

(4)
$\mathbf{V}_{emb}=concat(\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{i},...,\mathbf{e}_{f})$

where $f$ denotes the number of fields, and $\mathbf{e}_{i}\in\mathbb{R}^{k}$
denotes the embedding of one field. Although the feature lengths of input
instances can be various, their embedding are of the same length $f\times k$,
where $k$ is the dimension of field embedding.

We use instance-guided mask to introduce the multiplicative operation into DNN
ranking system and here the so-called ”instance” means the feature embedding
layer of current input instance in the following part of this paper.

###  3.2. Instance-Guided Mask

We utilize the global information collected from input instance by instance-
guided mask to dynamically highlight the informative elements in feature
embedding and feed-forward layer. For feature embedding, the mask lays stress
on the key elements with more information to effectively represent this
feature. For the neurons in hidden layer, the mask helps those important
feature interactions to stand out by considering the contextual information in
the input instance. In addition to this advantage, the instance-guided mask
also introduces the multiplicative operation into DNN ranking system to
capture complex feature cross more efficiently.

Figure 1. Neural Structure of Instance-Guided Mask

As depicted in Figure 1, two fully connected (FC) layers with identity
function are used in instance-guided mask. Notice that the input of instance-
guided mask is always from the input instance, that is to say, the feature
embedding layer.

The first FC layer is called ”aggregation layer” and it is a relatively wider
layer compared with the second FC layer in order to better collect the global
contextual information in input instance. The aggregation layer has parameters
$W_{d1}$ and here $d$ denotes the $d$-th mask. For feature embedding and
different MLP layers, we adopt different instance-guided mask owning its
parameters to learn to capture various information for each layer from input
instance.

The second FC layer named ”projection layer” reduces dimensionality to the
same size as feature embedding layer $V_{emb}$ or hidden layer $V_{hidden}$
with parameters $W_{d2}$, Formally,

(5) $V_{mask}=W_{d2}(Relu(W_{d1}V_{emb}+\beta_{d1}))+\beta_{d2}$

where $V_{emb}\in\mathbb{R}^{m=f\times k}$ refers to the embedding layer of
input instance, $W_{d1}\in\mathbb{R}^{t\times m}$ and
$W_{d2}\in\mathbb{R}^{z\times t}$ are parameters for instance-guided mask, $t$
and $z$ respectively denotes the neural number of aggregation layer and
projection layer, $f$ denotes the number of fields and $k$ is the dimension of
field embedding. $\beta_{d1}\in\mathbb{R}^{t\times m}$ and
$\beta_{d2}\in\mathbb{R}^{z\times t}$ are learned bias of the two FC layers.
Notice here that the aggregation layer is usually wider than the projection
layer because the size of the projection layer is required to be equal to the
size of feature embedding layer or MLP layer. So we define the size $r=t/z$ as
reduction ratio which is a hyper-parameter to control the ratio of neuron
numbers of two layers.

Element-wise product is used in this work to incorporate the global contextual
information aggregated by instance-guided mask into feature embedding or
hidden layer as following:

(6)
$\begin{split}\mathbf{V}_{maskedEMB}&=\mathbf{V}_{mask}\odot\mathbf{V}_{emb}\\\
\mathbf{V}_{maskedHID}&=\mathbf{V}_{mask}\odot\mathbf{V}_{hidden}\end{split}$

where $\mathbf{V}_{emb}$ denotes embedding layer and $\mathbf{V}_{hidden}$
means the feed-forward layer in DNN model, $\odot$ means the element-wise
production between two vectors as follows:

(7) $V_{i}\odot V_{j}=[V_{i1}\cdot V_{j1},V_{i2}\cdot V_{j2},...,V_{iu}\cdot
V_{ju}]$

here $u$ is the size of vector $V_{i}$ and $V_{j}$

The instance-guided mask can be regarded as a special kind of bit-wise
attention or gating mechanism which uses the global context information
contained in input instance to guide the parameter optimization during
training. The bigger value in $V_{mask}$ implies that the model dynamically
identifies an important element in feature embedding or hidden layer. It is
used to boost the element in vector $V_{emb}$ or $V_{hidden}$. On the
contrary, small value in $V_{mask}$ will suppress the uninformative elements
or even noise by decreasing the values in the corresponding vector $V_{emb}$
or $V_{hidden}$.

The two main advantages in adopting the instance-guided mask are: firstly, the
element-wise product between the mask and hidden layer or feature embedding
layer brings multiplicative operation into DNN ranking system in unified way
to more efficiently capture complex feature interaction. Secondly, this kind
of fine-gained bit-wise attention guided by input instance can both weaken the
influence of noise in feature embedding and MLP layers while highlight the
informative signals in DNN ranking systems.

###  3.3. MaskBlock

To overcome the problem of the inefficiency of feed-forward layer to capture
complex feature interaction in DNN models, we propose a basic building block
named MaskBlock for DNN ranking systems in this work, as shown in Figure 2 and
Figure 3. The proposed MaskBlock consists of three key components: layer
normalization module ,instance-guided mask, and a feed-forward hidden layer.
The layer normalization can ease optimization of the network. The instance-
guided mask introduces multiplicative interactions for feed-forward layer of a
standard DNN model and feed-forward hidden layer aggregate the masked
information in order to better capture the important feature interactions. In
this way, we turn the widely used feed-forward layer of a standard DNN model
into a mixture of addictive and multiplicative feature interactions.

First, we briefly review the formulation of LayerNorm.

Layer Normalization:
In general, normalization aims to ensure that signals have zero mean and unit
variance as they propagate through a network to reduce ”covariate shift”
(Ioffe and Szegedy, 2015). As an example, layer normalization (Layer Norm or
LN)(Ba et al., 2016) was proposed to ease optimization of recurrent neural
networks. Specifically, let $x=(x_{1},x_{2},...,x_{H})$ denotes the vector
representation of an input of size $H$ to normalization layers. LayerNorm re-
centers and re-scales input $\mathbf{x}$ as

(8) $\begin{split}&\mathbf{h}=\mathbf{g}\odot N(\mathbf{x})+\mathbf{b},\quad
N(\mathbf{x})=\frac{\mathbf{x}-\mu}{\delta},\\\
&\mu=\frac{1}{H}\sum^{H}_{i=1}x_{i},\quad\delta=\sqrt{\frac{1}{H}\sum^{H}_{i=1}(x_{i}-\mu)^{2}}\end{split}$

where $h$ is the output of a LayerNorm layer. $\odot$ is an element-wise
production operation. $\mu$ and $\delta$ are the mean and standard deviation
of input. Bias $\mathbf{b}$ and gain $\mathbf{g}$ are parameters with the same
dimension $H$.

As one of the key component in MaskBlock, layer normalization can be used on
both feature embedding and feed- forward layer. For the feature embedding
layer, we regard each feature’s embedding as a layer to compute the mean,
standard deviation, bias and gain of LN as follows:

(9)
$LN\\_EMB(\mathbf{V}_{emb})=concate\left(LN(\mathbf{e}_{1}),LN(\mathbf{e}_{2}),...,LN(\mathbf{e}_{i}),...,LN(\mathbf{e}_{f})\right)$

As for the feed-forward layer in DNN model, the statistics of $LN$ are
estimated among neurons contained in the corresponding hidden layer as
follows:

(10) $LN\\_HID(\mathbf{V}_{hidden})=ReLU(LN(\mathbf{W}_{i}\mathbf{X}))$

where $\mathbf{X}\in\mathbb{R}^{t}$ refers to the input of feed-forward layer,
$\mathbf{W}_{i}\in\mathbb{R}^{m\times t}$ are parameters for the layer, $t$
and $m$ respectively denotes the size of input layer and neural number of
feed-forward layer. Notice that we have two places to put normalization
operation on the MLP: one place is before non-linear operation and another
place is after non-linear operation. We find the performance of the
normalization before non-linear consistently outperforms that of the
normalization after non-linear operation. So all the normalization used in MLP
part is put before non-linear operation in our paper as formula (4) shows.

Figure 2. MaskBlock on Feature Embedding

MaskBlock on Feature Embedding:
We propose MaskBlock by combining the three key elements: layer normalization,
instance-guided mask and a following feed-forward layer. MaskBlock can be
stacked to form deeper network. According to the different input of each
MaskBlock, we have two kinds of MaskBlocks: MaskBlock on feature embedding and
MaskBlock on Maskblock. We will firstly introduce the MaskBlock on feature
embedding as depicted in Figure 2 in this subsection.

The feature embedding $\mathbf{V}_{emb}$ is the only input for MaskBlock on
feature embedding. After the layer normalization operation on embedding
$\mathbf{V}_{emb}$. MaskBlock utilizes instance-guided mask to highlight the
informative elements in $\mathbf{V}_{emb}$ by element-wise product, Formally,

(11) $\begin{split}\mathbf{V}_{maskedEMB}=\mathbf{V}_{mask}\odot
LN\\_EMB(\mathbf{V}_{emb})\end{split}$

where $\odot$ means an element-wise production between the instance-guided
mask and the normalized vector $LN_{E}MB(\mathbf{V}_{emb})$,
$\mathbf{V}_{maskedEMB}$ denote the masked feature embedding. Notice that the
input of instance-guided mask $\mathbf{V}_{mask}$ is also the feature
embedding $V_{emb}$.

Figure 3. MaskBlock on MaskBlock  Figure 4. Structure of Serial Model and
Parallel Model

We introduce a feed-forward hidden layer and a following layer normalization
operation in MaskBlock to better aggregate the masked information by a
normalized non-linear transformation. The output of MaskBlock can be
calculated as follows:

(12) $\begin{split}\mathbf{V}_{output}&=LN\\_HID(W_{i}V_{maskdEMB})\\\
&=ReLU(LN(\mathbf{W}_{i}(\mathbf{V}_{mask}\odot
LN\\_EMB(\mathbf{V}_{emb}))))\end{split}$

where $\mathbf{W}_{i}\in\mathbb{R}^{q\times n}$ are parameters of the feed-
forward layer in the $i$-th MaskBlock, $n$ denotes the size of
$\mathbf{V}_{maskedEMB}$ and $q$ means the size of neural number of the feed-
forward layer.

The instance-guided mask introduces the element-wise product into feature
embedding as a fine-grained attention while normalization both on feature
embedding and hidden layer eases the network optimization. These key
components in MaskBlock help the feed-forward layer capture complex feature
cross more efficiently.

MaskBlock on MaskBlock:
In this subsection, we will introduce MaskBlock on MaskBlock as depicted in
Figure 3. There are two different inputs for this MaskBlock: feature embedding
$V_{emb}$ and the output $V_{output}^{p}$ of the previous MaskBlock. The input
of instance-guided mask for this kind of MaskBlock is always the feature
embedding $V_{emb}$. MaskBlock utilizes instance-guided mask to highlight the
important feature interactions in previous MaskBlock’s output $V_{output}^{p}$
by element-wise product, Formally,

(13) $V_{maskedHID}=V_{mask}\odot V_{output}^{p}$

where $\odot$ means an element-wise production between the instance-guided
mask $V_{mask}$ and the previous MaskBlock’s output $V_{output}^{p}$,
$V_{maskedHID}$ denote the masked hidden layer.

In order to better capture the important feature interactions, another feed-
forward hidden layer and a following layer normalization are introduced in
MaskBlock . In this way, we turn the widely used feed-forward layer of a
standard DNN model into a mixture of addictive and multiplicative feature
interactions to avoid the ineffectiveness of those addictive feature cross
models. The output of MaskBlock can be calculated as follows:

(14) $\begin{split}\mathbf{V}_{output}&=LN\\_HID(W_{i}V_{maskdHID})\\\
&=ReLU(LN(\mathbf{W}_{i}(\mathbf{V}_{mask}\odot\mathbf{V}_{output}^{p})))\end{split}$

where $W_{i}\in\mathbb{R}^{q\times n}$ are parameters of the feed-forward
layer in the $i$-th MaskBlock, $n$ denotes the size of
$\mathbf{V}_{maskedHID}$ and $q$ means the size of neural number of the feed-
forward layer.

###  3.4. MaskNet

Based on the MaskBlock, various new ranking models can be designed according
to different configurations. The rank model consisting of MaskBlock is called
MaskNet in this work. We also propose two MaskNet models by utilizing the
MaskBlock as the basic building block.

Serial MaskNet:
We can stack one MaskBlock after another to build the ranking system , as
shown by the left model in Figure 4. The first block is a MaskBlock on feature
embedding and all other blocks are MaskBlock on Maskblock to form a deeper
network. The prediction layer is put on the final MaskBlock’s output vector.
We call MaskNet under this serial configuration as SerMaskNet in our paper.
All inputs of instance-guided mask in every MaskBlock come from the feature
embedding layer $\mathbf{V}_{emb}$ and this makes the serial MaskNet model
look like a RNN model with sharing input at each time step.

Parallel MaskNet:
We propose another MaskNet by placing several MaskBlocks on feature embedding
in parallel on a sharing feature embedding layer, as depicted by the right
model in Figure 4. The input of each block is only the shared feature
embedding $\mathbf{V}_{emb}$ under this configuration. We can regard this
ranking model as a mixture of multiple experts just as MMoE(Ma et al., 2018)
does. Each MaskBlock pays attention to specific kind of important features or
feature interactions. We collect the information of each expert by
concatenating the output of each MaskBlock as follows:

(15)
$\mathbf{V}_{merge}=concate(\mathbf{V}^{1}_{output},\mathbf{V}^{2}_{output},...,\mathbf{V}^{i}_{output},...,\mathbf{V}^{u}_{output})$

where $\mathbf{V}^{i}_{output}\in\mathbb{R}^{q}$ is the output of the $i$-th
MaskBlock and $q$ means size of neural number of feed-forward layer in
MaskBlock, $u$ is the MaskBlock number.

To further merge the feature interactions captured by each expert, multiple
feed-forward layers are stacked on the concatenation information
$\mathbf{V}_{merge}$. Let $\mathbf{H}_{0}=\mathbf{V}_{merge}$ denotes the
output of the concatenation layer, then $\mathbf{H}_{0}$ is fed into the deep
neural network and the feed forward process is:

(16) $\mathbf{H}_{l}=ReLU(\mathbf{W}_{l}\mathbf{H}_{l-1}+\beta_{l})$

where $l$ is the depth and ReLU is the activation function.
$\mathbf{W}_{t},\beta_{t},\mathbf{H}_{l}$ are the model weight, bias and
output of the $l$-th layer. The prediction layer is put on the last layer of
multiple feed-forward networks. We call this version MaskNet as ”ParaMaskNet”
in the following part of this paper.

###  3.5. Prediction Layer

To summarize, we give the overall formulation of our proposed model’ s output
as:

(17) $\hat{y}=\delta(w_{0}+\sum^{n}_{i=1}w_{i}x_{i})$

where $\hat{y}\in(0,1)$ is the predicted value of CTR, $\delta$ is the sigmoid
function, $n$ is the size of the last MaskBlock’s output(SerMaskNet) or feed-
forward layer(ParaMaskNet), $x_{i}$ is the bit value of feed-forward layer and
$w_{i}$ is the learned weight for each bit value.

For binary classifications, the loss function is the log loss:

(18)
$\mathcal{L}=-\frac{1}{N}\sum^{N}_{i=1}y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i})$

where $N$ is the total number of training instances, $y_{i}$ is the ground
truth of $i$-th instance and $\hat{y}_{i}$ is the predicted CTR. The
optimization process is to minimize the following objective function:

(19) $\mathfrak{L}=\mathcal{L}+\lambda\|\Theta\|$

where $\lambda$ denotes the regularization term and $\Theta$ denotes the set
of parameters, including those in feature embedding matrix, instance-guided
mask matrix, feed-forward layer in MaskBlock, and prediction part.

##  4\. Experimental Results

In this section, we evaluate the proposed approaches on three real-world
datasets and conduct detailed ablation studies to answer the following
research questions:

  * •

RQ1 Does the proposed MaskNet model based on the MaskBlock perform better than
existing state-of-the-art deep learning based CTR models?

  * •

RQ2 What are the influences of various components in the MaskBlock
architecture? Is each component necessary to build an effective ranking
system?

  * •

RQ3 How does the hyper-parameter of networks influence the performance of our
proposed two MaskNet models?

  * •

RQ4 Does instance-guided mask highlight the important elements in feature
embedding and feed-forward layers according to the input instance?

In the following, we will first describe the experimental settings, followed
by answering the above research questions.

###  4.1. Experiment Setup

####  4.1.1. Datasets

The following three data sets are used in our experiments:

  1. (1)

Criteo111Criteo http://labs.criteo.com/downloads/download-terabyte-click-logs/
Dataset: As a very famous public real world display ad dataset with each ad
display information and corresponding user click feedback, Criteo data set is
widely used in many CTR model evaluation. There are $26$ anonymous categorical
fields and $13$ continuous feature fields in Criteo data set.

  2. (2)

Malware222Malware https://www.kaggle.com/c/microsoft-malware-prediction
Dataset: Malware is a dataset from Kaggle competitions published in the
Microsoft Malware prediction. The goal of this competition is to predict a
Windows machine’s probability of getting infected. The malware prediction task
can be formulated as a binary classification problem like a typical CTR
estimation task does.

  3. (3)

Avazu333Avazu http://www.kaggle.com/c/avazu-ctr-prediction Dataset: The Avazu
dataset consists of several days of ad click- through data which is ordered
chronologically. For each click data, there are $23$ fields which indicate
elements of a single ad impression.

We randomly split instances by $8:1:1$ for training , validation and test
while Table 1 lists the statistics of the evaluation datasets.

Table 1. Statistics of the evaluation datasets Datasets | #Instances | #fields | #features
---|---|---|---
Criteo | 45M | 39 | 30M
Avazu | 40.43M | 23 | 9.5M
Malware | 8.92M | 82 | 0.97M

####  4.1.2. Evaluation Metrics

AUC (Area Under ROC) is used in our experiments as the evaluation metric.
AUC’s upper bound is $1$ and larger value indicates a better performance.

RelaImp is also as work (Xie et al., 2020) does to measure the relative AUC
improvements over the corresponding baseline model as another evaluation
metric. Since AUC is $0.5$ from a random strategy, we can remove the constant
part of the AUC score and formalize the RelaImp as:

(20) $RelaImp=\frac{AUC(Measured\ Model)-0.5}{AUC(Base\ Model)-0.5}-1$

Table 2. Overall performance (AUC) of different models on three datasets(feature embedding size=10,our proposed two models both have 3 MaskBlocks with same default settings.) | Criteo | Malware | Avazu
---|---|---|---
| AUC | RelaImp | AUC | RelaImp | AUC | RelaImp
FM | 0.7895 | 0.00% | 0.7166 | 0.00% | 0.7785 | 0.00%
DNN | 0.8054 | +5.35% | 0.7246 | +3.70% | 0.7820 | +1.26%
DeepFM | 0.8057 | +5.46% | 0.7293 | +5.86% | 0.7833 | +1.72%
DCN | 0.8058 | +5.49% | 0.7300 | +6.19% | 0.7830 | +1.62%
xDeepFM | 0.8064 | +5.70% | 0.7310 | +6.65% | 0.7841 | +2.01%
AutoInt | 0.8051 | +5.39% | 0.7282 | +5.36% | 0.7824 | +1.40%
SerMaskNet | 0.8119 | +7.74% | 0.7413 | +11.40% | 0.7877 | +3.30%
ParaMaskNet | 0.8124 | +7.91% | 0.7410 | +11.27% | 0.7872 | +3.12%

####  4.1.3. Models for Comparisons

We compare the performance of the following CTR estimation models with our
proposed approaches: FM, DNN, DeepFM, Deep&Cross Network(DCN), xDeepFM and
AutoInt Model, all of which are discussed in Section 2. FM is considered as
the base model in evaluation.

####  4.1.4. Implementation Details

We implement all the models with Tensorflow in our experiments. For
optimization method, we use the Adam with a mini-batch size of $1024$ and a
learning rate is set to $0.0001$. Focusing on neural networks structures in
our paper, we make the dimension of field embedding for all models to be a
fixed value of $10$. For models with DNN part, the depth of hidden layers is
set to $3$, the number of neurons per layer is $400$, all activation function
is ReLU. For default settings in MaskBlock, the reduction ratio of instance-
guided mask is set to $2$. We conduct our experiments with $2$ Tesla $K40$
GPUs.

###  4.2. Performance Comparison (RQ1)

The overall performances of different models on three evaluation datasets are
show in the Table 2. From the experimental results, we can see that:

  1. (1)

Both the serial model and parallel model achieve better performance on all
three datasets and obtains significant improvements over the state-of-the-art
methods. It can boost the accuracy over the baseline FM by $3.12\%$ to
$11.40\%$, baseline DeepFM by $1.55\%$ to $5.23\%$, as well as xDeepFM
baseline by $1.27\%$ to $4.46\%$. We also conduct a significance test to
verify that our proposed models outperforms baselines with the significance
level $\alpha=0.01$.

Though maskNet model lacks similar module such as CIN in xDeepFM to explicitly
capture high-order feature interaction, it still achieves better performance
because of the existence of MaskBlock. The experiment results imply that
MaskBlock indeed enhance DNN Model’s ability of capturing complex feature
interactions through introducing multiplicative operation into DNN models by
instance-guided mask on the normalized feature embedding and feed-forward
layer.

  2. (2)

As for the comparison of the serial model and parallel model, the experimental
results show comparable performance on three evaluation datasets. It
explicitly proves that MaskBlock is an effective basic building unit for
composing various high performance ranking systems.

###  4.3. Ablation Study of MaskBlock (RQ2)

In order to better understand the impact of each component in MaskBlock, we
perform ablation experiments over key components of MaskBlock by only removing
one of them to observe the performance change, including mask module, layer
normalization(LN) and feed-forward network(FFN). Table 3 shows the results of
our two full version MaskNet models and its variants removing only one
component.

From the results in Table 3, we can see that removing either instance-guided
mask or layer normalization will decrease model’s performance and this implies
that both the instance-guided mask and layer normalization are necessary
components in MaskBlock for its effectiveness. As for the feed-forward layer
in MaskBlock, its effect on serial model or parallel model shows difference.
The Serial model’s performance dramatically degrades while it seems do no harm
to parallel model if we remove the feed-forward layer in MaskBlock. We deem
this implies that the feed-forward layer in MaskBlock is important for merging
the feature interaction information after instance-guided mask. For parallel
model, the multiple feed-forward layers above parallel MaskBlocks have similar
function as feed-forward layer in MaskBlock does and this may produce
performance difference between two models when we remove this component.

Table 3. Overall performance (AUC) of MaskNet models removing different component in MaskBlock on Criteo dataset(feature embedding size=$10$, MaskNet model has $3$ MaskBlocks.) Model Name | SerMaskNet | ParaMaskNet
---|---|---
Full | 0.8119 | 0.8124
-w/o Mask | 0.8090 | 0.8093
-w/o LN | 0.8106 | 0.8103
-w/o FFN | 0.8085 | 0.8122

###  4.4. Hyper-Parameter Study(RQ3)

In the following part of the paper, we study the impacts of hyper-parameters
on two MaskNet models, including $1)$ the number of feature embedding size;
$2)$ the number of MaskBlock; and $3)$ the reduction ratio in instance-guided
mask module. The experiments are conducted on Criteo dataset via changing one
hyper-parameter while holding the other settings. The hyper-parameter
experiments show similar trend in other two datasets.

##### Number of Feature Embedding Size.

The results in Table 4 show the impact of the number of feature embedding size
on model performance. It can be observed that the performances of both models
increase when embedding size increases at the beginning. However, model
performance degrades when the embedding size is set greater than $50$ for
SerMaskNet model and $30$ for ParaMaskNet model. The experimental results tell
us the models benefit from larger feature embedding size.

Table 4. Overall performance (AUC) of different feature embedding size of MaskNet Models on Criteo dataset(the number of MaskBlock is $3$) Embedding Size | 10 | 20 | 30 | 50 | 80
---|---|---|---|---|---
SerMaskNet | 0.8119 | 0.8123 | 0.8121 | 0.8125 | 0.8121
ParaMaskNet | 0.8124 | 0.8128 | 0.8131 | 0.8129 | 0.8129

##### Number of MaskBlock.

For understanding the influence of the number of MaskBlock on model’s
performance, we conduct experiments to stack MaskBlock from $1$ to $9$ blocks
for both MaskNet models. The experimental results are listed in the Table 5.
For SerMaskNet model, the performance increases with more blocks at the
beginning until the number is set greater than $5$. While the performance
slowly increases when we continually add more MaskBlock into ParaMaskNet
model. This may indicates that more experts boost the ParaMaskNet model’s
performance though it’s more time consuming.

Table 5. Overall performance (AUC) of different numbers of MaskBlocks in MaskNet model on Criteo dataset(embedding size$=10$) Block Number | 1 | 3 | 5 | 7 | 9
---|---|---|---|---|---
SerMaskNet | 0.8110 | 0.8119 | 0.8126 | 0.8117 | 0.8115
ParaMaskNet | 0.8113 | 0.8124 | 0.8127 | 0.8128 | 0.8132
Table 6. Overall performance (AUC) of different size of Hidden Layer in Mask Module of MastBlock on Criteo dataset.(embedding size$=10$, number of MaskBlock is $3$) Reduction ratio | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
SerMaskNet | 0.8118 | 0.8119 | 0.8120 | 0.8117 | 0.8119
ParaMaskNet | 0.8124 | 0.8124 | 0.8122 | 0.8122 | 0.8124

##### Reduction Ratio in Instance-Guided Mask.

In order to explore the influence of the reduction ratio in instance-guided
mask, We conduct some experiments to adjust the reduction ratio from $1$ to
$5$ by changing the size of aggregation layer. Experimental results are shown
in Table 6 and we can observe that various reduction ratio has little
influence on model’s performance. This indicates that we can adopt small
reduction ratio in aggregation layer in real life applications for saving the
computation resources.

###  4.5. Instance-Guided Mask Study(RQ4)

As discussed in Section in 3.2, instance-guided mask can be regarded as a
special kind of bit-wise attention mechanism to highlight important
information based on the current input instance. We can utilize instance-
guided mask to boost the informative elements and suppress the uninformative
elements or even noise in feature embedding and feed-forward layer.

To verify this, we design the following experiment: After training the
SerMaskNet with $3$ blocks, we input different instances into the model and
observe the outputs of corresponding instance-guided masks.

Figure 5. Distribution of Mask Values

Figure 6. Mask Values of Two Expamples

Firstly, we randomly sample $100000$ different instances from Criteo dataset
and observe the distributions of the produced values by instance-guided mask
from different blocks. Figure 5 shows the result. We can see that the
distribution of mask values follow normal distribution. Over 50% of the mask
values are small number near zero and only little fraction of the mask value
is a relatively larger number. This implies that large fraction of signals in
feature embedding and feed-forward layer is uninformative or even noise which
is suppressed by the small mask values. However, there is some informative
information boosted by larger mask values through instance-guided mask.

Secondly, we randomly sample two instances and compare the difference of the
produced values by instance-guided mask. The results are shown in Figure 6. We
can see that: As for the mask values for feature embedding, different input
instances lead the mask to pay attention to various areas. The mask outputs of
instance A pay more attention to the first few features and the mask values of
instance B focus on some bits of other features. We can observe the similar
trend in the mask values in feed-forward layer. This indicates the input
instance indeed guide the mask to pay attention to the different part of the
feature embedding and feed-forward layer.

##  5\. Conclusion

In this paper, we introduce multiplicative operation into DNN ranking system
by proposing instance-guided mask which performs element-wise product both on
the feature embedding and feed-forward layers. We also turn the feed-forward
layer in DNN model into a mixture of addictive and multiplicative feature
interactions by proposing MaskBlock by bombing the layer normalization,
instance-guided mask, and feed-forward layer. MaskBlock is a basic building
block to be used to design new ranking model. We also propose two specific
MaskNet models based on the MaskBlock. The experiment results on three real-
world datasets demonstrate that our proposed models outperform state-of-the-
art models such as DeepFM and xDeepFM significantly.

## References

  * (1)
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016\.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ (2016).
  * Beutel et al. (2018) Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H Chi. 2018.  Latent cross: Making use of context in recurrent recommender systems. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 46–54.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. ACM, 7–10.
  * Dauphin et al. (2017) Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017.  Language modeling with gated convolutional networks. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 933–941.
  * Graepel et al. (2010) Thore Graepel, Joaquin Quinonero Candela, Thomas Borchert, and Ralf Herbrich. 2010\.  Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine. Omnipress.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_ (2017).
  * He et al. (2014) Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014\.  Practical lessons from predicting clicks on ads at facebook. In _Proceedings of the Eighth International Workshop on Data Mining for Online Advertising_. ACM, 1–9.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7132–7141.
  * Huang et al. (2019) Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. 2019\.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 169–177.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _arXiv preprint arXiv:1502.03167_ (2015).
  * Juan et al. (2016) Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. 2016\.  Field-aware factorization machines for CTR prediction. In _Proceedings of the 10th ACM Conference on Recommender Systems_. ACM, 43–50.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 8 (2009), 30–37.
  * Lian et al. (2018) Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018\.  xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, 1754–1763.
  * Ma et al. (2019) Chen Ma, Peng Kang, and Xue Liu. 2019.  Hierarchical gating networks for sequential recommendation. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 825–833.
  * Ma et al. (2018) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018\.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1930–1939.
  * McMahan et al. (2013) H. Brendan McMahan, Gary Holt, D. Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, and et al. 2013.  Ad Click Prediction: A View from the Trenches. In _Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ _(KDD ’13)_. Association for Computing Machinery, New York, NY, USA, 1222–1230.  https://doi.org/10.1145/2487575.2488200
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization machines. In _2010 IEEE International Conference on Data Mining_. IEEE, 995–1000.
  * Rendle et al. (2020) Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020.  Neural Collaborative Filtering vs. Matrix Factorization Revisited.  _arXiv preprint arXiv:2005.09683_ (2020).
  * Song et al. (2019) Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.  Autoint: Automatic feature interaction learning via self-attentive neural networks. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1161–1170.
  * Srivastava et al. (2015) Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015.  Highway networks.  _arXiv preprint arXiv:1505.00387_ (2015).
  * Wang et al. (2017) Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017\.  Deep & cross network for ad click predictions. In _Proceedings of the ADKDD’17_. ACM, 12\.
  * Xiao et al. (2017) Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. 2017\.  Attentional factorization machines: Learning the weight of feature interactions via attention networks.  _arXiv preprint arXiv:1708.04617_ (2017).
  * Xie et al. (2020) Ruobing Xie, Cheng Ling, Yalong Wang, Rui Wang, Feng Xia, and Leyu Lin. 2020\.  Deep Feedback Network for Recommendation. 2491–2497.  https://doi.org/10.24963/ijcai.2020/345
  * Zhang et al. (2016) Weinan Zhang, Tianming Du, and Jun Wang. 2016.  Deep learning over multi-field categorical data. In _European conference on information retrieval_. Springer, 45–57.