# Electron-phonon mediated spin-flip as driving mechanism for ultrafast
magnetization dynamics in 3$d$ ferromagnets

Theodor Griepe  Dahlem Center for Complex Quantum Systems and Fachbereich
Physik, Freie Universität Berlin, 14195 Berlin, Germany  Instituto de Ciencia
de Materiales de Madrid, CSIC, Cantoblanco, 28049 Madrid, Spain  Unai Atxitia
<EMAIL_ADDRESS>Dahlem Center for Complex Quantum Systems and Fachbereich
Physik, Freie Universität Berlin, 14195 Berlin, Germany  Instituto de Ciencia
de Materiales de Madrid, CSIC, Cantoblanco, 28049 Madrid, Spain

###### Abstract

Despite intense experimental effort, theoretical proposals and modeling
approaches, a lack of consensus exists about the intrinsic mechanisms driving
ultrafast magnetization dynamics in 3$d$ ferromagnets. In this work, we find
evidence of electron-phonon mediated spin-flip as the driving mechanism for
the ultrafast magnetization dynamics in all three 3$d$ ferromagnets; nickel,
iron and cobalt. We use a microscopic three temperature model with parameters
calculated from first-principles, which has been validated by direct
comparison to the electron and lattice dynamics extracted from previous
experiments. By direct comparison to the experimentally measured magnetization
dynamics for different laser fluence, we determine the spin-flip probability
of each material. In contrast to previous findings but in agreement to ab-
initio predictions, we find that relatively small values of the spin-flip
probability enable ultrafast demagnetization in all three 3$d$ ferromagnets.

††preprint: APS/123-QED

The discovery of femtosecond spin dynamics in the 3$d$ ferromagnet nickel
opened the door for magnetic non-volatile data processing on ultrafast time
scales[1]. Since then, ultrafast demagnetization has been demonstrated in
other transition metals [2, 3], rare-earth metals [4, 5, 6, 7] and their
alloys [8], semiconductors [9] and insulators [10]. The fundamental mechanisms
behind laser-induced ultrafast magnetization dynamics, such as the nature of
spin-flip scattering or the role of spin-polarized transport, are still object
of debate [11, 12, 13, 14, 15, 16]. In the first works, Koopmans et al. [11,
12] proposed electron-phonon scattering mediated spin-flip as the main
mechanism driving the ultrafast magnetization dynamics in 3$d$ ferromagnets.
This idea was supported by fitting the so-called microscopic three temperature
model (M3TM) to ultrafast demagnetization traces in nickel and cobalt [12].
Later on, it was argued that other mechanisms should significantly contribute
to the demagnetization since the ab-initio estimated values for the spin-flip
rates were too small to produce the observed demagnetization [14, 17]. The
main drawback of the M3TM as originally used by Koopmans et al. [12] is the
large number of fitting parameters. This has been partially solved in
subsequent works in nickel [18, 15] where the temperature dependence of the
electron, lattice and spin specific heats at equilibrium were directly taken
from experiments[19]. By doing so, the spin-flip rates necessary to fit the
experimental results reduced and approached to ab-initio estimations. Besides
electron-phonon spin-flip scattering, other alternative mechanisms have been
put forward, such as Coulomb scattering of Elliot-Yafet type [3] and electron-
magnon scattering [2, 20]. The latter aims to describe angular momentum
conservation due to an increase of the orbital momenta upon creation of a
magnon. However, experimental studies suggest that angular momentum
dissipation to orbital momenta cannot explain the magnetization quenching in
iron and cobalt on this timescale and support the claim of the lattice to
absorb excess angular momentum[21, 22, 23, 24]. Superdiffusive spin currents
have been also proposed as dominant mechanism driving the ultrafast
demagnetization, however this process is only efficient when magnets are
interfaced by nonmagnetic metals [13, 25]. Whereas in thin films on insulating
substrates, superdiffusive transport plays no significant role in the
demagnetization process of 3$d$ ferromagnets [15]. We note that recent ab-
initio time-dependent density functional theory have shown to be accurate in
the early stages of the optically induced demagnetization process before
lattice vibrations set in [26]. However, experimental evidence suggests that
angular momentum from the magnetic system is transferred to the lattice [24,
23]. These processes are not within reach of ab-initio methods, and,
therefore, their investigation needs to rely on semi-phenomenological models.

Femtosecond-resolved experimental techniques that measure the magnetization
dynamics as well as the electronic and lattice dynamics are nowadays
accessible. The electron energy distribution has been measured in nickel using
time- and angle-resolved photo-emission spectroscopy (tr-ARPES) [27, 28]. It
has been demonstrated that only a few tens of femtoseconds after the laser
hits the sample, the electron system thermalizes into a Fermi-Dirac
distribution with a well-defined electron temperature. The lattice energy
dynamics has been measured using time-resolved electron diffraction in the
three 3$d$ ferromagnets [29, 30, 24]. It has been demonstrated that the so-
called two- temperature model (2TM) describes well the measured electron and
lattice temperature dynamics for parameters calculated from first principles,
and by including the energy exchange between the spin and electron systems
within an atomistic spin model framework [29, 30]. However, the lack of a
theoretical model describing the magnetization dynamics questions the full
validity of the atomistic spin model.

In this work, we demonstrate that the M3TM is able to quantitatively describe
the ultrafast dynamics in 3$d$ ferromagnets, including electron, lattice and
spin degrees of freedom. To do so, we replicate the dynamics of the
experimentally measured electron and lattice temperatures as well as
magnetization for a range of laser fluences in all three 3$d$ ferromagnetic
transition metals iron, nickel and cobalt. Notably, we are able to reduce the
number of fit parameters to only the spin-flip probability. We achieve so by
using system parameters that have been calculated from ab-initio methods and
experimentally validated. By contrast to previous works, we find that the
spin-flip probabilities agree with to those calculated via ab-initio [14, 17].
We conclude that our work evidences electron-phonon spin-flips as intrinsic
mechanism driving ultrafast magnetization dynamics in 3$d$ ferromagnets.

Our model is based on an extension to the M3TM [12, 31]. The energy flow
dynamics are described by the two-temperature model (see Fig. 1),

$\displaystyle{C_{\rm{e}}}\frac{dT_{\rm{e}}}{dt}$ $\displaystyle=$
$\displaystyle{g_{\rm{ep}}}(T_{\rm{p}}-T_{\rm{e}})+S(t)+\dot{Q}_{\rm{e-s}}$
(1) $\displaystyle{C_{\rm{p}}}\frac{dT_{\rm{p}}}{dt}$ $\displaystyle=$
$\displaystyle-{g_{\rm{ep}}}(T_{\rm{p}}-T_{\rm{e}}).$ (2)

When a metallic thin film is subjected to an optical laser pulse, only the
electrons are excited by the photon electric field. Initially, the absorbed
energy is barely transferred to the lattice and consequently the electron
system heats up. On the timescale of 100 fs, the electron temperature will
rise far above the critical temperature, $T_{\rm{c}}$. The magnetic system
responds to this fast temperature rise by reducing its magnetic order on
similar time scales. The electron and lattice systems are assumed to be
thermalized so that their energy can be described by a temperature,
$T_{\rm{e}}$ for the electrons and $T_{\rm{p}}$ for the phonons. In Eq. (1),
the absorbed laser pulse power is represented by a Gaussian function,
$S(t)=S_{0}*G(\tau_{p})$, where $\tau_{p}$ is the pulse duration. The electron
heat capacity is $C_{e}$. The electron-phonon coupling allows for temperature
equilibration of hot electrons and the lattice on the time scale determined by
the ratio $g_{\rm{ep}}/C_{\rm{e}}$. Since $g_{\rm{ep}}$ accounts only for
spin-conserving scattering events, in Eq. (1) we include a term that accounts
for the finite energy cost (gain) of a spin-flip and couple it to the electron
dynamics: $\dot{Q}_{\rm{e-s}}=J_{0}m\dot{m}/V_{\rm{at}}$, with
$J_{0}/3=[S/(S+1)]k_{\rm{B}}T_{c}$ in the MFA, $m$ is the reduced
magnetization, $J_{0}$ is the exchange energy, $V_{\rm{at}}$ the atomic volume
and effective spin $S$. In phase I ($dm/dt<0$) the energy cost of an electron-
phonon mediated spin-flip of probability $a_{\rm{sf}}$ in minority direction
is deducted from the electronic energy, while in phase II ($dm/dt>0$) the
direction of energy flow is reverted (Fig. 1). In other works the energy flow
between the spin and the electron systems has been taken into account by
adding it to the electron specific heat, $C_{\rm{e}}\rightarrow
C_{\rm{e}}+C_{\rm{s}}$, where $C_{\rm{s}}$ is the equilibrium spin specific
heat [29, 27]. However, recent works suggest that this energy flow needs to be
calculated through the spin Hamiltonian for non-equilibrium spin
configurations [29, 30]. An important aspect of the two-temperature model in
Eqs. (1) and (2) is the exact value of $C_{\rm{e}}$, $C_{\rm{p}}$, and
$g_{\rm{ep}}$. In this work, we use the ab-initio calculated parameters, which
were already used before in Ref. [29, 30]. The parameters are temperature
dependent and were already validated by direct comparison to the ultrafast
lattice dynamics in iron, nickel and cobalt [29, 30]. The electronic and
lattice heat capacities agree very well with experimental data [32, 19]. The
magnetization dynamics is calculated using the M3TM for finite spin values,
for details of the M3TM see Supplemental Material and Ref. [31]. Within this
model, the rate parameter defining the magnetization dynamics scales linearly
with the so-called spin-flip probability $a_{\rm{sf}}$. Here, we use $S=1/2$
for nickel, $S=2$ for iron and $S=3/2$ for cobalt [33]. For these values,
within the MFA, the equilibrium magnetization as function of temperature is
well reproduced for all three $3d$ ferromagnets.

Figure 1:  Energy absorbed by the spinless electrons $S(t)$ is distributed to
the lattice ($g_{\rm{ep}}$) and localized spins ($\dot{Q}_{\rm{e-s}}$). Energy
is transferred from electron to spin during demagnetization (I), and spin to
electrons during remagnetization (II). The angular momentum from electron-
phonon mediated spin-flips, with probability $a_{\rm{sf}}$, is implicitly
exchanged with lattice. Figure 2: Ultrafast dynamics in nickel; comparison
between experiments (symbols) and model (solid line). For completeness, dashed
line corresponds to the model with $\dot{Q}_{\rm{e-s}}=0$, absorbed energies
from our simulations are indicated, the simulated pump fluences were reduced
to fit simulations with $\dot{Q}_{\rm{e-s}}=0$. (a) Magnetization dynamics in
nickel measured by MOKE [34]. (b) Electron temperature dynamics measured by
tr-ARPES [27]. (c) Lattice temperature dynamics measured by electron
diffraction [29]. The orange vertical lines mark the transition between phases
I and II.

Figure 2 shows an example of the comparison of our model to experimental data
for the magnetization [Fig. 2 (a)], electron [Fig. 2 (b)] and lattice [Fig. 2
(c)] temperature dynamics in nickel. Vertical orange line denotes the
transition from phase I ($dm/dt<0$) to phase II ($dm/dt>0$). The electron
temperature is retrieved from tr-ARPES measurements on a $400$ nm nickel film,
where the electronic energy distribution around the Fermi edge is measured and
fitted to the Fermi-Dirac distribution, yielding the electronic temperature
[27]. The lattice dynamics are retrieved from time resolved electron
diffraction experiments on $20$ nm thin films on a Si3N4 substrate. Intensity
changes in the diffraction patterns reveal the mean square displacement, from
which the lattice temperature can be deducted [29]. Finally, the magnetization
dynamics were retrieved from time-resolved MOKE experiments on $15$ nm thin
films on glass wafer substrate, where the Kerr rotation, proportional to the
sample magnetization, was recovered by amplitude changes in the hysteresis
loops [34]. By direct comparison of the magnetization dynamics to the model
[solid line in Fig. 2 (b)], we obtain a value for $a_{\rm{sf}}$. Figure 2 (a)
shows the magnetization dynamics calculated with the M3TM for
$a_{\rm{sf}}=0.09$ (solid line), and experiment (symbols) [27]. The exact
value of the absorbed fluence in experiments is difficult to estimate. We use
a value in our model for the absorbed fluence such that the final temperature,
and therefore the final magnetic state for one exemplary fluence, matches the
experiment. The dashed line in Fig. 2 shows the result of the model by
assuming $\dot{Q}_{\rm{e-s}}=0$ in Eq. (1). Although the effect of this term
on the magnetization dynamics is marginal [Fig. 2 (a)], it becomes critical
for the correct description of the electron and lattice temperature dynamics
[Fig. 2 (b) and (c)]. Symbols in Figs. 2 (b) and (c) show the electron and
lattice temperature dynamics, measured in Ref. [27] and [29], respectively. We
note that for $\dot{Q}_{\rm{e-s}}=0$, the electron system cools down slower
than experimental data while the lattice heats up faster. This behaviour
cannot be described by adjustment of $g_{\rm{ep}}$, which will accelerate the
cool down of the electron temperature, but also the heat up of lattice
temperature. The correction provided by $\dot{Q}_{\rm{e-s}}\sim\dot{m}$
strongly depends on the magnetization dynamics, which in turn is determined by
$a_{\rm{sf}}$ ($\dot{m}\sim a_{\rm{sf}}$). Thus, one can regulate through
$a_{\rm{sf}}$ the speed-up of the electron temperature cooling by increasing
the transfer of energy from the electron to the magnetic system. This
procedure has been used in previous works within the framework of atomistic
spin dynamics simulations [29, 30], however we emphasize here that in those
works a direct comparison to experimentally measured magnetization dynamics in
similar samples and conditions was missing. Here, we use the data recorded by
Borchert et al. [34] for the three 3$d$ ferromagnets, for which both the
samples and the conditions are practically the same as the one used to measure
the lattice temperature dynamics [30]. First, we set the constraint of using a
value for $a_{\rm{sf}}$ that reproduces the observed magnetization dynamics
[Fig. 2(a)]. Second, we find that by using exactly the same set of parameters
that reproduces the magnetization dynamics, the lattice and electron
temperature dynamics are also very well reproduced by our model [Figs.
2(b,c)]. A second step to validate our model for nickel is to extend the
comparison to experimental data for different fluences.

Figure 3(a) shows magnetization dynamics, tr-MOKE measurements in symbols and
model results in lines, for nickel upon irradiation with pump fluences ranging
from $0.5-15\rm{mJ}/\text{cm}^{2}$ [34]. As the effect of incident fluences
are hardly comparable across different experimental setups, we denote the fits
by the absorbed energy density used in the model calculations. We note that
all the parameters in the model are the same as those used in Fig. 2. The only
parameter that changes is the value of the absorbed power, $S_{0}$, which
exactly reflects the ratio of incident fluences from experiments. Our model
replicates the experimental magnetization data perfectly [Fig. 3(a)] for
different incident fluences. The lattice temperature dynamics is also
reproduced by our model to a high level of accuracy [Fig. 3(b)]. We find some
discrepancy between the model and lattice temperature dynamics for the highest
value of the absorbed energies. We argue that this comes as a result of the
MFA expression for the rate of the electron-spin energy flow
$\dot{Q}_{\rm{e-s}}$. For high fluence, the system reach elevated
temperatures. This is reflected by the low values of the magnetization that
are achieved. For these situations critical spin fluctuations become
important, which in turn are poorly captured by the MFA. We note that this
discrepancy was not observed in a previous work using ASD simulations [29].
Atomistic spin dynamics methods account correctly for the contribution to the
energy of the spin fluctuations at elevated temperature. However, ASD
simulations are unable to reproduce the experimental magnetization dynamics.
By contrast, the M3TM model, with only one free parameter, $a_{\rm{sf}}$,
provides an excellent agreement to experimental observations. One remaining
question for nickel is the origin of the value of $a_{\rm{sf}}=0.09$. With a
generalization of spin- and energy-dependent Eliashberg functions, Carva et
al. [14] have computed the spin-flip probabilities upon electron-phonon
scattering of Elliott Yafet type in relativistic DFT. For nickel, they have
found a range of values, $a_{\rm{sf}}=0.04-0.09$, which agrees with our model.
The transition rates are strongly dependent on the occupation-difference of
electronic majority and minority states, which only arises for non-thermal
electron distributions. Our model treats the electron and lattice systems in
thermal equilibrium throughout the process. The spin system however is neither
at equilibrium with the electronic system, nor internally. Thus, it describes
spin-flips and magnon excitations of which the energetic cost is compensated
by a shift in the close to equilibrium electron distribution. The influence of
non-thermal electron distributions is an ongoing discussion [35, 27, 14]. A
final question is about the applicability of this model to the other two 3$d$
ferromagnets, iron and cobalt.

Figure 3:  Magnetization and lattice dynamics in nickel for a range of pump
fluences. (a) Magnetization dynamics in nickel from experiments by Borchert et
al. [34] (dots) and model (lines). The legend shows the total absorbed energy
density in units of J/cm3. (b) Lattice dynamics in nickel from experiments by
Zahn et al. [29] (dots) and model (lines). The color scheme is matched to
similar absorbed energies as in panel (a). Figure 4: (a) and (b):
Magnetization and lattice dynamics in cobalt. Simulations (lines) in
comparison to experimental data (dots) from Borchert et al. [34] and Zahn et
al. [30], respectively. The color coding corresponds to similar absorbed pump
energies, listed in methods section. (c) and (d): Magnetization and lattice
dynamics in iron.

The model describes the experimentally measured magnetization and lattice
temperature dynamics in iron and cobalt. The parameters in the 2TM [Eqs. (1)
and (2)] have been estimated through ab-initio calculations and validated in a
previous work [30]. We estimate the spin-flip probabilities for iron and
cobalt by comparing the magnetization dynamics to the model for one single
laser fluence, similar to the procedure used for nickel [see Fig. 2]. Figure 4
shows the comparison between the model and experiments for both the
magnetization and lattice temperature dynamics in iron and cobalt. The color
coding in Fig. 4 scales with the amount of absorbed energy density from the
laser pulse used in Eqs. (1) and (2). For iron, the magnetization dynamics is
well reproduced by the model [Fig. 4 (a)]. We note that in literature,
experimentally measured magnetization dynamics deviate significantly from one
another depending on the sample characteristic and experimental technique[2,
35, 36]. The lattice temperature dynamics is reproduced with a satisfactory
degree of accuracy [Fig. 4 (b)]. From our model, the estimated spin-flip
probability for iron $a_{\rm{sf}}=0.07$ agrees well with ab-initio
calculations of $a_{\rm{sf}}=0.04-0.07$ [17]. For cobalt, the agreement
between the experimentally measured magnetization dynamics and model is
excellent [Fig. 4 (a)]. The agreement to the lattice temperature dynamics is
also excellent. Our estimation for the spin-flip probability
$a_{\rm{sf}}=0.07$ is larger than the ab-initio calculations,
$a_{\rm{sf}}=0.01-022$, albeit in the same order of magnitude. Overall, our
model provides a very good match for the spin-flip probability in comparison
to the ab-initio calculations for the three 3$d$ ferromagnets (see Table 1),
which strongly supports our claim that the electron-phonon mediated spin-flip
is the main microscopic mechanism driving the ultrafast magnetization dynamics
in 3$d$ ferromagnets.

To summarize, we have found evidence of electron-phonon mediated spin-flips as
the intrinsic driving mechanism for the ultrafast magnetization dynamics in
all three 3$d$ ferromagnets, nickel, iron and cobalt. By direct comparison
between our model and experimental electron and lattice temperature as well as
magnetization dynamics, we determine the spin-flip probability for each
material, which in turn agrees with ab-initio predictions. Our model supports
ultrafast transfer of spin angular momentum from the spin to lattice system,
in line with the experimental work of Tauchert et al. [24] in nickel. They
have found that the angular momentum loss during the ultrafast demagnetization
is transferred to the lattice in the form of chiral phonons.

Table 1: Spin-flip probabilities $a_{\rm{sf}}$ from ab-initio calculations [17], previous M3TM fits [12] and this work. 3$d$-ferromag. | Carva et al.[17] | Koopmans et al. [12] | this work
---|---|---|---
Nickel | 0.04-0.09 | 0.17-0.2 | 0.09
Iron | 0.04-0.07 |  | 0.07
Cobalt | 0.01-0.022 | 0.135-0.165 | 0.07

_Acknowledgments.–_ We gratefully acknowledges funding by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation)—Project-ID
328545488—TRR 227, Project No. A08. U. A. gratefully acknowledges support by
grant PID2021-122980OB-C55 and the grant RYC-2020-030605-I funded by
MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe” and “ESF
Investing in your future”.

## References

  * Beaurepaire  _et al._ [1996] E. Beaurepaire, J.-C. Merle, A. Daunois, and J.-Y. Bigot, Ultrafast Spin Dynamics in Ferromagnetic Nickel, Phys. Rev. Lett. 76, 4250 (1996).
  * Carpene  _et al._ [2008] E. Carpene, E. Mancini, C. Dallera, M. Brenna, E. Puppin, and S. De Silvestri, Dynamics of Electron-Magnon Interaction and Ultrafast Demagnetization in Thin Iron Films, Phys. Rev. B 78, 174422 (2008).
  * Krauß  _et al._ [2009] M. Krauß, T. Roth, S. Alebrand, D. Steil, M. Cinchetti, M. Aeschlimann, and H. C. Schneider, Ultrafast demagnetization of ferromagnetic transition metals: The role of the coulomb interaction, Phys. Rev. B 80, 180407 (2009).
  * Wietstruk  _et al._ [2011] M. Wietstruk, A. Melnikov, C. Stamm, T. Kachel, N. Pontius, M. Sultan, C. Gahl, M. Weinelt, H. A. Dürr, and U. Bovensiepen, Hot-Electron-Driven Enhancement of Spin-Lattice Coupling in Gd and Tb $4f$ Ferromagnets Observed by Femtosecond X-Ray Magnetic Circular Dichroism, Phys. Rev. Lett. 106, 127401 (2011).
  * Frietsch  _et al._ [2015] B. Frietsch, J. Bowlan, R. Carley, M. Teichmann, S. Wienholdt, D. Hinzke, U. Nowak, K. Carva, P. M. Oppeneer, and M. Weinelt, Disparate ultrafast dynamics of itinerant and localized magnetic moments in gadolinium metal, Nature Communications 6, 8262 (2015).
  * Frietsch  _et al._ [2020] B. Frietsch, A. Donges, R. Carley, M. Teichmann, J. Bowlan, K. Döbrich, K. Carva, D. Legut, P. Oppeneer, U. Nowak, and M. Weinelt, The role of ultrafast magnon generation in the magnetization dynamics of rare-earth metals, Science Advances 6, eabb1601 (2020).
  * Windsor  _et al._ [2022] Y. W. Windsor, S.-E. Lee, D. Zahn, V. Borisov, D. Thonig, K. Kliemt, A. Ernst, C. Schüßler-Langeheine, N. Pontius, U. Staub, C. Krellner, D. V. Vyalikh, O. Eriksson, and L. Rettig, Exchange scaling of ultrafast angular momentum transfer in 4f antiferromagnets, Nature Materials 21, 514 (2022).
  * Radu  _et al._ [2011] I. Radu, K. Vahaplar, C. Stamm, T. Kachel, N. Pontius, H. A. Dürr, T. A. Ostler, J. Barker, R. F. L. Evans, R. W. Chantrell, a Tsukamoto, A. Itoh, A. Kirilyuk, T. Rasing, A. V. Kimel, and H. A. Dürr, Transient ferromagnetic-like state mediating ultrafast reversal of antiferromagnetically coupled spins., Nature 472, 205 (2011).
  * Matsubara  _et al._ [2015] M. Matsubara, A. Schroer, A. Schmehl, A. Melville, C. Becher, M. Trujillo-Martinez, D. G. Schlom, J. Mannhart, J. Kroha, and M. Fiebig, Ultrafast optical tuning of ferromagnetism via the carrier density, Nature Communications 6, 6724 (2015).
  * Maehrlein  _et al._ [2018] S. F. Maehrlein, I. Radu, P. Maldonado, A. Paarmann, M. Gensch, A. M. Kalashnikova, R. V. Pisarev, M. Wolf, P. M. Oppeneer, J. Barker, and T. Kampfrath, Dissecting spin-phonon equilibration in ferrimagnetic insulators by ultrafast lattice excitation, Science Advances 4, eaar5164 (2018).
  * Koopmans  _et al._ [2005] B. Koopmans, J. J. M. Ruigrok, F. D. Longa, and W. J. M. de Jonge, Unifying Ultrafast Magnetization Dynamics, Phys. Rev. Lett. 95, 267207 (2005).
  * Koopmans  _et al._ [2010] B. Koopmans, G. Malinowski, F. Dalla Longa, D. Steiauf, M. Fähnle, T. Roth, M. Cinchetti, and M. Aeschlimann, Explaining the paradoxical diversity of ultrafast laser-induced demagnetization, Nature Materials 9, 259 (2010).
  * Battiato  _et al._ [2010] M. Battiato, K. Carva, and P. M. Oppeneer, Superdiffusive Spin Transport as a Mechanism of Ultrafast Demagnetization, Phys. Rev. Lett. 105, 027203 (2010).
  * Carva  _et al._ [2011] K. Carva, M. Battiato, and P. M. Oppeneer, Ab Initio Investigation of the Elliott-Yafet Electron-Phonon Mechanism in Laser-Induced Ultrafast Demagnetization, Phys. Rev. Lett. 107, 207201 (2011).
  * Schellekens  _et al._ [2013] A. J. Schellekens, W. Verhoeven, T. N. Vader, and B. Koopmans, Investigating the contribution of superdiffusive transport to ultrafast demagnetization of ferromagnetic thin films, Applied Physics Letters 102, 252408 (2013).
  * Dewhurst  _et al._ [2021] J. K. Dewhurst, S. Shallcross, P. Elliott, S. Eisebitt, C. v. K. Schmising, and S. Sharma, Angular momentum redistribution in laser-induced demagnetization, Phys. Rev. B 104, 054438 (2021).
  * Carva  _et al._ [2013] K. Carva, M. Battiato, D. Legut, and P. M. Oppeneer, Ab initio theory of electron-phonon mediated ultrafast spin relaxation of laser-excited hot electrons in transition-metal ferromagnets, Phys. Rev. B 87, 184425 (2013).
  * Roth  _et al._ [2012] T. Roth, A. J. Schellekens, S. Alebrand, O. Schmitt, D. Steil, B. Koopmans, M. Cinchetti, and M. Aeschlimann, Temperature Dependence of Laser-Induced Demagnetization in Ni: A Key for Identifying the Underlying Mechanism, Phys. Rev. X 2, 021006 (2012).
  * Meschter  _et al._ [1981] P. J. Meschter, J. W. Wright, C. R. Brooks, and T. G. Kollie, Physical contributions to the heat capacity of nickel, Journal of Physics and Chemistry of Solids 42, 861 (1981).
  * Tveten  _et al._ [2015] E. G. Tveten, A. Brataas, and Y. Tserkovnyak, Electron-magnon scattering in magnetic heterostructures far out of equilibrium, Phys. Rev. B 92, 180412 (2015).
  * Stamm  _et al._ [2007] C. Stamm, T. Kachel, N. Pontius, R. Mitzner, T. Quast, K. Holldack, S. Khan, C. Lupulescu, E. Aziz, M. Wietstruk, H. Dürr, and W. Eberhardt, Femtosecond modification of electron localization and transfer of angular momentum in nickel, Nature Materials 6, 740 (2007).
  * Cinchetti  _et al._ [2006] M. Cinchetti, M. Sánchez Albaneda, D. Hoffmann, T. Roth, J.-P. Wüstenberg, M. Krauß, O. Andreyev, H. C. Schneider, M. Bauer, and M. Aeschlimann, Spin-Flip Processes and Ultrafast Magnetization Dynamics in Co: Unifying the Microscopic and Macroscopic View of Femtosecond Magnetism, Phys. Rev. Lett. 97, 177201 (2006).
  * Dornes  _et al._ [2019] C. Dornes, Y. Acremann, M. Savoini, M. Kubli, M. J. Neugebauer, E. Abreu, L. Huber, G. Lantz, C. A. F. Vaz, H. Lemke, E. M. Bothschafter, M. Porer, V. Esposito, L. Rettig, M. Buzzi, A. Alberca, Y. W. Windsor, P. Beaud, U. Staub, D. Zhu, S. Song, J. M. Glownia, and S. L. Johnson, The ultrafast Einstein–de Haas effect, Nature 565, 209 (2019).
  * Tauchert  _et al._ [2022] S. R. Tauchert, M. Volkov, D. Ehberger, D. Kazenwadel, M. Evers, H. Lange, A. Donges, A. Book, W. Kreuzpaintner, U. Nowak, and P. Baum, Polarized phonons carry angular momentum in ultrafast demagnetization, Nature 602, 73 (2022).
  * Eschenlohr  _et al._ [2013] A. Eschenlohr, M. Battiato, R. Maldonado, N. Pontius, T. Kachel, K. Holldack, R. Mitzner, A. Föhlisch, P. M. Oppeneer, and C. Stamm, Ultrafast spin transport as key to femtosecond demagnetization, Nature Materials 12, 332 (2013).
  * Shokeen  _et al._ [2017] V. Shokeen, M. Sanchez Piaia, J.-Y. Bigot, T. Müller, P. Elliott, J. K. Dewhurst, S. Sharma, and E. K. U. Gross, Spin Flips versus Spin Transport in Nonthermal Electrons Excited by Ultrashort Optical Pulses in Transition Metals, Phys. Rev. Lett. 119, 107203 (2017).
  * Tengdin  _et al._ [2018] P. Tengdin, W. You, C. Chen, X. Shi, D. Zusin, Y. Zhang, C. Gentry, A. Blonsky, M. Keller, P. M. Oppeneer, H. C. Kapteyn, Z. Tao, and M. M. Murnane, Critical behavior within 20 fs drives the out-of-equilibrium laser-induced magnetic phase transition in nickel, Science Advances 4, eaap9744 (2018).
  * You  _et al._ [2018] W. You, P. Tengdin, C. Chen, X. Shi, D. Zusin, Y. Zhang, C. Gentry, A. Blonsky, M. Keller, P. M. Oppeneer, H. Kapteyn, Z. Tao, and M. Murnane, Revealing the Nature of the Ultrafast Magnetic Phase Transition in Ni by Correlating Extreme Ultraviolet Magneto-Optic and Photoemission Spectroscopies, Phys. Rev. Lett. 121, 077204 (2018).
  * Zahn  _et al._ [2021] D. Zahn, F. Jakobs, Y. W. Windsor, H. Seiler, T. Vasileiadis, T. A. Butcher, Y. Qi, D. Engel, U. Atxitia, J. Vorberger, and R. Ernstorfer, Lattice dynamics and ultrafast energy flow between electrons, spins, and phonons in a 3d ferromagnet, Phys. Rev. Research 3, 023032 (2021).
  * Zahn  _et al._ [2022] D. Zahn, F. Jakobs, H. Seiler, T. A. Butcher, D. Engel, J. Vorberger, U. Atxitia, Y. W. Windsor, and R. Ernstorfer, Intrinsic energy flow in laser-excited $3d$ ferromagnets, Phys. Rev. Research 4, 013104 (2022).
  * Beens  _et al._ [2019] M. Beens, M. L. M. Lalieu, A. J. M. Deenen, R. A. Duine, and B. Koopmans, Comparing all-optical switching in synthetic-ferrimagnetic multilayers and alloys, Phys. Rev. B 100, 220409 (2019).
  * Hofmann  _et al._ [1956] J. Hofmann, A. Paskin, K. Tauer, and R. Weiss, Analysis of ferromagnetic and antiferro-magnetic second-order transitions, Journal of Physics and Chemistry of Solids 1, 45 (1956).
  * Köbler  _et al._ [2003] U. Köbler, J. Englich, O. Hupe, and J. Hesse, Effective spin quantum numbers in iron, cobalt and nickel, Physica B Condensed Matter 339, 156 (2003).
  * Borchert  _et al._ [2020] M. Borchert, C. von Korff Schmising, D. S. Schick, D. Engel, S. Sharma, S. Shallcross, and S. Eisebitt, Uncovering the role of the density of states in controlling ultrafast spin dynamics, arXiv: 2008.12612 (2020).
  * Chekhov  _et al._ [2021] A. Chekhov, Y. Behovits, J. Heitz, C. Denker, D. Reiss, M. Wolf, M. Weinelt, P. Brouwer, M. Münzenberg, and T. Kampfrath, Ultrafast Demagnetization of Iron Induced by Optical versus Terahertz Pulses, Physical Review X 11, 041055 (2021).
  * Zahn [2021] D. Zahn,  _Structural dynamics and microscopic energy flow in ferromagnetic metals and in an anisotropic layered material_ , Ph.D. thesis, Freie Universität Berlin (2021).

Supplementary Material to "Electron-phonon mediated spin-flip as driving
mechanism for ultrafast magnetization dynamics in 3$d$ ferromagnets"

##  S1 S1. Microscopic three temperature model: magnetization dynamics

The magnetization dynamics are determined by the statistical change of
occupation numbers $f_{m_{s}}$ of the $S_{z}$ component $m_{s}$:

$\displaystyle\frac{dm}{dt}$ $\displaystyle=$
$\displaystyle-\frac{1}{S}\sum_{ms=-S}^{ms={+S}}m_{s}\frac{df_{m_{s}}}{dt}$
(S1) $\displaystyle\frac{df_{m_{s}}}{dt}$ $\displaystyle=$
$\displaystyle-(W^{+}_{m_{s}}+W^{-}_{m_{s}})f_{m_{s}}+W^{+}_{m_{s-1}}f_{m_{s-1}}+W^{-}_{m_{s+1}}f_{m_{s+1}}$
(S2) $\displaystyle W^{\pm}_{m_{s}}$ $\displaystyle=$ $\displaystyle
R\frac{Jm}{4Sk_{B}T_{c}}\frac{T_{p}}{T_{c}}\frac{\text{e}^{\mp\tfrac{\tiny{Jm}}{\tiny{2Sk_{B}T_{e}}}}}{\text{sinh}(\frac{\tiny{Jm}}{\tiny{2Sk_{B}T_{e}}})}$
(S3) $\displaystyle\cdot(S(S+1)-m_{s}(m_{s}\pm 1)),$

The analytical description of the transition rates $W^{\pm}_{m_{s}}$ was
introduced by Beens et al. [31]. $J=3k_{B}T_{C}S/(S+1)$ is the mean flied
exchange parameter for a system of spin $S$ and $T_{C}$ the Curie temperature.
The rate parameter

$\displaystyle
R=\frac{8a_{\rm{sf}}{g_{\rm{ep}}}k_{B}T^{2}_{C}V_{\rm{at}}}{(\mu_{\rm{at}}{E^{2}_{\rm{D}}})}$
(S4)

is proportional to the microscopic spin-flip probability $a_{\rm{sf}}$, where
$V_{\rm{at}}$ is the atomic volume in the lattice and $E_{D}=k_{B}T_{D}$ the
Debye energy.


# A Multi-task Learning Framework for Product Ranking with BERT

Xuyang Wu  Santa Clara UniversitySanta ClaraCAUSA<EMAIL_ADDRESS>,  Alessandro
Magnani  Walmart Global TechSunnyvaleCAUSA<EMAIL_ADDRESS>,
Suthee Chaidaroon  Walmart Global TechSunnyvaleCAUSA
<EMAIL_ADDRESS>,  Ajit Puthenputhussery  Walmart Global
TechSunnyvaleCAUSA<EMAIL_ADDRESS>,  Ciya Liao  Walmart
Global TechSunnyvaleCAUSA<EMAIL_ADDRESS>and  Yi Fang  Santa Clara
UniversitySanta ClaraCAUSA<EMAIL_ADDRESS>
(2022)

###### Abstract.

Product ranking is a crucial component for many e-commerce services. One of
the major challenges in product search is the vocabulary mismatch between
query and products, which may be a larger vocabulary gap problem compared to
other information retrieval domains. While there is a growing collection of
neural learning to match methods aimed specifically at overcoming this issue,
they do not leverage the recent advances of large language models for product
search. On the other hand, product ranking often deals with multiple types of
engagement signals such as clicks, add-to-cart, and purchases, while most of
the existing works are focused on optimizing one single metric such as click-
through rate, which may suffer from data sparsity. In this work, we propose a
novel end-to-end multi-task learning framework for product ranking with BERT
to address the above challenges. The proposed model utilizes domain-specific
BERT with fine-tuning to bridge the vocabulary gap and employs multi-task
learning to optimize multiple objectives simultaneously, which yields a
general end-to-end learning framework for product search. We conduct a set of
comprehensive experiments on a real-world e-commerce dataset and demonstrate
significant improvement of the proposed approach over the state-of-the-art
baseline methods.

Product Search, Multi-task Learning, Neural Information Retrieval

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
ACM Web Conference 2022; April 25–29, 2022; Virtual Event, Lyon,
France††booktitle: Proceedings of the ACM Web Conference 2022 (WWW ’22), April
25–29, 2022, Virtual Event, Lyon, France††price: 15.00††doi:
10.1145/3485447.3511977††isbn: 978-1-4503-9096-5/22/04††ccs: Computing
methodologies Multi-task learning††ccs: Information systems Retrieval models
and ranking

##  1\. Introduction

Online shopping has now become an integral part of people’s daily life. With
an ever increasing catalog size, product search systems have been playing a
crucial role in serving customers shopping on online e-commerce platforms, by
benefiting both users and suppliers. Many approaches have been proposed for
product search, ranging from adaptations of general web search models (Duan et
al., 2013) to applying learning to rank to the e-commerce domain (Trotman et
al., 2017). Recently, neural information retrieval (IR) methods based on deep
neural networks have been introduced and utilized in product search (Magnani
et al., 2019; Zhang et al., 2019), which has demonstrated promising results by
learning semantic representations of queries and products. There exist several
challenges in applying neural IR to product search. First of all, product
search often deals with multiple types of engagement signals such as clicks,
add-to-cart, and purchases, while most of the existing work is focused on
optimizing one single metric such as click-through rate. In addition, neural
IR generally requires a huge amount of training data and one source of data
may not be able to support sufficient model training. Last but not least, the
vocabulary mismatch between query and document in product search may be a
larger problem than in other IR domains (Gysel et al., 2016). The existing
work does not fully utilize the recent advances of large language models for
product search as mediocre results are reported when using BERT in product
ranking (Sarvi et al., 2020).

In this paper, we tackle the above challenges by proposing a multi-task
learning framework for product ranking with BERT. The proposed framework aims
at optimizing multiple objectives at the same time such as click, add-to-cart,
and purchase. One of the major challenges in applying multi-task learning is
that the intrinsic conflicts between different tasks can hurt the model
performance on some of the tasks, especially when model parameters are heavily
shared among the tasks. As a result, many deep learning based multi-task
learning models are sensitive to factors such as the data distributions and
relationships among tasks (Ma et al., 2018b). Our proposed multi-task learning
architecture was inspired by the Mixture-of-Experts (MoE) layer (Shazeer et
al., 2017) and the recent MMoE model (Ma et al., 2018b). Compared with roughly
shared parameters in MMoE, we explicitly separate shared and task-specific
experts to alleviate harmful parameter interference between common and task-
specific knowledge.

Given a user query, a ranked list of products is shown and the user may click
on the relevant items, add some of them to the shopping cart, and eventually
purchase one or more items. The user engagement action generally follows a
sequential order, impression $\rightarrow$ click $\rightarrow$ add-to-cart
$\rightarrow$ purchase. In other words, the later step may occur only if the
former steps are completed with positive feedback. In this paper, we utilize
probability transfer (Wen et al., 2020) over the sequential user behaviors to
leverage all impression samples over the entire space and extra abundant
supervisory signals from post-click behaviors, efficiently addressing the data
sparsity issue. Specifically, we model the respective probabilities of click,
add-to-cart, and purchase simultaneously in a multi-task learning framework
according to the conditional probability rule defined on the user behavior
graph.

On the other hand, the vocabulary mismatch between query and document poses a
fundamental challenge in search engines in general. In product search, the
vocabulary gap may be a larger problem than in other information retrieval
domains (Sarvi et al., 2020; Gysel et al., 2016). Queries and product titles
are usually short, and titles are sometimes phrases or simple combinations of
keywords instead of well-structured sentences. There is a growing interest in
utilizing neural learning to match methods for bridging the vocabulary gap in
product search. These methods go beyond lexical matching by representing
queries and documents as dense semantic vectors and learning their degree of
similarity in the semantic space (Mitra et al., 2018). However, the existing
work has not fully utilized the advantages of large pre-trained language
models such as BERT (Devlin et al., 2019). A recent study has explored the
state-of-the-art BERT-based model for product search but demonstrated mediocre
results when comparing with traditional methods (Sarvi et al., 2020). In this
paper, we pre-train a domain-specific BERT model with fine-tuning and show
much promising results. The main contributions of the paper can be summarized
as follow:

  * •

We propose a novel multi-task learning framework for product ranking based on
neural information retrieval. To the best of our knowledge, there exists no
prior work on integrating multiple types of engagement signals with neural IR
for product search.

  * •

To model the relationships between different tasks and alleviate harmful
parameter interference between common and task-specific knowledge, the
proposed framework utilizes the mixture-of-experts layer and probability
transfer between tasks to harness the abundant engagement data.

  * •

The proposed framework integrates semantic match with traditional ranking
features in an end-to-end learning manner. We leverage a domain-specific BERT
for semantic match with fine-tuning and demonstrate promising results.

  * •

We conduct a comprehensive set of experiments on a real-world product search
dataset in e-commerce and show the effectiveness of the proposed approach over
the competitive baseline methods.

##  2\. Related Work

###  2.1. Product Search and Ranking

Product search problem is more challenging than the traditional web search
problem (Duan et al., 2013) as the user queries tend to be short and there are
millions of potentially related products (Sarvi et al., 2020). Some
researchers proposed an iterative approach with two or more steps where an
initial set of candidate items is retrieved. The retrieved items were then
iteratively ordered (re-ranked) and reduced in size by picking the top ones in
the list (Trotman et al., 2017). Long et al. (Long et al., 2012) proposed a
model that incorporates best-selling products data to rank the search results.
Also, the model in (Parikh and Sundaresan, 2011) considered the diversity of
the product results to enhance the user experience. As various signals could
be used in e-commerce to measure the quality of the search results, some works
(Bi et al., 2019; Wu et al., 2018) optimized the search results based on user
engagements, such as click-through rate and conversion rate. Unfortunately,
user engagement data, is often very sparse and it can therefore limit the
performance of the model. Deep neural-based models (Mitra et al., 2018; Onal
et al., 2018) were also used in both the retrieval and ranking steps. Magnani
et al. (Magnani et al., 2019) enhanced the deep learning-based model using
different types of text representation and loss function. Zhang et al. (Zhang
et al., 2019) added to the ranking model, interaction features between the
query and a graph of products that captured relationship between similar
products.

###  2.2. Neural Information Retrieval

In neural information retrieval, ranking models can be categorized into two
main groups, representation-based and interaction-based models.
Representation-based models learn an embedding for queries and items
respectively and then measure the relevance of a product for a given query, by
computing a distance between the query and item embedding. DSSM (Huang et al.,
2013) computed query and item embeddings by averaging word embeddings from
query text and document text fields, respectively. CLSM (Palangi et al., 2016)
and LSTM-RNN (Shen et al., 2014) used CNN (LeCun et al., 1989) and LSTM
(Hochreiter and Schmidhuber, 1997) networks. NRM-F (Zamani et al., 2018)
achieved a better performance by encoding multiple product fields (e.g. title,
description, color) in the embedding. Representation-based methods were often
limited by the embedding size and consequently they could not summarize all
the information from the original data and capture precise lexical matching.

Interaction-based models looked at interactions between word pairs in queries
and items. DRMM (Guo et al., 2016) computed the cosine similarity between each
word embedding in the query and item. In recent years, BERT-based (Devlin et
al., 2019) models have achieved the state-of-the-art performance (Dai and
Callan, 2019; Nogueira and Cho, 2019; Qiao et al., 2019) in ranking. They
usually concatenated the query and document string as one sentence which was
then fed to multiple transformer layers. The attention mechanism (Vaswani et
al., 2017) within the network could attend to each word pair of the query and
product.

###  2.3. Multi-task Learning

Multi-Task learning aims to improve generalization by leveraging domain-
specific information in the training signals of related tasks (Vandenhende et
al., 2021). It has several advantages over traditional single-task learning.
Due to their inherent layer sharing, the resulting memory efficiency can be
substantially reduced and the inference speed can be improved. Moreover, the
associated tasks can benefit from each other if they share complementary
information, or act as a regularizer for one another.

The early multi-task learning (MTL) work mainly focused on hard parameter
sharing (Ruder, 2017). This is also a very common type of MTL models. The
output of the shared layers fed unique modules for different tasks. When tasks
were highly correlated, this structure could often achieve good results. When
the tasks were not so correlated, there could be a negative migration
phenomenon. Some works, such as MMoE (Ma et al., 2018b) and PLE (Tang et al.,
2020), addressed this issue by utilizing multiple experts on a shared bottom
structure. Based on the gating mechanism, different tasks could filter the
output of different experts, shared experts, and task-specific experts. This
type of models mainly learned in the shared architecture at the bottom but did
not exchange more information at the top.

Some other ideas, such as ESMM (Ma et al., 2018a) and $ESM^{2}$ (Wen et al.,
2020) models, used probability transfer based on the sequential order between
different tasks at the top of the model to optimize the model effect and
achieve better results in click-through rate and conversion rate estimation
tasks. In (Gao et al., 2019, 2021), neural collaborative filtering was
extended to the setting of multi-task learning. The above models mainly used
probability transfer, in which only simple probability information was
transferred between adjacent tasks. Xi et al. (Xi et al., 2021) proposed AITM,
which modeled the sequential dependence among multi-step conversions and
adaptively learned what and how much information to transfer for different
conversion stages.

##  3\. Multi-task Learning For Product Ranking

Figure 1. The architecture of the proposed multi-task learning framework for
product ranking (MLPR).

In this section, we present the proposed multi-task learning framework for
product ranking (MLPR). Let
$\mathcal{Q}=\\{q_{1},q_{2},...,q_{n},...,q_{N}\\}$ denote the collection of
$N$ user queries, and $\mathcal{I}=\\{i_{1},i_{2},...,i_{m},...,i_{M}\\}$
denote the collection of $M$ products (items). Given the search results, we
consider three types of user activities: click, add-to-cart (ATC), and
purchase, which follow a sequential order as follow: impression $\rightarrow$
click $\rightarrow$ add-to-cart $\rightarrow$ purchase. Our task is to predict
the probability of each engagement activity given a query $q_{n}$ and product
$i_{m}$ pair, mathematically as follow

(1) $\displaystyle\hat{y}^{k}_{n,m}=\mathcal{F}^{k}(\phi(q_{n}),\psi(i_{m}))$

where $\phi(\cdot)$ and $\psi(\cdot)$ denote the query encoder and the item
encoder, respectively. $\mathcal{F}^{k}(\cdot)$ denotes the prediction
probability function for the task $k$ based on query $q_{n}$ and product
$i_{m}$ pair. Since there are three types of engagement activities, we
formulate them as multi-task learning problem by optimizing all these three
objectives simultaneously.

Figure 1 illustrates the architecture of the proposed framework, which
consists of five stages: deep & wide feature generation, multi-experts with
shared gating control, specific-experts with customized gating control, tower
network and attention unit, and probability transfer. Given $K$ tasks, the
deep & wide feature generation stage create the input features based on the
raw data, which is followed by the two-stage extraction networks designed as a
shared-bottom structure. The tower networks with the attention units are built
upon the output of the extraction networks. They generate the output for the
corresponding task $k$. We will present each stage in details in the following
subsections.

###  3.1. Deep & Wide Feature Generation

The deep features include query embedding, product embedding, and their
interactions. In MLPR, we leverage a domain-specific BERT for learning the
embeddings, which are pre-trained on the e-commerce domain data. The query
embedding is generated by the domain-specific BERT from the query text field.
The product embedding is generated from the title field, type field, brand
field, color field, and gender field of the product. After obtaining the
embedding features, we also compute the interactions between query embedding
and product embedding based on Cosine similarity
$\dfrac{\mathbf{\mu}_{q}^{T}\cdot\mathbf{\mu}_{i}}{\parallel\mathbf{\mu}_{q}\parallel\cdot\parallel\mathbf{\mu}_{i}\parallel}$,
Hadamard (element-wise) product $\mathbf{\mu}_{q}\circ\mathbf{\mu}_{i}$, and
concatenation $\mathbf{\mu}_{q}\oplus\mathbf{\mu}_{i}$.

##### Domain-specific BERT

We utilize the fine-tuned BERT model to generate the input query and item
embedding. We first initialized the BERT model with the pre-trained weights
taken from the distillbert-base-uncased111https://huggingface.co/distilbert-
base-uncased. Then the BERT model was fine-tuned based on the user engagement
logs collected from the e-commerce website. Each row of the log file consists
of one query and a list of clicked, add-to-cart and purchased items. The
training objective is to estimate the optimal order of these items by using
the numbers of clicks, add-to-cart and purchases as the ground truth. For each
query, we also injected randomly sampled items. The ratio of relevant and
sampled items are 1:20. We use a raw query and item attributes such as title,
color, brand, and product types as the inputs to the model. The model has both
query and item encoders. The last layer of the encoder outputs 256-dimensional
query and item vectors.

Wide features directly come from the production side ranking features, which
are the traditional features used in learning to rank. They can be generally
grouped into the following categories: query item level engagement features
(e.g., query item CTR, ATC, order ratio, etc.), item attributes (e.g.,
category, price, rating score, review count, etc.), iteration features (e.g.,
similarity score, matching score) and so on. In our experiments, we remove the
engagement ranking features to avoid any potential data leak. We obtained a
total of 243 ranking features. In addition, we use the z-score to normalize
all the ranking features with mean and standard deviation computed from
training data. The concatenated features after deep & wide feature generation
are denoted as follows:

(2)
$\mathbf{x}_{n,m}=concat(\phi(q_{n}),\psi(i_{m}),\varphi(\phi(q_{n}),\psi(i_{m})),\mathbf{r}_{n,m}),$

where $\phi(\cdot)$, $\psi(\cdot)$ and $\varphi(\cdot)$ denote the query
encoder, item encoder and interaction feature encoder, respectively.
$\mathbf{r}_{n,m}$ denotes the ranking features for each query and item pair
$(q_{n},i_{m})$. $\mathbf{x}_{n,m}$ denote the concatenated feature vector
generated from query and item input field.

###  3.2. Two-stage Extraction Networks

Many multi-task learning models in the existing work contain a shared layer at
the bottom, which can learn common knowledge from different tasks. In
addition, the shared experts can continuously absorb the joint hidden
information from different tasks. This structure may help alleviate
overfitting, but it could negatively affect model performance due to task
dependence and data distribution. We propose two stages in the the extraction
networks and explicitly separate shared and task-specific experts to avoid
harmful parameter interference.

####  3.2.1. Multi-Experts with Shared Gating Control

In this stage, the model utilizes the gating network mechanism at the bottom
of the model based on the principle of MMoE (Ma et al., 2018b). Each task uses
a separate gating network. The gating network of each task achieves the
selective utilization in different task networks through different final
output weights. Various schemes of gating networks can learn different
patterns of combined experts, and thus the model will consider the relevance
and difference of each task. For each input from the previous stage, the
current stage can select the partial meaningful experts by the gating network
conditioned on the input. Each expert network is a simple multi-layer feed-
forward network with batch normalization and ReLu activation function. The
gating network is designed as a single-layer feed-forward network with a
Softmax activation function. The output of each gating network is formulated
as:

(3) $\displaystyle w^{k}(\mathbf{x}_{n,m})$
$\displaystyle=Softmax(\mathbf{W}^{k}_{g}\mathbf{x}_{n,m})$ $\displaystyle
g^{k}(\mathbf{x}_{n,m})$
$\displaystyle=w^{k}(\mathbf{x}_{n,m})E^{k}(\mathbf{x}_{n,m})$

where $\mathbf{x}_{n,m}$ is the concatenated feature vector from the deep &
wide feature generation layer, $\mathbf{W}^{k}_{g}$ is the trainable parameter
matrix for task $k$, $w^{k}(\mathbf{x}_{n,m})$ is the weighting function which
obtains the weighted vector of task $k$ by a linear layer with the Softmax
activation function. $E^{k}(\mathbf{x}_{n,m})$ is the expert network.

####  3.2.2. Specific-Experts with Customized Gating Control

In this stage, our model applies the specific experts with customized gating
controllers to extract the task-specific hidden information. The shared expert
module and the task-specific expert module will obtain the input from the
previous stage. The parameters in the shared expert are affected by all the
tasks. They are in the task-specific expert affected by the corresponding task
(Tang et al., 2020).

(4) $\displaystyle w^{k}(\mathbf{x})$
$\displaystyle=Softmax(\mathbf{W}^{k}_{g}\mathbf{x})$ $\displaystyle
H^{k}(\mathbf{x})$ $\displaystyle={[E^{k}(\mathbf{x}),E^{s}(\mathbf{x})]}^{T}$
$\displaystyle g^{k}(\mathbf{x})$
$\displaystyle=w^{k}(\mathbf{x})H^{k}(\mathbf{x})$

where $H^{k}(\mathbf{x})$ denotes the vector of the combination of shared
experts $E^{s}(\mathbf{x})$ and the task $k$’s specific experts
$E^{k}(\mathbf{x})$, $\mathbf{x}$ denotes the previous layer’s output
$g^{k}(\mathbf{x}_{n,m})$. Then the model uses the gating network to calculate
the weighted sum of the selected vectors, which is the same structure as Eqn.
(3) in the previous stage with different parameter matrix $\mathbf{W}^{k}_{g}$
and input experts $H^{k}(\mathbf{x})$.

###  3.3. Tower Network & Attention Unit

In the upper stage, tower networks obtain the prediction corresponding to each
task. Each tower network is a simple multi-layer feed-forward network, and it
can be extended to any advanced structure. The attention units learn more
task-driven confidential information within the tower network. For a task $k$,
those units could adaptively transfer helpful information from the former
task. Given the $K$ tasks, the output $\mathbf{t}^{k}$ of the tower network
for each task $k$ is defined as follows:

(5) $\mathbf{t}^{k}={MLP}^{k}(\mathbf{v})$

where $\mathbf{t}^{k}(\cdot)$ denotes the tower network and input $\mathbf{v}$
is the output of the shared-bottom stage, the output of Eqn. (4).

For the attention units, there are two inputs from the adjacent tasks $k-1$
and $k$, respectively, and the output of attention unit $\mathbf{a}^{k}$ of
the task $k$ is defined as:

(6)
$\mathbf{a}^{k}=Attention(\mathbf{t}^{k},\mathbf{a}^{k-1})=softmax(\frac{\mathbf{QK}^{T}}{\sqrt{d_{k}}})\mathbf{V}$

where $Attention(\cdot,\cdot)$ function is the similar design with self-
attention mechanism (Vaswani et al., 2017), and $\mathbf{t}^{k}$ is the tower
network’s output, $\mathbf{a}^{k-1}$ is the attention unit output from the
former task.
$\mathbf{Q}=\mathbf{W_{Q}}(\mathbf{t}^{k}\oplus\mathbf{a}^{k-1}),\mathbf{K}=\mathbf{W_{K}}(\mathbf{t}^{k}\oplus\mathbf{a}^{k-1}),\mathbf{V}=\mathbf{W_{V}}(\mathbf{t}^{k}\oplus\mathbf{a}^{k-1})$
is a simple single-layer feed-forward network with different weight matrix
$\mathbf{W_{Q},W_{K},W_{V}}$, respectively. For the first task without former
task, $\mathbf{a}^{1}=Attention(\mathbf{t}^{1},\emptyset)$.

The output of attention unit $a^{k}$ feeds into a single-layer feed-forward
network ${MLP}^{k}(\cdot)$ to obtain the corresponding prediction probability
$\hat{p}^{k}$ for each task $k$.

(7) $\hat{p}^{k}=sigmoid({MLP}^{k}(\mathbf{a}^{k}))$

###  3.4. Probability Transfer

To alleviate the data sparsity and the bias of sample space, the proposed
framework adopts the probability transfer mechanism (Ma et al., 2018a), which
is defined on the user behavior graph impression $\rightarrow$ click
$\rightarrow$ add-to-cart $\rightarrow$ purchase. Given the impression
$\mathbf{x}$, the model prediction probability transfer is defined as:

(8) $\displaystyle\hat{y}^{Click}$
$\displaystyle=\hat{p}^{ctr}=p(y^{Click}=1|\mathbf{x})$
$\displaystyle\hat{y}^{ATC}$ $\displaystyle=\hat{p}^{ctr}\times\hat{p}^{avr}$
$\displaystyle=p(y^{Click}=1|\mathbf{x})\times
p(y^{ATC}=1|y^{Click}=1,\mathbf{x})$ $\displaystyle\hat{y}^{Purchase}$
$\displaystyle=\hat{p}^{ctr}\times\hat{p}^{avr}\times\hat{p}^{cvr}$
$\displaystyle=p(y^{Click}=1|\mathbf{x})\times
p(y^{ATC}=1|y^{Click}=1,\mathbf{x})$ $\displaystyle\times
p(y^{Purchase}=1|y^{Click}=1,y^{ATC}=1,\mathbf{x})$

where $y^{Click}=1$, $y^{ATC}=1$, $y^{Purchase}=1$ denote whether click or
add-to-cart or purchase event occurs, respectively. $\hat{y}^{Click}$,
$\hat{y}^{ATC}$, and $\hat{y}^{Purchase}$ denote the final outputs of the
model, respectively. $\hat{p}^{ctr}=p(y^{Click}=1|\mathbf{x})$ denotes the
post-view click-through rate.
$\hat{p}^{avr}=p(y^{ATC}=1|y^{Click}=1,\mathbf{x})$ denotes the click-through
add-to-cart conversion rate, which is defined as the conditional probability
of the product being added to cart given that it has been clicked. Similarly,
$\hat{p}^{cvr}=p(y^{Purchase}=1|y^{Click}=1,y^{ATC}=1,\mathbf{x})$ denotes the
click-through conversion rate, defined as the conditional probability of the
product being purchased given that it has been added into cart, which depicts
the complete behavior sequence: impression $\rightarrow$ click $\rightarrow$
add-to-cart $\rightarrow$ purchase.

###  3.5. Loss Optimization

The final loss is a linear combination of the losses of the individual tasks:

(9) $\mathcal{L}_{MTL}=\sum_{k}w_{k}\cdot\mathcal{L}_{k}$

where $w_{k}$ is the task-specific weight and $\mathcal{L}_{k}$ is the task-
specific loss function. In MLPR, we adopt the uncertainty weighting of the
loss optimization (Kendall et al., 2018) which uses the homoscedastic
uncertainty to balance the single-task losses. The model’s homoscedastic
uncertainty or task-dependent uncertainty is not output but a quantity that
remains constant for different input examples of the same task. The
optimization procedure is carried out to maximize a Gaussian likelihood
objective that accounts for the homoscedastic uncertainty. In the model the
uncertainty loss can be formulated as:

(10)
$\displaystyle\mathcal{L}_{MTL}(\mathbf{W},\sigma_{1},\sigma_{2},\sigma_{3})=$
$\displaystyle\dfrac{1}{2\sigma_{1}^{2}}\mathcal{L}_{1}(\mathbf{W})+\dfrac{1}{2\sigma_{2}^{2}}\mathcal{L}_{2}(\mathbf{W})+\dfrac{1}{2\sigma_{3}^{2}}\mathcal{L}_{3}(\mathbf{W})$
$\displaystyle+\log\sigma_{1}\sigma_{2}\sigma_{3}$

where $\mathcal{L}_{1}$, $\mathcal{L}_{2}$, and $\mathcal{L}_{3}$ represent
the losses of the three tasks, respectively. $\sigma_{1}$, $\sigma_{2}$, and
$\sigma_{3}$ are the corresponding noise parameters and can balance the task-
specific losses. The trainable parameters should be automatically updated
during the training process.

##  4\. Experimental Setup

###  4.1. Dataset

The e-commerce dataset was collected from Walmart.com during one continuous
month in Oct 2020, which contains the user search queries, the corresponding
products in the search results, and the user engagement data for each query-
item pair including the number of clicks, the number of adding to the shopping
cart, and the number of purchases. We filtered out the query-item pairs with
less than or equal to five impressions. Then, we divided the data into
training, validation, and test sets, with the percentage of 80%, 10%, and 10%,
respectively. Each query-item pair is associated with one or more types of
engagement: clicks, ATC, and purchases. Table 1 shows the data statistics.

Query | Items | Query-Item pairs | Impressions
---|---|---|---
467,622 | 4,286,211 | 14,856,350 | 312,926,929
Table 1. Statistics of the e-commerce dataset.

###  4.2. Evaluation Metrics

We aim to evaluate two aspects of the proposed work: prediction and ranking.
First of all, the proposed multi-task learning model predicts the probability
for each query-item pair on each of the three types of user engagement
(clicks, ATC, and purchase). We use Area Under the Curve (AUC) of Receiver
Operating Characteristic (ROC) for the prediction tasks as it is widely used
for evaluating classification/prediction models (Manning et al., 2008). To
evaluate the ranking results for each test query, we apply Normalized
Discounted Cumulative Gain (NDCG) which is suitable for product search where
users are usually sensitive to the ranked position of the relevant
products(Manning et al., 2008).

Model | AUC | NDCG@1 | NDCG@5
---|---|---|---
Click | ATC | Purchase | Click | ATC | Purchase | Click | ATC | Purchase
$MLP_{Single}$ | +3.93% | +2.06% | +0.01% | +8.06% | +3.39% | +0.06% | +5.46% | +2.20% | +1.36%
$MLP_{MTL}$ | +3.78% | +2.70% | +0.03% | +8.81% | +4.97% | -0.28% | +5.85% | +3.54% | +1.36%
$ESM^{2}$ | +1.48% | +0.28% | -0.70% | +2.64% | -2.66% | -2.04% | +1.42% | -2.79% | +0.03%
MMoE | -0.73% | -0.15% | -1.01% | -1.70% | -5.23% | -4.52% | -1.66% | -3.79% | -1.99%
PLE | +5.80% | +3.63% | +0.56% | +10.14% | +6.31% | +3.28% | +7.84% | +4.86% | +3.69%
AITM | +5.86% | +3.98% | +0.64% | +9.88% | +6.93% | +3.13% | +7.73% | +4.86% | +3.49%
MLPR | $\bf+6.48\%^{\dagger}$ | $\bf+4.66\%^{\dagger}$ | $\bf+1.03\%^{\dagger}$ | $\bf+17.22\%^{\dagger}$ | $\bf+10.61\%^{\dagger}$ | $\bf+5.36\%^{\dagger}$ | $\bf+10.48\%^{\dagger}$ | $\bf+8.10\%^{\dagger}$ | $\bf+5.65\%^{\dagger}$
Table 2. Experimental results in terms of percentage lift over XGBoost in AUC,
NDCG@1, and NDCG@5 for the tasks: Click, Add-to-cart (ATC) and Purchase. The
best results on each task are highlighted. $\dagger$ denotes statistically
significant improvement from XGBoost to MLPR with the p-value $<0.0001$ using
the two-tailed t-test.

###  4.3. Baseline Methods

We compare MLPR with the following competitive baselines:

  * •

XGBoost (Chen and Guestrin, 2016): XGBoost is a gradient boosting framework
that uses tree-based learning algorithms. It has been widely-used in
industrial ranking systems. In the experiments, only relative metric
improvements over XGBoost instead of absolute values are presented, due to the
company confidential policy.

  * •

$\mathbf{MLP_{Single}}$: This is a single-task learning model with the basic
multi-layer perceptron (MLP) used for each task.

  * •

$\mathbf{MLP_{MTL}}$: We use shared-bottom structure at the bottom and tower
network at the top. The structure of shared-bottom and tower network are
multi-layer perceptron (Goodfellow et al., 2016).

  * •

$\mathbf{ESM^{2}}$ (Wen et al., 2020): The ESMM (Ma et al., 2018a) and
$ESM^{2}$ with probability transfer pattern were designed for solving the non-
end-to-end post-click conversion rate via training on the entire space to
relieve the sample selection bias problem.

  * •

MMoE (Ma et al., 2018b): The MMoE with Expert-Bottom pattern is designed to
integrate experts via multiple gates in the Gate Control.

  * •

PLE (Tang et al., 2020): The Progressive Layered Extraction (PLE) with Expert-
Bottom pattern separates task-shared experts and task-specific experts
explicitly under different task correlations.

  * •

AITM (Xi et al., 2021): The AITM model with adaptive information module
transfers the knowledge from different conversion stages in the vector space.

##  5\. Experimental Results

###  5.1. Baseline Comparison

In this section, we compare the proposed MLPR model with the baseline methods
on the three types of user engagement: clicks, add-to-cart (ATC), and
purchases. Table 2 contains the results and we have the following
observations. First of all, MLPR achieved the best results on all the three
tasks in both metrics and outperformed all the competitive baselines with a
significant margin. The deep learning based models yielded better results than
the traditional XGBoost method, indicating the advantages of the neural
methods in utilizing a large amount of data for model training. We can then
compare a basic multi-task learning model $MLP_{MTL}$ with the single-task
learning models $MLP_{Single}$ and XGBoost. The results showed that
$MLP_{MTL}$ achieved better results in most of the metrics, which indicated
the effectiveness of a multi-task learning approach in transferring knowledge
between different tasks.

Based on the experts-bottom-based structure, the standard MMoE model could not
perform well on the dataset. It performed even worse than the $MLP_{MTL}$
model, as it only controls the shared knowledge among different tasks.
However, the PLE model with the specific-experts layer could improve
significantly by transferring the shared information and task-specific
knowledge among various tasks. The AUC of Click was improved by 5.8% over
XGBoost. The ${ESM^{2}}$ and AITM models optimize the performance in the upper
level of the model structure. The simple probability transfer learning
structure in ${ESM^{2}}$ transfers the knowledge with a simple conditional
probability between adjacent tasks. The AITM model with attention module could
obtain more gains by the sequential dependence tasks, which achieved
competitive results. Our proposed MLPR model obtained significant improvement
compared to various state-of-the-art baseline models and demonstrated the
effectiveness of the proposed multi-task learning architecture with the
domain-specific BERT for product ranking.

###  5.2. Ablation Study

This section discusses the effect of different components and stages of the
MLPR model.

####  5.2.1. Domain-specific BERT with Fine-tuning

We fine-tune the domain-specific BERT with the downstream multi-task learning.
The experimental results from Table 3 demonstrate that the fine-tuning of BERT
has significant improvement on each prediction task, either with the basic
$MLP_{MTL}$ model or with the MLPR model. Especially on the CTR prediction
task, even the basic $MLP_{MTL}$ model with fine-tuning yielded $1.41\%$ of
improvement over the one without fine-tuning in AUC score, and $7.03\%$ of
improvement in NDCG@1.

We investigated some specific queries and their search results from XGBoost
and our model respectively. For example, given the query “half bed for kids”,
the top result returned by XGBoost was the product with the title “The
cincinnati Kid POSTER (22x28) (1965) (Half Sheet Style A)”, due to the lexical
matching between the query and the title, but this item was not relevant at
all. On the other hand, our model returned the product “Bedz King Stairway
Bunk Beds Twin over Full with 4 Drawers in the Steps and a Twin Trundle, Gray”
as the top result. As we can see, the title did not have as much lexical
overlap with the given query as the previous product had, but it was
semantically relevant to the query. This example demonstrated the
effectiveness of the proposed BERT based method in bridging the vocabulary
gap.

Model | AUC | AUC Gain | NDCG@1 | NDCG@1 Gain |
---|---|---|---|---|---
Click | ATC | Purchase | Click | ATC | Purchase |
$MLP_{MTL}$ W/O Fine-tuning | +3.77% | +2.78% | +0.20% | N/A, N/A, N/A | +8.28% | +4.68% | +0.28% | N/A, N/A, N/A |
$MLP_{MTL}$ With Fine-tuning | +5.23% | +3.24% | +0.59% | 1.41%,0.45%,0.39% | +15.91% | +9.03% | +3.93% | 7.03%, 4.14%, 3.74% |
MLPR W/O Fine-tuning | +5.63% | +4.05% | +0.66% | N/A, N/A, N/A | +10.14% | +7.22% | +3.28% | N/A, N/A, N/A |
MLPR With Fine-tuning | +6.48% | +4.66% | +1.03% | 0.81%,0.58%,0.37% | +17.22% | +10.61% | +5.36% | 6.44%,3.19%,2.02% |
Table 3. Experimental results of the domain-specific BERT with fine-tuning vs without (W/O) fine-tuning in the basic MTL model and the MLPR model. AUC and NDCG@1 are reported in terms of the percentage lift over XGBoost. AUC Gain and NDCG@1 Gain are reported in terms of the percentage lift for fine-tuning over without fine-tuning. Model | AUC | AUC Gain | NDCG@1 | NDCG@1 Gain
---|---|---|---|---
Click | ATC | Purchase | Click | ATC | Purchase
$MLP_{MTL}$ | +3.78% | +2.70% | +0.03% | N/A, N/A, N/A | +8.81% | +4.97% | -0.28% | N/A, N/A, N/A
+Uncertainty Loss | +3.77% | +2.78% | +0.20% | -0.01%, 0.08%, 0.17% | +8.28% | +4.68% | +0.28% | -0.48%, -0.28%, 0.56%
+Specific-Experts | +5.80% | +3.63% | +0.56% | 1.94%, 0.91%, 0.53% | +10.14% | +6.31% | +3.28% | 1.22%, 1.26%, 3.57%
+Attention Units | +5.39% | +3.78% | +0.63% | 1.55%, 1.06%, 0.60% | +9.85% | +6.80% | +3.31% | 0.95%, 1.75%, 3.60%
+Probability Transfer | +5.69% | +4.03% | +0.67% | 1.84%, 1.30%, 0.64% | +9.85% | +6.85% | +3.31% | 0.96%, 1.80%, 3.59%
+Fine-tuning | +6.48% | +4.66% | +1.03% | 2.60%, 1.91%, 1.00% | +17.22% | +10.61% | +5.36% | 7.74%, 5.39%, 5.65%
Table 4. Experimental results of incrementally adding individual components to
the base $MLP_{MTL}$ model. AUC and NDCG@1 are reported in terms of the
percentage lift over XGBoost. AUC Gain and NDCG@1 Gain are reported in terms
of the percentage lift over the base $MLP_{MTL}$ model. The best results on
each task are highlighted.

####  5.2.2. The Effect of Different Stages of MLPR

In order to understand the performance of each stage of our framework, we
investigate the individual components of the MLPR model by incrementally
adding a new component to the base multi-task learning model. We define each
component in our model as follows:

  * •

$\mathbf{MLP_{MTL}}$ The base model. The shared-bottom stage design as multi-
layer feed-forward network and the upper stage design as tower network for
each task.

  * •

+Uncertainty Loss The same structure with $MLP_{MTL}$ but with a different
loss function (uncertainty loss) for training.

  * •

+Specific-Experts Based on the previous $MLP_{MTL}$ structure and the
uncertainty loss, the model adds a new component (specific-experts with
customized gating) to the shared-bottom stage.

  * •

+Attention Units Based on the previous model structure, the model adds the
attention units after the tower model in the upper stage of the model.

  * •

+Probability Transfer The model implements the probability transfer component
based on the previous design, which regularizes the predicted results from the
attention units.

  * •

+Fine-tuning The fine-tuning process is applied based on the previous model.
The parameter within the domain-specific BERT will be updated by the top-level
optimization function.

Table 4 contains the experimental results. As we can see, with the Uncertainty
Loss, the model obtained slightly better while comparable results overall.
Further with the Specific-Experts layer, the model performance significantly
improved, especially for the AUC score of CTR prediction, with a 1.94%
increase. Because the specific experts could extract more confidential
information than the simple shared-bottom design, the specific experts stage
not only extracts the common knowledge from different tasks but also learns
the specific information for each individual task. The model with Attention
Units on the upper level also demonstrated good improvement over the base
model. Moreover, the probability transfer component showed positive results,
as it optimized the joint predictions for the multiple tasks. Last but not the
least, with the benefits of domain-specific BERT, the model learned valuable
information from the text field. With the fine-tuning process, the model
gained the best result, which demonstrated the effectiveness of using the
fine-tuned BERT for product search.

Model | Deployment Strategy | Time
---|---|---
XGBoost | W/O query/product embedding | 58ms
$MLP_{MTL}$ | product pre-computing | 96ms
MLPR | W/O product pre-computing | 171ms
MLPR | product pre-computing | 112ms
Table 5. The latency in milliseconds (ms) at 99 percentile on product ranking.

###  5.3. Latency Performance

To understand the efficiency of MLPR in inference, we conducted an analysis on
latency by experimenting with four models. We used the 99th-percentile (P99)
of product ranking time as the latency metric, which was measured from the
time the model received the query to the time it returned a ranked list of 100
products. The experiments were performed on an Intel(R) Xeon(R) CPU E5-2660 v4
@ 2.00GHz machine and NVIDIA Tesla V100 GPU with 16G memory.

The offline P99 latency on the test set is reported in Table 5. As we can see,
XGBoost had the lowest running time among the four models, since it did not
compute query and product embeddings. The MLPR model could save a significant
amount of time when using pre-computing of product embeddings, as the
inference time was dropped from 171ms to 112ms. With product pre-computing,
MLPR was slightly slower than $MLP_{MTL}$ while MLPR has a more sophisticated
architecture with a higher accuracy as shown in Table 2. The experimental
results demonstrated the efficiency of the proposed multi-task learning with
the pre-computing strategy.

###  5.4. Transfer Knowledge Gain

In order to understand the transfer knowledge gained through different tasks,
we compared the model performance with different sampling strategies to
demonstrate that our model has a robust generalization ability. Firstly, we
sorted the query-item pairs in the test dataset according to the number of
impressions and divided the test dataset into three groups based on the
percentiles, i.e., top 0%-25%, 25%-75%, and 75%-100% of the sorted test
dataset. The top 0%-25% data corresponds to the portion of the test query-item
pairs with the least impressions and the top 75%-100% data includes the test
instances with the most impressions.

As the results shown in Figure 2, our proposed model demonstrated relatively
stable performance across the three different groups. It could achieve
reasonable performance even when less engagement data was used. On the other
hand, compared to the baseline models, our model has improved on different
tasks in both AUC and NDCG metrics, especially in the top 0%-25% group.

0%-25%25%-75%75%-100%$1$$2$$3$$4$$5$$\cdot 10^{-2}$Percentile
GroupAUC$MLP_{Single}$$MLP_{MTL}$MLPR (a) AUC percentage lift

0%-25%25%-75%75%-100%$1.5$$2$$2.5$$3$$\cdot 10^{-2}$Percentile
GroupNDCG@1$MLP_{Single}$$MLP_{MTL}$MLPR (b) NDCG@1 percentage lift

Figure 2. Transfer knowledge gain of each percentile group with different
models over XGBoost model.

###  5.5. Hyperparameter Analysis

In this section, we perform an analysis on two important hyper-parameters of
the proposed deep learning architecture: the dropout ratio and the number of
the hidden layers.

####  5.5.1. Dropout Ratio

If the model has too many parameters and too few training samples, the trained
model is likely to overfit (Srivastava et al., 2014b). As a widely used
technique to alleviate overfitting in neural nets, the dropout mechanism
(Srivastava et al., 2014a) can randomly deactivate some neural nodes. This
method can diminish the interaction between hidden layer nodes, and improve
the model’s generalization ability. We experimented with different dropout
ratios, ranged from 0.2 to 0.8. As we can see from the results shown in Figure
3(a), when the dropout ratio is 0.2, the model obtained the best performance.
In all the other experiments, we used 0.2 as the dropout ratio for MLPR unless
specified otherwise.

####  5.5.2. Number of the Hidden Layers and Nodes

In a deep neural model, increasing the number of layers of the network can
generally increase the model capacity (Lin et al., 2014). However, it will
also increase the number of model parameters, which may result in overfitting.
In the experiments, we tried different number of layers ranged from 2 to 4 in
the MLP component of the underlying expertise network. As shown in Figure
3(b), with the increase of the number of layers, the model performance was
improved at the beginning but then declined, which indicated the network may
start to overfit. Thus, we chose a 3-layer network as the MLP component
structure. In addition, we tested the number of nodes in different hidden
layers. We found that with the increase of the number of hidden layer nodes,
the model generally performed better. In all the other experiments, we used
[512, 256, 128] as the numbers of the nodes in the hidden layers.

$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$$0.8$$0$$1$$2$$3$$4$$5$$6$$\cdot
10^{-2}$AUCClickATCPurchase (a) Dropout ratio

$2$$3$$4$$0$$1$$2$$3$$4$$5$$6$$\cdot 10^{-2}$ClickATCPurchase (b) The number
of hidden layers

Figure 3. The AUC percentage lift of the MLPR model over the XGBoost model
with various values of the dropout ratio and the number of hidden layers.

##  6\. Conclusion and Future Work

In this paper, we propose a novel multi-task learning framework for product
ranking by integrating multiple types of engagement signals with neural
information retrieval. The proposed end-to-end learning framework incorporates
a domain-specific BERT for semantic match with traditional ranking features.
The comprehensive experiments on a real-world e-commerce dataset have
demonstrated the effectiveness of the proposed approach.

This work is an initial step towards a promising research direction. The
proposed framework allows flexible configuration, such as input data, text
embedding extraction, mixture-of-experts, loss function, etc. In the future,
we will integrate other types of input data into the model such as product
images. In addition, we will conduct A/B testing to validate the performance
of the proposed model in the online environment. Last but no the least, we
will apply the proposed framework to other search and ranking tasks.

## References

  * (1)
  * Bi et al. (2019) Keping Bi, Choon Hui Teo, Yesh Dattatreya, Vijai Mohan, and W. Bruce Croft. 2019.  Leverage Implicit Feedback for Context-aware Product Search. In _SIGIR_ _(CEUR Workshop, Vol. 2410)_.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A Scalable Tree Boosting System. In _SIGKDD_. ACM, 785–794.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper Text Understanding for IR with Contextual Neural Language Modeling. In _SIGIR_. ACM, 985–988.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT_. Association for Computational Linguistics, 4171–4186.
  * Duan et al. (2013) Huizhong Duan, ChengXiang Zhai, Jinxing Cheng, and Abhishek Gattani. 2013.  Supporting Keyword Search in Product Database: A Probabilistic Approach.  _VLDB_ 6, 14 (2013), 1786–1797.
  * Gao et al. (2019) Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, and Depeng Jin. 2019\.  Neural Multi-task Recommendation from Multi-behavior Data. In _ICDE_. IEEE, 1554–1557.
  * Gao et al. (2021) Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, Lina Yao, Yang Song, and Depeng Jin. 2021\.  Learning to Recommend With Multiple Cascading Behaviors.  _IEEE TKDE_ 33, 6 (2021), 2588–2601.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep learning_.  MIT press.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016\.  A Deep Relevance Matching Model for Ad-hoc Retrieval. In _CIKM_. ACM, 55–64.
  * Gysel et al. (2016) Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2016.  Learning Latent Vector Spaces for Product Search. In _CIKM_. ACM, 165–174.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _CIKM_. ACM, 2333–2338.
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018.  Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In _CVPR_. IEEE, 7482–7491.
  * LeCun et al. (1989) Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989.  Backpropagation applied to handwritten zip code recognition.  _Neural computation_ 1, 4 (1989), 541–551.
  * Lin et al. (2014) Min Lin, Qiang Chen, and Shuicheng Yan. 2014.  Network In Network.  _CoRR_ abs/1312.4400 (2014).
  * Long et al. (2012) Bo Long, Jiang Bian, Anlei Dong, and Yi Chang. 2012\.  Enhancing product search by best-selling prediction in e-commerce. In _CIKM_. ACM, 2479–2482.
  * Ma et al. (2018b) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018b.  Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts. In _SIGKDD_. ACM, 1930–1939.
  * Ma et al. (2018a) Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018a.  Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In _SIGIR_. ACM, 1137–1140.
  * Magnani et al. (2019) Alessandro Magnani, Feng Liu, Min Xie, and Somnath Banerjee. 2019\.  Neural Product Retrieval at Walmart.com. In _WWW_. ACM, 367–372.
  * Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008.  _Introduction to Information Retrieval_.  Cambridge University Press, USA.
  * Mitra et al. (2018) Bhaskar Mitra, Nick Craswell, et al. 2018\.  An introduction to neural information retrieval.  _Foundations and Trends in Information Retrieval_ (2018).
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _CoRR_ abs/1901.04085 (2019).  arXiv:1901.04085 http://arxiv.org/abs/1901.04085
  * Onal et al. (2018) Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md. Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, Aaron Angert, Edward Banner, Vivek Khetan, Tyler McDonnell, An Thanh Nguyen, Dan Xu, Byron C. Wallace, Maarten de Rijke, and Matthew Lease. 2018\.  Neural information retrieval: at the end of the early years.  _Information Retrieval Journal_ 21, 2-3 (2018), 111–182.
  * Palangi et al. (2016) Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab K. Ward. 2016\.  Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 24, 4 (2016), 694–707.
  * Parikh and Sundaresan (2011) Nish Parikh and Neel Sundaresan. 2011.  Beyond relevance in marketplace search. In _CIKM_. ACM, 2109–2112.
  * Qiao et al. (2019) Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019\.  Understanding the Behaviors of BERT in Ranking.  _CoRR_ abs/1904.07531 (2019).  arXiv:1904.07531 http://arxiv.org/abs/1904.07531
  * Ruder (2017) Sebastian Ruder. 2017\.  An Overview of Multi-Task Learning in Deep Neural Networks.  _CoRR_ abs/1706.05098 (2017).  arXiv:1706.05098 http://arxiv.org/abs/1706.05098
  * Sarvi et al. (2020) Fatemeh Sarvi, Nikos Voskarides, Lois Mooiman, Sebastian Schelter, and Maarten de Rijke. 2020\.  A Comparison of Supervised Learning to Match Methods for Product Search. In _SIGIR_. ACM.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In _ICLR_.
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval. In _CIKM_. ACM, 101–110.
  * Srivastava et al. (2014a) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014a.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  _JMLR_ 15, 56 (2014), 1929–1958.  http://jmlr.org/papers/v15/srivastava14a.html
  * Srivastava et al. (2014b) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014b.  Dropout: a simple way to prevent neural networks from overfitting.  _JMLR_ 15, 1 (2014), 1929–1958.
  * Tang et al. (2020) Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020\.  Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. In _RecSys_. ACM, 269–278.
  * Trotman et al. (2017) Andrew Trotman, Jon Degenhardt, and Surya Kallumadi. 2017\.  The Architecture of eBay Search. In _SIGIR_ _(CEUR Workshop, Vol. 2311)_.
  * Vandenhende et al. (2021) Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. 2021.  Multi-Task Learning for Dense Prediction Tasks: A Survey.  _IEEE PAMI_ PP (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NeurIPS_. 5998–6008.
  * Wen et al. (2020) Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. 2020.  Entire Space Multi-Task Modeling via Post-Click Behavior Decomposition for Conversion Rate Prediction. In _SIGIR_. ACM, 2377–2386.
  * Wu et al. (2018) Liang Wu, Diane Hu, Liangjie Hong, and Huan Liu. 2018\.  Turning Clicks into Purchases: Revenue Optimization for Product Search in E-Commerce. In _SIGIR_. ACM, 365–374.
  * Xi et al. (2021) Dongbo Xi, Zhen Chen, Peng Yan, Yinger Zhang, Yongchun Zhu, Fuzhen Zhuang, and Yu Chen. 2021.  Modeling the Sequential Dependence among Audience Multi-step Conversions with Multi-task Learning in Targeted Display Advertising. In _SIGKDD_. ACM, 3745–3755.
  * Zamani et al. (2018) Hamed Zamani, Bhaskar Mitra, Xia Song, Nick Craswell, and Saurabh Tiwary. 2018.  Neural Ranking Models with Multiple Document Fields. In _WSDM_. ACM, 700–708.
  * Zhang et al. (2019) Yuan Zhang, Dong Wang, and Yan Zhang. 2019.  Neural IR Meets Graph Embedding: A Ranking Model for Product Search. In _WWW_. ACM, 2390–2400.