# Hidden Real Topology and Unusual Magnetoelectric Responses in Monolayer
Antiferromagnetic Cr2Se2O

Jialin Gong  School of Physical Science and Technology, Southwest University,
Chongqing 400715, China  Yang Wang  Key Lab of advanced optoelectronic quantum
architecture and measurement (MOE), Beijing Key Lab of Nanophotonics $\&$
Ultrafine Optoelectronic Systems, and School of Physics, Beijing Institute of
Technology, Beijing 100081, China  Yilin Han  Key Lab of advanced
optoelectronic quantum architecture and measurement (MOE), Beijing Key Lab of
Nanophotonics $\&$ Ultrafine Optoelectronic Systems, and School of Physics,
Beijing Institute of Technology, Beijing 100081, China  Zhenxiang Cheng
Institute for Superconducting and Electronic Materials (ISEM), University of
Wollongong, Wollongong 2500, Australia  Xiaotian Wang  School of Physical
Science and Technology, Southwest University, Chongqing 400715, China
Institute for Superconducting and Electronic Materials (ISEM), University of
Wollongong, Wollongong 2500, Australia  Zhi-Ming Yu  Key Lab of advanced
optoelectronic quantum architecture and measurement (MOE), Beijing Key Lab of
Nanophotonics $\&$ Ultrafine Optoelectronic Systems, and School of Physics,
Beijing Institute of Technology, Beijing 100081, China  Yugui Yao  Key Lab of
advanced optoelectronic quantum architecture and measurement (MOE), Beijing
Key Lab of Nanophotonics $\&$ Ultrafine Optoelectronic Systems, and School of
Physics, Beijing Institute of Technology, Beijing 100081, China

###### Abstract

Recently, the real topology has been attracting widespread interest in two
dimensions (2D). Here, based on first-principles calculations and theoretical
analysis, we reveal the monolayer Cr2Se2O (ML-CrSeO) as the first material
example of a 2D antiferromagnetic (AFM) real Chern insulator (RCI) with
topologically protected corner states. Unlike previous RCIs, we find that the
real topology of the ML-CrSeO is rooted in one certain mirror subsystem of the
two spin channels, and can not be directly obtained from all the valence bands
in each spin channel as commonly believed. In particular, due to
antiferromagnetism, the corner modes in ML-CrSeO exhibit strong corner-
contrasted spin polarization, leading to spin-corner coupling (SCC). This SCC
enables a direct connection between spin space and real space. Consequently,
large and switchable net magnetization can be induced in the ML-CrSeO nanodisk
by electrostatic means, such as potential step and in-plane electric field,
and the corresponding magnetoelectric responses behave like a sign function,
distinguished from that of the conventional multiferroic materials. Our work
considerably broadens the candidate range of RCI materials, and opens up a new
direction for topo-spintronics and 2D AFM materials research.

Materials with nontrivial real band topology Zhao and Lu (2017); Ahn  _et al._
(2018); Wu  _et al._ (2019); Luo  _et al._ (2022) have become a focus of
current physics research. As a definition, the real topological phases feature
real band eigenstates Zhao and Lu (2017); Ahn  _et al._ (2018), which can be
guaranteed by certain symmetries, such as the presence of the spacetime
inversion symmetry (${\cal{PT}}$) and the absence of spin-orbit coupling (SOC)
effect. Furthermore, for 2D systems both with and without SOC, the combined
operator ${C_{2z}\cal{T}}$ (with $z$ being the direction normal to the 2D
plane) can also protect the real band topology Ahn and Yang (2019); Zhu  _et
al._ (2022).

Various real topological phases are proposed in nonmagnetic systems without
SOC, including 2D and 3D RCIs, Z2 nodal lines, Z2 nodal surfaces, real Weyl
and Dirac points Wu  _et al._ (2018); Ahn  _et al._ (2018); Sheng  _et al._
(2019); Zhao (2020); Wang  _et al._ (2020); Chen  _et al._ (2021, 2022); Pan
_et al._ (2022); Zhu  _et al._ (2022); Pan  _et al._ (2023); Xue  _et al._
(2023); Xiang  _et al._ (2023); Li  _et al._ (2023); Zhang  _et al._ (2023a),
and some of them have been experimentally realized Pan  _et al._ (2023); Xue
_et al._ (2023); Xiang  _et al._ (2023). Recently, Zhang _et. al._ Zhang  _et
al._ (2023a) demonstrated that the 2D RCI could also appear in ferromagnetic
systems regardless of the SOC strength. However, the 2D RCI with AFM order has
never been studied. Particularly, due to the magnetic but spin neural
property, one can expect that the AFM RCI may exhibit signatures distinct from
the conventional and ferromagnetic RCIs, especially for the magnetoelectric
responses Ma  _et al._ (2021); Zhang  _et al._ (2023b).

Generally, the real topology of a 2D insulator is characterized by a
$\mathbb{Z}_{2}$ real Chern number (also known as the second Stiefel-Whitney
number) $\nu_{R}$ Zhao and Lu (2017); Ahn  _et al._ (2018), which can be
evaluated from the eigenvalues of two-fold rotation $C_{2z}$ or spatial
inversion ${\cal{P}}$ of the valence bands at the four $C_{2z}$ (${\cal{P}}$)
invariant momentum points $\Gamma_{i}$ ($i=1,2,3,4$) Ahn  _et al._ (2018,
2019)

$(-1)^{\nu_{\mathrm{R}}}=\prod_{i=1}^{4}(-1)^{\left\lfloor\left(n_{i,-}\right)/2\right\rfloor},$
(1)

where $\lfloor\cdots\rfloor$ is the floor function, and $n_{i,-}$ is the
number of valence states at four momentum points $\Gamma_{i}$ which have
negative eigenvalues. This formula (1) successfully predicts many RCI and
topological Z2 nodal line materials Chen  _et al._ (2022, 2021); Sheng  _et
al._ (2019); Zhu  _et al._ (2022); Li  _et al._ (2023); Pan  _et al._ (2022);
Wang  _et al._ (2020); Pan  _et al._ (2023); Xue  _et al._ (2023); Xiang  _et
al._ (2023); Li  _et al._ (2023); Zhang  _et al._ (2023a).

In this work, we show that there is a caveat for calculating $\nu_{R}$ of the
system that can be divided into two or multiple _independent_ subsystems, due
to the floor function. For example, assuming the numbers of the valence states
with negative $C_{2z}$ eigenvalue in two independent subsystems are
respectively $n^{(1)}_{-}=\\{6,6,5,4\\}$, and $n^{(2)}_{-}=\\{4,4,5,6\\}$ at
the four $C_{2z}$-invariant points. For each subsystem, one can obtain a real
Chern number by Eq. (1), which gives $\nu_{R}^{(1)}=0$ and $\nu_{R}^{(2)}=1$.
Therefore, the whole system is topologically nontrivial. However, this real
topology is missed if one directly applies Eq. (1) to the whole system, as
$n_{-}=n^{(1)}_{-}+n^{(2)}_{-}=\\{10,10,10,10\\}$, which gives a trivial real
Chern number $\nu_{R}=0$. This simple case tells us that for the materials
hosting multiple independent subsystems, since Eq. (1) does not naturally
guarantee $\nu_{R}=\sum_{i}\nu_{R}^{(i)}\ \text{mod}\ 2$ (with $\nu_{R}^{(i)}$
denoting the real Chern number of the $i$-th subsystem), the real topology of
the whole system may be hidden in one certain subsystem, and needs to be
carefully examined.

Via first-principles calculations and theoretical analysis, we demonstrate the
aforementioned hidden real topology in ML-CrSeO, which has a horizontal mirror
$M_{z}$, and then two independent mirror subsystems. Particularly, the ground
state of the ML-CrSeO exhibits out-of-plane intralayer AFM order. This means
that the ML-CrSeO is also the first material example of a 2D AFM RCI. Compared
with the previously reported nonmagnetic and ferromagnetic RCIs Sheng  _et
al._ (2019); Zhu  _et al._ (2022); Chen  _et al._ (2021); Li  _et al._ (2023);
Pan  _et al._ (2022); Zhang  _et al._ (2023a), the topological corner states
in ML-CrSeO are spin-polarized, however, the direction of the spin
polarization is corner-dependent, leading to a unique SCC effect in 2D AFM
RCIs. The SCC can be directly observed by spin-resolved scanning tunneling
spectroscopy (STS) Wiesendanger (2009). Furthermore, it can fundamentally
affect the magnetoelectric responses of the systems. We find that by applying
electrostatic fields, the net magnetization of the ML-CrSeO can be efficiently
changed from zero to a finite constant. Remarkably, the sign of the finite
constant is solely determined by the direction of the electrostatic fields,
and is switched when changing the direction of the fields. Hence, the
magnetoelectric responses in ML-CrSeO behave as a sign function, distinguished
from those in the conventional multiferroic materials Eerenstein  _et al._
(2006) and the previously reported RCIs. This sign-function response also
indicates an electric switching of the net magnetization in ML-CrSeO, which is
important for device applications Matsukura  _et al._ (2015); Wadley  _et al._
(2016); Song  _et al._ (2017).

Figure 1:  (a) Side view of the crystalline structure of ML-CrSeO. (b) The
first Brillouin zone (BZ) of ML-CrSeO. The orange and blue regions are the two
irreducible parts of the first BZ. (c) Calculated phonon spectrum of ML-CrSeO.

_Structure and magnetic ordering.–_ The ML-CrSeO is a kind of oxyselenides.
The crystal structure of the ML-CrSeO is shown in Fig. 1(a), in which three
atomic sublayers with two Cr and one O atoms as the middle layer and two Se
atoms as the upper and lower layers are obvious. The crystal lattice has space
group (SG) P4/mmm (No. 123), which preserves $C_{4z}$, $C_{2x}$ and $M_{z}$
symmetries. The optimized lattice constant is calculated as $a=b=4.02$ Å. We
also find that the ML-CrSeO is both dynamically and thermodynamically stable,
as shown in Fig. 1(c) and the supplemental material (SM) SM .

Since the ML-CrSeO has transition metal elements, one can expect that it may
exhibit magnetic ordering in its ground state. Our calculations show each Cr
atom indeed has a finite magnetic moment, which is obtained as $\sim
3.1~{}\mu{{}_{B}}$. By comparing the energies of several typical types of
magnetic ordering SM , we find that the ground state of ML-CrSeO is a C-type
AFM with intralayer out-of-plane magnetization, i.e., the two Cr atoms in the
unit cell have opposite spin polarization, as shown in Fig. 2(b). With
magnetic ordering, the ML-CrSeO belongs to magnetic SG No. 123.342, which
keeps $C_{4z}{\cal{T}}$, $C_{2z}$ and $M_{z}$ symmetries, but breaks both
$C_{4z}$ and ${\cal{T}}$ Liu  _et al._ (2022); Yu  _et al._ (2022); Zhang  _et
al._ (2022). The combined operator $C_{4z}{\cal{T}}$ guarantees the spin
neutrality of the monolayer.

Figure 2: (a) Spin-resolved band structures for ML-CrSeO without SOC. (b) The
ground state of the ML-CrSeO, which is a intralayer out-of-plane (C-type) AFM.

_Electronic band and hidden real topology.–_ We then investigate the
electronic band structure of the ML-CrSeO with the AFM ground state. Since the
SOC effect is rather small in the ML-CrSeO SM , we focus on the band structure
without SOC.

When SOC is neglected, the two spin channels of the ML-CrSeO are decoupled but
not independent, as they are connected by $C_{4z}{\cal{T}}$. Each spin channel
can be regarded as an effective spinless system persevering ${\cal{T}}$.
Obviously, each spin channel does not have $C_{4z}$ symmetry and then the
crystal symmetry of SG 123. By detailed analysis, we find that it is the
crystal symmetry of SG 47 that is respected by each spin channel. The
generators of SG 47 are $C_{2x}$, $C_{2z}$ and $M_{z}$. Hence, the Hamiltonian
of the ML-CrSeO (without SOC) can be written as

${\cal{H}}={\cal{H}}_{\uparrow}\oplus{\cal{H}}_{\downarrow}={\cal{H}}_{0}\oplus
C_{4z}{\cal{H}}_{0}C_{4z}^{-1},$ (2)

where ${\cal{H}}_{0}$ denotes a spinless and nonmagnetic Hamiltonian
respecting the symmetry of SG 47. Since ${\cal{H}}_{0}$ has $C_{2x}$ and
${\cal{T}}$, one knows that the two spin channels should be degenerate along
$\Gamma$-M path.

Table 1:  $C_{2z}$ eigenvalues of all occupied bands at $\Gamma$, X, Y, and M points of the ML-CrSeO.  Spin-up | Spin-down |
---|---|---
| $\Gamma$ | X | Y | M |  |  | $\Gamma$ | X | Y | M
$n_{+}$ | 10 | 10 | 11 | 11 |  | $n_{+}$ | 10 | 11 | 10 | 11
$n_{-}$ | 11 | 11 | 10 | 10 |  | $n_{-}$ | 11 | 10 | 11 | 10
|  |  $\nu_{\mathrm{R}}$ = 0 |  |  |  |  |  |  $\nu_{\mathrm{R}}$ = 0 |  |
Table 2:  Mirror-resolved $C_{2z}$ eigenvalues of all occupied bands at $\Gamma$, X, Y, and M points of the ML-CrSeO. Both spin channels are considered.  Spin-up |  |  | Spin-down
---|---|---|---
| $M_{z}=1$ |  | $M_{z}=-1$ |  |  | $M_{z}=1$ |  | $M_{z}=-1$
| $\Gamma$ | X | Y | M |  | $\Gamma$ | X | Y | M |  |  | $\Gamma$ | X | Y | M |  | $\Gamma$ | X | Y | M
$n_{+}$ | 5 | 6 | 8 | 7 |  | 5 | 4 | 3 | 4 |  | $n_{+}$ | 5 | 8 | 6 | 7 |  | 5 | 3 | 4 | 4
$n_{-}$ | 8 | 7 | 5 | 6 |  | 3 | 4 | 5 | 4 |  | $n_{-}$ | 8 | 5 | 7 | 6 |  | 3 | 5 | 4 | 4
| $\nu_{R}=0$ |  | $\nu_{R}=1$ |  |  | $\nu_{R}=0$ |  | $\nu_{R}=1$

The spin-resolved band structures for the ML-CrSeO are shown in Fig. 2(a), in
which the spin-up and spin-down bands are degenerate on the high-symmetry line
$\Gamma$-M (M’), consistent with the above symmetry analysis. Besides, the
spin-up and spin-down bands are split at other momenta in BZ. Recently, the
systems with spin-splitting band structures are also termed as altermagnetic
materials Šmejkal  _et al._ (2022a); Feng  _et al._ (2022); Šmejkal  _et al._
(2022, 2022b); Bai  _et al._ (2023); Ouassou  _et al._ (2023). For both spin
channels, large gaps of 1.12 eV appear. Then, we can define a real Chern
number $\nu_{R}^{\sigma}$ (with $\sigma=\uparrow,\downarrow$) for each spin
subspace, as it preserves both $C_{2z}$ and ${\cal{T}}$ symmetries. Since
$C_{4z}$ commutes with $C_{2z}$, one always has
$\nu_{R}^{\uparrow}=\nu_{R}^{\downarrow}$. The results of the $C_{2z}$
eigenvalues of valence bands at the four $C_{2z}$-invariant momentum points
are listed in Table 1. Obviously, according to Eq. (1), both spin channels are
topologically trivial with $\nu_{R}^{\uparrow}=\nu_{R}^{\downarrow}=0$.
However, as aforementioned, this result may not be correct if the spin
channels can be further divided into independent subsystems.

The ML-CrSeO is exactly the case, as it has a horizontal mirror $M_{z}$ and
then can be divided into two independent subsystems based on the eigenvalue of
$M_{z}$, i.e.,

${\cal{H}}_{\uparrow(\downarrow)}={\cal{H}}_{\uparrow(\downarrow)}^{+}\oplus{\cal{H}}_{\uparrow(\downarrow)}^{-},$
(3)

where ${\cal{H}}_{\uparrow(\downarrow)}^{\pm}$ denotes the subsystem with
$M_{z}=\pm 1$ of spin-up (spin-down) channel. All the mirror subsystems also
have both $C_{2z}$ and ${\cal{T}}$. By analysing the mirror eigenvalue of the
bands in each spin channel, we obtain a mirror-resolved band structure for the
ML-CrSeO [see Fig. 3(a)]. For each mirror subsystem, we count the bands with
negative $C_{2z}$ eigenvalue, and the result is listed in Table 2. Using Eq.
(1), we find that for both spin-up and spin-down channels, the mirror
subsystem with $M_{z}=-1$ has a nontrivial $\nu_{R}^{-}=1$, while the
$M_{z}=1$ subsystem is trivial with $\nu_{R}^{+}=0$. Consequently, the _exact_
real Chern number for each spin channel is
$\nu_{R}^{\uparrow}=\nu_{R}^{\downarrow}=\nu_{R}^{+}+\nu_{R}^{-}=1$. This
confirms that the ML-CrSeO is a material candidate for AFM RCI with hidden
real topology.

Figure 3: (a) Mirror-resolved band structure and density of states (DOS) of
the spin-up channel of the ML-CrSeO. The spin-down channel has a similar band
structure, as the two spin channels are connected by $C_{4z}{\cal{T}}$
symmetry. (b) Mirror-resolved energy spectrum for the spin-up channel on a
quadrangle-shaped disk. Orange and green curves and dots in (a) and (b) denote
the states with $M_{z}=-1$ and $M_{z}=1$, respectively. We also plot the
spectrum and the spatial charge distribution of the corner states as insets in
(b). (c) The energy spectrum for the whole ML-CrSeO on a quadrangle-shaped
disk, where four corner states (the colored dots) as a group can be observed.
(d) Spatial distribution of the corner states. The blue and red dots (circles)
in (c) [(d)] correspond to the spin-up and spin-down corner states,
respectively.

_Corner states and spin-corner coupling.–_ According to the bulk-boundary
correspondence, the nontrivial real Chern number $\nu_{R}^{-}=1$ will lead to
protected corner states on a pair of $C_{2z}$-related intersections in each
spin channel, and the corner states should have a mirror eigenvalue of
$M_{z}=-1$. Moreover, since the spin-up and spin-down channels are connected
by $C_{4z}{\cal{T}}$, there genetically exhibit four corner states as a group
in the bulk band gap: two from spin-up channels and two from spin-down
channels.

To demonstrate this feature, a wannier tight-binding model for a quadrangle
nanodisk composed of 64 unit cells was constructed based on the ML-CrSeO with
the preservation of $C_{4z}{\cal{T}}$ symmetry, as shown in Fig. 3(d). The
obtained discrete energy spectrum of the quadrangle nanodisk is plotted in
Fig. 3(c), in which four degenerate states in the bulk band gap are observed.
By analysing the wave function of the four states, we confirm that they are
indeed corner states, as they are localized at the four corners of the disk
[see Fig. 3(d)]. Moreover, all these corner states are from the $M_{z}=-1$
subsystems [see Fig. 3(b)] and are fully spin polarized, as the SOC effect is
neglected during the calculations. We would like to point out that both the
existence and the spin polarization of the corner states are robust against
the SOC effect SM .

Remarkably, the corner states at $x$-axis (labelled as X corner) are spin-
down, whereas those at $y$-axis (labelled as Y corner) are spin-up, as shown
in Fig. 3(d). This corner-contrasted spin polarization directly couples the
real space and spin space, leading to a novel SCC effect. Since the corner
states are well separated in real space (for example, the distance between
every two corners in Fig. 3(d) is larger than $3.2$ nm), this provides great
convenience for experimentally controlling the spins at each corner
independently. Moreover, as electronic states at the Fermi level play a key
role in many material properties, the independent control of the spin in each
corner is expected to exhibit distinctive signatures.

Figure 4: SCC-induced electric control of the magnetization. (a) Schematic
figure showing potential-step control of the magnetization of the ML-CrSeO
disk. (b) The energy spectrum for the ML-CrSeO disk under a potential step
$U=0.01$ eV. (c) Net magnetization as a function of $U$. (d) With corner-
dependent spin polarization, an in-plane electric field can lead to net
magnetization in the ML-CrSeO disk. (e) The energy spectrum for the ML-CrSeO
disk under an electric field of $E_{x}=0.00025$ eV/Å, corresponding to
$\theta=0$. (f) Net magnetization as a function of $\theta$. The blue and red
circles (dots) are spin-up and spin-down corner states.

_Novel magnetoelectric responses.–_ We then investigate the magnetoelectric
responses in the ML-CrSeO. A unique feature of AFM materials is that it is
magnetic but spin-neutral Baltz  _et al._ (2018); Yan  _et al._ (2020). This
provides an ideal platform for generating net magnetization and spin current
by nonmagnetic methods. As studied in previous works Šmejkal  _et al._ (2017);
Ma  _et al._ (2021); Shao  _et al._ (2021); González-Hernández  _et al._
(2021); Zhang  _et al._ (2023b); Shao  _et al._ (2023), these novel
magnetoelectric responses generally require certain bulk band structures like
spin-layer coupling Zhang  _et al._ (2023b) and C-paired spin-valley locking
Ma  _et al._ (2021), which, however, are absent in the ML-CrSeO.

In contrast, we investigate the magnetoelectric responses of a ML-CrSeO
nanodisk with a novel SCC effect. While the corner states in ML-CrSeO are
fully spin polarized, the nanodisk does not have net magnetization as
guaranteed by the $C_{4z}{\cal{T}}$ symmetry. Hence, by breaking the
$C_{4z}{\cal{T}}$, the ML-CrSeO nanodisk generally has finite net
magnetization. There are many nonmagnetic ways to realize this symmetry
breaking, such as uniaxial strain, applying nonmagnetic substrate without
$C_{4z}$ symmetry and interface engineering. Among them, the electrostatic
means may be the most convenient in experiments and applications Yu  _et al._
(2020). Here, we propose two electrostatic means to achieve net magnetization
in the ML-CrSeO nanodisk.

As the four corners of the nanodisk are well separated, we can apply a
potential step to make each corner to have different potential energy.
Specifically, we assume that the potential step satisfies $V=-U$ for the X
corners and $V=+U$ for the Y corners, as illustrated in Fig. 4(a). The
calculated spectrum of the nanodisk with $U=0.01$ eV is shown in Fig. 4(b). As
expected, the degenerate of the X and Y corners is broken. The Y (X) corners
increase (decrease) in energy, but the bulk states are less affected. Hence,
with half doping of the corner states, only the X corners are occupied, and
then the net magnetization of the system is $M=-2\mu_{B}$. Moreover, the net
magnetization can be switched by reversing the value of $U$. Consequently, the
net magnetization takes the form of a sign function by varying the
electrostatic potential, as shown in Fig. 4(c).

We can also use in-plane electric field
${{\bm{E}}}_{\parallel}=\\{E\cos\theta,E\sin\theta\\}$ to produce a corner-
dependent electrostatic potential [see Fig. 4(d)]. When the electric field is
along the $x$-direction ($\theta=0$), the four corner states are split into
three groups in energy: the -X (+X) corner has the lowest (highest) energy,
while the two Y corners are in the middle [see Fig. 4(e)]. Then, with $1/4$
doping of the corner states, only the -X corner is occupied, and then the net
magnetization of the system is $M=-1\mu_{B}$. Similarly, if $\theta=\pi/2$,
one would have $M=1\mu_{B}$. Our detailed calculations show that the net
magnetization has a sign-function dependence on the rotation angle of the
electric field, as shown in Fig. 4(f). These sign functions of the
magnetoelectric responses are completely different from the conventional
magnetoelectric responses, suggesting the potential of the AFM RCIs for novel
device operations.

_Discussion.–_ In this work, we have unveiled a hidden real topology in a 2D
AFM material with mirror symmetry. The underlying physics is to study the real
topology in each independent subsystem, rather than the whole system. We then
predict the ML-CrSeO as the first material example of the AFM RCI with hidden
real topology. We also show that the 2D AFM RCI exhibits the SCC effect, which
enables a direct control of the net magnetization of the system by both local
and global electric methods.

Besides the ML-CrSeO, we find six other 2D AFM materials, including Cr2Br2,
Mn2Br2, V2Br2, Mn2F2, Cr2Cl2, and V2Se2O as the AFM RCIs with hidden real
topology SM . Interestingly, the Cr2Br2, Cr2Cl2, and V2Se2O are also
valleytronics materials Schaibley  _et al._ (2016); Li  _et al._ (2019); Chu
_et al._ (2021); Liu  _et al._ (2023) with two valleys located at X and Y
points in the BZ. Last but not least, it is necessary to re-examine the
previous 2D materials with a horizontal mirror. With the hidden real topology
proposed here, one can expect that many candidates previously diagnosed as
trivial may have a nontrivial real Chern number.

## References

  * Zhao and Lu (2017) Y. X. Zhao and Y. Lu, Physical Review Letters 118, 056401 (2017).
  * Ahn  _et al._ (2018) J. Ahn, D. Kim, Y. Kim, and B.-J. Yang, Physical Review Letters 121, 106403 (2018).
  * Wu  _et al._ (2019) Q. Wu, A. A. Soluyanov, and T. Bzdušek, Science 365, 1273 (2019).
  * Luo  _et al._ (2022) A. Luo, Z. Song, and G. Xu, npj Computational Materials 8, 26 (2022).
  * Ahn and Yang (2019) J. Ahn and B.-J. Yang, Physical Review B 99, 235125 (2019).
  * Zhu  _et al._ (2022) J. Zhu, W. Wu, J. Zhao, C. Chen, Q. Wang, X.-L. Sheng, L. Zhang, Y. X. Zhao, and S. A. Yang, Physical Review B 105, 085123 (2022).
  * Wu  _et al._ (2018) W. Wu, Y. Liu, S. Li, C. Zhong, Z.-M. Yu, X.-L. Sheng, Y. X. Zhao, and S. A. Yang, Physical Review B 97, 115125 (2018).
  * Sheng  _et al._ (2019) X.-L. Sheng, C. Chen, H. Liu, Z. Chen, Z.-M. Yu, Y. X. Zhao, and S. A. Yang, Physical Review Letters 123, 256402 (2019).
  * Zhao (2020) Y. Zhao, Frontiers of Physics 15, 1 (2020).
  * Wang  _et al._ (2020) K. Wang, J.-X. Dai, L. B. Shao, S. A. Yang, and Y. X. Zhao, Physical Review Letters 125, 126403 (2020).
  * Chen  _et al._ (2021) C. Chen, W. Wu, Z.-M. Yu, Z. Chen, Y. X. Zhao, X.-L. Sheng, and S. A. Yang, Physical Review B 104, 085205 (2021).
  * Chen  _et al._ (2022) C. Chen, X.-T. Zeng, Z. Chen, Y. X. Zhao, X.-L. Sheng, and S. A. Yang, Physical Review Letters 128, 026405 (2022).
  * Pan  _et al._ (2022) M. Pan, D. Li, J. Fan, and H. Huang, npj Comput. Mater. 8, 1 (2022).
  * Pan  _et al._ (2023) Y. Pan, C. Cui, Q. Chen, F. Chen, L. Zhang, Y. Ren, N. Han, W. Li, X. Li, Z.-M. Yu, _et al._ , Nature Communications 14, 6636 (2023).
  * Xue  _et al._ (2023) H. Xue, Z. Chen, Z. Cheng, J. Dai, Y. Long, Y. Zhao, and B. Zhang, arXiv preprint arXiv:2304.06304 (2023).
  * Xiang  _et al._ (2023) X. Xiang, X. Ni, F. Gao, X. Wu, Z. Chen, Y.-G. Peng, and X.-F. Zhu, arXiv preprint arXiv:2304.12735 (2023).
  * Li  _et al._ (2023) J. Li, Y. Liu, J. Bai, C. Xie, H. Yuan, Z. Cheng, W. Wang, X. Wang, and G. Zhang, Applied Physical Review 10, 031416 (2023).
  * Zhang  _et al._ (2023a) X. Zhang, T. He, Y. Liu, X. Dai, G. Liu, C. Chen, W. Wu, J. Zhu, and S. A. Yang, Nano Letters 23, 7358 (2023a).
  * Ma  _et al._ (2021) H.-Y. Ma, M. Hu, N. Li, J. Liu, W. Yao, J.-F. Jia, and J. Liu, Nature Communications 12, 2846 (2021).
  * Zhang  _et al._ (2023b) R.-W. Zhang, C. Cui, R. Li, J. Duan, L. Li, Z.-M. Yu, and Y. Yao, arXiv preprint arXiv:2306.08902 (2023b).
  * Ahn  _et al._ (2019) J. Ahn, S. Park, D. Kim, Y. Kim, and B.-J. Yang, Chinese Physics B 28, 117101 (2019).
  * Wiesendanger (2009) R. Wiesendanger, Reviews of Modern Physics 81, 1495 (2009).
  * Eerenstein  _et al._ (2006) W. Eerenstein, N. Mathur, and J. F. Scott, Nature 442, 759 (2006).
  * Matsukura  _et al._ (2015) F. Matsukura, Y. Tokura, and H. Ohno, Nature Nanotechnology 10, 209 (2015).
  * Wadley  _et al._ (2016) P. Wadley, B. Howells, J. Železnỳ, C. Andrews, V. Hills, R. P. Campion, V. Novák, K. Olejník, F. Maccherozzi, S. Dhesi, _et al._ , Science 351, 587 (2016).
  * Song  _et al._ (2017) C. Song, B. Cui, F. Li, X. Zhou, and F. Pan, Progress in Materials Science 87, 33 (2017).
  * (27) See Supplemental Material (SM) for the computational methods, Monte Carlo simulations of temperature-dependent magnetization, spin-orbit coupling effect, and other AFM RCIs with hidden real topology, etc.
  * Liu  _et al._ (2022) G.-B. Liu, Z. Zhang, Z.-M. Yu, S. A. Yang, and Y. Yao, Phys. Rev. B 105, 085117 (2022).
  * Yu  _et al._ (2022) Z.-M. Yu, Z. Zhang, G.-B. Liu, W. Wu, X.-P. Li, R.-W. Zhang, S. A. Yang, and Y. Yao, Science Bulletin 67, 375 (2022).
  * Zhang  _et al._ (2022) Z. Zhang, G.-B. Liu, Z.-M. Yu, S. A. Yang, and Y. Yao, Phys. Rev. B 105, 104426 (2022).
  * Šmejkal  _et al._ (2022a) L. Šmejkal, J. Sinova, and T. Jungwirth, Physical Review X 12, 040501 (2022a).
  * Feng  _et al._ (2022) Z. Feng, X. Zhou, L. Šmejkal, L. Wu, Z. Zhu, H. Guo, R. González-Hernández, X. Wang, H. Yan, P. Qin, _et al._ , Nature Electronics 5, 735 (2022).
  * Šmejkal  _et al._ (2022) L. Šmejkal, A. H. MacDonald, J. Sinova, S. Nakatsuji, and T. Jungwirth, Nature Reviews Materials 7, 482 (2022).
  * Šmejkal  _et al._ (2022b) L. Šmejkal, J. Sinova, and T. Jungwirth, Physical Review X 12, 031042 (2022b).
  * Bai  _et al._ (2023) H. Bai, Y. C. Zhang, Y. J. Zhou, P. Chen, C. H. Wan, L. Han, W. X. Zhu, S. X. Liang, Y. C. Su, X. F. Han, F. Pan, and C. Song, Physical Review Letters 130, 216701 (2023).
  * Ouassou  _et al._ (2023) J. A. Ouassou, A. Brataas, and J. Linder, Physical Review Letters 131, 076003 (2023).
  * Baltz  _et al._ (2018) V. Baltz, A. Manchon, M. Tsoi, T. Moriyama, T. Ono, and Y. Tserkovnyak, Reviews of Modern Physics 90, 015005 (2018).
  * Yan  _et al._ (2020) H. Yan, Z. Feng, P. Qin, X. Zhou, H. Guo, X. Wang, H. Chen, X. Zhang, H. Wu, C. Jiang, _et al._ , Advanced Materials 32, 1905603 (2020).
  * Šmejkal  _et al._ (2017) L. Šmejkal, J. Železný, J. Sinova, and T. Jungwirth, Phys. Rev. Lett. 118, 106402 (2017).
  * Shao  _et al._ (2021) D.-F. Shao, S.-H. Zhang, M. Li, C.-B. Eom, and E. Y. Tsymbal, Nature Communications 12, 7061 (2021).
  * González-Hernández  _et al._ (2021) R. González-Hernández, L. Šmejkal, K. Výborný, Y. Yahagi, J. Sinova, T. c. v. Jungwirth, and J. Železný, Phys. Rev. Lett. 126, 127701 (2021).
  * Shao  _et al._ (2023) D.-F. Shao, Y.-Y. Jiang, J. Ding, S.-H. Zhang, Z.-A. Wang, R.-C. Xiao, G. Gurung, W. J. Lu, Y. P. Sun, and E. Y. Tsymbal, Phys. Rev. Lett. 130, 216702 (2023).
  * Yu  _et al._ (2020) Z.-M. Yu, S. Guan, X.-L. Sheng, W. Gao, and S. A. Yang, Phys. Rev. Lett. 124, 037701 (2020).
  * Schaibley  _et al._ (2016) J. R. Schaibley, H. Yu, G. Clark, P. Rivera, J. S. Ross, K. L. Seyler, W. Yao, and X. Xu, Nature Reviews Materials 1, 1 (2016).
  * Li  _et al._ (2019) L. Li, M. Wu, and X. C. Zeng, Advanced Functional Materials 29, 1905752 (2019).
  * Chu  _et al._ (2021) J. Chu, Y. Wang, X. Wang, K. Hu, G. Rao, C. Gong, C. Wu, H. Hong, X. Wang, K. Liu, _et al._ , Advanced Materials 33, 2004469 (2021).
  * Liu  _et al._ (2023) Y. Liu, Y. Feng, T. Zhang, Z. He, Y. Dai, B. Huang, and Y. Ma, Advanced Functional Materials , 2305130 (2023).


# Learning to Select Historical News Articles for Interaction based Neural
News Recommendation

Peitian Zhang<EMAIL_ADDRESS>Renmin University of ChinaBeijingChina100872 ,
Zhicheng Dou<EMAIL_ADDRESS>Renmin University of ChinaBeijingChina100872 and
Jing Yao<EMAIL_ADDRESS>Renmin University of ChinaBeijingChina100872

(2021)

###### Abstract.

The key to personalized news recommendation is to match the user’s interests
with the candidate news precisely and efficiently. Most existing approaches
embed user interests into a representation vector then recommend by comparing
it with the candidate news vector. In such a workflow, fine-grained matching
signals may be lost. Recent studies try to cover that by modeling fine-grained
interactions between the candidate news and each browsed news article of the
user. Despite the effectiveness improvement, these models suffer from much
higher computation costs online. Consequently, it remains a tough issue to
take advantage of effective interactions in an efficient way. To address this
problem, we proposed an end-to-end Selective Fine-grained Interaction
framework (SFI) with a learning-to-select mechanism. Instead of feeding all
historical news into interaction, SFI can quickly select informative
historical news w.r.t. the candidate and exclude others from following
computations. We empower the selection to be both sparse and automatic, which
guarantees efficiency and effectiveness respectively. Extensive experiments on
the publicly available dataset MIND validates the superiority of SFI over the
state-of-the-art methods: with only five historical news selected, it can
significantly improve the AUC by 2.17% over the state-of-the-art interaction-
based models; at the same time, it is four times faster.

News Recommendation, Interaction-based, Selection

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Nowadays, people are overwhelmed with information, exhausted to seek things
they’re interested in. Online news platforms e.g. MSN
News111https://www.msn.com/en-us/news greatly alleviate this information
overload problem by recommending news articles according to user’s specific
interests (Zheng et al., 2018; Wu et al., 2019b; Lian et al., 2020; Liu et
al., 2010). The key technology of these news platforms is personalized news
recommendation (Lavie et al., 2010). Due to the particular large-scale and
time-sensitive property of news, the news recommenders must be both effective
and efficient so that it can be deployed in real production systems.

A lot of existing news recommendation approaches (Okura et al., 2017; An et
al., 2019; Yu et al., 2019; Wu et al., 2019d, f, c; An et al., 2019; Wu et
al., 2019e, b; Lian et al., 2020) follow a representation-based matching
strategy. They learn a representation vector for the candidate news and encode
the user’s history news into a vector to form the user representation in the
same semantic space. The matching score between these vectors is calculated as
the click probability. However, the user vector is an aggregation of multiple
historical articles, so it hardly keeps the fine-grained information and may
contain noise in the articles. For example in Figure 1, the candidate news
matches user’s fine-grained interests Biden and Hillary (fine-grained
interaction happens), which motivates the user’s current click. Unfortunately,
the aggregated user vector mixes all terms in $\mathbf{h}_{1},\mathbf{h}_{2}$
and $\mathbf{h}_{3}$ and blurs these fine-grained interests, thus degrades the
capacity of user modeling. Even worse, noises such as wind and wildfires are
also included and they are unrelated to the current click. Though some recent
“multi-channel” methods (Liu et al., 2019; Chen et al., 2018) attempt to cover
richer information by maintaining multiple representation vectors, they are
still limited to modeling fine-grained interaction in an explicit and reliable
manner.

Figure 1. Example of a user’s behavior log in MSN. Three historical news
articles of the user are shown and $\mathrm{c}$ is the news she actually
clicked afterward. The text marked in the same color is the fine-grained
(term-term) matching signals. The darker the color, the more they match.

Illustration

In order to capture fine-grained matching signals between the candidate news
and the user, Wang et al. (Wang et al., 2020) proposed an interaction-based
model. It computes similarity matrices between the candidate news and every
historical news piece of the user at word level to derive the click
probability. Despite the effectiveness improvement, the model is especially
slow. It has to recompute term-level interaction matrices with every
historical article when scoring each candidate, which is far more expensive
than dot product used in representation-based methods. Intuitively, we
shouldn’t involve all the historical articles in interaction. For example,
$\mathbf{h}_{2}$ should be excluded within this click because it is irrelevant
to the current candidate $\mathbf{c}$ and there would be no interactions
between them. Tailoring the user history to several recent browsed news items
seems to be a straightforward solution to save efficiency. However, blindly
interacting with only the latest browsed news limits the recommendation
effectiveness, mainly because: 1) When the length of the kept browsing history
is short, there isn’t sufficient news for the model to learn the user’s
interests well. Back to the Figure 1, if we cut off earlier history news
$\mathrm{h}_{2}$ and $\mathrm{h}_{3}$, the interaction quality would greatly
decrease since the most informative one, $\mathrm{h}_{3}$, is lost. 2) When
the capacity becomes larger, irrelevant historical news, such as
$\mathrm{h}_{2}$, is involved. As mentioned above, such news is harmful to the
recommendation accuracy. It hardly contributes to motivate the click and would
act like noise, reducing the matching score of the candidate that the user’s
truly interested in.

To tackle the above problem, in this paper, we propose SFI, a Selective Fine-
grained Interaction framework. The key idea of it is to select a small number
of historical news articles with higher informativeness, then perform fine-
grained interaction over them only.

The biggest challenge of SFI is to select the informative history news
sparsely, precisely, and efficiently. Most previous works about feature
selection employ gating operator (Guo et al., 2016; Liu et al., 2020; Yuan et
al., 2019). But it cannot be directly used in SFI because gating doesn’t
eliminate zero entries so the total computations are not lessened. Two-stage
training (Hofstätter et al., 2021) is not desirable either, because no ground-
truth label (indicates which historical news interacts with the candidate) is
available to train the selector. Last but not least, we’d better manipulate
news-level representations to guide selection for the sake of efficiency. In
this work, we design the learning-to-select mechanism to fulfill all our
goals. Specifically, SFI learns a selection vector for each news article, and
computes the cosine similarity between candidate news and every history news
in the selection space, taking the result as the informativeness of each
historical article. Next, we design two successive selection networks. The
hard-selection network enforces sparsity. It selects $K$ most informative news
and excludes others from following interactions. Within the output of the
hard-selection, the following soft-selection network masks news whose
informativeness is below a given threshold and attaches different weights to
the unmasked ones. This refinement allows the gradient flow through to
optimize the selection vectors, so that the model can learn to highlight
valuable features for selection hence achieve higher effectiveness.

Extensive experiments on the publicly available dataset MIND show that SFI
outperforms all baselines in terms of both effectiveness and efficiency: with
only five historical news articles selected, it significantly improves the
recommendation effectiveness by 2.17% over the state-of-the-art interaction-
based models with four times faster speed (almost reaches the fastest speed of
representation-based methods and outperforms it by $2.71\%$ in AUC). We also
comprehensively compare SFI with its naive recent $K$ counterpart and
investigate the efficiency effectiveness trade-off brought by SFI.

The main contributions of this paper can be summarized into three aspects:

(1) We propose SFI, a selective fine-grained interaction framework, to take
full advantage of the fine-grained interaction in a highly efficient way.

(2) We design the learning-to-select mechanism to sparsely and automatically
select informative historical news w.r.t. the candidate.

(3) We conduct extensive ablation studies to verify the advantage of
selection; and further investigate the efficiency-effectiveness trade-off that
SFI achieves.

##  2\. Related Work

In this section, we first review the traditional recommendation methods, then
the neural news recommendation methods.

###  2.1. Traditional News Recommendation

A lot of traditional recommendations methods (Das et al., 2007; Li et al.,
2011; Koren et al., 2009) are based on collaborative filtering (CF). CF-based
methods cluster users by “co-visitation” relationships to recommend news to
similar users (Das et al., 2007). Another line of CF studies apply Matrix
Factorization (Koren et al., 2009) and Factorization Machine (Rendle, 2010) to
model the interaction between users and items. However, these methods face the
problem of cold-start and sparsity, which is severe in news domain. They also
require difficult and labor-consuming feature engineering. As the counterpart
of CF, content-based recommendation methods become the main focus of news
recommendation (Liu et al., 2010; Li et al., 2010) because of the rich text
information in news articles.

###  2.2. Neural News Recommendation

In recent years, deep learning techniques are widely used in news
recommendation systems and achieve better results than traditional methods.
They can be categorized as follows:

####  2.2.1. Feature-based Methods

Following traditional recommendation approaches, feature-based methods feed
the model with news content together with manually designed features, then
employ neural networks to model the complex interactions among all the
features (Cheng et al., 2016; Guo et al., 2017; Lian et al., 2018). For
example, Cheng et al. (Cheng et al., 2016) propose to combine shallow and deep
neural networks to extract valuable information from a variety of manual
features. Guo et al. (Guo et al., 2017) add deep layers over the factorization
machine to model high order interactions.

####  2.2.2. Representation-based Methods

More methods proposed to learn representations of news and users from raw
texts and browsing histories respectively (Okura et al., 2017; Huang et al.,
2013; Wu et al., 2019d, f, c; An et al., 2019; Yu et al., 2019; Wu et al.,
2019b, e; Lian et al., 2020). Numerous well designed models are proposed:
multi-layer perceptron over tri-grams (Huang et al., 2013), denoising auto-
encoder (Okura et al., 2017), convolution neural networks (Wu et al., 2019d,
c; An et al., 2019; Yu et al., 2019; Wu et al., 2019b, e; Lian et al., 2020),
and various attention-based methods (Wu et al., 2019d, f, c). Multi-channel
structure is also explored (Liu et al., 2019). Besides, some approaches (Wu et
al., 2019a; Hu et al., 2020; Wu et al., 2019g) focused on using graph neural
networks to represent news and users with their neighbors. Several methods
(Wang et al., 2018, 2019) proposed to incorporate knowledge to construct
knowledge-aware representations of news and users.

In spite of the improvements these methods have made, all of them embed the
news and the user into one or several one-fold vectors in the semantic space,
where the fine-grained information is limited. And the representations can
only meet each other in the prediction phase, which may impair fine-grained
matching signals between the user and the candidate news.

####  2.2.3. Interaction-based Methods

To address the above problem, interaction-based models, which match user’s
interests with the candidate news at more delicate levels, are proposed. Wang
et al. (Wang et al., 2020) designed the state-of-the-art interaction-based
method for news recommendation. They constructed segment-to-segment similarity
matrices between the candidate news and every historical news article of the
user from 3 different granularities. Then they use 3D-CNN to highlight salient
matching signals to make recommendations. Although FIM achieves better results
feature- and representation-based methods, its main drawback is the especially
slow inference speed.

In this work, we explore the interaction-based methods and aim to efficiently
select fewer historical articles with higher value to perform interactions.
Several works in other fields have proposed to select important features for
interaction (Guo et al., 2016; Liu et al., 2020; Yuan et al., 2019), but they
all employ the gating operator, which fails to discard the unselected items
hence cannot be directly used to achieve our goal. The most related work
(Hofstätter et al., 2021) splits selection to another training stage,
forbidding the model learning to select. Our proposed learning-to-select
mechanism effectively addresses these issues.

##  3\. Our Approach

Figure 2. The architecture of our SFI model.

The model selects informative historical news using the efficient
representation based matching, then carry out effective fine-grained term-
level interaction over the selected news for final recommendation.

First we formulate the news recommendation problem. Given a user $u$, we have
a set of historical news articles browsed by her at the platform, denoted as
$\mathcal{H}=\\{\mathrm{h}_{1},\mathrm{h}_{2},\cdots,\mathrm{h}_{M}\\}$. For a
candidate news $\mathrm{c}$, our goal is to infer the probability that the
user clicks this news article based on her browsing history $\mathcal{H}$,
denoted as $p(\mathrm{c}|\mathcal{H})$.

The architecture of SFI is presented in Figure 2. Specifically, it contains
four major modules. The news encoder module learns word- and news-level
representations, the fine-grained ones are intended for interaction and the
coarse-grained ones are further transformed for selection. The following
history selector module manipulates news-level representation to efficiently
and precisely select informative news from the user’s browsing history. The
fine-grained representations of selected news are fed into the news interactor
module to compute interactions. The coarse-grained matching signals are also
modeled in this module. Finally the click predictor module incorporates all
matching signals to predict the click probability $p(\mathrm{c}|\mathcal{H})$.
Next, we introduce each component in our model, especially the history
selector.

###  3.1. News Encoder Module

Since users’ click decisions on news platforms are usually based on the title
of news articles (Wu et al., 2019d), the news encoder learns the news
representation from title only. Denote the word sequence of a news title as
$S=\\{\mathrm{w}_{1},\mathrm{w}_{2},\cdots,\mathrm{w}_{N}\\}$, where $N$ is
the length of $S$. First of all, we transform $S$ into a sequence of vectors
$E=\\{\mathbf{e}_{1},\mathbf{e}_{2},\cdots,\mathbf{e}_{N}\\}$ by word
embedding matrix $\mathbf{W_{e}}\in\mathbb{R}^{V\times D}$. $V$ is the
vocabulary size and $D$ is the dimension of embeddings. Usually, the local
contexts of a word across different spans play a big role in representing the
word (Wu et al., 2019d, c; Wang et al., 2020). Therefore, we employ a
hierarchical dilated convolution (Yu and Koltun, 2016) to extract context
features from different semantic granularities. In the $l$-th convolution
layer, the representation of the $i$-th word is calculated as:

(1)
$r_{i}^{l}=\mathsf{ReLU}\left(\mathbf{F_{w}}\times\bigoplus_{k=0}^{w}\mathbf{e}_{i\pm
k\delta}+\mathbf{b}\right)\in\mathbb{R}^{f_{s}},$

where $\bigoplus$ means the concatenation operation for vectors.
$\mathbf{F_{w}}$ is the convolution kernel of size $2w+1$, $\delta$ denotes
dilation rate, $\mathbf{b}$ denotes bias and $f_{s}$ denotes the number of
filters. A detailed description of dilated convolution can be found in (Wang
et al., 2020). By hierarchically stacking dilated convolutions with expanding
dilation rate, local contexts of different distances are fused into the word
representations. Afterward, the output of each convolution layer is appended
to the final representation of $i$-th word:
$\mathbf{r}_{i}=\\{\mathbf{r}_{i}^{l}\\}_{l=0}^{L}$, where
$\left\\{\cdot\right\\}$ denotes the vertical alignment of a matrix, and $L$
denotes the total number of the stacked convolution layers.

The representation of each semantic level may contain information of different
importance for matching. For example, in the news title “Restaurants to
Satisfy Late Night Cravings in Louisville and Beyond”, phrase-level local
contexts “Late Night Craving” for the word “Night” matter more than that of
sentence-level, e.g. “Restaurants … Night … And”. Therefore, we use an
attentive pooling technique (Yu et al., 2019; Wu et al., 2019c; An et al.,
2019) to highlight the important local contexts of a single word.
Specifically, a trainable vectors $\mathbf{q_{l}}\in\mathbb{R}^{f_{s}}$ is
introduced as the query of attention. The representation
$\mathbf{r}_{i}^{\prime}$ of the word $\mathrm{w}_{i}$ that fuses information
across every semantic level is computed as:

(2)
$\displaystyle\mathbf{r}_{i}^{\prime}=\sum_{l=1}^{L}a_{il}\mathbf{r}_{i}^{l},\qquad\text{where
}\quad
a_{il}=\frac{\exp(\mathbf{q_{l}}^{\top}\mathbf{r}_{i}^{l})}{\sum_{j=1}^{L}\exp(\mathbf{q_{l}}^{\top}\mathbf{r}_{j}^{l})}.$

Similarly, different words may contribute differently in expressing the news.
For example, “Louisville” is more informative than “Beyond” because it reveals
the location. We use another query vector
$\mathbf{q}_{w}\in\mathbb{R}^{f_{s}}$ to highlight the informative words in
the news title and obtain the overall representation of the entire news:

(3)
$\displaystyle\mathbf{r}=\sum_{i=1}^{N}a_{i}\mathbf{r}_{i}^{\prime},\qquad\text{where
}\quad
a_{i}=\frac{\exp(\mathbf{q_{w}}^{\top}\mathbf{r}^{\prime}_{i})}{\sum_{j=1}^{L}\exp(\mathbf{q_{w}}^{\top}\mathbf{r}^{\prime}_{j})}.$

So far, the fine-grained representation of each word
$\mathbf{r}_{i}=\\{\mathbf{r}_{i}^{l}\\}_{l=0}^{L}$ and the coarse-grained
representation of the entire news $\mathbf{r}$ are generated by _news
encoder_. We further explore other kinds of state-of-the-art encoders, and
study their performance in Section 5.3.

###  3.2. History Selector Module

The history selector is the core component of our model. It selects the
informative historical news sparsely, automatically, and efficiently with a
learning-to-select mechanism. Then the selected news pieces are fed into next
module for fine-grained interactions.

Denote the news-level representation of $i$-th clicked news in the user’s
history as $\mathbf{h}_{i}$, and that of candidate news as $\mathbf{c}$.
Recall that the news-level representation is attentively aggregated from the
word vectors in the title, which are optimized for the final matching and thus
aren’t selection-oriented. Therefore, directly using $\mathbf{h}_{i}$ and
$\mathbf{c}$ for selection may lead to sub-optimal results. In learning-to-
select, we project all the news-level representations into the same selection
space using a fully-connected network to mitigate such conflicts:

(4) $\displaystyle\mathsf{Proj}(\mathbf{r})$
$\displaystyle=\mathbf{W_{p}}\mathbf{r}+\mathbf{b},$

where $\mathbf{r}$ could be either $\mathbf{h}_{i}$ or $\mathbf{c}$.
Considering selection efficiency, we define the candidate-aware
informativeness of every historical article as the cosine similarity between
selection vectors:

(5) $\displaystyle\mathbf{s}$
$\displaystyle=\\{\mathrm{cos}\left(\mathsf{Proj}(\mathbf{h}_{i}),\mathsf{Proj}(\mathbf{c})\right)\\}_{i=1}^{M}.$

In selection, no supervision signals are available for $\mathbf{s}$, so it’s
critical to optimize the parameters end-to-end to allow the model learn
valuable features for selection. Besides, sparsity must be enforced, otherwise
the model would still be slow. Keeping both constraints in mind, we design two
complementary selection networks.

####  3.2.1. Hard-Selection Network

This sub-module enforces sparsity: it keeps the top $K$ most informative
history news and discards others. Formally,

(6) $\displaystyle\mathbf{x}$
$\displaystyle=\mathsf{argTOPK}(\mathbf{s})\in\mathbb{R}^{K},$

where $\mathsf{argTOPK}(\mathbf{s})$ gets the index of the top $K$ value in
the vector $\mathbf{s}$. The corresponding history news sliced by $\mathbf{x}$
is selected and its fine-grained representations will get involved in the
fine-grained interaction later. To do that, we first transform $\mathbf{x}$ to
one-hot encoding matrix
$\mathbf{X}=\mathsf{one\\_hot}(\mathbf{x})\in\mathbb{R}^{K\times M}$, where
the $i$-th row in $\mathbf{X}$ is the one-hot vector of $\mathbf{x}_{i}$, then
use matrix multiplication to prune the browsing history to a smaller size:

(7) $\displaystyle\hat{H}=\mathbf{X}\otimes
H,\qquad\hat{\mathbf{s}}=\mathbf{X}\otimes\mathbf{s}.$

where
$H=\\{\\{\mathbf{h}_{i}^{l}\\}_{l=0}^{L}\\}_{i=1}^{M}\in\mathbb{R}^{M\times
L\times f_{s}}$ is the fine-grained representation tensor of user’s browsed
news, $\hat{H}\in\mathbb{R}^{K\times L\times f_{s}}$ is that of the selected
news. By regulating hyper parameter $K$, we can elastically control the
model’s efficiency.

However, this sub-module has three defects that may limit the effectiveness:
1) Because $\mathbf{X}$ is sparse, the gradient cannot be passed to optimize
$\mathbf{W_{p}}$; 2) Because the informativeness distributions of different
users vary greatly, some noisy news articles are not filtered out among top
$K$; 3) All of the selected news articles are weighted equally even though
some of them are more informative. To tackle the above problems, a soft
refinement is proposed.

####  3.2.2. Soft-Selection Network

This sub-module makes the gradient flow through selection. It is essentially a
gating operator with a threshold (Yuan et al., 2019), which further rules out
noise (namely authentically uninformative news) and improves effectiveness.
Given the output of hard-selection, the soft-selection network masks the news
whose informativeness is below the threshold and attaches different importance
to the unmasked ones:

(8) $\displaystyle\tilde{H}$
$\displaystyle=\hat{H}\odot\mathsf{Expand}(\mathbf{\tilde{s}}),\qquad\mathbf{\tilde{s}}=\sigma(\mathbf{\hat{s}}),$
(9) $\displaystyle\sigma(s_{i})$
$\displaystyle=\begin{cases}0&s_{i}<\gamma,\\\
h_{i}&\mathrm{otherwise}.\end{cases}$

$\odot$ is Hadamard Product, and $\gamma$ denotes the threshold.
$\sigma(\cdot)$ is element-wise. $\mathsf{Expand}(\tilde{\mathbf{s}})$ repeats
the elements in $\tilde{\mathbf{s}}$, expanding it into $\mathbb{R}^{K\times
L\times f_{s}}$. $\tilde{H}$ is the refined $\hat{H}$, where all
representations of the news whose informativeness is lower than $\gamma$ are
masked as $0$.

The number of the authentically informative news articles is floating per
candidate, so a dynamic quantity of news items is kept. This entitles more
flexibility to the selection operation. Meanwhile, with $\tilde{s}$ attached
to $\hat{H}$, the news interactor can attend to more informative ones, and the
parameters in $\mathsf{Proj}(\cdot)$ can be optimized by the selecting step
since the element-wise multiplication is differentiable. This helps SFI to
learn features that are important for selection and will enhance the
effectiveness remarkably.

In back propagation, gradient from the loss function is applied to the news
interactor, then to the selected fine-grained representation tensor
$\tilde{H}$. For simplification, $\tilde{H}$ is reshaped into a vector
$\tilde{\mathbf{R}}\in\mathbb{R}^{1\times(K\times d)}$ where $d=L\times
f_{s}$, together with its gradient
$\nabla_{\tilde{\mathbf{R}}}=\mathsf{reshape}(\nabla_{\tilde{H}})\in\mathbb{R}^{1\times(K\times
d)}$. The same operation is taken for $\hat{H}$, forming
$\hat{\mathbf{R}}\in\mathbb{R}^{1\times(K\times d)}$. Then the gradient for
$\mathbf{W_{p}}$ is:

(10) $\displaystyle\nabla_{\mathbf{W_{p}}}$
$\displaystyle=\mathbf{Z}^{\top}\otimes\nabla_{\mathbf{s}}\otimes\mathbf{c}_{1}^{\prime}+\mathbf{c}^{\top}\otimes\nabla_{\mathbf{s}}^{\top}\otimes\mathbf{Z}_{1},$
$\displaystyle\nabla_{\mathbf{s}}$
$\displaystyle=\mathbf{X}^{\top}\otimes\nabla_{\mathbf{\hat{s}}}$
$\displaystyle=\mathbf{X}^{\top}\otimes\left(\nabla_{\tilde{\mathbf{s}}}\odot
g(\hat{\mathbf{s}})\right)$ (11)
$\displaystyle=\mathbf{X}^{\top}\otimes\left(\left\\{\sum_{j=1}^{d}\left(\nabla_{\tilde{\mathbf{R}}}\otimes\mathsf{Diag}(\hat{\mathbf{R}})\right)[i,j]\right\\}_{i=1}^{K}\odot
g(\hat{\mathbf{s}})\right),$

where $\mathsf{Diag}(\hat{\mathbf{R}})$ is the matrix with the elements of
$\hat{\mathbf{R}}$ as the diagonal.
$\mathbf{Z}=\\{\mathbf{h}_{i}\\}_{i=1}^{M}\in\mathbb{R}^{M\times f_{s}}$ is
the news-level representation matrix of the historical news, and
$\mathbf{Z}_{1}=\\{\mathsf{Proj}(\mathbf{h}_{i})\\}_{i=1}^{M},\mathbf{c}_{1}=\mathsf{Proj}(\mathbf{c})$
are the corresponding selection vectors. $g(s_{i})$ is the derivative for
$\sigma(s_{i})$:

$g(s_{i})=\begin{cases}0&s_{i}<\gamma,\\\ 1&\mathrm{otherwise}.\end{cases}$

In this way, the gradient safely flows through the selection stage and reaches
$\mathbf{s}$, to increase the score of the useful news pieces and vice versa.
It is further spread to optimize $\mathbf{W_{p}}$ to achieve the above
adjustment, however, only from the selected entries.

###  3.3. News Interactor Module

The selected historical news articles are fed into this module to perform
fine-grained interactions with the current candidate. We denote the
representation of the words in $v$-th selected news as
$\mathbf{d}_{v}=\\{\mathbf{t}_{i}\\}_{i=1}^{N}$ where
$\mathbf{t}_{j}=\\{\mathbf{t}_{j}^{l}\\}_{l=0}^{L}\in\mathbb{R}^{L\times
f_{s}}$ is the stacked representation of $j$-th word. Similarly, the
representation of each word in the current candidate news is
$\mathbf{c}^{\mathrm{f}}=\\{\mathbf{p}_{j}\\}_{j=1}^{N}$. Resembling FIM (Wang
et al., 2020), we construct pair-to-pair similarity matrix
$\mathbf{M}^{l}_{v,\mathrm{c}}$ of $l$-th semantic granularity, where each
entry is the scaled dot product between the fine-grained representations of
$v$-th selected news and the candidate news:

(12)
$\mathbf{M}_{v,\mathrm{c}}^{l}[i,j]=\frac{\mathbf{t}_{i}[l]^{\top}\mathbf{p}_{j}[l]}{\sqrt{f_{s}}}\in\mathbb{R}^{N\times
N}.$

Next, the similarity matrices of each granularity across all the selected
history news are fused into a 3D cube $O\in\mathbb{R}^{L\times K\times N\times
N}$, where a series of 3D CNN and 3D max pooling is applied to highlight the
significant matching signals. Outputs of the final pooling layer are flattened
as the vector containing fine-grained interactive information across the user
and candidate news, denoted as $\phi_{\mathrm{u},\mathrm{c}}$. Other state-of-
the-art interactors are studied in Section5.3.

In SFI, fine-grained matching information $\phi_{\mathrm{u},\mathrm{c}}$ only
engage selected news articles. However, it is important not to leave out the
unselected ones. Although conducting fine-grained interactions on them is
unnecessary, we still value the coarse-grained matching signals of them, which
come from the matching between news-level representations:

(13)
$\displaystyle\psi_{\mathrm{u},\mathrm{c}}=\\{\psi_{\mathbf{h}_{1},\mathbf{c}},\psi_{\mathbf{h_{2}},\mathbf{c}},\cdots,\psi_{\mathbf{h}_{M},\mathbf{c}}\\},\qquad\psi_{\mathbf{h}_{i},\mathbf{c}}=\mathbf{h}_{i}^{\top}\mathbf{c}.$

$\psi_{\mathrm{u},\mathrm{c}}$ gives an overall matching degree of the user
and the candidate news and is complementary to $\phi_{\mathrm{u},\mathrm{c}}$.
It facilitates the model to learn more precise correspondences between the
matching signals and the click probability. Another critical point is that by
involving $\psi_{\mathrm{\mathrm{u},\mathrm{c}}}$ to score the candidate, the
gradient can be delivered by all of the historical news articles rather than
only the selected ones.

###  3.4. Click Predictor

The _click predictor_ module incorporates the output from news interactor then
predicts the probability of a user clicking on a candidate news article. The
news articles with higher click probability are ranked higher in the final
user interface.

Given vectors containing course- and fine-grained matching information,
$\psi_{\mathrm{u},\mathrm{c}}$ and $\phi_{\mathrm{u},\mathrm{c}}$
respectively, we propose to incorporate both by:

(14)
$y_{\mathrm{u},\mathrm{c}}=\mathbf{W_{c}}\left\\{\phi_{\mathrm{u},\mathrm{c}},\psi_{\mathrm{u},\mathrm{c}}\right\\}+\mathbf{b}.$

Following (Huang et al., 2013; Wu et al., 2019d), we use negative sampling to
simulate the unbalanced distribution of clicked news in an impression. For
each ground-truth candidate, we randomly sample $m$ news that is not clicked
by her in the same impression as negative samples:

(15)
$\hat{p}(\mathrm{c}|\mathcal{H})=\frac{\exp(y_{\mathrm{u},\mathrm{c}}^{+})}{\exp(y_{\mathrm{u},\mathrm{c}}^{+})+\sum_{j=1}^{m}\exp(y_{\mathrm{u},\mathrm{c}}^{-})}.$

Thus, it is converted to a $m+1$ classification problem, and the negative log
likelihood loss is going to be minimized when training:

(16)
$\mathcal{L}=-\sum_{\mathrm{c}\in\mathcal{S}}\log\hat{p}(\mathrm{c}|\mathcal{H}),$

where $\mathbf{c}$ is the ground-truth news piece which the user clicked, and
$\mathcal{S}$ denotes all training samples.

Finally, we jointly train the _news encoder_ , _history selector_ , _news-
interactor_ and _click predictor_ through the final click signal. In such a
way, the model can better learn dependencies among modules.

##  4\. Experimental

###  4.1. Datasets and Experimental Settings

Table 1. Statistics of the MIND dataset #users |  1,000,000 |  #news |  161,013
---|---|---|---
#impressions |  15,777,377 |  #clicks |  24,155,470
avg. title len |  11.52 |  avg. his len |  32.99

Our experiments are conducted on MIND (Wu et al., 2020), a large-scale dataset
collected from the users’ click logs of the Microsoft News platform from Oct.
12 to Nov. 22, 2019. The statistics of MIND are shown in Table 1. We use the
same training-testing partition as (Wu et al., 2020).

In our experiments, the dimension $D$ of word embeddings is set to $300$. We
use the pre-trained Glove embeddings (Pennington et al., 2014), to initialize
the embedding matrix $\mathbf{W_{e}}$. The maximum length of news titles is
set at $20$. the maximum number of clicked news for learning user
representations was set to $50$. In the _news encoder_ , we stack $3$
convolution layers with dilation rates $[1,2,3]$. The kernel size and the
number of filters is set to $3$ and $150$ respectively. We employ a 2-layer
composition for _news-interactor_ module, the output channels and the window
size is set at $32-[3,3,3]$ and $16-[3,3,3]$. Each convolution component is
followed by a max pooling layer with size $[3,3,3]$ and stride $[3,3,3]$. We
apply the dropout strategy (Srivastava et al., 2014) to the word embedding
layer to mitigate overfitting. The dropout rate is set at $0.2$. Adam (Kingma
and Ba, 2015) is used as the optimization algorithm.

The batch size is set to $100$ when training and $400$ when predicting, and
the encoding process is executed offline when predicting. Since there are $40$
kernels in total on our machine, we set $40$ parallel threads to load data in
order to minimize the latency caused by processing data. We independently
repeat each experiment for $5$ times and report the average performance. We
conduct all experiments on a machine with Xeon(R) Silver 4114 CPUs and a TITAN
V GPU222We will release the code and scripts based upon the acceptance of the
paper..

###  4.2. Evaluation Metrics

Following existing studies, we use the average AUC, MRR, nDCG@5, and nDCG@10
scores over all impressions to evaluate the effectiveness of the models. All
results come from the official test entry. Moreover, given the same batch
size, we use the prediction speed i.e. iterations per second to evaluate the
efficiency. In one iteration, the batch size of candidate news articles is
scored.

###  4.3. Baselines

We compare SFI with the following baseline methods:

(1) General Recommendation Methods: LibFM (Rendle, 2012), a state-of-the-art
feature-based matrix factorization approach for recommendation333The TF-IDF
features are used.; DSSM (Huang et al., 2013), a deep structured semantic
model that uses multiple dense layers upon tri-grams. All of the users’
clicked news are concatenated as the query, and the candidate news is regarded
as documents; Wide&Deep (Cheng et al., 2016), a widely used recommendation
method that uses the combination of a wide channel and a deep channel for
memorization and generalization; DeepFM (Guo et al., 2017), a popular neural
recommendation method which combines factorization machine with deep neural
networks;

(2) Representation-based Methods: DFM (Lian et al., 2018), which uses dense
layers for different channels and attentively fuse outputs; GRU (Okura et al.,
2017), which learn news representations with an auto-encoder and utilizes GRU
to learn user representations; Hi-Fi Ark (Liu et al., 2019), a multi-channel
representation approach for recommendation; NPA (Wu et al., 2019d), which
highlights informative words and news with personalized attention; NRMS (Wu et
al., 2019f), which learns delicate representations of news and users by multi-
head self-attention; LSTUR (An et al., 2019), which models long- and short-
term user interests with GRU;

(3) Interaction-based Methods: FIM (Wang et al., 2020), the state-of-the-art
interaction-based approach for neural news recommendation, which encodes news
by hierarchical dilated CNN and performs interaction between each of the user
browsed news articles and the candidate. Recent($K$), the naive counterpart of
SFI, which keeps the recent $K$ historical news for interaction only
(Recent($50$) equals FIM).

##  5\. Experimental Results and Analysis

###  5.1. Overall Results of Effectiveness

Table 2. The performance of different methods for news recommendation. The number of news items involving in interactions is bolded in ( ) for interaction-based methods. The result with superscript * is referencing the one in (Wu et al., 2020) where MIND is presented. $\dagger$ indicates a significant improvement over all baselines with paired t-test ($p<0.01$). Type | Methods | AUC | MRR | NDCG@5 |  NDCG@10
---|---|---|---|---|---
General methods | LibFM* | $59.93$ | $0.2823$ | $30.05$ |  $35.74$
DSSM* | $64.31$ | $0.3047$ | $33.86$ |  $38.61$
Wide&Deep* | $62.16$ | $0.2931$ | $31.38$ |  $37.12$
DeepFM* | $60.30$ | $0.2819$ | $30.02$ |  $35.71$
Represent. based | DFM* | $62.28$ | $0.2942$ | $31.52$ |  $37.22$
GRU* | $65.42$ | $0.3124$ | $33.76$ |  $39.42$
Hi-Fi Ark | $65.87$ | $0.3119$ | $33.64$ |  $39.25$
NPA* | $66.69$ | $0.3224$ | $34.98$ |  $40.68$
LSTUR* | $67.73$ | $0.3277$ | $35.59$ |  $41.34$
| NRMS* | $67.76$ | $0.3305$ | $35.94$ |  $41.63$
Interaction based | FIM | $68.12$ | $0.3354$ | $36.45$ |  $42.11$
Recent ($\mathbf{5}$) | $65.39$ | $0.3164$ | $34.14$ |  $39.78$
SFI ($\mathbf{5}$) | $69.60^{\dagger}$ | $0.3475^{\dagger}$ | ${37.86}^{\dagger}$ |  ${43.51}^{\dagger}$
SFI ($\mathbf{50}$) | $\mathbf{69.95}^{\dagger}$ | $\mathbf{0.3503}^{\dagger}$ | $\mathbf{38.31}^{\dagger}$ |  $\mathbf{43.97}^{\dagger}$

The overall recommendation effectiveness of all models is shown in Table 2.
Based on the results, we have the following observations:

(1) Our proposed model SFI consistently outperforms other baselines in terms
of all metrics. On the one hand, SFI captures fine-grained interactions to
model user interests, gaining $2.71\%$ up to $3.23\%$ AUC improvements over
all of the state-of-the-art representation-based methods. On the other hand,
SFI$(K)$ outperform Recent($K$) baseline by $6.4\%$ and $2.17\%$ when $K=5$
and $50$ respectively. This result substantiates the power of the learning-to-
select mechanism.

(2) The variant SFI($50$) that keeps the entire browsing history outperforms
SFI($5$) that selects only $5$ historical news. This is as expected because
after removing noise, SFI($50$) covers richer information to model the user.
Interestingly, the improvement is tiny compared with the margin between
Recent($50$) i.e. FIM and Recent($5$). We will study this phenomenon in detail
in Section 5.4.1.

(3) The interaction-based methods for news recommendation outperform all
representation-based methods, which validates the benefit of capturing fine-
grained matching signals. However, simply pruning the user’s history to a
smaller size to save speed is not feasible because it hurts the effectiveness
seriously.

Without Bert (Devlin et al., 2019), expanded SFI (with extra news abstract)
ranks among the top $15$ on the official testing leaderboard.

###  5.2. Results of Efficiency

Since the motivation of SFI is mainly concerned with efficiency, we further
compare the inference speed between SFI and several baselines. Results in
Table 3 substantiates the superiority of SFI: with the selection capacity of
$\mathbf{5}$, it can infer almost four times faster than the state-of-the-art
interaction-based method, while significantly improving the recommendation
effectiveness. SFI($5$) also achieves comparable speed with the state-of-the-
art representation-based method NRMS, and outperforms it by $\mathbf{2.71}\%$
in AUC. The efficiency of SFI($K$) and Recent($K$) significantly drops from
$K=5$ to $K=25$, which will be further studied in Section 5.4.1.

Table 3. The inference speed comparison of different methods for news recommendation. The improvement over FIM is given in the bracket. Methods |  Inference Speed | AUC | nDCG@5
---|---|---|---
NRMS |  $121.54$ | $67.76$ | $35.94$
FIM |  $20.53$ | $68.12$ | $36.45$
Recent ($\mathbf{5}$) |  $125.96(\uparrow 517\%)$ | $65.39(\downarrow 4.01\%)$ | $34.14(\downarrow 6.34\%)$
Recent ($\mathbf{25}$) |  $40.05(\uparrow 96\%)$ | $67.32(\downarrow 1.17\%)$ | $35.16(\downarrow 3.54\%)$
SFI ($\mathbf{5}$) |  $\mathbf{99.57(\uparrow 385\%)}$ | $69.60(\uparrow 2.17\%)$ | $37.86(\uparrow 3.87\%)$
SFI ($\mathbf{25}$) |  $33.11(\uparrow 66\%)$ | $69.75(\uparrow 2.39\%)$ | $38.01(\uparrow 4.28\%)$

###  5.3. Ablation Study

Since SFI is essentially a flexible framework, we conduct extensive ablation
studies to gain comprehensive insights into every module. In each subsection,
we pose our claim first before explanations.

####  5.3.1. HDCNN and 3DCNN are the most effective encoder and interactor
among a variety of state-of-the-art architectures.

Table 4. The effectiveness of SFI with different news encoders and news interactors. Encoders Interactors | 2D-CNN | 3D-CNN | MHAI | KNRM
---|---|---|---|---
PCNN | $63.71$ | $63.95$ | $65.54$ | $63.24$
HDCNN | $68.45$ | $\mathbf{69.56}$ | $63.66$ | $60.01$
MHA | $66.44$ | $65.56$ | $61.31$ | $63.92$
LSTM | $68.58$ | $67.91$ | $68.39$ | $64.12$

In the interaction-based workflow, the history selector can be easily inserted
between any kind of news encoder and interactor. This flexibility motivates us
to study how state-of-the-art encoders and interactors would perform. We
compare among 1D-CNN with Personalized Attention (Wu et al., 2019d) (denoted
as PCNN), Hierarchical Dilated CNN (Wang et al., 2020) (denoted as HDCNN),
Multi-head Self Attention (Wu et al., 2019f) (denoted as MHA), LSTM for the
news encoder and 2D-CNN, 3D-CNN (Wang et al., 2020), KNRM (Xiong et al.,
2017), Multi-Head Self Attention (Wu et al., 2019f) (denoted as MHAI) for the
news interactor. The AUC scores are reported in Table 4. We find Hiearchical
Dilated CNN combined with 3D-CNN is the best setting.

####  5.3.2. Every sub-module in history selector is critical to improving
effectiveness.

Figure 3. The effectiveness of history selector and coarse-grained
information.

The learning-to-select mechanism comprises three parts: a selection
projection, a hard selection, and a soft refinement. The hard selection is the
cornerstone of our work so we no longer verify its impact. For the other two
components, we compare SFI with the variant that applies only hard-selection,
and that applies hard-selection followed by soft-selection without learning
extra selection vectors. The result is reported in Figure 3. As we observe,
the soft-selection network improves the effectiveness. This is because it
filters the authentically uninformative history news, and makes the gradient
flow through to optimize the representation vectors used for selection.
However, without selection projection, these news-level representations are
optimized for two incompatible goals: selecting and matching, which may
decrease the recommendation accuracy. Experiments validates our claim: the
model benefits a lot from selection projection. Thanks to it, SFI can encode
selective features into selection vectors, leaving the news-level
representations to focus on the final matching.

####  5.3.3. The coarse matching signals of the unselected articles are also
important.

In Figure 3, SFI outperforms its variant that totally abandons the coarse-
grained matching signals. This observation verifies that the coarse-grained
matching signals are complementary. Note that doing so won’t reduce efficiency
because batched matrix multiplication is fast on GPU.

####  5.3.4. SFI benefits from end-to-end training.

When deployed in production, the news encoding process could be done offline
to speed up inference, known as a pipeline convention. It’s natural to migrate
it to the training phase, where we first pre-train SFI without the _history
selector_ to acquire coarse-and-fine representations of news. Then we replace
the _news encoder_ with a lookup table constructed from these representations
and fine-tune them with _history selector_ applied. End-to-end training is the
counterpart, in which we jointly train the _news encoder_ , _history filter_ ,
_news-interactor_ and _click predictor_ by the final classification loss
simultaneously. In Figure 3, the first four bars in every group are the
performance of SFI trained in pipeline, and the last bar is that of trained
end-to-end under the same setting. As expected, end-to-end training leads to
better effectiveness. It’s because optimizing the parameters rather than
directly updating the vectors can make the news encoder learn more precise
representations for both selection and interaction. Also, the modules can
better learn the correspondence among them in end-to-end training.

###  5.4. Hyper Parameter Analysis

Figure 4. The efficiency and effectiveness of SFI with different numbers of
selected news. The number next to the marker indicates the selected news
count. Figure 5. The informativeness score of history news at each position.
Smaller x-axis represents more recent history.

####  5.4.1. Influence of the Interaction Capacity

The predefined interaction capacity $K$ is the most important hyper parameter
since the efficiency-effectiveness trade-off is up to it. We study its
influence by drawing the inference speed curve against the AUC score of the
model with different $K$ in Figure 4. Motivated by several observations in
Section 4, we also include Recent($K$), which only interacts with the latest
$K$ historical news to save efficiency. We add two dashed lines to mark the
best effectiveness and efficiency that baseline models ever achieve. The
optimal result would be situated at the upper right corner.

According to the figure, we find: First, from $K=5$ to $K=50$, SFI($K$) is far
more effective than its naive counterpart Recent($K$), this again validates
the effectiveness of history selector. However, due to the time consumption of
selection, SFI($K$) is a bit slower. Overall, SFI($5$) is the optimal setting
because it greatly outperforms NRMS and all Recent($K$) including FIM, while
providing a much higher efficiency over FIM. Second, when $K$ is growing, the
effectiveness of both Recent($K$) and SFI($K$) is improving, which is because
a bigger capacity keeps richer information to learn the user’s interests. As a
side effect, the model becomes slower. Third, as $K$ increases, the
effectiveness of SFI($K$) grows slower than Recent($K$) and is about saturated
at $K=40$. Intuitively, with the learning-to-select mechanism, SFI($K$) can
consistently select the most effective articles for interaction, so increasing
the capacity only brings a little more valuable information. In contrast,
Recent($K$) cannot access the informative historical news in earlier history
unless the capacity is big enough.

These observations motivate us to study what informativeness scores of the
historical articles at different positions are learnt by the model itself. We
report the informativeness score at each history position (averaged from $K=5$
to $K=50$) in Figure 5. The blue line is the mean value of informativeness,
and the shade indicates standard deviation. The horizontal black line marks
the threshold of the soft-selection. According to the figure, the increasing
mean value of informativeness tells us that more recent reading history helps
more in expectation. At the same time, the significantly high variance
confirms that historical news at each position has the potential to interact
with the candidate. Therefore, with a small $K$, Recent($K$) cannot access
earlier historical news that tends to be useful in interaction. Rather, SFI is
quite able to inspect them and involve them in fine-grained interactions
intelligently. Moreover, the informativeness of the history news whose
position is farther than $40$ hardly reaches the threshold. So they are
considered authentically uninformative and masked even though they are among
the top $K\geq 40$. This explains the saturation of SFI’s effectiveness and
justifies our intuition.

####  5.4.2. Influence of the Informativeness Threshold

Figure 6. The effectiveness of SFI with different value of $\gamma$.

Another crucial factor of SFI is the informativeness threshold in the soft-
selection network. The effectiveness of SFI with different threshold settings
is shown in Figure 6. In summary, the threshold shouldn’t be too large or too
small. When $\gamma<0.1$, almost all history news articles are considered
informative, so the selection fails. When $gamma>0.3$, the _history selector_
rules out too many history news articles, including the valuable ones. The
gradient cannot be passed adequately, either. Hence the model’s effectiveness
declines. When it reaches $1$, all fine-grained representations are masked as
$0$, completely disabling the news interactor. Recall that the coarse-grained
matching signals persist, leading to better results than random
recommendation. Overall, $\gamma=0.2$ is the optimal configuration.

##  6\. Conclusion and Future Work

Capturing fine-grained interactions brings more accuracy and higher online
costs for news recommenders. In this work, we proposed a selective fine-
grained interaction framework to select a small number of valuable historical
articles for interaction, drawing a good balance between efficiency and
effectiveness. With the help of the learning-to-select mechanism, the
selection can be performed efficiently, sparsely, and automatically.
Experimental results show SFI can significantly improve the recommendation
effectiveness by $2.17\%$ over the state-of-the-art models with four times
faster speed. We experimented a lot to provide comprehensive insights of SFI
and studied the efficiency-effectiveness trade-off it achieves. In the future,
we will dig deeper into representing users with terms to further improve the
efficiency while keeping the effectiveness.

## References

  * (1)
  * An et al. (2019) Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019\.  Neural News Recommendation with Long- and Short-term User Representations. In _ACL,2019_. Association for Computational Linguistics, 336–345.
  * Chen et al. (2018) Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018.  Sequential Recommendation with User Memory Networks. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018_ , Yi Chang, Chengxiang Zhai, Yan Liu, and Yoelle Maarek (Eds.). ACM, 108–116.  https://doi.org/10.1145/3159652.3159668
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide & Deep Learning for Recommender Systems. In _DLRS@RecSys, 2016_. ACM, 7–10.
  * Das et al. (2007) Abhinandan Das, Mayur Datar, Ashutosh Garg, and Shyamsundar Rajaram. 2007.  Google news personalization: scalable online collaborative filtering. In _WWW,2010_. ACM, 271–280.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT,2019_. Association for Computational Linguistics, 4171–4186.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In _IJCAI,2017_. ijcai.org, 1725–1731.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016\.  A Deep Relevance Matching Model for Ad-hoc Retrieval. In _Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016_ , Snehasis Mukhopadhyay, ChengXiang Zhai, Elisa Bertino, Fabio Crestani, Javed Mostafa, Jie Tang, Luo Si, Xiaofang Zhou, Yi Chang, Yunyao Li, and Parikshit Sondhi (Eds.). ACM, 55–64.  https://doi.org/10.1145/2983323.2983769
  * Hofstätter et al. (2021) Sebastian Hofstätter, Bhaskar Mitra, Hamed Zamani, Nick Craswell, and Allan Hanbury. 2021\.  Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_ , Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (Eds.). ACM, 1349–1358.  https://doi.org/10.1145/3404835.3462889
  * Hu et al. (2020) Linmei Hu, Siyong Xu, Chen Li, Cheng Yang, Chuan Shi, Nan Duan, Xing Xie, and Ming Zhou. 2020\.  Graph Neural News Recommendation with Unsupervised Preference Disentanglement. In _ACL,2020_. Association for Computational Linguistics, 4255–4264.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _CIKM,2013_. ACM, 2333–2338.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR,2015_.
  * Koren et al. (2009) Yehuda Koren, Robert M. Bell, and Chris Volinsky. 2009\.  Matrix Factorization Techniques for Recommender Systems.  _Computer_ 42, 8 (2009), 30–37.
  * Lavie et al. (2010) Talia Lavie, Michal Sela, Ilit Oppenheim, Ohad Inbar, and Joachim Meyer. 2010.  User attitudes towards news content personalization.  _Int. J. Hum. Comput. Stud._ 68, 8 (2010), 483–495.
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010\.  A contextual-bandit approach to personalized news article recommendation. In _Proceedings of the 19th International Conference on World Wide Web, WWW 2010, Raleigh, North Carolina, USA, April 26-30, 2010_ , Michael Rappa, Paul Jones, Juliana Freire, and Soumen Chakrabarti (Eds.). ACM, 661–670.  https://doi.org/10.1145/1772690.1772758
  * Li et al. (2011) Lei Li, Dingding Wang, Tao Li, Daniel Knox, and Balaji Padmanabhan. 2011.  SCENE: a scalable two-stage personalized news recommendation system. In _SIGIR,2011_. ACM, 125–134.
  * Lian et al. (2020) Defu Lian, Yongji Wu, Yong Ge, Xing Xie, and Enhong Chen. 2020.  Geography-Aware Sequential Location Recommendation. In _KDD,2020_. ACM, 2009–2019.
  * Lian et al. (2018) Jianxun Lian, Fuzheng Zhang, Xing Xie, and Guangzhong Sun. 2018\.  Towards Better Representation Learning for Personalized News Recommendation: a Multi-Channel Deep Fusion Approach. In _IJCAI,2018_. ijcai.org, 3805–3811.
  * Liu et al. (2020) Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. 2020.  AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction. In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_ , Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (Eds.). ACM, 2636–2645.  https://doi.org/10.1145/3394486.3403314
  * Liu et al. (2010) Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. 2010\.  Personalized news recommendation based on click behavior. In _IUI,2010_. ACM, 31–40.
  * Liu et al. (2019) Zheng Liu, Yu Xing, Fangzhao Wu, Mingxiao An, and Xing Xie. 2019.  Hi-Fi Ark: Deep User Representation via High-Fidelity Archive Network. In _IJCAI,2019_. ijcai.org, 3059–3065.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017\.  Embedding-based News Recommendation for Millions of Users. In _SIGKDD,2017_. ACM, 1933–1942.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global Vectors for Word Representation. In _ACL,2014_. ACL, 1532–1543.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization Machines. In _ICDM,2010_. IEEE Computer Society, 995–1000.
  * Rendle (2012) Steffen Rendle. 2012\.  Factorization Machines with libFM.  _ACM Trans. Intell. Syst. Technol._ 3, 3 (2012), 57:1–57:22.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _J. Mach. Learn. Res._ 15, 1 (2014), 1929–1958.
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020\.  Fine-grained Interest Matching for Neural News Recommendation. In _ACL,2020_. Association for Computational Linguistics, 836–845.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018\.  DKN: Deep Knowledge-Aware Network for News Recommendation. In _WWW,2018_. ACM, 1835–1844.
  * Wang et al. (2019) Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. 2019.  Knowledge Graph Convolutional Networks for Recommender Systems. In _WWW,2019_. ACM, 3307–3313.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019c.  Neural News Recommendation with Attentive Multi-View Learning. In _IJCAI,2019_. ijcai.org, 3863–3869.
  * Wu et al. (2019d) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019d.  NPA: Neural News Recommendation with Personalized Attention. In _SIGKDD,2019_. ACM, 2576–2584.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Yongfeng Huang, and Xing Xie. 2019b.  Neural News Recommendation with Topic-Aware News Representation. In _ACL,2019_. Association for Computational Linguistics, 1154–1159.
  * Wu et al. (2019e) Chuhan Wu, Fangzhao Wu, Mingxiao An, Tao Qi, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019e.  Neural News Recommendation with Heterogeneous User Behavior. In _EMNLP-IJCNLP,2019_. Association for Computational Linguistics, 4873–4882.
  * Wu et al. (2019f) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019f.  Neural News Recommendation with Multi-Head Self-Attention. In _EMNLP-IJCNLP,2019_. Association for Computational Linguistics, 6388–6393.
  * Wu et al. (2019g) Chuhan Wu, Fangzhao Wu, Tao Qi, Suyu Ge, Yongfeng Huang, and Xing Xie. 2019g.  Reviews Meet Graphs: Enhancing User and Item Representations for Recommendation with Hierarchical Attentive Graph Neural Network. In _EMNLP-IJCNLP,2019_. Association for Computational Linguistics, 4883–4892.
  * Wu et al. (2020) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020.  MIND: A Large-scale Dataset for News Recommendation. In _ACL,2020_. Association for Computational Linguistics, 3597–3606.
  * Wu et al. (2019a) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019a.  Session-Based Recommendation with Graph Neural Networks. In _AAAI,2019_. AAAI Press, 346–353.
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In _SIGIR,2017_. ACM, 55–64.
  * Yu and Koltun (2016) Fisher Yu and Vladlen Koltun. 2016.  Multi-Scale Context Aggregation by Dilated Convolutions. In _ICLR,2016_.
  * Yu et al. (2019) Zeping Yu, Jianxun Lian, Ahmad Mahmoody, Gongshen Liu, and Xing Xie. 2019.  Adaptive User Modeling with Long and Short-Term Preferences for Personalized Recommendation. In _IJCAI,2019_. ijcai.org, 4213–4219.
  * Yuan et al. (2019) Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, and Songlin Hu. 2019.  Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots. In _EMNLP-IJCNLP,2019_. Association for Computational Linguistics, 111–120.
  * Zheng et al. (2018) Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018.  DRN: A Deep Reinforcement Learning Framework for News Recommendation. In _WWW,2018_. ACM, 167–176.