Harmonic chain far from equilibrium: single-file diffusion, long-range order,
and hyperuniformity

Harukuni Ikeda1$\star$

1 Department of Physics, Gakushuin University, 1-5-1 Mejiro, Toshima-ku, Tokyo
171-8588, Japan

⋆<EMAIL_ADDRESS>
## Abstract

In one-dimensional many-particle systems, diffusion is strongly suppressed
because particles cannot bypass each other. For this reason, the mean-square
displacement (MSD) increases in proportion to the square of time, ${\rm
MSD}(t)\sim t^{1/2}$ in equilibrium. This phenomenon is called single-file
diffusion. Single-file diffusion is observed even in the harmonic lattice,
which prevents the existence of long-range order in one dimension. We here
study single-file diffusion of the harmonic chain far from equilibrium. We
consider the following four types of driving forces that do not satisfy the
detailed balance: (i) temporally correlated noise with the power-low Fourier-
spectrum $D(\omega)\sim\omega^{-2\alpha}$, (ii) center-of-mass conserving
noise, (iii) periodic driving force, and (iv) periodic deformations of
particles. For the driving force (i) with $\alpha>-1/4$, we observe ${\rm
MSD}(t)\sim t^{1/2+2\alpha}$ for large $t$. On the other hand, for the driving
forces (i) with $\alpha<-1/4$ and (ii)-(iv), MSD remains finite. As a
consequence, the harmonic chain exhibits the long-range order even in one
dimension, which is prohibited by the Mermin-Wagner theorem in equilibrium. We
also discuss that the model shows hyperuniformity when driven by the driving
forces (i) with $\alpha<0$ and (ii)-(iv).

###### Contents

  1. 1 Introduction
  2. 2 Model and physical quantities
    1. 2.1 Model
    2. 2.2 Physical quantities
  3. 3 Temporally correlated noise
    1. 3.1 Settings
    2. 3.2 Mean-squared displacement
    3. 3.3 Order parameter
    4. 3.4 Giant number fluctuation and hyperuniformity
  4. 4 Center of mass conserving dynamics
    1. 4.1 Settings
    2. 4.2 Mean-squared displacement
    3. 4.3 Order parameter
    4. 4.4 Hyperuniformity
  5. 5 Periodically driven system
    1. 5.1 Settings
    2. 5.2 Mean-squared displacement
    3. 5.3 Order parameter
    4. 5.4 Hyperuniformity
  6. 6 Periodically deforming particles
    1. 6.1 Settings
    2. 6.2 Mean-squared displacement
    3. 6.3 Order parameter
    4. 6.4 Hyperuniformity
  7. 7 Summary

##  1 Introduction

In equilibrium, the mean-squared displacement (MSD) of a particle increases
linearly with time $t$, ${\rm MSD}(t)\sim t$ [1]. In one-dimensional many-
particle systems, particles cannot bypass one another, resulting in a strong
suppression of diffusion. Consequently, the MSD exhibits the sub-diffusion
${\rm MSD}\sim t^{1/2}$ [2, 3, 4, 5, 6]. This phenomenon is known as single-
file diffusion. The simplest model to observe single-file diffusion is the
one-dimensional harmonic chain, where point-like particles on a line are
connected by harmonic springs [2]. The model is often recognized as a toy
model of a one-dimensional crystal [7, 8]. However, as proved by Mermin and
Wagner, the long-range order cannot exist in one and two dimensions [9, 8].
This implies that the particles will diffuse away from their lattice positions
after a sufficiently long time. In one dimension, this diffusion proportional
to the square root of the time $t$, ${\rm MSD}\sim t^{1/2}$, as in the case of
standard single-file diffusion [2]. The aim of this study is to elucidate how
the above equilibrium behaviors are modified for systems far from equilibrium.
For this purpose, we consider the harmonic chain driven by the following four
types of driving forces that do not satisfy the detailed balance [1].

Firstly, we consider noise with power-law temporal correlation. The existence
of temporal correlations in the noise implies that the noise itself has
internal degrees of freedom. In particular, in non-equilibrium systems showing
self-organized criticality, the Fourier spectrum of noise for small frequency
$\omega$ shows $D(\omega)\sim\omega^{-2\alpha}$ with positive exponent
$\alpha>0$ [10]. The Fourier spectrum with negative $\alpha$ has also been
observed for one-dimensional quasi-periodic patterns [11,
oǧuz2019hyperuniformity], and thus may give useful insights for systems driven
by quasi-periodic driving forces. We show that for $\alpha>-1/4$, the mean-
squared displacement behaves as ${\rm MSD}\sim t^{1/2+2\alpha}$ for large $t$.
For $\alpha<-1/4$, on the contrary, the diffusion is completely suppressed,
and ${\rm MSD}$ converges to a finite value in the long time limit. As a
consequence of the absence of diffusion, the model exhibits the long-range
translational order, which is prohibited in equilibrium [9, 13]. Furthermore,
we show that the static structure factor $S(q)$ for a small wave number $q$
behaves as $S(q)\sim q^{-4\alpha}$. For $\alpha>0$, $S(q)$ diverges in the
limit $q\to 0$, meaning that the large-scale density fluctuation is
anomalously enhanced. This property is referred to as the giant number
fluctuation [14, 15]. On the contrary, for $\alpha<0$, $S(q)\to 0$ in the
limit $q\to 0$, meaning that the large-scale density fluctuation is highly
suppressed. This property is known as hyperuniformity [16].

Secondly, we consider the noise that conserves the center of mass. In previous
work, Hexner and Levine have shown that the density fluctuation is highly
suppressed for a system far from equilibrium, conserving the center of mass
[17]. Recently, Galliano et al. [18] have shown that a center-of-mass
conserving system shows the long-range crystalline order even in $d=2$, as a
consequence of the suppression of the fluctuation. Does the long-range order
also emerge in $d=1$? The dynamics considered by Galliano et al. is hard to
implement to the lattice model, so we here introduce a modified simpler
dynamics that also conserves the center of mass. We show that the harmonic
chain driven by this dynamics indeed possesses the long-range translational
order [18]. We also show that the static structure factor for small $q$
behaves as $S(q)\sim q^{2}$, meaning that the model shows hyperuniformity
[16].

Thirdly, we consider periodic driving forces. We investigate the harmonic
chain driven by temporally periodic but spatially uncorrelated driving forces.
We show that ${\rm MSD}$ oscillates with the same frequency as that of the
driving force, and the order parameter takes a finite value. We also show
$S(q)\sim q^{2}$ for $q\ll 1$, meaning that the model shows hyperuniformity
[16].

Finally, we consider periodically deforming particles in one dimension, which
was originally introduced as a model to describe dense biological tissues
[19]. The driving force of the model oscillates with constant frequency and
simultaneously conserves the center of mass. Under the harmonic approximation,
the model can be reduced to the one-dimensional harmonic chain with
oscillating natural lengths. We show that the mean-squared displacement
oscillates with the same frequency as that of the driving force, and the
translational order parameter takes a finite value. We also show that the
model exhibits stronger hyperuniformity than that of the center-of-mass
conserving noise and periodic driving forces: $S(q)\sim q^{4}$ for $q\ll 1$
[16].

##  2 Model and physical quantities

Here, we introduce the model and define important physical quantities.

###  2.1 Model

We consider the harmonic chain driven by the following dynamics [1]:

$\displaystyle\dot{x}_{j}=K(x_{j+1}+x_{j-1}-2x_{j})+\xi_{j}(t).$ (1)

where $x_{j}$, $K$, and $\xi_{j}$ denote the position of the $j$-th particle,
spring constant, and deriving force, respectively. Let $a$ be the lattice
constant, $R_{j}=ja$ be the equilibrium position of the $j$-th particle, and
$u_{j}=x_{j}-R_{j}$ be the displacement from the equilibrium position. The
dynamical equation for $u_{j}$ is then written as

$\displaystyle\dot{u}_{j}=K(u_{j+1}+u_{j-1}-2u_{j})+\xi_{j}(t).$ (2)

It is convenient to introduce the Fourier and inverse Fourier transformations
of the displacement $u_{j}(t)$ [1, 7]:

$\displaystyle u_{k}(t)=\frac{1}{\sqrt{N}}\sum_{j=1}^{N}q_{j}(t)e^{\frac{2\pi
i}{N}jk},$ $\displaystyle
q_{k}(t)=\frac{1}{\sqrt{N}}\sum_{j=1}^{N}u_{j}(t)e^{-\frac{2\pi i}{N}jk},$ (3)

where $N$ denotes the number of particles. We impose the periodic boundary
condition $x_{N+1}=x_{1}$. Eq. (2) is diagonalized in the Fourier space:

$\displaystyle\dot{q}_{k}(t)=-\lambda_{k}q_{k}(t)+f_{k}(t),$ (4)

where

$\displaystyle\lambda_{k}=2K\left[1-\cos\left(\frac{2\pi k}{N}\right)\right],$
(5)

and

$\displaystyle
f_{k}(t)=\frac{1}{\sqrt{N}}\sum_{j=1}^{N}\xi_{j}(t)e^{-\frac{2\pi i}{N}jk}$
(6)

denotes the Fourier transform of the driving force $\xi_{j}$. To satisfy the
translational symmetry, we assume that the mean and variance of $f_{k}(t)$ are
given by

$\displaystyle\overline{f_{k}(t)}=0,$
$\displaystyle\overline{f_{k}(t)f_{l}(t^{\prime})}=\delta_{k,-l}D_{k}(t-t^{\prime}).$
(7)

Using the Fourier transformation w.r.t $t$, Eq. (4) can be solved as

$\displaystyle q_{k}(\omega)=\frac{f_{k}(\omega)}{i\omega+\lambda_{k}}.$ (8)

The two point correlation is then calculated as

$\displaystyle\left\langle
q_{k}(\omega)q_{l}(\omega^{\prime})\right\rangle=2\pi\delta_{k,-l}\delta(\omega+\omega^{\prime})C_{k}(\omega),$
(9)

where

$\displaystyle
C_{k}(\omega)=\frac{D_{k}(\omega)}{\omega^{2}+\lambda_{k}^{2}}.$ (10)

The inverse Fourier transform w.r.t. $\omega$ yields

$\displaystyle\left\langle
q_{k}(t)q_{-k}(0)\right\rangle=\frac{1}{2\pi}\int_{-\infty}^{\infty}d\omega
e^{i\omega
t}\frac{D_{k}(\omega)}{\omega^{2}+\lambda_{k}^{2}}=\frac{1}{\pi}\int_{0}^{\infty}d\omega\frac{D_{k}(\omega)\cos(\omega
t)}{\omega^{2}+\lambda_{k}^{2}},$ (11)

where we used the time-reversal symmetry of the correlation:
$D_{k}(\omega)=D_{k}(-\omega)$.

###  2.2 Physical quantities

Using the Parseval’s identity, the mean-squared displacement is calculated as
follows:

$\displaystyle{\rm MSD}(t)$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\left\langle\Delta
u_{i}(t)^{2}\right\rangle$
$\displaystyle=\frac{1}{N}\sum_{k}\left\langle\Delta q_{k}(t)\Delta
q_{-k}(t)\right\rangle$ $\displaystyle=\frac{1}{N}\sum_{k}\left\langle
q_{k}(t)q_{-k}(t)\right\rangle+\frac{1}{N}\sum_{k}\left\langle
q_{k}(t)q_{-k}(t)\right\rangle-\frac{2}{N}\sum_{k}{\rm Re}\left\langle
q_{k}(t)q_{-k}(0)\right\rangle$
$\displaystyle=\frac{2}{N}\sum_{k}\frac{1}{\pi}\int_{0}^{\infty}d\omega
D_{k}(\omega)\frac{1-\cos(\omega t)}{\omega^{2}+\lambda_{k}^{2}},$ (12)

where we used abbreviations: $\Delta u_{i}(t)=u_{i}(t)-u_{i}(0)$ and $\Delta
q_{k}(t)=q_{k}(t)-q_{k}(0)$.

To quantify the translational order, we define the order parameter as follows:

$\displaystyle
R=\frac{1}{N}\left\langle\sum_{i=1}^{N}e^{i\frac{2\pi}{a}x_{i}}\right\rangle=\left\langle
e^{i\frac{2\pi u_{1}}{a}}\right\rangle.$ (13)

If the system has the long-range order, $R$ remains finite in the
thermodynamic limit $N\to\infty$.

For perfect crystals, the static structure factor $S(q)$ in the limit of the
small wave number vanishes: $\lim_{q\to 0+}S(q)=0$. In other words, the
density fluctuation is highly suppressed for small $q$. This property is
referred to as hyperuniformity [16]. In equilibrium, on the contrary,
$\lim_{q\to 0+}S(q)$ converges to a finite value, namely, the thermal
fluctuation destructs hyperuniformity [20]. Does hyperuniformity survive under
the athermal noise considered in this work? To answer this question, we
calculate the static structure factor $S(q)$ for a small wave number $q=2\pi
k/(Na)\ll 1$ [20]:

$\displaystyle S(q)$
$\displaystyle=\left\langle\frac{1}{N}\left|\sum_{i=1}^{N}e^{iqx_{i}}\right|^{2}\right\rangle$
$\displaystyle\approx\frac{1}{N}\left\langle\left|\sum_{j=1}^{N}e^{iqR_{j}}\right|^{2}\right\rangle+q^{2}\left\langle\frac{1}{N}\left|\sum_{j=1}^{N}u_{j}e^{iqR_{j}}\right|^{2}\right\rangle$
$\displaystyle=S_{0}(q)+q^{2}\left\langle q_{k}q_{-k}\right\rangle$
$\displaystyle\approx q^{2}\left\langle q_{k}q_{-k}\right\rangle.$ (14)

where
$S_{0}(q)=\left\langle\left|\sum_{j=1}^{N}e^{iqR_{j}}\right|^{2}\right\rangle/N$
denotes the static structure factor of the one-dimensional lattice, which has
delta peaks at $q=2\pi n/a$, $n=0,1,2,\cdots$ and can be ignored for
sufficiently small but finite $q$. Eq. (14) allows us to discuss
hyperuniformity from the scaling behavior of $\left\langle
q_{k}q_{-k}\right\rangle$ for small $k$.

##  3 Temporally correlated noise

###  3.1 Settings

Here we consider the Gaussian color noise of zero mean and variance

$\displaystyle\left\langle\xi_{i}(t)\xi_{j}(t^{\prime})\right\rangle=\delta_{ij}D(t).$
(15)

In previous work, single-file diffusion of active particles has been
investigated [21]. In that case, the correlation of the noise $D(t)$ decays
exponentially, which leads to the same scaling as that in equilibrium ${\rm
MSD}(t)\sim t^{1/2}$ [21]. As we will see later, the scaling is altered if
$D(t)$ has the power-low tail. For simplicity, we assume that $D(t)$ in the
Fourier space is written as

$\displaystyle
D(\omega)=\frac{2T\left|\omega\right|^{-2\alpha}}{\sec(\alpha\pi)},$
$\displaystyle-1/2<\alpha<1/2.$ (16)

Here, the pre-factor $1/\sec(\alpha\pi)$ has been chosen to simplify the final
result, and the value of $\alpha$ is restricted to $-1/2<\alpha<1/2$ to
converge the correlation function, as we will see later. When $\alpha=0$, the
model satisfies the detailed balance [1], and thus $\xi_{i}$ can be identified
with the thermal noise at temperature $T$. The power-low spectrum of the noise
appears for non-equilibrium systems showing self-organized criticality [10]
and is often referred to as the $1/f$ noise [22]. The power-low spectrum also
appears for the Fourier spectrum of quasi-periodic patterns. In Ref.
[oǧuz2019hyperuniformity], the authors showed that the Fourier spectrum of
one-dimensional quasi-periodic patterns exhibits the power-low behaviors for
small $\omega$ with $\alpha\in[-3/2,1]$. Also, in Ref. [20], the authors
argued that small perturbations to one-dimensional periodic patterns yield the
power-low spectrum for $\alpha\in[-1,0]$. Therefore, the model driven by the
noise with correlation Eq. (16) would give useful insights for quasi-
periodically and periodically driven systems.

###  3.2 Mean-squared displacement

In the thermodynamic limit $N\to\infty$, the mean-squared displacement Eq.
(12) is calculated as

$\displaystyle{\rm MSD}(t)$
$\displaystyle=\frac{1}{\pi^{2}}\int_{0}^{2\pi}d\theta\int_{0}^{\infty}d\omega
D(\omega)\frac{1-\cos(\omega
t)}{\omega^{2}+\left[2K(1-\cos\theta)\right]^{2}}$
$\displaystyle=\frac{2T}{\pi^{2}\sec({\pi\alpha})}\int_{0}^{2\pi}d\theta\int_{0}^{\infty}d\omega\left|\omega\right|^{-2\alpha}\frac{1-\cos(\omega
t)}{\omega^{2}+\left[2K(1-\cos\theta)\right]^{2}}.$ (17)

The integral w.r.t $\theta$ can be calculated as

$\displaystyle\int_{0}^{2\pi}d\theta\frac{1}{\omega^{2}+\left[2K(1-\cos\theta)\right]^{2}}=2\pi\sqrt{\frac{\omega+\sqrt{\omega^{2}+16K^{2}}}{2\omega^{3}(\omega^{2}+16K^{2})}}\sim\begin{cases}\frac{\pi}{\sqrt{2K\omega^{3}}}&\left|\omega\right|\ll
1\\\ \frac{2\pi}{\omega^{2}}&\left|\omega\right|\gg 1.\end{cases}$ (18)

Using this result, one can deduce the scaling behavior of ${\rm MSD}$ for
$t\ll 1$ as follows:

$\displaystyle{\rm MSD}(t)\sim At^{1+2\alpha}\ (t\ll 1),$ (19)

where $A$ denotes a constant. The scaling exponent agrees with that of a free-
particle driven by the temporally correlated noise [23]. For $t\gg 1$ and
$\alpha>-1/4$, we get

$\displaystyle{\rm MSD}(t)\sim Bt^{\frac{1}{2}+2\alpha}\ (t\gg 1),$ (20)

where $B$ denotes a constant. For $\alpha=0$, one recovers the scaling of
single-file diffusion in equilibrium ${\rm MSD}\sim t^{1/2}$ [2, 6]. For
$\alpha<-1/4$, the ${\rm MSD}$ in the long time limit converges to
$\lim_{t\to\infty}{\rm MSD}(t)=2\left\langle u_{1}^{2}\right\rangle$. We plot
${\rm MSD}$ for several $\alpha$ in Fig. 1.

Figure 1: Mean-squared displacement of harmonic chain driven by temporally
correlated noise. Markers denote exact results. Dashed and solid lines
represent short and long-time asymptotic behaviors, respectively. For
simplicity, we set $K=1$ and $T=1$.

###  3.3 Order parameter

The equal time correlation in the Fourier space is

$\displaystyle\left\langle
q_{k}q_{-k}\right\rangle=\frac{1}{\pi}\int_{0}^{\infty}\frac{D(\omega)d\omega}{\omega^{2}+\lambda_{k}^{2}}=\frac{2T}{\pi\sec(\alpha\pi)(\lambda_{k})^{1+2\alpha}}\int_{0}^{\infty}dx\frac{\left|x\right|^{-2\alpha}}{x^{2}+1}.=\frac{T}{(\lambda_{k})^{1+2\alpha}}.$
(21)

Note that the integral converges only when $-1/2<\alpha<1/2$. When $\alpha=0$,
we get the law of equipartition:

$\displaystyle\left\langle q_{k}q_{-k}\right\rangle=\frac{T}{\lambda_{k}}.$
(22)

In the real space, we get

$\displaystyle\left\langle u_{1}^{2}\right\rangle$
$\displaystyle=\frac{1}{N}\sum_{j=1}^{N}\left\langle u_{j}^{2}\right\rangle$
$\displaystyle=\frac{1}{N}\sum_{k=1}^{N}\left\langle q_{k}q_{-k}\right\rangle$
$\displaystyle=\frac{T}{N}\sum_{k=1}^{N}\frac{1}{\lambda_{k}^{1+2\alpha}}$
$\displaystyle=\frac{T}{\pi}\int_{0}^{\pi}d\theta\frac{1}{\left[2K(1-\cos\theta)\right]^{1+2\alpha}}.$
(23)

For $\alpha<-1/4$, the integral converges to

$\displaystyle\frac{\left\langle
u_{1}^{2}\right\rangle}{T}=-\frac{\alpha\Gamma(-1/2-2\alpha)}{2^{1+4\alpha}K^{1+2\alpha}\pi^{1/2}\Gamma(1-2\alpha)},$
(24)

while for $\alpha\geq-1/4$, the fluctuation diverges, $\left\langle
u_{1}^{2}\right\rangle\to\infty$. We plot $\left\langle
u_{1}^{2}\right\rangle$ in Fig. 2 (a). Since $\xi_{i}$ is a Gaussian random
number, the solution of the linear differential equation Eq. (2), $u_{1}$,
also becomes a Gaussian random number [1]. Therefore, the order parameter can
be calculated as

$\displaystyle R=\left\langle e^{i\frac{2\pi
u_{i}}{a}}\right\rangle=\exp\left[-\frac{2\pi^{2}}{a^{2}}\left\langle
u_{1}^{2}\right\rangle\right].$ (25)

We plot $R$ in Fig. 2 (b). The order parameter $R$ has a finite value for
$\alpha<-1/4$ and vanishes for $\alpha\geq-1/4$. This result is consistent
with the spherical model driven by the temporally correlated noise [24].

Figure 2: Physical quantities of harmonic chain driven by temporally
correlated noise. (a) $\alpha$ dependence of fluctuation $\left\langle
u_{1}^{2}\right\rangle$. $\left\langle u_{1}^{2}\right\rangle$ has a finite
value for $\alpha<-1/4$ and diverges at $\alpha=-1/4$. (b) $\alpha$ dependence
of the order parameter for several temperatures. The order parameter has a
finite value for $\alpha<-1/4$ and vanishes at $\alpha=-1/4$. For simplicity,
we set $K=1$ and $a=1$.

###  3.4 Giant number fluctuation and hyperuniformity

For a small wave number $q=2\pi k/(Na)\ll 1$, $S(q)$ is approximated as

$\displaystyle S(q)\approx q^{2}\left\langle
q_{k}q_{-k}\right\rangle=T\frac{q^{2}}{\lambda_{k}^{1+2\alpha}}\approx\frac{Tq^{-4\alpha}}{(Ka^{2})^{1+2\alpha}}.$
(26)

For $\alpha>0$, $S(q)\to\infty$ in the limit of small $q$, meaning that the
large-scale density fluctuation is anomalously enhanced. This property is
referred to as the giant number fluctuation [14, 15]. On the contrary, for
$\alpha<0$, $S(q)\to 0$ in the limit $q\to 0$, i.e., the large-scale
fluctuation is highly suppressed. This property is referred to as the
hyperuniformity [16].

The large $t$ scaling behavior of ${\rm MSD}(t)$ is primarily determined by
the behavior of $D(\omega)$ at small $\omega$. So, one can remove the
ultraviolet divergence of the correlation function for $\alpha<-1/2$ by
introducing the phenomenological cut-off to the spectrum: $D(\omega)=0$ for
$\omega>\omega_{c}$. Then the correlation Eq. (21) for $q=2\pi k/(Na)\ll 1$
can be calculated as

$\displaystyle\left\langle
q_{k}q_{-k}\right\rangle=\frac{T}{\pi}\int_{0}^{\omega_{c}}d\omega\frac{\left|\omega\right|^{-2\alpha}}{\omega^{2}+\lambda_{k}^{2}}\sim\frac{T}{\pi}\int_{0}^{\omega_{c}}d\omega\left|\omega\right|^{-2\alpha-2},$
$\displaystyle\alpha<-1/2.$ (27)

The integral converges to a constant value for $\alpha<-1/2$. Therefore, we
get $S(q)\approx q^{2}\left\langle q_{k}q_{-k}\right\rangle\sim q^{2}$, which
is consistent with the limit $\alpha\to-1/2$ of Eq. (26). As we will see
later, this result is also consistent with the periodically driven system,
meaning that the temporally correlated noise with $\alpha<-1/2$ can be
identified with the periodic driving force at the level of the linear model
considered in this manuscript.

##  4 Center of mass conserving dynamics

###  4.1 Settings

Hyperuniformity is a phenomenon where the density fluctuation becomes much
smaller than what would be expected from the central limit theorem.
Hyperuniformity has been reported in various systems, such as crystals,
quasicrystals [20, oǧuz2019hyperuniformity, 25], and chiral active matter [26,
27, 28]. In general, the physical mechanisms causing hyperuniformity would
depend on the details of the systems. However, Hexner and Levine have shown
that a system far from equilibrium can generally show hyperuniformity if it
conserves the center of mass [17]. Recently, Galliano et al. [18] argued that
the suppression of the density fluctuation also stabilizes the long-range
crystalline order even in $d=2$, which is prohibited by the Mermin-Wagner
theorem in equilibrium [9]. Here, we argue that the same scenario also holds
in $d=1$.

To preserve the center of mass, we here consider the following driving force:

$\displaystyle\xi_{j}(t)=\eta_{j}(t)-\eta_{j-1}(t),$ (28)

where $\eta_{j}(t)$ is a Gaussian random number of zero mean and variance:

$\displaystyle\left\langle\eta_{i}(t)\eta_{j}(t^{\prime})\right\rangle=T\delta_{ij}\delta(t-t^{\prime}).$
(29)

Then, under the periodic boundary condition, one can see that the dynamics Eq.
(1) preserves the center of mass:

$\displaystyle\sum_{j=1}^{N}\dot{x}_{j}(t)=0.$ (30)

The Fourier transform of $\xi_{j}(t)$ satisfies

$\displaystyle\left\langle f_{k}(t)\right\rangle=0,$
$\displaystyle\left\langle
f_{k}(t)f_{l}(t)\right\rangle=4\delta_{k,-l}T\left[1-\cos\left(\frac{2\pi
k}{N}\right)\right]\delta(t-t^{\prime})=\delta_{k,-l}D_{k}(t-t^{\prime}),$
(31)

where

$\displaystyle D_{k}(t)=\frac{2T\lambda_{k}}{K}\delta(t-t^{\prime}).$ (32)

###  4.2 Mean-squared displacement

The mean-squared displacement in the thermodynamics limit $N\to\infty$ is
calculated as

$\displaystyle{\rm MSD}(t)$
$\displaystyle=\frac{1}{\pi^{2}}\int_{0}^{2\pi}d\theta\int_{0}^{\infty}d\omega
D_{k}(\omega)\frac{1-\cos(\omega
t)}{\omega^{2}+\left[2K(1-\cos\theta)\right]^{2}}$
$\displaystyle=\frac{2T}{\pi^{2}K}\int_{0}^{\infty}d\omega(1-\cos(\omega
t))\int_{0}^{2\pi}d\theta\frac{2K(1-\cos\theta)}{\omega^{2}+\left[2K(1-\cos\theta)\right]^{2}}.$
(33)

We plot the mean-squared displacement in Fig. 3. One can see that ${\rm MSD}$
converges to a finite value in the long-time limit, $\lim_{t\to\infty}{\rm
MSD}=2\left\langle u_{1}^{2}\right\rangle$.

Figure 3: Mean-squared displacement of harmonic chain driven by center-of-mass
conserving noise. Markers denote exact results. Solid lines represent long
time asymptotic behavior: ${\rm MSD}\sim 2\left\langle
u_{1}^{2}\right\rangle$. For simplicity, we set $K=1$ and $T=1$.

###  4.3 Order parameter

Repeating the same analysis as in Eq. (11), we get

$\displaystyle\left\langle q_{k}q_{-k}\right\rangle=\frac{T}{K}.$ (34)

The cage size is then

$\displaystyle\left\langle
u_{1}^{2}\right\rangle=\frac{1}{N}\sum_{k=1}^{N}\left\langle
q_{k}q_{-k}\right\rangle=\frac{T}{K}.$ (35)

Since $\xi_{j}(t)$ is a Gaussian random number, $u_{1}$ also follows the
Gaussian distribution. Thus, the order parameter is

$\displaystyle R=\frac{1}{N}\sum_{j=1}^{N}e^{\frac{2\pi
i}{a}x_{j}}=\left\langle e^{\frac{2\pi
i}{a}u_{1}}\right\rangle=\exp\left[-\frac{2\pi^{2}T}{a^{2}K}\right].$ (36)

The order parameter has a finite value, meaning that the model driven by the
center of mass conserving dynamics has the long-order even in $d=1$.

###  4.4 Hyperuniformity

Hexner et al. argued that the density fluctuation is anomalously suppressed in
the center of mass conserving dynamics [17]. To see this, we calculate the
static structure factor $S(q)$ for small $q=2\pi k/N\ll 1$:

$\displaystyle S(q)\approx q^{2}\left\langle
q_{k}q_{-k}\right\rangle=\frac{T}{K}q^{2}.$ (37)

$S(q)$ vanishes in the limit $q\to 0$, meaning that the large-scale
fluctuation is highly suppressed. This is the signature of the hyperuniformity
[16].

##  5 Periodically driven system

###  5.1 Settings

Here, we consider the periodic driving force. For a concrete example, we
consider chiral active particles in one dimension. Chiral active particles are
particles that exhibit circular motions [29]. A popular mathematical model to
describe this circular motion is [30]

$\displaystyle\dot{x}=\sqrt{2T}\cos\theta+\xi_{x},$
$\displaystyle\dot{y}=\sqrt{2T}\sin\theta+\xi_{y},$
$\displaystyle\dot{\theta}=\omega_{0}+\xi_{\theta},$ (38)

where $\xi_{x,y,\theta}$ denotes the noise. For simplicity, hereafter, we only
consider the model in the absence of the noise $\xi_{x,y,\theta}=0$. If a
chiral active particle is confined in a one-dimensional channel along the $x$
direction, one can only consider the motion along that direction:
$\dot{x}=\sqrt{2T}\cos(\omega_{0}t+\theta(0))$. How does this periodic nature
of the driving force affect the collective motion? To model the corrective
excitation of chiral active particles in one dimension, we consider the
harmonic chain Eq. (1) driven by the following periodic function [31]:

$\displaystyle\xi_{j}(t)=\sqrt{2T}\cos(\omega_{0}t+\psi_{j}),$ (39)

where $\psi_{j}$ denotes a random number uniformly distributed in $[0,2\pi]$.
The mean and variance of $\xi_{j}(t)$ are

$\displaystyle\left\langle\xi_{j}(t)\right\rangle=0,$
$\displaystyle\left\langle\xi_{i}(t)\xi_{j}(t^{\prime})\right\rangle=\delta_{ij}T\cos(\omega_{0}(t-t^{\prime})).$
(40)

When $\omega_{0}=0$, $\xi_{j}(t)=\sqrt{2T}\cos\psi_{j}$ plays the role of the
random field and destructs the long-range order in $d\leq 4$ as predicted by
Imry and Ma [32]. What will happen when $\omega_{0}\neq 0$?

###  5.2 Mean-squared displacement

By using Eq. (12), the MSD in the thermodynamic limit $N\to\infty$ is
calculated as

$\displaystyle{\rm MSD}(t)=2\left[1-\cos(\omega_{0}t)\right]\left\langle
u_{1}^{2}\right\rangle,$ (41)

where

$\displaystyle\left\langle
u_{1}^{2}\right\rangle=\frac{T}{\pi}\int_{0}^{\pi}d\theta\frac{1}{\omega_{0}^{2}+\left[2K(1-\cos\theta)\right]^{2}}.$
(42)

MSD oscillates with the same frequency as that of the driving force
$\omega_{0}$. We plot ${\rm MSD}(t)$ for several $\omega_{0}$ in Fig. 4.

Figure 4: Mean-squared displacement of harmonic chain driven by periodic
driving force. Figure 5: Cage size of harmonic chain driven by periodic
driving force. The cage size diverges in the limit $\omega_{0}\to 0$.

In Fig. 5, we plot $\left\langle u_{1}^{2}\right\rangle$ Eq. 42. The cage size
$\left\langle u_{1}^{2}\right\rangle$ has a finite value for $\omega_{0}>0$
and diverges at $\omega_{0}=0$.

###  5.3 Order parameter

Because the current driving force Eq. (39) is not a Gaussian random variable,
one can not easily calculate the order parameter $R$. So instead, we use the
following inequality:

$\displaystyle R=\left\langle e^{i\frac{2\pi
u_{1}}{a}}\right\rangle=\left\langle\cos\left(\frac{2\pi
u_{1}}{a}\right)\right\rangle\geq 1-\frac{2\pi^{2}}{a^{2}}\left\langle
u_{1}^{2}\right\rangle.$ (43)

Since dynamics Eq. (2) does not depend on $a$, whether or not long-range order
exists is also independent of $a$. In other words, it is sufficient to show
the existence of the long-range order at a particular value of $a$. We chose
$a$ so that $a>\pi\sqrt{2\left\langle u_{1}^{2}\right\rangle}$, then Eq. (43)
leads to $R>0$; the model has the long-range order. The result is consistent
with the recent theoretical study of the spherical model driven by the
periodic driving forces [31].

###  5.4 Hyperuniformity

The static structure factor for $q=2\pi k/(Na)\ll 1$ is calculated as

$\displaystyle S(q)\sim q^{2}\left\langle
q_{k}q_{-k}\right\rangle=\frac{Tq^{2}}{\omega_{0}^{2}+\lambda_{k}^{2}}\sim\begin{cases}Tq^{2}/\omega_{0}^{2}&\omega_{0}>0,\\\
Tq^{-2}/(Ka^{2})^{2}&\omega_{0}=0.\end{cases}$ (44)

The model exhibits hyperuniformity $S(q)\sim q^{2}$ for $\omega_{0}>0$. This
result is consistent with the temporally correlated noise with
$\alpha\leq-1/2$, see the discussion below Eq. (27). Interestingly, the
similar behavior of $S(q)$ has been previously reported for chiral active
matter in $d=2$ [33, 26, 27, 28]. For $\omega_{0}=0$, on the contrary, one
observes $S(q)\sim q^{-2}$. Therefore, $S(q)$ diverges in the limit of the
small $q$. This anomalous enhancement of the large-scale density fluctuation
is referred to as the giant number fluctuation [14, 15]. A similar power-low
divergence of $S(q)$ has been previously reported in active matter in quenched
random potentials [34].

##  6 Periodically deforming particles

###  6.1 Settings

Figure 6: Sketch of (a) periodically deforming particles in one dimension and
(b) harmonic chain where particle interactions are replaced by linear springs.

What will happen if the driving force oscillates with constant frequency and
simultaneously conserves the center of mass? To answer this question, we
consider the model introduced by Tjhung and Berthier [19]. Tissues are often
fluidized by periodic deformation of cells [35]. To model this behavior,
Tjhung and Berthier introduced periodically deforming particles [19]. The one-
dimensional version of the model is written as

$\displaystyle\dot{x}_{j}(t)=-\frac{\partial V_{N}}{\partial x_{j}},$ (45)
$\displaystyle V_{N}=\sum_{i<j}^{N}v(h_{ij}),$ (46)

where $v(h_{ij})$ denotes the one-sided harmonic potential [36]:

$\displaystyle v(h_{ij})=\frac{Kh_{ij}^{2}\theta(-h_{ij})}{2},$ $\displaystyle
h_{ij}=\left|x_{i}-x_{j}\right|-\frac{r_{i}(t)+r_{j}(t)}{2}.$ (47)

Here the diameter of the $i$-th particle $r_{i}(t)$ oscillates with frequency
$\omega_{0}$ [19]:

$\displaystyle r_{i}(t)=a+\sigma\cos(\omega_{0}t+\psi_{i}),$ (48)

where $\psi_{i}$ is a random number distributed uniformly in $[0,2\pi]$. When
$\omega_{0}=0$, $\sigma\cos\psi_{i}$ plays the role of the polydispersity, and
thus, the model clearly does not have the long-range translational order. The
force term in Eq. (45) satisfies Newton’s third law and thus conserves the
center of mass [37].

For sufficiently high density and small $\sigma$, the harmonic approximation
would be justified, and thus the one-sided harmonic potential would be
replaced by the harmonic potential (see Fig. 6):

$\displaystyle v(r_{ij})\approx\frac{Kh_{ij}^{2}}{2}.$ (49)

Taking only the nearest neighbor interactions, one can approximate Eq. (45) as

$\displaystyle\dot{x}_{j}\approx
K(x_{j+1}+x_{j-1}-2x_{j})+K\frac{r_{j+1}-r_{j-1}}{2}.$ (50)

Then, the dynamics of the displacement $u_{j}$ is

$\displaystyle\dot{u}_{j}=K(u_{j+1}+u_{j-1}-2u_{j})+\xi_{j},$ (51)

where

$\displaystyle\xi_{j}(t)=\frac{K\sigma}{2}\left[\cos(\omega_{0}t+\psi_{j+1})-\cos(\omega_{0}t+\psi_{j-1})\right].$
(52)

The Fourier transform of $\xi_{j}$ satisfies

$\displaystyle\left\langle f_{k}(t)\right\rangle=0,$
$\displaystyle\left\langle
f_{k}(t)f_{l}(t^{\prime})\right\rangle=T\left[\sin\left(\frac{2\pi
k}{N}\right)\right]^{2}\delta_{k,-l}\cos(\omega_{0}(t-t^{\prime})),$ (53)

where we defined $T=(K\sigma)^{2}/4$.

###  6.2 Mean-squared displacement

Figure 7: Cage size of periodically deforming particles. The case size has a
finite value for $\omega_{0}>0$ and diverges at $\omega_{0}=0$.

Repeating the same analysis as in the previous sections, we get

$\displaystyle{\rm MSD}(t)=2(1-\cos(\omega_{0}t))\left\langle
u_{1}^{2}\right\rangle,$ (54)

where

$\displaystyle\left\langle
u_{1}^{2}\right\rangle=\frac{T}{\pi}\int_{0}^{\pi}d\theta\frac{(\sin\theta)^{2}}{\omega_{0}^{2}+\left[2K(1-\cos\theta)\right]^{2}}.$
(55)

Therefore, ${\rm MSD}$ shows the periodic motion as in the case of the model
considered in Sec. 5. A similar periodic motion of ${\rm MSD}$ has been
previously reported by a numerical simulation of the periodically deforming
particles in $d=2$ [38].

###  6.3 Order parameter

We plot $\left\langle u_{1}^{2}\right\rangle$ in Fig. 7. The cage size
$\left\langle u_{1}^{2}\right\rangle$ has a finite value for $\omega_{0}>0$.
In this case, using Eq. (43) and repeating the same argument in the previous
section, we can conclude that the model possesses the long-range order. In the
limit $\omega_{0}\to 0$, the cage size diverges $\left\langle
u_{1}^{2}\right\rangle\to\infty$, and thus one can not prove the existence of
the long-range order. This is a natural result because when $\omega_{0}=0$,
the polydispersity $\sigma\cos\psi_{i}$ destructs the long-range order.

###  6.4 Hyperuniformity

The static structure factor for small $q=2\pi k/(Na)$ is

$\displaystyle S(q)\sim q^{2}\left\langle
q_{k}q_{-k}\right\rangle=\frac{Tq^{2}\sin(aq)^{2}}{\omega_{0}^{2}+\lambda_{k}^{2}}\sim\begin{cases}Ta^{2}q^{4}/\omega_{0}^{2}&\omega_{0}>0\\\
T/(Ka)^{2}&\omega_{0}=0.\end{cases}$ (56)

For $\omega_{0}>0$, we observe $S(q)\sim q^{4}$, which is much smaller than
that of the center-of-mass conserving dynamics Eq. (37) and periodic driving
force Eq. (44). This is a consequence of the fact that the driving force Eq.
(52) is a periodic function and simultaneously conserves the center of mass.
For $\omega_{0}=0$, $S(q)$ converges to a finite value for small $q$, meaning
that the polydispersity destructs hyperuniformity.

##  7 Summary

In this work, we investigated the one-dimensional harmonic chain far from
equilibrium. We considered the four types of driving forces that do not
satisfy the detailed balance: (i) temporally correlated noise with power-low
spectrum $D(\omega)\sim\omega^{-2\alpha}$, (ii) center-of-mass conserving
noise, (iii) periodic driving force, (iv) periodic deformation. For the
driving force (i), we found that the model undergoes the anomalous diffusion
${\rm MSD}(t)\sim t^{1/2+2\alpha}$ for $\alpha>-1/4$. On the contrary, for the
driving force (i) with $\alpha<-1/4$, and (ii)–(iv), we found that MSD(t)
remains finite. As a consequence, the translational order parameter has a
finite value, unlike the equilibrium systems where the Mermin-Wagner theorem
prohibits the long-range order in $d=1$. We also discussed hyperuniformity by
observing the small $q$ behavior of the static structure factor $S(q)$. We
found $S(q)\sim q^{-4\alpha}$ for the driving force (i), $S(q)\sim q^{2}$ for
(ii) and (iii), and $S(q)\sim q^{4}$ for (iv). Therefore, the driving force
(i) with $\alpha<0$, and (ii)-(iv) yield hyperuniformity.

For the harmonic potential, the dynamics of the relative displacement $u_{j}$,
Eq. (2), does not depend on the lattice spacing $a$. Therefore, the
qualitative behavior of the model does not depend on the density. What will
happen for more realistic interaction potentials, such as the Lennard-Jones
potential [39], one-sided harmonic potential, and Hertzian potential [36]?
Does the model undergo the fluid-solid phase transition in one dimension? What
is the nature of the transition? Further theoretical and numerical studies
would be beneficial.

#### Funding information

This project has received JSPS KAKENHI Grant Numbers 23K13031.

## References

  * [1] R. Zwanzig,  _Nonequilibrium statistical mechanics_ ,  Oxford university press (2001).
  * [2] S. Alexander and P. Pincus,  _Diffusion of labeled particles on one-dimensional chains_ ,  Phys. Rev. B 18, 2011 (1978),  10.1103/PhysRevB.18.2011.
  * [3] J. Kärger,  _Straightforward derivation of the long-time limit of the mean-square displacement in one-dimensional diffusion_ ,  Phys. Rev. A 45, 4173 (1992),  10.1103/PhysRevA.45.4173.
  * [4] M. Kollmann,  _Single-file diffusion of atomic and colloidal systems: Asymptotic laws_ ,  Phys. Rev. Lett. 90, 180602 (2003),  10.1103/PhysRevLett.90.180602.
  * [5] B. Lin, M. Meron, B. Cui, S. A. Rice and H. Diamant,  _From random walk to single-file diffusion_ ,  Phys. Rev. Lett. 94, 216001 (2005),  10.1103/PhysRevLett.94.216001.
  * [6] A. Taloni, O. Flomenbom, R. Castañeda-Priego and F. Marchesoni,  _Single file dynamics in soft materials_ ,  Soft Matter 13(6), 1096 (2017).
  * [7] N. W. Ashcroft and N. D. Mermin,  _Solid state physics_ ,  Cengage Learning (2022).
  * [8] A. Altland and B. D. Simons,  _Condensed matter field theory_ ,  Cambridge university press (2010).
  * [9] N. D. Mermin and H. Wagner,  _Absence of ferromagnetism or antiferromagnetism in one- or two-dimensional isotropic heisenberg models_ ,  Phys. Rev. Lett. 17, 1133 (1966),  10.1103/PhysRevLett.17.1133.
  * [10] P. Bak, C. Tang and K. Wiesenfeld,  _Self-organized criticality: An explanation of the 1/f noise_ ,  Phys. Rev. Lett. 59, 381 (1987),  10.1103/PhysRevLett.59.381.
  * [11] E. C. Oğuz, J. E. S. Socolar, P. J. Steinhardt and S. Torquato,  _Hyperuniformity of quasicrystals_ ,  Phys. Rev. B 95, 054119 (2017),  10.1103/PhysRevB.95.054119.
  * [12] E. OÇ§uz, J. E. Socolar, P. J. Steinhardt and S. Torquato,  _Hyperuniformity and anti-hyperuniformity in one-dimensional substitution tilings_ ,  Acta Crystallographica Section A: Foundations and Advances 75(1), 3 (2019).
  * [13] H. Nishimori and G. Ortiz,  _Elements of phase transitions and critical phenomena_ ,  Oup Oxford (2010).
  * [14] V. Narayan, S. Ramaswamy and N. Menon,  _Long-lived giant number fluctuations in a swarming granular nematic_ ,  Science 317(5834), 105 (2007),  10.1126/science.1140414.
  * [15] H. P. Zhang, A. Be’er, E.-L. Florin and H. L. Swinney,  _Collective motion and density fluctuations in bacterial colonies_ ,  Proceedings of the National Academy of Sciences 107(31), 13626 (2010),  10.1073/pnas.1001651107.
  * [16] S. Torquato,  _Hyperuniform states of matter_ ,  Physics Reports 745, 1 (2018).
  * [17] D. Hexner and D. Levine,  _Noise, diffusion, and hyperuniformity_ ,  Phys. Rev. Lett. 118, 020601 (2017),  10.1103/PhysRevLett.118.020601.
  * [18] L. Galliano, M. E. Cates and L. Berthier,  _Two-dimensional crystals far from equilibrium_ ,  Phys. Rev. Lett. 131, 047101 (2023),  10.1103/PhysRevLett.131.047101.
  * [19] E. Tjhung and L. Berthier,  _Discontinuous fluidization transition in time-correlated assemblies of actively deforming particles_ ,  Phys. Rev. E 96, 050601 (2017),  10.1103/PhysRevE.96.050601.
  * [20] J. Kim and S. Torquato,  _Effect of imperfections on the hyperuniformity of many-body systems_ ,  Phys. Rev. B 97, 054105 (2018),  10.1103/PhysRevB.97.054105.
  * [21] P. Dolai, A. Das, A. Kundu, C. Dasgupta, A. Dhar and K. V. Kumar,  _Universal scaling in active single-file dynamics_ ,  Soft Matter 16(30), 7077 (2020),  10.1039/d0sm00687d.
  * [22] E. Milotti,  _1/f noise: a pedagogical review_ ,  arXiv preprint physics/0204033 (2002).
  * [23] I. Eliazar and J. Klafter,  _A unified and universal explanation for lévy laws and 1/f noises_ ,  Proceedings of the National Academy of Sciences 106(30), 12251 (2009).
  * [24] H. Ikeda,  _Correlated noise and critical dimensions_ ,  arXiv preprint arXiv:2302.13666 (2023).
  * [25] D. Chen, Y. Zheng, L. Liu, G. Zhang, M. Chen, Y. Jiao and H. Zhuang,  _Stone–wales defects preserve hyperuniformity in amorphous two-dimensional networks_ ,  Proceedings of the National Academy of Sciences 118(3), e2016862118 (2021).
  * [26] M. Huang, W. Hu, S. Yang, Q.-X. Liu and H. Zhang,  _Circular swimming motility and disordered hyperuniform state in an algae system_ ,  Proceedings of the National Academy of Sciences 118(18), e2100493118 (2021).
  * [27] B. Zhang and A. Snezhko,  _Hyperuniform active chiral fluids with tunable internal structure_ ,  Phys. Rev. Lett. 128, 218002 (2022),  10.1103/PhysRevLett.128.218002.
  * [28] Y. Kuroda and K. Miyazaki,  _Microscopic theory for hyperuniformity in two-dimensional chiral active fluid_ ,  arXiv preprint arXiv:2305.06298 (2023).
  * [29] B. Liebchen and D. Levis,  _Chiral active matter_ ,  Europhysics Letters 139(6), 67001 (2022).
  * [30] A. Callegari and G. Volpe,  _Numerical Simulations of Active Brownian Particles_ , pp. 211–238,  Springer International Publishing, Cham,  ISBN 978-3-030-23370-9,  10.1007/978-3-030-23370-9_7 (2019).
  * [31] H. Ikeda and Y. Kuroda,  _Does spontaneous symmetry breaking occur in periodically driven low-dimensional non-equilibrium classical systems?_ ,  arXiv preprint arXiv:2304.14235 (2023).
  * [32] Y. Imry and S.-k. Ma,  _Random-field instability of the ordered state of continuous symmetry_ ,  Phys. Rev. Lett. 35, 1399 (1975),  10.1103/PhysRevLett.35.1399.
  * [33] Q.-L. Lei and R. Ni,  _Hydrodynamics of random-organizing hyperuniform fluids_ ,  Proceedings of the National Academy of Sciences 116(46), 22983 (2019).
  * [34] S. Ro, Y. Kafri, M. Kardar and J. Tailleur,  _Disorder-induced long-ranged correlations in scalar active matter_ ,  Phys. Rev. Lett. 126, 048003 (2021),  10.1103/PhysRevLett.126.048003.
  * [35] S. M. Zehnder, M. Suaris, M. M. Bellaire and T. E. Angelini,  _Cell volume fluctuations in MDCK monolayers_ ,  Biophysical Journal 108(2), 247 (2015),  10.1016/j.bpj.2014.11.1856.
  * [36] C. S. O’Hern, L. E. Silbert, A. J. Liu and S. R. Nagel,  _Jamming at zero temperature and zero applied stress: The epitome of disorder_ ,  Phys. Rev. E 68, 011306 (2003),  10.1103/PhysRevE.68.011306.
  * [37] H. Goldstein, C. Poole and J. Safko,  _Classical mechanics_ (2002).
  * [38] E. Tjhung and T. Kawasaki,  _Excitation of vibrational soft modes in disordered systems using active oscillation_ ,  Soft Matter 13(1), 111 (2017),  10.1039/c6sm00788k.
  * [39] J.-P. Hansen and I. R. McDonald,  _Theory of simple liquids: with applications to soft matter_ ,  Academic press (2013).


# Bridging Language and Items for Retrieval and Recommendation

Yupeng Hou1$\dagger$ Jiacheng Li1$\dagger$ Zhankui He1 An Yan1 Xiusi Chen2
Julian McAuley1
UC San Diego1 UC Los Angeles2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper introduces BLaIR, a series of pretrained sentence embedding models
specialized for recommendation scenarios. BLaIR is trained to learn
correlations between item metadata and potential natural language context,
which is useful for retrieving and recommending items. To pretrain BLaIR, we
collect Amazon Reviews 2023, a new dataset comprising over 570 million reviews
and 48 million items from 33 categories, significantly expanding beyond the
scope of previous versions. We evaluate the generalization ability of BLaIR
across multiple domains and tasks, including a new task named complex product
search, referring to retrieving relevant items given long, complex natural
language contexts. Leveraging large language models like ChatGPT, we
correspondingly construct a semi-synthetic evaluation set, Amazon-C4.
Empirical results on the new task, as well as conventional retrieval and
recommendation tasks, demonstrate that BLaIR exhibit strong text and item
representation capacity. Our datasets, code, and checkpoints are available at:
https://github.com/hyp1231/AmazonReviews2023.

$\dagger$$\dagger$footnotetext: Equal contribution.

##  1 Introduction

Language is an important modality for describing and showcasing products in
e-commerce platforms, and it plays a critical role in tasks such as
item/product retrieval Ai et al. (2017); Bi et al. (2020); Yan et al. (2022)
and recommendation Hou et al. (2022, 2023); Li et al. (2023). Early methods
developed for recommendation scenarios do not capture the rich semantics of
natural language, as they heavily rely on keyword-based discrete features
Zhang et al. (2019); Zhou et al. (2020a, b). With large language models (LLMs)
showing emerging abilities to capture rich semantics OpenAI (2023); Hoffmann
et al. (2022); Rae et al. (2021); Anil et al. (2023); Chowdhery et al. (2023);
Touvron et al. (2023a, b); Zhao et al. (2023), there is a growing interest in
leveraging LLMs to deal with more language-heavy recommendation tasks. These
tasks present higher demands for the model’s abilities to model language and
items jointly. Examples include conversational recommender systems (CRS) He et
al. (2023) and recommendations based on complex instructions He et al. (2022);
Li et al. (2023); Zhang et al. (2023b); Ren et al. (2023); Bao et al. (2023);
Hou et al. (2024).

Despite their promising abilities, integrating practical scales (_e.g.,_
millions) of items into existing LLMs remains challenging. A practical idea is
to develop specialized lightweight models to connect natural language with
items. In this way, extensive items can be integrated with LLMs in training-
free paradigms like retrieval-augmented generation. Considering existing
methods, end-to-end neural models that are trained on task-specific data fail
to generalize across various tasks and domains (_e.g.,_ predefined item
categories) Bi et al. (2020); Zhou et al. (2020b); He et al. (2022). Another
line is to encode natural language and items using pretrained language models
(PLMs) to generate text embeddings in the same embedding space Hou et al.
(2022, 2023); Yuan et al. (2023). Although these PLMs can achieve acceptable
results under certain circumstances, they are not specialized for
recommendation scenarios, leading to suboptimal performance and generalization
issues.

In this work, we aim to pretrain language models on large-scale corpora that
incorporate collaborative information, with the goal of enabling these models
to adapt to a variety of recommendation domains and tasks. Training such
models, however, presents non-trivial challenges: (1) Objective Design: We
need to formulate training objectives that bridge items and potential natural
language inputs, and the objectives should allow for the easy collection of
large-scale data for pretraining. (2) Data Collection: To ensure practicality,
it is better to use more up-to-date training corpora to match the knowledge
cutoff date of existing LLMs, _e.g.,_ April 2023 of GPT-4
Turbo111https://openai.com/blog/new-models-and-developer-products-announced-
at-devday. (3) Data Curation: It is necessary to carefully curate the
collected datasets to prevent data contamination Oren et al. (2023); Zhou et
al. (2023) and ensure fair evaluation.

To overcome the above challenges, we introduce BLaIR (Bridging Language and
Items for Retrieval and Recommendation). Specifically, we first collect a new
dataset Amazon Reviews 2023. The dataset includes user reviews and item
metadata that are up-to-date (as of September 2023) and is more than twice the
size of previous Amazon reviews datasets. Based on the newly collected
dataset, we continually pretrain a series of sentence embedding models using a
contrastive objective that pairs user reviews with their corresponding item
metadata. Such an objective enables the series of models to effectively link
items with natural language contexts that are potentially mentioned in user
reviews. To prevent data contamination, we fix a timestamp to divide the data
into training and evaluation sets across all domains, and pretrain BLaIR only
using the corpora in the training split.

To thoroughly assess BLaIR, we conduct experiments on three tasks, including a
new task named complex product search, which retrieves relevant items given
complex natural language contexts. Leveraging the power of LLMs, we also
construct a semi-synthetic evaluation set, Amazon-C4, for the newly introduced
task. The evaluation encompasses multiple tasks and domains, allowing us to
validate the generalizability of BLaIR.

Our main contributions are as follows:

  * •

We collect an extensive and up-to-date dataset named Amazon Reviews 2023. The
dataset comprises over 570 million reviews and 48 million items from 33
categories, providing essential resources for future research.

  * •

We present BLaIR, a series of sentence embedding models that are pretrained
over user reviews and item metadata pairs with a contrastive objective. The
derived text representations can be applied to different language-item
correlated tasks across different domains.

  * •

We introduce a new task, complex product search, which retrieves items given
complex natural language contexts. We construct a semi-synthetic evaluation
set using the power of LLMs.

  * •

We define a comprehensive evaluation benchmark based on Amazon Reviews 2023
and evaluate the proposed models. We find that BLaIR improves over existing
methods across multiple domains and tasks, highlighting the strong
generalization of BLaIR.

Figure 1: The overview of BLaIR.

##  2 BLaIR

We present BLaIR, a series of pretrained models that bridge natural language
and items for retrieval and recommendation. We start with introducing the
Amazon Reviews 2023 dataset that has been newly collected and curated (Section
2.1), then we outline the architecture of BLaIR (Section 2.2) and describe in
detail how BLaIR are trained using the new dataset (Section 2.3).

Table 1: Statistics of different versions of Amazon Reviews datasets.

Version | Amazon’13 | Amazon’14 | Amazon’18 | Amazon’23
---|---|---|---|---
#Categories | 28 | 24 | 29 | 33
Customer Reviews
#Reviews | 34,686,771 | 82,456,877 | 233,055,327 | 571,544,897
#Users | 6,643,669 | 21,128,805 | 43,531,850 | 54,514,264
#Items | 2,441,053 | 9,857,241 | 15,167,257 | 48,185,153
#Tokens | 5.9B | 9.1B | 15.7B | 30.1B
Min Time | Jun 1995 | Jun 1996 | Jun 1996 | Jun 1996
Max Time | Mar 2013 | July 2014 | Oct 2018 | Sep 2023
Product Metadata
#Tokens | - | 4.1B | 7.9B | 30.7B
# Meta | - | 9,430,088 | 14,741,571 | 35,393,189

Table 2: Statistics of representative large-scale recommendation datasets.

Dataset | #Items | #Users | #Interactions
---|---|---|---
Netflix Bennett et al. (2007) | 17,770 | 480,189 | 100,480,507
Tmall | 2,353,207 | 963,923 | 44,528,127
Amazon’18 Ni et al. (2019) | 15,167,257 | 43,531,850 | 233,055,327
Google Local Yan et al. (2023) | 4,963,111 | 113,643,107 | 666,324,103
Tenrec Yuan et al. (2022) | 3,753,436 | 5,022,750 | 142,321,193
MicroLens Ni et al. (2023) | 1,142,528 | 34,492,051 | 1,006,528,709
Amazon Reviews 2023 | 48,185,153 | 54,514,264 | 571,544,897

###  2.1 Amazon Reviews 2023

To ensure that the training corpora are up to date to match the knowledge
cutoff date of existing LLMs, we collect the new Amazon Reviews 2023 dataset.
Table 1 shows the comparison of statistics between Amazon Reviews 2023 and
previous Amazon Reviews datasets. In Table 2, we also present statistics for
our Amazon Reviews 2023 and other representative large-scale datasets. The
advantages of Amazon Reviews 2023 can be summarized as follows:

  * •

Larger Size: Amazon Reviews 2023 is notably more extensive than its
predecessors in every dimension, encompassing reviews, users, items, and
metadata. In particular, the new dataset features $3.18$ times the number of
items and $2.4$ times the number of reviews and item metadata compared to
Amazon Reviews 2018.

  * •

Newer Interactions: Amazon Reviews 2023 contains more recent reviews from
Amazon, extending the previous version with new data ranging from Oct 2018 to
Sep 2023.

  * •

Richer and Cleaner Metadata: We parse the original HTML pages into JSON format
to obtain the product metadata. Our metadata contains more descriptive fields
(_e.g.,_ Description and Features of items) and multi-modal fields (_e.g.,_
Videos and Images with different resolutions) than previous versions.

  * •

Finer-Grained Timestamps: The timestamps in prior Amazon Reviews datasets were
accurate only to the day, which could lead to certain inaccuracies for time-
sensitive tasks, including sequential recommendation. The 2023 version of the
dataset offers timestamps with up to millisecond precision, affording much
finer temporal resolution for a variety of applications.

###  2.2 Architecture

BLaIR is designed as a series of sentence embedding models that bridge natural
language contexts and items. Intuitively, we can use the architectures of any
off-the-shelf pretrained Transformer-based sentence embedding models,
including encoder-only models like BERT Devlin et al. (2018) and RoBERTa Liu
et al. (2019) (_e.g.,_ SimCSE Gao et al. (2021)), and encoder-decoder models
like T5 Raffel et al. (2020) (_e.g.,_ InstructOR Su et al. (2023)).

Concretely, we instantiate the backbone model of BLaIR with RoBERTa Liu et al.
(2019). Given one sentence $s$ as input, BLaIR encodes it into sentence
embeddings:

$\displaystyle\bm{s}$
$\displaystyle=\operatorname{\textsc{BLaIR}}([\texttt{[CLS]};s]),$ (1)

where “$[;]$” denotes the concatenation operation and [CLS] is a special token
concatenated at the beginning of the input sentence $s$.
$\bm{s}\in\mathbb{R}^{d}$ is the normalized hidden state that corresponds to
[cls] in the last layer of BLaIR and $d$ is the dimension of the sentence
embedding vectors.

###  2.3 Training Objective

To jointly model language and items, we consider directly optimizing the model
on pairs of natural language context $c$ and item metadata $m$. The metadata
$m$ is a single sentence that contains a concise description of an item. The
language context $c$ is the text associated with the corresponding item. One
instance of $c$ and $m$ in the collected Amazon Reviews 2023 dataset is the
pair of user reviews (language context) and item features (item metadata).
User reviews are large-scale, naturally diverse, and easy to collect. Reviews
have rich correlations with items, including _fine-grained_ correlations like
descriptions and user experiences, and _coarse-grained_ correlations like
backgrounds and motivations.

We optimize BLaIR by aligning the sentence embeddings of context $c$ and item
metadata $m$ in the embedding space to bridge language and items. To be more
specific, BLaIR is trained primarily using a supervised contrastive loss. We
use in-batch instances as negative samples. For a batch of $B$ pairs of
sentences, we first encode them into sentence embeddings using Equation 1 to
obtain $\\{(\bm{c}_{1},\bm{m}_{1}),\ldots,(\bm{c}_{B},\bm{m}_{B})\\}$. The
contrastive loss can be expressed as follows:

$\mathcal{L}_{\mathit{CL}}=-\sum_{i}^{B}\log\frac{\exp(\bm{c}_{i}\cdot\bm{m}_{i}/\tau)}{\sum_{j}^{B}\exp(\bm{c}_{i}\cdot\bm{m}_{j}/\tau)},$
(2)

where $\tau$ is the temperature hyperparameter. To stabilize model training,
we also use the original pretraining objective of the backbone model as an
auxiliary loss function $\mathcal{L}_{\mathit{PT}}$, _e.g.,_ masked language
modeling (MLM) if the backbone model is RoBERTa or BERT and denoising auto-
encoding if the backbone model is T5. To this end, the overall training
objective $\mathcal{L}$ is expressed as follows:

$\mathcal{L}=\mathcal{L}_{\mathit{CL}}+\lambda\mathcal{L}_{\mathit{PT}},$ (3)

where $\lambda$ is a hyperparameter to balance the above two training
objectives.

##  3 Experiments

In this section, we first present the pretraining details (Section 3.1). In
what follows, we evaluate the performance of BLaIR on three tasks across
multiple domains. The tasks include sequential recommendation (Section 3.2),
conventional product search (Section 3.3), and a newly introduced task named
complex product search (Section 3.3). Then we make some further analyses on
the generalizability and training dynamics of BLaIR (Section 3.4).

###  3.1 Training Setups

We first introduce the steps to process the raw Amazon Reviews 2023 dataset
for pretraining. Then we describe the four released checkpoints and their
corresponding implementation details.

Data processing. We process the raw Amazon Reviews 2023 dataset for training
BLaIR.

  * •

_Data split._ Recommender systems in the real world only access interactions
that occurred before a specific timestamp, and aim to predict future
interactions. To better align with such scenarios, we split the reviews into
training, validation, and test sets by _absolute timestamps_ , rather than
predicting the latest few interacted items of each user. To be specific, we
find two timestamps and split all the reviews in a ratio of $8:1:1$. These two
timestamps are used to split data for both pretraining and all downstream
evaluation tasks.

  * •

_Data format._ Each training instance is a pair of two sentences. The first
sentence is the _language context_ , concatenating the title and content of
user reviews. The second is _item metadata_ , which is a concatenation of the
title, features, and description of one item.

  * •

_Data filtering._ For the sake of data quality, we filter out training
instances with context or item metadata less than 30 characters. Limited by
the computing budget, we downsample 10% of the total training instances to
train our first release of BLaIR. The downsampled training set consists of
$3.08\times 10^{7}$ training instances.

Table 3: Statistics of the processed datasets in sequential recommendation task. “Avg. L” denotes the average length of interaction sequences. “Avg. $|t_{i}|$” denotes the average number of characters in the item metadata. Dataset | #Items | #Inters | Avg. L | Avg. $|t_{i}|$
---|---|---|---|---
Beauty | 43,982 | 105,310 | 2.98 | 205.80
Games | 115,815 | 2,550,503 | 4.54 | 661.59
Baby | 179,133 | 3,599,935 | 4.32 | 518.74

Model | Beauty | Games | Baby
---|---|---|---
R@10 | N@10 | R@50 | N@50 | R@10 | N@10 | R@50 | N@50 | R@10 | N@10 | R@50 | N@50
ID-based methods
GRU4Rec | 0.35 | 0.22 | 1.18 | 0.39 | 2.19 | 1.19 | 5.21 | 1.84 | 1.34 | 0.67 | 4.02 | 1.24
SASRec | 0.77 | 0.65 | 1.26 | 0.76 | 2.23 | 1.24 | 4.92 | 1.83 | 1.53 | 0.83 | 3.87 | 1.34
Text-based methods
SASRec (Text) |  |  |  |  |  |  |  |  |  |  |  |
RoBERTabase (123M) | 0.62 | 0.39 | 1.76 | 0.64 | 1.67 | 0.92 | 4.09 | 1.45 | 0.96 | 0.46 | 3.23 | 0.94
UniSRec (Text) |  |  |  |  |  |  |  |  |  |  |  |
RoBERTabase (123M) | 2.42 | 1.58 | 4.10 | 1.94 | 2.22 | 1.22 | 5.47 | 1.92 | 1.22 | 0.62 | 3.63 | 1.14
BLaIR-MLMbase (123M) | 2.69 | 1.77 | 4.85 | 2.24 | 2.49 | 1.40 | 5.90 | 2.14 | 1.43 | 0.75 | 3.99 | 1.30
BLaIRbase (123M) | 3.52 | 2.22 | 5.41 | 2.63 | 2.72 | 1.46 | 6.43 | 2.27 | 1.66 | 0.87 | 4.59 | 1.50
SASRec (Text) |  |  |  |  |  |  |  |  |  |  |  |
RoBERTalarge (354M) | 0.62 | 0.31 | 1.80 | 0.57 | 1.56 | 0.86 | 3.88 | 1.37 | 1.03 | 0.51 | 3.51 | 1.04
UniSRec (Text) |  |  |  |  |  |  |  |  |  |  |  |
RoBERTalarge (354M) | 2.15 | 1.36 | 3.65 | 1.68 | 1.95 | 1.07 | 5.17 | 1.77 | 1.37 | 0.69 | 4.04 | 1.26
SimCSElarge (354M) | 3.90 | 2.48 | 5.08 | 2.74 | 2.73 | 1.48 | 6.40 | 2.27 | 1.61 | 0.82 | 4.55 | 1.45
BLaIR-MLMlarge (354M) | 3.27 | 2.05 | 5.04 | 2.43 | 2.51 | 1.39 | 6.18 | 2.18 | 1.65 | 0.86 | 4.48 | 1.47
BLaIRlarge (354M) | 3.09 | 2.03 | 5.24 | 2.50 | 2.82 | 1.52 | 6.54 | 2.32 | 1.53 | 0.83 | 3.87 | 1.34
_Improvement_ | – | – | +3.2% | – | +3.3% | +2.7% | +2.2% | +2.2% | +3.1% | +6.1% | +0.9% | +3.5%

Table 4: Performance comparison of different methods on the sequential
recommendation task. The best performance score is denoted in bold. The
numbers reported are Recall@10 and NDCG@10.

Implementation details. In our first release, we train four variants of the
BLaIR model, namely BLaIRbase, BLaIRlarge, BLaIR-MLMbase, and BLaIR-MLMlarge.
All four checkpoints follow RoBERTa’s architecture Liu et al. (2019). The
checkpoints BLaIRbase and BLaIRlarge are trained with the objective described
in Equation 3, while BLaIR-MLMbase and BLaIR-MLMlarge are trained with only
MLM loss. The two checkpoints that are trained using only MLM loss are used
mainly as ablations of BLaIRbase and BLaIRlarge.

We use the RoBERTa tokenizer and truncate sentences with a maximum of 64
tokens. BLaIRbase is initialized with RoBERTabase (123M). We train BLaIRbase
on two NVIDIA A100 (80G) GPUs with a per-device batch size of 384 for one
epoch, taking about 24 hours. BLaIRlarge is initialized with RoBERTalarge
(354M) and trained on three NVIDIA A100 (80G) GPUs with a per-device batch
size of 192 for one epoch, taking around two days. We set the hyperparameters
$\tau=0.05,\lambda=0.1$ and optimize the models with a learning rate of
$5\times 10^{-5}$.

###  3.2 Sequential Recommendation

To evaluate BLaIR’s ability to model a single modality (item), we use BLaIR as
the item feature encoder. The encoded representations (embeddings) are later
fed into the sequential recommendation models as input during the training
process.

Task and dataset. We consider the sequential recommendation task. Given the
historical interaction sequence of one user $\\{i_{1},\dots,i_{l}\\}$, the
task is to predict the next item of interest $i_{l+1}$, where $l$ is the
length of the interaction sequence for the user. The items in the interaction
sequence have been ordered chronologically. Here, each item $i$ is associated
with a sentence that represents the metadata of the corresponding item. Note
that in practice, the items in the interaction sequences and the items to
predict are usually from the same domain defined in the Amazon Reviews
datasets Kang and McAuley (2018); Hou et al. (2022).

For our experiments, we include three domains, _i.e.,_ All Beauty (Beauty),
Video Games (Games), and Office Products (Office). The most recent $50$
interactions of each user are used as the historical interaction sequence. The
training, validation and test sets are split using the same timestamps as
described in Section 3.1. We evaluate the models on the test set using the
model that achieves the best ranking performance (NDCG@10) on the validation
set. Existing work usually applies a filter on user-item interactions to
extract a dense subset for both training and testing (_e.g.,_ 5-core filtering
Hou et al. (2022)). However, performing this practice may introduce a
distribution shift between the raw data and the extracted subset, _e.g.,_ only
popular items remain. Recognizing this problem, we use all available
interactions in the raw data for a more realistic evaluation. The statistics
of the processed datasets are summarized in Table 3.

Baselines. For model architecture, we mainly consider two categories: (1) ID-
based methods that assign a learnable embedding vector to each unique item ID
as the corresponding item representation. These methods include GRU4Rec Hidasi
et al. (2016) and SASRec Kang and McAuley (2018); (2) Text-based methods that
encode item metadata into text representations as item representations.
Representative methods include SASRec (Text) Hou et al. (2022) and UniSRec Hou
et al. (2022). For text-based methods, _e.g.,_ UniSRec, we can also fix the
model architecture and evaluate how different text encoders contribute to the
final performance. We select RoBERTa Liu et al. (2019) and supervised SimCSE
Gao et al. (2021) as the PLM baselines. In our experiments, we only use the
architecture of UniSRec instead of initializing the behavior encoder with
pretrained checkpoints. We implement the recommendation models using an open-
source recommendation library RecBole Zhao et al. (2021).

Results. Table 4 illustrates the performance comparison of different methods
in the sequential recommendation task. From Table 4, we have the following
observations:

  * •

BLaIR can serve as item metadata encoders and derive strong text-based item
representations for recommendations. We can see that UniSRec together with
BLaIR achieves the best performance on 9 or 12 metrics, demonstrating the
effectiveness of BLaIR.

  * •

Text-based methods generally achieve better performance than ID-based methods
on all evaluated datasets. We attribute this to well-trained PLMs that
specialize in recommendation scenarios. This observation further demonstrates
BLaIR’s strong capacity to represent items. Note that on the two large
datasets Games and Baby, which have millions of interactions, text-based
methods with BLaIR still outperform ID-based methods like SASRec.

  * •

There are scaling effects for both text representation models and
recommendation models. When scaling the number of parameters of text
representation models from 123M to 354M, the recommendation performance
generally improves. When scaling the number of parameters in the adaptor
modules via mixture-of-experts techniques from SASRec to UniSRec, the
performance also improves.

Table 5: An example of user queries and the corresponding item metadata in ESCI and Amazon-C4 datasets. Item ASIN/ID | B07RJZNY5C
---|---
Query (ESCI) | salt gun
Query (Amazon-C4) |  I want a gun that I can use while garden- ing to get rid of stink bugs, ants, flies, and spiders in my house. It needs to be amazing and help me feel less scared.
Metadata | BUG-A-SALT 3.0 Black Fly Edition.
Table 6: Statistics of the datasets of product search tasks after
preprocessing. “Avg. $|q|$” denotes the average number of characters in the
queries. “Avg. $|t_{i}|$” denotes the average number of characters in the item
metadata.

Dataset | #Queries | #Items | Avg. $|q|$ | Avg. $|t_{i}|$
---|---|---|---|---
ESCI | 27,643 | 1,367,729 | 22.46 | 544.28
Amazon-C4 | 21,223 | 1,058,417 | 229.89 | 538.97

Model | ESCI | Amazon-C4
---|---|---
All | Games | Baby | Office | Sports | All | Games | Baby | Office | Sports
Sparse retrieval method
BM25 | 1.10 | 0.60 | 0.76 | 1.35 | 1.57 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Dense retrieval methods
RoBERTabase (123M) | 0.08 | 0.09 | 0.00 | 0.03 | 0.25 | 0.25 | 0.44 | 0.07 | 0.33 | 0.37
SimCSEbase (123M) | 5.77 | 2.54 | 4.53 | 7.14 | 6.60 | 3.30 | 6.81 | 6.07 | 3.66 | 4.76
BLaIR-MLMbase (123M) | 0.54 | 0.49 | 0.45 | 0.85 | 0.50 | 0.51 | 0.71 | 0.33 | 0.45 | 0.72
BLaIRbase (123M) | 11.76 | 8.62 | 12.38 | 12.56 | 14.26 | 14.46 | 20.32 | 20.57 | 17.84 | 21.07
RoBERTalarge (354M) | 0.04 | 0.00 | 0.00 | 0.08 | 0.00 | 0.06 | 0.05 | 0.00 | 0.00 | 0.08
SimCSElarge (354M) | 8.69 | 7.41 | 9.17 | 9.53 | 9.65 | 5.05 | 9.49 | 7.61 | 5.84 | 9.01
BLaIR-MLMlarge (354M) | 0.45 | 0.33 | 0.22 | 0.29 | 0.17 | 0.30 | 0.29 | 0.00 | 0.16 | 0.47
BLaIRlarge (354M) | 12.12 | 11.53 | 12.18 | 12.81 | 13.90 | 17.18 | 25.67 | 23.39 | 20.07 | 25.68
Table 7: Performance comparison of different methods on conventional product search (ESCI) and complex product search (Amazon-C4) tasks. We report the NDCG@100 metric. “All” averages performance scores over all 33 domains. The best performance score is denoted in bold. Variant | Games - UniSRec | ESCI | Amazon-C4
---|---|---|---
R@10 | N@10 | R@50 | N@50 | All | Games | All | Games
RoBERTabase (123M) | 2.22 | 1.22 | 5.47 | 1.92 | 0.08 | 0.09 | 0.25 | 0.44
BLaIR-MLMbase (123M) |  |  |  |  |  |  |  |
\+ Games | 2.29 | 1.27 | 5.70 | 2.01 | 0.08 | 0.09 | 0.36 | 1.09
\+ All domains | 2.49 | 1.40 | 5.90 | 2.14 | 0.54 | 0.49 | 0.51 | 0.71
Table 8: Performance comparison w.r.t. pretraining datasets. “+ Games” denotes that the model is trained on the in-domain dataset (Games), while “+ All domains” denotes that the model is trained on all the domains of Amazon Reviews 2023. Variant | Games - UniSRec | ESCI | Amazon-C4
---|---|---|---
R@10 | N@10 | R@50 | N@50 | All | Games | All | Games
RoBERTabase (123M) | 2.22 | 1.22 | 5.47 | 1.92 | 0.08 | 0.09 | 0.25 | 0.44
BLaIRbase (123M) |  |  |  |  |  |  |  |
Init. from RoBERTa | 2.72 | 1.46 | 6.43 | 2.27 | 11.76 | 8.62 | 14.46 | 20.32
Init. from BLaIR-MLM | 2.63 | 1.44 | 6.28 | 2.23 | 8.20 | 6.69 | 7.35 | 14.50
Table 9: Performance comparison w.r.t. data curriculum strategies. We mainly
compare two BLaIR variants that are initialized from RoBERTa and BLaIR-MLM,
respectively.

###  3.3 Product Search

To evaluate BLaIR’s capacity of jointly modeling multiple modalities
(language-item), we consider two more product search tasks. The first is
conventional product search, _i.e.,_ retrieve relevant items based on keyword-
based queries. The second is a new task named complex product search.

Conventional product search. Given the user query $q$, which is usually a
short phrase or a combination of keywords, the task is to predict the next
item of interest. We use a conventional product search dataset named ESCI
Reddy et al. (2022), which contains real user queries on the Amazon platform.
ESCI contains multi-granularity labels, and only those <query, item> pairs
with the label Exact are used. There are two versions namely ESCI-small and
ESCI-large. We use the small version that has queries of better quality. We
filter out queries that are not in English and only keep queries with
timestamps that fall within the time range of our predefined test set. For
each ground-truth item, we link the item ID (ASIN) with their metadata in
Amazon Reviews 2023.

We sample a large pool of candidate items that contains items from all domains
of Amazon Reviews 2023. For each <query, item> pair, we randomly sample $50$
in-domain items (as of the ground-truth item) into the final candidate item
pool. The ranking scores are calculated on all these multi-domain candidate
items for each user query.

Complex Product Search. Benefiting from the strong natural-language
understanding and generating abilities of LLMs, we create a new product search
dataset, the Amazon-C4 (Complex Contexts Created by ChatGPT). This dataset is
designed to assess a model’s ability to comprehend complex language contexts
and retrieve relevant items. In conventional product search, users may input
short, straightforward keywords to retrieve desired items. In the new product
search task with complex contexts, the input is longer and more detailed, but
not always directly relevant to the item metadata. Examples of such input
include multi-round dialogues and complex user instructions.

The Amazon-C4 dataset is created by prompting ChatGPT OpenAI (2022) to
generate complex contexts as queries. During data construction, 5-star-rated
user reviews on items are treated as satisfactory interactions, while reviews
with at least 100 characters are considered valid for conveying sufficient
information to be rewritten as complex contextual queries. Considering the
limited budget for API calls, we uniformly sample a subset (around 22,000) of
user reviews from the test set of Amazon Reviews 2023 that meet the rating and
review length requirements. ChatGPT rephrases user reviews as complex contexts
with a first-person tone, serving as queries in our subsequent experiments.
After filtering out failed queries, we summarize the dataset statistics in
Table 6. An example of queries in Amazon-C4 can be found in Table 5.

Like the ESCI dataset Reddy et al. (2022), we also sample a large candidate
item pool that contains items from all domains of Amazon Reviews 2023. For
each <context, item> pair, we randomly sample $50$ items from the domain of
the ground-truth item into the multi-domain candidate item pool. The ranking
scores are calculated for all these in-domain and cross-domain candidate items
for each user query.

Baselines. We select BM25 Robertson et al. (2009) as a sparse retrieval
baseline method. For dense retrieval Ni et al. (2022); Zhao et al. (2022); Su
et al. (2023); Yao et al. (2023), we mainly compare with several PLMs as
baselines. Specifically, we encode each query and item metadata with the
compared PLMs, including RoBERTa, supervised SimCSE, and BLaIR. Then we
calculate the cosine similarities between query representations and item
representations. We calculate the NDCG metric of the ground-truth item over
the top 100 ranked candidate items.

Results. Table 7 shows the performance comparison of different PLMs on the two
product search tasks. From the table, we observe that:

  * •

The sparse retrieval method BM25 performs well on conventional product search
dataset ESCI. However, on Amazon-C4 dataset where queries are long and
complex, the performance of BM25 sharply drops to around zero. We need models
with stronger natural-language understanding abilities to understand the long
context and find relevant items.

  * •

PLMs that are trained with MLM only (_e.g.,_ RoBERTa and BLaIR-MLM) perform
poorly on retrieval tasks. The reason is that the hidden state that
corresponds to [cls] is not well-trained in the MLM task. The results indicate
that we need sentence representation models for zero-shot retrieval tasks.

  * •

The BLaIR model achieves the best performance on all domains among the
compared methods, especially on the Amazon-C4 dataset. This observation
indicates that the designed language-item contrastive task and large-scale
multi-domain pretraining on the Amazon Reviews 2023 dataset enables BLaIR to
learn correlations between language contexts and items.

###  3.4 Further Experiments

Does multi-domain training perform better than in-domain training? We
investigate whether training on multiple recommendation domains improves the
model performance. The “Games” domain is chosen for analysis. We compare two
variants of BLaIR-MLM: one trained on all domains and the other exclusively on
the Games domain. As evidenced in Table 8, the model trained on all domains
generally performs better. These findings suggest that multi-domain
pretraining improves BLaIR’s generalizability, despite potential differences
in data distribution compared to the target downstream task.

What is the better data curriculum for training BLaIR? We also examine which
arrangement of different data sources, such as context-item pairs and item
metadata, improves model training. We compare two variants: the first uses
RoBERTa for initialization and is trained directly on context-item pairs,
while the second begins with RoBERTa and is initially trained on item metadata
before being trained on context-item pairs (in other words, initialized from
BLaIR-MLM before training on context-item pairs). As shown in Table 9, the
data curriculum “metadata $\rightarrow$ <context, metadata>” negatively
impacts model performance. We hypothesize that training solely on metadata
with an MLM loss could lead to overfitting, thereby degrading the following
training and eventually resulting in suboptimal model performance.

##  4 Related Work

Large-scale datasets for recommendation. The availability of large-scale
datasets has played a pivotal role in the advancements of recommendation
research. In early years, the prominent Netflix Prize Bennett et al. (2007)
dataset spurred innovations in collaborative filtering Koren et al. (2009); Su
and Khoshgoftaar (2009); Liang et al. (2018). Most recently, the investigation
into scaling laws for recommendation models necessitates cleaner and larger-
scale extensive datasets Ardalani et al. (2022); Zhang et al. (2023a).
Currently, large-scale recommendation datasets span across various domains,
including E-Commerce (_e.g.,_ Amazon Reviews 2018 Ni et al. (2019) and Tmall
2016222https://tianchi.aliyun.com/competition/entrance/231532), Books (_e.g.,_
GoodReads Wan and McAuley (2018)), Movies (_e.g.,_ Netflix Prize Bennett et
al. (2007)), Points of Interest (_e.g.,_ Google Local Reviews Yan et al.
(2023)), and Streaming (_e.g.,_ Twitch Rappaz et al. (2021), Tenrec Yuan et
al. (2022), and MicroLens Ni et al. (2023)). In Table 2, we present statistics
for our Amazon Reviews 2023 and other representative datasets, highlighting
the largest sizes of item pools to date.

Foundation model-powered retrieval and recommendation. The rise of pretrained
models has led to the emergence of foundation models tailored for retrieval
and recommendation tasks. These models provide the versatility to address
multiple tasks or domains within a unified architecture. For retrieval tasks,
two-tower models leverage contrastive learning on extensive query-document
pairs. These models embed user queries and candidate documents into a shared
space, enabling similarity calculations for text-based retrieval Reimers and
Gurevych (2019); Gao et al. (2021, ); Ni et al. (2022); Karpukhin et al.
(2020); Su et al. (2023) or cross-modal retrieval Radford et al. (2021); Xu et
al. (2021). The generative retrieval paradigm offers an alternative, aiming to
store document knowledge directly within foundation model parameters for text
retrieval Tay et al. (2022). Unlike the textual or visual nature of retrieval
documents, recommendation tasks face the challenge of diverse item attributes,
and the non-transferability of item indices. Addressing this, models have been
developed to incorporate item textual attributes for recommendation Hou et al.
(2022); Li et al. (2023), demonstrating strong transferability. Others, such
as Hou et al. (2023), propose shared item indices with semantic meanings,
enabling the construction of transferable recommendation systems. Meanwhile,
recognizing the prevalence of large language models (LLMs), research
investigates building unified recommender systems built upon LLMs Bao et al.
(2023); Zhang et al. (2023b). Particularly, for the applications in the
e-commerce field, Zhang et al. (2020) introduced Adaptive Hybrid Masking and
Neighbor Product Reconstruction to learn phrase-level knowledge and product-
level knowledge respectively. Shin et al. (2022) built a vision-language “two-
tower” model for e-commerce specific tasks including category classification,
attribute extraction, product matching, product clustering.

##  5 Conclusion

In this work, we aim to bridge the gap between natural language and items for
language-heavy recommendation tasks. We propose a new task named complex
product search that complements the previously widely studied conventional
product search tasks. We introduce Amazon Reviews 2023, a new multi-modal
multi-domain reviews dataset, featuring what is, to the best of our knowledge,
the largest item pool available in public datasets. We further offer BLaIR, a
series of pretrained language models that jointly capture item metadata and
user reviews for product retrieval and recommendation. To comprehensively
benchmark the performance of previous PLMs and BLaIR, we conduct experiments
and demonstrate that BLaIR achieves overall better performance in a zero-shot
setting than the compared methods. Finally, we release the new dataset and
checkpoints of BLaIR in the hope of encouraging more community efforts in the
direction of language-heavy product retrieval and recommendation.

## References

  * Ai et al. (2017) Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W Bruce Croft. 2017.  Learning a hierarchical embedding model for personalized product search.  In _SIGIR_.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_.
  * Ardalani et al. (2022) Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. 2022.  Understanding scaling laws for recommendation models.  _arXiv preprint arXiv:2208.08489_.
  * Bao et al. (2023) Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023.  Tallrec: An effective and efficient tuning framework to align large language model with recommendation.  In _RecSys_.
  * Bennett et al. (2007) James Bennett, Stan Lanning, et al. 2007.  The netflix prize.  In _Proceedings of KDD cup and workshop_ , volume 2007, page 35. New York.
  * Bi et al. (2020) Keping Bi, Qingyao Ai, and W Bruce Croft. 2020.  A transformer-based embedding model for personalized product search.  In _SIGIR_.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.  Palm: Scaling language modeling with pathways.  _JMLR_ , 24(240):1–113.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * (9) Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan.  Precise zero-shot dense retrieval without relevance labels.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  In _emnlp_.
  * He et al. (2023) Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Majumder, Nathan Kallus, and Julian McAuley. 2023.  Large language models as zero-shot conversational recommenders.  In _CIKM_.
  * He et al. (2022) Zhankui He, Handong Zhao, Zhaowen Wang, Zhe Lin, Ajinkya Kale, and Julian Mcauley. 2022.  Query-aware sequential recommendation.  In _CIKM_.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based recommendations with recurrent neural networks.  In _ICLR_.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_.
  * Hou et al. (2023) Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. 2023.  Learning vector-quantized item representation for transferable sequential recommenders.  In _TheWebConf_.
  * Hou et al. (2022) Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022.  Towards universal sequence representation learning for recommender systems.  In _KDD_.
  * Hou et al. (2024) Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024.  Large language models are zero-shot rankers for recommender systems.  In _ECIR_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation.  In _ICDM_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009.  Matrix factorization techniques for recommender systems.  _Computer_ , 42(8):30–37.
  * Li et al. (2023) Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley. 2023.  Text is all you need: Learning language representations for sequential recommendation.  In _KDD_.
  * Liang et al. (2018) Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018.  Variational autoencoders for collaborative filtering.  In _Proceedings of the 2018 world wide web conference_ , pages 689–698.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_ , pages 188–197.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022.  Large dual encoders are generalizable retrievers.  In _EMNLP_.
  * Ni et al. (2023) Yongxin Ni, Yu Cheng, Xiangyan Liu, Junchen Fu, Youhua Li, Xiangnan He, Yongfeng Zhang, and Fajie Yuan. 2023.  A content-driven micro-video recommendation dataset at scale.  _arXiv preprint arXiv:2309.15379_.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  _OpenAI Blog_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Oren et al. (2023) Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. 2023.  Proving test set contamination in black box language models.  _arXiv preprint arXiv:2310.17623_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rappaz et al. (2021) Jérémie Rappaz, Julian McAuley, and Karl Aberer. 2021.  Recommendation on live-streaming platforms: Dynamic availability and repeat consumption.  In _Proceedings of the 15th ACM Conference on Recommender Systems_ , pages 390–399.
  * Reddy et al. (2022) Chandan K. Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022.  Shopping queries dataset: A large-scale ESCI benchmark for improving product search.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Ren et al. (2023) Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2023.  Representation learning with large language models for recommendation.  _arXiv preprint arXiv:2310.15950_.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Shin et al. (2022) Wonyoung Shin, Jonghun Park, Taekang Woo, Yongwoo Cho, Kwangjin Oh, and Hwanjun Song. 2022.  e-clip: Large-scale vision-language representation learning in e-commerce.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, pages 3484–3494.
  * Su et al. (2023) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2023.  One embedder, any task: Instruction-finetuned text embeddings.  In _ACL_.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ , 2009.
  * Tay et al. (2022) Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. 2022.  Transformer memory as a differentiable search index.  _Advances in Neural Information Processing Systems_ , 35:21831–21843.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wan and McAuley (2018) Mengting Wan and Julian McAuley. 2018.  Item recommendation on monotonic behavior chains.  In _Proceedings of the 12th ACM conference on recommender systems_ , pages 86–94.
  * Xu et al. (2021) Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6787–6800.
  * Yan et al. (2022) An Yan, Chaosheng Dong, Yan Gao, Jinmiao Fu, Tong Zhao, Yi Sun, and Julian McAuley. 2022.  Personalized complementary product recommendation.  In _Companion Proceedings of the Web Conference 2022_ , pages 146–151.
  * Yan et al. (2023) An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. 2023.  Personalized showcases: Generating multi-modal explanations for recommendations.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2251–2255.
  * Yao et al. (2023) Feng Yao, Jingyuan Zhang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Yun Liu, and Weixing Shen. 2023.  Unsupervised legal evidence retrieval via contrastive learning with approximate aggregated positive.  In _AAAI_.
  * Yuan et al. (2022) Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, et al. 2022.  Tenrec: A large-scale multipurpose benchmark dataset for recommender systems.  _Advances in Neural Information Processing Systems_ , 35:11480–11493.
  * Yuan et al. (2023) Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023.  Where to go next for recommender systems? id-vs. modality-based recommender models revisited.  In _SIGIR_.
  * Zhang et al. (2020) Denghui Zhang, Zixuan Yuan, Yanchi Liu, Fuzhen Zhuang, Haifeng Chen, and Hui Xiong. 2020.  E-bert: A phrase and product knowledge enhanced language model for e-commerce.  _arXiv e-prints_ , pages arXiv–2009.
  * Zhang et al. (2023a) Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. 2023a.  Scaling law of large sequential recommendation models.  _arXiv preprint arXiv:2311.11351_.
  * Zhang et al. (2023b) Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2023b.  Recommendation as instruction following: A large language model empowered recommendation approach.  _arXiv preprint arXiv:2305.07001_.
  * Zhang et al. (2019) Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, Xiaofang Zhou, et al. 2019.  Feature-level deeper self-attention network for sequential recommendation.  In _IJCAI_.
  * Zhao et al. (2022) Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2022.  Dense text retrieval based on pretrained language models: A survey.  _arXiv preprint arXiv:2211.14876_.
  * Zhao et al. (2021) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2021.  Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms.  In _CIKM_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zhou et al. (2020a) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020a.  S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization.  In _CIKM_.
  * Zhou et al. (2020b) Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020b.  Improving conversational recommender systems via knowledge graph based semantic fusion.  In _KDD_.
  * Zhou et al. (2023) Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023.  Don’t make your llm an evaluation benchmark cheater.  _arXiv preprint arXiv:2311.01964_.