# Joint Delay and Phase Precoding Under True-Time Delay Constraint for THz
Massive MIMO

Dang Qua Nguyen and Taejoon Kim This work was supported in part by the
National Science Foundation (NSF) under Grants CNS1955561. Department of
Electrical Engineering and Computer Science
The University of Kansas, Lawrence, KS 66045 USA
Email: {quand<EMAIL_ADDRESS>
###### Abstract

A new approach is presented to the problem of compensating the beam squint
effect arising in wideband terahertz (THz) hybrid massive multiple-input
multiple-output (MIMO) systems, based on the joint optimization of the phase
shifter (PS) and true-time delay (TTD) values under per-TTD device time delay
constraints. Unlike the prior approaches, the new approach does not require
the unbounded time delay assumption; the range of time delay values that a TTD
device can produce is strictly limited in our approach. Instead of focusing on
the design of TTD values, we jointly optimize both the TTD and PS values to
effectively cope with the practical time delay constraint. Simulation results
that illustrate the performance benefits of the new method for the beam squint
compensation are presented. Through simulations and analysis, we show that our
approach is a generalization of the prior TTD-based precoding approaches.

###### Index Terms:

Wideband THz massive MIMO, beam squint effect, hybrid precoding, true-time
delay.

##  I Introduction

Communications in the terahertz (THz) band (0.1-10 THz) have recently
attracted significant interests from academia and industry due to the
availability of the tens or hundreds of gigahertz bandwidth[1]. In THz
communications, data rates on the orders of 10 to 100 Gbps are achievable
using the currently available digital modulation techniques. To deal with the
high path losses, power consumption, and inter-symbol interference found in
the THz communication channels, the combination of hybrid massive multiple-
input multiple-output (MIMO) and orthogonal frequency division multiplexing
(OFDM) technologies has been popularly discussed recently.

It is well known that as the number of antennas grows, the system can be
greatly simplified in terms of beamforming and precoding complexity [2]. This
has engendered a significant interest in massive MIMO systems at sub-$6$ GHz
[2, 3, 4] and millimeter-wave (mmWave) frequencies [5, 6, 7]. The underlying
assumption behind several strong results [2, 3, 4, 5, 6, 7] was narrowband.
Unlike the narrowband systems, wideband THz OFDM systems suffer from the array
gain loss across different subcarriers as the number of antennas grows due to
beam squint [8, 9, 10]. Beam squint refers to a phenomenon in which deviation
occurs in the spatial direction of each OFDM subcarrier when wideband OFDM is
used in a very large antenna array system. This causes a sizable array gain
loss, which potentially demotivates the use of OFDM in THz massive MIMO.

###  I-A Related works

To deal with the beam squint effect, beam broadening techniques have been
previously studied in mmWave massive MIMO systems [9, 11]. While they show
efficacy in the mmWave bands, these techniques cannot be directly extended to
THz due to the extremely narrow pencil beam requirements imposed by one or two
orders of magnitude higher carrier frequencies. Traditionally, the beam squint
effect has been independently studied in the radar community (e.g., [12, 13],
and references therein). These traditional works [12, 13] proposed to employ
true-time delay (TTD) lines to relieve the beam squint effect. Recently, the
TTD methods have been proposed for THz hybrid massive MIMO-OFDM systems by
introducing delay-phase subarray architectures [14, 15, 16]. Most of these
prior works have focused on (i) the design of TTD values with fixed phase
shifter (PS) values and (ii) the assumption that the TTD values could increase
linearly with the number of antennas without bounds. However, it is
practically difficult to implement these approaches. First, the unbounded TTD
assumption cannot be realized in practice due to the inevitable limitation of
a TTD device. The range of time delay values that a TTD device can produce is
strictly limited (e.g., $0$-$508$ ps [17]). Second, to cope with the practical
TTD constraint, it is much desirable to jointly optimize both the TTD and PS
values to compensate the beam squint. To the best of our knowledge, the
practical TTD constraint and joint PS and time delay optimization for beam
squint compensation have not been fully addressed in the THz communication
literature.

###  I-B Overview of methodology and contributions

In this paper, we present an approach to the problem of beam squint
compensation in wideband THz hybrid massive MIMO-OFDM systems, based on the
joint optimization of the PS and TTD values subject to per-TTD device time
delay constraints. First, assuming a fully-digital array, we characterize the
optimal fully-digital precoder that completely compensates the beam squint
effect. Then, the problem is formulated as a joint PS and TTD optimization
problem that minimizes the distance between the optimal fully-digital precoder
and the product of PS and TTD precoders. Although the formulated problem is
non-convex and thus difficult to solve directly, we show that by transforming
the problem into the phase domain, the original problem is converted to an
equivalent convex problem, which allows us to find a closed-form of the global
optimal solution. Our analysis reveals the amount of time delay and the number
of transmit antennas required given an acceptable level of beam squint
compensation.

Notation: A bold lower case letter ${\mathbf{x}}$ is a column vector and a
bold upper case letter ${\mathbf{X}}$ is a matrix. ${\mathbf{X}}^{T}$,
${\mathbf{X}}^{H}$, $\|{\mathbf{X}}\|_{F}$, $X(i,j)$, and $|x|$ are,
respectively, the transpose, conjugate transpose, Frobenius norm, $i$th row
and $j$th column entry of ${\mathbf{X}}$, and the modulus of
$x\in{\mathbb{C}}$.
$\mathop{\mathrm{blk}}[{\mathbf{x}}_{1},{\mathbf{x}}_{2},\dots,{\mathbf{x}}_{N}]$
is an $nN\times N$ block diagonal matrix such that its main-diagonal blocks
contain ${\mathbf{x}}_{i}\in{\mathbb{R}}^{n}$, for $i=1,2,\dots,N$ and all
off-diagonal blocks are zero. ${\mathcal{F}}_{m,n}$ denotes the set of all
${\mathbf{X}}\in{\mathbb{C}}^{m\times n}$ such that
$|X(i,j)|=\frac{1}{\sqrt{m}},\forall i,j$. $\mathbf{0}_{n}$, $\mathbf{1}_{n}$,
and ${\mathbf{I}}_{n}$ denote, respectively, the $n\times 1$ all-zero vector,
$n\times 1$ all-one vector, and $n\times n$ identity matrix. Given
${\mathbf{x}}\in{\mathbb{R}}^{n}$, $e^{j{\mathbf{x}}}$ denotes
$[e^{jx_{1}},e^{jx_{2}},\dots,e^{jx_{n}}]^{T}\in{\mathbb{C}}^{n}$.

##  II Channel Model and Motivation

In this section, we describe the channel model and the beam squint effect of
large antenna array systems.

###  II-A Channel Model

We consider the downlink of a THz hybrid massive MIMO-OFDM system where the
base station is equipped with an $N_{t}$-element uniform linear array (ULA)
with element spacing $d$. The ULA is fed by $N_{RF}$ radio frequency (RF)
chains to simultaneously transmit $N_{s}$ data streams to an $N_{r}$-antenna
user. The dimensions of $N_{t}$, $N_{r}$, $N_{s}$, and $N_{RF}$ satisfy
$N_{r}=N_{s}\leq N_{RF}\ll N_{t}$. Herein, $f_{c}$ and $B$ denote the central
(carrier) frequency and bandwidth of the OFDM system, respectively. We let $K$
be the number of subcarriers and an odd number. Then, the $k$th subcarrier
frequency is given by $f_{k}=f_{c}+\frac{B}{K}(k-1-\frac{K-1}{2}).$ The
frequency domain MIMO-OFDM channel at the $k$th subcarrier is represented by

${\mathbf{H}}_{k}=\sqrt{\frac{N_{t}N_{r}}{L}}\sum_{l=1}^{L}\alpha_{l}e^{-j2\pi\tau_{l}f_{k}}{\mathbf{f}}(N_{t},\psi_{k,l}){\mathbf{f}}^{H}(N_{r},\phi_{k,l}),$
(1)

where $L$ denotes the number of spatial paths ($L\leq N_{RF}$), and
$\psi_{k,l}=2d\frac{f_{k}}{\nu}\sin({\Psi_{l}})$ and
$\phi_{k,l}=2d\frac{f_{k}}{\nu}\sin({\varphi_{l}})$ are the spatial directions
of the $k$th subcarrier on the $l$th path at the transmitter and receiver,
respectively, where $\nu=3\times 10^{8}$ m/s is the speed of light, and
$\Psi_{l}\in[-\frac{\pi}{2},\frac{\pi}{2}]$ and
$\varphi_{l}\in[-\frac{\pi}{2},\frac{\pi}{2}]$ are the angel-of-departure
(AoD) and angel-of-arrival (AoA) of the $l$th path, respectively. The
$\alpha_{l}\in{\mathbb{C}}$ and $\tau_{l}\in{\mathbb{R}}$ in (1) represent the
gain and delay of the $l$th path, respectively, and
${\mathbf{f}}(N,\psi)=\frac{1}{\sqrt{N}}[1,e^{-j\pi\psi},\dots,e^{-j\pi(N-1)\psi}]^{T}$
is the array response vector of an $N$-element ULA at the direction $\psi$.

Assuming $d=\frac{\nu}{2f_{c}}$, the spatial directions at the central
frequency $f_{c}$ are simplified to $\psi_{c,l}=\sin(\Psi_{l})$ and
$\phi_{c,l}=\sin(\varphi_{l})$, $\forall l$. Hence, setting
$\zeta_{k}=\frac{f_{k}}{f_{c}}$ leads to $\psi_{k,l}=\zeta_{k}\psi_{c,l}$ and
$\phi_{k,l}=\zeta_{k}\phi_{c,l}$, $\forall k,l$. For ease of exposition, we
assume that $L=N_{RF}$, which is equivalent setting $\alpha_{l}=0$, $\forall
l>L$ in (1) whenever $L<N_{RF}$.

###  II-B Motivation

As aforementioned in _Section_ I, when wideband OFDM is employed in a massive
MIMO system, a substantial array gain loss at each subcarrier could occur due
to beam squint. To describe the beam squint effect, we consider the $l$th path
of the channel in (1). We denote the beam forming vector matched to the array
response vector with the AoD $\Psi_{l}$ as
${\mathbf{f}}_{l}={\mathbf{f}}(N_{t},\psi_{c,l})$. The array gain at the $k$th
subcarrier of the $l$th path is then given by
$g({\mathbf{f}}_{l},\psi_{k,l})=|{\mathbf{f}}^{H}(N_{t},\psi_{k,l}){\mathbf{f}}_{l}|$,
i.e.,

$g({\mathbf{f}}_{l},\psi_{k,l})\\!=\\!\frac{1}{N_{t}}\bigg{|}\\!\sum\limits_{n=0}^{N_{t}-1}\\!\\!e^{jn\pi(\psi_{k,l}-\psi_{c,l})}\bigg{|}\\!=\\!\left|\frac{\sin\left(N_{t}\Delta_{k,l}\right)}{N_{t}\sin\left(\Delta_{k,l}\right)}\\!\right|\\!,$
(2)

where $\Delta_{k,l}=\frac{\pi}{2}(\psi_{k,l}-\psi_{c,l})$. It is not difficult
to observe that at the central frequency, the array gain is
$g({\mathbf{f}}_{l},\psi_{c,l})=1$, because $\lim_{x\rightarrow
0}\frac{\sin(N_{t}x)}{\sin(x)}=N_{t}$. However, when $f_{k}\neq f_{c}$,
$\Delta_{k,l}$ deviates from zero. As a result, any non-central subcarrier
suffers from the array gain loss. The implication of beam squint in the
spatial domain is that the beams at non-central subcarriers may completely
squint.

The beam squint effect becomes severe either when the bandwidth B increases or
when $N_{t}$ grows. The following lemma quantifies the asymptotic array gain
loss as $N_{t}\rightarrow\infty$.

######  _Lemma_ 1.

Suppose that $\psi_{k,l}$ is the spatial direction at the $k$th subcarrier
($f_{k}\neq f_{c}$) of the $l$th path. Then, the array gain in (2) converges
to $0$ as $N_{t}$ tends to infinity, i.e.,
$\lim_{N_{t}\rightarrow\infty}g({\mathbf{f}}_{l},\psi_{k,l})=0.$

###### Proof.

The array gain in (2) can be rewritten as
$g({\mathbf{f}}_{l},\psi_{k,l})=\frac{1}{N_{t}}\left|\frac{\sin(N_{t}\Delta_{k,l})}{\pi\Delta_{k,l}}\frac{\pi\Delta_{k,l}}{\sin(\Delta_{k,l})}\right|,$
where $\Delta_{k,l}\neq 0$ because $f_{k}\neq f_{c}$. Then the lemma follows
from the definition of the Dirac delta function
$\lim_{N_{t}\rightarrow\infty}\frac{\sin(N_{t}\Delta_{k,l})}{\pi\Delta_{k,l}}=\delta(\Delta_{k,l})$
[18], which completes the proof. ∎

Figure 1: Array gain vs. subcarriers for different numbers of transmit
antennas ($N_{t}$).

Fig. 1 illustrates the convergence trend of _Lemma_ 1\. The array gain
patterns are calculated for $f_{c}=300$ GHz, $K=129$, $\psi_{c,l}=0.8$, and
$B=30$ GHz. As $N_{t}$ tends to be large, the maximum array gain is only
achieved at the central frequency, i.e., $65$th subcarrier in Fig.1, while
other subcarriers suffer from nearly 100% array gain losses. This is quite
opposite to the traditional narrowband massive MIMO system in which the array
gain grows as $N_{t}\rightarrow\infty$.

Wideband THz massive MIMO research is in its early stages. In order to truly
unleash the potential of THz communications, a hybrid precoding mechanism that
can effectively compensate for the beam squint effect under practical
constraints is of paramount importance.

##  III Hybrid Precoding Under TTD Contraint

We consider a TTD-based hybrid precoding architecture [14, 15], where each RF
chain drives $M$ TTDs and each TTD is connected to $N=\frac{N_{t}}{M}$ phase
shifters (PSs) as shown in Fig. 2.

Figure 2: TTD-based hybrid precoding architecture

The $N_{t}$-element ULA is divided into $M$ subarrays with $N$ antennas per
subarray ($N_{t}=MN$). A TTD delays the $k$th subcarrier by $t$ ($0\leq t\leq
t_{\max}$), which corresponds to the $-2\pi f_{k}t$ phase rotation. The
$t_{\max}$ is the maximum time delay value that a TTD device can produce. The
received signal ${\mathbf{y}}_{k}\in{\mathbb{C}}^{N_{r}}$ at the $k$th
subcarrier is then expressed by

${\mathbf{y}}_{k}={\mathbf{H}}^{H}_{k}{\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1}){\mathbf{W}}_{k}{\mathbf{s}}_{k}+{\mathbf{n}}_{k},$
(3)

where ${\mathbf{s}}_{k}\in{\mathbb{C}}^{N_{s}}$,
${\mathbf{W}}_{k}\in{\mathbb{C}}^{N_{RF}\times N_{s}}$, and
${\mathbf{n}}_{k}\in{\mathbb{C}}^{N_{r}}$ are, respectively, the transmitted
data stream, baseband digital precoder, and noise. Herein,
${\mathbf{F}}_{1}$=$\frac{1}{\sqrt{N_{t}}}$$[{\mathbf{G}}_{1},{\mathbf{G}}_{2},\dots,{\mathbf{G}}_{N_{RF}}]$$\in{\mathbb{C}}^{N_{t}\times
MN_{RF}}$ is the PS precoding matrix, where
${\mathbf{G}}_{l}=\mathop{\mathrm{blk}}[e^{j\pi{\mathbf{x}}^{(l)}_{1}},e^{j\pi{\mathbf{x}}^{(l)}_{2}},\dots,e^{j\pi{\mathbf{x}}^{(l)}_{M}}]\in{\mathbb{C}}^{N_{t}\times
M}$ is the PS submatrix and
${\mathbf{x}}^{(l)}_{m}=[x^{(l)}_{1,m},x^{(l)}_{2,m},\dots,x^{(l)}_{N,m}]^{T}\in{\mathbb{R}}^{N}$
is the PS vector with the PSs connected to the $m$th TTD on the $l$th RF
chain, $\forall l,m$. The
${\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})\\!=\\!\mathop{\mathrm{blk}}[e^{-j2\pi
f_{k}{\mathbf{t}}_{1}},e^{-j2\pi f_{k}{\mathbf{t}}_{2}},\dots,e^{-j2\pi
f_{k}{\mathbf{t}}_{N_{RF}}}]\in{\mathbb{C}}^{MN_{RF}\times N_{RF}}$ is the
time delay precoding matrix, where
${\mathbf{t}}_{l}=[t^{(l)}_{1},t^{(l)}_{2},\dots,t^{(l)}_{M}]^{T}\in{\mathbb{R}}^{M}$
is the time delay vector on the $l$th RF chain, $\forall l$.

The prior methods [14, 15] fixed the PS precoder ${\mathbf{F}}_{1}$ and
designed the time delay precoder
${\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})$, where
${\mathbf{t}}_{l}$ were determined to increase proportionally to the number of
antennas. As the number of antennas grows by two or three orders of magnitude
in THz massive MIMO, the required time delay values in the prior work may
exceed the operational limit of a TTD device, causing degradation. This
motivates us to impose a practical constraint on the range of the TTD values
and study an approach that jointly optimize PS and TTD precoders.

##  IV Preliminaries

In this section, we describe the sign invariant property of the array gain and
identify the optimal fully-digital precoder that will be incorporated in
_Section_ V.

###  IV-A Sign Invariance of Array Gain

Denoting the $l$th column of
${\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})$ in
(3) as ${\mathbf{g}}^{(l)}_{k}=\frac{1}{\sqrt{N_{t}}}{\mathbf{G}}_{l}e^{-2\pi
f_{k}{\mathbf{t}}_{l}}$, the array gain associated with
${\mathbf{g}}^{(l)}_{k}$ is given, based on (2), by

$g({\mathbf{g}}^{(l)}_{k},\psi_{k,l})=\frac{1}{N_{t}}\bigg{|}\\!\sum\limits_{m=1}^{M}\\!\sum\limits_{n=1}^{N}\\!e^{j\pi\zeta_{k}\gamma^{(l)}_{n,m}}e^{j\pi
x^{(l)}_{n,m}}e^{-j\pi\zeta_{k}\vartheta^{(l)}_{m}}\\!\bigg{|},$ (4)

where $\gamma^{(l)}_{n,m}=((m-1)N+n-1)\psi_{c,l}$ and
$\vartheta^{(l)}_{m}=2f_{c}t^{(l)}_{m}\in[0,\vartheta_{\max}]$ with
$\vartheta_{\max}=2f_{c}t_{\max}$. The $g({\mathbf{g}}^{(l)}_{k},\psi_{k,l})$
in (4) is invariant to the multiplication of negative signs to
$\gamma_{n,m}^{(l)}$, $x_{n,m}^{(l)}$, and $\vartheta^{(l)}_{m}$. To be
specific, given $\psi_{c,l}>0$ (i.e., $\gamma^{(l)}_{n,m}>0$, $\forall m$,
$n$), we denote $\\{x_{m,n}^{(l)\star}\\}$ and
$\\{\vartheta^{(l)\star}_{m}\\}$ as the optimal values that maximize
$g({\mathbf{g}}^{(l)}_{k},\psi_{k,l})$. Then, it is not difficult to observe
that $\\{-x^{(l)\star}_{n,m}\\}$ and
$\\{\vartheta_{\max}-\vartheta^{(l)\star}_{m}\\}$ also maximize
$g({\mathbf{g}}^{(l)}_{k},-\psi_{k,l})$. This is a straightforward property
when we simultaneously change the sign of the phase in every term inside the
double sums in (4). Leveraging this sign invariant property, in what follows,
we assume that $\psi_{c,l}\geq 0$, $\forall l$. This sign invariant property
will be exploited when deriving the solution to our optimization problem in
_Section_ V and _Appendix_ B.

###  IV-B Optimal Fully-Digital Precoder

We identify an optimal fully-digital precoder that maximizes the array gain of
each subcarrier by completely compensating the beam squint effect. The purpose
is to provide a reference design. To this end, we define
${\mathbf{F}}_{k}\in{\mathcal{F}}_{N_{t},{N_{RF}}}$ as a fully-digital
precoder where ${\mathbf{f}}^{(l)}_{k}\in{\mathcal{F}}_{N_{t},1}$ denotes the
$l$th column of ${\mathbf{F}}_{k}$. The array gain obtained by
${\mathbf{f}}^{(l)}_{k}$ is given by
$g({\mathbf{f}}^{(l)}_{k},\psi_{k,l})=\big{|}{\mathbf{f}}^{H}(N_{t},\psi_{k,l}){\mathbf{f}}^{(l)}_{k}\big{|}$.
By the Cauchy-Schwarz inequality,
$g({\mathbf{f}}^{(l)}_{k},\psi_{k,l})\leq\left\|{\mathbf{f}}(N_{t},\psi_{k,l})\right\|_{2}\|{\mathbf{f}}^{(l)}_{k}\|_{2}=1$,
where the equality holds if and only if
${\mathbf{f}}^{(l)}_{k}={\mathbf{f}}(N_{t},\psi_{k,l})$, resulting in
${\mathbf{F}}^{\star}_{k}=[{\mathbf{f}}(N_{t},\psi_{k,1}),{\mathbf{f}}(N_{t},\psi_{k,2}),\dots,{\mathbf{f}}(N_{t},\psi_{k,N_{RF}})]$,
where the $((m-1)N+n-1)$th row and $l$th column entry of
${\mathbf{F}}^{\star}_{k}$ is

$F^{\star}_{k}((m-1)N+n-1,l)=\frac{1}{\sqrt{N_{t}}}e^{-j\pi\zeta_{k}\gamma^{(l)}_{n,m}},\forall
k,l,m,n.$ (5)

It is noteworthy to point out that the entries in (5) is only realizable when
$N_{RF}=N_{t}$, i.e., each antenna is fed by its dedicated RF chain, which is
highly impractical for THz massive MIMO systems. Therefore, we attempt to best
approximate the optimal fully-digital precoder in (5) as a product of
${\mathbf{F}}_{1}$ and
${\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}_{l=1}^{N_{RF}})$.

##  V Joint Phase and Delay Precoding under TTD Constraint

Ideally, we wish to find $\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1}$ and
${\mathbf{F}}_{1}$ satisfying
${\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})={\mathbf{F}}^{\star}_{k}$,
$\forall k$, i.e., $e^{j\pi
x^{(l)}_{n,m}-j\pi\zeta_{k}\vartheta^{(l)}_{m}}=e^{-j\pi\zeta_{k}\gamma^{(l)}_{n,m}}$,
$\forall k,l,m,n$. However, given fixed $l$, $m$, and $n$, solving $K$-coupled
matrix equations is an ill-posed problem, because PSs only generate fixed
phase values (frequency-independent), while TTDs generate frequency-dependent
phase values. To overcome this issue, we approach to formulate a problem that
optimizes ${\mathbf{F}}_{1}$ and $\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1}$ by
minimizing the difference between
${\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})$ and
${\mathbf{F}}^{\star}_{k}$, $\forall k$:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\min_{{\mathbf{F}}_{1},\\{{\mathbf{t}}_{l}\\}_{l=1}^{N_{RF}}}\\!\\!\\!\\!$
$\displaystyle\\!\\!\\!\\!\frac{1}{K}\sum_{k=1}^{K}\left\|{\mathbf{F}}^{\star}_{k}-{\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})\right\|^{2}_{F},$
(6a) subject to $\displaystyle\\!\\!\\!\\!0\leq t^{(l)}_{m}\leq
t_{max},\forall l,m,$ (6e)
$\displaystyle\\!\\!\\!\\!|{\mathbf{F}}_{1}(i,j)|=\frac{1}{\sqrt{N_{t}}},~{}\text{
if }|{\mathbf{F}}_{1}(i,j)|\neq 0,\forall i,j,$
$\displaystyle\\!\\!\\!\\!|{\mathbf{F}}_{2k}(p,q)|=1,~{}\text{ if
}|{\mathbf{F}}_{2k}(p,q)|\neq 0,\forall p,q,$
$\displaystyle\\!\\!\\!\\!{\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})\in{\mathcal{F}}_{N_{t}\times
N_{RF}},\forall k.$

The constraint (6e) indicates the range of time delay values of the per-TTD
device. The constraint (6e) describes the constant modulus property of the
product
${\mathbf{F}}_{1}{\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})$ at
every subcarrier. We note here that while we are focusing on precoding in this
paper, a similar optimization problem can be induced to design receive
combiners to deal with the beam squint effect at the receiver.

The constraints in (6e), (6e), and (6e) and the coupling between
${\mathbf{F}}_{1}$ and
${\mathbf{F}}_{2k}(\\{{\mathbf{t}}_{l}\\}^{N_{RF}}_{l=1})$ in (6a) make the
problem difficult to solve. Besides, (6) can be viewed as a matrix
factorization problem under non-convex constraints, which has been also
studied in wireless communications in the context of hybrid analog-digital
precoding [5, 7, 19, 20, 21]. A common approach was applying block coordinate
descent (BCD) and relaxing the constraints [5, 7, 19, 20, 21] to deal with the
non-convexity. Unlike the prior approaches, we show in this work that the
original non-convex problem in (6) can be readily converted into an equivalent
convex problem.

######  _Lemma_ 2.

For $x_{0}\in{\mathbb{R}}$ and $y\in{\mathbb{R}}$, the following equality
holds
$\mathop{\mathrm{argmin}}\limits_{y\mathrel{\mathop{\ordinarycolon}}\mathop{\mathrm{mod}}(y,\pi)\neq
x_{0}}|e^{jx_{0}}-e^{jy}|$
$=\mathop{\mathrm{argmin}}\limits_{y\mathrel{\mathop{\ordinarycolon}}\mathop{\mathrm{mod}}(y,\pi)\neq
x_{0}}|x_{0}-y|$, where $\mathop{\mathrm{mod}}(y,\pi)$ is $y$ modulo $\pi$.

###### Proof.

See Appendix A. ∎

Rewriting the objective function in (6) gives

$\frac{1}{N_{t}}\frac{1}{K}\sum_{k=1}^{K}\\!\sum_{l=1}^{N_{RF}}\\!\sum_{m=1}^{M}\\!\sum_{n=1}^{N}\bigg{|}e^{-j\pi\zeta_{k}\gamma^{(l)}_{n,m}}-e^{j\pi
x^{(l)}_{n,m}}e^{-j\pi\zeta_{k}\vartheta^{(l)}_{m}}\bigg{|}^{2}\\!\\!\\!,\\!\\!\\!\\!$
(7)

and incorporating _Lemma_ 2 into (7) converts (6) into the following
equivalent problem:

$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\min_{\\{x^{(l)}_{n,m}\\},\\{\vartheta^{(l)}_{m}\\}}\\!\\!\\!\\!$
$\displaystyle\frac{1}{K}\\!\sum\limits_{k=1}^{K}\\!\\!\sum\limits_{l=1}^{N_{RF}}\\!\\!\sum\limits_{m=1}^{M}\\!\sum\limits_{n=1}^{N}\left|x^{(l)}_{n,m}\\!\\!-\\!\zeta_{k}\vartheta^{(l)}_{m}\\!\\!+\\!\zeta_{k}\gamma^{(l)}_{n,m}\right|^{2}\\!\\!\\!\\!,$
(8a) subject to $\displaystyle
0\leq\vartheta^{(l)}_{m}\leq\vartheta_{\max},~{}\forall l,m.$ (8b)

The PS and TTD variables in (8) are collected into a composite matrix
${\mathbf{A}}_{l}=[{\mathbf{a}}^{(l)}_{1},\dots,{\mathbf{a}}^{(l)}_{M}]\in{\mathbb{R}}^{(N+1)\times
M}$, where
${\mathbf{a}}^{(l)}_{m}=[{\mathbf{x}}^{(l)T}_{m},\vartheta^{(l)}_{m}]^{T}\in{\mathbb{R}}^{N+1}$.
Containing the fully-digital counterpart in (5) in a matrix
${\mathbf{B}}^{(l)}_{k}\in{\mathbb{R}}^{N\times M},$ where
$B^{(l)}_{k}(n,m)=-\zeta_{k}\gamma^{(l)}_{n,m}$, $\forall k,l,n,m$, the
problem (8) is succinctly

$\displaystyle\underset{\\{{\mathbf{A}}_{l}\\}^{N_{RF}}_{l=1}}{\min}$
$\displaystyle\frac{1}{K}\sum\limits_{k=1}^{K}\sum\limits_{l=1}^{N_{RF}}\left\|{\mathbf{C}}_{k}{\mathbf{A}}_{l}-{\mathbf{B}}^{(l)}_{k}\right\|^{2}_{F},$
(9a) subject to
$\displaystyle\mathbf{0}^{T}_{M}\preceq{\mathbf{e}}^{T}_{N+1}{\mathbf{A}}_{l}\preceq\vartheta_{\max}\mathbf{1}^{T}_{M},~{}\forall
l,$ (9b)

where
${\mathbf{C}}_{k}=\begin{bmatrix}{\mathbf{I}}_{N},-\zeta_{k}\mathbf{1}_{N}\end{bmatrix}\in{\mathbb{R}}^{N\times(N+1)}$
and ${\mathbf{e}}_{N+1}=[\mathbf{0}^{T}_{N},1]^{T}\in{\mathbb{R}}^{N+1}$. The
notation $\preceq$ denotes an entry-wise vector inequality.

By introducing
${\mathbf{C}}=\frac{1}{K}\sum_{k=1}^{K}{\mathbf{C}}^{T}_{k}{\mathbf{C}}_{k}\in{\mathbb{R}}^{(N+1)\times(N+1)}$
and
${\mathbf{D}}_{l}=\frac{1}{K}\sum_{k=1}^{K}{\mathbf{C}}^{T}_{k}{\mathbf{B}}^{(l)}_{k}\in{\mathbb{R}}^{(N+1)\times
M}$, the problem (9) is equivalently

$\displaystyle\underset{{\mathbf{a}}^{(l)}_{m}}{\min}~{}$
$\displaystyle{\mathbf{a}}^{(l)T}_{m}{\mathbf{C}}{\mathbf{a}}^{(l)}_{m}-2{\mathbf{d}}^{(l)T}_{m}{\mathbf{a}}^{(l)}_{m},$
(10a) subject to $\displaystyle
0\leq{\mathbf{e}}_{N+1}^{T}{\mathbf{a}}^{(l)}_{m}\leq\vartheta_{\max},\forall
l,m,$ (10b)

where ${\mathbf{d}}^{(l)}_{m}$ in (10a) is the $m$th column of
${\mathbf{D}}_{l}$. The problem in (10) is solved globally and the solution to
it is summarized below.

######  _Theorem_ 1.

The optimal solution ${\mathbf{a}}^{(l)\star}_{m}=[{\mathbf{x}}^{(l)\star
T}_{m},\vartheta^{(l)\star}_{m}]^{T}$ to (10) is given by

$x^{(l)\star}_{n,m}=\left\\{\\!\\!\\!\begin{array}[]{ll}\frac{N-2n+1}{2}\psi_{c,l},&\\!\\!\\!\\!\text{
if }0\leq\psi_{c,l}\leq\frac{4f_{c}t_{\max}}{(2m-1)N-1},\\\
\vartheta_{\max}-\gamma^{(l)}_{n,m},&\\!\\!\\!\\!\text{ otherwise},\forall
l,n,m,\end{array}\right.$ (11)

and $\vartheta^{(l)\star}_{m}=2f_{c}t^{(l)\star}_{m}$, where the
$t^{(l)\star}_{m}$ is

$t^{(l)\star}_{m}=\left\\{\\!\\!\\!\begin{array}[]{ll}\frac{(2m-1)N-1}{4f_{c}}\psi_{c,l},\\!\\!\\!\\!&\text{
if }0\leq\psi_{c,l}\leq\frac{4f_{c}t_{\max}}{(2m-1)N-1},\\\
t_{\max},\\!\\!\\!\\!&\text{ otherwise},\forall l,m.\end{array}\right.$ (12)

###### Proof.

See Appendix B. ∎

######  Remark 1.

The PS and TTD solutions in (11) and (12) are different from those in prior
works. For instance, the PS values in [14] were not optimized and given by
$x^{(l)}_{n,m}=-(n-1)\psi_{c,l}$, $\forall n$, which is distinguishable from
(11). The time delay value of the $m$th TTD was
$t^{(l)}_{m}=m\frac{N\psi_{c,l}}{2f_{c}}$ in [14]; as $m$ increases, the
$t^{(l)}_{m}$ could be greater than $t_{\max}$, in which case such
$t^{(l)}_{m}$ needs to be knocked down to the $t_{\max}$, resulting in
performance deterioration. On the other hand, as will be discussed in
_Section_ VI in detail, when all TTD values are smaller than $t_{\max}$, the
approaches in [14, 15] achieve the same array gain performance as the proposed
approach, meaning that the designs in [14, 15] is a special case of _Theorem_
1.

Focusing on the large $t_{\max}$, the TTD value in (12) becomes
$t^{(l)\star}_{m}=\frac{((2m-1)N-1)\psi_{c,l}}{4f_{c}},$ which is proportional
to the delay increase at every subcarrier and thereby, best compensating the
beam squint. On the basis of this observation, we obtain selection criteria
(rule of thumb) on the required number of transmit antennas and value of
maximum time delay based on (12) as follows.

_$N_{t}$ Selection Criterion:_ Given $t_{\max}$ value determined by the
employed TTD devices, choose $N_{t}$ such that

$N_{t}\leq\frac{M}{2m-1}+\frac{4Mf_{c}t_{\max}}{(2m-1)\psi_{c,l}},\forall
l,m.$ (13)

_$t_{\max}$ Selection Criterion:_ Equivalently, given $N_{t}$ value determined
by the transmission array, the TTD device should be chosen to satisfy

$t_{\max}\geq\psi_{c,l}\frac{(2m-1)N_{t}-M}{4Mf_{c}},\forall l,m.$ (14)

These bounds are once again approximations, but they give insight into the
values of $N_{t}$ and $t_{\max}$ required. For example, these bounds depend on
a random variable $\psi_{c,l}$, which can be characterized by the angle spread
statistics $\psi_{c,l}\in[\psi_{\min},\psi_{\max}]$ of the deployed array
antenna systems, leading to sufficient conditions for (13) and (14) by
replacing $\psi_{c,l}$ with $\psi_{\max}$.

##  VI Simulations

In this section, we conduct numerical simulations to evaluate the average
array gain performance of the proposed joint PS and TTD precoding and verify
our analysis in _Section_ V. The average array gain of the $l$th path is
computed as $\frac{1}{K}\sum_{k=1}^{K}g({\mathbf{g}}^{(l)}_{k},\psi_{c,l})$.
For the numerical simulation, we consider $B=30$ GHz, $f_{c}=300$ GHz,
$K=129$, and $\psi_{c,l}=0.8$, and the number of TTDs to be $M=16$.

Figure 3: The average array gain vs. $N_{t}$ for $t_{\max}=340$ ps.

Fig. 3 displays the average array gain curves of the proposed approach and the
prior approach in [14] for the number of antennas
$N_{t}\in\\{32,64,256,512,1024\\}$ and $t_{\max}=340$ ps. When $N_{t}=32$ or
$64$, both methods obtain the same near-optimal array gain performance,
because the designed time delay values in both approaches are smaller than
$340$ ps. However, when $N_{t}$ increases further, e.g., $N_{t}\geq 256$, both
approaches suffer from array gain loss as shown in Fig. 3. Nevertheless, the
proposed approach reveals a rather reliable performance than the benchmark
because some TTD values produced by the prior approach are larger than the
$t_{\max}$ and this increases the array gain loss.

Figure 4: The average array gain vs. $t_{\max}$ for $N_{t}=256$.

In Fig. 4, the average array gain performance of the proposed approach is
compared with the prior approach in [14], in which we increase the $t_{\max}$
values from $200$ ps to $400$ ps while fixing $N_{t}=256$. When
$t_{\max}\in[200,340]$ ps, as seen from the figure, the proposed approach
reveals consistent performance increase while the benchmark [14] shows
inconsistency because of the infeasible TTD values designed. When
$t_{\max}\geq 330$ ps, the proposed approach steadily converges to the average
array gain performance around 94%. Meanwhile, the benchmark achieves the same
performance when $t_{\max}\geq 350$ ps.

Overall, the trends in Figs. 3-4 reaffirm _Remark_ 1 stating that the prior
design [14] is a special case of the proposed approach. Figs. 3-4 also reveal
the intuition behind the selection criteria in (13) and (14), in which
$N_{t}\leq 256$ and $t_{\max}\geq 330$ ps are chosen to meet the criteria,
respectively.

##  VII Conclusion

A TTD-based hybrid precoding approach was proposed to compensate the beam
squint effect in the wideband THz massive MIMO-OFDM system by jointly
optimizing the PS and TTD precoders under the per-TTD device time delay
constraint. The joint optimization problem was initially formulated in the
context of minimizing the distance between the optimal fully-digital precoder
and the product of the PS and TTD precoders. By transforming the original
problem to a tractable phase domain, we identified the global optimal
solution. Based on the closed-form expression of our solution, we proposed the
selection criteria for the required number of transmit antennas and value of
maximum time delay. Through analysis and simulations, we affirmed that our
proposed design (_Theorem_ 1) is a generalization of the prior TTD precoding
approaches.

##  Appendix A Proof of lemma 2

Without loss of generality, we assume that $0<|x_{0}-y|<\pi$. Then,
$\mathop{\mathrm{argmin}}_{0<|x_{0}-y|<\pi}|e^{jx_{0}}-e^{jy}|$ =
$\underset{0<|x_{0}-y|<\pi}{\mathop{\mathrm{argmin}}}\Big{|}\sin\left(\frac{x_{0}-y}{2}\right)\Big{|}$
$=\underset{0<|x_{0}-y|<\pi}{\mathop{\mathrm{argmin}}}\sin\Big{(}\Big{|}\frac{x_{0}-y}{2}\Big{|}\Big{)}$
$=\underset{0<|x_{0}-y|<\pi}{\mathop{\mathrm{argmin}}}|x_{0}-y|$, where the
last step follows from the fact that $\sin(\cdot)$ is a strictly increasing
function in $(0,\frac{\pi}{2})$. This completes the proof.

##  Appendix B Proof of theorem 1

To show _Theorem_ 1, we need to first provide closed-form expressions of
${\mathbf{e}}^{T}_{N+1}\\!{\mathbf{C}}^{-1}\\!{\mathbf{e}}_{N+1}$ and
${\mathbf{e}}_{N+1}^{T}\\!{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}$, which will
be later used in this proof. To this end, we first claim that

$\displaystyle{\mathbf{e}}^{T}_{N+1}{\mathbf{C}}^{-1}{\mathbf{e}}_{N+1}\\!\\!\\!\\!$
$\displaystyle=$ $\displaystyle\\!\\!\\!\\!\frac{1}{\eta},$ (15a)
$\displaystyle{\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}\\!\\!\\!\\!$
$\displaystyle=$ $\displaystyle\\!\\!\\!\\!\frac{(2m-1)N-1}{2}\psi_{c,l},$
(15b)

where $\eta$$=$$\frac{NB^{2}}{f^{2}_{c}}\frac{(K^{2}-1)}{12K^{2}}.$ We first
prove (15a). Given ${\mathbf{C}}_{k}$ in (9), we have
${\mathbf{C}}^{T}_{k}{\mathbf{C}}_{k}\\!=\\!\begin{bmatrix}{\mathbf{I}}_{N}\\!&\\!-\zeta_{k}\mathbf{1}_{N}\\\
-\zeta_{k}\mathbf{1}^{T}_{N}\\!&\\!N\zeta^{2}_{k}\end{bmatrix}$, and . After
some algebraic manipulations, it is readily verified that
${\mathbf{C}}=\begin{bmatrix}{\mathbf{I}}_{N}&-\mathbf{1}_{N}\\\
-\mathbf{1}^{T}_{N}&\Gamma\end{bmatrix}$ and
${\mathbf{C}}^{-1}=\begin{bmatrix}({\mathbf{I}}_{N}+\frac{1}{\eta}\mathbf{1}_{N}\mathbf{1}^{T}_{N})&\frac{1}{\eta}\mathbf{1}_{N}\\\
\frac{1}{\eta}\mathbf{1}^{T}_{N}&\frac{1}{\eta}\end{bmatrix}$, where
$\Gamma=N+\eta$. Now, it is straightforward to conclude
${\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{e}}_{N+1}=\frac{1}{\eta}$ and
${\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}=\frac{\mathbf{1}^{T}_{N+1}{\mathbf{d}}^{(l)}_{m}}{\eta}.$
From the definition of ${\mathbf{d}}^{(l)}_{m}$ in (10), we obtain
${\mathbf{d}}^{(l)}_{m}=\frac{1}{K}\sum_{k=1}^{K}{\mathbf{C}}^{T}_{k}{\mathbf{b}}^{(l)}_{k,m}$,
where
${\mathbf{b}}^{(l)}_{k,m}=[-\zeta_{k}\gamma^{(l)}_{n,m},\dots,-\zeta_{k}\gamma^{(l)}_{N,m}]^{T}$
is the $m$th column of ${\mathbf{B}}^{(l)}_{k}$ in (9). Since
${\mathbf{d}}^{(l)}_{m}=\frac{1}{K}\sum_{k=1}^{K}[{\mathbf{I}}_{N},-\zeta_{k}\mathbf{1}^{T}_{N}]^{T}{\mathbf{b}}^{(l)}_{k,m}=\frac{1}{K}\sum_{k=1}^{K}[{\mathbf{b}}^{(l)}_{k,m},-\zeta_{k}\mathbf{1}^{T}_{N}{\mathbf{b}}^{(l)}_{k,m}]^{T},$
we have
${\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}=\frac{\mathbf{1}^{T}_{N+1}{\mathbf{d}}^{(l)}_{m}}{\eta}=\frac{(2m-1)N-1}{2}\psi_{c,l},$
which completes the proof of (15b).

Now, we are ready to prove _Theorem_ 1. We start by formulating the Lagrangian
of (10),

${\mathcal{L}}({\mathbf{a}}^{(l)}_{m},\lambda_{1},\lambda_{2})={\mathbf{a}}^{(l)T}_{m}{\mathbf{C}}{\mathbf{a}}^{(l)}_{m}-2{\mathbf{d}}^{(l)T}_{m}{\mathbf{a}}^{(l)}_{m}+\\\
\lambda_{1}({\mathbf{e}}_{N+1}^{T}{\mathbf{a}}^{(l)}_{m}-\vartheta_{\max})+\lambda_{2}(-{\mathbf{e}}_{N+1}^{T}{\mathbf{a}}^{(l)}_{m}),$
(16)

where $\lambda_{1}\geq 0$ and $\lambda_{2}\geq 0$ are the Largrangian
multipliers. After incorporating the first order necessary condition for
${\mathbf{a}}_{m}^{(l)}$ in (16), the KKT conditions of (10) are given by:

$\left\\{\begin{array}[]{ll}2{\mathbf{C}}{\mathbf{a}}^{(l)}_{m}-2{\mathbf{d}}^{(l)}_{m}+\lambda_{1}{\mathbf{e}}_{N+1}-\lambda_{2}{\mathbf{e}}_{N+1}&=0,\\\
\lambda_{1}({\mathbf{e}}_{N+1}^{T}{\mathbf{a}}^{(l)}_{m}\\!-\\!\vartheta_{\max})&=0,\\\
\lambda_{2}(-{\mathbf{e}}_{N+1}^{T}{\mathbf{a}}^{(l)}_{m})&=0,\\\
\lambda_{1}\geq 0,\lambda_{2}\geq 0.\end{array}\right.$ (17)

It is obvious that when $\lambda_{1}>0$ and $\lambda_{2}>0$, there is no
solution. When $\lambda_{1}=\lambda_{2}=0$, solving (17) yields
${\mathbf{a}}^{(l)}_{m}={\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}$ for
$0\leq{\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}\leq\vartheta_{\max}$.
When $\lambda_{1}>0$ and $\lambda_{2}=0$, (17) yields
${\mathbf{a}}^{(l)}_{m}={\mathbf{C}}^{-1}({\mathbf{d}}^{(l)}_{m}-\lambda_{1}{\mathbf{e}}_{N+1})$
for
${\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{b}}^{(l)}_{m}>\vartheta_{\max}$.
However, when $\lambda_{1}=0$ and $\lambda_{2}>0$, we obtain
${\mathbf{a}}^{(l)}_{m}={\mathbf{C}}^{-1}({\mathbf{d}}^{(l)}_{m}+\lambda_{2}{\mathbf{e}}_{N+1})$
for ${\mathbf{e}}_{N+1}^{T}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}<0$, which
contradicts with (15), because from the sign invariance property of array
gain, we have $\frac{(2m-1)N-1}{2}\psi_{c,l}\geq 0$, $\forall\psi_{c,l}\geq
0.$ In short, solving (17) leads to

${\mathbf{a}}^{(l)\star}_{m}\\!\\!=\\!\\!\left\\{\\!\\!\\!\begin{array}[]{ll}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m},\\!\\!\\!\\!\\!\\!\\!\\!&\text{
if
}0\leq{\mathbf{e}}^{T}_{N+1}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}\leq\vartheta_{\max},\\\
{\mathbf{C}}^{-1}({\mathbf{d}}^{(l)}_{m}\\!-\\!\lambda_{1}{\mathbf{e}}_{N+1}),\\!\\!\\!\\!\\!\\!\\!&\text{
if
}{\mathbf{e}}^{T}_{N+1}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}>\vartheta_{\max},\end{array}\right.$
(18)

where
$\lambda_{1}=\frac{{\mathbf{e}}^{T}_{N+1}{\mathbf{C}}^{-1}{\mathbf{d}}^{(l)}_{m}-\vartheta_{\max}}{{\mathbf{e}}^{T}_{N+1}{\mathbf{C}}^{-1}{\mathbf{e}}_{N+1}}$.
Substituting (15) into (18) results in the closed-form solution of (10) as in
(11) and (12), which concludes the proof.

## References

  * [1] I. F. Akyildiz, J. M. Jornet, and C. Han, “Terahertz band: Next frontier for wireless communications,” _Physical Communication_ , vol. 12, pp. 16–32, 2014.
  * [2] T. L. Marzetta, “Noncooperative cellular wireless with unlimited numbers of base station antennas,” _IEEE Transactions on Wireless Communications_ , vol. 9, no. 11, pp. 3590–3600, 2010.
  * [3] F. Rusek, D. Persson, B. K. Lau, E. G. Larsson, T. L. Marzetta, O. Edfors, and F. Tufvesson, “Scaling up MIMO: Opportunities and challenges with very large arrays,” _IEEE Signal Processing Magazine_ , vol. 30, no. 1, pp. 40–60, 2013.
  * [4] E. G. Larsson, O. Edfors, F. Tufvesson, and T. L. Marzetta, “Massive mimo for next generation wireless systems,” _IEEE Communications Magazine_ , vol. 52, no. 2, pp. 186–195, 2014.
  * [5] O. E. Ayach, S. Rajagopal, S. Abu-Surra, Z. Pi, and R. W. Heath, “Spatially sparse precoding in millimeter wave MIMO systems,” _IEEE Transactions on Wireless Communications_ , vol. 13, no. 3, pp. 1499–1513, 2014.
  * [6] A. Alkhateeb, O. El Ayach, G. Leus, and R. W. Heath, “Channel estimation and hybrid precoding for millimeter wave cellular systems,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 8, no. 5, pp. 831–846, 2014.
  * [7] H. Ghauch, T. Kim, M. Bengtsson, and M. Skoglund, “Subspace estimation and decomposition for large millimeter-wave mimo systems,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 10, no. 3, pp. 528–542, 2016.
  * [8] C. Han, L. Yan, and J. Yuan, “Hybrid beamforming for terahertz wireless communications: Challenges, architectures, and open problems,” _IEEE Wireless Communications_ , vol. 28, no. 4, pp. 198–204, 2021.
  * [9] M. Cai, K. Gao, D. Nie, B. Hochwald, J. N. Laneman, H. Huang, and K. Liu, “Effect of wideband beam squint on codebook design in phased-array wireless systems,” in _2016 IEEE Global Communications Conference (GLOBECOM)_ , 2016, pp. 1–6.
  * [10] B. Wang, M. Jian, F. Gao, G. Y. Li, and H. Lin, “Beam squint and channel estimation for wideband mmwave massive MIMO-OFDM systems,” _IEEE Transactions on Signal Processing_ , vol. 67, no. 23, pp. 5893–5908, 2019.
  * [11] X. Liu and D. Qiao, “Space-time block coding-based beamforming for beam squint compensation,” _IEEE Wireless Communications Letters_ , vol. 8, no. 1, pp. 241–244, 2019.
  * [12] R. Mailloux, _Phased Array Antenna Handbook, Third Edition_ , 2017.
  * [13] M. Longbrake, “True time-delay beamsteering for radar,” in _2012 IEEE National Aerospace and Electronics Conference (NAECON)_ , 2012, pp. 246–249.
  * [14] J. Tan and L. Dai, “Delay-phase precoding for thz massive MIMO with beam split,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , 2019, pp. 1–6.
  * [15] F. Gao, B. Wang, C. Xing, J. An, and G. Y. Li, “Wideband beamforming for hybrid massive MIMO terahertz communications,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 6, pp. 1725–1740, 2021.
  * [16] K. Dovelos, M. Matthaiou, H. Q. Ngo, and B. Bellalta, “Channel estimation and hybrid combining for wideband terahertz massive MIMO systems,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 6, pp. 1604–1620, 2021\.
  * [17] M.-K. Cho, I. Song, and J. D. Cressler, “A true time delay-based sige bi-directional T/R chipset for large-scale wideband timed array antennas,” in _2018 IEEE Radio Frequency Integrated Circuits Symposium (RFIC)_ , 2018, pp. 272–275.
  * [18] R. Bracewell, _The Fourier Transform and its Applications_ , 3rd ed. McGraw-Hill, New York, 2000.
  * [19] W. Zhang, T. Kim, D. J. Love, and E. Perrins, “Leveraging the restricted isometry property: Improved low-rank subspace decomposition for hybrid millimeter-wave systems,” _IEEE Transactions on Communications_ , vol. 66, no. 11, pp. 5814–5827, 2018.
  * [20] X. Yu, J. Zhang, and K. B. Letaief, “Alternating minimization for hybrid precoding in multiuser OFDM mmWave systems,” in _2016 50th Asilomar Conference on Signals, Systems and Computers_ , 2016, pp. 281–285.
  * [21] F. Sohrabi and W. Yu, “Hybrid analog and digital beamforming for mmWave OFDM large-scale antenna arrays,” _IEEE Journal on Selected Areas in Communications_ , vol. 35, no. 7, pp. 1432–1443, 2017.


# Dynamic Memory based Attention Network for Sequential Recommendation

Qiaoyu Tan 1, Jianwei Zhang 2, Ninghao Liu 1, Xiao Huang 3
Hongxia Yang 2, Jingren Zhou 2, Xia Hu 1

###### Abstract

Sequential recommendation has become increasingly essential in various online
services. It aims to model the dynamic preferences of users from their
historical interactions and predict their next items. The accumulated user
behavior records on real systems could be very long. This rich data brings
opportunities to track actual interests of users. Prior efforts mainly focus
on making recommendations based on relatively recent behaviors. However, the
overall sequential data may not be effectively utilized, as early interactions
might affect users’ current choices. Also, it has become intolerable to scan
the entire behavior sequence when performing inference for each user, since
real-world system requires short response time. To bridge the gap, we propose
a novel long sequential recommendation model, called Dynamic Memory-based
Attention Network (DMAN). It segments the overall long behavior sequence into
a series of sub-sequences, then trains the model and maintains a set of memory
blocks to preserve long-term interests of users. To improve memory fidelity,
DMAN dynamically abstracts each user’s long-term interest into its own memory
blocks by minimizing an auxiliary reconstruction loss. Based on the dynamic
memory, the user’s short-term and long-term interests can be explicitly
extracted and combined for efficient joint recommendation. Empirical results
over four benchmark datasets demonstrate the superiority of our model in
capturing long-term dependency over various state-of-the-art sequential
models.

## Introduction

Recommender systems have become an important tool in various online systems
such as E-commerce, social media, and advertising systems to provide
personalized services (Hidasi et al. 2015; Ying et al. 2018b). One core stage
of live industrial systems is candidate selection and ranking (Covington,
Adams, and Sargin 2016), which is responsible for retrieving a few hundred
relevant items from a million or even billion scale corpus. Previously,
researchers resort to collaborative filtering approaches (Sarwar et al. 2001)
to solve it by assuming that like-minded users tend to exhibit similar
preferences on items. Typical examples including models based on matrix
factorization (Sarwar et al. 2001), factorization machines (Rendle 2010), and
graph neural networks (Ying et al. 2018b; Wang et al. 2019b; Tan et al. 2020).
However, these methods ignore the temporal dynamics of user behaviors.

To capture sequential dynamics for user modeling, various sequential
recommendation methods have been proposed to make recommendations based on
user’s past behaviors (Hidasi et al. 2015; Tang and Wang 2018; Tan et al.
2021). They aim to predict the next item(s) that a user is likely to interact
with, given her/his historical interactions. Recently, a myriad of attempts
that build upon sequential neural networks, such as recurrent neural network
(RNNs), convolutional neural networks (CNNs), and self-attention networks,
have achieved promising results in various recommendation scenarios (Hidasi
and Karatzoglou 2018; Yuan et al. 2019; Yan et al. 2019; Sun et al. 2019;
Zhang et al. 2018). The basic paradigm is to encode a user’s historical
interactions into a vector using various sequential modules based on the
behavior sequence, which is obtained by sorting her/his past behaviors in
chronological order.

However, as many E-commerce and social media systems keep accumulating users’
records, the behavior sequences have become extraordinarily long. For example,
more than twenty thousand customers on the Alibaba e-commerce platform have
interacted with over one thousand items from April to September 2018 (Ren et
al. 2019). Despite the fertile information contained in these long behavior
sequences, existing sequential recommendation algorithms would achieve sub-
optimal performance in modeling long behavior sequences. The reason is that
standard sequential architectures (e.g., RNNs, CNNs, attention networks) are
insufficient to capture long-term dependencies confirmed in sequence learning
(Graves, Wayne, and Danihelka 2014; Yu and Koltun 2015; Dai et al. 2019).
Directly applying them to model long behavior sequences would result in
significant performance degeneration. Thus, in this paper, we target at
exploring sequential recommendation with extraordinary long user behavior
sequences.

There are three major challenges in learning from long behavior sequences. 1)
Given that the response time in real-world systems is limited, it has become
expensive to scan over the entire behavior sequence at each prediction time
(Zhu et al. 2018). Existing sequential recommender systems often require read
the whole behavior sequence. A few recent long sequential recommendation
methods explore splitting the whole input behavior sequence into short-term
and long-term behavior sequences and then explicitly extracting a user’s
temporal and long-term preferences (Ying et al. 2018a; Lv et al. 2019).
Despite their simplicity, they still suffer from high computation complexity
since they need to scan over the whole behavior sequence during inference. 2)
It is crucial to model the whole behavior sequence for a more accurate
recommendation. A few attempts have been made to focus only on short-term
actions (Li, Wang, and McAuley 2020; Hidasi and Karatzoglou 2018) and abandon
previous user behaviors. Nevertheless, studies (Ren et al. 2019; Belletti,
Chen, and Chi 2019) have demonstrated that user preferences may be influenced
by her/his early interactions beyond the short-term behavior sequence. 3) It
is hard to explicitly control the contributions of long-term or short-term
interests for user modeling. Some studies resort to memory neural network
(Graves, Wayne, and Danihelka 2014) to implicitly preserve the long-term
intentions for efficient sequential modeling (Chen et al. 2018; Ren et al.
2019). But they may suffer from long-term knowledge forgetting (Sodhani,
Chandar, and Bengio 2018), due to that the memory is optimized by predicting
the next-item. Therefore, an advanced sequential model is needed to explicitly
model both long-term and short-term preferences, as well as supporting
efficient inference.

To address the limitations above, we propose a novel dynamic memory-based
self-attention network, dubbed DMAN, to model long behavior sequence data. It
offers standard self-attention networks to capture long-term dependencies for
user modeling effectively. To improve model efficiency, DMAN truncates the
whole user behavior sequence into several successive sub-sequences and
optimizes the model sequence by sequence. Specifically, a recurrent attention
network is derived to utilize the correlation between adjacent sequences for
short-term interest modeling. Meanwhile, another attention network is
introduced to measure dependencies beyond consecutive sequences for long-term
interest modeling based on a dynamic memory, which preserves user behaviors
before the adjacent sequences. Finally, the two aspect interests are
adaptively integrated via a neural gating network for the joint
recommendation. To enhance the memory fidelity, we further develop a dynamic
memory network to effectively update the memory blocks sequence by sequence
using an auxiliary reconstruction loss. To summarize, the main contributions
of this paper are as follows:

>   * •
>
> We propose a dynamic memory-based attention network DMAN for modeling long
> behavior sequences, which conducts an explicit and adaptive user modeling
> and supports efficient inference.
>
>   * •
>
> We derive a dynamic memory network to dynamically abstract a user’s long-
> term interests into an external memory sequence by sequence.
>
>   * •
>
> Extensive experiments on several challenging benchmarks demonstrate our
> method’s effectiveness in modeling long user behavior data.
>
>

Figure 1: Illustration of DMAN for one layer. It takes a series of sequences
as input and trains the model sequence by sequence. When processing the $n$-th
sequence $\mathcal{S}_{n}$, the recurrent attention network is applied to
extract short-term user interest by using the previous hidden state
$\widetilde{\mathbf{H}}_{n-1}$ as context. Meanwhile, the long-term attention
network is utilized to extract long-term interest based on the memory blocks
$\mathbf{M}$. Next, the short-term and long-term interests are combined via a
neural gating network for joint user modeling. Finally, the dynamic memory
network updates the memory blocks via fusing the information in
$\widetilde{\mathbf{H}}_{n-1}$, and the model continues to process the next
sequence. The overall model is optimized by maximizing the likelihood of
observed sequence, while the dynamic memory network is trained based on a
local reconstruction loss $\mathcal{L}_{ae}$.

## The Proposed DMAN Model

In this section, we first introduce the problem formulation and then discuss
the proposed framework in detail.

### Notations and Problem Formulation

Assume $\mathcal{U}$ and $\mathcal{V}$ denote the sets of users and items,
respectively.
$\mathcal{S}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{|\mathcal{S}|}\\}$
represents the behavior sequence in chronological order of a user.
$\mathbf{x}_{t}\in\mathcal{V}$ records the $t$-th item interacted by the user.
Given an observed behavior sequence
$\\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{t}\\}$, the sequential
recommendation task is to predict the next items that the user might be
interacted with. Notations are summarized in Table 1.

In our setting, due to the accumulated behavior sequence $\mathcal{S}$ is very
long, we truncate it into a series of successive sub-sequences with fixed
window size $T$, i.e., $\mathcal{S}=\\{\mathcal{S}_{n}\\}_{n=1}^{N}$, for the
model to process efficiently.
$\mathcal{S}_{n}=\\{\mathbf{x}_{n,1},\mathbf{x}_{n,2},\ldots,\mathbf{x}_{n,T}\\}$
denotes the $n$-th sequence. Traditional sequential recommendation methods
mainly rely on a few recent behaviors $\mathcal{S}_{N}$ for user modeling. Our
paper focuses on leveraging the whole behavior sequence for a comprehensive
recommendation. We first illustrate how to explicitly extract short-term and
long-term user interests from historical behaviors, and then describe an
adaptive way to combine them for joint recommendation. Finally, we introduce a
novel dynamic memory network to preserve user’s long-term interests for
efficient inference effectively.

Table 1: Notations summary. Notation | Description
---|---
$u$ | a user
$t$ | an item
$\mathbf{x}$ | an interaction record
$\mathcal{U}$ | the set of users
$\mathcal{V}$ | the set of items
$\mathcal{S}_{n}$ | the $n$-th behavior sequence
$K$ | the number of candidate items
$N$ | the number of sliced sequences
$L$ | the number of self-attention layers
$m$ | the number of memory slots
$D$ | the number of embedding dimension
$\widetilde{\mathbf{H}}$ | the short-term interest embedding
$\widehat{\mathbf{H}}$ | the long-term interest embedding
$\mathbf{M}$ | the memory embedding matrix
$\mathbf{V}$ | the output user embedding

### Recurrent Attention Network

This subsection introduces the proposed recurrent attention network for short-
term interest modeling. Given an arbitrary behavior sequence $\mathcal{S}_{n}$
as input, an intuitive way to estimate a user’s short-term preferences is only
consider her/his behaviors within the sequence. However, the first few items
in each sequence may lack necessary context for effective modeling, because
previous sequences are not considered.

To address this limitation, we introduce the notion of recurrence in RNNs into
self-attention network and build a sequence-level recurrent attention network,
enabling information flow between adjacent sequences. In particular, we use
the hidden state computed for last sequence as additional context for next
sequence modeling. Formally, let $\mathcal{S}_{n-1}$ and $\mathcal{S}_{n}$ be
two successive sequences, and
$\widetilde{\mathbf{H}}^{l}_{n-1}\in\mathbb{R}^{T\times D}$ denote the $l$-th
layer hidden state produced for sequence $\mathcal{S}_{n-1}$. We calculate the
hidden state of sequence $\mathcal{S}_{n}$ as follows.

$\displaystyle\widetilde{\mathbf{H}}_{n}^{l}=\text{Atten}_{\text{rec}}^{l}(\widetilde{\mathbf{Q}}_{n}^{l},\widetilde{\mathbf{K}}_{n}^{l},\widetilde{\mathbf{V}}_{n}^{l})=\text{softmax}(\widetilde{\mathbf{Q}}_{n}^{l}(\widetilde{\mathbf{K}}_{n}^{l})^{\top})\widetilde{\mathbf{V}}_{n}^{l},$
(1)
$\displaystyle\widetilde{\mathbf{Q}}_{n}^{l}=\widetilde{\mathbf{H}}_{n}^{l-1}\widetilde{\mathbf{W}}_{Q}^{\top},\text{\quad
and
\quad}\widetilde{\mathbf{K}}_{n}^{l}=\mathbf{H}_{n}^{l-1}\widetilde{\mathbf{W}}_{K}^{\top},$
$\displaystyle\widetilde{\mathbf{V}}_{n}^{l}=\mathbf{H}_{n}^{l-1}\widetilde{\mathbf{W}}_{V}^{\top},$
$\displaystyle\mathbf{H}_{n}^{l-1}=\widetilde{\mathbf{H}}_{n}^{l-1}\mathbin{\|}\text{SG}(\widetilde{\mathbf{H}}_{n-1}^{l-1}),$

where $\text{Atten}_{\text{rec}}^{l}(\cdot,\cdot,\cdot)$ is the $l$-th layer
self-attention network, in which the query, key and value matrices are denoted
by $\mathbf{Q}$, $\mathbf{K}$ and $\mathbf{V}$, respectively. The input of the
first layer is the sequence embedding matrix
$\mathbf{X}_{n}=[\mathbf{x}_{n,1},\ldots,\mathbf{x}_{n,T}]\in\mathbb{R}^{T\times
D}$. Intuitively, the attention layer calculates a weighted sum of embeddings,
where the attention weight is computed between query $i$ in $\mathcal{S}_{n}$
and value $j$ obtained from previous sequences. The function
$\text{SG}(\cdot)$ stands for stop-gradient from previous hidden state
$\widetilde{\mathbf{H}}_{n-1}^{l-1}$, and $\mathbin{\|}$ denotes
concatenation. In our case, we use the extended context as key and value and
adopt three linear transformations to improve the model flexibility, where
$\\{\widetilde{\mathbf{W}}_{Q},\widetilde{\mathbf{W}}_{K},\widetilde{\mathbf{W}}_{V}\\}\in\mathbb{R}^{D\times
D}$ denote the parameters. The extended context not only provides precious
information for recovering the first few items, but also allows our model to
capture the dependency across sequences. In practice, instead of computing the
hidden states from scratch at each time point, we cache the hidden state of
last sequence for reuse. Besides, masking strategy and positional embeddings
are also included to avoid the future information leakage problem (Yuan et al.
2019) and capture sequential dynamics (Kang and McAuley 2018; Wang et al.
2019a). The final short-term interest embedding is defined as
$\widetilde{\mathbf{H}}_{n}=\widetilde{\mathbf{H}}_{n}^{L}$.

### Long-term Attention Network

In this subsection, we present another attention network for long-term
interest modeling. With the recurrent connection mechanism defined in Eq. 1,
our model can capture correlations between adjacent sequences for interest
modeling. However, longer-range dependencies beyond successive sequences may
still be ignored, since the recurrent connection mechanism is limited in
capturing longer-range correlations (Sodhani, Chandar, and Bengio 2018;
Sukhbaatar et al. 2015). Hence, additional architecture is needed to
effectively capture long-term user preferences.

To this end, we maintain an external memory matrix
$\mathbf{M}\in\mathbb{R}^{m\times D}$ to explicitly memorize a user’s long-
term preferences, where $m$ is the number of memory slots. Each user is
associated with a memory. Ideally, the memory complements with the short-term
interest modeling, with the aim to capture dependencies beyond adjacent
sequences. We leave how to effectively update the memory in later section and
now focus on how to extract long-term interests from the memory. Specifically,
let $\mathbf{M}^{l}\in\mathbb{R}^{m\times D}$ denote the $l$-th layer memory
matrix, we estimate the long-term hidden state of sequence $\mathcal{S}_{n}$
using another self-attention network as

$\displaystyle\widehat{\mathbf{H}}_{n}^{l}$
$\displaystyle=\text{Atten}^{l}(\widehat{\mathbf{Q}}_{n}^{l},\widehat{\mathbf{K}}_{n}^{l},\widehat{\mathbf{V}}_{n}^{l}),$
(2)
$\displaystyle\widehat{\mathbf{Q}}_{n}^{l},\widehat{\mathbf{K}}_{n}^{l},\widehat{\mathbf{V}}_{n}^{l}$
$\displaystyle=\widehat{\mathbf{H}}_{n}^{l-1}\widehat{\mathbf{W}}_{Q}^{\top},\mathbf{M}^{l-1}\widehat{\mathbf{W}}_{K}^{\top},\mathbf{M}^{l-1}\widehat{\mathbf{W}}_{V}^{\top}.$

Similarly, $\text{Atten}^{l}(\cdot,\cdot,\cdot)$ is a self-attention network.
It takes the last layer hidden state $\widehat{\mathbf{H}}_{n}^{l-1}$ as query
and uses the layer-wise memory matrix $\mathbf{M}^{l-1}$ as key (value). The
input of the query is $\mathbf{X}_{n}$. By doing so, the output hidden state
$\widehat{\mathbf{H}}^{l}_{n}\in\mathbb{R}^{T\times D}$ is a selective
aggregation of $m$ memory blocks, where the selection weight is query-based
and varies across different queries.
$\\{\widehat{\mathbf{W}}_{Q},\widehat{\mathbf{W}}_{K},\widehat{\mathbf{W}}_{V}\\}$
are trainable transformation matrices to improve model capacity. Since the
memory is maintained to cache long-term user interests that beyond adjacent
sequences, we refer to the above attention network as long-term interest
modeling. The final long-term interest embedding for sequence
$\mathcal{S}_{k}$ is denoted as
$\widehat{\mathbf{H}}_{n}=\widehat{\mathbf{H}}_{n}^{L}$.

### Neural Gating Network

After obtaining the short-term and long-term interest embeddings, the next aim
is to combine them for comprehensive modeling. Considering that a user’s
future intention can be influenced by early behaviors, while short-term and
long-term interests may contribute differently for next-item prediction over
time (Ma et al. 2019), we apply a neural gating network to adaptively control
the importance of the two interest embeddings.

$\displaystyle\mathbf{V}_{n}$
$\displaystyle=\mathbf{G}_{n}\odot\widetilde{\mathbf{H}}_{n}+(1-\mathbf{G}_{n})\odot\widehat{\mathbf{H}}_{n},$
(3) $\displaystyle\mathbf{G}_{n}$
$\displaystyle=\sigma(\widetilde{\mathbf{H}}_{n}\mathbf{W}_{short}+\widehat{\mathbf{H}}_{n}\mathbf{W}_{long}),$

where $\mathbf{G}_{n}\in\mathbb{R}^{T\times D}$ is the gate matrix learned by
a non-linear transformation based on short-term and long-term embeddings.
$\sigma(\cdot)$ indicates the sigmoid activation function, $\odot$ denotes
element-wise multiplication, and $\mathbf{W}_{short}$,
$\mathbf{W}_{long}\in\mathbb{R}^{D\times D}$ are model parameters. The final
user embedding $\mathbf{V}_{n}\in\mathbb{R}^{T\times D}$ is obtained by a
feature-level weighted sum of two types of interest embeddings controlled by
the gate.

### Dynamic Memory Network

In this subsection, we describe how to effectively update the memory
$\mathbf{M}$ to preserve long-term user preferences beyond adjacent sequences.
One feasible solution is to maintain a fixed-size FIFO memory to cache long-
term message. This strategy is sub-optimal for user modeling due to two
reasons. First, the oldest memories will be discarded if the memory is full,
whether it is important or not. This setting is reasonable in NLP task (Rae et
al. 2019) as two words that too far away in the sentence are often not
correlated. But it is not held in recommendation because behavior sequence is
not strictly ordered (Yuan et al. 2019) and users often exhibit monthly or
seasonal periodic behaviors. Second, the memory is redundant and not
effectively utilized, since user interests in practice is often bounded in
tens (Li et al. 2019).

To avoid these limitations, we propose to abstract a user’s long-term
interests from the past actively. Assume the model was processed sequence
$\mathcal{S}_{n}$, then the memory is updated as

$\displaystyle\mathbf{M}^{l}\leftarrow
f_{abs}^{l}(\mathbf{M}^{l},\widetilde{\mathbf{H}}^{l}_{n-1}),$ (4)

where $f_{abs}^{l}:\mathbb{R}^{(m+T)\times D}\rightarrow\mathbb{R}^{m\times
D}$ is the $l$-th layer abstraction function. It takes the old memory and the
context state $\widetilde{\mathbf{H}}^{l}_{n-1}$ as input and updates memory
$\mathbf{M}^{l}$ to represent user interests. In theory, $f_{abs}$ requires to
effectively preserve the primary interests in old memories and merges
contextual information. Basically, $f_{abs}$ can be trained with the next-item
prediction task end-to-end. Nevertheless, memories that differ from the target
item may be discarded. Therefore, we consider train the abstraction function
with an auxiliary attention-based reconstruction loss as follows.

$\displaystyle\mathcal{L}_{ae}$
$\displaystyle=\min\sum_{l=1}^{L}||\text{attent}_{rec}^{l}(\widetilde{\mathbf{Q}}^{l},\widetilde{\mathbf{K}}^{l},\widetilde{\mathbf{V}}^{l})-\text{attent}_{rec}^{l}(\widetilde{\mathbf{Q}}^{l},\widehat{\mathbf{K}}^{l},\widehat{\mathbf{V}}^{l})||^{2}_{F}$
(5) $\displaystyle\widetilde{\mathbf{Q}}^{l}$
$\displaystyle=\widetilde{\mathbf{H}}^{l}_{n},\
\widetilde{\mathbf{K}}^{l}=\widetilde{\mathbf{V}}^{l}=\mathbf{M}^{l}\mathbin{\|}\widetilde{\mathbf{H}}_{n-1}^{l},\
\widehat{\mathbf{K}}^{l}=\widehat{\mathbf{V}}^{l}=\mathbf{M}^{l},$

where $\text{atten}_{rec}^{l}(\cdot,\cdot,\cdot)$ is the self-attention
network defined in Eq. (1). We reuse the recurrent attention network but keep
the parameters fixed and not trainable here. We employ the hidden state
$\widetilde{\mathbf{H}}^{l}_{n}$ of $\mathcal{S}_{n}$ as query for two
attention networks. The first attention outputs a new representation for the
query via a weighted sum from the old and new memories, while the second from
the abstracted memories. By minimizing the reconstruction loss, we expect the
primary interests can be extracted by $f_{abs}$ as much as possible. Note that
we consider a lossy objective here because the information that is no longer
attended to in $\mathcal{S}_{n}$ can be discarded in order to capture the
shifting of user interests to some extent.

#### Implementation of abstraction function $f_{abs}$

We parameterize $f_{abs}$ with the dynamic routing method in CapsNet (Sabour,
Frosst, and Hinton 2017) for its promising results in capturing user’s diverse
interests in recommendation (Li et al. 2019). Suppose we have two layers of
capsules, we refer capsules from the first layer and the second layer as
primary capsules and interest capsules, respectively. The goal of dynamic
routing is to calculate the values of interest capsules given the primary
capsules in an iterative fashion. In each iteration, given primary capsules
vectors $\mathbf{x}_{i}$ (input vector), $i\in\\{1,\ldots,T+m\\}$ and interest
capsules $\bar{\mathbf{x}}_{j}$ (output vector), $j\in\\{1,\ldots,m\\}$, the
routing logit $b_{ij}$ between primary capsule $i$ and interest capsule $j$ is
computed by

$\displaystyle
b_{ij}=\bar{\mathbf{x}}_{j}^{\top}\mathbf{W}_{ij}\mathbf{x}_{i},$ (6)

where $\mathbf{W}_{ij}$ is a transformation matrix. Given routing logits,
$\mathbf{s}_{j}$ is computed as weighted sum of all primary capsules

$\displaystyle\mathbf{s}_{j}=\sum_{i=1}^{m+T}\alpha_{ij}\mathbf{W}_{ij}\mathbf{x}_{i},$
(7)

where
$\alpha_{ij}=\exp(b_{ij})/\sum_{j^{\prime}=1}^{m+T}\exp(b_{ij^{\prime}})$ is
the connection weight between primary capsule $i$ and interest capsule $j$.
Finally, a non-linear ”squashing” function (Sabour, Frosst, and Hinton 2017)
is proposed to obtain the corresponding vectors of interest capsules as

$\displaystyle\bar{\mathbf{x}}_{j}=\text{squash}(\mathbf{s}_{j})=\frac{\mathbin{\|}\mathbf{s}_{j}\mathbin{\|}^{2}}{1+\mathbin{\|}\mathbf{s}_{j}\mathbin{\|}^{2}}\frac{\mathbf{s}_{j}}{\mathbin{\|}\mathbf{s}_{j}\mathbin{\|}}.$
(8)

The routing process between Eq. (6) and Eq. (8) usually repeats three times to
converge. When routing finishes, the output interest capsules of user $u$ are
then used as the memory, i.e.,
$\mathbf{M}=[\bar{\mathbf{x}}_{1},\ldots,\bar{\mathbf{x}}_{m}]$.

### Model Optimization

As the data is derived from the user implicit feedback, we formulate the
learning problem as a binary classification task. Given the training sample
$(u,t)$ in a sequence $\mathbf{S}_{n}$ with the user embedding vector
$\mathbf{V}_{n,t}$ and target item embedding $\mathbf{x}_{t}$, we aim to
minimize the following negative likelihood

$\displaystyle\mathcal{L}_{like}$
$\displaystyle=-\sum_{u\in\mathcal{U}}\sum_{t\in\mathcal{S}_{n}}\log
P(\mathbf{x}_{n,t}|\mathbf{x}_{n,1},\mathbf{x}_{n,2},\cdots,\mathbf{x}_{n,t-1})$
(9)
$\displaystyle=-\sum_{u\in\mathcal{U}}\sum_{t\in\mathcal{S}_{n}}\log\frac{\exp(\mathbf{x}_{t}^{\top}\mathbf{V}_{n,t})}{\sum_{j\in\mathcal{V}}\exp(\mathbf{x}_{j}^{\top}\mathbf{V}_{n,t}))}.$

The loss above is usually intractable in practice because the sum operation of
the denominator is computationally prohibitive. Therefore, we adopt a negative
sampling strategy to approximate the softmax function in experiments. When the
data volume is large, we leverage Sampled Softmax technique (Covington, Adams,
and Sargin 2016; Jean et al. 2014) to further accelerate the training. Note
that Eqs. (9) and (6) are separately updated in order to preserve long-term
interests better. Specifically, we first update Eq. (9) by feeding a new
sequence and then updating the abstraction function’s parameters by minimizing
Eq. (6).

## Experiments and Analysis

Table 2: The dataset statistics.  Dataset | #Users | #Items | $T$ | $K$
---|---|---|---|---
MovieLens | 6,040 | 3,952 | 20 | 10
Taobao | 987,994 | 4,162,024 | 20 | 10
JD.com | 1,608,707 | 378,457 | 20 | 10
XLong | 20,000 | 3,269,017 | 50 | 20
Table 3: Sequential recommendation performance over three benchmarks. $*$ indicates the model only use the latest behavior sequence for training; otherwise, the whole behavior sequence. The second best results are underlined. Models | MovieLens | Taobao | JD.com
---|---|---|---
HR@10 | HR@50 | NDCG@100 | HR@50 | HR@100 | NDCG@100 | HR@10 | HR@50 | NDCG@100
GRU4Rec∗ | 17.69 | 43.13 | 16.90 | 10.42 | 14.01 | 4.23 | 27.65 | 38.73 | 23.40
Caser∗ | 18.98 | 45.64 | 17.62 | 13.71 | 16.51 | 6.89 | 29.27 | 40.16 | 24.25
SASRec∗ | 21.02 | 47.28 | 19.05 | 16.41 | 22.83 | 9.23 | 33.98 | 44.89 | 27.41
GRU4Rec | 19.78 | 47.40 | 18.75 | 13.48 | 16.53 | 5.81 | 35.28 | 47.52 | 27.64
Caser | 20.80 | 48.12 | 19.28 | 15.55 | 17.91 | 7.35 | 36.76 | 49.13 | 28.35
SASRec | 22.96 | 50.09 | 20.36 | 20.47 | 24.48 | 9.84 | 38.99 | 52.64 | 31.32
SHAN | 21.34 | 49.52 | 19.55 | 18.87 | 21.94 | 8.73 | 37.72 | 50.55 | 29.80
HPMN | 22.84 | 50.54 | 19.77 | 19.98 | 24.37 | 9.66 | 39.14 | 53.22 | 32.24
SDM | 23.42 | 51.26 | 20.44 | 21.66 | 25.42 | 10.22 | 40.68 | 55.30 | 34.82
DMAN | 25.18 | 53.24 | 22.03 | 24.92 | 29.37 | 11.13 | 44.58 | 58.82 | 36.93
Improv. | 7.51% | 3.86% | 7.77% | 15.05% | 15.53% | 8.90% | 9.58% | 6.36% | 6.05%

### Datasets

We conduct experiments over four public benchmarks. Statistics of them are
summarized in Table 2. MovieLens
111https://grouplens.org/datasets/movielens/1m/ collects users’ rating scores
for movies. JD.com (Lv et al. 2019) is a collection of user browsing logs over
e-commerce products collected from JD.com. Taobao (Zhu et al. 2018) and XLong
(Ren et al. 2019) are datasets of user behaviors from the commercial platform
of Taobao. The behavior sequence in XLong is significantly longer than other
three datasets, thus making it difficult to model.

### Baselines

To evaluate the performance of DMAN, we include three groups of baseline
methods. First, traditional sequential methods. To evaluate the effectiveness
of our model in dealing with long behavior sequence, three state-of-the-art
recommendation algorithms for sequences with a normal length have been
employed, including GRU4Rec (Tang and Wang 2018), Caser (Kang and McAuley
2018) and SASRec (He and Chua 2017). Second, long sequential methods. To
evaluate the effectiveness of our model in extracting long-term user interests
with dynamic memory, we include SDM (Lv et al. 2019) and SHAN (Ying et al.
2018a), which are tailored for modeling long behavior sequences. To evaluate
the effectiveness of our model in explicitly capturing user’s short-term and
long-term interests, we also set HPMN (Ren et al. 2019) as a baseline. It is
based on the memory network. Thrid, DMAN variants. To analyze the contribution
of each component of DMAN, we consider three variants. DMAN-XL discards the
long-term attention network to verify the effectiveness of capturing long-term
interests. DMAN-FIFO adopts a FIFO strategy to validate the usefulness of the
abstraction function in extracting primary interests. DMAN-NRAN replaces the
recurrent attention network with vanilla attention network to demonstrate the
effectiveness of extending context for effective user modeling.

### Experimental Settings

We obtain the behavior sequence by sorting behaviors in chronological order.
Following the traditional way (Kang and McAuley 2018), we employ the last and
second last interactions for testing and validation, respectively, and the
remaining for training. We follow the widely-adopted way (Li et al. 2017; Lv
et al. 2019) and split the ordered training sequence into $L$ consecutive
sequences. The maximum length of a sequence is $T$. The statistics of four
datasets are listed in Table 2. We repeatedly run the model five times and
report the average results.

#### Evaluation metrics

For each user in the test set, we treat all the items that the user has not
interacted with as negative items. To estimate the performance of top-$K$
recommendations, we use Hit Rate (HR$@K$) and Normalized Discounted Cumulative
Gain (NDCG$@K$) metrics, which are widely used in the literature (He and Chua
2017).

#### Parameter settings

For baselines, we use the source code released by the authors, and their
hyper-parameters are tuned to be optimal based on the validation set. To
enable a fair comparison, all methods are optimized with the number of samples
equals $5$ and the number of embedding dimensions $D$ equals $128$. We
implement DMAN with Tensorflow and the Adam optimizer is utilized to optimize
the model with learning rate equals $0.001$. The batch size is set to $128$
and the maximum epoch is $8$. The number of memory slots $m$ and attention
layers $L$ are searched from $\\{2,4,6,8,10,20,30\\}$ and $\\{1,2,3,4,5\\}$,
respectively.

Table 4: Performance on long user behavior data XLong. Method | Recall@200 | Recall@500
---|---|---
GRU4Rec∗ | 0.079 | 0.098
Caser∗ | 0.084 | 0.105
SASRec∗ | 0.105 | 0.123
GRU4Rec | 0.046 | 0.063
Caser | 0.023 | 0.041
SASRec | 0.061 | 0.096
SHAN | 0.091 | 0.115
HPMN | 0.118 | 0.136
SDM | 0.107 | 0.129
DMAN | 0.132 | 0.163

### Comparisons with SOTA

In this section, we compare our model with different baselines. Tables 3 and 4
report the results. In general, we have three aspects of observations.

#### Influence of modeling long behavior sequence for traditional sequential
methods

From Table 3, we observe that GRU4Rec, Caser, and SASRec improve their
performance when considering longer behavior sequence. Therefore, modeling
longer behavior sequence has proved to be effective for user modeling.
Besides, different sequential modules have varied abilities in handling long
behavior sequence. Specifically, SASRec, GRU4Rec, and Caser improve 24.74%,
29.36%, and 13.42% on Taobao in terms of $\text{HR}@50$, while SASRec
consistently performs the best. It indicates the ability of self-attention
network in extracting sequential patterns, and also validates our motivation
to extend self-attention network for long behavior sequence modeling.

#### Comparison with baselines on general datasets

As shown in Table 3, our model DMAN achieves better results than baselines
across three datasets. In general, long sequential models perform better than
traditional sequential methods, excepting SASRec. SASRec performs better than
SHAN and comparable to HPMN in most cases. This further implies the
effectiveness of self-attention network in capturing long-range dependencies.
The improvement of SDM over SASRec shows that explicitly extract long-term and
short-term interests from long sequence is beneficial. Considering DMAN and
SDM, DMAN consistently outperforms SDM over all evaluation metrics. This can
be attributed to that DMAN utilizes a dynamic memory network to actively
extract long-term interests into a small set of memory blocks, which is easier
for the attention network to effectively attend relative information than from
a long behavior sequence.

#### Comparison with baselines on long behavior dataset

Table 4 summarizes the results of all methods on XLong, where the length of
behavior sequence is larger than $1000$ on average. Obviously, DMAN
significantly outperforms other baselines. Compared with the findings in Table
3, one interest observation is that traditional sequential methods, i.e.,
GRU4Rec, Caser, and SASRec, performs poorly when directly handling long
behavior sequence, and lose to long sequential models in all cases. These
results demonstrate the necessity of developing new architectures tailored for
long behavior sequence modeling. Another unexpected observation is that HPMN
outperforms SDM on average. It further implies the ineffectiveness of
attention network in attending relative messages over long sequence. By
equipping attention network with the dynamic memory, our model allows us to
actively update user’s long-term interests in the memory and outperforms HPMN.

Table 5: Ablation study of DMAN. Dataset | Method | Recall@100 | NDCG@100
---|---|---|---
Taobao | DMAN-XL | 0.237 | 0.094
DMAN-FIFO | 0.263 | 0.108
DMAN-NRNA | 0.257 | 0.104
DMAN | 0.293 | 0.111
XLong | DMAN-XL | 0.021 | 0.013
DMAN-FIFO | 0.036 | 0.017
DMAN-NRAN | 0.043 | 0.019
DMAN | 0.054 | 0.022

### Ablation Study

We also conduct experiments to investigate the effectiveness of several core
components of the proposed DMAN. Table 5 reports the results on two
representative datasets. Obviously, DMAN significantly outperforms the other
three variants. The substantial difference between DMAN and DMAN-XL shows that
recurrent connection is not enough to capture user’s long-term interests. The
improvement of DMAN over DMAN-FIFO validates that the proposed abstraction
function is effective to extract user’s primary long-term interests. Besides,
DMAN outperforms DMAN-NRAN in general, which verifies the usefulness of
extending current context with previous hidden sequence state for short-term
interest extraction.

(a) Memory slots $m$

(b) Layer size $L$

(c) Learning curve

Figure 2: The proposed DMAN analysis

### Hyper-parameter Analysis

We further study the impacts of our model w.r.t. the number of memory slots
$m$ and attention layers $L$ on MovieLens. As we can see in Figure 2(a), DMAN
achieves satisfactory results when $m=20$ and the gain slows down with less
than 2% improvement when $m$ further increases. In experiments, we found 20 is
enough for MovieLens, Taobao, JD.com and XLong. From Figure 2(b), we observe
that the number of attention layers has positive impacts in our model. To
trade-off between memory costs and performance, we set $L=2$ for all datasets
since it already achieves satisfactory results. Besides, we also plot the
learning curve of DMAN on Taobao dataset in Figure 2(c), we can observe that
DMAN converges quickly after about 2 epochs. Similar observations have been
observed on other datasets. Specifically, DMAN tends to converge after 2
epochs on Taobao, JD.com and XLong datasets, while 50 epochs for MovieLens
data. These results demonstrate the training efficiency of our model.

## Related Work

### General Recommendation

Early recommendation works largely focused on explicit feedback (Koren 2008).
The recent research focus is shifting towards implicit data (Li and She 2017;
Hu, Koren, and Volinsky 2008). The typical examples include collaborative
filtering (Sarwar et al. 2001; Schafer et al. 2007), matrix factorization
techniques (Koren, Bell, and Volinsky 2009), and factorization machines
(Rendle 2010). The main challenge lies in representing users or items with
latent embedding vectors to estimate their similarity. Due to their ability to
learn salient representations, neural network-based models (Guo et al. 2017;
Su and Khoshgoftaar 2009; Tan, Liu, and Hu 2019) are also attracted much
attention recently. Some efforts adopt neural networks to extract side
attributes for content-aware recommendation (Kim et al. 2016), while some aim
to equip matrix factorization with non-linear interaction function (He and
Chua 2017) or graph convolutional aggregation (Wang et al. 2019b; Liu et al.
2019). In general, deep learning-based methods perform better than traditional
counterparts (Sedhain et al. 2015; Xue et al. 2017).

### Sequential Recommendation

Sequential recommendation takes as input the chronological behavior sequence
for user modeling. Typical examples belong to three categories. The first
relies on temporal matrix factorization (Koren 2009) to model user’s drifting
preferences. The second school uses either first-order (Rendle, Freudenthaler,
and Schmidt-Thieme 2010; Cheng et al. 2013) or hider-order (He and McAuley
2016; He et al. 2016; Yan et al. 2019) Markov-chains to capture the user state
dynamics. The third stream applies deep neural networks to enhance the
capacity of feature extraction (Yuan et al. 2019; Sun et al. 2019; Hidasi and
Karatzoglou 2018). For example, Caser (Tang and Wang 2018) applies CNNs to
process the item embedding sequence, while GRU4Rec (Hidasi et al. 2015) uses
gated recurrent unit GRU for session-based recommendation. Moreover, SASRec
(Kang and McAuley 2018) employs self-attention networks (Vaswani et al. 2017)
to selectively aggregate relevant items for user modeling.

However, these methods mainly focus on making recommendations based on
relatively recent behaviors. Recently, a few efforts attempt to model long
behavior sequence data. For instance, SDM (Lv et al. 2019) and SHAN (Ying et
al. 2018a) split the whole behavior sequence into short-term and long-term
sequences and then explicitly extract long-term and short-term interest
embeddings from them. But they are difficult to capture long-term interests
shifting and suffer from high computation complexity. HPMN (Ren et al. 2019)
uses the memory network (Graves, Wayne, and Danihelka 2014; Chen et al. 2018)
to memorize important historical behaviors for next-item prediction.
Nevertheless, memory network may suffer from long-term dependency forgetting
dilemma, as the memory is optimized by recovering the next item. Our model
focuses on combing external memory and attention networks for effective long
user behavior sequence modeling, which conducts an explicit and adaptive
modeling process.

## Conclusions

In this paper, we propose a novel dynamic memory-based attention network DMAN
for sequential recommendation with long behavior sequence. We truncate a
user’s overall behavior sequence into a series of sub-sequences and train our
model in a dynamic manner. DMAN can explicitly extract a user’s short-term and
long-term interests based on the recurrent connection mechanism and a set of
external memory blocks. To improve the memory fidelity, we derive a dynamic
memory network to actively abstract a user’s long-term interests into the
memory by minimizing a local reconstruction loss. Empirical results on real-
world datasets demonstrate the effectiveness of DMAN in modeling long user
behavior sequences.

## References

  * Belletti, Chen, and Chi (2019) Belletti, F.; Chen, M.; and Chi, E. H. 2019.  Quantifying Long Range Dependence in Language and User Behavior to improve RNNs.  In _KDD_ , 1317–1327.
  * Chen et al. (2018) Chen, X.; Xu, H.; Zhang, Y.; Tang, J.; Cao, Y.; Qin, Z.; and Zha, H. 2018.  Sequential recommendation with user memory networks.  In _WSDM_ , 108–116.
  * Cheng et al. (2013) Cheng, C.; Yang, H.; Lyu, M. R.; and King, I. 2013.  Where you like to go next: Successive point-of-interest recommendation.  In _IJCAI_.
  * Covington, Adams, and Sargin (2016) Covington, P.; Adams, J.; and Sargin, E. 2016.  Deep neural networks for youtube recommendations.  In _RecSys_ , 191–198.
  * Dai et al. (2019) Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019\.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_ .
  * Graves, Wayne, and Danihelka (2014) Graves, A.; Wayne, G.; and Danihelka, I. 2014.  Neural turing machines.  _arXiv preprint arXiv:1410.5401_ .
  * Guo et al. (2017) Guo, H.; Tang, R.; Ye, Y.; Li, Z.; and He, X. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_ .
  * He et al. (2016) He, R.; Fang, C.; Wang, Z.; and McAuley, J. 2016.  Vista: a visually, socially, and temporally-aware model for artistic recommendation.  In _RecSys_ , 309–316.
  * He and McAuley (2016) He, R.; and McAuley, J. 2016.  Fusing similarity models with markov chains for sparse sequential recommendation.  In _ICDM_ , 191–200. IEEE.
  * He and Chua (2017) He, X.; and Chua, T.-S. 2017.  Neural factorization machines for sparse predictive analytics.  In _SIGIR_ , 355–364.
  * Hidasi and Karatzoglou (2018) Hidasi, B.; and Karatzoglou, A. 2018.  Recurrent neural networks with top-k gains for session-based recommendations.  In _CIKM_ , 843–852.
  * Hidasi et al. (2015) Hidasi, B.; Karatzoglou, A.; Baltrunas, L.; and Tikk, D. 2015.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ .
  * Hu, Koren, and Volinsky (2008) Hu, Y.; Koren, Y.; and Volinsky, C. 2008.  Collaborative filtering for implicit feedback datasets.  In _ICDM_ , 263–272. Ieee.
  * Jean et al. (2014) Jean, S.; Cho, K.; Memisevic, R.; and Bengio, Y. 2014.  On using very large target vocabulary for neural machine translation.  _arXiv preprint arXiv:1412.2007_ .
  * Kang and McAuley (2018) Kang, W.-C.; and McAuley, J. 2018.  Self-attentive sequential recommendation.  In _ICDM_ , 197–206. IEEE.
  * Kim et al. (2016) Kim, D.; Park, C.; Oh, J.; Lee, S.; and Yu, H. 2016.  Convolutional matrix factorization for document context-aware recommendation.  In _RecSys_ , 233–240.
  * Koren (2008) Koren, Y. 2008.  Factorization meets the neighborhood: a multifaceted collaborative filtering model.  In _KDD_ , 426–434.
  * Koren (2009) Koren, Y. 2009.  Collaborative filtering with temporal dynamics.  In _KDD_ , 447–456.
  * Koren, Bell, and Volinsky (2009) Koren, Y.; Bell, R.; and Volinsky, C. 2009.  Matrix factorization techniques for recommender systems.  _Computer_ (8): 30–37.
  * Li et al. (2019) Li, C.; Liu, Z.; Wu, M.; Xu, Y.; Zhao, H.; Huang, P.; Kang, G.; Chen, Q.; Li, W.; and Lee, D. L. 2019.  Multi-interest network with dynamic routing for recommendation at Tmall.  In _CIKM_ , 2615–2623.
  * Li et al. (2017) Li, J.; Ren, P.; Chen, Z.; Ren, Z.; Lian, T.; and Ma, J. 2017.  Neural attentive session-based recommendation.  In _CIKM_ , 1419–1428.
  * Li, Wang, and McAuley (2020) Li, J.; Wang, Y.; and McAuley, J. 2020.  Time Interval Aware Self-Attention for Sequential Recommendation.  In _WSDM_ , 322–330.
  * Li and She (2017) Li, X.; and She, J. 2017.  Collaborative variational autoencoder for recommender systems.  In _GKDD_ , 305–314.
  * Liu et al. (2019) Liu, N.; Tan, Q.; Li, Y.; Yang, H.; Zhou, J.; and Hu, X. 2019.  Is a single vector enough? exploring node polysemy for network embedding.  In _KDD_ , 932–940.
  * Lv et al. (2019) Lv, F.; Jin, T.; Yu, C.; Sun, F.; Lin, Q.; Yang, K.; and Ng, W. 2019.  SDM: Sequential deep matching model for online large-scale recommender system.  In _CIKM_ , 2635–2643.
  * Ma et al. (2019) Ma, C.; Ma, L.; Zhang, Y.; Sun, J.; Liu, X.; and Coates, M. 2019.  Memory Augmented Graph Neural Networks for Sequential Recommendation.  _arXiv preprint arXiv:1912.11730_ .
  * Rae et al. (2019) Rae, J. W.; Potapenko, A.; Jayakumar, S. M.; and Lillicrap, T. P. 2019.  Compressive Transformers for Long-Range Sequence Modelling.  _arXiv preprint arXiv:1911.05507_ .
  * Ren et al. (2019) Ren, K.; Qin, J.; Fang, Y.; Zhang, W.; Zheng, L.; Bian, W.; Zhou, G.; Xu, J.; Yu, Y.; Zhu, X.; et al. 2019.  Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction.  In _SIGIR_ , 565–574.
  * Rendle (2010) Rendle, S. 2010.  Factorization machines.  In _ICDM_ , 995–1000. IEEE.
  * Rendle, Freudenthaler, and Schmidt-Thieme (2010) Rendle, S.; Freudenthaler, C.; and Schmidt-Thieme, L. 2010.  Factorizing personalized markov chains for next-basket recommendation.  In _WWW_ , 811–820.
  * Sabour, Frosst, and Hinton (2017) Sabour, S.; Frosst, N.; and Hinton, G. E. 2017.  Dynamic routing between capsules.  In _NIPS_ , 3856–3866.
  * Sarwar et al. (2001) Sarwar, B.; Karypis, G.; Konstan, J.; and Riedl, J. 2001.  Item-based collaborative filtering recommendation algorithms.  In _WWW_ , 285–295. ACM.
  * Schafer et al. (2007) Schafer, J. B.; Frankowski, D.; Herlocker, J.; and Sen, S. 2007.  Collaborative filtering recommender systems.  In _The adaptive web_ , 291–324. Springer.
  * Sedhain et al. (2015) Sedhain, S.; Menon, A. K.; Sanner, S.; and Xie, L. 2015.  Autorec: Autoencoders meet collaborative filtering.  In _WWW_ , 111–112.
  * Sodhani, Chandar, and Bengio (2018) Sodhani, S.; Chandar, S.; and Bengio, Y. 2018.  On training recurrent neural networks for lifelong learning.  _CoRR, abs/1811.07017_ .
  * Su and Khoshgoftaar (2009) Su, X.; and Khoshgoftaar, T. M. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009\.
  * Sukhbaatar et al. (2015) Sukhbaatar, S.; Weston, J.; Fergus, R.; et al. 2015.  End-to-end memory networks.  In _NeurIPS_ , 2440–2448.
  * Sun et al. (2019) Sun, F.; Liu, J.; Wu, J.; Pei, C.; Lin, X.; Ou, W.; and Jiang, P. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer.  In _CIKM_ , 1441–1450.
  * Tan, Liu, and Hu (2019) Tan, Q.; Liu, N.; and Hu, X. 2019.  Deep Representation Learning for Social Network Analysis.  _Frontiers in Big Data_ 2: 2.
  * Tan et al. (2020) Tan, Q.; Liu, N.; Zhao, X.; Yang, H.; Zhou, J.; and Hu, X. 2020.  Learning to Hash with Graph Neural Networks for Recommender Systems.  In _WWW_ , 1988–1998.
  * Tan et al. (2021) Tan, Q.; Zhang, J.; Yao, J.; Liu, N.; Zhou, J.; Yang, H.; and Hu, X. 2021.  Sparse-interest network for sequential recommendation.  In _WSDM_.
  * Tang and Wang (2018) Tang, J.; and Wang, K. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding.  In _WSDM_ , 565–573.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NIPS_ , 5998–6008.
  * Wang et al. (2019a) Wang, B.; Zhao, D.; Lioma, C.; Li, Q.; Zhang, P.; and Simonsen, J. G. 2019a.  Encoding word order in complex embeddings.  _arXiv preprint arXiv:1912.12333_ .
  * Wang et al. (2019b) Wang, X.; He, X.; Wang, M.; Feng, F.; and Chua, T.-S. 2019b.  Neural graph collaborative filtering.  In _SIGIR_ , 165–174.
  * Xue et al. (2017) Xue, H.-J.; Dai, X.; Zhang, J.; Huang, S.; and Chen, J. 2017.  Deep Matrix Factorization Models for Recommender Systems.  In _IJCAI_ , volume 17, 3203–3209. Melbourne, Australia.
  * Yan et al. (2019) Yan, A.; Cheng, S.; Kang, W.-C.; Wan, M.; and McAuley, J. 2019.  CosRec: 2D Convolutional Neural Networks for Sequential Recommendation.  In _CIKM_ , 2173–2176.
  * Ying et al. (2018a) Ying, H.; Zhuang, F.; Zhang, F.; Liu, Y.; Xu, G.; Xie, X.; Xiong, H.; and Wu, J. 2018a.  Sequential recommender system based on hierarchical attention network.  In _IJCAI_.
  * Ying et al. (2018b) Ying, R.; He, R.; Chen, K.; Eksombatchai, P.; Hamilton, W. L.; and Leskovec, J. 2018b.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems.  In _KDD_ , 974–983. ACM.
  * Yu and Koltun (2015) Yu, F.; and Koltun, V. 2015.  Multi-scale context aggregation by dilated convolutions.  _arXiv preprint arXiv:1511.07122_ .
  * Yuan et al. (2019) Yuan, F.; Karatzoglou, A.; Arapakis, I.; Jose, J. M.; and He, X. 2019.  A simple convolutional generative network for next item recommendation.  In _WSDM_ , 582–590.
  * Zhang et al. (2018) Zhang, S.; Tay, Y.; Yao, L.; and Sun, A. 2018.  Next item recommendation with self-attention.  _arXiv preprint arXiv:1808.06414_ .
  * Zhu et al. (2018) Zhu, H.; Li, X.; Zhang, P.; Li, G.; He, J.; Li, H.; and Gai, K. 2018.  Learning tree-based deep model for recommender systems.  In _KDD_ , 1079–1088.