empty

# Secrecy Rate Maximization for Reconfigurable Intelligent Surface Aided
Millimeter Wave System with Low-resolution DACs

Yue Xiu, Jun Zhao, , Wei Sun, , Zhongpei Zhang Yue Xiu, and Zhongpei Zhang are
with the University of Electronic Science and Technology of China, Chengdu,
China (E-mail: xiuyue@std.uestc.edu.cn). Jun Zhao is with Nanyang
Technological University, Singapore (E-mail: junzhao@ntu.edu.sg). Wei Sun is
with the School of Computer Science and Engineering, Northeastern University,
Shenyang 110819, China (E-mail: weisun@stumail.neu.edu.cn). This work was
supported by 1) Guangdong province Key Project of science and
Technology(2018B010115001), 2) Major Project of the Ministry of Industry and
Information Technology of China under Grant (TC190A3WZ-2), Nanyang
Technological University (NTU) Startup Grant, 3) Alibaba-NTU Singapore Joint
Research Institute (JRI), 4) Singapore National Research Foundation (NRF)
under its Strategic Capability Research Centres Funding Initiative: Strategic
Centre for Research in Privacy-Preserving Technologies & Systems (SCRIPTS).

###### Abstract

In this letter, we investigate the secrecy rate of a reconfigurable
intelligent surface (RIS)-aided millimeter-wave (mmWave) system with low-
resolution digital-to-analog converters (LDACs). Compared to the RIS-aided
systems in most existing works, we consider how to alleviate the hardware loss
and improve the secrecy rate by using RIS. In particular, we formulate a
secrecy rate maximization problem with hardware constraints. Then, the RIS
phase shifts and the transmit beamforming are optimized to maximize the
secrecy rate. Due to the nonconvexity of the problem, the formulated problem
is intractable. To handle the problem, an alternating optimization (AO)-based
algorithm is proposed. Specifically, we first use the successive convex
approximation (SCA) method to obtain the transmit beamforming. Then, the
element-wise block coordinate descent (BCD) method is used to obtain the RIS
phase shifts. Numerical results demonstrate that the RIS can mitigate the
hardware loss, and the proposed AO-based algorithm with low complexity
outperformances the baselines.

###### Index Terms:

Reconfigurable intelligent surface, millimeter-wave, low-resolution digital-
to-analog converter, alternating optimization algorithm.

##  I Introduction

Millimeter-wave (mmWave) technologies have played an important role in 5G
communication systems. Compared to the microwave, the mmWave can achieve a
large system capacity and security performance [1]. However, the blockage
issue needs to be tackled before the commercial application of mmWave
technologies. Specifically, mmWave frequencies are susceptible to blockage,
which means mmWave communications are difficult to be applied in urban areas
with dense buildings [2].

To handle this problem, reconfigurable intelligent surface (RIS) is proposed
as a promising technology [3, 4, 5, 6, 7]. In particular, by adjusting the
reflection matrix of RIS, the propagation direction of the transmitted signal
can be changed, mitigating mmWave communication systems’ blockage problem.
Therefore, RIS-aided mmWave systems are getting growing interest from both
academia and industry [8, 9].

Physical layer security is a key technology for solving privacy protection
problems in the physical layer and has been a hot topic in the past decade. To
improve security performance, the basic idea is to improve legitimate users’
achievable rate or deteriorate that of the eavesdroppers (Eve). In a RIS-aided
wireless system, the received signal can be suppressed at Eve while being
boosted at the legitimate user. Thus, deploying the RIS bring a new degree of
freedom (DoF) in the space domain, and the secrecy rate can be further
improved.

Recently, the secure problem of RIS-aided mmWave systems has been investigated
in [10, 11, 12]. Specifically, in [10], the authors maximized RIS-aided
communication systems’ secrecy rate by optimizing the RIS phase shifts and the
transmit beamforming. In [11], a low-complexity iterative algorithm was
proposed to solve the sum secrecy rate maximization problem for RIS-aided
multi-user mmWave system. In [12], the authors optimized the hybrid precoding
at APs and phase shifting at the RIS to maximize RIS-aided mmWave systems’
secrecy rate.

However, the RIS-aided mmWave security systems with hardware limitations have
not been investigated yet. In general, since much more power consumption is
needed for the high-resolution digital-to-analog converts (HDACs) [13], to cut
the hardware cost and power budget, low-resolution digital-to-analog converts
(LDACs) have been widely used in the mmWave system. However, the hardware
imperfections of LDACs and RIS phase noise caused by finite discrete phase
shifts of the RIS are another crucial hardware impairment in RIS-aided systems
[14, 15]. Hence, we focus on a RIS-aided mmWave security system with LDACs and
phase noise in this paper. Specifically, we maximize the secrecy rate under
these hardware constraints by jointly optimizing the transmit beamforming and
the RIS phase shifts. Since the objective function and the feasible set are
non-convex, the formulated problem is intractable. To cope with these
difficulties, we propose an alternating optimization (AO)-based algorithm
based on the successive convex approximation (SCA) method and the element-wise
block coordinate descent (BCD) method.

##  II System Model

Figure 1: A RIS-aided mmWave secure communication system with hardware
limitations.

We consider a RIS-aided massive MIMO mmWave downlink, as shown in Fig. 1,
where the AP with $N_{t}$ antennas and $N_{RF}$ RF chains sends data to the
user and Eve. The user and Eve are equipped with a single antenna,
respectively. Considering $N_{t}$ being typically large, we assume the hybrid
beamforming at the AP with 1-bit DACs to degrade the complexity and infinite-
resolution analog-to-digital converters (ADCs) at the user and Eve, due to
relatively smaller single antenna. Moreover, the RIS is equipped with $N_{r}$
reflection elements. In this paper, we adopt a geometric model for mmWave
channels [8]. $\bm{G}\in\mathbb{C}^{N_{r}\times N_{t}}$ denotes the AP-to-RIS
mmWave channel. $\bm{F}_{RF}\in\mathbb{C}^{N_{t}\times N_{RF}}$ and
$\bm{w}\in\mathbb{C}^{N_{RF}\times 1}$ are the analog beamforming codebook and
digital beamforming vector, respectively. $\bm{F}_{RF}$ adopts the semi-
unitary codebook in [16]. $\mathcal{Q}(\cdot)$ denotes the 1-bit quantizer.

We define the RIS reflection matrix as
$\bm{\Theta}=\mathrm{diag}(\bm{\theta})\in\mathbb{C}^{N_{r}\times N_{r}}$,
where $\bm{\theta}=[\beta e^{j\phi_{1}},\ldots,\beta
e^{j\phi_{N_{r}}}]\in\mathbb{C}^{1\times N_{r}}$,
$\phi_{i}\in[0,2\pi],~{}\forall~{}i=1,\ldots,N_{r}$ and $\beta=1$ denote the
phase shifts and the reflection coefficient of the passive element at the RIS,
respectively. According to [15], due to the hardware limitations, the RIS
phase shift $\phi_{i}$ can only choose its phase shifts from a finite number
of discrete values
$\mathcal{G}=\\{0,\Delta\theta,\ldots,(L-1)\Delta\theta\\}$, where $L$ is the
number of discrete values, and $\Delta\theta=\frac{2\pi}{L}$. Due to
unfavorable propagation conditions (obstacles, buildings), the direct link
from the AP to the user is ignored. Then, the RIS reflects the signal to the
user and Eve, and the received signals at the user and Eve can be expressed as

$\displaystyle\textstyle{y=\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\mathcal{Q}(\bm{w}s)+n},$
(1)

and

$\displaystyle\textstyle{y_{e}=\bm{h}_{e}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\mathcal{Q}(\bm{w}s)+n_{e}},$
(2)

in which $n\sim\mathcal{CN}(0,\sigma^{2})$ and
$n_{e}\sim\mathcal{CN}(0,\sigma_{e}^{2})$ are the additive white Gaussian
noise, $\bm{h}\in\mathbb{C}^{N_{r}\times 1}$ and
$\bm{h}_{e}\in\mathbb{C}^{N_{r}\times 1}$ are the channel matrix between the
RIS and the user, Eve, respectively.

##  III Linear Quantization Models

We consider linear additive quantization noise model (AQNM) schemes [17] for
the non-linear quantization operator $\mathcal{Q}(\cdot)$. The linearization
approximation is expressed as

$\displaystyle\textstyle{\mathcal{Q}(\bm{w}s)\approx
b_{Q}\bm{w}s+\bm{q}_{Q}},$ (3)

where $b_{Q}$ is the weight. And it is expressed as

$\displaystyle\textstyle{b_{Q}=1-\eta_{b}},$ (4)

where $\eta_{b}$ is the distortion factor, which is generally approximated by
$\eta_{b}=\frac{\pi\sqrt{3}}{2}2^{-2b}$ for $b$-bit quantization. The more
accurate value for 1-bit quantization is $\eta_{b}\approx 0.3634$ [17]. In
(3), $\bm{q}_{Q}$ stands for the quantization distortion with the following
covariance

$\displaystyle\textstyle{\bm{A}_{Q}=b_{Q}(1-b_{Q})\mathrm{diag}(\bm{w}\bm{w}^{H})}.$
(5)

Then, the achievable rate at the user can be expressed as

$\displaystyle\textstyle{R=\log_{2}\left(1+\frac{|b_{Q}\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\bm{w}|^{2}}{b_{Q}(1-b_{Q})\|\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})\|^{2}+\sigma^{2}}\right)}.$
(6)

The achievable rate at the Eve can be expressed as

$\displaystyle\textstyle{R_{e}=\log_{2}\left(1+\frac{|b_{Q}\bm{h}_{e}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\bm{w}|^{2}}{b_{Q}(1-b_{Q})\|\bm{h}_{e}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})\|^{2}+\sigma_{e}^{2}}\right)}.$
(7)

The system secrecy rate can be written as

$\displaystyle\textstyle{R_{s}=[R-R_{e}]^{+}},$ (8)

where $[x]^{+}=\max(0,x)$. Then, the problem is formulated as

$\displaystyle\textstyle{\max\limits_{\bm{\theta},\bm{w}}}~{}$
$\displaystyle\textstyle{R_{s}}$ (9a) s.t.
$\displaystyle\textstyle{\textstyle{|\theta_{i}|=1}}$ (9b)
$\displaystyle\textstyle{\|\bm{F}_{RF}\mathcal{Q}(\bm{w}s)\|^{2}\leq P},$ (9c)

where (9b) denotes the unit-modulus constraint. (9c) is the power constraint,
and $P$ is the maximum transmit power. Incorporating (9c) and (5), the
transmit power constraint is rewritten as

$\displaystyle\textstyle{\|\bm{F}_{RF}\mathcal{Q}(\bm{w}s)\|^{2}}$
$\displaystyle\textstyle{=\|b_{Q}\bm{F}_{RF}\bm{w}\|^{2}+\mathrm{tr}(\bm{F}_{RF}\bm{A}_{Q}\bm{F}_{RF})},$
$\displaystyle\textstyle{=\|b_{Q}\bm{w}\|^{2}+\mathrm{tr}(\bm{A}_{Q})},$ (10)

which follows from the semi-unitary $\bm{F}_{RF}$. Therefore, problem (9) is
rewritten as

$\displaystyle\textstyle{\max\limits_{\bm{\Theta},\bm{w}}}~{}$
$\displaystyle\textstyle{R_{s}}$ (11a) s.t.
$\displaystyle\textstyle{\text{(9b)},}$ (11b)
$\displaystyle\textstyle{\|b_{Q}\bm{w}\|^{2}+\mathrm{tr}(\bm{A}_{Q})\leq P}.$
(11c)

##  IV Alternating Optimization Algorithm

Since the problem in (11) is non-convex, we adopt an AO-based algorithm to
optimize $\bm{w}$ and $\bm{\Theta}$, respectively. Specifically, we first
optimize $\bm{w}$ with fixed $\bm{\Theta}$, then we fix $\bm{w}$ and optimize
$\bm{\Theta}$.

###  IV-A Digital Beamforming Optimization

Under given $\bm{\Theta}$, problem (11) is rewritten as

$\displaystyle\textstyle{\max\limits_{\bm{w}}}~{}$
$\displaystyle\textstyle{\log_{2}\left(1+\frac{|\bm{D}\bm{w}|^{2}}{\omega}\right)-\log_{2}\left(1+\frac{|\bm{D}_{e}\bm{w}|^{2}}{\omega_{e}}\right)}$
(12a) s.t. $\displaystyle\textstyle{\text{(11c)},}$ (12b)

where

$\displaystyle\textstyle{\bm{D}=b_{Q}\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF},\bm{D}_{e}=b_{Q}\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}},$
$\displaystyle\textstyle{\omega=|\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\bm{A}_{Q}\mathrm{diag}(\bm{w})|^{2}+\sigma^{2}},$
$\displaystyle\textstyle{\omega_{e}=|\bm{h}_{e}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\bm{A}_{Q}\mathrm{diag}(\bm{w})|^{2}+\sigma_{e}^{2}}.$
(13)

Then, to deal with the non-convex objective function in (12a), we introduce an
auxiliary variable $t$ to rewrite (12) as

$\displaystyle\textstyle{\max\limits_{\bm{w}}}~{}$
$\displaystyle\textstyle{\log_{2}\left(1+\frac{|\bm{D}\bm{w}|^{2}}{\omega}\right)-t}$
(14a) s.t. $\displaystyle\textstyle{\text{(11c)}},$ (14b)
$\displaystyle\textstyle{t=\log_{2}\left(1+\frac{|\bm{D}_{e}\bm{w}|^{2}}{\omega_{e}}\right)}.$
(14c)

It is not difficult to find that the objective function is still non-convex.
To handle the non-convex objective function, we use $\rho$ to further
transform (14a) as

$\displaystyle\textstyle{\max\limits_{\bm{w},\rho}\log_{2}\left(1+\rho\right)-t}$
(15)

where

$\displaystyle\textstyle{\rho\leq\frac{|\bm{D}\bm{w}|^{2}}{\omega}}.$ (16)

According to the Schur complement in [18], we have

$\displaystyle\textstyle{\left[\begin{matrix}1&z\\\
z&|\bm{D}\bm{w}|^{2}\end{matrix}\right]\succeq\bm{0}},\textstyle{~{}~{}\rho\leq\frac{z^{2}}{\omega}},$
(17)

where $z$ is an auxiliary variable. Due to $\rho\leq\frac{z^{2}}{\omega}$ is
non-convex with respect to $\rho$ and $\omega$, we use the SCA method based on
the first-order Taylor expansion [19] to tackle
$\rho\leq\frac{z^{2}}{\omega}$. In particular, for any fixed points
$(\bar{z},\bar{\omega})$, we have

$\displaystyle\textstyle{\frac{z^{2}}{\omega}\geq\frac{2\bar{z}}{\bar{\omega}}z-\frac{\bar{z}^{2}}{\omega^{2}}\omega\geq\rho}.$
(18)

By applying the concept of the SCA [19],[20], we iteratively update $\bar{z}$
and $\bar{\omega}$ in the $n$-th iteration as

$\displaystyle\textstyle{\bar{\omega}^{(n)}=\omega^{(n-1)},\bar{z}^{(n)}=z^{(n-1)}}.$
(19)

Now, we handle the non-convex constraint in (14c), similarly, we use the Schur
complement to deal with (14c) and (14c) can be transformed into the following
equivalent forms.

$\displaystyle\textstyle{\left[\begin{matrix}2^{t}-1&r\\\
r&\omega_{e}\end{matrix}\right]\succeq\bm{0}},$ (20)
$\displaystyle\textstyle{r^{2}\geq(2^{t}-1)\omega_{e}},$ (21)

and

$\displaystyle\textstyle{r^{2}-|\bm{D}_{e}\bm{w}|^{2}=0}.$ (22)

In order to deal with the bilinear function on the left-hand side of (21), the
SCA method based on the arithmetic geometric mean (AGM) inequality is adopted,
so (21) can be rewritten as

$\displaystyle\textstyle{\frac{1}{2}\left((\omega_{e}\eta)^{2}+(\frac{2^{t}-1}{\eta})^{2}\right)-\bar{r}(r-\bar{r})\leq
0},$ (23)

where $\eta$ is a feasible point. To tighten the upper bound, $\eta$ is
iteratively updated. The update formulation in the $n$-th iteration is
expressed as

$\displaystyle\textstyle{\eta^{(n)}=\sqrt{\frac{2^{t^{(n-1)}}-1}{\rho^{(n-1)}}}}.$
(24)

Then, we deal with non-convex constraint (22) and use the SCA method to
transform (22) as the following convex constraints.

$\displaystyle\textstyle{|\bm{D}_{e}\bm{w}|^{2}-\bar{r}(r-\bar{r})\leq
0},~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
$\displaystyle\textstyle{r^{2}-|\bm{D}_{e}\bar{\bm{w}}|^{2}-2\mathrm{Re}(\bar{\bm{w}}^{H}\bm{D}_{e}^{H}\bm{D}_{e}(\bm{w}-\bar{\bm{w}}))\leq
0.}$ (25)

The non-convex constraint in (11c) can be rewritten as the following form by
using an similar manner with (25).

$\displaystyle(|b_{Q}\bar{\bm{w}}|^{2}-2\mathrm{Re}(b_{Q}\bar{\bm{w}}^{H}(\bm{w}-\bar{\bm{w}})))+\mathrm{tr}(\bm{A_{Q}})\leq
P.$ (26)

Now, the problem in (12) is transformed to the following convex approximation
problem.

$\displaystyle\textstyle{\max\limits_{t,r,\omega,\rho,z,\bm{w}}}~{}$
$\displaystyle\textstyle{\log_{2}\left(1+\rho\right)-t}$ (27a) s.t.
$\displaystyle\left[\begin{matrix}1&z\\\
z&|\bm{D}\bm{w}|^{2}\end{matrix}\right]\succeq\bm{0},$ (27b)
$\displaystyle\textstyle{\text{(18)},\text{(20)},\text{(23)},\text{(25)},\text{(26)}.}$
(27c)

The SCA-based algorithm is summarized in Algorithm 1 for solving (12).

1 Initialization: $\bar{r}^{(0)}$, $\bar{\omega}^{(0)}$, $\bar{z}^{(0)}$,
$\bar{\bm{w}}^{(0)}$.

2 Repeat

3 Update $\\{\bm{w}^{(n)},r^{(n)},\rho^{(n)},\omega^{(n)},z^{(n)},t^{(n)}\\}$
with fixed $\bar{r}$, $\bar{\omega}$, $\bar{t}$, $\bar{\bm{w}}$ by solving
(27).

4 Update $\eta^{(n+1)}$, $\bar{\omega}^{(n+1)}$, $\bar{z}^{(n+1)}$ based on
(19) and (24).

5 Update $n=n+1$.

6 Until Convergence.

7 Output: $\bm{w}^{*}$.

Algorithm 1 SCA-based algorithm for problem (12).

###  IV-B Discrete RIS Phase Shifts Optimization

Substituting the transmit beamforming vector $\bm{w}$ obtained in the previous
section into problem (11). Then the sub-problem of phase shifts matrix can be
rewritten at the top of next page.

Since the variable $\phi_{i}$ belongs to a finite number of values, problem
(28) can be resolved by using the exhaustive search method. However, feasible
set $\mathcal{G}$ is large and the complexity of exhaustive search method is
very high. To reduce complexity, an element-wise BCD algorithm is proposed.

To obtain the closed form solution of phase shifts, we employ the element-wise
BCD algorithm [19] to solve this problem. We assume $\phi_{i}$ as one block in
the BCD and iteratively derive the continuous solutions of $\phi_{i}$ by using
Theorem 1.

######  Theorem 1.

There exists solution $\phi_{i}^{*}$ to maximize the value of (28) by solving
the following equation

$\displaystyle\textstyle{\frac{(\mu_{i}-\bar{\mu}_{i})t-\tilde{\mu}_{i}}{\mu_{i}(1+t^{2})+\bar{\mu}_{i}(1-t^{2})-\tilde{\mu}_{i}2t}-\frac{(\eta_{i}-\bar{\eta}_{i})t-\tilde{\eta}_{i}}{\eta_{i}(1+t^{2})+\bar{\eta}_{i}(1-t^{2})-\tilde{\eta}_{i}2t}}$
$\displaystyle\textstyle{+\frac{(\lambda_{i}-\bar{\lambda}_{i})t-\tilde{\lambda}_{i}}{\lambda_{i}(1+t^{2})+\bar{\lambda}_{i}(1-t^{2})-\tilde{\lambda}_{i}2t}-\frac{(\rho_{i}-\bar{\rho}_{i})t-\tilde{\rho}_{i}}{\rho_{i}(1+t^{2})+\bar{\rho}_{i}(1-t^{2})-\tilde{\rho}_{i}2t}=0}.$
(29)

We use one dimension search method in [21] to solve the equation in (29).
$\phi_{i}^{*}$ can be expressed as

$\displaystyle\phi_{i}^{*}=2\arctan(t).$ (30)

The proof is given in Appendix A.

According to Theorem 1, the discrete solution $\bar{\phi}_{i}^{*}$ can be
calculated by the following formulation

$\displaystyle\bar{\phi}_{i}^{*}=\arg\min_{\phi_{i}\in\mathcal{G}}|\phi_{i}^{*}-\phi_{i}|.$
(31)

The algorithm based on element-wise BCD can be summarized in Algorithm 2.

$\displaystyle\textstyle{\max\limits_{\bm{\Theta}}}~{}$
$\displaystyle\textstyle{\frac{\left(\frac{b_{Q}(1-b_{Q})\|\bm{\theta}^{T}\mathrm{diag}(\bm{h}^{H})\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})\|^{2}+\sigma^{2}+|b_{Q}\bm{h}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\bm{w}|^{2}}{b_{Q}(1-b_{Q})\|\bm{\theta}^{T}\mathrm{diag}(\bm{h}^{H})\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})\|^{2}+\sigma^{2}}\right)}{\left(\frac{b_{Q}(1-b_{Q})\|\bm{\theta}^{T}\mathrm{diag}(\bm{h}_{e}^{H})\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})\|^{2}+\sigma^{2}+|b_{Q}\bm{h}_{e}^{H}\bm{\Theta}\bm{G}\bm{F}_{RF}\bm{w}|^{2}}{b_{Q}(1-b_{Q})\|\bm{\theta}^{T}\mathrm{diag}(\bm{h}_{e}^{H})\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})\|^{2}+\sigma_{e}^{2}}\right)}}$
(28a) s.t. (9b) (28b)

1 Initialization: $t=0$, $\bm{\Theta}^{0}$.

2 Repeat:

3 for: $i=1,\ldots,N_{r}$

4 Calculate $\phi_{i}^{t}$ based on Theorem 1.

5 End:

6 Set $t=t+1$

7 Until: Convergence.

Output: $\bm{\Theta}^{*}$

Algorithm 2 Element-wise BCD-based algorithm for problem (28).

Based on the above analysis, we can obtain the AO-based algorithm for solving
problem (9). Following the results in [19], since each sub-algorithm converges
to a local optimal solution, we can guarantee that the AO-based algorithm can
converge to a local optimal solution.

1 Initialization: $\bm{w}^{(0)}$, $\bm{\Theta}^{(0)}$.

2 Repeat

3 Update $\bm{w}^{(j)}$ by using Algorithm 1.

4 Update $\bm{\Theta}^{(j)}$ by using Algorithm 2.

5 Update $j=j+1$.

6 Until Convergence.

7 Output: $\bm{w}^{*}$, $\bm{\Theta}^{*}$.

Algorithm 3 AO-based Algorithm for problem (9).

###  IV-C Complexity Analysis

The complexity of the proposed method is about
$\mathcal{O}(N_{r}(N_{r}N_{t}+L_{P})+N_{t}^{2})$, which depends on the
computational complexity of $\phi_{i}$ and $\bm{w}$. We compare the complexity
of these algorithms in TABLE 1, it is not difficult to find that the AO-based
algorithm has the lowest complexity.

TABLE I: Complexity Comparison of Algorithms  Symbol |  Total Complexity
---|---
Proposed AO-based algorithm | $\mathcal{O}(N_{r}(N_{r}N_{t}+L_{P})+N_{t}^{2})$
the exhaustive method | $\mathcal{O}(N_{r}^{L_{P}+1}+N_{r}^{2}+N_{r}N_{t})$
SDP-based algorithm [22] | $\mathcal{O}(\zeta N_{r}^{8}+N_{t}^{2})$

##  V Numerical Results

As shown in Fig.1, we consider a RIS-aided mmWave system with hardware
limitations, where AP’s coordinate is $(0,0,0)$ m and IRS is located at
$(0,60,20)$ m. While the user and the Eve are located at $(5,60,0)$ m and
$(5,80,0)$ m, respectively. We set $N_{t}=64$, $N_{r}=16$, $N_{RF}=8$, $b=1$,
$\sigma^{2}=-110$ dBm. The mmWave channels from the AP to the RIS, the RIS to
the $k$th user are respectively expressed as

$\displaystyle\textstyle{\bm{G}=\sqrt{\frac{1}{\beta
L_{1}}}\sum_{l=0}^{L_{1}-1}\alpha\bm{a}_{T}(N_{T},\theta)\bm{a}_{R}^{T}(N_{r},\varphi,\phi),}$
(32)
$\displaystyle\textstyle{\bm{h}_{k}=\sqrt{\frac{1}{\hat{\beta}_{k}L_{k}}}\sum_{l=0}^{L_{k}-1}\hat{\alpha}_{k}\bm{a}_{R}(N_{r},\vartheta_{k}),~{}k\in\\{user,Eve\\},}$
(33)

where $\beta$ and $\hat{\beta}_{k}$ denote the large-scale fading
coefficients. They are generated by

$\displaystyle 72+29.2\log_{10}d+\zeta,$ (34)

where $d$ denotes the signal propagation distance,
$\zeta\sim\mathcal{CN}(0,1)$ is the log-normal shadowing, $\alpha$ and
$\hat{\alpha}_{k}$ denote the small-scale fading coefficients which follow
$\mathcal{CN}(0,1)$ [15]. $\bm{a}_{T}(\cdot)$ and $\bm{a}_{R}(\cdot)$
represent array steering vectors at the AP and the RIS, respectively. Three
schemes with fixed quantization bits for each DACs are considered for
comparison:

  * •

MRT-BCD: In this scheme, the transmit beamforming is designed based on maximum
ratio transmission (MRT). Then, the RIS phase shifts is obtained by using BCD
algorithm.

  * •

NO-RIS: In this scheme, the mmWave do not use the RIS to assist communication.
Moreover, the transmit beamforming is optimized by using Algorithm 1.

  * •

Upper Bound: In this scheme, we consider the RIS-aided mmWave system without
hardware limitations as an upper bound. Moreover, the transmit beamforming
vector and the RIS phase shifts are optimized by using Algorithm 1 and
Algorithm 2, respectively.

(a) Convergence of Algorithm 3

(b) Secrecy rate versus $N_{r}$

(c) Secrecy rate versus $b$

The convergence behavior of the proposed AO-based algorithm under different
iteration times is given in Fig.2. We observe that the secrecy rate increases
monotonically with the increase of the number of iterations. In addition, the
algorithm converges quickly, and can achieve a high security rate at the 5th
iteration.

Fig. 3 shows the secrecy rate under different number of reflecting elements of
the RIS. As expected, compared with the MRT-BCD and NO-RIS schemes, the
proposed algorithm can achieve the best performance. Moreover, although the
resolution of DACs in the upper bound scheme is higher than that of the
proposed scheme, we find that as the number of reflecting elements of RIS
increases, both the proposed scheme and the upper bound scheme increase at the
same time. This finding validates the feasibility of using RIS mitigates the
influence of hardware limitations.

Fig. 4 depicts the secrecy rate versus quantization bits under different
schemes. Compared with the MRT-BCD scheme and NO-RIS scheme, the proposed AO-
based algorithm can achieve the best security performance. It is not difficult
to find that the schemes with the RIS can outperformance the NO-RIS scheme.
From the perspective of maximizing the secrecy rate, when the system is
assisted by RIS, there is no need to deploy high-resolution DACs. This result
demonstrates that the RIS can suppress the hardware loss as well.

##  VI Conclusion

The secrecy rate maximization problem for mmWave communications with hardware
limitations at the AP and RIS was investigated in this paper. The RIS phase
shifts, transmit beamforming vector were jointly optimized to maximize the
secrecy rate under the hardware constraint and unit-modulus constraints. To
solve this problem, we proposed an AO-based algorithm. Numerical results have
shown that the proposed AO-based algorithm outperforms conventional schemes in
terms of secrecy rate. Moreover, the numerical results also show that when the
mmWave system is equipped with RIS, the mmWave system does not need to equip
with excessive RF chains and high-resolution DACs. In this paper, the CSI is
assumed as perfect. In fact, the CSI is difficult to obtain due to the passive
feature of the RIS. Therefore, we will design a channel estimation scheme
111We carefully studied the channel estimation scheme in [23, 24]. They are
useful to improve our future work, and we will combine the channel estimation
results to design the beamforming for RIS-aided mmWave system.in the future
works, then we use the proposed algorithm in this paper to design beamforming
for maximizing secrecy rate in a RIS-aided mmWave system.

##  Appendix A The proof of theorem 1

We assume $\phi_{i}$ as one block of BCD, (28a) can be rewritten as

$\displaystyle\textstyle{R_{s}=\left(\frac{|\bm{\theta}^{T}\bm{c}|^{2}+\|\bm{\theta}^{T}\bm{A}\|^{2}+\sigma^{2}}{|\bm{\theta}^{T}\bm{d}|^{2}+\|\bm{\theta}^{T}\bm{B}\|^{2}+\sigma_{e}^{2}}\right)\left(\frac{\|\bm{\theta}^{T}\bm{B}\|^{2}+\sigma_{e}^{2}}{\|\bm{\theta}^{T}\bm{A}\|^{2}+\sigma^{2}}\right)},$
(35)

where

$\displaystyle\textstyle{\bm{c}=b_{Q}\mathrm{diag}(\bm{h}^{H})\bm{G}\bm{F}_{RF}\bm{w},\bm{d}=b_{Q}\mathrm{diag}(\bm{h}_{e}^{H})\bm{G}\bm{F}_{RF}\bm{w}},$
$\displaystyle\textstyle{\bm{A}=b_{Q}(1-b_{Q})\mathrm{diag}(\bm{h}^{H})\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})},$
$\displaystyle\textstyle{\bm{B}=b_{Q}(1-b_{Q})\mathrm{diag}(\bm{h}_{e}^{H})\bm{G}\bm{F}_{RF}\mathrm{diag}(\bm{w})}.$
(36)

To simplify (35), we expanse $|\bm{\theta}^{T}\bm{c}|^{2}$,
$|\bm{\theta}^{T}\bm{d}|^{2}$, $|\bm{\theta}^{T}\bm{A}|^{2}$, and
$|\bm{\theta}^{T}\bm{B}|^{2}$ as

$\displaystyle\textstyle{|\bm{\theta}^{T}\bm{c}|^{2}=|e^{j\phi_{i}}c_{i}+p_{i}|^{2}=|c_{i}|^{2}+|p_{i}|^{2}}$
$\displaystyle\textstyle{+(\mathrm{Re}\\{c_{i}\\}\mathrm{Re}\\{p_{i}\\}+\mathrm{Im}\\{c_{i}\\}\mathrm{Im}\\{p_{i}\\})\cos(\phi_{i})}$
$\displaystyle\textstyle{-(\mathrm{Re}\\{c_{i}\\}\mathrm{Im}\\{p_{i}\\}+\mathrm{Im}\\{c_{i}\\}\mathrm{Re}\\{p_{i}\\})\sin(\phi_{i})},$
$\displaystyle\textstyle{|\bm{\theta}^{T}\bm{d}|^{2}=|e^{j\phi_{i}}\bar{c}_{i}+\bar{p}_{i}|^{2}=|\bar{c}_{i}|^{2}+|\bar{p}_{i}|^{2}}$
$\displaystyle\textstyle{+(\mathrm{Re}\\{\bar{c}_{i}\\}\mathrm{Re}\\{\bar{p}_{i}\\}+\mathrm{Im}\\{\bar{c}_{i}\\}\mathrm{Im}\\{\bar{p}_{i}\\})\cos(\phi_{i})}$
$\displaystyle\textstyle{-(\mathrm{Re}\\{\bar{c}_{i}\\}\mathrm{Im}\\{\bar{p}_{i}\\}+\mathrm{Im}\\{\bar{c}_{i}\\}\mathrm{Re}\\{\bar{p}_{i}\\})\sin(\phi_{i})},$
$\displaystyle\textstyle{|\bm{\theta}^{T}\bm{A}|^{2}=\sum_{k=1}^{N_{t}}|e^{j\phi_{i}}q_{i,k}+v_{k}|^{2}=\sum_{k=1}^{N_{t}}|q_{i,k}|^{2}+\sum_{k=1}^{N_{t}}|v_{k}|^{2}}$
$\displaystyle\textstyle{+\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{q_{i,k}\\}\mathrm{Re}\\{v_{k}\\}+\mathrm{Im}\\{q_{i,k}\\}\mathrm{Im}\\{v_{k}\\})\cos(\phi_{i})}$
$\displaystyle\textstyle{-\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{q_{i,k}\\}\mathrm{Im}\\{v_{k}\\}+\mathrm{Im}\\{q_{i,k}\\}\mathrm{Re}\\{v_{k}\\})\sin(\phi_{i})},$
$\displaystyle\textstyle{|\bm{\theta}^{T}\bm{B}|^{2}=\sum_{k=1}^{N_{t}}|e^{j\phi_{i}}\bar{q}_{i,k}+\bar{v}_{k}|^{2}=\sum_{k=1}^{N_{t}}|\bar{q}_{i,k}|^{2}+\sum_{k=1}^{N_{t}}|\bar{v}_{k}|^{2}}$
$\displaystyle\textstyle{+\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{\bar{q}_{i,k}\\}\mathrm{Re}\\{\bar{v}_{k}\\}+\mathrm{Im}\\{\bar{q}_{i,k}\\}\mathrm{Im}\\{\bar{v}_{k}\\})\cos(\phi_{i})}$
$\displaystyle\textstyle{-\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{\bar{q}_{i,k}\\}\mathrm{Im}\\{\bar{v}_{k}\\}+\mathrm{Im}\\{\bar{q}_{i,k}\\}\mathrm{Re}\\{\bar{v}_{k}\\})\sin(\phi_{i})},$
(37)

where

$\displaystyle\textstyle{p_{i}=\sum_{j\neq
i}^{N_{r}}e^{j\phi_{j}}c_{j},\bar{p}_{i}=\sum_{j\neq
i}^{N_{r}}e^{j\phi_{j}}d_{j},\bar{c}_{i}=d_{i}},$
$\displaystyle\textstyle{q_{i,k}=a_{i,k},\bar{q}_{i,k}=b_{i,k}}$
$\displaystyle\textstyle{v_{k}=\sum_{j\neq
1}^{N_{r}}e^{j\phi_{j}}a_{k,j},\bar{v}_{k}=\sum_{j\neq
1}^{N_{r}}e^{j\phi_{j}}b_{k,j}}$ (38)

We continue to simplify (35) by introducing auxiliary variables $\mu_{i}$,
$\bar{\mu}$, $\eta_{i}$, $\bar{\eta}_{i}$, $\lambda_{i}$, $\bar{\lambda}_{i}$,
$\rho_{i}$, and $\bar{\rho}_{i}$.

$\displaystyle\textstyle{R_{s}=\left(\frac{\mu_{i}+\bar{\mu}_{i}\cos(\phi_{i})-\tilde{\mu}_{i}\sin(\phi_{i})}{\eta_{i}+\bar{\eta}_{i}\cos(\phi_{i})-\tilde{\eta}_{i}\sin(\phi_{i})}\right)\left(\frac{\lambda_{i}+\bar{\lambda}_{i}\cos(\phi_{i})-\tilde{\lambda}_{i}\sin(\phi_{i})}{\rho_{i}+\bar{\rho}_{i}\cos(\phi_{i})-\tilde{\rho}_{i}\sin(\phi_{i})}\right)},$
(39)

where

$\displaystyle\textstyle{\mu_{i}=|c_{i}|^{2}+|p_{i}|^{2}+\sum_{k=1}^{N_{t}}|q_{i,k}|^{2}+\sum_{k=1}^{N_{t}}|v_{k}|^{2}+\sigma^{2}},$
$\displaystyle\textstyle{\bar{\mu}_{i}=(\mathrm{Re}\\{c_{i}\\}\mathrm{Re}\\{p_{i}\\}+\mathrm{Im}\\{c_{i}\\}\mathrm{Im}\\{p_{i}\\})}$
$\displaystyle\textstyle{+\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{q_{i,k}\\}\mathrm{Re}\\{v_{k}\\}+\mathrm{Im}\\{q_{i,k}\\}\mathrm{Im}\\{v_{k}\\})},$
$\displaystyle\textstyle{\tilde{\mu}_{i}=\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{q_{i,k}\\}\mathrm{Im}\\{v_{k}\\}+\mathrm{Im}\\{q_{i,k}\\}\mathrm{Re}\\{v_{k}\\})},$
$\displaystyle\textstyle{\eta_{i}=|\bar{c}_{i}|^{2}+|\bar{p}_{i}|^{2}+\sum_{k=1}^{N_{t}}|\bar{q}_{i,k}|^{2}+\sum_{k=1}^{N_{t}}|\bar{v}_{k}|^{2}+\sigma^{2}},$
$\displaystyle\textstyle{\bar{\eta}_{i}=(\mathrm{Re}\\{\bar{c}_{i}\\}\mathrm{Re}\\{\bar{p}_{i}\\}+\mathrm{Im}\\{\bar{c}_{i}\\}\mathrm{Im}\\{\bar{p}_{i}\\})}$
$\displaystyle\textstyle{+\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{\bar{q}_{i,k}\\}\mathrm{Re}\\{\bar{v}_{k}\\}+\mathrm{Im}\\{\bar{q}_{i,k}\\}\mathrm{Im}\\{\bar{v}_{k}\\})},$
$\displaystyle\textstyle{\tilde{\eta}_{i}=\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{\bar{q}_{i,k}\\}\mathrm{Im}\\{\bar{v}_{k}\\}+\mathrm{Im}\\{\bar{q}_{i,k}\\}\mathrm{Re}\\{\bar{v}_{k}\\})},$
$\displaystyle\textstyle{\rho_{i}=\sum_{k=1}^{N_{t}}|q_{i,k}|^{2}+\sum_{k=1}^{N_{t}}|v_{k}|^{2}+\sigma^{2}},$
$\displaystyle\textstyle{\bar{\rho}_{i}=\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{q_{i,k}\\}\mathrm{Re}\\{v_{k}\\}+\mathrm{Im}\\{q_{i,k}\\}\mathrm{Im}\\{v_{k}\\})},$
$\displaystyle\textstyle{\tilde{\rho}_{i}=\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{q_{i,k}\\}\mathrm{Im}\\{v_{k}\\}+\mathrm{Im}\\{q_{i,k}\\}\mathrm{Re}\\{v_{k}\\}},$
$\displaystyle\textstyle{\lambda_{i}=\sum_{k=1}^{N_{t}}|\bar{q}_{i,k}|^{2}+\sum_{k=1}^{N_{t}}|\bar{v}_{k}|^{2}+\sigma^{2}},$
$\displaystyle\textstyle{\bar{\lambda}_{i}=\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{\bar{q}_{i,k}\\}\mathrm{Re}\\{\bar{v}_{k}\\}+\mathrm{Im}\\{\bar{q}_{i,k}\\}\mathrm{Im}\\{\bar{v}_{k}\\})},$
$\displaystyle\textstyle{\tilde{\lambda}_{i}=\sum_{k=1}^{N_{t}}(\mathrm{Re}\\{\bar{q}_{i,k}\\}\mathrm{Im}\\{\bar{v}_{k}\\}+\mathrm{Im}\\{\bar{q}_{i,k}\\}\mathrm{Re}\\{\bar{v}_{k}\\}}.$

Let $\tan(\frac{\phi_{i}}{2})=t$, we have $\sin(\phi_{i})=\frac{2t}{1+t^{2}}$
and $\cos(\phi_{i})=\frac{1-t^{2}}{1+t^{2}}$. (39) is rewritten as

$\displaystyle\textstyle{R_{s}=\left(\frac{\mu_{i}(1+t^{2})+\bar{\mu}_{i}(1-t^{2})-\tilde{\mu}_{i}2t}{\eta_{i}(1+t^{2})+\bar{\eta}_{i}(1-t^{2})-\tilde{\eta}_{i}2t}\right)}\textstyle{\left(\frac{\lambda_{i}(1+t^{2})+\bar{\lambda}_{i}(1-t^{2})-\tilde{\lambda}_{i}2t}{\rho_{i}(1+t^{2})+\bar{\rho}_{i}(1-t^{2})-\tilde{\rho}_{i}2t}\right)}.$
(41)

When the derivative of (41) is $0$, the optimal solution $\phi_{i}^{*}$ can be
determined by the following equation

$\displaystyle\textstyle{\frac{(\mu_{i}-\bar{\mu}_{i})t-\tilde{\mu}_{i}}{\mu_{i}(1+t^{2})+\bar{\mu}_{i}(1-t^{2})-\tilde{\mu}_{i}2t}-\frac{(\eta_{i}-\bar{\eta}_{i})t-\tilde{\eta}_{i}}{\eta_{i}(1+t^{2})+\bar{\eta}_{i}(1-t^{2})-\tilde{\eta}_{i}2t}}$
$\displaystyle\textstyle{+\frac{(\lambda_{i}-\bar{\lambda}_{i})t-\tilde{\lambda}_{i}}{\lambda_{i}(1+t^{2})+\bar{\lambda}_{i}(1-t^{2})-\tilde{\lambda}_{i}2t}-\frac{(\rho_{i}-\bar{\rho}_{i})t-\tilde{\rho}_{i}}{\rho_{i}(1+t^{2})+\bar{\rho}_{i}(1-t^{2})-\tilde{\rho}_{i}2t}=0},$
(42)

Then, the equation (42) can be solved using a one dimension search in [21].
Finally, the Theorem 1 is proved.

## References

  * [1] M. Di Renzo, A. Zappone, M. Debbah, et al, “Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and road ahead,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 11, pp. 2450-2525, Nov. 2020.
  * [2] S. Zhou, W. Xu, K. Wang, et al, “Spectral and energy efficiency of IRS-assisted MISO communication with hardware impairments,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 9, pp. 1366-1369, Sept. 2020.
  * [3] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 11, pp. 5394–5409, Nov. 2019.
  * [4] Q. Wu and R. Zhang, “Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network,” _IEEE Commun. Mag._ , vol. 58, no. 1, pp. 106–112, Jan. 2020.
  * [5] Q. Wu, S. Zhang, B. Zheng, et al, “Intelligent reflecting surface aided wireless communications: A tutorial,”arXiv preprint arXiv: 2007.02795, 2020.
  * [6] S. Abeywickrama, R. Zhang, Q. Wu, et al, “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” _IEEE Trans. Wireless Commun._ , vol. 68, no. 9, pp. 5849–5863, Jun. 2020.
  * [7] R. Karasik, O. Simeone, M. Di Renzo, et al, “Beyond max-SNR: Joint encoding for reconfigurable intelligent surfaces,”arXiv preprint arXiv: 1911.09443, 2019.
  * [8] Y. Xiu, Y. Zhao, Y. Liu, et al, “IRS-assisted millimeter wave communications: Joint power allocation and beamforming design,”arXiv preprint arXiv:2001.07467, 2020.
  * [9] C. Pradhan, A. Li, L. Song, et al, “Hybrid precoding design for reconfigurable intelligent surface aided mmwave communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 7, pp. 1041–1045, Jul. 2020.
  * [10] H. Shen, W. Xu, S. Gong, et al, “Secrecy rate maximization for intelligent reflecting surface assisted multi-antenna communications,” _IEEE Commun. Lett._ , vol. 9, no. 23, pp. 1488-1492, Jul. 2019.
  * [11] J. Chen, Y. Liang, Y. Pei, et al, “Intelligent reflecting surface: A programmable wireless environment for physical layer security,” _IEEE Access_ , vol. 7, pp. 82599-82612, 2019.
  * [12] X. Lu, W. Yang, X. Guan, et al, “Robust and secure beamforming for intelligent reflecting surface aided mmWave MISO systems,” arXiv preprint arXiv: 2003.11195, 2020.
  * [13] J. Zhang, L. Dai, Z. He, et al., “Performance analysis of mixed-ADC massive MIMO systems over Rician fading channels,” _IEEE J. Sel. Areas Commun._ , vol. 35, no. 6, pp. 1327-1338, Jun. 2017.
  * [14] S. Zhou, W. Xu, K. Wang, et al, “Spectral and energy efficiency of IRS-assisted MISO communication with hardware impairments,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 9, pp. 1366-1369, Sept. 2020.
  * [15] K. Zhi, C. Pan, H. Ren, et al. “Uplink achievable rate of intelligent reflecting surface-aided millimeter-wave communications with low-resolution ADC and phase noise,” arXiv preprint arXiv: 2008.00437, 2020.
  * [16] J. Mo, A. Alkhateeb, S. Abu-Surra, et al., “Hybrid architectures with few-bit ADC receivers: Achievable rates and energy-rate tradeoffs,” _IEEE Trans. Wireless Commun._ , vol. 16, no. 4, pp. 2274–2287, 2017.
  * [17] O. Orhan, E. Erkip, and S. Rangan, “Low power analog-to-digital conversion in millimeter wave systems: Impact of resolution and bandwidth on performance,” in _Proc. IEEE ITA_ , Feb. 2015, pp. 191–198.
  * [18] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge univer-sity Press, 2004.
  * [19] C. Pan, H. Ren, K. Wang, et al, “Intelligent reflecting surface aided MIMO broadcasting for simultaneous wireless information and power transfer,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1719-1734, Aug. 2020.
  * [20] G. Zhou, C. Pan, H. Ren, et al, “Robust beamforming design for intelligent reflecting surface aided MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 10, pp. 1658-1662, Oct. 2020.
  * [21] Y. Cai, Y. Xu, Q. Shi, et al, “Robust joint hybrid transceiver design for millimeter wave full-duplex MIMO relay systems,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 2, pp. 1199-1215, Feb. 2019.
  * [22] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network: Joint active and passive beamforming design,” in _Proc. IEEE GLOBECOM_ , Abu Dhabi, UAE, Dec. IEEE, 2018, pp. 1–6.
  * [23] C. Liu, X. Liu, D. W. K. Ng, et al, “Deep residual learning for channel estimation in intelligent reflecting surface-assisted multi-user communications,” arXiv preprint arXiv: 2009.01423, 2020.
  * [24] C. Liu, Z. Wei, D. W. K. Ng, et al, “Deep transfer learning for signal detection in ambient backscatter communications,” _IEEE Trans. Wireless Commun._ , doi: 10.1109/TWC.2020.3034895.


#  MentSum: A Resource for Exploring
Summarization of Mental Health Online Posts

###### Abstract

Mental health remains a significant challenge of public health worldwide. With
increasing popularity of online platforms, many use the platforms to share
their mental health conditions, express their feelings, and seek help from the
community and counselors. Some of these platforms, such as Reachout, are
dedicated forums where the users register to seek help. Others such as Reddit
provide subreddits where the users publicly but anonymously post their mental
health distress. Although posts are of varying length, it is beneficial to
provide a short, but informative summary for fast processing by the
counselors. To facilitate research in summarization of mental health online
posts, we introduce Mental Health Summarization dataset, MentSum, containing
over 24k carefully selected user posts from Reddit, along with their short
user-written summary (called Tldr) in English from 43 mental health
subreddits. This domain-specific dataset could be of interest not only for
generating short summaries on Reddit, but also for generating summaries of
posts on the dedicated mental health forums such as Reachout. We further
evaluate both extractive and abstractive state-of-the-art summarization
baselines in terms of Rouge scores, and finally conduct an in-depth human
evaluation study of both user-written and system-generated summaries,
highlighting challenges in this research.

Keywords: Text Summarization, Summarization Dataset, Mental Health
Summarization

languageresourceLanguage Resources

MentSum: A Resource for Exploring
Summarization of Mental Health Online Posts

Sajad Sotudeh${}^{\text{1,2}}$*††thanks:  *Equal contribution, Nazli
Goharian${}^{\text{1,2}}$*11footnotemark: 1, Zachary Young${}^{\text{2}}$
---
${}^{\text{1}}$Information Retrieval Lab, Georgetown University
${}^{\text{2}}$Department of Computer Science, Georgetown University
{sajad<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Mental health has been a global public health challenge for many years and
even more so since the COVID-19 pandemic [Holmes et al., 2020, Pfefferbaum and
North, 2020, Otu et al., 2020]. Social media has served as a viable platform
for many to share their frustrations, emotions, depressions, and also their
already diagnosed mental disorders. Figure 1 depicts the growing popularity
(measured by the number of subscribers) of discussion forums dedicated to
three mental disorders in Reddit social discussion website over the years.
111Statistics from https://subredditstats.com/.

Online social platforms such as Reddit 222https://www.reddit.com/ and Reachout
333https://au.reachout.com/ have become increasingly popular over the recent
years due to the vital networking facets that they offer to the community
users. These platforms provide users with an opportunity to share different
types of user-curated and user-generated content, ranging from daily
updates/statuses to sharing personal anecdotes and mental conditions. Users
can also interact with other users, carry on conversations through which they
can express their feelings and views regarding a specific topic. Platforms
such as Reachout are not public, requiring users to register; users’ content
are not visible to anyone but to the permitted users and counselors. On the
other hand, in the public platforms such as Reddit, users can openly exchange
information with each other through community-based subreddits, each of which
specified with a certain theme or condition, such as suicide watch, mental
health, alcoholism, attention-deficit/hyperactivity disorder (ADHD),
depression, anxiety, etc. Each post in any of these subreddits, however, may
report more than one past or present condition and what the user is distressed
about.

Figure 1: Growing popularity of mental health related forums in Reddit.

The user-generated content on many of such platforms might be of varying
length. Longer posts may address multitude of issues of concern or simply be a
lengthy elaboration of the user on the situation. The longer a post is, the
more time it requires a counselor for reading the post which leads to fatigue
and/or delay in a timely response. Our hypothesis is that a short yet
informative summary of each user’s post provides the counselors with the
important information of the post in a glimpse before reading the details.
Hence, in this research we create a dataset resource for the research
community to be utilized in the short text (known as Tldr) summarization of
mental health related social media posts.

A great deal of research studies in social media mental health domain have
focused on developing classification models and their needed datasets to
either triage the severity of the potential harm or to identify the type of
mental disorders; among these efforts are [Choudhury et al., 2013, Coppersmith
et al., 2014a, Yates et al., 2017, Coppersmith et al., 2018, Cohan et al.,
2018, Garg et al., 2021]. Our goal is not to undermine classification of
disorders but potentially serve as yet another additional form of guidance to
the readers/counselors of the posts through short summaries of the posts. To
this end, we had to create a domain-specific dataset to contain social media
mental health related posts, along with their short user-written summary as
the gold standard. This dataset, MentSum, contains over 24k posts with
pairwise user-written summaries. We hope that MentSum would expedite future
work in the social media mental health text summarization task. In short, our
contributions are:

  * •

Creating Mental Health Summarization dataset, called MentSum that includes
over 24k user posts from the online mental health discussion forums along with
their user-written short summaries.

  * •

Providing the results of existing strong baselines in summarization, covering
both extractive and abstractive approaches.

  * •

Carrying out a human evaluation and error analysis in terms of fluency,
informativeness, and conciseness of the system generated summaries, on its
own, and in comparison with the user-written summaries.

We believe that insights from our human evaluation study can be further used
by future work to investigate more sophisticated models for moving the field
forward.

## 2\. Related work

Online social platforms provide a considerable wealth of textual data,
attracting attention of those who study the users’ mental conditions in social
environments. Early works in mental health research have put their focus on
understanding and identifying mental health conditions in social media
platforms such as Reddit and Twitter [Choudhury et al., 2013, Resnik et al.,
2013, Coppersmith et al., 2014b, Mowery et al., 2017b]; particularly,
predicting mental state of the users in online media [Hao et al., 2013, Wang
et al., 2013, Mowery et al., 2017a], exploring the types of mental health
condition [Wilson et al., 2014], detecting the severity level of mental
disorders [O’Dea et al., 2015, Chancellor et al., 2016], studying mental
health discourse [Choudhury and De, 2014], studying language of users and
identifying those with a high risk of mental illness [Milne et al., 2016,
Cohan et al., 2017], and analyzing the impact of conversation between a target
user and participants [Soldaini et al., 2018]. Some research focused on
creating large-scale mental health datasets such as rsdd [Yates et al., 2017],
smhd [Cohan et al., 2018] to detect potential of mental health conditions
through the general language of users, and rsdd-time [MacAvaney et al., 2018]
to study the temporal information of diagnoses. Unlike existing work whose
main focus has been on classification tasks, we define a text summarization
task over users’ mental health content in online social media platforms.

While summarization of clinical reports has already attracted the attention of
researchers [Mishra et al., 2014, Goldstein and Shahar, 2016, MacAvaney et
al., 2019, Zhang et al., 2020, Sotudeh et al., 2020], the summarization of
social media mental health posts has not been explored previously, which could
be due to lack of large-scale mental health summarization datasets. The
closest work but yet different than ours is done by ?) that aim to summarize
the mental health diagnostic interviews on a small-scale conversational
dataset (189 patient interviews) without human-written gold summaries. Hence,
to the best of our knowledge, we are the first to propose a relatively large
scale mental health text summarization dataset based off social media users’
content with user-written short summaries.

## 3\. MentSum dataset

In this section, we elaborate on construction of our dataset, provide dataset
statistics, and analyze the characteristics of the data. Subsequently, we
provide the ethics and privacy of the dataset.

### 3.1. Dataset construction

Our constructed dataset is based off Reddit mental health related posts of the
users along with their user-written short summaries (called Tldr) as the
ground-truth. Note that the author of the post and Tldr is the same; hence,
the goldness of this ground-truth Tldr, in respect to its fluency, and
completeness might be impacted by the emotional state of the post’s author.
The choice of Reddit for building our dataset is motivated by being a public
and popular platform, namely its public content and also the availability of
the short summary ground-truth for each post. Reddit is a social media
platform that supports communities called subreddits, each dedicated to a
specific topic.

We used Pushshift [Baumgartner et al., 2020] which is a social media data
repository containing recent and historical dumps of posted content on Reddit,
which are made publicly available to the Natural Language Processing (NLP)
community for research studies. We downloaded the Reddit data dumps covering
the period of 2005-2021, and filtered the posts based on a set of pre-defined
43 mental health subreddits. 444Subreddits are available at
https://ir.cs.georgetown.edu/resources/data/mentsum/ As not all of these
users’ posts have short summaries (i.e., Tldr), using regular expression, we
harvested posts that contain a Tldr summary as done in [Völske et al., 2017,
Sotudeh et al., 2021]. The regular expression matches keywords that begin with
uncased “TL” and end with uncased “DR”, allowing up to three characters in
between.

Social media texts are generally unstructured and noisy in terms of having
chunky sentences, and grammatical errors [Baldwin et al., 2013]. This is due
to the fact that users can freely express themselves [Liu et al., 2016]. To
further preserve high-quality instances, we applied the following filtering
using a set of hand-crafted filtering rules listed below.

  1. 1.

Token filtering: We remove a set of markup characters such as “&lt”, “&gt”,
“amp” , etc. that frequently occur within the harvested instances. URLs are
also removed and replaced with “@http ” tokens. We further remove all non-
ASCII characters that may happen within the social media text; hence,
preventing their negative effect in the summarization process. We further
replace the user IDs or users’ names with “@user” tokens to hinder the
possibility of users’ identities being disclosed.

  2. 2.

Instance filtering  We define two instance sampling criteria which should be
met by each instance to be included in the final MentSum dataset. First, we
identify the most frequent word bigram of the post’s Tldr; if it occurs more
than 3 times (empirically determined), we exclude the instance from the final
dataset, otherwise, we keep the instance. This is based on our observation
that Tldr summaries with more than 3 identical bigrams contain redundant
information, not conveying enough information about the posted content in a
short summary. Second, we apply a filtering rule based on the compression
ratio 555$\text{Compression ratio}=\frac{\text{count of words in user's
post}}{\text{count of words in {Tldr}}}$ of instances. Specifically, we only
retain instances whose compression ratio falls in the range of [2-13]
666[2-13] was decided empirically in our experiments. (i.e., user’s post
should be between 2x and 13x longer than the associated Tldr summary). This
filtering decision is based on the notion that to have short summaries we do
not want too small of compression ratio; to have informative enough summaries
we do not want too large of compression ratio.

The pipeline that we mentioned above reduced the initial set of 42k instances
to the final dataset with 24,119 English post-Tldr user-written summary pairs.

| Dataset size | 24,119 posts
---|---
Training set size | 21,695 posts
Subreddit coverage | 43 subreddits
|
Average post length (word/sent.) | 327.5 / 16.9
Min post length (word/sent.) | 50 / 1
Max post length (word/sent.) | 2,979 / 192
Median post length (word/sent.) | 267.0 / 14.0
|
Average Tldr length (word/sent.) | 43.5 / 2.6
Min Tldr length (word/sent.) | 15 / 1
Max Tldr length (word/sent.) | 596 / 36
Median Tldr length (word/sent.) | 35.0 / 2.0
|
Average sentence length (post/Tldr) | 19.4 / 18.3 words
Median sentence length (post/Tldr) | 17.0 / 16.0 words
|
Compression ratio (average) | 7.5

(a)
| Total vocabulary size | 76,411 words
---|---
Occurring 10+ times | 17,486 words
Training vocabulary size | 49,037 words
Validation vocabulary size | 13,609 words
Test vocabulary size | 13,765 words
Training/Test vocabulary overlap | 91.4%
(b)
Table 1: (a) Statistics of MentSum dataset. (b) Vocabulary statistics over
distinct and uncased vocabulary terms.

### 3.2. Dataset statistics

The overall dataset statistics are shown in Tables 1 (a). As shown, the
compression ratio of this dataset amounts to 7.5 which shows that Tldr
summaries are the extremely short version of their associated users’ post.
Table 1 (b) presents the statistics on distinct and uncased vocabularies
included in the dataset. As shown, about 23% of the vocabularies occur more
than 10 times within the dataset. Figure 2 depicts the count and percentage of
the posts for top 10 subreddits that have the highest count of posts. As
indicated, ADHD has the highest count of posts in MentSum, followed by
depression, anxiety, SuicideWatch, and the rest777While our dataset covers a
range of 43 mental health subreddits, we only show top 10 of them in terms of
post frequency due to space constraints..

In order to make different sets for training and evaluating summarization
models, we randomly divide the data into 21,695 training (90%), 1,209
validation (5%) and 1,215 test (5%) instances.

Figure 2: Count (y-axis) and proportion (above each bar) of posts of top 10
frequent subreddits in MentSum dataset. “Other” includes the posts in the
remaining 33 mental health subreddits.  Figure 3: The relative position of
word bigrams of the user-written Tldr summary across users’ post in MentSum
dataset.  Figure 4: Percentage of novel n-grams (y-axis) across increasing
word n-grams (x-axis) in user-written Tldrs. The heat extent shows the
frequency of word n-grams for a certain percentage bin.

### 3.3. Dataset analysis

In this section, we provide an analysis of MentSum dataset in terms of lead
bias, abstractive, and extractive characteristics of data.

Lead bias.  Lead bias [Hong and Nenkova, 2014] is a common phenomenon in News
summarization datasets, where the early parts of the source document contain
the most salient information. However, this characteristic does not hold in
the social media posts as the salient information are scattered in the entire
user’s post [Kim et al., 2019]; hence, imposing a challenge for the
summarization task. Figure 3 demonstrates the relative position of word
bigrams of the Tldr summary within users’ posts in MentSum dataset. As
observed, the Tldr summary’s bigrams are uniformly distributed along the
users’ post text, exhibiting a weak lead bias.

Abstractiveness. We further measure the abstractiveness of the purposed
MentSum dataset to verify its applicability for abstractive text summarization
models. Figure 4 shows the percentage of novel words for different n-grams. As
observed, the heat is mainly populated in the upper bins (i.e., higher
probabilities), particularly for $\\{n|n=2,3,4,5,6\\}$, which shows strong
abstractive characteristics of the MentSum dataset, making it suitable for
abstractive summarization task.

Extractiveness. Figure 5 shows the density estimation diagram of oracle
sentences’ relative position in the users’ post in regard to the Rouge score
888We have taken average of Rouge-1, Rouge-2, and Rouge-L scores w.r.t the
Tldr summary. of the oracle sentences. Oracle sentences are up to 3 summary-
worthy sentences which are labelled using a greedy labeling approach proposed
in ?). As indicated, the oracle sentences appear across various positions of
the users’ post. Considering the diagram and the range of oracle sentences’
scores, MentSum dataset still expresses extractive characteristics in addition
to its high abstractiveness.

Figure 5: Kernel density estimation diagram of oracle sentences’ relative
position and the Rouge score of oracle sentences in respect to user-written
Tldrs.

### 3.4. Ethics and privacy

Although we use publicly available Reddit data in our research to construct
the MentSum dataset, mental health is a sensitive topic and special care
should be taken when such data is used in social media research [Thomas et
al., 2002, Moreno et al., 2013, Suster et al., 2017, Benton et al., 2017,
Nicholas et al., 2020]. Hence, we made no attempt to identify and contact the
users, or discover user relations with other social media accounts. In
preprocessing step, we have de-identified the usernames with @user tokens to
prevent the user identities from being known. The MentSum dataset can be
accessed through a Data Usage Agreement (DUA)
999https://ir.cs.georgetown.edu/resources/. The DUA particularly ensures that
no attempts should be made to distribute portions of dataset (which could
result in revealing users’ identity), identify users, and contact users.

## 4\. Experimental setup

To evaluate the quality of MentSum dataset for the summarization task of
mental health related posts, and to provide strong baselines for further
research, we explored several baselines. In this section, we present the
baselines and the implementation details.

### 4.1. Baselines

We explored various extractive and abstractive baselines which are listed
below.

  * -

Lead-2: A simple extractive baseline that selects the first two leading
sentences as the summary.

  * -

LSA [Steinberger and Jez̈ek, 2004]: A non-neural extractive vector-based model
that adopts the mathematical concept of Singular Value Decomposition (SVD) to
find hidden semantic structures of words and sentences.

  * -

LexRank [Erkan and Radev, 2004]: An unsupervised extractive model that makes
use of graph centrality network to find important sentences and concatenate
them to form the final summary.

  * -

BertSumExt [Liu and Lapata, 2019]: A neural extractive model that fine-tunes
Bert [Devlin et al., 2019] language model on extractive summarization text.
This model runs by appending [CLS] tokens to the start of each input sentence,
and use the representations associated with [CLS] tokens to predict sentence
importance. [CLS] is the classification head in Bert model that aggregates the
contextualized embeddings of preceding tokens.

  * -

MatchSum [Zhong et al., 2020]: A state-of-the-art extractive summarization
model that first composes candidate summaries given the salient set of source
sentences scored by BertSumExt model, and then ranks them using the Siamese
neural networks. Top-ranked candidate summary is retrieved as the final
extractive summary of the post.

  * -

BertSumAbs [Liu and Lapata, 2019]: The abstractive variant of BertSum
framework, where the encoder is simply a Bert model, which is trained
alongside a Transformers-based [Vaswani et al., 2017] decoder from scratch.

  * -

BertSumExtAbs [Liu and Lapata, 2019]: A two-stage fine-tuned abstractive model
that exploits a pre-trained BertSumExt summarizer (i.e., first stage) which is
further fine-tuned along with a decoder on abstractive summarization task
(i.e., second stage).

  * -

Bart [Lewis et al., 2020]: An abstractive model that is currently amongst the
most powerful state-of-the-art summarization models. Bart extends the Bert’s
intuition by adding up a couple of pre-training objectives including token
deletion, text infilling, sentence permutation, and document rotation. Unlike
Bert, Bart utilizes a pre-trained encoder-decoder framework for language
generation task, summarization being one of them.

### 4.2. Implementation details

We used Sumy package 101010https://github.com/miso-belica/sumy for running
non-neural extractive models. To find extractive oracle labels (i.e.,
important sentences) of users’ post, we ran a greedy labeling approach [Liu
and Lapata, 2019] over the entire set of source sentences and retrieved up to
3 sentences as the extractive summary. For BertSum models, we used the
official codebase 111111https://github.com/nlpyang/PreSumm with BERT-base-
uncased and ran all of the models with default hyper-parameters as suggested
by ?) besides the learning rate of $1e-3$ and warmup steps of 5k. For
MatchSum, we used RoBERTa-base as the encoder with the same default hyper-
parameters as initialized in the original paper [Zhong et al., 2020] and
candidate summaries of lengths 2 and 3. We utilized Huggingface Transformers’
[Wolf et al., 2020] implementation for training Bart model. All the
experimented neural models were trained for 5 epochs and then the best
checkpoint that attains the highest Rouge-L score during validation was picked
for inference time. As the optimizer, we used AdamW [Loshchilov and Hutter,
2019] initialized with learning ratio of $3e-5$,
$(\beta_{1},\beta_{2})=(0.9,0.98)$, and a weight decay of 0.01. Cross-entropy
loss was employed for all of the experimented models. We also utilized Weights
& Biases toolkit [Biewald, 2020] to keep track of training and validation
progress.

Method | RG-1 | RG-2 | RG-L
---|---|---|---
Lead-2 | 20.09 | 3.57 | 13.70
OracleExt | 35.98 | 11.59 | 23.21
|  |  |
LSA [Steinberger and Jez̈ek, 2004] | 22.96 | 3.98 | 14.50
LexRank [Erkan and Radev, 2004] | 23.21 | 4.42 | 15.19
BertsumExt [Liu and Lapata, 2019] | 24.64 | 5.83 | 16.66
MatchSum [Zhong et al., 2020] | 26.29 | 6.32 | 17.12
|  |  |
BertsumAbs [Liu and Lapata, 2019] | 25.35 | 6.49 | 17.11
BertsumExtAbs [Liu and Lapata, 2019] | 25.75 | 6.80 | 17.48
Bart [Lewis et al., 2020] | 29.13 | 7.98 | 20.27
Table 2: Rouge (F1) results on MentSum’s test set.

## 5\. Results and discussion

Table 2 presents the performance of summarization models on the test set of
MentSum dataset in terms of widely adopted Rouge (F1) metric [Lin, 2004] that
measures the text overlap between the system-generated Tldr summaries and
user-written Tldr summaries. As expected and was shown in the data analysis,
Lead-2 is unable to perform well on MentSum dataset. As addressed earlier,
this is due to scattered salient information throughout the post.

Comparing extractive models with the abstractive ones, we notice that
extractive models lag behind the best performing abstractive model (i.e.,
Bart) by a huge margin; this is not surprising as summaries are paraphrased by
users, as also confirmed by Figure 4 in Section 3.3. Even when comparing
abstractive variants of BertSum model (i.e., BertSumAbs and BertSumExtAbs)
with the extractive models, we see a similar trend of gaining more significant
improvements via abstractive models.

While comparing the extractive models with each other, we observe that
BertSumExt model considerably outperforms the other unsupervised models (i.e.,
LSA amd LexRank), though, it lags behind the MatchSum model as the best-
perfomring extractive model. This shows the effect of learning salient
sentences in supervised manner, accounting for the improvements of both
BertSumExt and MatchSum models. Noting that OracleExt shows the upper bound
performance of extractive summarization models, it is noticeable that we see a
large gap between the extractive systems and OracleExt’s performances. This
observation is a clear justification that there is still a large room for
improvement in extractive summarization setting, which can be further
approached by future work.

While looking at the performances of abstractive summarization models, we see
a remarkable improvement of Bart over other systems; particularly, with
relative improvements of 13.12% (RG-1), 17.35% (RG-2), and 15.96% (RG-L)
compared to BertSumExtAbs model 121212The relative improvement of system A
over system B is calculated as
$\frac{\text{Rouge}_{\text{A}}-\text{Rouge}_{\text{B}}}{\text{Rouge}_{\text{B}}}$.
This is expected as Bart uses pre-trained encoder-decoder framework, unlike
abstractive models of BertSum that only employ the pre-trained encoder (i.e.,
Bert) while training the decoder from scratch. It is also observed that
BertSumExtAbs model surpasses BertSumAbs model, showing the effectiveness of
extractive objectives in abstractive summary generation process as also
suggested by ?).

Figure 6: Percentage of novel n-grams across increasing word n-grams in Bart’s
system-generated Tldrs on test set. The heat extent shows the frequency of
word n-grams for a certain percentage bin.

Figure 6 demonstrates the percentage of novel n-grams in Bart-generated Tldrs
over the test set.131313The novel n-grams diagram on user-written Tldrs in
test set is quite similar to Figure 4 and is not shown due to space
limitations. As indicated, with increasing n-grams, the rate of novel words
generation grows, showing that Bart does the abstraction by
rephrasing/paraphrasing the source sentences. Also, compared to the user-
written Tldrs, Bart does more extraction (on word level) than users do;
however, it is still hard to attribute this observation to the quality of
generated Tldrs as the goldness of ground-truth Tldrs may be influenced by the
emotional state of the author at the time of writing as noted in Section 3.1.

## 6\. Human analysis

A few prior studies have recognized the limitations of widely used Rouge
metric on qualitative evaluation as it is rather biased towards surface
lexical similarities [Ng and Abrecht, 2015, Cohan and Goharian, 2016]. To shed
light on the qualities and limitations of the state-of-the-art summarizer
(i.e., Bart), and provide insightful directions for future work, we carried
out a human evaluation study. To this end, we randomly selected 100 posts
along with their associated user-written and system-generated Tldr summaries
from the test set. Following prior work [Grusky et al., 2018, Zhang et al.,
2020, Cho et al., 2021, Sotudeh et al., 2021], we defined three qualitative
metrics: (1) Fluency: is the summary well-written and easy to understand?; (2)
Informativeness: does the Tldr summary provide useful information (i.e., the
most important information) about user’s post?; (3) Conciseness: does the
summary briefly provide comprehensive information (i.e., majority of important
information) about user’s post? Two human annotators evaluated the provided
cases using 5-point Likert scale (1 = worst, 5 = best). For disagreement
cases, where the assigned scores are 2 levels or more different from each
other, a third annotator broke the disagreement and made the final decision.
We then provided selected cases through the following multi-stage pipeline:

  1. 1.

In the first stage, we provided 100 posts with one randomly selected summary
(from either user-written or system-generated), and asked the annotators to
independently score the given summary in terms of the qualitative criteria.

  2. 2.

In the second stage, we provided the same 100 posts but with the other summary
that was not shown to the annotators in the first stage, and asked the
annotators to independently evaluate the given summary on the qualitative
criteria.

  3. 3.

In the final stage, we provided the same 100 posts with both summaries (i.e.,
user-written and system-generated) side-by-side, shuffled, non known to
annotators and asked the annotators to specify which summary they prefer the
most. The annotators were further asked to stipulate which major pieces of
information are captured by or missing from each given summary. We will
discuss the details of this stage in Section 7 (i.e., Error Analysis).

While stages (1) and (2) could have been combined into one stage, to avoid any
bias in scoring these two summaries, we decided it was better that the
annotators would score each of these two summaries independently without any
comparison. It has to be mentioned that the order of summaries was shuffled;
that is, the annotators were not aware of which of the two summaries they were
evaluating.

System | Fluency | Info. | Conc.
---|---|---|---
Bart | 4.60 | 3.65 | 3.51
User | 4.28 | 3.46 | 3.15

Table 3: Results of the human evaluation comparing the systems in terms of
Fluency, Informativeness, and Conciseness. Winning scores are shown in bold.
Scores are in 5-point Likert scale (1=worst, 5=best).

System | Fluency | Info. | Conc.
---|---|---|---
Bart | 17.4% | 28.2% | 28.0%
User | 19.5% | 25.2% | 24.6%

Table 4: System-wise Cohen’s kappa inter-rater agreement.

The evaluation scores are averaged and shown in Table 3. As shown, the Bart
summarizer outperforms the user-written Tldr summaries across all metrics with
relative improvements of 7.47% (Fluency), 5.49% (Informativeness), and 11.42%
(Conciseness). Observing the significant improvement gain by the state-of-the-
art summarizer, it does seem interesting that contextualized language
modelling based approaches such as Bart are becoming a firm touchstone to be
compared with the human system. This finding is consistent with the
observations that have been made recently by a prior work [Fabbri et al.,
2021].

Table 4 reports the Cohen’s kappa [Cohen, 1960] inter-rater agreement for both
system-generated and user-written Tldr summaries. With regard to the Cohen’s
kappa range interpretation [McHugh, 2012], the agreements obtained on
informativeness and Conciseness metrics fall into the “fair” agreement, while
there is a “slight” agreement observed on fluency. The slight agreement on
fluency could be attributed to the subjective nature of this metric in
evaluation process [van der Lee et al., 2021], leading to a high variability
in assigned scores by the annotators. In addition to the overall promising
performance of Bart, we observed that the nature of data is also impactful in
the process of human study. In other words, users in social media are free to
publish their content of discussion in whatever style they like to as there is
usually not any supervision over the posted content in terms of quality, which
leads to having gibberish Tldrs in some cases.

As the results of stage (3) of the evaluation, the annotators reported that
they prefer system-generated summaries in 59% of the cases, and user-written
summaries in 41% of the cases with an agreement rate of 59.6% (moderate).

## 7\. Error analysis

In this section, we showcase where the system summarizer lacks in or improves
compared to the user-written summaries in more detail. As reported by the
annotators, the distribution of qualitative criteria for determining which
summary is more preferable is shown in Table 5. For instance, annotators pick
the Bart-generated summaries in 59% of cases. In 64% of Bart’s win cases, the
system-generated summary was preferred due to its informativeness as compared
to the user-written summary. As indicated, out of 3 evaluation metrics,
informativeness is the most dominant one in the selection of Bart-generated
and user-written summaries.

System | Win rate | Fluency | Info. | Conc.
---|---|---|---|---
Bart | 59% | 17% | 64% | 19%
User | 41% | 8% | 81% | 11%
Table 5: Distribution of leading criteria for summary selection in human
evaluation process.

We report the findings for each of the three metrics of fluency,
informativeness, and conciseness 141414Examples are truncated in the paper for
extra privacy of the users.:

  * -

Fluency.  Few reasons contributed to annotators’ lower score for user-written
Tldrs in terms of fluency. One of them is grammatical issues in the users’
language in user-written Tldr summaries. Examples of such grammatical error is
“...good friend and ex girlfriend has...”. Another observation showed the
complex structure of the user-written sentence which makes the summary less
understandable such as in “...she can’t be arsed trying to communicate with
myself...”. Interestingly, the system-generated Tldrs did not suffer from
grammatical errors. This might be due to the fact that the system summarizer
is exposed to the structured, grammatical, and fluent textual data from books
and Wikipedia during pre-training, and hence with further fine-tuning on the
MentSum dataset, it is yet able to produce understandable Tldrs. On the other
hand, for the cases where system-generated summaries underperform the user-
written ones, we observed examples of repetition such as in “i’m having
massive mood swings and mood swings...”, and complexity without specifying
sentence boundaries such as in “...i was @ageX i am @ageY now and i haven’t
been able to find a therapist until my @ageZ birthday and i want to offer
myself...”.

  * -

Informativeness.  We observed that the users provide some history of their
mental health and disorders within their posts. It is important for these
vital information to be captured in the system-generated summaries.
Interestingly, the system summarizer attends more to such mental disorders and
includes them into the generated summaries. For instance, in the system-
generated summary “...diagnosed with anxiety, GAD, and agoraphobia...” all
mental illnesses of the user are captured. However, the user-written Tldr
“...was diagnosed with anxiety...” does not include “GAD, and agoraphobia”.
Furthermore, we observed that parts of user-written summaries contain
information that is not directly mentioned in the user’s post, but was
inferred from it. Such cases are still a challenge for the system summarizers.
As an example, consider user-written Tldr of “...I’m holding it all in which
is making me and everyone around me worse...”, which conveys information that
is not directly mentioned in the post, but could be inferred from the post.
While user-written Tldr contains such salient information, the system-
generated Tldr was not able to infer and generate that. In underperformed
cases, we observed that the model focuses on a few (one or two) important
points of the user’s post, while the user-written Tldr includes more important
points from diverse parts of the user’s post.

  * -

Conciseness.  The aforementioned limitations of informativeness affect the
conciseness negatively. That is, some of important information that are
required to be in a concise summary are missed. In particular, for the cases
in which user-written Tldrs are scored lower than system-generated ones in
terms of conciseness, the user-written Tldrs are verbose and very detailed.
However, the system is able to skip unnecessary information, focus on
discerning important information, and verbalize them within a comparably
shorter text. For instance, consider system-generated Tldr “...been on meds
and seeing a psych for a while, now finally going to a dr for help...” in
comparison with its user-written Tldr “...saw a psych about it but then
stopped. felt like garbage last year. seeing a doctor about it now. happy
hopeful noises...”. As seen, the system verbalizes the important points using
a comparably briefer usage of language.

## 8\. Summary and conclusion

Mental health and illnesses have become a major challenge in public health.
The prevalence of user-generated and user-curated content in online social
media platforms has provided a large amount of accessible information for
those who seek to study user behavioural patterns in social settings. While
most previous research studies in mental health domain have focused on
developing datasets for the classification tasks to triage and detect the type
and severity of mental health concerns, there has not been any dataset to
support the research in summarizing mental health related social media posts.
In this paper, we introduced our large-scale novel dataset resource, MentSum
for the task of summarization of users’ mental health posts on Reddit social
media. Although the dataset is gathered from 43 publicly available Reddit
subreddits, it will be provided to researchers through a Data Usage Agreement
(DUA) to protect the privacy of the users. Our detailed dataset analyses
revealed characteristics such as weak lead bias, strong abstractiveness, and
extractiveness. We further evaluated various state-of-the-art extractive and
abstractive summarization models on MentSum dataset. This study showed that
while the abstractive method (i.e., Bart) outperformed the extractive models,
the gap between the extractive models’ performance and OracleExt (i.e., the
upper bound performance of extractive methods) indicates a large room for
improvement in extractive settings, calling for future research. Our human
evaluation over 100 randomly selected post summary pairs in terms of fluency,
informativeness, and conciseness of system-generated and user-written
summaries revealed interesting information; the system-generated summaries
were preferred on average over user-written ones, showing the promising
performance of contextualized language modelling based summarization
approaches. Finally, we provided an error analysis to show the current
challenges. We hope our provided MentSum dataset paves the path for further
exploration in summarization research of social media mental health posts.

## 9\. Bibliographical References

## References

  * Baldwin et al., 2013 Baldwin, T., Cook, P., Lui, M., MacKinlay, A. D., and Wang, L.  (2013).  How noisy social media text, how diffrnt social media sources?  In IJCNLP.
  * Baumgartner et al., 2020 Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., and Blackburn, J.  (2020).  The pushshift reddit dataset.  In ICWSM.
  * Benton et al., 2017 Benton, A., Coppersmith, G. A., and Dredze, M.  (2017).  Ethical research protocols for social media health research.  In EthNLP@EACL.
  * Biewald, 2020 Biewald, L.  (2020).  Experiment tracking with weights and biases.  Software available from wandb.com.
  * Chancellor et al., 2016 Chancellor, S., Lin, Z. J., Goodman, E. L., Zerwas, S., and Choudhury, M. D.  (2016).  Quantifying and predicting mental illness severity in online pro-eating disorder communities.  Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing.
  * Cho et al., 2021 Cho, S., Dernoncourt, F., Ganter, T., Bui, T., Lipka, N., Chang, W., Jin, H., Brandt, J., Foroosh, H., and Liu, F.  (2021).  Streamhover: Livestream transcript summarization and annotation.  ArXiv preprint, abs/2109.05160.
  * Choudhury and De, 2014 Choudhury, M. D. and De, S.  (2014).  Mental health discourse on reddit: Self-disclosure, social support, and anonymity.  In ICWSM.
  * Choudhury et al., 2013 Choudhury, M. D., Gamon, M., Counts, S., and Horvitz, E.  (2013).  Predicting depression via social media.  In ICWSM.
  * Cohan and Goharian, 2016 Cohan, A. and Goharian, N.  (2016).  Revisiting summarization evaluation for scientific articles.  In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 806–813, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Cohan et al., 2017 Cohan, A., Young, S., Yates, A., and Goharian, N.  (2017).  Triaging content severity in online mental health forums.  Journal of the Association for Information Science and Technology, 68.
  * Cohan et al., 2018 Cohan, A., Desmet, B., Yates, A., Soldaini, L., MacAvaney, S., and Goharian, N.  (2018).  Smhd: a large-scale resource for exploring online language usage for multiple mental health conditions.  In COLING.
  * Cohen, 1960 Cohen, J.  (1960).  A coefficient of agreement for nominal scales.  Educational and Psychological Measurement, 20:37 – 46.
  * Coppersmith et al., 2014a Coppersmith, G. A., Dredze, M., and Harman, C.  (2014a).  Quantifying mental health signals in twitter.  In CLPsych@ACL.
  * Coppersmith et al., 2014b Coppersmith, G. A., Harman, C., and Dredze, M.  (2014b).  Measuring post traumatic stress disorder in twitter.  In ICWSM.
  * Coppersmith et al., 2018 Coppersmith, G. A., Leary, R., Crutchley, P., and Fine, A. B.  (2018).  Natural language processing of social media as screening for suicide risk.  Biomedical Informatics Insights, 10.
  * Devlin et al., 2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Erkan and Radev, 2004 Erkan, G. and Radev, D. R.  (2004).  Lexrank: Graph-based lexical centrality as salience in text summarization.  J. Artif. Intell. Res., 22:457–479.
  * Fabbri et al., 2021 Fabbri, A. R., Kryscinski, W., McCann, B., Socher, R., and Radev, D.  (2021).  Summeval: Re-evaluating summarization evaluation.  Transactions of the Association for Computational Linguistics, 9:391–409.
  * Garg et al., 2021 Garg, S., Taylor, J., Sherief, M. E., Kasson, E., Aledavood, T., Riordan, R., Kaiser, N. M., Cavazos-Rehg, P. A., and Choudhury, M. D.  (2021).  Detecting risk level in individuals misusing fentanyl utilizing posts from an online community on reddit.  Internet Interventions, 26.
  * Gehrmann et al., 2018 Gehrmann, S., Deng, Y., and Rush, A.  (2018).  Bottom-up abstractive summarization.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Goldstein and Shahar, 2016 Goldstein, A. and Shahar, Y.  (2016).  An automated knowledge-based textual summarization system for longitudinal, multivariate clinical data.  Journal of biomedical informatics, 61:159–75.
  * Grusky et al., 2018 Grusky, M., Naaman, M., and Artzi, Y.  (2018).  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hao et al., 2013 Hao, B., Li, L., Li, A., and Zhu, T.  (2013).  Predicting mental health status on social media - a preliminary study on microblog.  In HCI.
  * Holmes et al., 2020 Holmes, E. A., O’Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., Ballard, C. G., Christensen, H., Silver, R. C., Everall, I., Ford, T. J., John, A., Kabir, T., King, K., Madan, I., Michie, S., Przybylski, A. K., Shafran, R., Sweeney, A., Worthman, C. M., Yardley, L., Cowan, K. C., Cope, C., Hotopf, M., and Bullmore, E. T.  (2020).  Multidisciplinary research priorities for the covid-19 pandemic: a call for action for mental health science.  The Lancet. Psychiatry, 7:547 – 560.
  * Hong and Nenkova, 2014 Hong, K. and Nenkova, A.  (2014).  Improving the estimation of word importance for news multi-document summarization.  In EACL.
  * Kim et al., 2019 Kim, B., Kim, H., and Kim, G.  (2019).  Abstractive summarization of Reddit posts with multi-level memory networks.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519–2531, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Lewis et al., 2020 Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L.  (2020).  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin, 2004 Lin, C.-Y.  (2004).  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata, 2019 Liu, Y. and Lapata, M.  (2019).  Text summarization with pretrained encoders.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al., 2016 Liu, H., Morstatter, F., Tang, J., and Zafarani, R.  (2016).  The good, the bad, and the ugly: uncovering novel research opportunities in social media mining.  International Journal of Data Science and Analytics, 1:137–143.
  * Loshchilov and Hutter, 2019 Loshchilov, I. and Hutter, F.  (2019).  Decoupled weight decay regularization.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  * MacAvaney et al., 2018 MacAvaney, S., Desmet, B., Cohan, A., Soldaini, L., Yates, A., Zirikly, A., and Goharian, N.  (2018).  Rsdd-time: Temporal annotation of self-reported mental health diagnoses.  In CLPsych@NAACL-HTL.
  * MacAvaney et al., 2019 MacAvaney, S., Sotudeh, S., Cohan, A., Goharian, N., Talati, I. A., and Filice, R. W.  (2019).  Ontology-aware clinical abstractive summarization.  Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
  * Manas et al., 2021 Manas, G., Aribandi, V., Kursuncu, U., Alambo, A., Shalin, V. L., Thirunarayan, K., Beich, J., Narasimhan, M., and Sheth, A.  (2021).  Knowledge-infused abstractive summarization of clinical diagnostic interviews: Framework development study.  JMIR Mental Health, 8.
  * McHugh, 2012 McHugh, M. L.  (2012).  Interrater reliability: the kappa statistic.  Biochemia Medica, 22:276 – 282.
  * Milne et al., 2016 Milne, D. N., Pink, G., Hachey, B., and Calvo, R. A.  (2016).  Clpsych 2016 shared task: Triaging content in online peer-support forums.  In CLPsych@HLT-NAACL.
  * Mishra et al., 2014 Mishra, R., Bian, J., Fiszman, M., Weir, C. R., Jonnalagadda, S. R., Mostafa, J., and Fiol, G. D.  (2014).  Text summarization in the biomedical domain: A systematic review of recent research.  Journal of biomedical informatics, 52:457–67.
  * Moreno et al., 2013 Moreno, M. A., Goniu, N., Moreno, P. S., and Diekema, D.  (2013).  Ethics of social media research: Common concerns and practical considerations.  Cyberpsychology, behavior and social networking, 16 9:708–13.
  * Mowery et al., 2017a Mowery, D. L., Bryan, C. J., and Conway, M.  (2017a).  Feature studies to inform the classification of depressive symptoms from twitter data for population health.  ArXiv, abs/1701.08229.
  * Mowery et al., 2017b Mowery, D. L., Smith, H., Cheney, T., Stoddard, G., Coppersmith, G. A., Bryan, C. J., and Conway, M.  (2017b).  Understanding depressive symptoms and psychosocial stressors on twitter: A corpus-based study.  Journal of Medical Internet Research, 19.
  * Ng and Abrecht, 2015 Ng, J.-P. and Abrecht, V.  (2015).  Better summarization evaluation with word embeddings for ROUGE.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics.
  * Nicholas et al., 2020 Nicholas, J., Onie, S., and Larsen, M. E.  (2020).  Ethics and privacy in social media research for mental health.  Current Psychiatry Reports, 22:1–7.
  * O’Dea et al., 2015 O’Dea, B., Wan, S., Batterham, P. J., Calear, A. L., Paris, C., and Christensen, H.  (2015).  Detecting suicidality on twitter.  Internet Interventions, 2:183–188.
  * Otu et al., 2020 Otu, A. A., Charles, C. H., and Yaya, S.  (2020).  Mental health and psychosocial well-being during the covid-19 pandemic: the invisible elephant in the room.  International Journal of Mental Health Systems, 14.
  * Pfefferbaum and North, 2020 Pfefferbaum, B. and North, C. S.  (2020).  Mental health and the covid-19 pandemic.  The New England journal of medicine.
  * Resnik et al., 2013 Resnik, P., Garron, A., and Resnik, R.  (2013).  Using topic modeling to improve prediction of neuroticism and depression in college students.  In EMNLP.
  * Soldaini et al., 2018 Soldaini, L., Walsh, T., Cohan, A., Han, J., and Goharian, N.  (2018).  Helping or hurting? predicting changes in users’ risk of self-harm through online community interactions.  In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 194–203, New Orleans, LA, June. Association for Computational Linguistics.
  * Sotudeh et al., 2020 Sotudeh, S., Goharian, N., and Filice, R.  (2020).  Attend to medical ontologies: Content selection for clinical abstractive summarization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1899–1905, Online. Association for Computational Linguistics.
  * Sotudeh et al., 2021 Sotudeh, S., Deilamsalehy, H., Dernoncourt, F., and Goharian, N.  (2021).  TLDR9+: A large scale resource for extreme summarization of social media posts.  In Proceedings of the Third Workshop on New Frontiers in Summarization, pages 142–151, Online and in Dominican Republic, November. Association for Computational Linguistics.
  * Steinberger and Jez̈ek, 2004 Steinberger, J. and Jez̈ek, K.  (2004).  Using latent semantic analysis in text summarization and summary evaluation.  In ISIM.
  * Suster et al., 2017 Suster, S., Tulkens, S., and Daelemans, W.  (2017).  A short review of ethical challenges in clinical natural language processing.  In EthNLP@EACL.
  * Thomas et al., 2002 Thomas, J. C., Sage, M., Dillenberg, J., and Guillory, V. J.  (2002).  A code of ethics for public health.  American journal of public health, 92 7:1057–9.
  * van der Lee et al., 2021 van der Lee, C., Gatt, A., van Miltenburg, E., and Krahmer, E. J.  (2021).  Human evaluation of automatically generated text: Current trends and best practice guidelines.  Comput. Speech Lang., 67:101151.
  * Vaswani et al., 2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  (2017).  Attention is all you need.  In Isabelle Guyon, et al., editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.
  * Völske et al., 2017 Völske, M., Potthast, M., Syed, S., and Stein, B.  (2017).  TL;DR: Mining Reddit to learn automatic summarization.  In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wang et al., 2013 Wang, X., Zhang, C., and Sun, L.  (2013).  An improved model for depression detection in micro-blog social network.  2013 IEEE 13th International Conference on Data Mining Workshops, pages 80–87.
  * Wilson et al., 2014 Wilson, M. L., Ali, S., and Valstar, M. F.  (2014).  Finding information about mental health in microblogging platforms: a case study of depression.  Proceedings of the 5th Information Interaction in Context Symposium.
  * Wolf et al., 2020 Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A.  (2020).  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  * Yates et al., 2017 Yates, A., Cohan, A., and Goharian, N.  (2017).  Depression and self-harm risk assessment in online forums.  In EMNLP.
  * Zhang et al., 2020 Zhang, Y., Merck, D., Tsai, E., Manning, C. D., and Langlotz, C.  (2020).  Optimizing the factual correctness of a summary: A study of summarizing radiology reports.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5108–5120, Online. Association for Computational Linguistics.
  * Zhong et al., 2020 Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., and Huang, X.  (2020).  Extractive summarization as text matching.  In ACL.

lrec2022-bib languageresource