# Learning and Evaluating Sparse Interpretable Sentence Embeddings

Valentin Trifonov
ETH Zürich, Switzerland
<EMAIL_ADDRESS>
&Octavian-Eugen Ganea
ETH Zürich, Switzerland
<EMAIL_ADDRESS>
Anna Potapenko
National Research University
Higher School of Economics, Russia111Work done during an internship at ETH
Zürich.
<EMAIL_ADDRESS>
&Thomas Hofmann
ETH Zürich, Switzerland
<EMAIL_ADDRESS>

###### Abstract

Previous research on word embeddings has shown that sparse representations,
which can be either learned on top of existing dense embeddings or obtained
through model constraints during training time, have the benefit of increased
interpretability properties: to some degree, each dimension can be understood
by a human and associated with a recognizable feature in the data. In this
paper, we transfer this idea to sentence embeddings and explore several
approaches to obtain a sparse representation. We further introduce a novel,
quantitative and automated evaluation metric for sentence embedding
interpretability, based on topic coherence methods. We observe an increase in
interpretability compared to dense models, on a dataset of movie dialogs and
on the scene descriptions from the MS COCO dataset.

##  1 Introduction

In the word embeddings literature, it has previously been of interest to find
interpretable representations: individual dimensions should capture a distinct
semantic meaning, such that humans are able to understand why a word is
encoded in a particular vector. With a cognitive plausibility argument from
Murphy et al. (2012), the interpretability can be linked to sparse
representations: they argue that the representation should model a wide range
of features in the data and that every sample should be characterized by the
presence of a small number of key features. Arora et al. (2016) use this idea
to recover and disentangle the different meanings of polysemous words.

The above/named approaches, as well as those by Subramanian et al. (2017);
Faruqui et al. (2015), recover an interpretable sparse representation in a
separate, post-processing step on top of the uninterpretable, dense embeddings
of the original model (often word2vec or GloVe). This is commonly done using
sparse coding or a downstream model. Additionally to understanding a model’s
intermediate representation, there has been work on constructing models that
inherently use a sparse embedded representation by learning it during the
training process (Sun et al., 2016; Chen et al., 2017). This is motivated by
the idea that the model should include the prior that each word is a sparse
combination of disentangled features from the very beginning. In contrast,
when computing dense embeddings first, it is less likely that this
representation will be easily disentanglable in the post-processing step.

Goh (2016) argues that sparse representations can be used to explain image and
sentence embeddings as well. To be precise, the author focuses on encoder-
decoder neural networks and uses sparse coding to recover interpretable
features in the latent spaces of a variational autoencoder (Kingma and
Welling, 2013) and an image captioning system based on (Vinyals et al., 2015).

In this paper, we aim to use sparse methods to disentangle sentence
embeddings’ dimensions. We focus on a simple sentence autoencoder model, and
apply both a sparse-coding-based post-processing technique, as well as model
constraints during training time, to obtain sparse vector representations of
sentences. We aim to increase the understanding of the latent space, which
helps us gain insight into how the inference and learning process works by
identifying the patterns in the data that the model learns to recognize and
encode in this representation.

To compare our different approaches, as well as measure the improvement
compared to the baseline of a dense autoencoder model, we introduce a novel,
quantitative and automated metric of the mentioned interpretability
properties. It is based on the notion of topic coherence and further develops
it for the case of sentences. We observe that the new measure reflects our
manual judgment on the interpretability of the embeddings. Additionally, we
track reconstruction quality and performance in downstream tasks, showing that
sparse approaches can obtain a remarkable increase in interpretability at a
moderate cost in quality.

##  2 Models

Our models are based on a standard recurrent neural network autoencoder
following the _Sequence-to-Sequence_ architecture (seq2seq; Sutskever et al.,
2014). This architecture is based on the encoder-decoder scheme, where an
encoder network maps the input to a dense, embedded representation
$\mathbf{z}$, and a decoder net reconstructs the input from $\mathbf{z}$. In
Section 3, we give a more detailed description of our experimental setup.

###  2.1 Enforcing Sparsity by Post-Processing Dense Embeddings

Consider a dataset $x_{1},\ldots,x_{N}$ of $N$ sentences. We train a dense
autoencoder net with a hidden state size $D^{\prime}=500$ to convergence, and
compute
$\mathbf{Z}=[\mathbf{z}_{1},\ldots,\mathbf{z}_{N}]^{\mathsf{T}}\in\mathbb{R}^{N\times
D^{\prime}}$, the vector representations of the data arranged as the rows of a
matrix. We follow the approaches by Arora et al. (2016); Goh (2016) and
compute a sparse representation of size $D=2000$ on top of $\mathbf{Z}$, where
all but $k$ values have to be zero. We do this by solving the following sparse
dictionary learning problem:

$\begin{split}\mathbf{E},\mathbf{U}=\arg\min_{\mathbf{E},\mathbf{U}}||\mathbf{E}\mathbf{U}-\mathbf{Z}||_{F}^{2},\\\
s.t.\;||\mathbf{e}_{i}||_{0}\leq k,||\mathbf{u}_{j}||_{2}=1,\;\forall
i,j,\end{split}$ (1)

whereby we obtain
$\mathbf{E}=[\mathbf{e}_{1},\ldots,\mathbf{e}_{N}]^{\mathsf{T}}\in\mathbb{R}^{N\times
D}$, a set of new, sparse vector representations, and
$\mathbf{U}=[\mathbf{u}_{1},\ldots,\mathbf{u}_{D}]^{\mathsf{T}}\in\mathbb{R}^{D\times
D^{\prime}}$, a dictionary of _atoms_ found in $\mathbf{Z}$. We solve this
problem with the $k$-SVD algorithm (Aharon et al., 2006, we use an open source
implementation called _pyksvd_ 222https://github.com/hoytak/pyksvd).

The intuition behind this sparse coding approach is as follows. The atoms
$\mathbf{U}$ are intended to represent a wide range of the 2000 most important
features that explain the data in the dense latent space of the model. By
solving this problem we decompose the intermediate representation
$\mathbf{z}_{i}$ of a sample $x_{i}$ into a linear combination
$\mathbf{e}_{i}$ of atoms. By constraining $\mathbf{e}_{i}$ to a fixed and low
sparsity level $k$ we aim to disentangle this representation and therefore
increase interpretability. We refer to this representation as the $k$-SVD
model.

###  2.2 Enforcing Sparsity during Embedding Learning

The $k$-SVD model proposed in the previous section obtains sparse
representations through solving two independent problems: finding a fixed-size
vector representation for sentences with a neural model and, in a separate
step, mapping it to an interpretable, sparse representation. As we mention in
the introduction, we conjecture that interpretability can be further increased
with an end-to-end approach. In this section, we introduce modifications to
the model architecture that will force the neural nets to encode and
understand sparse representations of the data during training time.

We propose an additional layer that is inserted between the encoder and
decoder net. We map the vector representation $\mathbf{z}$ to a vector
$\mathbf{e}$ of the same size in a _sparsity transformation_. The only
requirements for this mapping are that the output $\mathbf{e}$ is sparse and
differentiable (or that we can define a custom gradient) to allow
backpropagation through it. We then feed $\mathbf{e}$ through the decoder net
instead of $\mathbf{z}$ and train the whole net end-to-end. In the rest of
this section, we propose two mappings for such a sparsity transformation.

####  $k$-Sparse.

For this model, we draw inspiration from the _$k$ -Sparse Autoencoder_ by
Makhzani and Frey (2013). We again introduce a hyperparameter $k$ and define
$\mathbf{e}$ by keeping the $k$ largest activations in $\mathbf{z}$, the
_support set_ , and setting all other units to zero. We backpropagate only
through the support set. This is a simple way of enforcing a hard
specification for the sparsity level as an integral part of the model.

#### Sparsemax.

The $k$-Sparse approach has the drawback of requiring a fixed sparsity level
for all samples. To allow for a variable, per-sample sparsity level, we use
the _Sparsemax_ layer, introduced by Martins and Astudillo (2016). Sparsemax
is an alternative to Softmax—however, unlike Softmax, it is able to return
sparse probability distributions. It is defined as:

$\text{Sparsemax}(\mathbf{z})=\arg\min\limits_{\mathbf{p}\in\Delta^{D-1}}||\mathbf{z}-\mathbf{p}||_{2}^{2},$
(2)

where $D$ is the dimensionality of $\mathbf{z}$ and $\Delta^{D-1}$ is the
$(D-1)$-dimensional simplex
$=\\{\mathbf{p}\in\mathbb{R}^{D}\,|\,\mathbf{1}^{\mathsf{T}}\mathbf{p}=1,\mathbf{p}\geq
0\\}$.

Similar to Softmax, Sparsemax supports a temperature mechanism, where a
hyperparameter $\tau$ trades off the “confidence” in the output probability of
the largest input unit. To be precise, as $\tau$ approaches $0$, the
probability distribution
$\text{Sparsemax}\left(\frac{\mathbf{z}}{\tau}\right)$ approaches the
distribution peaked on the maximum components of $\mathbf{z}$. Additionally to
Softmax, Sparsemax has the property that this output distribution becomes
increasingly sparse.

Putting this together, we introduce a hyperparameter $\tau$ and define a
sparsity transformation by defining
$\mathbf{e}=\text{Sparsemax}\left(\frac{\mathbf{z}}{\tau}\right)$.

##  3 Experiments

###  3.1 Training Details

In our experiments, we use a vocabulary size of 20,000, with the symbolic
words <person>, <unk>, and <eos> for names and out-of-vocabulary (OOV) words
in the dataset, and the end-of-sentence marker, respectively. We convert words
to 100-dimensional word embeddings by looking them up in a trainable matrix
$\mathbf{V}$ (note that, in general, this matrix is not sparse—sparsity is
only imposed on the latent space of the sentences333On a side note, sparsity
can be imposed on the word embeddings by adding an L1/regularizer to
$\mathbf{V}$ (Sun et al., 2016; Chen et al., 2017).).

Our encoder and decoder nets are recurrent neural networks that use a single
GRU (Cho et al., 2014) layer. They have the same hidden dimensionality but
share no parameters. We obtain the model predictions as the Softmax of a
learned, affine transformation to 20,000-dimensional space at every time step
of the decoder net. We minimize the mean cross-entropy loss over all
timesteps. We use a batch size of 64 and the Adam optimization algorithm.

###  3.2 Data

We train our models on the Cornell Movie-Dialogs Corpus and MS Common Objects
in Context datasets (respectively Danescu-Niculescu-Mizil and Lee, 2011; Lin
et al., 2014).

The Movie-Dialogs Corpus is a collection of movie lines, therefore it contains
a wide variety of different utterences and allows us to explore general-
purpose sentence embeddings. We preprocess this data by splitting the movie
lines into separate sentences, thereby obtaining more than 500,000 samples.
This dataset has no predefined split; we define a validation and test set by
setting aside 50,000 samples each.

The MS COCO dataset contains images showing scenes with objects in numerous
configurations. Every image contains 5 human-annotated variations of a caption
that describe the scene. In our experiments, we use only these captions and
refer to this as the COCO Captions data. They total over 600,000 samples and
allow us to explore sentence embeddings of a more narrow language: since they
merely describe objects and scenes, they tend to follow the same, simple
sentence structure. The dataset comes with a predefined training/validation
split.

For tokenizing and splitting movie lines into sentences we use the NLP library
SpaCy444https://spacy.io. All our models are implemented in TensorFlow (Abadi
et al., 2015).

original |  a room with blue walls and a white sink and door .
---|---
reconstruction |  a room with blue walls and a white sink and windows .
original |  two cars parked on the sidewalk on the street
reconstruction |  two buses parked on the curb on the street
original |  two women waiting at a bench next to a street .
reconstruction |  two women sit at a park next to a street .
original |  a car that seems to be parked illegally behind a legally parked car
reconstruction |  a car that seems to be parked close to a police officer and talking
original |  a bathroom sink and various personal hygiene items .
reconstruction |  a bathroom sink and various other hygiene items .
original |  this is an open box containing four cucumbers .
reconstruction |  this is an open box makes delicious doughnuts .
Table 1: Typical sentence reconstruction errors by the $k$-Sparse, $k=15$
model, trained on the COCO Captions data.

##  4 A Quantitative and Automated Evaluation Metric

The most common quantitative interpretability measure for embeddings (in
particular word embeddings) is the intrusion test, first introduced in (Chang
et al., 2009). This test involves generating 5-tuples of samples, where
according to the embeddings model four are related and one stands out. The
better human judges identify the intruder, the more interpretable the model is
considered.

This evaluation method has the drawback of requiring human attention, thereby
it is expensive and slow to evaluate. For our evaluation, we introduce an
automated interpretability test, based on topic coherence, that does not
require human attention. We describe our method in this section.

A _topic model_ defines a set of topics in a corpus of documents and allows us
to find the top $n$ most likely words that belong to each topic. _Topic
coherence_ is an automated evaluation method of the interpretability of topic
models, which has been shown to correlate well with human assessments (Newman
et al., 2010; Mimno et al., 2011). Given a symmetric similarity measure of two
words (e.g. pointwise mutual information), the coherence of a topic is defined
as the mean pairwise similarity of all pairs of words. The total topic
coherence of the model is the mean coherence over all topics.

We devise an evaluation scheme based on topic coherence. Instead of looking at
words in topics, we consider the highest-ranked sentences in the dimensions of
our embeddings and replace the word similarity measure with a sentence
similarity measure. Let $x_{d}^{(p)}$ be the sample that has rank $p$ in the
order given by the $d$-th dimension in the embedding. For a similarity measure
$sim_{*}$, the coherence of a single dimension $d$ is defined as:

$\begin{multlined}coh_{*}(d)=\frac{2}{n\cdot(n-1)}\sum_{p=1}^{n-1}\sum_{q=p+1}^{n}sim_{*}(x_{d}^{(p)},x_{d}^{(q)}).\end{multlined}coh_{*}(d)=\frac{2}{n\cdot(n-1)}\sum_{p=1}^{n-1}\sum_{q=p+1}^{n}sim_{*}(x_{d}^{(p)},x_{d}^{(q)}).$
(3)

The coherence of the model is defined as the mean coherence over all
dimensions:

$coh_{*}(1,\ldots,D)=\frac{1}{D}\sum_{d}coh_{*}(d).$ (4)

In addition, to determine how much the coherence deteriorates when looking
beyond the top ranks, we consider all non-zero samples of a dimension and we
evaluate Equation 3 on $n$ sentences sampled at random and without replacement
from $\\{x_{i}\mid e_{i,d}\neq 0\\}$ instead of
$x_{d}^{(1)},\ldots,x_{d}^{(n)}$.

We compute this on the validation set of our data. We strip all stop words
from all sentences. We consider $n=10$ sentences per dimension, unless a
dimension has a non-zero value for less than $n$ samples, in which case we
compute Equation 3 on all pairs of sentences. In the following, we define
three choices for a sentence similarity measure $sim_{*}$.

##### Jaccard Similarity.

We regard the sentences as sets of words and compute the Jaccard similarity:

$sim_{\text{J}}(x_{i},x_{j})=\frac{|x_{i}\cap x_{j}|}{|x_{i}\cup x_{j}|}.$ (5)

##### BoW Similarity.

We consider the Bag-of-Words (BoW) vectors $\mathbf{b}_{i},\mathbf{b}_{j}$ of
the two sentences, i.e. the vectors with the number of occurrences of each
vocabulary word in $x_{i},x_{j}$, respectively. The similarity is defined as
the cosine of the angle between these vectors:

$sim_{\text{BoW}}(x_{i},x_{j})=\frac{\mathbf{b}_{i}^{\mathsf{T}}\mathbf{b}_{j}}{||\mathbf{b}_{i}||_{2}\,||\mathbf{b}_{j}||_{2}}.$
(6)

##### WMD Similarity.

The Jaccard and BoW similarity measures have a drawback in that they do not
take semantic relatedness of different words into account. The _Word Mover’s
Distance_ (WMD; Kusner et al., 2015) remedies this problem: the authors define
a document distance measure that relies on the word2vec latent space to make a
better assessment of the semantic distance of sentences, based on the distance
of the words they consist of. We use the negative WMD to obtain a similarity
measure:

$sim_{\text{WMD}}(x_{i},x_{j})=-\text{WMD}(x_{i},x_{j}).$ (7)

##  5 Results

###  5.1 Reconstruction Quality

$e_{i,d_{1}}$ |  $x_{i}$
---|---
$1.00$ |  a person laying on a couch with a laying on him
$1.00$ |  a cat laying on top of a suitcase laying on the floor .
$1.00$ |  a man laying on top of a sandy beach laying next to a surfboard .
$1.00$ |  a person laying on a couch with a cat laying in their arms, covering part of the face .
$1.00$ |  a woman is laying on a couch with a boy laying his head on her belly, and a cat between her legs .
$1.00$ |  some cats laying on a dock with their chins laying over the end
$1.00$ |  a man laying in bed with a gray cat laying on top of him .
$1.00$ |  a number of cows laying in a lot near cars
$1.00$ |  a number of items laying on a surface near one another
$1.00$ |  two cows laying out together underneath a tree .
(a) This dimension clearly corresponds to sentences that describe a
configuration of an object laying on another, whether that be people on the
couch or items on a surface. Coherence score: $coh_{\text{WMD}}(d_{1})=-2.32$.

$e_{i,d_{2}}$ |  $x_{i}$
---|---
$0.94$ |  a motorcycle parked outside the doors of a building
$0.94$ |  a blue motorcycle parked outside of a building .
$0.94$ |  traffic lights on the road showing the street
$0.93$ |  food in a bowl sitting on a table
$0.92$ |  a yellow train in an outside train station .
$0.92$ |  a motorcycle sits on a sidewalk near a building
$0.92$ |  a car that is outside in the dirt .
$0.92$ |  a red truck parked outside in the snow .
$0.91$ |  a boy sitting on a bench at the park
$0.91$ |  a black motorcycle is parked on a sidewalk
(b) This dimension seems to capture, with some false positives, different
kinds of motor vehicles (motorcycle, train, car, truck) that are parked (sit,
sitting, is outside) somewhere. Coherence score:
$coh_{\text{WMD}}(d_{2})=-2.83$.

$e_{i,d_{3}}$ |  $x_{i}$
---|---
$0.79$ |  herd of goats in grassy area with herder .
$0.64$ |  herd of five zebras grazing in a field
$0.63$ |  people are sitting in lounge chairs on the beach .
$0.63$ |  a close up of many large kites near the ground
$0.63$ |  cows lounge in a field with a mountain backdrop .
$0.62$ |  close up of the flower extending from a banana tree stalk
$0.61$ |  a group of object on top of a muddy river .
$0.61$ |  many plants and umbrellas on the side of the street .
$0.58$ |  a close up view of sheets that are on a bed
$0.58$ |  room with cramped quarters holding dining table set and extra chairs .
(c) It is not clear which features this dimension captures. Coherence score:
$coh_{\text{WMD}}(d_{3})=-3.12$.

Table 2: Examples of selected dimensions $d_{1},d_{2},d_{3}$ of our
$k$-Sparse, $k=15$ model, trained on the COCO Captions data. We show the 10
highest-ranked samples $x_{i}$ and the coherence $coh_{\text{WMD}}(d)$ of each
dimension $d$. We give more examples of high-coherence dimensions in the
appendix, in Table 5.

We start off by looking at the amount of information lost by our models due to
sparsity constraints.

In general, we observe that as the sparsity level is decreased, the
reconstructions start to deteriorate. At low values of $k$, our sparse models
often fail to restore the exact meaning or phrasing, but still generate
sentences with correct grammar and related topics. For example, they turn
“waiting at a bench” into “sit at a park”, “sidewalk” into “curb”, “two cars”
into “two buses”, and similar. The $k$-SVD model generally does this less than
the other models but in some cases it fails as well. See examples of typical
reconstruction errors by our $k$-Sparse, $k=15$ model in Table 1.

###  5.2 Highest-Ranked Samples

We examine the top samples in the dimensions of our embedding models and
observe that sparse models often group sentences s.t. they have a common
syntactic element or talk about a common concept. For example, in our
$k$-Sparse, $k=15$ model trained on the COCO Captions dataset, we identify
dimensions that represent sentences about objects in water, people holding
things, horse (and occasionally bicycle) riders, sentences starting with
common prefixes such as there is a […], etc. We give examples in Table 2 and
in the appendix in Table 5. For some dimensions, this pattern is not only
recognizable in the top ranks but for all samples $x_{i}$ with $e_{i,d}\neq
0$.

We are able to find such patterns in all sparse models, but the lower the
sparsity level, the more apparent these patterns become. $k$-SVD based models
exhibit these properties to a lesser extent.

Embeddings model | Jaccard | BoW | WMD
---|---|---|---
COCO Captions | $0.05$ | $0.10$ | $-3.12$
Movie-Dialogs | $0.08$ | $0.16$ | $-2.06$
(a) Mean similarity of random sentences

| Top 10 samples | Random 10 samples
---|---|---
Embeddings model | Jaccard | BoW | WMD | Jaccard | BoW | WMD
dense, 500 dim. AE | $0.08$ | $0.14$ | $-3.00$ | $0.06$ | $0.10$ | $-3.12$
$k$-SVD, $k=100$ | $0.07$ | $0.12$ | $-3.08$ | $0.06$ | $0.10$ | $-3.11$
$k$-SVD, $k=50$ | $0.08$ | $0.13$ | $-3.03$ | $0.06$ | $0.11$ | $-3.10$
$k$-SVD, $k=20$ | $0.11$ | $0.18$ | $-2.88$ | $0.06$ | $0.11$ | $-3.08$
$k$-SVD, $k=15$ | $\mathbf{0.11}$ | $\mathbf{0.19}$ | $-2.86$ | $0.07$ | $0.12$ | $-3.06$
$k$-Sparse, $k=100$ | $0.08$ | $0.14$ | $-3.02$ | $0.06$ | $0.11$ | $-3.09$
$k$-Sparse, $k=50$ | $0.09$ | $0.15$ | $-2.96$ | $0.07$ | $0.12$ | $-3.06$
$k$-Sparse, $k=20$ | $0.11$ | $0.17$ | $\mathbf{-2.85}$ | $0.08$ | $0.14$ | $\mathbf{-3.00}$
$k$-Sparse, $k=15$ | $0.11$ | $0.18$ | $-2.86$ | $\mathbf{0.08}$ | $\mathbf{0.14}$ | $-3.01$
Sparsemax, $\tau=50$ | $0.04$ | $0.07$ | $-3.25$ | $0.03$ | $0.06$ | $-3.27$
Sparsemax, $\tau=20$ | $0.04$ | $0.06$ | $-3.29$ | $0.03$ | $0.05$ | $-3.35$
Sparsemax, $\tau=10$ | $0.04$ | $0.07$ | $-3.25$ | $0.03$ | $0.06$ | $-3.31$
(b) COCO Captions dataset

| Top 10 samples | Random 10 samples
---|---|---
Embeddings model | Jaccard | BoW | WMD | Jaccard | BoW | WMD
dense, 500 dim. AE | $0.20$ | $0.31$ | $-1.85$ | $0.09$ | $0.16$ | $-2.02$
$k$-SVD, $k=100$ | $0.17$ | $0.24$ | $-1.99$ | $0.09$ | $0.16$ | $-2.01$
$k$-SVD, $k=50$ | $0.17$ | $0.24$ | $-2.01$ | $0.10$ | $0.16$ | $-2.01$
$k$-SVD, $k=20$ | $0.20$ | $0.28$ | $-1.91$ | $0.11$ | $0.18$ | $-2.01$
$k$-SVD, $k=15$ | $0.20$ | $0.29$ | $-1.88$ | $0.12$ | $0.19$ | $-1.98$
$k$-Sparse, $k=100$ | $0.16$ | $0.25$ | $-2.01$ | $0.10$ | $0.18$ | $-2.08$
$k$-Sparse, $k=50$ | $0.16$ | $0.25$ | $-1.95$ | $0.11$ | $0.19$ | $-2.05$
$k$-Sparse, $k=20$ | $0.19$ | $0.30$ | $-1.82$ | $0.13$ | $0.22$ | $-1.99$
$k$-Sparse, $k=15$ | $\mathbf{0.22}$ | $\mathbf{0.33}$ | $\mathbf{-1.76}$ | $0.14$ | $\mathbf{0.23}$ | $-1.98$
Sparsemax, $\tau=50$ | $0.12$ | $0.19$ | $-2.13$ | $0.12$ | $0.19$ | $-2.02$
Sparsemax, $\tau=20$ | $0.13$ | $0.21$ | $-2.02$ | $\mathbf{0.16}$ | $0.23$ | $\mathbf{-1.89}$
Sparsemax, $\tau=10$ | $0.15$ | $0.22$ | $-2.01$ | $0.15$ | $0.22$ | $-1.96$
(c) Movie-Dialogs dataset

Table 3: Interpretability of our models, as measured by our topic-coherence-
based metric in Equation 4. We evaluate this equation using three different
notions of sentence similarity $sim_{*}$. In Equation 3, we consider 10 random
non-zero samples in addition to the 10 highest-ranked samples.

###  5.3 Quantitative Evaluation

In Table 2 we additionally report the coherence $coh_{\text{WMD}}(d)$ of the
presented dimensions $d$ (see Equations 3, 7). We observe that this score
correlates with our empirical assessment of the interpretability of the
dimension. For example, we observe on the COCO dataset that, while unrelated
groups of sentences usually have a coherence score of $<-3$, sentences with
common or semantically related subjects and objects have higher coherence
scores (usually between $-2.8$ and $-2.2$). Groups of sentences with very
close semantic meaning or large common prefixes have coherence scores around
$-2$ or higher.

We report the topic coherence of our models (Equation 4) in Table 3. As rough
reference values for the metrics, we include the mean similarity of pairs of
random sentences from the dataset (estimated on 500 randomly sampled pairs),
and the topic coherence of a 500-dimensional dense autoencoder model.

In accordance with our empirical observations, we see an increase in
interpretability in the sparse models. For example, on the COCO Captions data,
a random pair of sentences has a WMD-based similarity of -3.12, and the WMD-
based coherence score of a dense autoencoder model is -3. With the additional
sparse coding step on top of that, we can increase the coherence to -2.86.

###  5.4 Downstream Tasks

Embeddings model | CR | MR | SUBJ | MPQA | SST2 | SST5 | TREC | SICK-E | SICK-R | STS14 | MRPC
---|---|---|---|---|---|---|---|---|---|---|---
dense, 500 dim. AE | $65.99$ | $\mathbf{59.37}$ | $76.24$ | $\mathbf{73.01}$ | $60.63$ | $28.96$ | $\mathbf{77.60}$ | $\mathbf{75.50}$ | $\mathbf{0.61}$ | $\mathbf{0.42}$ | $67.25$
$k$-SVD, $k=100$ | $60.48$ | $54.63$ | $69.61$ | $70.73$ | $59.58$ | $25.07$ | $68.20$ | $56.36$ | $0.34$ | $0.18$ | $59.65$
$k$-SVD, $k=50$ | $62.54$ | $55.01$ | $70.47$ | $70.70$ | $57.66$ | $25.84$ | $69.80$ | $58.09$ | $0.34$ | $0.17$ | $60.58$
$k$-SVD, $k=20$ | $62.41$ | $55.53$ | $70.60$ | $71.16$ | $58.76$ | $25.20$ | $70.40$ | $60.26$ | $0.33$ | $0.17$ | $59.94$
$k$-SVD, $k=15$ | $62.91$ | $55.48$ | $70.63$ | $71.25$ | $57.33$ | $23.89$ | $70.20$ | $59.71$ | $0.33$ | $0.16$ | $61.04$
$k$-Sparse, $k=100$ | $65.22$ | $56.09$ | $\mathbf{76.47}$ | $72.04$ | $58.98$ | $27.69$ | $72.80$ | $70.33$ | $0.56$ | $0.37$ | $66.72$
$k$-Sparse, $k=50$ | $64.64$ | $57.13$ | $74.74$ | $71.51$ | $59.86$ | $27.42$ | $73.80$ | $71.36$ | $0.55$ | $0.32$ | $66.38$
$k$-Sparse, $k=20$ | $64.53$ | $55.98$ | $73.00$ | $71.65$ | $58.43$ | $26.24$ | $75.60$ | $68.20$ | $0.50$ | $0.25$ | $67.48$
$k$-Sparse, $k=15$ | $\mathbf{67.63}$ | $58.24$ | $75.52$ | $71.87$ | $\mathbf{62.16}$ | $\mathbf{30.14}$ | $76.80$ | $72.50$ | $0.55$ | $0.23$ | $65.45$
Sparsemax, $\tau=50$ | $64.58$ | $54.60$ | $66.33$ | $69.13$ | $55.46$ | $27.69$ | $65.80$ | $64.28$ | $0.53$ | $0.19$ | $\mathbf{67.88}$
Sparsemax, $\tau=20$ | $63.58$ | $54.58$ | $66.45$ | $70.21$ | $52.94$ | $26.92$ | $64.20$ | $63.95$ | $0.49$ | $0.19$ | $66.90$
Sparsemax, $\tau=10$ | $63.44$ | $54.60$ | $63.07$ | $69.29$ | $54.53$ | $26.92$ | $61.40$ | $63.02$ | $0.48$ | $0.17$ | $66.49$
Table 4: Evaluation of the embeddings from Movie-Dialogs models on various
downstream tasks. The values measure classification accuracy or spearman
correlation with human-labeled ground truth (see Section 5.4); larger values
are better.

Additionally to the interpretability properties of sparse sentence embeddings,
it is of interest whether sparsity decreases their usefulness in downstream
tasks. To evaluate this, we use the _SentEval_ framework (Conneau and Kiela,
2018), which learns downstream models on top of the provided sentence
embeddings to solve a variety of transfer tasks.

We report the accuracy on the standard classification problems the framework
provides, namely binary sentiment of movie reviews (MR), movie lines (SST2)
and product reviews (CR), five-class sentiment of movie lines (SST5),
subjectivity/objectivity (SUBJ), binary opinion polarity (MPQA), and six-class
question type (TREC) classification. To look at semantic entailment/similarity
of pairs of sentences, we report Spearman correlation with human-labelled
ground truth on the five-class semantic relatedness (STS14, SICK-R), and
three-class semantic entailment (SICK-E) tasks, and accuracy on the binary
paraphrase detection (MRPC) task.

We configure the framework to use Logistic Regression for downstream models.
More details on the tasks and evaluation methods can be found in the SentEval
paper. We evaluate these tasks on the Movie-Dialogs models only, because COCO
is unsuited for general-purpose sentence embeddings.

We show the results of this evaluation in Table 4. We again observe that
sparse representations perform, in many cases, worse than their dense
equivalent, therefore, trading quality for interpretability555On a side note,
we address the noticeable fact that the transfer tasks are solved with low
accuracy in general. For numbers comparable to the state of the art
literature, more powerful sentence embedding models (such as self-attentive
networks, InferSent, SkipThought etc., see Conneau et al., 2017; Kiros et al.,
2015) with a higher latent dimensionality, more layers, and a larger and more
diverse dataset are required. Further, SentEval provides slower but more
powerful MLP downstream models instead of Logistic Regression.. However, this
does not occur on all tasks: for example, SST2 and SST5 clearly benefit from a
sparse representation.

###  5.5 Discussion

The results of our quantitative evaluation method confirm the tendencies we
observed in our empirical evaluation. It appears that embedding dimensions
generated by sparse models are coherent to a higher extent—in particular, the
lower the sparsity level, the more apparent topics can be found in the
embedding dimensions.

The price of good interpretability is a higher reconstruction error. As we
impose more sparsity in the representation, the model is forced to “cut
corners” and single slots in the embedding are designated broader collections
of traits in the data. This results in more coherent topics, however, the
narrow information bandwidth reduces the decoder net’s ability to reconstruct
the exact sentence. The fact that sparse representations carry less
information may also explain the lower utility in some of the downstream
tasks. Other tasks (e.g. sentiment classification) can be solved with greater
accuracy, which suggests that a sparse and interpretable representation
discovers more useful features for a simple downstream model like Logistic
Regression.

As we force a model to deal with a sparse representation already during the
training phase, finding sensible atoms gets incorporated into the encoding and
decoding mechanism. We found that, in comparison to extracting this
information from a dense model’s intermediate representation, this results in
an observable and measurable boost in interpretability. We can explain this by
the fact that this architecture makes it _part of the model’s task_ to find a
sparse and accurate representation of the data, whereas in a post-processing
approach the model focuses on reconstruction only.

On the other side of the coin, this modification interferes with the training
process. To be more precise, the model follows a more complex objective, and
the sparse layer is limited in the amount of information that can be forward
and backpropagated through it at a time—hence we observe convergence at a
higher loss value and bigger reconstruction errors.

We note that our Sparsemax-based approach does not perform particularly well
in our evaluation, although in some cases it outperforms other approaches when
we consider samples beyond the 10 highest-ranked. This can be explained by the
fact that the sparsity level is not fixed, and that due to the Sparsemax
layer, the embeddings $\mathbf{e}_{i}$ are valid probability distributions. A
high value $e_{i,d}$ does, therefore, not necessarily indicate a strong
presence of feature $d$ in sample $i$, but also a lack of other features. On
the other hand, Sparsemax is better at determining feature presence/absence in
general, due to not being constrained to find an exact number of features.

##  6 Related Work

As aforesaid, there has been work in the NLP literature on the
interpretability of word embeddings. Murphy et al. (2012) suggest that sparse
embeddings can be linked to a disentangled, and thus interpretable,
representations. This idea is also applied in (Arora et al., 2016; Faruqui et
al., 2015; Subramanian et al., 2017), commonly by solving a sparse dictionary
learning problem on top of dense embeddings. In the papers (Sun et al., 2016;
Chen et al., 2017; Luo et al., 2015), the authors learn sparse word embeddings
during the training phase. Goh (2016) applies above-named approaches to image
embeddings, and the intermediate representation of an image captioning model.

Makhzani and Frey (2013) define the $k$-Sparse autoencoder. They use a
$k$-Sparse layer in a shallow autoencoder trained on the MNIST and NORB
datasets, focusing on unsupervised feature learning, improvement in
classification accuracy, and a fast alternative to sparse coding. Martins and
Astudillo (2016) develop Sparsemax as an alternative to the Softmax layer that
is able to output exactly zero probabilities, their work is focused on
classification problems and attention mechanisms.

Interpretability metrics are usually of interest for word embeddings, where
the predominant evaluation method is the word intrusion test (Chang et al.,
2009). Our interpretability metric is based on topic coherence (Newman et al.,
2010), a comparison of different variants of this method can be found in
(Röder et al., 2015).

##  7 Conclusion

Being able to understand the intermediate representation of a neural net
increases our model understanding. In this paper we have taken a step towards
this goal by introducing several sparse methods for a sentence autoencoder,
inspired by previous work on word embeddings. The evaluation of our proposed
models supports our hypothesis that sparse methods benefit the
interpretability of the embedding. It is intuitive that a vector restricted to
many zero values inevitably carries less information, and indeed we have found
that this increase in interpretability comes at a cost in reconstruction
quality and, in some cases, utility in downstream tasks. It is, however,
possible to strike a balance and achieve good interpretability without a large
penalty.

We have devised a novel, automated method of quantifying said
interpretability, based on topic coherence. In our experiments, we observe
that this evaluation corresponds to our manual assessment of interpretability.
It is fully automated, and therefore cheap and fast to run. It can easily be
extended by using different sentence similarity metrics or other topic
coherence variants.

An interpretable sentence representation has further applications beyond model
understanding: for example, it allows us to develop a sentence similarity
measure, that can justify _why_ two sentences are similar. It can also help us
understand downstream models on top of sentence embeddings. For example,
consider the case of a linear classification model: we can inspect the largest
positive and negative weights and understand which features in a source
sentence influence the model’s decision.

For future work, it suggests itself to apply sparsity constraints to more
sophisticated sentence embedding models such as _SkipThought_ or _InferSent_
(respectively Kiros et al., 2015; Conneau et al., 2017). Our methods can also
be used to construct sparse encoder-decoder models for further tasks, such as
image captioning, machine translation, or recommender systems.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Aharon et al. (2006) Michal Aharon, Michael Elad, and Alfred Bruckstein. 2006.  $rmk$-svd: An algorithm for designing overcomplete dictionaries for sparse representation.  _IEEE Transactions on signal processing_ , 54(11):4311–4322.
  * Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016.  Linear algebraic structure of word senses, with applications to polysemy.  _arXiv preprint arXiv:1601.03764_.
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. 2009.  Reading tea leaves: How humans interpret topic models.  In _Advances in neural information processing systems_ , pages 288–296.
  * Chen et al. (2017) Yunchuan Chen, Ge Li, and Zhi Jin. 2017.  Learning sparse overcomplete word vectors without intermediate dense representations.  In _International Conference on Knowledge Science, Engineering and Management_ , pages 3–15. Springer.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  Senteval: An evaluation toolkit for universal sentence representations.  _arXiv preprint arXiv:1803.05449_.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017\.  Supervised learning of universal sentence representations from natural language inference data.  _arXiv preprint arXiv:1705.02364_.
  * Danescu-Niculescu-Mizil and Lee (2011) Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011.  Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011_.
  * Faruqui et al. (2015) Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. 2015\.  Sparse overcomplete word vector representations.  _arXiv preprint arXiv:1506.02004_.
  * Goh (2016) Gabriel Goh. 2016.  Decoding the thought vector.  http://gabgoh.github.io/ThoughtVectors/.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Skip-thought vectors.  In _Advances in neural information processing systems_ , pages 3294–3302.
  * Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015.  From word embeddings to document distances.  In _International Conference on Machine Learning_ , pages 957–966.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Luo et al. (2015) Hongyin Luo, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2015.  Online learning of interpretable word embeddings.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1687–1692.
  * Makhzani and Frey (2013) Alireza Makhzani and Brendan Frey. 2013.  K-sparse autoencoders.  _arXiv preprint arXiv:1312.5663_.
  * Martins and Astudillo (2016) Andre Martins and Ramon Astudillo. 2016.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  In _International Conference on Machine Learning_ , pages 1614–1623.
  * Mimno et al. (2011) David Mimno, Hanna M Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011.  Optimizing semantic coherence in topic models.  In _Proceedings of the conference on empirical methods in natural language processing_ , pages 262–272. Association for Computational Linguistics.
  * Murphy et al. (2012) Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012.  Learning effective and interpretable semantic models using non-negative sparse embedding.  _Proceedings of COLING 2012_ , pages 1933–1950.
  * Newman et al. (2010) David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010.  Automatic evaluation of topic coherence.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 100–108. Association for Computational Linguistics.
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015.  Exploring the space of topic coherence measures.  In _Proceedings of the eighth ACM international conference on Web search and data mining_ , pages 399–408. ACM.
  * Subramanian et al. (2017) Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. 2017.  Spine: Sparse interpretable neural embeddings.  _arXiv preprint arXiv:1711.08792_.
  * Sun et al. (2016) Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng. 2016.  Sparse word embeddings using l1 regularized online learning.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , pages 2915–2921. AAAI Press.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in neural information processing systems_ , pages 3104–3112.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on_ , pages 3156–3164. IEEE.

$e_{i,d_{4}}$ |  $x_{i}$
---|---
$0.98$ |  a black cat drinking water out of a water faucet .
$0.98$ |  the boats are outside on the water sailing .
$0.98$ |  several cows drinking water from a water receptacle .
$0.98$ |  a boat speeds down open water spraying water behind it .
$0.98$ |  two elephants drink water out of a body of water
$0.98$ |  a large body of water covered with boats .
$0.98$ |  a person stands on water skis in the water .
$0.98$ |  a woman is on the water on water skis .
$0.98$ |  small boats on water with setting sun behind distant hills .
$0.98$ |  a power boat on a body of water with a large water spray behind .
(a) $coh_{\text{WMD}}(d_{4})=-2.21$

$e_{i,d_{5}}$ |  $x_{i}$
---|---
$0.99$ |  white vase holding holding an assortment of flowers
$0.99$ |  a man holding holding a tennis racquet on a tennis court .
$0.99$ |  a man holding holding a giant remote control .
$0.99$ |  two bears holding each other outside the surroundings .
$0.99$ |  snowboarder holding a pink board being hugged by man in costume .
$0.99$ |  baby holding a teething toy in his hand
$0.99$ |  a countertop holding a <unk> bowl across from a shelf holding stemware .
$0.99$ |  a bird holding a fish in it’s mouth .
$0.99$ |  a oven holding two trays of food baking .
$0.99$ |  two glasses holding red wine sit on a piece of paper on a wooden surface .
(b) $coh_{\text{WMD}}(d_{5})=-2.77$

$e_{i,d_{6}}$ |  $x_{i}$
---|---
$0.97$ |  person riding their skateboard on the street with the cars .
$0.94$ |  person riding a skateboard while pushing a stroller
$0.94$ |  person riding a horse while the sun sets
$0.93$ |  person riding a horse while another horse stands in a field .
$0.92$ |  person riding a bicycle while walking two dogs .
$0.92$ |  person riding a four wheeler on a beach towards a bridge .
$0.91$ |  person riding an elephant as it crosses through a river .
$0.91$ |  person riding a horse along shore of a body of water .
$0.91$ |  a person riding their bike down a path to a gate with a stop sign .
$0.91$ |  person riding down snowy hill on a pair of skis
(c) $coh_{\text{WMD}}(d_{6})=-2.21$

$e_{i,d_{7}}$ |  $x_{i}$
---|---
$0.99$ |  there is a brown box on the toilet
$0.99$ |  there is a blender with a green mixture in it
$0.99$ |  there is a brown bear walking through the woods alone
$0.99$ |  there is a clock that is above the building doors
$0.99$ |  there is a clock inside of a curvy blue sculpture .
$0.99$ |  there is a truck that has something mounted on the top
$0.99$ |  there is a boy playing with a tie
$0.99$ |  there is a person playing a nintendo wii
$0.99$ |  there is a boy playing baseball at the base ball field
$0.99$ |  there is a clock on the wall between the two arches .
(d) $coh_{\text{WMD}}(d_{7})=-3.47$. Note that this dimension has low
coherence because the common feature it brings out (there is a) consists of
stop words, which are not considered in our metrics.

$e_{i,d_{8}}$ |  $x_{i}$
---|---
$1.00$ |  a person is surfing on a shallow wave .
$1.00$ |  a person is surfing on a medium sized wave .
$1.00$ |  a person is surfing in a on a wave
$1.00$ |  a person is surfing on a wave in the ocean .
$1.00$ |  a person is surfing a huge wave while staying upright .
$1.00$ |  a person is surfing on the waves of an empty ocean .
$1.00$ |  a person is surfing on a board at the beach
$1.00$ |  a person is surfing in a wave pool .
$1.00$ |  a person is surfing on a wave at the beach
$1.00$ |  a person is surfing a wave on a surfboard .
(e) $coh_{\text{WMD}}(d_{8})=-1.47$

Table 5: Highest-ranked samples in a selection of dimensions of our
$k$-Sparse, $k=15$ model, trained on the COCO Captions data, along with the
coherence of the dimension.


# Long-Term CSI-based Design for RIS-Aided Multiuser MISO Systems Exploiting
Deep Reinforcement Learning

Hong Ren, Cunhua Pan, Liang Wang, Zhoubing Kou, and Kezhi Wang (Corresponding
author: Cunhua Pan) H. Ren and Zhoubing Kou are with Southeast University,
Nanjing 210096, China. (e-mail: hren, zbkou@seu.edu.cn). C. Pan is with Queen
Mary University of London, London E1 4NS, U.K. (e-mail: c.pan@qmul.ac.uk). K.
Wang and Liang Wang are with Northumbria University, UK. (e-mail: kezhi.wang,
Liang.wang@northumbria.ac.uk).

###### Abstract

In this paper, we study the transmission design for reconfigurable intelligent
surface (RIS)-aided multiuser communication networks. Different from most of
the existing contributions, we consider long-term CSI-based transmission
design, where both the beamforming vectors at the base station (BS) and the
phase shifts at the RIS are designed based on long-term CSI, which can
significantly reduce the channel estimation overhead. Due to the lack of
explicit ergodic data rate expression, we propose a novel deep deterministic
policy gradient (DDPG) based algorithm to solve the optimization problem,
which was trained by using the channel vectors generated in an offline manner.
Simulation results demonstrate that the achievable net throughput is higher
than that achieved by the conventional instantaneous-CSI based scheme when
taking the channel estimation overhead into account.

###### Index Terms:

Reconfigurable intelligent surface (RIS), intelligent reflecting surface
(IRS), deep reinforcement learning.

##  I Introduction

Due to its appealing features of low hardware cost and energy consumption,
reconfigurable intelligent surface (RIS) has been widely regarded as one of
the promising technologies in the future wireless networks [1, 2, 3]. To reap
the benefits brought by the RIS, the active beamforming at the base station
(BS) and phase shifts at the RIS should be carefully designed. Hence,
extensive research efforts have been devoted to the transmission design for
RIS-aided various wireless communication systems, such as multicell networks
[4], simultaneous wireless information and power transfer (SWIPT) [5],
physical layer security [6], and mobile edge computing [7]. The above
contributions were based on the assumption that perfect instantaneous channel
state information (CSI) is available at the BS, which will entail excessive
amount of channel estimation overhead that is proportional to the number of
reflecting elements.

One alternative method to address the above challenge is to adopt the long-
term CSI [8, 9, 10, 11, 12, 13], which varies much more slowly than the
instantaneous CSI and is relatively easy to obtain. Specifically, the
contributions in [8, 9, 10, 11, 12] focused on the two-timescale design, in
which the phase shifts at the RIS were designed based on the long-term CSI
such as angle information or/and channel covariance matrices, while the BS
still needs the instantaneous aggregate CSI for the active beamforming design.
Most recently, the authors in [13] proposed a transceiver design, where both
the active beamforming at the BS and phase shifts were designed based on the
long-term CSI. However, only one user was considered in [13], and closed-form
solution can be readily derived. Most recently, the authors studied the
ergodic sum rate maximization problem in [14, 15] for multi-user communication
systems based on fully long-term CSI and Jensen’s inequality was adopted to
approximate the ergodic data rate expression. However, the fairness issue
cannot be guaranteed in [14, 15].

Against the above background, the main contributions of this paper are
summarized as follows:

  * •

We first consider the fairness design for RIS-aided multi-user communications
based on long-term CSI with the aim of maximizing the minimum ergodic user
data rate.

  * •

Unlike the single-user case in [13], the explicit closed-form expression of
the ergodic data rate cannot be obtained. In addition, the max-min objective
function is non-differentiable. To address this issue, we propose a novel deep
deterministic policy gradient (DDPG) based algorithm to solve the optimization
problem. In specific, we first estimate the long-term CSI. Then, we generate a
set of channels based on the Rician distribution, which were used to train the
DDPG networks. The solution produced by the DDPG networks can then be used in
the remaining channel coherence time intervals where the long-term CSI keeps
fixed.

##  II System Model

Consider a downlink MISO system where an $M$-antenna BS communicates with $K$
single-antenna users. An RIS with $N$ reflecting elements is deployed between
the users and the BS to improve the communication quality. The RIS
configuration matrix can be denoted as
$\mathbf{\Phi}=\text{diag}\left\\{\phi_{1},\phi_{2},\cdots,\phi_{N}\right\\}$,
where $\phi_{n}$ is the phase shift of the $n$-th element and
$|\phi_{n}|=1,n=1,2,\cdots,N$. Let us denote $\mathbf{h}_{k}$ and
$\mathbf{g}_{k}$ as the channel vector between the $k$-th user and the RIS and
the direct channel between the $k$-th user and the BS, respectively. The
channel matrix between the BS and the RIS is denoted as $\mathbf{G}$.

The beamforming vector for the $k$-th user is denoted as $\mathbf{w}_{k}$. The
set of all beamforming vectors is denoted as
$\mathbf{W}=\left[{{{\bf{w}}_{1}},\cdots,{{\bf{w}}_{K}}}\right]$. Then, the
transmitted complex baseband signal $\mathbf{x}\in\mathbb{C}^{M\times 1}$ is
given by $\mathbf{x}=\mathbf{W}\mathbf{s}$, where
$\mathbf{s}\in\mathbb{C}^{K\times 1}=[s_{1},s_{2},\dots,s_{K}]^{T}$ denotes
data streams to $K$ users, and
${\mathbb{E}}\\{{\bf{s}\bf{s}}^{H}\\}={\bf{I}}_{K}$.

The received signal of the $k$-th user is given by

$y_{k}=({\mathbf{h}_{k}^{T}\mathbf{\Phi}\mathbf{G}}+{\mathbf{g}_{k}^{T}})\sum\nolimits_{j=1}^{K}\mathbf{w}_{j}s_{j}+n_{k},$
(1)

where $n_{k}\sim\mathcal{CN}(0,\sigma^{2})$ is the additive white Gaussian
noise (AWGN) at the $k$-th user.

The Rician channel model is adopted, and thus all the channels can be
represented as

$\displaystyle\mathbf{G}$
$\displaystyle=\sqrt{\kappa}{\left(\sqrt{\frac{\delta}{\delta+1}}\bar{\mathbf{G}}+\sqrt{\frac{1}{\delta+1}}\tilde{\mathbf{G}}\right)},$
(2) $\displaystyle\mathbf{g}_{k}$
$\displaystyle=\sqrt{\beta_{k}}\left(\sqrt{\frac{\varepsilon_{k}}{\varepsilon_{k}+1}}\bar{\mathbf{g}}_{k}+\sqrt{\frac{1}{\varepsilon_{k}+1}}\tilde{\mathbf{g}}_{k}\right),$
(3) $\displaystyle\mathbf{h}_{k}$
$\displaystyle=\sqrt{\gamma_{k}}\left(\sqrt{\frac{\eta_{k}}{\eta_{k}+1}}\bar{\mathbf{h}}_{k}+\sqrt{\frac{1}{\eta_{k}+1}}\tilde{\mathbf{h}}_{k}\right),$
(4)

where $\kappa$ denotes the large-scale path-loss factor from the BS to the
RIS, $\beta_{k}$ and $\gamma_{k}$ are the large-scale path-loss factors from
the RIS and the BS to the $k$-th user, respectively. $\tilde{\mathbf{G}}$,
$\tilde{\mathbf{g}}_{k}$ and $\tilde{\mathbf{h}}_{k}$ denote the non-LoS
components that represent the scattering components, each element of which
follows zero mean and unit variance complex Gaussian distribution. $\delta$,
$\varepsilon_{k}$ and $\eta_{k}$ are Rician factors. $\bar{\mathbf{G}}$,
$\bar{\mathbf{g}}_{k}$ and $\bar{\mathbf{h}}_{k}$ represent LoS components
that are given by uniform linear antenna array response as follows

$\overline{\bf{G}}=\sum\limits_{i=1}^{I}{{{\bf{a}}_{N}}({\phi_{AoA,i}}){\bf{a}}_{M}^{T}({\phi_{AoD,i}})},$
(5) $\bar{\mathbf{g}}_{k}=\mathbf{a}_{N}(\theta_{AoD}^{k}),$ (6)
$\bar{\mathbf{h}}_{k}=\mathbf{a}_{M}(\psi_{AoD}^{k}),$ (7)

where $I$ is the number of links from the BS to the RIS, $\phi_{AoA}$ denotes
the arrival of angle (AoA) from the BS to the RIS, $\phi_{AoD}$ is the angle
of departure (AoD) from the BS to the RIS, $\theta_{AoD}^{k}$ is the AoD from
the RIS to the $k$-th user, $\psi_{AoD}^{k}$ is the AoD from the BS to the
$k$-th user, and
$\mathbf{a}_{x}(\theta)=[1,e^{j\theta},\dots,e^{j(x-1)\theta}]^{T}$.
Therefore, $\bar{\mathbf{G}}$, $\bar{\mathbf{g}}_{k}$ and
$\bar{\mathbf{h}}_{k}$ are only related to the AoD and AoA of signals that are
named as long-term CSI.

Herein, the $k$-th user’s instantaneous signal-to-interference-plus-noise
ratio (SINR) can be written as

${\text{SINR}}_{k}=\frac{|({\mathbf{h}_{k}^{T}\mathbf{\Phi}\mathbf{G}}+{\mathbf{g}_{k}^{T}})\mathbf{w}_{k}|^{2}}{\sum_{j=1,j\neq
k}^{K}|({\mathbf{h}_{k}^{T}\mathbf{\Phi}\mathbf{G}}+{\mathbf{g}_{k}^{T}})\mathbf{w}_{j}|^{2}+\sigma_{k}^{2}}.$
(8)

The instantaneous data rate of the $k$-th user is given by

$R_{k}=\log_{2}(1+{\text{SINR}}_{k}).$ (9)

In this paper, we aim to jointly optimize the precoding matrix $\mathbf{W}$
and the configuration matrix $\mathbf{\Phi}$ to maximize the minimum ergodic
transmission data rate of $K$ users, where the expectation is taken over the
non-LoS components of all channels. Mathematically, the optimization problem
is formulated as

$\displaystyle\underset{\mathbf{W},\mathbf{\Phi}}{\max}\;\underset{k}{\min}\;\;\;\mathbb{E}[R_{k}]$
(10) $\displaystyle\;\;\;\;\;\;\text{s.t.}\
\quad\;\;\text{C1}:{\rm{tr}}\\{\mathbf{W}\mathbf{W}^{H}\\}\leq P_{t},$
$\displaystyle\;\;\;\;\;\;\;\;\;\quad\quad\;\text{C2}:|\phi_{n}|=1,\forall
n=1,2,\dots,N,$

where C1 is the power constraint at the BS and $P_{t}$ is the maximum power
limit, C2 represents unit-modulus constraint of the reflecting elements. Since
the closed-form expression of the ergodic data rate is not available, Problem
(10) is challenging to solve. In addition, the max-min objective function is
non-differentiable and the existing algorithms developed for sum rate
maximization problem in [14, 15] are not applicable.

##  III Proposed Algorithm

In this section, we first introduce the transmission scheme and then provide a
DDPG based algorithm to solve the problem.

###  III-A Transmission Scheme

In this paper, we consider a series of consecutive channel coherence time
intervals (CCTIs) as shown in Fig. 1, in which the long-term CSI keeps
invariant and non-LoS components vary in each CCTI.

Figure 1: Two Transmission Schemes

Most of the existing contributions considered the first transmission scheme as
shown in Fig. 1, where the precoding matrix and the phase shift matrix are
designed based on the instantaneous CSI. In specific, in each CCTI, the
instantaneous CSI should be estimated, based on which the beamforming and
phase shift should be optimized. Finally, the data is transmitted in the
remaining part of the CCTI. However, this scheme has three drawbacks. First,
the training overhead is high as it needs to estimate the instantaneous
cascaded BS-RIS-user channel and the direct BS-user channel, the amount of
which is proportional to the number of reflecting elements that is excessive.
Second, in each CCTI, the BS needs to calculate the precoding matrix and phase
shift matrix, which incurs high computational complexity. Third, in each CCTI,
the calculated phase shift matrix needs to be sent back to the RIS controller,
which induces high feedback overhead.

To address the above issues, this paper considers the long-term CSI-based
transmission scheme shown in the lower part of Fig. 1. In this scheme, the BS
first estimates the long-term CSI (angle information). Based on the estimated
angle information, the precoding matrix and phase shift matrix are designed,
which will be used in the remaining CCTIs. Note that the precoding matrix and
the phase shift matrix only need to be calculated once in the considered time
intervals. In addition, the long-term CSI is only estimated once, which
significantly reduces the channel estimation overhead. Note that in the
considered time intervals, all the time slots are dedicated for data
transmission.

Next, we derive the ergodic net data rate by considering the channel
estimation overhead. Assume that there are a total of $T$ CCTIs, and there are
$\tau_{c}$ time slots in each CCTI.

For the instantaneous CSI-based scheme, one needs to estimate the cascaded
channel matrix $\mathbf{G}_{k}=\mathbf{G}\rm{diag}\\{\mathbf{h}_{k}\\}$ and
the direct channel matrix $\mathbf{g}_{k}$ for each CCTI. Based on [16], the
minimum number of time slots for channel training that is denoted as $\tau$
should satisfy:

$\tau\geq 2K+N-1.$ (11)

Then, at the $t$-th CCTI, the net instantaneous data rate is given by

$R_{k,{\rm{ins}}}^{(t)}=\left({1-\frac{{2K+N-1}}{{{\tau_{c}}}}}\right){\log_{2}}\left({1+{\rm{SINR}}_{k,{\rm{ins}}}^{(t)}}\right),$
(12)

where ${\rm{SINR}}_{k,{\rm{ins}}}^{(t)}$ is given by

${\rm{SINR}}_{k,{\rm{ins}}}^{(t)}=\frac{{|({\bf{h}}_{k}^{(t)T}{{\bf{\Phi}}^{(t)}}{{\bf{G}}^{(t)}}+{\bf{g}}_{k}^{(t)T}){\bf{w}}_{k}^{(t)}{|^{2}}}}{{\sum\limits_{j=1,j\neq
k}^{K}|({\bf{h}}_{k}^{(t)T}{{\bf{\Phi}}^{(t)}}{{\bf{G}}^{(t)}}+{\bf{g}}_{k}^{(t)T}){\bf{w}}_{j}^{(t)}{|^{2}}+\sigma_{k}^{2}}}.$

For the long-term CSI based design, we ignore the channel estimation overhead
incurred by the angle information estimation as it can be estimated with
limited overhead and only needs to be estimated at the start of the considered
time intervals. Hence, at the $t$-th CCTI, the net instantaneous data rate is
given by

$R_{k,{\rm{sta}}}^{(t)}={\log_{2}}\left({1+{\rm{SINR}}_{k,{\rm{lon}}}^{(t)}}\right).$
(13)

where ${\rm{SINR}}_{k,{\rm{lon}}}^{(t)}$ is given by

${\rm{SINR}}_{k,{\rm{lon}}}^{(t)}=\frac{{|({\bf{h}}_{k}^{(t)T}{\bf{\Phi}}{{\bf{G}}^{(t)}}+{\bf{g}}_{k}^{(t)T}){{\bf{w}}_{k}}{|^{2}}}}{{\sum\limits_{j=1,j\neq
k}^{K}|({\bf{h}}_{k}^{(t)T}{\bf{\Phi}}{{\bf{G}}^{(t)}}+{\bf{g}}_{k}^{(t)T}){{\bf{w}}_{j}}{|^{2}}+\sigma_{k}^{2}}}.$

###  III-B DRL description

As shown at the lower part of Fig. 1, one algorithm needs to be developed to
solve the optimization problem based on the long-term CSI. Since the closed-
form analytical expression of the ergodic data rate is not available, we
propose a novel DDPG-based method to solve Problem (10).

In a standard architecture of DRL, the optimization problem can be viewed as
the Markov Decision process (MDP) problem, which means an agent is deployed to
interact with the environment in discrete time steps (time slots). In each
time step, the agent consistently observes the state $s^{(t)}$, executes the
action $a^{(t)}$, and receives a reward $r^{(t)}$ from the environment. During
the training process, the agent learns to find the optimal policy
$\pi(\bm{a}|\bm{s})$ that maps the action set $\bm{a}$ and state set $\bm{s}$,
where $a^{(t)}\in\bm{a},s^{(t)}\in\bm{s}$, aiming at maximizing the
accumulated reward
$R^{(t)}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r^{(t^{\prime})}$, where
$T$ is the overall number of time steps, and $\gamma\in[0,1]$ is the discount
factor. Furthermore, a DNN is implemented to approximately calculate the
Q-value, which can be expressed as

$\displaystyle
Q(s^{(t)},a^{(t)})=\mathbb{E}\big{[}R^{(t)}|s^{(t)},a^{(t)}\big{]},$ (14)

where $\mathbb{E}\big{[}\cdot\big{]}$ denotes the expectation operator.

###  III-C Proposed Algorithm based on DDPG

In order to tackle the above optimization problem, we propose a novel
algorithm based on DDPG for determining the phase shift of reflecting elements
of RIS and the precoding matrix at the BS. The proposed framework is presented
in Fig. 2. To this end, we first generate a set of offline instantaneous CSI
$\\{\mathbf{G}^{(t)},\mathbf{g}_{k}^{(t)},\mathbf{h}_{k}^{(t)},t=1,2,\cdots,\forall
k\\}$ based on the Rician fading distribution and the long-term CSI.

Figure 2: The DRL-based precoding matrix and phase shift design framework
using DDPG

Specifically, in the framework of our proposed algorithm, we view the system
model as the environment, an agent is deployed at the BS. A DNN named actor
$\pi(a^{(t)},s^{(t)}|\omega^{\pi})$ is deployed to generate the optimal action
$a^{(t)}$ based on the given state $s^{(t)}$, another DNN named critic
$Q(s^{(t)},a^{(t)}|\omega^{Q})$ is deployed to generate Q-value, with the
input of $a^{(t)}$ and $s^{(t)}$, where $\omega^{\pi},\omega^{Q}$ are the
parameters of actor and critic separately. Besides, both actor and critic have
their target networks with the same structure, which can be expressed as
$\pi^{\prime}(a^{(t)},s^{(t)}|\omega^{\pi^{\prime}})$ and
$Q^{\prime}(s^{(t)},a^{(t)}|\omega^{Q^{\prime}})$ with parameters
$\omega^{\pi^{\prime}}$, $\omega^{Q^{\prime}}$. For achieving better
convergence performance and stabilizing the training process, a memory named
experience replay is deployed to store the experience which consists of
$\big{[}s^{(t)},a^{(t)},r^{(t)},s^{(t+1)}\big{]}$. We define the state
$s^{(t)}$, action $a^{(t)}$, and reward $r^{(t)}$ as follows:

  1. 1.

$\textbf{State}~{}s^{(t)}$: The state $s^{(t)}$ in the $t$-th time step is
defined as a vector containing three parts. The first part is the set that
includes the current real and imaginary part of precoding matrix $\bm{W}$; The
second part is the set that contains the real part and imaginary part of the
current phase shift of reflecting elements of RIS $\bm{\Phi}$; The third part
is the set that has the real part and imaginary part of the CSI, i.e.,
$\mathbf{G}^{(t)},\mathbf{g}_{k}^{(t)},\mathbf{h}_{k}^{(t)}$, $k=1,2,...,K$.

  2. 2.

$\textbf{Action}~{}{\bf{a}}^{(t)}$: The action ${\bf{a}}^{(t)}$ is defined as
a vector in $\mathbb{R}^{2MK+N}$ that contains two kinds of components. Note
that the activation function applied in the actor network is
${\rm{tanh}}(\cdot)$, the output value should be within $[-1,1]$. Thus, the
actor network can directly generate the real part and imaginary part of each
element in precoding matrix $\bm{W}$. Then, we normalize and reformulate the
optimized precoding matrix as follows:

$\displaystyle\bm{W}\leftarrow\frac{\sqrt{P_{t}}\bm{W}}{||\bm{W}||_{F}}.$ (15)

Additionally, as the continuous phase shift is applied in this paper and
according to the Euler’s formulation, the $n$-th phase shift $\phi_{n}$ can be
represented by an angle $\phi^{a}_{n}$ within $[0,2\pi]$, which can be
expressed as

$\displaystyle\phi_{n}={\rm{cos}}(\phi^{a}_{n})+j\cdot{\rm{sin}}(\phi^{a}_{n}).$
(16)

Then, given the $i$-th action value in ${\bf{a}}^{(t)}$ (denoted as
$a^{(t)}_{i}\in[-1,1]$) and the current phase shift angle $\phi^{a}_{n}$, the
optimized phase shift angle can be given by

$\displaystyle\phi^{a}_{n}\leftarrow\phi^{a}_{n}+a^{(t)}_{i}\pi.$ (17)

For the above defined action vector, the constraints in the optimization
problem such as C1 and C2 are always satisfied.

  3. 3.

$\textbf{Reward}~{}r^{(t)}$: The reward function is defined as follows:

$\displaystyle{r^{(t)}}=\mathop{\min}\limits_{k}{R_{k}^{(t)}},$ (18)

where $R_{k}^{(t)}$ can be calculated by using Eq. (9) and the $t$-th offline
instantaneous CSI.

We also give the overall algorithm design in Algorithm. 1.

Algorithm 1 Proposed Algorithm

1: Initialize actor $\pi(\cdot|\omega^{\pi})$, critic $Q(\cdot|\omega^{Q})$
networks with parameters $\omega^{\pi}$ and $\omega^{Q}$;

2: Initialize target networks with parameters
$\omega^{\pi^{\prime}}=\omega^{\pi}$ and $\omega^{Q^{\prime}}=\omega^{Q}$;

3: Initialize experience replay memory with size $Z$, sample size $V$;

4: for epoch = 1,2,… do

5: Initialize $s^{(t)}$;

6: for t = 1,2,… do

7: Obtain $s^{(t)}$;

8: Obtain action $\pi(a^{(t)},s^{(t)}|\omega^{\pi})+\lambda\rho^{a}$;

9: Execute action $a^{(t)}$;

10: Obtain reward $r^{(t)}$ according to Eq. (18);

11: Store experience $\big{[}s^{(t)},a^{(t)},r^{(t)},s^{(t+1)}\big{]}$ into
experience replay memory;

12: if the learning process starts then

13: Randomly sample $V$ experiences from experience replay memory;

14: Update critic according to Eq. (19);

15: Update actor according to Eq. (20);

16: Update target networks with rate $\eta$:

17:
$\omega^{\pi^{\prime}}\leftarrow\eta\omega^{\pi}+(1-\eta)\omega^{\pi^{\prime}}$

18: $\omega^{Q^{\prime}}\leftarrow\eta\omega^{Q}+(1-\eta)\omega^{Q^{\prime}}$

19: end if

20: end for

21: end for

Precisely, we first initialize the actor, critic networks, two target
networks, and experience replay memory separately. In each training epoch, the
state $s^{(t)}$ can be obtained from the environment. In Line 8, the action is
generated by the actor network. Note that a random noise $\rho^{a}$ is added,
which decays with rate $\lambda$, for better exploration. After executing the
action, the reward $r^{(t)}$ can be obtained by Eq. (18). Then, the experience
is stored into the memory. When the learning process starts, $V$ experiences
are randomly sampled for training the networks. The critic network is updated
through minimizing the loss function, which is expressed as:

$\displaystyle\\!\\!\\!\\!L\\!\\!=\\!\\!\frac{1}{V}\\!\\!\sum_{i=1}^{V}\big{(}r^{i}\\!\\!+\\!\gamma
Q^{\prime}(s^{(i+1)},a^{(i+1)}|\omega^{Q^{\prime}}\\!)\\!-\\!Q(s^{i},a^{i}|\omega^{Q})\big{)}^{2}.$
(19)

The actor network can be trained with the policy gradient, which is described
as follows:

$\displaystyle\triangledown_{\omega^{\pi}}J\approx\frac{1}{V}\sum\nolimits_{i=1}^{V}\triangledown_{a^{i}}Q(s^{i},a^{i}|\omega^{Q})\triangledown_{\omega^{\pi}}\pi(a^{i},s^{i}|\omega^{\pi}).$
(20)

Finally, we update two target networks with rate $\eta$.

##  IV Simulation Results

In this section, simulation results are provided to evaluate the performance
of our proposed algorithm. The BS and the RIS are located at (0, 0, 30
$\rm{m}$) and (100 $\rm{m}$, 20 $\rm{m}$, 10 $\rm{m}$), respectively. The
users are uniformly and randomly placed in a circle centered at (150 $\rm{m}$,
0, 1.5 $\rm{m}$) with radius of 20 $\rm{m}$. The large-scale path loss is
given by
$PL=P{L_{0}}-10\alpha{\log_{10}}\left({{d\mathord{\left/{\vphantom{d{{d_{0}}}}}\right.\kern-1.2pt}{{d_{0}}}}}\right)$,
where $P{L_{0}}$ is the path loss with the reference distance $d_{0}$,
$\alpha$ is the path loss exponent, and $d$ is the communication distance.
Unless otherwise stated, the simulation parameters are set as follows: Noise
power density of -174 dBm/Hz, channel bandwidth of 1 MHz, reference path loss
of $P{L_{0}}=-30\ {\rm{dB}}$, reference distance of $d_{0}=1$ $m$, the number
of antennas at the BS of $M=8$, Rician factors of $\delta=2.2$,
$\varepsilon_{k}=3.75$ and $\eta_{k}=2.2$. There are a total of $T=150$ CCTIs,
and $\tau_{c}=150$ time slots in each CCTI.

In Fig. 5, we first illustrate the convergence behaviour of the proposed DDPG
algorithm. The numbers of reflecting elements and users are set as $N=80$ and
$K=10$, respectively. The number of links from BS to RIS is $I=2$. There are
two curves in Fig. 5. One curve named as ‘Reward’ is obtained by:

${\tilde{r}^{(t)}}=\mathop{\min}\limits_{k}\frac{1}{t}\sum\nolimits_{l=1}^{t}{R_{k}^{(t)}}.$
(21)

Note that ${\tilde{r}^{(t)}}$ is different from that in (18) due to the fact
that the latter has lower computational complexity in the training process.
The second curve named as ‘Smooth processing’ can be obtained as follows:

$r_{{\rm{smooth}}}^{(t)}=w*r_{{\rm{smooth}}}^{(t-1)}+(1-w)*{r^{(t)}}.$ (22)

It can be observed from Fig. 5 that the DDPG converges rapidly and generally
1000 episodes are enough for the convergence.

Figure 3: Convergence behaviour of the DDPG algorithm.

Figure 4: Minimum average user data rate versus the number of reflecting
elements.

Figure 5: Minimum user data rate versus the number of reflecting elements.

In Fig. 5, we illustrate the minimum average user date (MAUR) versus the
number of RIS reflecting elements. Four users are considered, and $I$ is set
to one. Our proposed long-term CSI-based scheme is compared with the existing
scheme that is based on instantaneous CSI [17, 18]. The data rates for both
schemes are calculated based on (12) and (13). Results in Fig. 5 show that the
MAUR achieved by the long-term CSI-based algorithm always increases with the
number of reflecting elements. However, the MAUR achieved by the conventional
instantaneous CSI-based scheme first increases with the number of the
reflecting elements, and then decreases with it. The main reason is that, when
the number of reflecting elements is small, the increased passive beamforming
gain brought by the RIS outweigh the penalty due to the increased pilot
overhead. However, when additionally increasing the number of reflecting
elements, the detrimental effect of increased pilot overhead starts to
dominate the system performance.

Finally, we provide Fig. 5 to compare the computational complexity of the two
algorithms. It is observed that the total computational complexity of the
proposed algorithm is much lower than the conventional instantaneous CSI-based
algorithm. The reason is as follows. For the instantaneous CSI-based design,
the algorithm needs to be implemented in each CCTI, while the statistical-CSI-
based scheme only needs to run the DDPG algorithm once in the considered time
intervals.

##  V Conclusions

In this paper, we have proposed a DDPG algorithm to solve the RIS-aided
multiuser communication systems, where both the beamforming vectors at the BS
and the phase shifts at the RIS are designed based on the statistical CSI.
Simulation results show that the proposed algorithm converges rapidly. In
addition, compared with the conventional instantaneous CSI-based scheme, the
proposed algorithm has much lower computational complexity and higher net
throughput.

## References

  * [1] Q. Wu and R. Zhang, “Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network,” _IEEE Commun. Mag._ , vol. 58, no. 1, pp. 106–112, 2020.
  * [2] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, and C. Yuen, “Reconfigurable intelligent surfaces for energy efficiency in wireless communication,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 8, pp. 4157–4170, 2019.
  * [3] C. Pan, H. Ren, K. Wang, J. F. Kolb, M. Elkashlan, M. Chen, M. Di Renzo, Y. Hao, J. Wang, A. L. Swindlehurst, X. You, and L. Hanzo, “Reconfigurable intelligent surfaces for 6G systems: Principles, applications, and research directions,” _IEEE Commun. Mag._ , vol. 59, no. 6, pp. 14–20, June 2021\.
  * [4] C. Pan, H. Ren, K. Wang, W. Xu, M. Elkashlan, A. Nallanathan, and L. Hanzo, “Multicell MIMO communications relying on intelligent reflecting surfaces,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 8, pp. 5218–5233, 2020.
  * [5] C. Pan, H. Ren, K. Wang, M. Elkashlan, A. Nallanathan, J. Wang, and L. Hanzo, “Intelligent reflecting surface aided MIMO broadcasting for simultaneous wireless information and power transfer,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1719–1734, 2020.
  * [6] S. Hong, C. Pan, H. Ren, K. Wang, and A. Nallanathan, “Artificial-noise-aided secure MIMO wireless communications via intelligent reflecting surface,” _IEEE Trans. Commun._ , vol. 68, no. 12, pp. 7851–7866, 2020.
  * [7] T. Bai, C. Pan, Y. Deng, M. Elkashlan, A. Nallanathan, and L. Hanzo, “Latency minimization for intelligent reflecting surface aided mobile edge computing,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 11, pp. 2666–2682, 2020.
  * [8] Y. Han, W. Tang, S. Jin, C.-K. Wen, and X. Ma, “Large intelligent surface-assisted wireless communication exploiting statistical CSI,” _IEEE Trans. Veh. Tech._ , vol. 68, no. 8, pp. 8238–8242, 2019.
  * [9] M.-M. Zhao, Q. Wu, M.-J. Zhao, and R. Zhang, “Intelligent reflecting surface enhanced wireless networks: Two-timescale beamforming optimization,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 1, pp. 2–17, 2021.
  * [10] K. Zhi, C. Pan, H. Ren, and K. Wang, “Statistical CSI-based design for reconfigurable intelligent surface-aided massive MIMO systems with direct links,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 5, pp. 1128–1132, 2021\.
  * [11] A. Papazafeiropoulos, C. Pan, P. Kourtessis, S. Chatzinotas, and J. Senior, “Intelligent reflecting surface-assisted MU-MISO systems with imperfect hardware: Channel estimation, beamforming design.” [Online]. Available: https://arxiv.org/abs/2102.05333
  * [12] K. Zhi, C. Pan, H. Ren, and K. Wang, “Ergodic rate analysis of reconfigurable intelligent surface-aided massive MIMO systems with ZF detectors,” _IEEE Commun. Lett._ , pp. 1–1, 2021.
  * [13] X. Hu, J. Wang, and C. Zhong, “Statistical CSI based design for intelligent reflecting surface assisted miso systems,” _Science China Information Sciences_ , vol. 63, no. 12, pp. 1–10, 2020.
  * [14] J. Wang, H. Wang, Y. Han, S. Jin, and X. Li, “Joint transmit beamforming and phase shift design for reconfigurable intelligent surface assisted MIMO systems,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 7, no. 2, pp. 354–368, 2021.
  * [15] X. Gan, C. Zhong, C. Huang, and Z. Zhang, “RIS-assisted multi-user MISO communications exploiting statistical CSI,” _IEEE Trans. Commun._ , vol. 69, no. 10, pp. 6781–6792, 2021.
  * [16] Z. Wang, L. Liu, and S. Cui, “Channel estimation for intelligent reflecting surface assisted multiuser communications: Framework, algorithms, and analysis,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 10, pp. 6607–6620, 2020.
  * [17] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1839–1850, 2020.
  * [18] K. Feng, Q. Wang, X. Li, and C.-K. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 745–749, 2020\.