# An Attention-Aided Deep Learning Framework for Massive MIMO Channel
Estimation

Jiabao Gao, Mu Hu, Caijun Zhong, Geoffrey Ye Li, and Zhaoyang Zhang J. Gao, M.
Hu, C. Zhong, Z. Zhang are with the College of Information Science and
Electronic Engineering, Zhejiang University, Hangzhou, China (Email:
{gao_jiabao, muhu, caijunzhong}@zju.edu.cn). Geoffrey Ye Li is with the
Faculty of Engineering, Department of Electrical and Electronic Engineering,
Imperial College London, England (Email: Geoffrey.Li@imperial.ac.uk).

###### Abstract

Channel estimation is one of the key issues in practical massive multiple-
input multiple-output (MIMO) systems. Compared with conventional estimation
algorithms, deep learning (DL) based ones have exhibited great potential in
terms of performance and complexity. In this paper, an attention mechanism,
exploiting the channel distribution characteristics, is proposed to improve
the estimation accuracy of highly separable channels with narrow angular
spread by realizing the “divide-and-conquer” policy. Specifically, we
introduce a novel attention-aided DL channel estimation framework for
conventional massive MIMO systems and devise an embedding method to
effectively integrate the attention mechanism into the fully connected neural
network for the hybrid analog-digital (HAD) architecture. Simulation results
show that in both scenarios, the channel estimation performance is
significantly improved with the aid of attention at the cost of small
complexity overhead. Furthermore, strong robustness under different system and
channel parameters can be achieved by the proposed approach, which further
strengthens its practical value. We also investigate the distributions of
learned attention maps to reveal the role of attention, which endows the
proposed approach with a certain degree of interpretability.

Index Terms

Massive MIMO, channel estimation, deep learning, attention mechanism, hybrid
analog-digital, divide-and-conquer.

##  I Introduction

Massive multiple-input multiple-output (MIMO) is a key enabling technology for
future wireless communication systems due to its high spectral and energy
efficiency[1, 2]. However, the realization of various theoretical gains of
massive MIMO is critically dependent on the quality of channel state
information (CSI). Because of the large number of antennas and users, the CSI
acquisition has long been a major challenge in practical massive MIMO systems.

In the prior works, least square (LS) and minimal mean-squared error (MMSE)[3]
are two most commonly used estimators for channel estimation. The LS is
relatively simple and easy to implement while its performance is
unsatisfactory. On the other hand, MMSE can refine the LS estimation if
accurate channel correlation matrix (CCM) is available. However, the
complexity of MMSE estimation is much higher than that of LS estimation due to
the matrix inversion operation. On the other hand, to reduce the hardware and
energy cost, the hybrid analog-digital (HAD) architecture is usually adopted
in practical massive MIMO systems, where the multi-antenna array is connected
to only a limited number of radio-frequency (RF) chains through phase shifters
in analog domain[4, 5, 6]. With HAD, channel estimation becomes even more
difficult since the received signals at the BS are only a few linear
combinations of the original signals. If LS is used, multiple estimations are
required since only part of the antennas’ channels can be estimated once due
to limited number of RF chains. To avoid the dramatically increased overhead
of LS, the slowly changing directions of arrival of channel paths are obtained
first in the preamble stage in [7], then only channel gains of each path are
re-estimated in a long period. Another alternative is to exploit the channel
sparsity and estimate all antennas’ channels at once using the compressed
sensing (CS) based methods, such as orthogonal matching pursuit[8] and sparse
Bayesian learning[9]. In [10, 11], several improved CS algorithms have been
developed through embedding the structural characteristics of channel
sparsity, which can achieve better estimation performance without extra pilot
overhead. Nevertheless, CS algorithms require high computational complexity
and perform poor for channels with low sparsity. Therefore, it is highly
desirable to develop channel estimators with less requirement for prior
information and better performance-complexity trade-off.

Inspired by the great performance and the low complexity during online
prediction, deep learning (DL) has been applied to many wireless communication
problems[12, 13], such as spectrum sensing[14], resource management[15, 16,
17, 18], beamforming[19, 20], signal detection[21, 22, 23], and channel
estimation[26, 24, 25, 27, 28, 29, 30, 31, 32]. By exploiting the structural
characteristics of the modulated signals, the customized deep neural network
(DNN) in [14] significantly outperforms energy detection in spectrum sensing.
In [15], a DNN has been proposed for resource management, which can achieve
comparable performance as the iterative optimization algorithm. An
unsupervised learning-based beamforming network has been developed for
intelligent reconfigurable surface aided massive MIMO systems in [19]. In
[21], channel estimation and signal detection in orthogonal frequency division
multiplexing systems have been performed jointly by a DNN. Then, a model-
driven based approach is further proposed in [22] to exploit the advantages of
both conventional algorithms and DNN. In [23], rather than directly using a
black-box DNN, the conventional orthogonal approximate message passing
algorithm (OAMP) is unfolded for the detection network.

There are mainly two categories of approaches for DL-based massive MIMO
channel estimation. In the first category, “deep unfolding” methods unfold
various iterative optimization algorithms and enhance their estimation
performance by inserting learnable parameters. In [24], the AMP algorithm is
unfolded into a cascaded neural network for millimeter wave channel
estimation, where the denoiser is learned by a DNN. Thanks to the power of DL,
the proposed method can outperform a series of conventional denoising-AMP
based algorithms. In [25], the iterative shrinkage thresholding algorithm is
unfolded to solve sparse linear inverse problems, where massive MIMO channel
estimation is used as a case study. However, “unfolding” is only feasible to
the iterative algorithms with simple structures, and the computational
complexity is also high. In the other category, DL is used to directly learn
the mapping from available channel-related information to the CSI for
performance improvement or complexity reduction. In [26], a DNN has been
proposed to refine the coarse estimation in HAD massive MIMO systems, where
the channel correlation in the frequency and time domains is exploited for
further performance improvement. In [28], the estimation performance is
further improved by jointly training the pilot signals and channel estimator
with an autoencoder in downlink massive MIMO systems. In [29], graph neural
network has been used for massive MIMO channel tracking. Deep multimodal
learning has been used for massive MIMO channel estimation and prediction in
[30]. To reduce the complexity, the amplitudes of beamspace channels are
predicted by a DNN and the dominant entries are estimated by LS in [31], thus
avoiding the greedy search commonly adopted by CS algorithms. In [32], the
uplink-to-downlink channel mapping in frequency-division duplex (FDD) systems
is learned by a sparse complex valued network.

Nevertheless, current DL-based channel estimation methods have seldom
exploited the characteristics of channel distribution. In practice, the BS is
often located in a high altitude with few surrounding scatters[33], so the
angular spread of each user’s incident signal at the BS is narrow. Thus, the
global distribution of channels corresponding to different users in the entire
angular space can be viewed as the composition of many local distributions,
where each local distribution represents channels within a small angular
region. Due to narrow angular spread, a certain angular region contains much
fewer channel cases than the entire angular space because of the limited
angular range of channel paths, making the local distributions much simpler
than the global distribution. Besides, different local distributions can be
highly distinguishable from each other if the entire angular space is properly
segmented into different angular regions. Under such a condition, the classic
“divide-and-conquer” policy, which tackles a complex main problem by solving a
series of its simplified sub-problems, is very suitable. Specifically, the
estimation of channels in the entire angular space can be regarded as the main
problem and the estimation of channels in different small angular regions can
be regarded as different sub-problems. Motivated by this, in this paper, we
propose a novel attention-aided DL-based channel estimation framework, where
the “divide-and-conquer” policy is realized automatically through the dynamic
adaptation of attention maps. The main contributions of this paper are
summarized as follows:

  * •

An attention-aided DL-based channel estimation framework is proposed for
massive MIMO systems, which achieves better performance than its counterpart
without attention in simulation. To the best knowledge of the authors, this is
the first work that introduces the attention mechanism to DL-based channel
estimation111There are already some literature that uses attention-aided DL to
solve communication problems, such as CSI compression[34, 35] and joint source
and channel coding[36]. Nevertheless, the considered channel distribution in
[34] does not possess strong separable property, and the proposed method in
[36] requires extra side information. As for [35], the non-local neural
network model is utilized to exploit the self-attention in the spatial
dimension of channels..

  * •

We extend the above framework to the scenario with HAD and an embedding method
is proposed to effectively integrate the attention mechanism into the fully
connected neural network (FNN), which expands the application range of the
proposed approach.

  * •

We visually explain the “divide-and-conquer” policy reflected in the
distributions of learned attention maps, which enhances the interpretability
and rationality of the proposed approach.

  * •

Based on our results, the performance gain of attention mainly comes from the
narrow angular spread characteristic of channels. Therefore, the proposed
approach can be extended to many other problems apart from channel estimation
as long as the channel distribution has certain separability, such as multi-
user beamforming, FDD downlink channel prediction, and so forth.

The rest of this paper is organized as follows. Section II introduces the
system model, channel model, and problem formulation. Section III presents the
attention-aided DL-based channel estimation framework, which is extended to
the HAD scenario in Section IV. Extensive simulation results are demonstrated
in Section V. Eventually, the paper is concluded in Section VI.

Here are some notations used subsequently. We use italic, bold-face lower-case
and bold-face upper-case letter to denote scalar, vector, and matrix,
respectively. ${\bf A}^{T}$ and ${\bf A}^{H}$ denote the transpose and
Hermitian or complex conjugate transpose of matrix $A$, respectively.
$[\mbox{\boldmath$A$}]_{i,j}$ denotes the element at the $i$-th row and $j$-th
column of matrix $A$. ${\left\|{\bf{x}}\right\|}$ denotes the $l$-2 norm of
vector $x$, and $|a|$ denotes the amplitude of complex number $a$.
${\mathbb{C}^{x\times y}}$ denotes the ${x\times y}$ complex space.
$\mathcal{CN}(\mu,\sigma^{2})$ denotes the distribution of a circularly
symmetric complex Gaussian random variable with mean $\mu$ and variance
$\sigma^{2}$. $\mathcal{U}[a,b]$ denotes the uniform distribution between $a$
and $b$.

##  II System model and problem formulation

In this section, system model and channel model are first introduced. Then,
the conventional massive MIMO channel estimation problem is formulated.

###  II-A System Model

Consider a single cell massive MIMO system, where the BS is equipped with an
$N$-antenna uniform linear array (ULA) and $K$ single-antenna users are
randomly distributed in the cell of the corresponding BS, as illustrated in
Fig. 1.

Figure 1: Massive MIMO system without HAD.

###  II-B Channel Model

Following the same channel model as in [37], the uplink channel from user $k$
to the BS can be expressed as

$\mbox{\boldmath$h$}_{k}=\frac{1}{\sqrt{N_{p}}}\sum_{i=1}^{N_{p}}\alpha_{ki}\mbox{\boldmath$a$}(\theta_{ki})\in\mathbb{C}^{N\times
1},$ (1)

where $N_{p}$ is the number of paths, $\alpha_{ki}$ and $\theta_{ki}$ are the
complex gain and angle of arrival (AoA) at the BS of the $i$-th path from the
$k$-th user, respectively. Without loss of generality, we consider half-
wavelength antenna spacing in this paper, then the steering vector of the ULA
can be written as
$\mbox{\boldmath$a$}(\theta)=[1,e^{j\pi\sin(\theta)},\cdots,e^{j\pi\sin(\theta)(N-1)}]^{T}$.
Define the average AoA and the angular spread of user $k$’s channel paths as
$\bar{\theta}_{k}$ and $\bigtriangleup_{\theta}$, respectively, that is,
$\theta_{ki}$ follows a uniform distribution
$\mathcal{U}[\bar{\theta}_{k}-\bigtriangleup_{\theta},\bar{\theta}_{k}+\bigtriangleup_{\theta}]$.
As in [37, 11], the narrow angular spread assumption is adopted, i.e.,
$\bigtriangleup_{\theta}\ll\pi$.

To better understand this channel characteristic, we convert the original
channel to the angular domain by

$\displaystyle\mbox{\boldmath$x$}_{k}=\mbox{\boldmath$F$}\mbox{\boldmath$h$}_{k}\in\mathbb{C}^{N\times
1},$ (2)

where $\mbox{\boldmath$x$}_{k}$ denotes the angular domain channel of user
$k$, and $\mbox{\boldmath$F$}\in\mathbb{C}^{N\times N}$ is a shift-version
discrete Fourier transform matrix[11], with the $n$-th row given by
$\mbox{\boldmath$f$}_{n}=\frac{1}{\sqrt{N}}[1,e^{-j\pi\eta_{n}},\cdots,e^{-j\pi\eta_{n}(N-1)}]$,
for $\eta_{n}=\frac{-N+1}{N},\frac{-N+3}{N},\cdots,\frac{N-1}{N}$. Due to
narrow angular spread assumption, the angular domain channel exhibits the
spatial-clustered sparsity structure[11]. Specifically, as shown in the right
half of Fig. 1, $\mbox{\boldmath$x$}_{k}$ only has a few significant elements
appearing in a cluster. If properly exploited, such sparsity structure can
help to improve estimation performance and reduce estimation overhead.

###  II-C Problem Formulation

During the uplink training, orthogonal pilot sequences are sent by different
users. Denote the pilot sequence of the $k$-th user as
$\mbox{\boldmath$p$}_{k}\in{\mathbb{C}}^{1\times Lp}$, where $L_{p}\geq K$ is
the length of pilot sequences. Notice that the channel during pilot training
phase is assumed to be unchanged[11] since $L_{p}$ is relatively small.
Therefore, the superimposed received signal at the BS can be expressed as

$\displaystyle\mbox{\boldmath$Y$}=\sum^{K}_{k=1}\mbox{\boldmath$h$}_{k}\mbox{\boldmath$p$}_{k}+\mbox{\boldmath$N$}\in\mathbb{C}^{N\times
L_{p}},$ (3)

where
$\mbox{\boldmath$N$}\sim\mathcal{CN}(0,\sigma^{2})\in{\mathbb{C}}^{N\times
L_{p}}$ is the zero-mean additive white Gaussian noise at the BS with variance
$\sigma^{2}$. Without loss of generality, we fix the power of pilot sequences
to unit and adjust the transmit signal-to-noise ratio (SNR) by changing the
noise variance. Then, we have
$\mbox{\boldmath$p$}_{i}\mbox{\boldmath$p$}_{j}^{H}=0,\forall i\neq j$ and
$\mbox{\boldmath$p$}_{i}\mbox{\boldmath$p$}_{i}^{H}=1,\forall i$. Exploiting
the orthogonality of the pilot sequences, the LS estimation of user $k$’s
channel can be obtained as

$\hat{\mbox{\boldmath$h$}}_{k}=\mbox{\boldmath$Y$}\mbox{\boldmath$p$}_{k}^{H}=\mbox{\boldmath$h$}_{k}+\widetilde{\mbox{\boldmath$n$}}_{k}\in\mathbb{C}^{N\times
1},$ (4)

where
$\widetilde{\mbox{\boldmath$n$}}_{k}\triangleq\mbox{\boldmath$N$}\mbox{\boldmath$p$}_{k}^{H}$
is the effective noise for user $k$. For brevity, we will consider a specific
user from now on and omit subscript $k$. Besides, we use
$\hat{\mbox{\boldmath$h$}}_{\text{LS}}$ to denote the LS estimation.
Therefore, the goal of channel estimation222Here we use the term “channel
estimation” for consistency, actually “channel refinement” is more proper. is
to find a function that maps $\hat{\mbox{\boldmath$h$}}_{\text{LS}}$ to $h$.

One of the conventional methods is the MMSE estimation, where the LS
estimation is refined by the CCM. However, accurate CCM is hard to obtain in
practice and the complexity of matrix inversion in MMSE estimation is very
high, especially when the antenna number is large. In [38], DL-based methods
have been proposed to refine the channel estimation. In this paper, we will
develop an attention-aided DL framework for conventional massive MIMO channel
estimation by exploiting the characteristics of channel distribution.

##  III Attention-aided DL framework for massive MIMO channel estimation

In this section, input and output processing, network structure design, and
detailed network training method of the proposed framework are introduced.

###  III-A Input and Output Processing

Since channel parameters can be canonically expressed in the angular domain,
the input and output of the networks are all in the angular domain in the
proposed framework. In simulation, we find that the more sparse angular domain
input and output can lead to better channel estimation performance than the
original ones. Once the angular domain channel estimation,
$\hat{\mbox{\boldmath$x$}}$, is obtained, the original channel estimation can
be readily recovered by
$\hat{\mbox{\boldmath$h$}}=\mbox{\boldmath$F$}^{H}\hat{\mbox{\boldmath$x$}}$.
Besides, the real and imaginary parts have to be separately processed since
complex training is still not well supported by current DL libraries. To
promote efficient training, we also perform standard normalization
preprocessing on the input.

###  III-B Attention-Aided Channel Estimation Network Structure Design

As shown in Fig. 2, convolutional neural network (CNN) is a suitable choice
for the network structure to exploit the local correlation in the input data
due to the spatial-clustered sparsity structure of the angular domain channel.
In this paper, one-dimensional convolution (Conv1D) is used due to the shape
of input data. The input of a Conv1D layer is organized as a
$(F,C)$-dimensional feature matrix, where $C$ denotes the number of
channels333Here channel is a term in CNN representing a dimension of feature
matrix, not the communication channel. and $F$ denotes the number of features
in each channel. Then, the convolution operation slides $C^{\prime}$ filters
over the input feature matrix in certain strides to obtain the output feature
matrix, which is also the input of the next layer. Specifically, each filter
contains a $(L,C)$-dimensional trainable weight matrix and a scalar bias term,
where $L$ denotes the filter size. When a filter is located in a certain
position of the feature matrix, the cross-correlation between the
corresponding chunk of the feature matrix and the weight matrix of the filter
is computed and the bias is added to obtain the convolution output of the
position[39]. In the proposed channel estimation network, $N_{B}$
convolutional blocks and an output Conv1D layer are used to refine the LS
coarse channel estimation. As depicted in the dashed box, in each
convolutional block, a batch normalization (BN) layer to prevent gradient
explosion or vanishing[43] and a ReLU activation function are inserted after
the Conv1D layer. Besides, the Conv1D layer in the first block has $F$ filters
of size $L_{I}$ and the Conv1D layers in the next $N_{B}-1$ blocks have $F$
filters of size $L_{H}$. The optimal values of $N_{B}$ and $F$ can be
determined through simulation. Finally, the output Conv1D layer has 2 filters
of size $L_{O}$, corresponding to the real and imaginary parts of the channel
prediction, respectively. The stride is set to $S$ and all the Conv1D layers
pad zeros to keep the dimension $N$ of the feature matrix unchanged.

Figure 2: Structure of the channel estimation network.

To effectively exploit the distribution characteristics of channel, the
attention mechanism444Notice that, the term attention can refer to many
related methods including [40, 42, 41]. In this paper, we use the classic
“SENet” proposed in [40]. is applied in the network structure design. In the
original CNN, all the features are used for all data samples with equal
importance. However, certain features can definitely be more important or
informative than others to certain data samples in practice, especially for
highly separable data like narrow angular spread channel. For instance, key
features, which are only aimed at dealing with channel distribution in a
specific angular region, might be useless or even disruptive for the
estimation of channels in another region far apart. Therefore, the idea of
feature importance reweighting can be used here to improve network
performance.

As is demonstrated in Fig. 3, the original feature matrix is multiplied by an
attention map in a channel-wise manner to obtain the reweighted feature matrix
in the attention module, where more important or informative features to the
current data sample will be paid more “attention” to. For the learning process
of the attention map, global average pooling is performed first on the
original feature matrix, $\mbox{\boldmath$Z$}_{O}$, to embed the global
information into a $(1,C)$-dimensional squeezed feature matrix, $z$.
Specifically, the $c$-th element of $z$ is calculated by
$z_{c}=\sum_{f=1}^{F}[\mbox{\boldmath$Z$}_{O}]_{f,c}/F$[40]. Then, the
$(1,C)$-dimensional attention map, $m$, is predicted by a dedicated attention
network based on $z$. The attention network contains two fully connected (FC)
layers. The first FC layer with $C/r$ neurons is followed by a ReLU
activation, $f_{\text{ReLU}}(x)=\text{max}(0,x)$, where $r\geq 1$ denotes the
reduction ratio. The second FC layer with $C$ neurons is followed by a Sigmoid
activation, $f_{\text{Sigmoid}}(x)=1/(1+e^{-x})$, which limits the elements of
$m$ between 0 and 1. As can be seen in Fig. 2, an attention module is inserted
at the end of each convolutional block in the proposed channel estimation
network. Besides, $r$ is set to 2 to balance performance and complexity and
the FC layers in the attention network do not use bias to facilitate channel
dependency modeling.

Figure 3: Structure of the attention module.

###  III-C Network Training

To train the designed network, the mean-squared error (MSE) between the true
angular domain channel, $x$, and the predicted angular domain channel,
$\hat{\mbox{\boldmath$x$}}$, is used as the loss function, which can be
calculated by

$\text{MSE
Loss}=\frac{1}{n}\sum_{i=1}^{n}\left\|\hat{\mbox{\boldmath$x$}}_{i}-\mbox{\boldmath$x$}_{i}\right\|^{2},$
(5)

where subscript $i$ denotes the $i$-th data sample in a mini-batch and $n=500$
is the size of the mini-batch. Xavier[44] is used as the weight initializer
and Adam[45] is used as the weight optimizer. The initial learning rate is set
to 0.001. To balance the training complexity and testing performance, we
generate totally 200,000 data samples according to the adopted channel and
transmission models. Then, the generated dataset is split into training,
validation, and testing set with a ratio of 3:1:1. In order to accelerate loss
convergence at the beginning and reduce loss oscillation near the end of
training, the learning rate is set to decay 10 times if the validation loss
does not decrease in 10 consecutive epochs. Besides, early stopping [46] with
a patience of 25 epochs is applied to prevent overfitting and speed up the
training process.

##  IV Extension to the HAD scenario

In practice, the HAD architecture is often adopted in massive MIMO systems to
save hardware and energy cost. Due to the effect of phase shifters in the
analog domain in the HAD architecture, the problem formulation of channel
estimation changes and the channel estimation network structure has to be
customized correspondingly as well. In the HAD architecture, we assume there
is only $M\ll N$ RF chains available at the BS, as illustrated in Fig. 4.

Figure 4: Massive MIMO system with HAD.

###  IV-A Problem Reformulation with HAD

With HAD, the signals arriving at the antennas have to go through the phase
shifters first before received by the RF chains. So, the eventual received
signal on the baseband can be expressed as

$\displaystyle\mbox{\boldmath$Y$}_{\text{HAD}}=\mbox{\boldmath$WY$}\in\mathbb{C}^{M\times
L_{p}},$ (6)

where $\mbox{\boldmath$W$}\in{\mathbb{C}}^{M\times N}$ denotes the analog
combining matrix. As the phase shifters only change the phase of signals, we
have $|[\mbox{\boldmath$W$}]_{i,j}|=1/\sqrt{N}$, $\forall i,j$ after
normalization. We set $W$ to a matrix whose rows are length-$N$ Zadoff-Chu
sequences with different shifting steps as in[10]. Again, exploiting the
orthogonality of the pilot sequences, the received signal corresponding to
user $k$ can be obtained as

$\mbox{\boldmath$y$}_{k}=\mbox{\boldmath$Y$}_{\text{HAD}}\mbox{\boldmath$p$}_{k}^{H}=\mbox{\boldmath$Wh$}_{k}+\widetilde{\mbox{\boldmath$n$}}^{\prime}_{k}\in\mathbb{C}^{M\times
1},$ (7)

where
$\widetilde{\mbox{\boldmath$n$}}^{\prime}_{k}\triangleq\mbox{\boldmath$W$}\widetilde{\mbox{\boldmath$n$}}_{k}$
is the effective noise for user $k$ with HAD. Consider a specific user and
omit the subscript $k$, the goal of channel estimation now becomes to find a
function that maps $y$ to $h$.

Since the overhead of LS estimation increases dramatically due to limited
number of RF chains, CS algorithms are more often adopted to solve the channel
estimation problem in HAD massive MIMO systems conventionally. However, the
performance of CS algorithms is highly dependent on channel sparsity and the
computational complexity is relatively high due to complex operations and a
large number of iterations. Therefore, we extend the proposed framework to the
HAD scenario and use DL to overcome these issues.

###  IV-B Attention-Aided Channel Estimation Network Structure Design With HAD

Different from the former scenario, in the problem of channel estimation with
HAD, the input data becomes the received signal $y$, where little local
correlation exists due to the compression of matrix $W$. Therefore, FNN should
be used rather than CNN to achieve better performance. Although the attention
mechanism has been originally proposed in the area of computer vision and is
only compatible with CNN, its key idea, feature importance reweighting, is
actually independent of network structure. Therefore, to exploit the benefit
of the attention mechanism, we propose a simple but effective method to embed
it into FNN.

As introduced earlier, the attention module is inserted after a feature matrix
and the attention map is learned from the squeezed feature matrix obtained by
global average pooling. FNN can not directly use attention since all the
neurons of the neighboring FC layers are fully connected and features of FC
layers appear in the form of vectors instead of matrices. Therefore, as
depicted in the dashed box in Fig. 5, we reshape the feature vector of a FC
layer into a matrix first, like the feature matrix of a Conv1D layer. Then,
with the matrix-shaped feature, the original attention mechanism can be
normally applied. Finally, the reweighted feature vector can be obtained by
flattening the reweighted feature matrix.

The detailed network design is illustrated in Fig. 5. The first FC layer
consists of $F\times C$ neurons, which is followed by a ReLU activation and a
BN layer. The feature vector is then reshaped into a $(F,C)$ feature matrix,
where $C$ and $F$ can be regarded as the number of channels and the number of
features of each channel, respectively. Based on the feature matrix, the
original attention module is inserted to get the reweighted feature matrix,
which is then flattened back to the reweighted feature vector. Eventually, an
output FC layer with $2N$ neurons is used to obtain the real and imaginary
parts of the angular domain channel prediction. We only use one hidden FC
layer here since experiments indicate that more hidden FC layers are not
helpful to further improve the performance but increases the complexity
dramatically.

Figure 5: Channel estimation network structure with HAD.

###  IV-C Complexity Analysis

In this subsection, the complexity of various algorithms are analyzed. Two
metrics are used to measure the complexity, namely the required number of
floating point operations (FLOPs) and the total number of parameters. For
brevity, only multiplication is considered and one complex multiplication is
counted as four real multiplications when computing FLOPs, and the weights and
biases of BN layers are ignored and one complex parameter is counted as two
real parameters when computing parameter number. When analyzing the complexity
of neural networks, we ignore the offline training phase and focus on the
online testing phase since the network training only needs to be executed once
and the BS usually has sufficient computational ability in practice.

Using the notations in Section III, the FLOPs of the Conv1D layer and the
$l$-th FC layer are $LFCC^{\prime}$ and $N_{l-1}N_{l}$, respectively, where
$N_{l}$ denotes the number of neurons in the $l$-th FC layer. Without HAD, the
overall FLOPs of CNN can be readily obtained as
$(2L_{I}F+2L_{O}F+L_{H}F^{2}(N_{B}-1))N$ and the additional FLOPs of attention
modules is $N_{B}F(N+F+1)$. The FLOPs of MMSE estimation is $4(2N^{3}+N^{2})$.
Besides, for both algorithms, the LS estimation has to be obtained first,
which also requires $4NL_{p}^{2}$ FLOPs. In the scenario with HAD, the FLOPs
of the attention-aided FNN can be obtained as $FC(2M+2N+1)+C(C+1)$. In this
paper, structured variational Bayesian inference (S-VBI) is selected as the
CS-based baseline algorithm, whose FLOPs is
$I_{E}(\frac{2}{3}M^{3}+(2M+2)N^{2})$ with $I_{E}$ denoting the number of
iterations[11]. Again, for both algorithms, obtaining the received signal
corresponding to a single user requires $4ML_{p}^{2}$ FLOPs. Notice that in
both scenarios, the FLOPs of DL-based algorithms only scale linearly with $N$
and $M$, which is an attractive practical advantage, especially in large scale
systems. By contrast, the FLOPs of conventional algorithms are much higher and
grow cubically with $N$ and $M$.

As for the total number of parameters, the Conv1D layer and the $l$-th FC
layer contains $LCC^{\prime}$ and $N_{l-1}N_{l}$ parameters, respectively.
Without HAD, CNN contains totally $2(L_{I}+L_{0})F+L_{H}(N_{B}-1)F^{2}$
parameters and the additional number of parameters of attention modules is
$N_{B}F^{2}$. The CCM used in MMSE requires $2N^{2}$ parameters. In the
scenario with HAD, attention-aided FNN contains totally $FC(2M+2N)+C^{2}$
parameters, while S-VBI does not need any parameters.

##  V Simulation Results

In this section, extensive simulation results are presented to evaluate the
performance of the proposed DL-based channel estimation framework in scenarios
with and without HAD. MSE is adopted as the performance metric. Notice that,
converting the channel to angular domain does not change the MSE since $F$ is
a unitary matrix. Some of the parameters used in simulation are summarized in
Table I, unless otherwise specified. As for network hyper-parameters in the
scenario without HAD, $L_{I}$, $L_{H}$, and $L_{O}$ are set to 7, 5, and 1,
respectively, and $S$ is set to 1.

Parameter | Value
---|---
$N$ | 128
$M$ | 32
$N_{p}$ | 20
$\theta_{i}$ | $\mathcal{U}[0,2\pi]$
$\bigtriangleup_{\theta}$ | $5^{\circ}$
$\alpha_{i}$ | $\mathcal{CN}(0,1)$
SNR | 20 dB
TABLE I: Simulation parameters.

We compare the proposed algorithm with the following baseline algorithms. The
structures of all DL-based baselines are carefully determined by cross
validation out of fairness.

####  V-1 Without HAD

The following algorithms are selected as baselines:

  * •

MMSE Single: Refine the LS estimation by the CCM,
$\mbox{\boldmath$R$}_{hh}\triangleq\mathbb{E}({\mbox{\boldmath$h$}\mbox{\boldmath$h$}^{H}})\in\mathbb{C}^{N\times
N}$ as [3]

$\hat{\mbox{\boldmath$h$}}_{\text{MMSE}}=\mbox{\boldmath$R$}_{hh}(\mbox{\boldmath$R$}_{hh}+\mbox{\boldmath$I$}/\text{SNR})^{-1}\hat{\mbox{\boldmath$h$}}_{\text{LS}}.$
(8)

  * •

MMSE $\bm{3^{\circ}$}:$ Split the entire angular space into many
$3^{\circ}$-angular regions and estimate a dedicated CCM for each region with
only channel samples whose average AoAs are in the region. During the testing
process of a channel sample, the angular region it belongs to will be
estimated first555The angular region estimations of samples are assumed to be
accurate for simplicity. and the corresponding CCM will be selected for
channel refinement. Compared with using a single CCM for all channel samples,
using multiple CCMs matching different angular regions can effectively exploit
the narrow angular spread characteristic of channels and improve performance
significantly. Actually, it can be regarded as the manual implementation of
the “divide-and-conquer” policy, i.e., the channel samples are “divided” by
their angular regions and “conquered” by different corresponding CCMs.

  * •

FNN: The FNN structure consists of three FC layers with 512, 1024, and 256
neurons, respectively, with one BN layer inserted between every two FC layers.
The activation function of the first two FC layers is ReLU while the last FC
layer does not use activation.

  * •

CNN without attention: The same CNN structure but with all the attention
modules removed.

####  V-2 With HAD

The following algorithms are selected as baselines:

    * –

Separate LS: A total of $N/M$ estimates are executed. In each estimate, only
$M$ antennas are switched on by adjusting $W$, and their channels are obtained
by LS estimation[7].

    * –

S-VBI: One of the state-of-the-art CS-based algorithms designed for narrow
angular spread channel estimation in HAD massive MIMO systems, where the
spatial-clustered channel sparsity is embedded to improve the estimation
performance[11]. The source code is provided by the authors of [11].

    * –

FNN without attention: Adopt the same structure of FNN as in the former
scenario while the number of neurons reduces to 256, 512, and 256,
respectively, with smaller input dimension.

    * –

CNN: The structure of CNN is also similar to the former scenario, except that
the output layer is changed from Conv1D to FC for dimension conversion.

    * –

CNN without attention: The same CNN structure but with all the attention
modules removed.

###  V-A Impacts of Network Parameters

To determine the best network structures for two scenarios, we investigate the
impacts of key network parameters on network performance. Without HAD, the
structure of CNN is mainly determined by the number of convolutional blocks,
$N_{B}$, and the number of filters of each Conv1D layer, $F$. As illustrated
in Fig. 6(a), attention can improve the performance of CNNs with various
numbers of convolutional blocks and filters and the performance of a two-layer
attention-aided CNN is even better than a four-layer CNN without attention,
which indicates the superiority of the attention mechanism. In general, the
performance of networks is better with stronger representation capability
brought by more convolutional blocks. However, with enough filters, the
performance improvement of attention-aided CNN is marginal if the number of
filters keeps growing and it can even be harmful to CNN without attention
sometimes. Besides, deeper and wider CNNs also have heavier computing and
storage burdens. To strike a balance between performance and complexity, we
choose to use four convolutional blocks and 96 filters for each Conv1D layer.

With HAD, the structure of the attention-aided FNN is mainly determined by the
number of neurons of the hidden FC layer $F\times C$ and the way of reshaping
in the attention embedding module. As in Fig. 6(b), the network performs best
when $F\times C=3072$ and the performance will deteriorate with either too few
or many neurons. Besides, as can be indicated from the bowl shape of curves, a
medium number of features in each channel performs best when $F\times C$ is
fixed. The reason is that the number of channels is too small and there is not
enough degrees of freedom for dynamic adjustment of attention maps when $F$ is
too large, while each channel does not contain enough features to effectively
capture the global information[40] when $F$ is too small. So, we choose to
reshape the feature vector into 192 channels with 16 features in each channel.

(a) Without HAD

(b) With HAD

Figure 6: Impact of network parameters in the two considered scenarios.

###  V-B Impacts of System Parameters

In this subsection, the impacts of various system parameters are investigated
to validate the superiority and universality of the proposed approach.

####  V-B1 Impact of SNR

As illustrated in Fig. 7(a), without HAD, all DL-based methods can refine and
improve the channel quality of LS coarse estimation. The performance
improvement of FNN decreases as the SNR increases while CNN outperforms LS
significantly in various SNR regimes thanks to the exploitation of local
correlation of input data. Then, with the aid of attention, the MSE of CNN
further decreases moderately. Besides, the performance gain of the attention
mechanism increases with SNR. When SNR is 0 dB, the MSE of CNN with attention
is $89.55\%$ of that of CNN without attention while this ratio decreases to
$71.83\%$ when SNR is 20 dB. The reason is that the narrow angular spread
characteristic of the angular domain channel is more exposed and easier to be
exploited with less noise, thereby amplifying the benefits of attention. As
for MMSE, the performance improvement of MMSE Single is marginal while MMSE
$3^{\circ}$ performs much better due to the exploitation of the narrow angular
spread characteristic of channel. Nevertheless, the proposed attention-aided
CNN still slightly outperforms MMSE $3^{\circ}$, demonstrating its
superiority.

From Fig. 7(b), the performance of FNN is much better than CNN and outperforms
separate LS except in high SNR regimes when HAD is considered and attention is
not used, but it is still obviously inferior to S-VBI. However, with the aid
of attention, the performance of both CNN and FNN improves significantly. As
can be observed, attention-aided CNN outperforms S-VBI except when SNR is
higher than 15 dB while the attention-aided FNN is even better and outperforms
S-VBI consistently in all SNR regimes. Besides, compared with Fig. 7(a), the
performance gain of the attention mechanism is much more significant since the
attention mechanism can not only help denoise but also plays an important role
in reversing the effect of $W$ in the HAD scenario. Specifically, when
restoring the high-dimensional channel from the low-dimensional received
signal, the performance deterioration can be effectively reduced if the
approximate AoA range of channel paths is known. Thanks to the attention
mechanism, such processing can be automatically realized by the dynamic
adjustment of attention maps.

(a) Without HAD

(b) With HAD

Figure 7: Impact of SNR in the two considered scenarios.

####  V-B2 Impact of Angular Spread

As is illustrated in Fig. 8, attention-aided CNN has close performance to MMSE
$3^{\circ}$ and consistently outperforms LS significantly with various angular
spreads. As angular spread increases, the performance of all algorithms
decreases in both scenarios since the channel estimation problem becomes more
complex with less sparse angular domain channel. Besides, the performance gain
of attention also decreases because the channel distribution is less
separable, which makes the attention mechanism more difficult to realize the
“divide-and-conquer” policy. In the scenario with HAD, the performance of
attention-aided FNN is better than separate LS unless the angular spread is
too large while only $M/N$ resource overhead is required.

(a) Without HAD

(b) With HAD

Figure 8: Impact of angular spread in the two considered scenarios.

####  V-B3 Impacts of Antenna Number and RF Chain Ratio

As can be observed from Fig. 9, in both scenarios, the performance of all
algorithms improves as $N$ increases. Since the power leakage of angular
domain channel is inversely proportional to the antenna number[37], the
increased channel sparsity caused by more antennas can simplify channel
estimation. Without HAD, attention-aided CNN has close performance to MMSE
$3^{\circ}$ and the performance gain of attention can be amplified by sparser
channel. With HAD, the performance of all algorithms improve as the RF chain
ratio $M/N$ increases since more information is kept during the sensing phase.
Besides, attention-aided FNN outperforms S-VBI consistently with various $M$
and $N$ and the performance gap increases with less antennas with fixed RF
chain ratio, indicating that the DL-based approach is less dependent on
channel sparsity. From the perspective of resource saving, attention-aided FNN
is also superior to S-VBI. In particular, the MSE of attention-aided FNN with
only $1/4$ RF chains is comparable to that of S-VBI with $1/2$ RF chains. As a
result, the hardware and energy cost can be halved. Furthermore, given strict
target MSE performance and a limited number of RF chains, S-VBI may need to
estimate multiple times while attention-aided FNN completes the estimation at
once, saving more resources for data transmission. Such an advantage can be
very appealing in scenarios like high-mobility communication, where the
channel is fast time-varying with short channel coherence time.

(a) Without HAD

(b) With HAD

Figure 9: Impact of antenna number and RF chain ratio in the two considered
scenarios.

###  V-C Generalization Ability

The generalization ability to different parameters heavily influences the
practicality of neural networks. In the considered problem, there are two
categories of parameters, namely system parameters and channel parameters.
System parameters include the number of antennas, RF chains, and users, which
determine the input and output dimensions of the network. Channel parameters
include SNR, number of paths, angular spread, and gain distribution of channel
paths, which influence the input and output distributions of the network. For
system parameters, the numbers of antennas and RF chains are usually fixed in
practice, and different user numbers can also be handled by the same network
since a multi-user channel estimation problem is decomposed into multiple
single-user problems by exploiting the orthogonality of pilot sequences.
Therefore, we focus on the generalization performance of channel parameters.

The generalization to different SNRs is illustrated in Fig. 10. The legend
“trained with accurate SNRs” denotes that for each SNR, a dedicated model
trained with accurate SNR data is used for testing. In both scenarios, the
proposed networks can only handle tiny SNR mismatch between the training and
testing phases when the model is trained with a single SNR point and the
performance degradation can be very severe when the SNR mismatch is large. To
alleviate this issue, one common method is training with data under a variety
of SNRs, then the characteristics of different SNRs can be captured by a
single network. In simulation, we select five SNR points, namely 0, 5, 10, 15,
and 20 dB for training. Besides, the number of training samples from each SNR
point is kept same as when trained separately out of fairness. Based on our
simulation results, directly using MSE as loss can lead to poor performance
when different SNR points are trained together since the loss of high SNR data
will be overwhelmed by the loss of low SNR data. To ensure that all SNR
regimes get sufficient training, we use a heuristic loss function computed as

$\text{Weighted MSE
Loss}=\frac{1}{n}\sum_{i=1}^{n}(\text{SNR}_{i}\cdot\left\|\hat{\mbox{\boldmath$x$}}_{i}-\mbox{\boldmath$x$}_{i}\right\|^{2}),$
(9)

where the MSE is weighted by the SNR of data sample. As can be indicated by
the two close curves marked with circle and cross, networks trained with mixed
SNRs achieve similar performance as trained with accurate SNRs and
significantly outperform networks trained with a single SNR point.

(a) SNR generalization without HAD.

(b) SNR generalization with HAD.

Figure 10: Generalization to SNRs with different training methods in the two
considered scenarios.

As for the generalization to other parameters, detailed results are omitted
here due to space limitation while the trends and patterns are also similar.
In conclusion, through mixed parameters training and proper design of the loss
function, a single network with strong robustness can be obtained to handle
all situations during testing, which is very appealing in practical
applications.

###  V-D The Role of Attention

Although it is hard to rigorously analyze the representations learned by DNNs,
we still try to attain at least a primitive understanding of the role of
attention. Intuitively, the performance gain of attention can be considered to
come from the “divide-and-conquer” policy realized by the dynamic adjustment
of attention maps. In this way, sample-specific processing can be performed on
different data samples to improve the performance. Without attention, the
processing performed by the network is fixed for all data samples, which is
less advanced. Next, we would like to analyze the distributions of learned
attention maps to roughly corroborate this.

Due to the narrow angular spread characteristic, the channel distribution is
highly related to the average AoA parameter, or, more precisely, its sine
value. So, we select three sine value ranges for comparison, where the first
two ranges are close to each other and the third range is far away from the
first two ranges. The average attention maps of validation data samples whose
average AoAs are inside the three ranges are plotted in Fig. 11. The number of
elements of each attention map equals to the corresponding channel number of
the feature matrix and the values of the elements represent the scale factors
acting on the original features. Due to space limitation, only the 16-th to
the 48-th channels are displayed here. A larger scale factor indicates more
important channel of features. From the figure, we have the following
observations:

    * –

Without HAD, the role of attention is different in different depths of the
attention-aided CNN. Specifically, as is shown in the first two subfigures,
features are scaled in an angle-agnostic manner in shallower layers with small
differences among average attention maps of different sine value ranges while
the distributions of average attention maps become increasingly angle-specific
in deeper layers. Notice that, the mean value of the 38-th scale factor of the
third attention map varies significantly with sine value ranges. Reasonably,
it can be inferred as a key angle-related feature in the considered problem.
Such a phenomenon is also consistent with a typical discipline in DNNs that
earlier layer features are more general while later layer features exhibit
greater specificity[47].

    * –

The distributions of average attention maps of closer sine value ranges are
more similar. From the second subfigure, the curves of the first two ranges
are very close to each other, while the curve of the third range is apparently
different from them. It can be regarded as the embodiment of “divide-and-
conquer” since the channel estimation for data samples in the first two ranges
and the third range can be regarded as two different subproblems, which are
“divided” by different attention maps first and then “conquered” subsequently.

    * –

As is illustrated in the third subfigure, all scale factors in the fourth
attention map are 0.5, which is due to the zero output of the former ReLU
activation function and the Sigmoid activation function used to predict the
attention map. Therefore, the last attention module is actually useless and
can be removed during testing to further reduce the complexity[40].

    * –

From the fourth subfigure, the differences of average attention maps between
sine value ranges are bigger and the binarization level of scale factors is
higher in the HAD scenario. Only one attention module is used in the
attention-aided FNN, so the “divide” process has to be realized more
intensely, which is different from the attention-aided CNN used in the
scenario without HAD. Another reason might be that compared with the denoising
process in the former scenario, reversing the effect of $W$ is more angle-
related, therefore the “divide-and-conquer” policy is reflected more fully.
When dealing with a certain subproblem, only specific features are kept and
others are totally abandoned.

(a) The first attention map without HAD

(b) The third attention map without HAD

(c) The fourth attention map without HAD

(d) The attention map with HAD

Figure 11: Average attention maps of data samples in three ranges. The legend
$(a,b)$ denotes the range where the minimum and maximum sine values of average
AoAs are $a$ and $b$, respectively.

Apart from the statistical characteristics, Fig. 12 also presents the
attention maps of two exemplary data samples with close average AoAs. Although
the average AoAs are almost same, the attention maps of these two data samples
are still dramatically different, which reveals the sample-specific nature of
attention. The reason is that although average AoA can reflect most of the
channel’s characteristics, there are still some features, such as the specific
AoAs and gains of channel paths, which can also be exploited by attention for
further performance improvement.

(a) The third attention map without HAD

(b) The attention map with HAD

Figure 12: The attention maps of two exemplary data samples with very close
average AoAs.

###  V-E Complexity Comparison

Under typical system settings where $N=128$, $M=32$, $L_{p}=K=10$, and
$I_{E}=50$, the specific complexity of different algorithms is compared in
Table II. Notice that the last attention layer in attention-aided CNN is
removed during testing as mentioned above. Besides, for MMSE $3^{\circ}$, CCMs
computed by channel samples whose average AoAs have same sine values can be
shared to halve the number of parameters.

As we can see, without HAD, the number of parameters only increases $19.86\%$
with the use of attention, and the additional FLOPs overhead introduced by
attention is almost negligible. Although the FLOPs of attention-aided CNN are
slightly higher than MMSE currently, it will be much smaller than MMSE if the
antenna number keeps growing. Besides, the parameter number of MMSE
$3^{\circ}$ is also quite large since tens of CCMs are required to exploit the
narrow angular spread characteristic of channels.

In the scenario with HAD, we only compare three algorithms with practical
performance. Both attention-aided CNN and FNN have similar parameter numbers
while the FLOPs of attention-aided FNN is much lower. Remember that, its
performance is also better than attention-aided CNN, which indicates the
superiority of the proposed design. The FLOPs of S-VBI is significantly higher
than the DL-based methods. In simulation, when both run on CPU, attention-
aided FNN can be hundreds of times faster than S-VBI in terms of clock time
and the advantage is even more exaggerated if accelerated by GPU.

Metrics | Without HAD | With HAD
---|---|---
CNN | CNN+Att |  MMSE $3^{\circ}$ | CNN+Att | FNN+Att | S-VBI
FLOPs ($\times 10^{7}$) | 1.794 | $1.801$ | 1.689 | 0.531 | $0.103$ | 5.516
Parameters ($\times 10^{6}$) | 0.141 | $0.169$ | 1.966 | 1.002 | $1.072$ | 0
TABLE II: Complexity comparison of algorithms under typical system settings.

##  VI Conclusion

In this paper, we have proposed a novel attention-aided DL framework for
massive MIMO channel estimation. Both the scenarios without and with HAD are
considered and scenario-specific neural networks are customized
correspondingly. By integrating the attention mechanism into CNN and FNN, the
narrow angular spread characteristic of channel can be effectively exploited,
which is realized by the “divide-and-conquer” policy to dynamically adjust
attention maps. The proposed approach can significantly improve the
performance but is with relatively low complexity.

## References

    * [1] F. Rusek _et al_., “Scaling up MIMO: Opportunities and challenges with very large arrays,” _IEEE Signal Process. Mag_., vol. 30, no. 1, pp. 40–60, Jan. 2013.
    * [2] E. G. Larsson, O. Edfors, F. Tufvesson, and T. Marzetta, “Massive MIMO for next generation wireless systems,” _IEEE Commun. Mag_., vol. 52, no. 2, pp. 186–195, Feb. 2014.
    * [3] Y. S. Cho, J. Kim, W. Y. Yang, and C.-G. Kang, _MIMO-OFDM Wireless Communications with MATLAB_. Singapore: John Wiley $\&$ Sons (Asia) Pte Ltd. 2010.
    * [4] F. Sohrabi and W.Yu, “Hybrid digital and analog beamforming design for large-scale antenna arrays,” _IEEE J. Sel. Topics Signal Process_., vol. 10, no. 3, pp. 501–513, Apr. 2016.
    * [5] A. F. Molisch _et al_., “Hybrid beamforming for massive MIMO: A survey,” _IEEE Commun. Mag_., vol. 55, no. 9, pp. 134–141, Sep. 2017.
    * [6] S. Guo, H. Zhang, P. Zhang, P. Zhao, L. Wang, and M. Alouini, “Generalized beamspace modulation using multiplexing: A breakthrough in mmWave MIMO,” _IEEE J. Sel. Areas in Commun._ , vol. 37, no. 9, pp. 2014–2028, Jul. 2019.
    * [7] D. Fan _et al_., “Angle domain channel estimation in hybrid millimeter wave massive MIMO systems,” _IEEE Trans. Wireless Commun_., vol. 17, no. 12, pp. 8165–8179, Dec. 2018.
    * [8] J. Lee, G. Gil, and Y. Lee, “Channel estimation via orthogonal matching pursuit for hybrid MIMO systems in millimeter wave communications,” _IEEE Trans. Commun_., vol. 64, no. 6, pp. 2370–2386, Jun. 2016.
    * [9] S. Ji, Y. Xue, and L. Carin, “Bayesian compressive sensing,” _IEEE Trans. Signal Process_., vol. 56, no. 6, pp. 2346–2356, Jun. 2008.
    * [10] Y. Wang, A. Liu, X. Xia, and K. Xu, “Learning the structured sparsity: 3D massive MIMO channel estimation and adaptive spatial interpolation,” _IEEE Trans. Veh. Technol_., vol. 68, no. 11, pp. 10663–10678, Nov. 2019.
    * [11] X. Xia, K. Xu, S. Zhao, and Y. Wang, “Learning the time-varying massive MIMO channels: Robust estimation and data-aided prediction,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 8, pp. 8080–8096, Aug. 2020.
    * [12] Z. Qin, H. Ye, G. Y. Li, and B. F. Juang, “Deep Learning in Physical Layer Communications,” _IEEE Wireless Commun._ , vol. 26, no. 2, pp. 93–99, Apr. 2019.
    * [13] H. Ye, L. Liang, G. Y. Li, and B. Juang, “Deep learning-based end-to-end wireless communication systems with conditional GANs as unknown channels,” _IEEE Trans. Wireless Commun_., vol. 19, no. 5, pp. 3133–3143, May 2020.
    * [14] J. Gao, X. Yi, C. Zhong, X. Chen, and Z. Zhang, “Deep learning for spectrum sensing,” _IEEE Wireless Commun. Lett._ , vol. 8, no. 6, pp. 1727–1730, Dec. 2019.
    * [15] H. Sun, _et al_., “Learning to optimize: Training deep neural networks for wireless resource management”, IEEE Trans. Signal Process., vol. 66, no. 20, pp. 5438–5453, Oct. 2018.
    * [16] H. Ye, G. Y. Li, and B. F. Juang, “Deep reinforcement learning based resource allocation for V2V communications,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 4, pp. 3163–3173, Apr. 2019.
    * [17] L. Liang, H. Ye, and G. Y. Li, “Spectrum sharing in vehicular networks based on multi-agent reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 37, no. 10, pp. 2282–2292, Oct. 2019.
    * [18] L. Liang, H. Ye, G. Yu, and G. Y. Li, “Deep-learning-based wireless resource allocation with application to vehicular networks,” _Proc. IEEE_ , vol. 108, no. 2, pp. 341–356, Feb. 2020.
    * [19] J. Gao, C. Zhong, X. Chen, H. Lin, and Z. Zhang, “Unsupervised learning for passive beamforming”, IEEE Commun. Lett., vol. 24, no. 5, pp. 1052–1056, May 2020.
    * [20] H. Song, M. Zhang, J. Gao, and C. Zhong, “Unsupervised learning based joint active and passive beamforming design for recongurable intelligent surfaces aided wireless networks,” _IEEE Commun. Lett._ , early access, Dec. 2020. doi: 10.1109/LCOMM.2020.3041510.
    * [21] H. Ye, G. Y. Li, and B. Juang, “Power of deep learning for channel estimation and signal detection in OFDM systems,” _IEEE Wireless Commun. Lett._ , vol. 7, no. 1, pp. 114–117, Feb. 2018.
    * [22] P. Jiang, T. Wang, B. Han, X. Gao, J. Zhang, C. Wen, S. Jin, and G. Y. Li, “Artificial intelligence-aided OFDM receiver: Design and experimental results,” Dec. 2018, _arXiv:1812.06638_. [Online]. Available: https://arxiv.org/abs/1812.06638
    * [23] H. He, C. Wen, S. Jin, and G. Y. Li, “Model-driven deep learning for MIMO detection,” _IEEE Trans. Signal Process._ , vol. 68, pp. 1702–1715, Feb. 2020.
    * [24] H. He, C. Wen, S. Jin, and G. Y. Li, “Deep learning-based channel estimation for beamspace mmWave massive MIMO systems,” _IEEE Wireless Commun. Lett_., vol. 7, no. 5, pp. 852–855, Oct. 2018.
    * [25] W. Chen, B. Zhang, S. Jin, B. Ai, and Z. Zhong, “Solving sparse linear inverse problems in communication systems: A deep learning approach with adaptive depth,” _IEEE J. Sel. Areas Commun_., vol. 39, no. 1, pp. 4–17, Jan. 2021.
    * [26] P. Dong, H. Zhang, G. Y. Li, I. S. Gaspar, and N. NaderiAlizadeh, “Deep CNN-based channel estimation for mmWave massive MIMO systems,” _IEEE J. Sel. Topics Signal Process_., vol. 13, no. 5, pp. 989–1000, Sept. 2019.
    * [27] P. Wu and J. Cheng, “Acquiring measurement matrices via deep basis pursuit for sparse channel estimation in mmWave massive MIMO systems,” July 2020, _arXiv:2007.05177_. [Online]. Available: https://arxiv.org/abs/2007.05177
    * [28] X. Ma and Z. Gao, “Data-driven deep learning to design pilot and channel estimator for massive MIMO,” _IEEE Trans. Veh. Technol_., vol. 69, no. 5, pp. 5677–5682, May. 2020.
    * [29] Y. Yang, S. Zhang, F. Gao, J. Ma, and O. A. Dobre, “Graph neural network based channel tracking for massive MIMO networks,” _IEEE Commun. Lett._ , vol. 24, no. 8, pp. 1747–1751, Aug. 2020.
    * [30] Y. Yang, F. Gao, C. Xing, J. An, and A. Alkhateeb, “Deep multimodal learning: Merging sensory data for massive MIMO channel prediction,” Jul. 2020, _arXiv:2007.09366_. [Online]. Available: https://arxiv.org/abs/2007.09366
    * [31] W. Ma, C. Qi, Z. Zhang, and J. Cheng, “Sparse channel estimation and hybrid precoding using deep learning for millimeter wave massive MIMO,” _IEEE Trans. Commun_., vol. 68, no. 5, pp. 2838–2849, Jan. 2020.
    * [32] Y. Yang, F. Gao, G. Y. Li, and M. Jian, “Deep learning based downlink channel prediction for FDD massive MIMO system,” IEEE Commun. Lett., vol. 23, no. 11, pp. 1994–1998, Nov. 2019.
    * [33] A. Al-Hourani, S. Kandeepan, and A. Jamalipour, “Modeling air-to-ground path loss for low altitude platforms in urban environments,” in _Proc. IEEE Global Commun. Conf_., Austin, TX, 2014, pp. 2898–2904.
    * [34] Q. Cai, C. Dong, and K. Niu,“Attention model for massive MIMO CSI compression feedback and recovery,” _2019 IEEE WCNC_ , Marrakesh, Morocco, 2019, pp. 1–5.
    * [35] D. J. Ji and D. -H. Cho, “ChannelAttention: Utilizing attention layers for accurate massive MIMO channel feedback,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 5, pp. 1079–1082, May 2021.
    * [36] J. Xu, B. Ai, W. Chen, A. Yang, and P. Sun, “Wireless image transmission using deep source channel coding with attention modules,” Nov. 2020, _arXiv:2012.00533_. [Online]. Available: https://arxiv.org/abs/2012.00533
    * [37] H. Xie, F. Gao, S. Zhang, and S. Jin, “A unified transmission strategy for TDD/FDD massive MIMO systems with spatial basis expansion model,” IEEE Trans. Veh. Technol., vol. 66, no. 4, pp. 3170–3184, April 2017.
    * [38] Y. Yang, F. Gao, X. Ma, and S. Zhang, “Deep learning-based channel estimation for doubly selective fading hannels,” _IEEE Access_ , vol. 7, pp. 36579–36589, Mar. 2019.
    * [39] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016.
    * [40] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-Excitation networks,” Sept. 2017, _arXiv:1709.01507_. [Online]. Available: https://arxiv.org/abs/1709.01507
    * [41] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” Nov. 2017, _arXiv:1711.07971_. [Online]. Available: https://arxiv.org/abs/1711.07971
    * [42] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual Attention network for scene segmentation,” Sept. 2018, _arXiv:1809.02983_. [Online]. Available: https://arxiv.org/abs/1809.02983
    * [43] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _ICML 2015_ , Lille, France, Jul. 6-11, 2015, pp. 448–456.
    * [44] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _AISTATS 2010_ , vol. 9, pp. 249–256, May 2010.
    * [45] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Dec. 2014, _arXiv:1412.6980_. [Online]. Available: https://arxiv.org/abs/1412.6980
    * [46] R. Caruana, S. Lawrence, and L. Giles, “Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping,” in _NIPS 2000_ , Denver, CO, USA, Dec. 6-12, 2020.
    * [47] A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick, “On the importance of single directions for generalization,” in _ICLR 2018_ , Vancouver, BC, Canada, Apr. 30-May 3, 2018.