[126] |  MISO-DL-MU |  Continuous |  Rician fading |  Perfect |  Maximizing sum-rate |  CNN |  6 layers |  Generated by [27] |  CSI |  RIS phase shifts
|  |  |  |  |  |  FNN |  5 layers |  |  Effective channel matrix |  BS beamforming vector
[127] |  MIMO-DL-SU |  Continuous |  Rician fading |  Perfect |  Maximizing spectral efficiency |  DNN |  4 layers |  Generated by random exploration |  CSI |  RIS phase shifts
[128] |  Broadcasting Communications for IoTs |  Continuous |  Rician fading |  Statistical |  Maximizing spectral efficiency |  DNN |  4 layers |  Generated by random exploration |  CSI |  RIS phase shifts
[129] |  MISO-DL-SU |  Continuous |  Rayleigh fading |  Perfect |  Maximizing data rate |  FNN |  7 layers |  Generated as [17] |  CSI |  RIS phase shifts
[130] |  MISO-DL-MU |  Continuous/ discrete |  Geometry-based clustered delay line |  Estimated |  Maximizing sum-rate |  FNN |  6 layers |  Obtained from [131] |  CSI |  RIS phase shifts
|  |  |  |  |  |  k-means is used to cluster RIS elements based on their estimated cascaded channel coefficient without dataset. |  Estimated cascaded channel of RIS elements. |  RIS element clusters.

###  VI-B Unsupervised Learning-based Optimization

Supervised learning is data-demanding, but fine-grained labeled datasets may
be inaccessible in practice, preventing the application of supervised learning
algorithms. On the contrary, unsupervised learning can find hidden patterns of
unlabeled data without predefined targets or human intervention. Table XII
summarizes unsupervised learning algorithms for RIS-aided wireless networks.
It shows that neural networks are used in unsupervised manners for RIS phase-
shift configuration. Meanwhile, some classic unsupervised learning methods,
such as k-means, can also be applied for clustering RIS elements. This
subsection will introduce unsupervised neural networks and clustering
algorithms.

Figure 16: Unsupervised neural networks for optimizing RIS phase shifts.

####  VI-B1 Algorithm Training and Network Architecture of Unsupervised Neural
Networks

Supervised neural networks aim to minimize the loss between predicted results
and desired target, i.e., predicted and target data rate in the dataset.
However, in unsupervised neural networks, the loss function is directly
related to optimization objectives. Specifically, we consider a single-user
scenario as an example, and the user SNR is

$\eta_{k}=\frac{|(\bm{h}^{R}\bm{\Theta}\bm{G}+\bm{h}^{D})p|^{2}}{N_{0}^{2}},$
(27)

where $p$ is the transmit power at the BS, $\bm{G}$ indicates the channel gain
from BS antennas to RIS elements, $h^{R}$ indicates the channel gain from RIS
elements to the user, $h^{D}$ indicates the channel gain from BS antennas to
the user, $N_{0}^{2}$ is the noise power, and $\bm{\Theta}$ is the matrix of
RIS phase shifts. As shown in Fig. 16, the neural network considers the
channel state information as input, including $\bm{G}$, $\bm{h}^{R}$ and
$\bm{h}^{D}$. Then, the output is the predicted RIS phase shifts
$\bm{\Theta}$. The loss function is defined by

$Loss(w)=-\frac{1}{\mathcal{T}}\sum^{\mathcal{T}}_{i=1}(\bm{h}^{R}\bm{\Theta}\bm{G}+\bm{h}^{D}),$
(28)

where $\mathcal{T}$ is the minibatch size. To minimize the loss function
equation (28), $\bm{h}^{R}\bm{\Theta}\bm{G}+\bm{h}^{D}$ must be maximized.
This means that the neural network must predict appropriate RIS phase shifts
$\bm{\Theta}$ to maximize $\bm{h}^{R}\bm{\Theta}\bm{G}+\bm{h}^{D}$, and the
SNR will be maximized accordingly.

Table XII shows that most existing works apply 2 to 5 hidden layers. In
particular, the hidden layer numbers are related to the problem’s complexity.
The authors in [127] used 1 hidden layer with 40 nodes for $8\times 2$ MIMO,
and 2 hidden layers for $16\times 2$ MIMO, achieving satisfactory simulation
results without overfitting or underfitting. In addition, similar to
supervised learning, early stop is applied in [126] to prevent overfitting.

####  VI-B2 Clustering Algorithms

Clustering algorithms are usually unsupervised ML algorithms, i.e., k-means
and Density-based spatial clustering of applications with noise (DBSCAN).
These algorithms are designed to partition objects into multiple sets to
minimize the within-cluster sum of squares. Specifically, it aggregates
objects with the same hidden patterns. For instance, k-means is used in [130]
to group RIS elements according to estimated channel coefficients, and then
each group has the same RIS configurations to reduce the computational
complexity.

The main advantage of unsupervised learning is that it has no requirement on
predefined targets, which is more practical in real-world applications.
However, the absence of targets means that the model output is hard to
validate or verify, and the solution quality cannot be guaranteed.

TABLE XIII: Summary of reinforcement learning for RIS-aided wireless networks

Ref. |  Scenario |  Phase-shift resolution |  Channel settings |  CSI |  Objectives |  Algorithm |  State definition |  Action definition |  Reward function
---|---|---|---|---|---|---|---|---|---
[132] |  MISO-DL-MU NOMA |  Discrete |  Rayleigh fading |  Perfect |  Maximizing sum-rate |  DDPG |  Current RIS phases |  RIS phase shifts |  Sum-rate
[133] |  Point-to-point communications |  Discrete |  Wideband geometric |  Estimated |  Maximizing data-rate |  DRL |  CSI |  RIS phase shifts |  Data rate
[89] |  MISO-DL-MU UAV |  Continuous |  Saleh-Valenzuela |  Imperfect |  Maximizing secrecy rate |  DDPG |  CSI |  RIS phase shifts and BS beamforming vector |  Secrecy rate with penalty
[134] |  MISO-DL-MU with jammer |  Continuous |  Quasi-static flat-fading |  Perfect |  Maximizing sum-rate |  Fast-policy hill-climbing learning |  Previous jammer power and SINR, current CSI |  BS transmit power and RIS phase shifts |  Maximizing data-rate, decreasing BS power and SINR penalty
[135] |  MISO-DL-MU |  Continuous |  Rayleigh fading |  Delayed |  Maximizing secrecy rate |  DRL |  CSI, previous secrecy rate and transmission rate, QoS level |  BS beamforming vector and RIS phase shifts |  Maximize the system secrecy rate, guaranteeing QoS requirements.
[136] |  MISO-DL-SU |  Continuous |  Rayleigh fading |  Perfect |  Maximizing SNR |  DDPG |  SNR and current RIS phases |  RIS phase shifts |  Received SNR
[137] |  MISO-DL-MU THz |  Continuous |  Rayleigh distribution |  Perfect |  Maximizing sum-rate |  DDPG |  Current BS and RIS beamforming vectors, CSI |  BS beamforming vectors and RIS phase shifts |  Throughput and the penalty of adjusting the beamforming direction.
[138] |  MISO-DL-MU |  Continuous |  Quasi-static flat-fading |  Perfect |  Maximizing energy efficiency |  DRL |  CSI and energy level of RIS |  BS beamforming vector, RIS phase shifts and on/off |  Energy efficiency
[139] |  MISO-DL-SU |  Continuous |  Quasi-static flat-fading |  Perfect |  Power minimization |  DDPG |  CSI, previous outage events |  BS beamforming vector and RIS configurations |  Energy efficiency
[140] |  MISO-DL-MU |  Continuous |  Frequency flat fading |  Perfect |  Maximizing channel capacity |  DDPG |  Transmit and received power, previous action, and CSI |  BS beamforming vector and RIS phase shifts |  Channel capacity
[141] |  MISO-DL-MU mmWave |  Continuous |  3GPP model |  Perfect/ Imperfect |  Maximizing sum-rate |  Distributed RL |  CSI |  RIS phase shifts |  Data rate
[142] |  MISO-DL-MU NOMA |  Continous |  Rayleigh fading |  Perfect |  Maximizing energy efficiency |  Decaying DDQN |  RIS phases and positions, UE positions, and current BS power allocations |  RIS phase and position and BS beamforming changes |  Energy efficiency with penalty
[143] |  Multi-cell communications |  Continuous |  Rayleigh fading |  Imperfect |  Maximizing sum-rate |  Multi-agent DRL |  Local and neighbor CSI, local sum-rate |  RIS phase shifts, BS beamforming vector, and UE power changes |  Sum-rate with interference penalties
[144] |  MISO-UL-MU IoT UAV |  Continuous |  Rician fading |  Perfect |  Minimizing sum AoI |  DRL |  SNR and UAV height |  UAV altitude changes |  Negative summation of age of information
[145] |  MISO-DL-MU NOMA |  Continuous |  Rayleigh fading |  Perfect |  Maximizing sum-rate |  Object migration automation |  Current RIS phase |  Power allocation coefficient |  Sum-rate
|  |  |  |  |  |  DDPG |  Current RIS phase |  RIS Phase changes |  Sum-rate difference

###  VI-C Reinforcement Learning-based Optimization

Figure 17: DRL-empowered RIS-aided wireless networks

RL is the most widely applied ML technique for optimization, including model-
free (e.g., Q-learning, DQN, actor-critic learning) and model-based (i.e.,
dynamic programming) algorithms. However, defining the Markov decision process
(MDP) is fundamental to applying model-based or model-free RL
algorithms[146][147], and the RL agent interacts with the environment under an
MDP scheme to learn the best long-term policy. Given the current system state
$s$, the agent selects an action $a$ for implementation and receives a reward
$r$, and then the environment will move to the next state $s^{\prime}$. An MDP
model is critical to transforming the optimization problem into an RL context.
Specifically, environmental status, control variables, and optimization
objectives are defined as states, actions, and rewards, respectively. Then RL
algorithms can be used subsequently to maximize the reward and improve the
objective function. Table XIII summarizes existing studies that apply RL to
RIS-aided wireless networks. This subsection will first analyze the state,
action, and reward function definitions of these existing studies, and then
present the algorithm architecture and training methods.

####  VI-C1 State Definition

Table XIII shows that the state may be defined in various ways, e.g., CSI[133,
89, 134], current RIS phase[132, 137, 142], position[142, 144], energy
level[138], previous transmission rate[135]. Specifically, the state refers to
the environment status that should be considered for decision-making. For
example, the CSI has a great effect on the RIS phase shifts, and therefore CSI
is involved in the state definition of many studies [133, 89, 134]. Similarly,
RIS-aided UAVs are investigated in [144], and the UAV altitude is included in
the state definitions because the height will directly affect the channel
conditions.

####  VI-C2 Action Definition

In the context of MDP, the action indicates control variables that will change
the state, such as RIS phase shifts[132]-[145], BS beamforming [89, 134, 135,
137, 138], RIS positions[142] and elements on/off[138]. The control variables
in problem formulations are easily converted into actions. However, note that
many RL algorithms require discrete action spaces, but the control variables
in problem formulations are usually continuous as shown in Section III. The
first solution is to quantize the control variables. For instance, the BS
transmit power is quantized with an interval of 1 W[138], and the RIS phase
changes $\bigtriangleup\theta\in\\{-\frac{\pi}{10},0,\frac{\pi}{10}\\}$ in
[79]. Another solution is to apply the deep deterministic policy gradient
(DDPG) algorithm, which can handle continuous action-space problems[132, 89,
137, 139, 145].

####  VI-C3 Reward Functions

The reward function is a crucial part of RL. As shown in Table XIII, the
reward function definition mainly depends on the optimization objectives,
including data rate[132, 133, 141, 143], energy efficiency [138, 139, 142],
channel capacity[140], and SNR[136]. Moreover, the reward function can include
multiple objectives and constraints to balance the overall performance. As an
example, the reward function in [134] has data rate as a positive term to
maximize the data rate, while BS power consumption is a negative term to
reduce power consumption. RL focuses on the long-term accumulated reward,
which means it can better adapt to highly dynamic wireless environments
without requiring full knowledge of the defined problem.

####  VI-C4 Algorithm Architecture and Training

In Q-learning, the state-action values are updated by

$Q^{new}(s,a)=Q^{old}(s,a)+\alpha(r+\eta\max\limits_{a}Q(s^{\prime},a)-Q^{old}(s,a)),$
(29)

where $Q^{old}(s,a)$ and $Q^{new}(s,a)$ are old and new Q-values,
respectively. $\alpha$ is the learning rate ($0<\alpha<1$), and $\eta$ is the
discount factor $(0<\eta<1)$.

Equation (29) indicates that a Q-table is used to record all the state-action
values, leading to slow convergence for problems with large state-action
space. To this end, DQN is proposed to use neural networks for Q-value
estimation:

$Loss(\omega)=\mathscr{E}(r+\eta\max\limits_{a}Q(s^{\prime},a,\omega^{\prime})-Q(s,a,\omega)),$
(30)

where $\mathscr{E}$ represents the error between the predicted Q-value
$Q(s_{,}a,\omega)$ and target Q-value
$r+\eta\max\limits_{a}Q(s^{\prime},a,\omega^{\prime})$. $\omega$ and
$\omega^{\prime}$ are the weight of the main and target networks,
respectively. The main network is used to predict current Q-values by
$Q(s_{,}a,\omega)$, and the target network estimates target Q-values by
$Q(s^{\prime},a,\omega^{\prime})$.

In DQN, $\max\limits_{a}Q(s^{\prime},a,\omega^{\prime})$ indicates that the
target network will select the action and meanwhile evaluate the action, and
the maximizing operator will result in over-optimistic Q-value estimation.
Then double deep Q-learning (DDQN) is proposed to mitigate Q-value over-
estimation by

$\leavevmode\resizebox{413.38667pt}{}{$Loss(w)=\mathscr{E}(r+\eta
Q(s^{\prime},\arg\max\limits_{a}Q(s^{\prime},a,\omega),\omega^{\prime})-Q(s,a,\omega))$},$
(31)

where $\arg\max\limits_{a}Q(s^{\prime},a,\omega)$ means action selection of
the main network, and
$Q(s^{\prime},\arg\max\limits_{a}Q(s^{\prime},a,\omega),\omega^{\prime})$
indicates the action evaluation of the target network. Decoupling the action
selection and evaluation can provide more accurate Q-value prediction and
prevent over-estimation.

DRL has been used for RIS phase-shift optimization in [133, 136, 135]. In
these studies, continuous phase shifts are quantized to form discrete action
spaces for DQN or DDQN. On the contrary, DDPG can handle continuous action
spaces directly without quantization, which has been used for continuous RIS
phase-shift control in [132, 89, 137].

DDPG is considered a combination of actor-critic learning and DQN, in which
the actor network selects actions, and the critic network evaluates the state-
action values. The loss function of the critic network is defined as

$\leavevmode\resizebox{413.38667pt}{}{$Loss(w^{C})=\mathscr{E}(r+\eta
Q(s^{\prime},a(s^{\prime},\omega^{A^{\prime}}),\omega^{C^{\prime}})-Q(s,a,\omega^{C}))$},$
(32)

where $a(s^{\prime},\omega^{A^{\prime}})$ indicates that action $a$ is
selected by the target actor network with weight $\omega^{A^{\prime}}$, and
$Q(s^{\prime},a(s^{\prime},\omega^{A^{\prime}}),\omega^{C^{\prime}})$ means
the state-action value is evaluated by the target critic network with weight
$\omega^{C^{\prime}}$. For the actor network, the policy gradient is

$\displaystyle\nabla_{\omega^{A}}J\approx\frac{1}{\mathcal{T}}\sum^{\mathcal{T}}_{i=1}(\nabla_{a}Q(s,a,\omega^{C})$
$\displaystyle|_{s=s_{i},a=a(s_{i},\omega^{A})}$ (33)
$\displaystyle\cdot\nabla_{\omega^{A}}a(s_{i},\omega^{A})|_{s=s_{i}}),$

In equation (33), the critic network provides the Q-value $Q(s,a,\omega^{C})$,
and it represents the expected accumulated reward for a given pair $(s,a)$.
The actor network is trained to produce actions that can result in the maximum
state-action value as predicted by the critic network. Therefore, a common
approach to calculate the loss function of the actor network is

$Loss(w^{A})=-\frac{1}{\mathcal{T}}\sum^{\mathcal{T}}_{i=1}(Q(s_{i},a_{i},w^{C})),$
(34)

which is computed by using the negative mean of the Q-values predicted by the
critic network.

Figure 18: Comparison between conventional Air-FL and Air-FL with RISs

Fig. 17 shows DRL-empowered RIS-aided wireless networks, which include DDQN
and DDPG as two DRL examples. Based on the current state $s$, the agent
selects BS beamforming vectors and RIS phase shifts as the action $a$. Then
the action $a$ is implemented and rewards $r$ are collected, e.g., sum-rate,
energy efficiency, or power consumption. The system will arrive at a new state
$s^{\prime}$ that is indicated by CSI, user positions, or SNR. The experience
tuple $<s,a,r,s^{\prime}>$ is saved in the experience pool, and a mini-batch
is sampled for network training. For the DDQN algorithm, the main network is
trained as equation (31), and the target network will copy the weight of the
main network, providing a stable reference. By contrast, the actor and critic
networks are trained by equations (32) and (33) in DDPG, and it applies slow
update strategies for target networks.

Fig. 17 presents the application of DDQN and DDPG to joint active and passive
beamforming problems. Note that here the DDQN and DDPG algorithms can be
easily generalized to many other RL algorithms without loss of generality.
This scheme can also be applied to other RIS-related scenarios. For instance,
for the UAV-RIS joint optimization problem, one can include the UAV control
variables in the action definition, and add UAV altitude in the state.
Finally, there have been various reinforcement learning algorithms, but one
common deficiency is the low sampling efficiency. It requires substantial
numbers of interactions for agent training, leading to large costs in real-
world applications, e.g., hundreds of millions of samples.

TABLE XIV: Summary of federated learning and RIS-aided wireless networks

Ref. |  Scenario |  Phase-shift resolution |  Channel settings |  CSI |  FL-related objectives |  Control variables |  Constraints |  Algorithms
---|---|---|---|---|---|---|---|---
[148] |  AirFL with NOMA |  Continuous |  Rician channel |  Perfect |  Minimizing the gap between converged and optimal training loss. |  FL device selection, receiver beamforming and RIS phase shifts. |  Device selection, receiver beamform, and phase shifts constraints |  AO, Gibbs sampling, SCA
[149] |  AirFL with RISs |  Continuous |  Rician fading |  Perfect |  Minimizing global loss |  Transmit power and RIS phase shifts |  Power and dual constraints |  AO, QCQP, SDP
[150] |  AirFL with NOMA |  Continuous |  Rayleigh fading/ Rician fading |  Perfect |  Minimizing FL training gap |  BS transmit power and RIS configurations |  Transmit power, target rate, MSE tolerance, and RIS configuration constraints. |  AO, SCA, SDR
[151] |  RIS-enhanced FL |  Discrete |  Generated by simulator |  Perfect |  Minimizing average system power consumption |  RIS phase shifts and bits, bandwidth, and CPU frequency |  Training latency, convergence rate, and learning performance constraints |  Stochastic Lyapunov optimization, greedy algorithm
[152] |  AirFL with RISs |  Continuous |  Empirical channel fading |  Perfect/ Imperfect |  Minimizing the MSE of the aggregated AirFL model |  Receive and transmit beamformer, RIS phase shifts |  Total transmit power, phase shifts, and target rate constraints |  AO, SCA
[153] |  AirFL with RISs and NOMA |  Continuous |  Rayleigh fading |  Perfect |  Maximizing the achievable hybrid rate of FL and NOMA |  User transmit power, BS receive scalar, and RIS phase shifts |  Target rate and MSE, phase configuration and total transmit power constraints |  AO, SCA, SDR
[154] |  AirFL with RISs and NOMA |  Continuous |  Rayleigh fading |  Perfect |  Minimizing the FL MSE and cardinality |  Transmit power, receive scalar, reflection coefficients, and learning participants |  Total transmit power, phase configuration, target MSE, and the number of learning devices |  AO, SDR, SCA
[155] |  AirFL with RISs |  Continuous |  Obtained from [156] |  Perfect |  Minimizing the effect of device selection and the communication error on the convergence rate |  Device selection, over-the-air transceivers, and RIS phase shifts |  Device selection, receiver beamforming, and phase configurations |  Gibbs-sampling, SCA
[157] |  AirFL with RISs |  Continuous |  Obtained from testbed |  Perfect |  Maximizing FL utility |  RIS phase shifts, user-RIS association, and bandwidth allocation |  Bandwidth allocation, RIS phase configurations and association, target SNR constraints |  Matching game, bisection search
[158] |  AirFL with RISs |  Continuous |  Rayleigh fading |  Perfect |  Power minimization |  CPU frequency, power and bandwidth allocation, RIS configurations and accuracy design |  Task completion time, maximum transmit power, phase configuration, total bandwidth |  AO, SDP, MM
[159] |  FL-aided RIS optimization |  Continuous |  Wideband geometric/ Rayleigh fading |  Predicted |  Average rate maximization |  Local models: local devices train local DNNs to predict channel rate using sampled channel vectors; Global model: edge server aggregates local DNN models and average.1
[160] |  FL-aided mobile RIS optimization |  Continuous |  Rician fading |  Predicted |  Sum-rate maximization |  FL-DDPG is applied. Neural networks are trained at local agents and then aggregated to predict Q-values. Control variables include RIS positions, phase shifts, and AP power allocation.

  * 1

The columns are combined because [159] [160] are different from other studies
by using FL as an optimization approach, while FL in other works
of Table XIV is part of the optimization objectives. Therefore, instead of
showing control variables and constraints, it is essential to present the
local
and global models of FL-based optimization algorithms.

###  VI-D Federated Learning and RISs

Different from conventional centralized ML algorithms, FL trains the model
across multiple decentralized edge devices or servers that hold local datasets
without exchanging data. In FL, each edge device will train a local model
using local samples, and then a global model is formed by aggregating local
model parameters. Afterwards, edge devices download the global model to update
local models. Table XIV summarizes existing works focusing on FL and RIS-aided
wireless communications. This subsection first discusses RIS-enhanced over-
the-air FL (AirFL), and then introduces how to use FL optimization in RIS-
aided environments.

Figure 19: Graph learning for RIS-aided wireless networks

####  VI-D1 RIS-enhanced Over-the-air FL

The main advantage of FL is that it helps preserve data security and privacy,
and the distributed property makes wireless networks an ideal platform for FL
training. Therefore AirFL is proposed to combine FL with wireless
communications. In particular, AirFL implements FL in wireless networks, using
edge devices for local model training and edge servers for model aggregation.

However, the information exchange between local and global servers may be
affected by unreliable wireless links, limited bandwidth, signal distortion,
dynamic channel conditions, and so on. The uncontrollable signal propagation
path can degrade the FL performance, e.g., slow uploading of local models due
to low data rate. Therefore, RISs are combined with AirFL to realize the full
potential of FL.

As shown in Fig. 18, in conventional Air-FL, obstacles may lead to high
penetration loss between edge devices and edge servers, and then the low
channel capacity will result in slow model uploading and downloading. Finally,
the slow parameter exchange efficiency may degrade the convergence rate and
lower the accuracy of Air-FL. By contrast, in Air-FL with RISs, the indirect
transmission between UE-RIS-BS provides an alternative transmission path for
local model uploading or global model downloading. RISs improve the channel
capacity by manipulating the signal propagation path. Therefore, efficient
model uploading and downloading will improve the convergence rate and
precision of Air-FL.

There are a few works that investigate how to enhance FL performance in RIS-
aided wireless networks by minimizing global training loss[148, 149], MSE[150,
154], power consumption[158, 151], maximizing the FL utility[157]. In [149],
Yang et al. aim to minimize the global training loss of FL by controlling
transmit power and RIS phase shifts, and the optimization problem is solved by
AO-based QCQP and SDP. [157] proves that RISs can improve more than 30%
prediction accuracy of AirFL, and a 10 times lower AirFL test error is
reported in [154] by using multi-RIS. In these works, the FL performance is
improved by optimizing the resource allocation and user-RIS association, and
then edge users can efficiently upload the local models. Meanwhile, it is
worth noting that these works still rely on model-based optimization
algorithms, such as AO, QCQP[149], SCA[152, 153, 154], and MM[158].

####  VI-D2  FL for RIS-aided Wireless Communications

FL can also be used to optimize the performance of RIS-aided wireless
communications. For example, deploying a local FL model in RISs may reduce the
communication overhead between RISs and the BS, since only local model
parameters are shared instead of sharing the whole dataset. In addition, FL
can better protect private information such as user CSI, which may be used to
infer user locations. Specifically, FL is used in [159] and [160] for average
rate maximization, in which local models are deployed in user devices and the
global model is aggregated by edge servers. In [159], federated neural
networks consider sampled channel vectors as input to predict achievable
rates. FL and DDPG are combined in [160], and the local neural networks used
in DDPG will be aggregated and updated.

FL is an appealing technique for wireless networks as a distributed ML
algorithm. However, the distributed implementation also leads to high
communication overhead due to frequent parameter sharing. Meanwhile, the local
devices may have different computational capabilities and storage capacities,
and such heterogeneity may affect model aggregation and update in FL.

###  VI-E Graph Learning

Graph learning refers to a group of ML techniques in the graph domain,
including graph neural networks (GNN), graph attention networks (GAN) and
graph convolution networks (GCN). Compared with CNN, which operates on regular
Euclidean data like images (2D grid) and text (1D sequence), graph learning is
more efficient in describing graphs and structures. Graph learning aims to
transform nodes, edges, and their features into low-dimension vector spaces by
preserving properties such as graph structure[161].

Wireless networks are highly dynamic, and wireless data may be collected from
non-Euclidean domains, which is represented by graph structure with high
dependency on network topology. The conventional approach of data processing
is to convert the data with graph structure into Euclidean domain, but such
transformation leads to high complexity and extra overhead. By contrast, graph
learning enables the graph-structured data to be processed effectively, and
transforming the wireless network topology into graphs can better describe the
association and interference between network devices[162]. Therefore, graph
learning has been applied to power control and interference management [163],
resource allocation [164, 165], network slicing [166], and so on. In the
following, GNN is used as an example to introduce graph learning fundamentals,
and then we explain how to apply graph learning for RIS control and
optimizations.

####  VI-E1 GNN Fundamentals

The primary motivation for developing GNN is to extend the existing neural
network architecture into graph-related data processing capabilities[167]. In
a graph, each node is described by its features and related nodes. Suppose
that $z_{v}$ is a state vector to describe the features of node $v$, and it is
defined by

$z_{v}=f(y_{v},y_{v}^{ed},y_{v}^{ne},z_{v}^{ne}),$ (35)

where $y_{v}$ and $y_{v}^{ed}$ are the features of node $v$ and its edge, and
$z_{v}^{ne}$ and $y_{v}^{ne}$ are the state and features of neighbour nodes,
respectively. Then, $z_{v}$ and $y_{v}$ are used to produce an output $o_{v}$
by

$o_{v}=g(z_{v},y_{v}),$ (36)

where $g$ is the output function to map the relationship between states,
features, and outputs.

Similarly, by collecting all the states and features, we have

$Z^{l+1}=f(Z^{l+1},Y),$ (37) $O=g(Z,Y_{N}),$ (38)

where $Z^{l+1}$ indicates all the states at $l^{th}$ iteration, $Y$ indicates
all the features, $Y_{N}$ means the node features, and $O$ is the overall
output. Equation (37) shows that the system state is updated in an iterative
manner, which is inspired by Banach’s fixed point theorem [168]. Finally,
similar to conventional neural networks, GNN aims to minimize the loss
function.

####  VI-E2 Graph Learning for RIS Control and Optimizations

Interference control is an important technique for multi-user environments to
maximize the system sum-rate, and the interactions between RISs and UEs are
easily described by a graph. The graph in Fig. 19 includes $K+1$ nodes, in
which one node represents the RIS and the rest are $K$ UEs. Given this scheme,
GNN is applied to user scheduling and RIS configurations in [169] and [170].
In particular, GNN is trained in an unsupervised manner, and the inputs are
user weights and pilot sub-frames of the scheduled users, and the outputs are
RIS configurations and beamformers. Similarly, unsupervised GNN is applied in
[171] for network utility maximization, which takes pilot signals as input to
optimize the BS beamforming and RIS configurations.

In [169, 170, 171], a useful feature of GNN is used to reduce the interference
between users. Specifically, when updating one node in the GNN, all the
neighbour nodes will be included in the updating function, which means GNN can
better capture the mutual interference between users. Meanwhile, RIS node
updating is a function of all the user nodes, enabling GNN to configure RIS
elements to improve the channel capacity of all users. In addition, the
authors in [171] note that another key advantage of GNN is the generalization
capability. For instance, when the number of cell users constantly changes,
conventional FNN must be re-trained to handle various user numbers. In
contrast, a GNN can generalize to different numbers of users by simply adding
and removing components in its feature extraction and information exchange
stages. Such generalization capability can considerably alleviate ML model
training efforts.

Graph learning is one of the most state-of-the-art ML techniques. However, the
application to wireless networks is still in a very early stage. The real-time
wireless environment can produce dynamic and generative changing graphs, which
may prevent the application of graph learning.

###  VI-F Transfer Learning

Long training time and slow convergence are common issues of most ML
algorithms, and one of the main reasons is that the model must explore the
task from scratch. Fast decision-making is critical in wireless
communications, but the low sampling efficiency may prevent applying ML to
RIS-aided wireless networks. This subsection will introduce transfer learning
fundamentals and explain how transfer learning can improve ML-enabled wireless
networks with RISs.

####  VI-F1 Transfer Learning Fundamentals

Transfer learning can be combined with many ML algorithms, and here we
consider transfer reinforcement learning (TRL) as an example[172]. In
conventional RL, the decision-making $\mathcal{D}_{RL}$ of one agent is
described by

$\mathcal{D}_{RL}:s\times\mathscr{K}\rightarrow a,r,$ (39)

where $\mathscr{K}$ represents the agent’s knowledge, $s$, $a$, and $r$ are
the current state, selected action, and received reward, respectively. In
equation (39), the agent utilizes the collected knowledge $\mathscr{K}$ for
decision-making and action selection.

By contrast, the decision-making in TRL is

$\mathcal{D}_{TRL}:s\times\mathcal{M}(\mathscr{K}_{expert})\times\mathscr{K}_{learner}\rightarrow
a,r,$ (40)

where $\mathscr{K}_{expert}$ and $\mathscr{K}_{learner}$ are the knowledge of
the expert and learner agents, respectively. The learner is designed to solve
the target task, and the expert has some existing knowledge of related source
tasks. Considering the similarities between the source and target tasks, the
expert’s experience may be reused by the learner as prior knowledge. The
$\mathcal{M}$ in equation (40) defines a mapping function.
$\mathcal{M}(\mathscr{K}_{expert})$ indicates that the expert’s experience
will be transformed into digestible knowledge, boosting the learning process
of the learner. With existing prior knowledge, the learner can achieve a jump-
start at the exploration phase, achieving a higher exploration efficiency and
average reward with faster convergence[173].

Figure 20: Transfer reinforcement learning for RIS-aided wireless networks

####  VI-F2 Transfer Learning-boosted Wireless Networks with RISs

Wireless networks can be highly dynamic. For example, user numbers and CSI
patterns may change quickly in a short period of time, and then the RIS
control policy may need to be retrained to handle these dramatic changes.
However, ML algorithms usually require many training iterations, preventing
the application to dynamic wireless networks. To this end, TRL may become a
promising solution. Fig. 20 illustrates how TRL is used for RIS-aided wireless
networks, which includes source and target tasks. We assume that the expert
agent has existing knowledge of the source task, BS beamforming, and the
learner agent is designed for the target task, joint active and passive
beamforming. Due to the potential task similarities, the learner may reuse the
expert’s experience to better handle target tasks. However, note that the
expert’s knowledge may exist in various ways, e.g., state-action values and
action selections, and then the mapping function may be defined in different
manners. Fig. 20 provides an example by finding similar states and actions,
and a Q-value-based mapping function can be defined by

$\begin{aligned}
Q^{new}(s^{L},a^{L})=&Q^{E}(\mathcal{M}(s^{L}),\mathcal{M^{\prime}}(a^{L}))+Q^{old}(s^{L},a^{L})+\\\
&\alpha(r+\eta\max\limits_{a}Q(s^{\prime},a)-Q^{old}(s^{L},a^{L})),\end{aligned}$

(41)

where $s^{L}$ and $a^{L}$ are the learner’s state and action, $\mathcal{M}$
and $\mathcal{M^{\prime}}$ are the state and action map functions,
respectively, and $Q^{E}$ indicates the state-action value of the expert.
Compared with conventional RL, the main difference is that
$Q^{E}(\mathcal{M}(s^{L}),\mathcal{M^{\prime}}(a^{L}))$ is involved as an
extra reward for selecting $a^{L}$ under $s^{L}$. In particular, Fig. 20 shows
the steps of defining mapping functions for active and passive beamforming
tasks. Firstly, the state mapping function $\mathcal{M}$ is defined to find
$s^{E}=\mathcal{M}(s^{L})$, finding similar environment states such as CSI or
current BS beamforming vectors between the learner and expert agents.
Similarly, the action mapping function $\mathcal{M^{\prime}}$ aims to find
similar beamforming decisions between the learner and expert action spaces.
Finally, by finding these similar network states and beamforming decisions, as
shown in equation (41), good actions with high Q-values in the expert can
provide extra rewards for the learner. Then the learner is encouraged to
select better actions to achieve a higher sum-rate or energy efficiency.

With transfer learning, the RL agent can achieve higher exploration efficiency
and faster convergence, enhancing the efficiency of RIS-aided wireless
networks. Transfer learning has been used in [172] for joint resource
allocation of network slicing, and [174] for mmWave networks, achieving faster
convergence and better network performance. Similarly, transfer learning can
be applied to ML-enabled RIS optimization for faster convergence and achieving
prompt phase-shift responses. Transfer learning is a very useful technique to
mitigate ML model training effort. However, note that transfer learning relies
on existing experts to reuse prior knowledge, and the mapping function
definition may be difficult due to the inherent task difference between
experts and learners.

###  VI-G Hierarchical Learning

Figure 21: Hierarchical reinforcement learning for RIS-aided wireless networks

Hierarchical learning is another technique that can be used for optimizing
RIS-aided wireless networks. The main idea of hierarchical learning is to
decouple the long-term task into multiple achievable goals to increase
exploration efficiency[175]. In particular, it defines a meta-controller to
select goals and a sub-controller to achieve these goals. Based on the short-
term performance of the sub-controller, the meta-controller can adjust the
goal dynamically to guarantee the long-term performance of the whole system.
Hierarchical learning can also be applied to optimization problems that
include multiple control variables with different time scales[57]. For
instance, in [57], Zhou et al. consider a meta-controller for sleep control,
and sub-controllers for transmission power and RIS control, enabling control
variables with different time scales.

Fig. 21 shows how hierarchical reinforcement learning is applied to RIS-aided
wireless networks, and the agent consists of a meta-controller and a sub-
controller. Specifically, the sub-controller can generate long-term policy
instructions for the sub-controller, such as the maximum number of active RIS
elements that is available. Then, as shown in Fig. 21, given high-level goals,
the sub-controllers can select short-term decisions for RIS phase shifts.
Meanwhile, the meta-controller focuses on average power consumption in a
period as long-term network performance, and the sub-controller accounts for
delay or data rate as instant metrics. This scheme can coordinate control
variables with different time scales, balancing instant and long-term network
metrics. More specifically, the state-action value of the meta-controller is
updated by:

$\begin{aligned}
&Q_{meta}^{new}(s_{meta},g_{meta})=Q_{meta}^{old}(s_{meta},g_{meta})+\\\
&\alpha(r_{ex}+\eta\max\limits_{g}Q_{meta}(s_{meta}^{\prime},g)-Q_{meta}^{old}(s_{meta},g_{meta})),\end{aligned}$

(42)

where $s_{meta}$ and $s_{meta}^{\prime}$ is the current and next meta-states,
$g_{meta}$ is the goal, and $r_{ex}$ is the extrinsic reward, respectively.
$Q^{old}_{meta}$ and $Q^{new}_{meta}$ are old and new state-action values for
the meta-controller, indicating the accumulated reward by selecting $g_{meta}$
under state $s_{meta}$.

Similarly, the Q-value of the sub-controller is updated by

$\begin{aligned}
Q&{}_{sub}^{new}(s_{sub},g_{meta},a_{sub})=Q_{sub}^{old}(s_{sub},g_{meta},a_{sub})+\\\
&\alpha(r_{in}+\eta\max\limits_{a}Q_{sub}(s_{sub}^{\prime},g_{meta},a)-Q_{sub}^{old}(s_{sub},g_{meta},a_{sub})),\end{aligned}$

(43)

where $s_{sub}$ and $s_{sub}^{\prime}$ are current and the next sub-states,
$a_{sub}$ is the action, and $r_{in}$ is the intrinsic reward. $Q^{new}_{sub}$
and $Q^{old}_{sub}$ are defined similarly as the meta-controller, indicating
the expected reward of selecting $a_{sub}$ under state $s_{sub}$ and goal
$g_{meta}$. Equation (43) shows that the sub-controller is under the policy
control of the meta-controller.

Hierarchical learning is a promising technology to enable hierarchical
autonomy in RIS-aided wireless networks. However, one key challenge is to
define the relationship between different hierarchies, e.g., meta-controller
and sub-controllers. In addition, decoupling one task into multiple sub-tasks
can be difficult in highly-dynamic wireless networks, which may prevent the
application of hierarchical learning.

Figure 22: Meta-learning for RIS-aided wireless networks TABLE XV: Summary of
ML-based control and optimization algorithms for RIS-aided wireless networks.

ML techniques |  Typical algorithms |  Main features |  Difficulties |  RIS-related applications
---|---|---|---|---
Supervised learning |  Supervised DNN, CNN, decision trees, and support vector machine. |  The algorithm is trained to map the relationship between the given input and labeled output for classification and prediction. The input data is fed into the model, and then the model parameters are adjusted until the output is properly fitted. |  1) Supervised learning relies on fine-grained datasets to train the algorithm; 2) The algorithm training may be time-consuming; 3) The model is easy to be overfitted. |  Supervised learning is a promising technique if there exist fine-grained datasets, and then various neural networks may be used to predict full channel states [117] or optimal RIS phase shifts[121]\- [125].
Unsupervised learning |  k-means, DBSCAN, and unsupervised neural networks. |  Unsupervised learning algorithms aim to unveil hidden patterns of unlabeled datasets. |  The result performance is hard to be testified or explain. |  Unsupervised DNN can be directly used for optimization problems in RIS-aided networks without involving datasets, which designs RIS phase-shift by defining objectives as loss functions[127]-[129].
Reinforcement learning |  Q-learning |  The agent interacts with the environment under an MDP framework, recording experience by a Q-table. |  1)Long convergence time for large state-action problems; 2) Discrete states and actions only. |  RL is the most widely applied ML technique for the control and optimization of RIS-aided wireless networks, e.g., power minimization [139], sum-rate[132, 133, 144, 145], secrecy rate [89, 135], and energy efficiency[138, 142]. DDPG is especially useful considering the continuous RIS phase-shift control requirements [132, 89]. RL can also be combined with other ML techniques, e.g., transfer reinforcement learning, federated reinforcement learning, and meta reinforcement learning.
|  Actor-critic learning |  The actor is defined to select actions, while the critic evaluates the actions. |  1) Long convergence iterations; 2) Unstable performance due to the interaction between actor and critic. |
|  Deep reinforcement learning |  DRL applies neural networks to predict state-action values, solving the large state-action issue of tabular Q-learning. |  1) Hyperparameter tuning can be difficult when lacking experience. 2) The sampling efficiency is low. |
|  Double deep Q-learning |  DDQN provides a more accurate Q-value estimation by decoupling the action selection and evaluation. |  |
|  Multi-agent reinforcement learning |  Each agent applies RL or DRL independently to optimize its performance or achieve an overall goal. |  The coordination mechanism of multiple agents must be carefully designed. |
|  DDPG |  DDPG combines actor-critic with policy gradients, optimizing problems with continuous action space. |  1) Unstable and heavily dependent on appropriate hyperparameters; 2) Overestimation in critic network. |
Federated learning |  Federated deep learning, federated DRL |  Local models are first trained using local datasets, and then the parameters are aggregated to form a global model. Local devices will download the global model to update local models. User privacy is well protected in FL. |  1) High communication overhead due to parameter exchange; 2) The local device heterogeneity will affect the system performance. |  On the one hand, RISs can improve the AirFL performance by improving the channel capacity; on the other hand, FL is used to optimize RIS-aided network performance [159, 160].
Graph learning |  Graph neural networks, and graph attention networks. |  Graph learning refers to ML on graphs. It maps the graph features to vectors with the same dimensions in the embedding space, which is used for link prediction, matching and classification. |  1) Dynamic and generative changing graph; 2)Interpretability of graph learning. |  GNN is used for user schedule and RIS configurations in [169, 170, 171] to maximize network utility and sum-rate. The general application of graph learning is still an open issue.
Transfer learning |  Transfer reinforcement learning, transfer supervised learning |  Transfer learning aims to reuse the existing knowledge of experts to accelerate the learning process on target tasks, achieving faster convergence and less training efforts. |  1) The mapping function is hard to design, changing with different algorithms; 2) Transfer learning is vulnerable to adversarial attacks. |  When there are existing experts or source tasks, transfer learning may be used to accelerate ML algorithm training in RIS-aided wireless networks[172].
Hierarchical learning |  Hierarchical reinforcement learning, hierarchical deep learning |  Hierarchical learning decouples the task into multiple sub-tasks and goals, increasing the task exploration efficiency. |  1) The goal and sub-task selection require case-by-case analyses; 2) The relationship between the meta-controller and the sub-controller may be unstable. |  Hierarchical learning is used for optimizing RIS-aided networks with control variables that have different time scales or sparse rewards [57, 103].
Meta-learning |  Meta reinforcement learning, supervised meta-learning |  Using experience of former learning tasks to improve the performance on target tasks. The ML model will learn how to learn across tasks. |  1) Source task distribution must be carefully designed; 2) How to prevent overfitting and underfitting. |  Pre-training ML models at the BS for RIS channel estimation[176]; Model-agnostic meta-learning for joint RIS phase control and power allocation[177].

###  VI-H Meta-Learning

Meta-learning refers to ML algorithms that extract the experience of multiple
learning episodes, e.g., a distribution of related tasks, and then use such
prior training to improve the performance on target tasks[178]. In particular,
meta-learning is designed to learn how to learn across tasks, and this
learning-to-learn design can bring several benefits, such as improved training
and learning efficiency. In addition, it is better aligned with human learning
features, where learning skills are constantly improved on a lifetime
timescale and evolutionary policy[179].

RIS-aided networks may include diverse elements, such as RISs, BSs, UAVs, etc,
and it can be difficult to train ML models from scratch and meanwhile jointly
consider all these network elements. Fig. 22 shows an example of using meta-
learning schemes for UAV-aided joint active and passive beamforming, in which
RISs are deployed on the UAV for location flexibility. The ML model is first
pre-trained by three existing tasks such as RIS passive beamforming, BS
beamforming, and UAV trajectory design. Then, using prior experience, the ML
model is expected to learn quickly on the target task, which will jointly
consider RISs, BSs, and UAVs. Additionally, such a constant learning scheme
can be more useful when other future tasks are expected, and incoming new
tasks are always trained based on plenty of former knowledge. There are few
works on applying meta-learning to RIS-aided networks. For instance, Jung et
al. apply meta-learning for RIS channel estimation, and the ML model is pre-
trained at the BS by using pilot signals to rapidly estimate RIS
channels[176]. In [177], model-agnostic meta-learning is used for joint RIS
phase-shift control and power allocation, which has a faster convergence rate
than baseline ML algorithms.

However, meta-learning must balance the meta-training and self-learning
phases. Specifically, meta-training with a wide variety of tasks may lead to
underfitting, which means that the agent is unable to specialize to the target
task when self-learning. By contrast, if the meta-training tasks are too
specific, the knowledge learned on the source tasks may have difficulty in
generalizing to target tasks[179]. Therefore, the source task distribution in
the meta-training phase has to be carefully selected.

###  VI-I Discussions and Numerical Results

ML offers promising opportunities for optimizing RIS-aided wireless
communications. Table XV overviews various ML techniques 444Note that there
are many ML algorithms applied to wireless communications. Instead of
collecting all the existing ML algorithms, Table XV provides a compressed
taxonomy to understand the feature of each technique along with RIS control
applications..

Supervised learning trains ML models to best map the input to output, e.g.,
CSI and user position to RIS phase shifts. However, the model training relies
on fine-grained labeled datasets, which may be inaccessible in practice. By
contrast, unsupervised learning has no need for labeled datasets, and it
involves the objective function in the loss function for improvement. Such
unsupervised learning approaches can reduce the dependence on labeled
datasets, but the generated results are hard to validate due to the absence of
labeled data in most circumstances.

RL is the most widely applied ML technique for optimization problems, and each
RL algorithm has its own features and difficulties. For example, DDPG can
handle continuous action space of RIS but can be unstable[132, 89, 137, 139],
and DDQN can prevent overestimation but sampling efficiency is low [142]. FL
and graph learning are newly emerging ML techniques. Most existing works
consider RIS-enhanced AirFL, demonstrating that RISs can improve the training
efficiency and performance of FL [148, 149, 150, 154, 158, 151]. Graph
learning has shown great potential in many other fields, and wireless network
applications include power control and interference management [163], resource
allocation [164, 165], and network slicing [166]. Despite the advantages,
applying graph learning to wireless networks is still an open issue that
requires more effort.

Figure 23: Convergence performance of transfer deep reinforcement learning
(TDRL) and DRL. 1) TDRL: We assume there is an existing DRL agent that has
been trained under a limited number of UEs. Then a TDRL agent will reuse the
expert’s prior knowledge to adapt to the environment with more diverse UEs. 2)
DRL: conventional DQN-based RIS phase-shift control. Figure 24: Energy
efficiency performance by joint sleep control and RIS phase-shift design. We
consider a multi-BS and multi-RIS heterogeneous network. Sleep control is a
long-term decision to decide the BS on/off status, while RIS phase-shift
control is a short-term optimization based on dynamic channel status. Detailed
parameters can be found in [103].

Transfer learning and hierarchical learning are both promising ML techniques
for RIS-aided wireless networks. Transfer learning can reduce the model
training efforts, while hierarchical learning provides a novel architecture
for applying ML to wireless communications with hierarchical intelligence,
especially when optimization parameters have different timescales. However,
more research is needed on these techniques as they are used for RIS-aided
wireless networks. Both transfer learning and meta-learning involve source
tasks and prior experience. The core feature of meta-learning is learn-to-
learn, which is an appealing advantage for enabling rapid adaptation to
dynamic wireless environments. Compared with transfer learning, meta-learning
provides a scheme that can be used to facilitate transfer learning as well as
other techniques. In transfer learning, the prior knowledge is usually
extracted from the source task without defining a meta-objective. By contrast,
the prior experience in meta-learning is usually defined by an outer
optimization that evaluates the potential benefit of handling new tasks.
Meanwhile, meta-learning involves a wider range of meta-representation
problems than transfer learning.

Instead of applying one specific ML algorithm solely, note that these ML
algorithms may be jointly used. For instance, federated deep reinforcement
learning deploys DRL in each local server for decision-making, and then uses a
global server to aggregate the main networks for overall estimation and
coordination. Such integration can make the most of each algorithm’s
advantages, achieving better overall performance.

Finally, Fig. 23 and 24 present examples of using transfer learning and
hierarchical learning for RIS-related optimization, respectively. In
particular, Fig. 23 compares the convergence of transfer deep reinforcement
learning (TDRL) and DRL, and TDRL achieves faster convergence with higher
average reward. The main reason is that TDRL can reuse the former knowledge of
existing experts, which will considerably improve the exploration efficiency
of ML algorithms. Such improvement becomes more obvious when the number of RIS
elements increases, which indicates higher exploration difficulty for
conventional DRL algorithms. Meanwhile, Fig. 24 shows the energy efficiency of
hierarchical reinforcement learning-enabled joint sleep control and RIS phase-
shift optimization[103]. It includes a multi-BS and multi-RIS scenario, and
sleep control can decide the on/off status of BSs to reduce energy
consumption, while RISs can improve the channel capacity. Fig. 24 demonstrates
that combining sleep control with RISs can bring higher energy efficiency than
using each technique solely, and hierarchical reinforcement learning can well
coordinate different decisions with various time scales.

TABLE XVI: Summary of control and optimization techniques for RIS-aided
wireless networks

Optimization approaches |  Main features |  Advantage |  Drawbacks |  Difficulties |  Application scenarios for RISs
---|---|---|---|---|---
Model-based algorithms |  Model-based algorithms aim to find global optimal or at least sub-optimal results for target problems. They usually require full knowledge of the problem to find near-optimal solutions by using transformation, relaxation, and approximation. |  Model-based algorithms, i.e., SCA, MM, can provide detailed proofs and explanations for the optimality. Target problems are efficiently solved with guaranteed optimality once the closed-form solution is achieved. |  Model-based solutions are usually problem-specific with certain requirements such as convexity and continuity, indicating case-by-case analyses and design. It has difficulty adapting to dynamically changing environments. |  It has to apply transformations, division, and relaxation to convert the problem to specific forms. These transformations need a dedicated design for each problem. |  Numerous algorithms have been developed to solve RIS-aided optimization problems, e.g., AO to decouple the active and passive beamforming, SDR to relax the rank constraints, and SCA to estimate the sub-optimal results.
Heuristic algorithms |  It applies heuristic rules to find a trade-off between optimality and computational complexity. Heuristic algorithms focus on local optima and low-complexity solutions. |  Heuristic algorithms have much lower computational complexity. It has few requirements for the properties of target problems. |  It only presents local optima in the current stage, indicating a bad performance in some cases. |  Heuristic rules should be carefully selected and designed, directly affecting the algorithm performance. |  Considering the high complexity of RIS control problems, heuristic algorithms can provide low-complexity alternatives, i.e., sequential phase shift and on/off control using greedy rule, phase-shift optimization using GA.
ML techniques |  ML techniques are usually data-driven, providing unified control and optimization algorithms for certain types of problems. Most algorithms are easily applied without requiring dedicated design. |  Data-driven approaches avoid the complexity of building dedicated optimization models. It can better adapt to the dynamic wireless environment given the learning capability. |  It may require many iterations for the algorithm training. ML optimization techniques do not guarantee optimality. |  Algorithm training is the main difficulty of applying ML, which is data and computation-demanding. |  Various ML techniques have been applied for RIS-related optimizations, e.g., neural networks for CSI prediction and RIS phase control, and DDPG for continuous RIS phase-shift optimization.

Figure 25: Algorithm selection and relationship of model-based, heuristic and
ML approaches.

##  VII Comparison and Relationship between Model-based, Heuristic and ML
Approaches

This work has introduced three types of optimization techniques: model-based,
heuristic, and ML approaches. One intuitive question is how to evaluate the
advantages and difficulties of these techniques as well as their
relationships. To answer this question, we compare these approaches in Table
XVI, including main features, advantages, drawbacks, difficulties, and
applications for RISs. In addition, Fig. 25 summarizes algorithm selection of
applying various methods and their relationships. Note that Fig. 25 provides a
general overview for optimizing RIS-aided wireless networks, but the algorithm
selection and design should be combined with specific application scenarios.

1) Model-based method: Table XVI shows that model-based approaches can provide
efficient and stable solutions once the problem is properly reformulated,
especially when closed-form expressions are obtained. However, model-based
algorithms are usually complicated to design, indicating a series of
transformations and relaxations, e.g., decoupling the denominator and
numerator in SINR terms and relaxing integer constraints. As a result, the
approximation and relaxation can undermine the quality of solutions.
Additionally, environmental uncertainties can significantly affect the
performance of model-based algorithms, since they require full knowledge of
the optimization parameters. One possible solution is to assume environment
changes follow some specific distributions, but the optimization over
distributions will further increase the complexity. Another solution is to use
Monte Carlo sampling and repeat the optimization to achieve average results,
which is time-consuming. As illustrated in Fig. 25, given a joint optimization
problem with coupled control variables, one may use AO or BCD to decouple the
joint optimization into multiple sub-problems. Specifically, FP can be used to
eliminate fractional terms, e.g., SINR and energy efficiency, and SDR is
applied to relax non-convex constraints. Then, various techniques may be
applied, such as MM, SCA, SOCP and BnB, to solve each sub-problem under the
alternating framework.

2) Heuristic algorithms: The primary benefit of heuristic methods is the low
implementation complexity, i.e., optimizing RIS control in an element-by-
element manner, achieving a trade-off between optimality and computational
complexity. Heuristic methods also show a high generalization capability,
e.g., genetic algorithm and PSO apply unified fitness functions to represent
the optimization objectives. However, meta-heuristic algorithms are sensitive
to key parameters, e.g., population numbers and inertia weight in PSO, which
may require find-tuning efforts. But other heuristic methods, especially
greedy algorithms and matching theory, can be easily applied with little
tuning requirement. Meanwhile, Fig. 25 shows that heuristic algorithms can
also be used to solve sub-problems that are defined under an AO scheme,
indicating possible combinations between model-based and heuristic algorithms.
For instance, to maximize energy efficiency, Yang et al. define three sub-
problems, and SCA is deployed for active and passive beamforming, while a
greedy algorithm is used for RIS on/off control[54]. Such a combined scheme
demonstrates the capability of integrating model-based algorithms with
heuristic algorithms.

3) ML algorithms: Wireless networks are highly dynamic, and hence optimization
techniques must be robust to environmental uncertainties. With the learning
capability, ML algorithms can adapt well to dynamic environments. In
particular, ML algorithms present unified optimization schemes, which are
applied to diverse problems with few design requirements. For instance, most
optimization problems can be converted into unified MDPs that include state,
action, transition probability and rewards, and then reinforcement learning is
utilized to maximize the reward for a higher sum-rate or energy efficiency.
Meanwhile, as summarized in Fig. 25, reinforcement learning can be integrated
with other ML techniques to develop diverse optimization algorithms, such as
federated deep reinforcement learning, transfer reinforcement learning, and
hierarchical reinforcement learning. For example, transfer reinforcement
learning can achieve faster convergence and higher average reward than
conventional reinforcement learning algorithms. However, ML algorithm training
is usually computation-demanding, requiring a large number of computational
resources, e.g., iterative exploration of RL and back-propagation for neural
network training. Finally, datasets are crucial to applying data-driven ML
algorithms, especially for supervised learning. Model-based methods provide a
useful approach for labeled dataset generation, which indicates the potential
to combine model-based and ML algorithms. For example, Hu et al. first apply
the BCD method for RIS-aided mobile edge computing, and then the produced
results serve as datasets for location-based supervised learning algorithms
[125]. This reveals the potential benefit of integrating ML techniques with
model-based algorithms.

##  VIII RIS-assisted 6G Applications: Optimization Analyses and Challenges

This section analyzes control and optimization techniques for RIS-assisted 6G
applications, e.g., potential optimization difficulties and algorithm
selections. In addition, we identify several research challenges for the
optimization of RIS-aided wireless networks.

TABLE XVII: Control and optimization analyses for RIS-assisted 6G applications

6G applications |  Key features |  Motivations for integrating RISs |  Potential optimization difficulties |  Analyses of optimization algorithm selections
---|---|---|---|---
RIS-NOMA |  NOMA enables spectrum sharing among users, e.g., multiple users can use the same time and frequency resource blocks, improving user fairness and spectral efficiency. |  Due to the resource-sharing nature, the NOMA system is more vulnerable to security issues. Then RISs can be applied to reshape the signal propagation environment for security services against eavesdroppers. |  RIS-NOMA integration increases the overall optimization complexity. Specifically, the decoding order may be frequently changed due to the dynamic RIS configuration, increasing the difficulty of applying conventional model-based algorithms. |  ML techniques can be applied for intelligent decision-making in RIS-NOMA systems to handle the complexity and meanwhile improve long-term network performance. AO may be used to decouple the RIS control with NOMA optimization.
RIS-SWIPT |  SWIPT is an attractive solution to transmit electricity without using physical wire links, increasing the mobility, reliability, and safety of electronic devices. |  The low energy efficiency at the energy receiver is one of the main issues for practical SWIPT deployment, and RISs become a promising solution, e.g., increasing sum-rate, reducing transmit power, and maximizing the minimum received power. |  Existing studies mainly apply model-based methods for joint optimization of RISs and wireless power transfer, requiring dedicated model design. Although some low-complexity algorithms are proposed, i.e., MM [180] and bi-section search[181], they still require full knowledge of the defined problem. |  ML approaches can be promising alternatives to handle the complexity of joint optimizing RISs and SWIPT. For instance, the RL agent can explore the complicated IoT environment without any prior knowledge, and intelligently optimize energy and information transfer and RIS control.
RIS-mmWave and THz |  The increasing traffic demand and scarce bandwidth resources make mmWave and THz communications become appealing techniques. |  mmWave and THz communications are vulnerable to signal blockages and attenuation, leading to severe path loss and reduced cover range. RISs can be applied to manipulate the signal propagation environment when the direct transmission is blocked. |  The optimization of RIS-mmWave and RIS-THz systems rely on accurate and practical channel estimation to identify the performance limit. Therefore, robust optimization techniques should be developed to handle the uncertainty in channel estimation. |  Compared with heuristic or ML algorithms, model-based algorithms are more reliable in guaranteeing the worst-case performance for RIS-mmWave and THz systems, e.g., maximizing worst-case network performance, or considering outage probability constraints.
RIS-NTN |  The NTN complements the limitations of terrestrial networks, providing flexible and reliable support for remote areas by UAVs, high-altitude platforms (HAPs) and low earth orbit satellites. |  Frequent repositioning will increase the UAV power consumption, especially considering that UAVs are powered by a battery[79]. In this case, RISs can be applied to overcome this challenge, in which one can configure the RIS phase shifts instead of moving UAVs to save energy. |  UAV communications are highly dynamic due to their mobility, and ML algorithms can be used to handle such uncertainty. However, ML model training requires many iterations, which may prevent UAVs from making real-time responses to network dynamics. |  To overcome the tedious training iterations of conventional ML algorithms, transfer learning and meta-learning may be used to improve the training efficiency and make rapid responses to UAV dynamics, which is still an open issue.
RIS-V2X |  V2X is a key paradigm for envisioned 6G networks, enabling intelligent transportation with higher road safety and traffic efficiency. |  Latency and reliability are the most critical requirements of V2X networks for road safety. However, the V2X transmission can be unstable due to fast-moving vehicles and the dynamic nature of wireless communications. Therefore, RISs can be exploited to improve channel capacity, coverage, signal strength and reliability. |  A critical feature of V2X communications is the stringent requirement for reliability and safety, which means the proposed algorithm should guarantee the worst-case network performance. In addition, this indicates that the control and optimization algorithms should be efficient, robust and reliable. |  ML algorithms can quickly adapt to dynamic environments, but the model training is time-consuming. Meanwhile, model-based algorithms can produce stable results, but they require full knowledge of the environment. Therefore, developing efficient optimization algorithms for V2X networks is still challenging for current studies.
RIS-ISAC |  ISAC is recently emerging as a key technology to support ubiquitous wireless connectivity and accurate sensing[182]. |  Target detection and parameter estimation are two primary tasks in radar sensing[183], and RISs can be deployed to provide virtual LoS signal transmission, enabling the radar to sense targets in blocked areas[184]. |  ISAC offers significant potential by combining sensing and communication, but it also leads to extra complexity for network management, and such complexity further increases by involving RISs. |  It is crucial to develop efficient optimization techniques to realize the full potential of the RIS-ISAC system, e.g., decoupling joint optimization into multiple sub-problems using AO, and applying ML algorithms for joint control.

###  VIII-A Control and Optimization Analyses of RIS-assisted 6G applications

Table XVII summarizes RIS-assisted 6G applications, including NOMA, SWIPT,
mmWave and THz communications, NTNs, V2X communications, and ISAC. For
example, due to the resource-sharing nature, the NOMA system is more
vulnerable to security issues. Then RISs can be applied to reshape the signal
propagation environment for security services against eavesdroppers. The low
energy efficiency at the energy receiver is one of the main issues for
practical SWIPT deployment, and RISs become a promising solution to increase
sum-rate, reduce transmit power, and maximize the minimum received power. In
addition, Table XVII also summarizes the motivations for integrating RISs with
other 6G applications such as RIS-NTN, RIS-V2X, and RIS-ISAC.

However, integrating RISs with 6G techniques also increases the difficulties
for network management. In RIS-NOMA systems, the decoding order may be
frequently changed due to the dynamic RIS configuration, increasing the
difficulty of applying conventional model-based algorithms. For V2X networks,
a critical feature is the stringent requirement for reliability and safety,
which means the proposed algorithm should guarantee the worst-case network
performance. Such a requirement means that the control and optimization
algorithms should be efficient, robust and reliable. The potential
optimization difficulties of other RIS-assisted 6G applications are also
reviewed in Table XVII.

Finally, we analyze optimization algorithm selections for various RIS-assisted
6G applications. Integrating RISs will substantially increase the network
management complexity, since RIS phase-shift control is highly coupled with
other control variables such as decoding order in NOMA, beam selection in
mmWave networks, and UAV altitude control. ML algorithms become promising
solutions to handle such complexity, such as DDQN[142] and DDPG[132, 145]. In
particular, these studies apply unified schemes to optimize network
performance, overcoming the difficulties of reformulation and transformation
for convexity. However, conventional ML algorithms require many iterations for
model training, which may prevent the application to highly dynamic
environments such as RIS-UAV. To this end, transfer learning and meta-learning
may be used to improve training efficiency and make rapid responses. On the
other hand, other applications such as V2X have more stringent service
requirements to guarantee worst-case performance. In this case, model-based
methods can usually provide more stable performance than ML or heuristic
approaches, providing detailed proofs and explanations for the algorithm
output.

###  VIII-B Challenges and Future Directions

This subsection identifies research challenges and possible future directions.

1) Practical RIS Phase-shift Design: Most existing RIS optimization studies
rely on perfect CSI acquisition and static user conditions, which are
impractical assumptions in the real world. Specifically, the wireless
environment is highly dynamic due to various channel conditions and diverse
user demands. Therefore, developing robust and practical algorithms for RIS
control is of great importance for the real-world deployment of RISs, e.g.,
imperfect CSI acquisition and UEs with high mobility, which requires more
research efforts.

2) Low-overhead Control: Many advanced control and optimization techniques
have been proposed for RISs, but the communication and control overhead is
neglected in most works. For example, frequent parameter exchange between the
BS and RISs may lead to high overhead, and the model training overhead of ML
algorithms can hamper the system efficiency. These issues are still open
challenges, and agile optimization algorithms with low complexity and overhead
are yet to be developed.

3) ML-enabled Intelligent RIS Beamforming: ML is one of the most promising
techniques to facilitate future 6G networks, and integrating ML with RISs can
bring intelligent prediction, clustering, and decision-making for RIS-aided
wireless networks. Despite the significant potential, some critical questions,
e.g., algorithm deployment, offline or online training, and training cost, are
neglected in many existing studies. Addressing these problems can further
enable an intelligent future wireless network.

4) Practical RIS Location Optimization: In many existing studies, RIS location
is considered as a predefined parameter for simulation. However, RIS location
can considerably affect the system performance and therefore should be very
carefully handled. Moreover, the real-world environment is more complicated
when considering dense buildings and other obstacles. RIS location and scale
should be jointly optimized by considering the wireless environment, user
distribution, and service requirements, which still require research effort.

5) Flexible Control Framework: The former analyses have shown that each
optimization approach has its advantages and difficulties. Model-based methods
have higher stability and optimality, and heuristic methods have lower
complexity, while ML techniques are more robust. One intuitive direction is to
combine these methods to form a flexible optimization framework that can make
the most of each approach’s advantages and complement the difficulties.
However, many existing studies stick with one type of optimization technique,
and flexible control schemes are considered future challenges.

##  IX Conclusion

RIS technology is a key enabler for 6G networks, and control and optimization
techniques are critical to exploiting the full potential of RISs. In this
work, we have surveyed various approaches for optimizing RIS-aided wireless
networks, including model-based, heuristic, and ML approaches. We have
provided in-depth analyses of the algorithms’ features, difficulties, and
applications towards RIS, and we have further compared the advantages and
disadvantages of nearly 20 techniques. Our analyses reveal that model-based
methods exhibit satisfying performance and stability, but the algorithm design
is complicated with low generalization capability. Heuristic algorithms can
obtain low-complexity sub-optimal solutions, which are usually considered as
baselines or supplements for other techniques. ML techniques have high
generalization capability and optimality, but ML model training is
computationally demanding and requires experience. Finally, the algorithm
selection depends on specific optimization requirements, which should be
jointly considered based on application scenarios. It is hoped that this
survey will serve as a roadmap for researchers to investigate advanced
optimization techniques for RIS-aided wireless networks.

## References

  * [1] Z. Zhang, Y. Xiao, Z. Ma, M. Xiao, Z. Ding, X. Lei, G. K. Karagiannidis, and P. Fan, “6G wireless networks: Vision, requirements, architecture, and key technologies,” _IEEE Veh. Technol. Mag._ , vol. 14, no. 3, pp. 28–41, Sep. 2019.
  * [2] E. Basar, M. D. Renzo, J. D. Rosny, M. Debbah, M.-S. Alouini, and R. Zhang, “Wireless communications through reconfigurable intelligent surfaces,” _IEEE Access_ , vol. 7, pp. 116 753–116 773, Aug. 2019.
  * [3] E. Björnson, H. Wymeersch, B. Matthiesen, P. Popovski, L. Sanguinetti, and E. de Carvalho, “Reconfigurable intelligent surfaces: A signal processing perspective with wireless applications,” _IEEE Signal Process. Mag._ , vol. 39, no. 2, pp. 135–158, Feb. 2022.
  * [4] X. Mu, Y. Liu, L. Guo, J. Lin, and R. Schober, “Simultaneously transmitting and reflecting (STAR) RIS aided wireless communications,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 5, pp. 3083–3098, May 2021.
  * [5] X. Mu, Y. Liu, L. Guo, J. Lin, and H. V. Poor, “Intelligent reflecting surface enhanced multi-UAV NOMA networks,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 10, pp. 3051–3066, Oct. 2021.
  * [6] M. D. Renzo, A. Zappone, M. Debbah, M.-S. Alouini, C. Yuen, J. de Rosny, and S. Tretyakov, “Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and the road ahead,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 11, pp. 2450–2525, Jul. 2020.
  * [7] A. Almohamad, A. M. Tahir, A. Al-Kababji, H. M. Furqan, T. Khattab, M. O. Hasna, and H. Arslan, “Smart and secure wireless communications via reflecting intelligent surfaces: A short survey,” _IEEE Open J. Commun. Soc._ , vol. 1, pp. 1442–1456, Sep. 2020.
  * [8] S. Gong, X. Lu, D. T. Hoang, D. Niyato, L. Shu, D. I. Kim, and Y.-C. Liang, “Toward smart wireless communications via intelligent reflecting surfaces: A contemporary survey,” _IEEE Commun. Surveys Tuts._ , vol. 22, no. 4, pp. 2283–2314, 4th quarter 2020.
  * [9] M. A. ElMossallamy, H. Zhang, L. Song, K. G. Seddik, Z. Han, and G. Ye Li, “Reconfigurable intelligent surfaces for wireless communications: Principles, challenges, and opportunities,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 3, no. 6, pp. 990–1002, Sep. 2020.
  * [10] M. Z. Siddiqi and T. Mir, “Reconfigurable intelligent surface-aided wireless communications: An overview,” _Intell. and Converged Netw._ , vol. 3, no. 1, pp. 33–63, Mar. 2022.
  * [11] C. Pan, G. Zhou, K. Zhi, S. Hong, T. Wu, Y. Pan, H. Ren, M. D. Renzo, A. L. Swindlehurst, R. Zhang, and A. Y. Zhang, “An overview of signal processing techniques for RIS/RIS-aided wireless systems,” _IEEE J. Sel. Top. Signal Process._ , vol. 5, no. 16, pp. 883–917, Aug. 2022.
  * [12] R. Alghamdi, R. Alhadrami, D. Alhothali, H. Almorad, A. Faisal, S. Helal, R. Shalabi, R. Asfour, N. Hammad, A. Shams, N. Saeed, H. Dahrouj, T. Y. Al-Naffouri, and M.-S. Alouini, “Intelligent surfaces for 6G wireless networks: A survey of optimization and performance analysis techniques,” _IEEE Access_ , vol. 8, pp. 202 795–202 818, Oct. 2020.
  * [13] K. Faisal and W. Choi, “Machine learning approaches for reconfigurable intelligent surfaces: A survey,” _IEEE Access_ , vol. 10, pp. 27 343–27 367, Jul. 2022.
  * [14] Y. Liu, X. Liu, X. Mu, T. Hou, J. Xu, M. D. Renzo, and N. Al-Dhahir, “Reconfigurable intelligent surfaces: Principles and opportunities,” _IEEE Commun. Surveys Tuts._ , vol. 23, no. 3, pp. 1546–1577, 3rd quarter 2021.
  * [15] B. Zheng, C. You, W. Mei, and R. Zhang, “A survey on channel estimation and practical passive beamforming design for intelligent reflecting surface aided wireless communications,” _IEEE Commun. Surveys Tuts._ , vol. 24, no. 2, pp. 1035–1071, 2nd quarter 2022.
  * [16] Y. C. Eldar, A. Goldsmith, D. Gündüz, and H. V. Poor, _Machine Learning and Wireless Communications_. Cambridge, UK: Cambridge University Press, 2022.
  * [17] C. Huang, G. C. Alexandropoulos, C. Yuen, and M. Debbah, “Indoor signal focusing with deep learning designed reconfigurable intelligent surfaces,” in _Proc. Int. Workshop Signal Process. Adv. Wireless commun. (SPAWC)_ , Aug. 2019, pp. 1–5.
  * [18] H. Lu, Y. Zeng, S. Jin, and R. Zhang, “Aerial intelligent reflecting surface: Joint placement and passive beamforming design with 3D beam flattening,” _IEEE Trans. on Wireless Commun._ , vol. 20, no. 7, pp. 4128–4143, Jul 2021.
  * [19] K. Tekbıyık, G. K. Kurt, A. R. Ekti, and H. Yanikomeroglu, “Reconfigurable intelligent surfaces in action for nonterrestrial networks,” _IEEE Veh. Technol. Mag._ , vol. 17, no. 3, pp. 45–53, Sep. 2022.
  * [20] R. Su, L. Dai, J. Tan, M. Hao, and R. MacKenzie, “Capacity enhancement for irregular reconfigurable intelligent surface-aided wireless communications,” in _Proc. IEEE Global Commun. Conf._ , Dec. 2020, pp. 1–6.
  * [21] C. Pan, H. Ren, K. Wang, W. Xu, M. Elkashlan, A. Nallanathan, and L. Hanzo, “Multicell MIMO communications relying on intelligent reflecting surfaces,” _IEEE Trans. Wireless Commun._ , vol. 19, pp. 5218–5233, Oct. 2020.
  * [22] S. Zhang and R. Zhang, “Capacity characterization for intelligent reflecting surface aided MIMO communication,” _IEEE J. Sel. Areas Commun._ , vol. 38, pp. 1823–1838, Jun. 2020.
  * [23] Y. Zhang, C. Zhong, Z. Zhang, and W. Lu, “Sum rate optimization for two way communications with intelligent reflecting surface,” _IEEE Commun. Lett._ , vol. 24, pp. 1090–1094, Mar. 2020.
  * [24] M.-M. Zhao, Q. Wu, M.-J. Zhao, and R. Zhang, “Intelligent reflecting surface enhanced wireless networks: Two-timescale beamforming optimization,” _IEEE Trans. Wireless Commun._ , vol. 20, pp. 2–17, Jan. 2021.
  * [25] J. Yuan, Y.-C. Liang, J. Joung, G. Feng, and E. G. Larsson, “Intelligent reflecting surface-assisted cognitive radio system,” _IEEE Trans. Commun._ , vol. 69, pp. 675 – 687, Jan. 2021.
  * [26] B. Di, H. Zhang, L. Li, L. Song, Y. Li, and Z. Han, “Practical hybrid beamforming with finite-resolution phase shifters for reconfigurable intelligent surface based multi-user communications,” _IEEE Trans. Vehi. Technol._ , vol. 69, pp. 4565 – 4570, Apr. 2020.
  * [27] H. Guo, Y.-C. Liang, J. Chen, and E. G. Larsson, “Weighted sum-rate maximization for reconfigurable intelligent surface aided wireless networks,” _IEEE Trans. Wireless Commun._ , vol. 19, pp. 3064–3076, May 2020.
  * [28] C. You, B. Zheng, and R. Zhang, “Intelligent reflecting surface with discrete phase shifts: Channel estimation and passive beamforming,” in _Proc. IEEE Int. Conf. Commun. (ICC)_ , Jun. 2020, pp. 1–6.
  * [29] X. Mu, Y. Liu, L. Guo, J. Lin, and N. Al-Dhahir, “Exploiting intelligent reflecting surfaces in NOMA networks: Joint beamforming optimization,” _IEEE Trans. Wireless Commun._ , vol. 19, pp. 6884–6898, Oct. 2020.
  * [30] Y. Chen, Y. Wang, and L. Jiao, “Robust transmission for reconfigurable intelligent surface aided millimeter wave vehicular communications with statistical CSI,” _IEEE Trans. Wireless Commun._ , vol. 21, pp. 928–944, Aug. 2021.
  * [31] J. Dai, Y. Wang, C. Pan, K. Zhi, H. Ren, and K. Wang, “Reconfigurable intelligent surface aided massive MIMO systems with low-resolution DACs,” _IEEE Commun. Lett._ , vol. 25, no. 9, pp. 3124–3128, Sep. 2021.
  * [32] K. Zhi, C. Pan, H. Ren, and K. Wang, “Statistical CSI-based design for reconfigurable intelligent surface-aided massive MIMO systems with direct links,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 5, pp. 1128–1132, May 2021.
  * [33] M. Rivera, M. Chegini, W. Jaafar, S. Alfattani, and H. Yanikomeroglu, “Optimization of quantized phase shifts for reconfigurable smart surfaces assisted communications,” in _Proc. Consum. Commun. Netw. Conf. (CCNC)_ , Feb. 2022, pp. 901–904.
  * [34] W. Ni, X. Liu, Y. Liu, H. Tian, and Y. Chen, “Resource allocation for multi-cell RIS-aided NOMA networks,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 7, pp. 4253–4268, Jul. 2021.
  * [35] Y. Chen, Y. Wang, J. Zhang, and M. Di Renzo, “QoS-driven spectrum sharing for reconfigurable intelligent surfaces (RISs) aided vehicular networks,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 9, pp. 5969–5985, Sep. 2021.
  * [36] W. Yu and T. Lan, “Transmitter optimization for the multi-antenna downlink with per-antenna power constraints,” _IEEE Transactions on signal processing_ , vol. 55, no. 6, pp. 2646–2660, 2007.
  * [37] G. Yang, X. Xu, and Y.-C. Liang, “Intelligent reflecting surface assisted non-orthogonal multiple access,” in _Proc. IEEE Wireless Commun. Netw. Conf (WCNC)_ , May. 2020, pp. 1–6.
  * [38] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Trans. Wireless Commun._ , vol. 18, pp. 5394–5409, Aug. 2019.
  * [39] Q. Wu and R. Zhang, “Beamforming optimization for wireless network aided by intelligent reflecting surface with discrete phase shifts,” _IEEE Trans. Commun._ , vol. 68, pp. 1838–1851, Mar. 2020.
  * [40] G. Zhou, C. Pan, H. Ren, K. Wang, M. D. Renzo, and A. Nallanathan, “Robust beamforming design for intelligent reflecting surface aided MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, pp. 1658–1662, Oct. 2020.
  * [41] H. Han, J. Zhao, W. Zhai, Z. Xiong, D. Niyato, M. D. Renzo, Q.-V. Pham, W. Lu, and K.-Y. Lam, “Reconfigurable intelligent surface aided power control for physical-layer broadcasting,” _IEEE Trans. Commun._ , vol. 69, pp. 7821–7836, Nov. 2021.
  * [42] G. Zhou, C. Pan, H. Ren, K. Wang, and A. Nallanathan, “A framework of robust transmission design for RIS-aided MISO communications with imperfect cascaded channels,” _IEEE Trans. Signal Process._ , vol. 68, pp. 5092–5106, Aug. 2020.
  * [43] B. Zheng, Q. Wu, and R. Zhang, “Intelligent reflecting surface-assisted multiple access with user pairing: NOMA or OMA?” _IEEE Wireless Commun. Lett._ , vol. 24, pp. 753–757, Jan. 2020.
  * [44] Y. Li, M. Jiang, Q. Zhang, and J. Qin, “Joint beamforming design in multi-cluster MISO NOMA reconfigurable intelligent surface-aided downlink communication networks,” _IEEE Trans. Commun._ , vol. 69, pp. 664–674, Oct. 2020.
  * [45] M. Fu, Y. Zhou, Y. Shi, and K. B. Letaief, “Reconfigurable intelligent surface empowered downlink non-orthogonal multiple access,” _IEEE Trans. Commun._ , vol. 69, pp. 3802–3817, Jun. 2021.
  * [46] J. Zhu, Y. Huang, J. Wang, K. Navaie, and Z. Ding, “Power efficient IRS-assisted NOMA,” _IEEE Trans. Commun._ , vol. 69, pp. 900–913, Oct. 2020.
  * [47] Z. Yang, C. Huang, J. Shi, Y. Chau, W. Xu, Z. Zhang, and M. Shikh-Bahaei, “Optimal control for full-duplex communications with reconfigurable intelligent surface,” in _Proc. IEEE Int. Conf. Commun.(ICC)_ , Jun. 2021, pp. 1–6.
  * [48] Z. Li, M. Chen, Z. Yang, J. Zhao, Y. Wang, J. Shi, and C. Huang, “Energy efficient reconfigurable intelligent surface enabled mobile edge computing networks with NOMA,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 7, pp. 427–440, Mar. 2021.
  * [49] X. Cao, X. Hu, and M. Peng, “Joint mode selection and beamforming for RIS-aided maritime cooperative communication systems,” _IEEE Trans. Green Commun. Netw. (Early Access)_ , Mar. 2022.
  * [50] C. Huang, A. Zappone, G. C. Alexandropoulos, M. Debbah, and C. Yuen, “Reconfigurable intelligent surfaces for energy efficiency in wireless communication,” _IEEE Trans. Wireless Commun._ , vol. 18, pp. 4157–4170, Aug. 2019.
  * [51] L. Du, W. Zhang, J. Ma, and Y. Tang, “Reconfigurable intelligent surfaces for energy efficiency in multicast transmissions,” _IEEE Trans. Vehi. Technol._ , vol. 70, pp. 6266–6271, Jun. 2021.
  * [52] S. Jia, X. Yuan, and Y.-C. Liang, “Reconfigurable intelligent surfaces for energy efficiency in D2D communication network,” _IEEE Wireless Commun. Lett._ , vol. 10, pp. 683–687, Mar. 2021.
  * [53] Z. Yang, J. Shi, Z. Li, M. Chen, W. Xu, and M. Shikh-Bahaei, “Energy efficient rate splitting multiple access (RSMA) with reconfigurable intelligent surface,” in _Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops)_ , Jun 2020, pp. 1–6.
  * [54] Z. Yang, M. Chen, W. Saad, W. Xu, M. Shikh-Bahaei, H. V. Poor, and S. Cui, “Energy-efficient wireless communications with distributed reconfigurable intelligent surfaces,” _IEEE Trans. Wireless Commun._ , vol. 21, pp. 665–679, Jul. 2021.
  * [55] Y. Zhang, B. Di, H. Zhang, J. Lin, C. Xu, D. Zhang, Y. Li, and L. Song, “Beyond cell-free MIMO: Energy efficient reconfigurable intelligent surface aided cell-free MIMO communications,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 7, pp. 412–426, Feb. 2021.
  * [56] M. Zhang, M. Chen, Z. Yang, H. Asgari, and M. Shikh-Bahaei, “Joint user clustering and passive beamforming for downlink NOMA system with reconfigurable intelligent surface,” in _Proc. IEEE Int. Symp. Personal, Indoor and Mobile Radio Commun.(PIMRC)_ , Oct. 2020, pp. 1–6.
  * [57] H. Zhou, L. Kong, M. Elsayed, M. Bavand, R. Gaigalas, S. Furr, and M. Erol-Kantarci, “Hierarchical reinforcement learning for RIS-assisted energy-efficient RAN,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM)_ , Dec. 2022, pp. 3326–3331.
  * [58] G. Zhou, C. Pan, H. Ren, K. Wang, and A. Nallanathan, “Intelligent reflecting surface aided multigroup multicast MISO communication systems,” _IEEE Trans. Signal Process._ , vol. 68, pp. 3236–3251, Apr. 2020.
  * [59] Q.-U.-A. Nadeem, A. Kammoun, A. Chaaban, M. Debbah, and M.-S. Alouini, “Asymptotic max-min SINR analysis of reconfigurable intelligent surface assisted MISO systems,” _IEEE Trans. Wireless Commun._ , vol. 19, pp. 7748–7764, Dec. 2020.
  * [60] H. Xie, J. Xu, and Y.-F. Liu, “Max-min fairness in RIS-aided multi-cell MISO systems with joint transmit and reflective beamforming,” _IEEE Trans. Wireless Commun._ , vol. 20, pp. 1379–1393, Feb. 2021.
  * [61] M. Bashar, K. Cumanan, A. G. Burr, P. Xiao, and M. D. Renzo, “On the performance of reconfigurable intelligent surface-aided cell-free massive MIMO uplink,” in _Proc. IEEE Int. Conf. Commun. (ICC)_ , Dec. 2020, pp. 1–6.
  * [62] S. Gong, Z. Yang, C. Xing, J. An, and L. Hanzo, “Beamforming optimization for intelligent reflecting surface-aided SWIPT IoT networks relying on discrete phase shifts,” _IEEE Internet Things J._ , vol. 8, pp. 8585–8602, Dec. 2020.
  * [63] Z. Abdullah, G. Chen, S. Lambotharan, and J. A. Chambers, “Optimization of intelligent reflecting surface assisted full-duplex relay networks,” _IEEE Wireless Commun. Lett._ , vol. 10, pp. 363–367, Oct. 2020.
  * [64] M. Hua, Q. Wu, D. W. K. Ng, J. Zhao, and L. Yang, “Intelligent reflecting surface-aided joint processing coordinated multipoint transmission,” _IEEE Trans. Commun._ , vol. 69, pp. 1650–1665, Dec. 2020.
  * [65] K. Zhi, C. Pan, H. Ren, and K. Wang, “Power scaling law analysis and phase shift optimization of RIS-aided massive MIMO systems with statistical CSI,” _IEEE Trans. Commun._ , vol. 70, no. 5, pp. 3558–3574, May 2022.
  * [66] S. Atapattu, R. Fan, P. Dharmawansa, G. Wang, J. Evans, and T. A. Tsiftsis, “Reconfigurable intelligent surface assisted two-way communications: Performance analysis and optimization,” _IEEE Trans. Commun._ , vol. 68, no. 10, pp. 6552–6567, Oct. 2020.
  * [67] Z. Chu, W. Hao, P. Xiao, and J. Shi, “Intelligent reflecting surface aided multi-antenna secure transmission,” _IEEE Wireless Commun. Lett._ , vol. 9, pp. 108–112, Jan. 2020.
  * [68] H. Shen, W. Xu, S. Gong, Z. He, and C. Zhao, “Secrecy rate maximization for intelligent reflecting surface assisted multi-antenna communications,” _IEEE Wireless Commun. Lett._ , vol. 23, pp. 1488–1492, Jun. 2019.
  * [69] X. Yu, D. Xu, Y. Sun, D. W. K. Ng, and R. Schober, “Robust and secure wireless communications via intelligent reflecting surfaces,” _IEEE J. Sel. Areas Commun._ , vol. 38, pp. 2637–2652, Jul. 2020.
  * [70] B. Feng, Y. Wu, and M. Zheng, “Secure transmission strategy for intelligent reflecting surface enhanced wireless system,” in _Proc. Int. Conf. Wireless Commun. Signal Process. (WCSP)_ , Oct. 2019, pp. 1–6.
  * [71] M. Cui, G. Zhang, and R. Zhang, “Secure wireless communication via intelligent reflecting surface,” _IEEE Wireless Commun. Lett._ , vol. 8, pp. 1410–1414, May 2019.
  * [72] Z. Ji, P. L. Yeoh, D. Zhang, G. Chen, Y. Zhang, Z. He, H. Yin, and Y. Li, “Secret key generation for intelligent reflecting surface assisted wireless communication networks,” _IEEE Trans. Vehi. Technol._ , vol. 70, pp. 1030–1034, Dec. 2020.
  * [73] H.-M. Wang, J. Bai, and L. Dong, “Intelligent reflecting surfaces assisted secure transmission without eavesdropper’s CSI,” _IEEE Signal Process. Lett._ , vol. 27, pp. 1300–1304, Jul. 2020.
  * [74] W. Sun, Q. Song, L. Guo, and J. Zhao, “Secrecy rate maximization for intelligent reflecting surface aided SWIPT systems,” in _Proc. IEEE Int. Conf. Commun. in China (ICCC)_ , Aug. 2020, pp. 1–6.
  * [75] L. Dong and H.-M. Wang, “Secure MIMO transmission via intelligent reflecting surface,” _IEEE Wireless Commun. Lett._ , vol. 9, pp. 787–790, Jan. 2020.
  * [76] L. Dong and H.-M. Wang, “Enhancing secure MIMO transmission via intelligent reflecting surface,” _IEEE Trans. Wireless Commun._ , vol. 19, pp. 7543–7556, Nov. 2020.
  * [77] J. Chen, Y.-C. Liang, Y. Pei, and H. Guo, “Intelligent reflecting surface: A programmable wireless environment for physical layer security,” _IEEE Access_ , vol. 7, pp. 82 599–82 612, Jun. 2019.
  * [78] H. Niu, Z. Chu, F. Zhou, and Z. Zhu, “Simultaneous transmission and reflection reconfigurable intelligent surface assisted secrecy MISO networks,” _IEEE Commun. Lett._ , vol. 25, no. 11, pp. 3498–3502, Nov. 2021.
  * [79] X. Liu, Y. Liu, and Y. Chen, “Machine learning empowered trajectory and passive beamforming design in UAV-RIS wireless networks,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 7, pp. 2042–2055, Jul. 2021.
  * [80] C. Wu, X. Mu, Y. Liu, X. Gu, and X. Wang, “Resource allocation in STAR-RIS-aided networks: OMA and NOMA,” _IEEE Transactions on Wireless Commun._ , vol. 21, no. 9, pp. 7653–7667, Sep 2022.
  * [81] J. Zuo, Y. Liu, Z. Qin, and N. Al-Dhahir, “Resource allocation in intelligent reflecting surface assisted NOMA systems,” _IEEE Trans. Commun._ , vol. 68, no. 11, pp. 7170–7183, Nov. 2020.
  * [82] Y. Cao, T. Lv, W. Ni, and Z. Lin, “Sum-rate maximization for multi-reconfigurable intelligent surface-assisted device-to-device communications,” _IEEE Trans. Commun._ , vol. 69, no. 11, pp. 7283–7296, Nov. 2021.
  * [83] G. Yang, Y. Liao, Y.-C. Liang, O. Tirkkonen, G. Wang, and X. Zhu, “Reconfigurable intelligent surface empowered device-to-device communication underlaying cellular networks,” _IEEE Trans. Commun._ , vol. 69, no. 11, pp. 7790–7805, Nov. 2021.
  * [84] Z. Peng, R. Weng, Z. Zhang, C. Pan, and J. Wang, “Active reconfigurable intelligent surface for mobile edge computing,” _IEEE Wireless Commun. Lett._ , vol. 11, no. 12, pp. 2482–2486, Dec. 2022.
  * [85] A. Li, Y. Liu, M. Li, Q. Wu, and J. Zhao, “Joint scheduling design in wireless powered MEC IoT networks aided by reconfigurable intelligent surface,” in _Proc. Int. Conf. Commun. in China (ICCC Workshops)_ , Sep. 2021, pp. 159–164.
  * [86] X. Lu, W. Yang, X. Guan, Q. Wu, and Y. Cai, “Robust and secure beamforming for intelligent reflecting surface aided mmWave MISO systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 12, pp. 2068–2072, Dec. 2020.
  * [87] W. Hao, G. Sun, M. Zeng, Z. Chu, Z. Zhu, O. A. Dobre, and P. Xiao, “Robust design for intelligent reflecting surface-assisted MIMO-OFDMA terahertz IoT networks,” _IEEE Internet Things J._ , vol. 8, no. 16, pp. 13 052–13 064, Aug. 2021.
  * [88] M.-M. Zhao, A. Liu, and R. Zhang, “Outage-constrained robust beamforming for intelligent reflecting surface aided wireless communication,” _IEEE Trans. Signal Process._ , vol. 69, pp. 1301–1316, Feb. 2021.
  * [89] X. Guo, Y. Chen, and Y. Wang, “Learning-based robust and secure transmission for reconfigurable intelligent surface aided millimeter wave UAV communications,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 8, pp. 1795–1799, Aug. 2021.
  * [90] J. C. Bezdek and R. J. Hathaway, “Some notes on alternating optimization,” in _Proc. IEEE Int. Conf. Fuzzy Syst._ , Jan. 2002, pp. 288–300.
  * [91] M. S. Julie Nutini, Issam Laradji, “Let’s make block coordinate descent converge faster: Faster greedy rules, message-passing, active-set complexity, and superlinear convergence,” _arXiv:1712.08859_ , p. 1–50, Jun. 2022.
  * [92] Y. Sun, P. Babu, and D. P. Palomar, “Majorization-Minimization algorithms in signal processing, communications, and machine learning,” _IEEE Trans. Signal Process._ , vol. 65, pp. 794 – 816, Feb. 2017.
  * [93] G. Scutari, F. Facchinei, P. Song, D. P. Palomar, , and J.-S. Pang, “Decomposition by partial linearization: Parallel optimization of multi-agent systems,” _IEEE Trans. Signal Process._ , vol. 62, pp. 641– 656, Nov. 2014.
  * [94] S. Zhang, “Quadratic maximization and semidefinite relaxation,” _Math. Program._ , vol. 87, pp. 453–465, May 2000.
  * [95] Z.-Q. Luo, W.-K. Ma, A. M.-C. So, Y. Ye, and S. Zhang, “Semidefinite relaxation of quadratic optimization problems,” _IEEE Signal Processing Magazine_ , vol. 27, no. 3, pp. 20–34, May 2010.
  * [96] P. Wang, J. Fang, X. Yuan, Z. Chen, and H. Li, “Intelligent reflecting surface-assisted millimeter wave communications: Joint active and passive precoding design,” _IEEE Trans. Vehi. Technol._ , vol. 69, pp. 14 960–14 973, Oct. 2020.
  * [97] M. S. Lobo, L. Vandenberghe, S. Boyd, and H. Lebret, “Applications of second-order cone programming,” _Linear Algebra Appl._ , vol. 284, pp. 193–228, Nov. 1998.
  * [98] Y. Nesterov and A. Nemirovskii, _Interior-point polynomial algorithms in convex programming_. Philadelphia, Pennsylvania, US: SIAM, 1994.
  * [99] A. Zappone, E. Jorswieck _et al._ , “Energy efficiency in wireless networks via fractional programming theory,” _Found. Trends Commun. Inf. Theory_ , vol. 11, no. 3-4, pp. 185–396, Mar. 2015.
  * [100] W. Dinkelbach, “On nonlinear fractional programming,” _Manage Sci._ , vol. 13, no. 7, pp. 492–498, Mar. 1967.
  * [101] K. Shen and W. Yu, “Fractional programming for communication systems—Part I: Power control and beamforming,” _IEEE Trans. Signal Process._ , vol. 66, no. 10, pp. 2616–2630, May 2018.
  * [102] J. Clausen, “Branch and bound algorithms-principles and examples,” _Department of Computer Science, University of Copenhagen_ , pp. 1–30, Mar 1999.
  * [103] H. Zhou, M. Elsayed, M. Bavand, R. Gaigalas, S. Furr, and M. Erol-Kantarci, “Cooperative hierarchical deep reinforcement learning based joint sleep, power, and ris control for energy-efficient hetnet,” Apr 2023. [Online]. Available: https://arxiv.org/abs/2304.13226
  * [104] T. Lipp and S. Boyd, “Variations and extension of the convex–concave procedure,” _Optim. Eng._ , vol. 17, no. 2, pp. 263–287, Nov. 2016.
  * [105] A. L. Yuille and A. Rangarajan, “The concave-convex procedure (CCCP),” in _Adv. Neural Inf. Process. Syst._ , Dec. 2001, pp. 1–8.
  * [106] Y. Xu, T. Zhang, Y. Zou, and Y. Liu, “Reconfigurable intelligence surface aided UAV-MEC systems with NOMA,” _IEEE Commun. Lett._ , vol. 26, no. 9, pp. 2121–2125, Sep. 2022.
  * [107] O. Bozorg-Haddad, M. Solgi, and H. A. Loáiciga, _Meta-heuristic and evolutionary algorithms for engineering optimization_. Hoboken, NJ, US: John Wiley & Sons, 2017.
  * [108] Z. Beheshti and S. M. H. Shamsuddin, “A review of population-based meta-heuristic algorithms,” _Int. J. Adv. Soft Comput. Appl_ , vol. 5, no. 1, pp. 1–35, Mar. 2013.
  * [109] S. L. Martins and C. C. Ribeiro, “Metaheuristics and applications to optimization problems in telecommunications,” in _Handbook of optimization in telecommunications_. Springer, 2006, pp. 103–128.
  * [110] S. Tewes, M. Heinrichs, P. Staat, R. Kronberger, and A. Sezgin, “Full-duplex meets reconfigurable surfaces: RIS-assisted SIC for full-duplex radios,” in _Proc. IEEE Int. Conf. Commun. (ICC)_ , Aug. 2022, pp. 1106–1111.
  * [111] A. Li, L. Song, B. Vucetic, and Y. Li, “Interference exploitation precoding for reconfigurable intelligent surface aided multi-user communications with direct links,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 11, pp. 1937–1941, Nov. 2020.
  * [112] A. Muhammad, M. Elhattab, M. Shokry, and C. Assi, “Leveraging reconfigurable intelligent surface to minimize age of information in wireless networks,” in _Proc. IEEE Int. Conf. Commun. (ICC)_ , Aug. 2022, pp. 2525–2530.
  * [113] Z. Zhang and L. Dai, “A joint precoding framework for wideband reconfigurable intelligent surface-aided cell-free network,” _IEEE Trans. Signal Process._ , vol. 69, pp. 4085–4101, Jun. 2021.
  * [114] E. El Haber, M. Elhattab, C. Assi, S. Sharafeddine, and K. K. Nguyen, “Latency and reliability aware edge computation offloading in RIS-aided networks,” in _Proc. IEEE Int. Conf. Commun. (ICC)_ , Aug. 2022, pp. 5035–5040.
  * [115] E. M. Taghavi, A. Alizadeh, N. Rajatheva, M. Vu, and M. Latva-aho, “User association in millimeter wave cellular networks with intelligent reflecting surfaces,” in _Proc. IEEE 93rd Vehi. Technol. Conf. (VTC2021-Spring)_ , Jun. 2021, pp. 1–6.
  * [116] H. Zhou, M. Erol-Kantarci, Y. Liu, and H. V. Poor, “Heuristic algorithms for RIS-assisted wireless networks: Exploring heuristic-aided machine learning,” Jun 2023. [Online]. Available: https://arxiv.org/abs/2307.01205
  * [117] S. Zhang, S. Zhang, F. Gao, J. Ma, and O. A. Dobre, “Deep learning optimized sparse antenna activation for reconfigurable intelligent surface assisted communication,” _IEEE Trans. Commun._ , vol. 69, no. 10, pp. 6691–6705, Oct. 2021.
  * [118] K. Stylianopoulos, N. Shlezinger, P. Del Hougne, and G. C. Alexandropoulos, “Deep-learning-assisted configuration of reconfigurable intelligent surfaces in dynamic rich-scattering environments,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , Apr. 2022, pp. 8822–8826.
  * [119] A. Taha, M. Alrabeiah, and A. Alkhateeb, “Enabling large intelligent surfaces with compressive sensing and deep learning,” _IEEE Access_ , vol. 9, pp. 44 304–44 321, Mar. 2021.
  * [120] M. A. Aygül, M. Nazzal, and H. Arslan, “Deep learning-based optimal RIS interaction exploiting previously sampled channel correlations,” in _Proc. IEEE Wireless Commun. Netw. Conf (WCNC)_ , May 2021, pp. 1–6.
  * [121] G. C. Alexandropoulos, S. Samarakoon, M. Bennis, and M. Debbah, “Phase configuration learning in wireless networks with multiple reconfigurable intelligent surfaces,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM Workshops)_ , Dec. 2020, pp. 1–6.
  * [122] Ö. Özdoğan and E. Björnson, “Deep learning-based phase reconfiguration for intelligent reflecting surfaces,” in _Proc. Asilomar Conf. Signals Syst. Comput._ , Nov. 2020, pp. 707–711.
  * [123] B. Yang, X. Cao, C. Huang, C. Yuen, L. Qian, and M. Di Renzo, “Intelligent spectrum learning for wireless networks with reconfigurable intelligent surfaces,” _IEEE Trans. Vehi. Technol._ , vol. 70, no. 4, pp. 3920–3925, Apr. 2021.
  * [124] Y. Song, M. R. Khandaker, F. Tariq, K.-K. Wong, and A. Toding, “Truly intelligent reflecting surface-aided secure communication using deep learning,” in _Proc. IEEE Vehi. Technol. Conf. (VTC2021-Spring)_ , Jun. 2021, pp. 1–6.
  * [125] X. Hu, C. Masouros, and K.-K. Wong, “Reconfigurable intelligent surface aided mobile edge computing: From optimization-based to location-only learning-based solutions,” _IEEE Trans. Commun._ , vol. 69, no. 6, pp. 3709–3725, Mar. 2021.
  * [126] H. Song, M. Zhang, J. Gao, and C. Zhong, “Unsupervised learning-based joint active and passive beamforming design for reconfigurable intelligent surfaces aided wireless networks,” _IEEE Commun. Lett._ , vol. 25, no. 3, pp. 892–896, Mar. 2020.
  * [127] N. T. Nguyen, L. V. Nguyen, T. Huynh-The, D. H. Nguyen, A. L. Swindlehurst, and M. Juntti, “Machine learning-based reconfigurable intelligent surface-aided MIMO systems,” in _Proc. Int. Workshop Signal Process. Adv. Wireless commun. (SPAWC)_ , Nov. 2021, pp. 101–105.
  * [128] S. Dinh-Van, T. M. Hoang, R. Trestian, and H. X. Nguyen, “Unsupervised deep-learning-based reconfigurable intelligent surface-aided broadcasting communications in industrial IoTs,” _IEEE Internet of Things J._ , vol. 9, no. 19, pp. 19 515–19 528, Oct. 2022.
  * [129] J. Gao, C. Zhong, X. Chen, H. Lin, and Z. Zhang, “Unsupervised learning for passive beamforming,” _IEEE Commun. Lett._ , vol. 24, no. 5, pp. 1052–1056, May 2020.
  * [130] G. López-Lanuza, K. Chen-Hu, and A. G. Armada, “Deep learning-based optimization for reconfigurable intelligent surface-assisted communications,” in _Proc. IEEE Wireless Commun. Netw. Conf (WCNC)_ , May 2022, pp. 764–769.
  * [131] E. Björnson and L. Marcenaro, “Configuring an intelligent reflecting surface for wireless communications: Highlights from the 2021 ieee signal processing cup student competition [sp competitions],” _IEEE Signal Process. Mag._ , vol. 39, no. 1, pp. 126–131, Jan. 2021.
  * [132] Z. Yang, Y. Liu, Y. Chen, and J. T. Zhou, “Deep reinforcement learning for RIS-aided non-orthogonal multiple access downlink networks,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM)_ , Dec. 2020, pp. 1–6.
  * [133] A. Taha, Y. Zhang, F. B. Mismar, and A. Alkhateeb, “Deep reinforcement learning for intelligent reflecting surfaces: Towards standalone operation,” in _Proc. Int. Workshop Signal Process. Adv. Wireless commun. (SPAWC)_ , Aug. 2020, pp. 1–5.
  * [134] H. Yang, Z. Xiong, J. Zhao, D. Niyato, Q. Wu, H. V. Poor, and M. Tornatore, “Intelligent reflecting surface assisted anti-jamming communications: A fast reinforcement learning approach,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 3, pp. 1963–1974, Mar. 2020.
  * [135] H. Yang, Z. Xiong, J. Zhao, D. Niyato, L. Xiao, and Q. Wu, “Deep reinforcement learning-based intelligent reflecting surface for secure wireless communications,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 1, pp. 375–388, Jan. 2021.
  * [136] K. Feng, Q. Wang, X. Li, and C.-K. Wen, “Deep reinforcement learning based intelligent reflecting surface optimization for MISO communication systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 5, pp. 745–749, May 2020.
  * [137] C. Huang, Z. Yang, G. C. Alexandropoulos, K. Xiong, L. Wei, C. Yuen, and Z. Zhang, “Hybrid beamforming for RIS-empowered multi-hop terahertz communications: A DRL-based method,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM Workshops)_ , Dec. 2020, pp. 1–6.
  * [138] G. Lee, M. Jung, A. T. Z. Kasgari, W. Saad, and M. Bennis, “Deep reinforcement learning for energy-efficient networking with reconfigurable intelligent surfaces,” in _Proc. IEEE Int. Conf. Commun. (ICC)_ , Jul. 2020, pp. 1–6.
  * [139] J. Lin, Y. Zout, X. Dong, S. Gong, D. T. Hoang, and D. Niyato, “Deep reinforcement learning for robust beamforming in RIS-assisted wireless communications,” in _Proc. IEEE Global Commun. Conf. (GLOBECOM)_ , Dec. 2020, pp. 1–6.
  * [140] C. Huang, R. Mo, and C. Yuen, “Reconfigurable intelligent surface assisted multiuser MISO systems exploiting deep reinforcement learning,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1839–1850, Aug. 2020.
  * [141] Q. Zhang, W. Saad, and M. Bennis, “Millimeter wave communications with an intelligent reflector: Performance optimization and distributional reinforcement learning,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 3, pp. 1836–1850, Mar. 2021.
  * [142] X. Liu, Y. Liu, Y. Chen, and H. V. Poor, “RIS enhanced massive non-orthogonal multiple access networks: Deployment and passive beamforming design,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 4, pp. 1057–1071, Apr. 2020.
  * [143] J. Kim, S. Hosseinalipour, T. Kim, D. J. Love, and C. G. Brinton, “Multi-RIS-assisted multi-cell uplink MIMO communications under imperfect CSI: A deep reinforcement learning approach,” in _Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops)_ , Jul. 2021, pp. 1–7.
  * [144] M. Samir, M. Elhattab, C. Assi, S. Sharafeddine, and A. Ghrayeb, “Optimizing age of information through aerial reconfigurable intelligent surfaces: A deep reinforcement learning approach,” _IEEE Trans. Vehi. Technol._ , vol. 70, no. 4, pp. 3978–3983, Apr. 2021.
  * [145] Z. Yang, Y. Liu, Y. Chen, and N. Al-Dhahir, “Machine learning for user partitioning and phase shifters design in RIS-aided NOMA networks,” _IEEE Trans. Commun._ , vol. 69, no. 11, pp. 7414–7428, Nov. 2021.
  * [146] T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonker _et al._ , “Model-based reinforcement learning: A survey,” _Foundations and Trends® in Machine Learning_ , vol. 16, no. 1, pp. 1–118, 2023.
  * [147] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT Press, 2018.
  * [148] H. Liu, X. Yuan, and Y.-J. A. Zhang, “Reconfigurable intelligent surface enabled federated learning: A unified communication-learning design approach,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 11, pp. 7595–7609, Nov. 2021.
  * [149] Y. Yang, Y. Zhou, T. Wang, and Y. Shi, “Reconfigurable intelligent surface assisted federated learning with privacy guarantee,” in _Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops)_ , Jul. 2021, pp. 1–6.
  * [150] W. Ni, Y. Liu, Y. C. Eldar, Z. Yang, and H. Tian, “STAR-RIS integrated nonorthogonal multiple access and Over-the-Air federated learning: Framework, analysis, and optimization,” _IEEE Internet Things J._ , vol. 9, no. 18, pp. 17 136–17 156, Sep. 2022.
  * [151] C. Battiloro, M. Merluzzi, P. Di Lorenzo, and S. Barbarossa, “Dynamic resource optimization for adaptive federated learning empowered by reconfigurable intelligent surfaces,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , Apr. 2022, pp. 4083–4087.
  * [152] J. Zheng, H. Tian, W. Ni, W. Ni, and P. Zhang, “Balancing accuracy and integrity for reconfigurable intelligent surface-aided Over-the-Air federated learning,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 12, pp. 10 964–10 980, Dec. 2022.
  * [153] W. Ni, Y. Liu, Z. Yang, and H. Tian, “Over-the-air federated learning and non-orthogonal multiple access unified by reconfigurable intelligent surface,” in _Proc. IEEE Conf. on Comput. Commun. (INFOCOM Workshops)_ , Jul. 2021, pp. 1–6.
  * [154] W. Ni, Y. Liu, Z. Yang, H. Tian, and X. Shen, “Federated learning in multi-ris-aided systems,” _IEEE Internet of Things Journal_ , vol. 9, no. 12, pp. 9608–9624, Jun. 2021.
  * [155] H. Liu, X. Yuan, and Y.-J. A. Zhang, “Joint communication-learning design for RIS-assisted federated learning,” in _Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops)_ , Jul. 2021, pp. 1–6.
  * [156] W. Tang, M. Z. Chen, X. Chen, J. Y. Dai, Y. Han, M. Di Renzo, Y. Zeng, S. Jin, Q. Cheng, and T. J. Cui, “Wireless communications with reconfigurable intelligent surface: Path loss modeling and experimental measurement,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 1, pp. 421–439, Jan. 2021.
  * [157] B. Yang, X. Cao, C. Huang, C. Yuen, M. Di Renzo, Y. L. Guan, D. Niyato, L. Qian, and M. Debbah, “Federated spectrum learning for reconfigurable intelligent surfaces-aided wireless edge networks,” _IEEE Transactions on Wireless Communications_ , vol. 21, no. 11, pp. 9610–9626, Nov. 2022.
  * [158] T. Zhang and S. Mao, “Energy-efficient federated learning with intelligent reflecting surface,” _IEEE Trans. Green Commun. Netw._ , vol. 6, no. 2, pp. 845–858, Jun. 2021.
  * [159] L. Li, D. Ma, H. Ren, D. Wang, X. Tang, W. Liang, and T. Bai, “Enhanced reconfigurable intelligent surface assisted mmWave communication: A federated learning approach,” _China Commun._ , vol. 17, no. 10, pp. 115–128, Oct. 2020.
  * [160] R. Zhong, X. Liu, Y. Liu, Y. Chen, and Z. Han, “Mobile reconfigurable intelligent surfaces for NOMA networks: Federated learning approaches,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 11, pp. 10 020–10 034, Nov. 2022.
  * [161] P. Cui, X. Wang, J. Pei, and W. Zhu, “A survey on network embedding,” _IEEE Trans. Knowl. Data Eng._ , vol. 31, no. 5, pp. 833–852, May 2018.
  * [162] S. He, S. Xiong, Y. Ou, J. Zhang, J. Wang, Y. Huang, and Y. Zhang, “An overview on the application of graph neural networks in wireless networks,” _IEEE Open J. Commu. Soc._ , vol. 2, pp. 2547–2565, Nov. 2021.
  * [163] N. Naderializadeh, M. Eisen, and A. Ribeiro, “Wireless power control via counterfactual optimization of graph neural networks,” in _Proc. Int. Workshop Signal Process. Adv. Wireless commun. (SPAWC)_ , Aug. 2020, pp. 1–5.
  * [164] M. Eisen and A. Ribeiro, “Optimal wireless resource allocation with random edge graph neural networks,” _IEEE Trans. Signal Process._ , vol. 68, pp. 2977–2991, Apr. 2020.
  * [165] H. Jiang, H. He, and L. Liu, “Dynamic spectrum access for femtocell networks: A graph neural network based learning approach,” in _Proc. Int. Conf. Comput., Netw. and Commun. (ICNC)_ , Mar. 2020, pp. 927–931.
  * [166] H. Wang, Y. Wu, G. Min, and W. Miao, “A graph neural network-based digital twin for network slicing management,” _IEEE Trans. Industr. Inform._ , vol. 18, no. 2, pp. 1367–1376, Feb. 2020.
  * [167] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” _AI Open_ , vol. 1, pp. 57–81, Apr. 2020.
  * [168] M. A. Khamsi and W. A. Kirk, _An introduction to metric spaces and fixed point theory_. Hoboken, NJ, US: John Wiley & Sons, 2011.
  * [169] Z. Zhang, T. Jiang, and W. Yu, “Learning based user scheduling in reconfigurable intelligent surface assisted multiuser downlink,” _IEEE J. Sel. Topics Signal Process._ , vol. 16, no. 5, pp. 1026–1039, Feb. 2022.
  * [170] Z. Zhang, T. Jiang, and W. Yu, “User scheduling using graph neural networks for reconfigurable intelligent surface assisted multiuser downlink communications,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , Apr. 2022, pp. 8892–8896.
  * [171] T. Jiang, H. V. Cheng, and W. Yu, “Learning to reflect and to beamform for intelligent reflecting surface with implicit channel estimation,” _IEEE J. Sel. Areas Commun._ , vol. 39, no. 7, pp. 1931–1945, Jul. 2021.
  * [172] H. Zhou, M. Erol-Kantarci, and H. V. Poor, “Learning from peers: Deep transfer reinforcement learning for joint radio and cache resource allocation in 5G RAN slicing,” _IEEE Trans. Cogn. Commun. Netw._ , vol. 8, no. 4, pp. 1925–1941, Sep. 2022.
  * [173] H. Zhou, M. Erol-Kantarci, and V. Poor, “Knowledge transfer and reuse: A case study of AI-enabled resource management in RAN slicing,” _IEEE Wireless Commun._ , pp. 1–10, Dec. 2022.
  * [174] M. Elsayed, M. Erol-Kantarci, and H. Yanikomeroglu, “Transfer reinforcement learning for 5G new radio mmWave networks,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 5, pp. 2838–2849, May 2021.
  * [175] S. Pateria, B. Subagdja, A.-h. Tan, and C. Quek, “Hierarchical reinforcement learning: A comprehensive survey,” _ACM Comput. Surveys (CSUR)_ , vol. 54, no. 5, pp. 1–35, Jun. 2021.
  * [176] M. Jung and W. Saad, “Meta-learning for 6G communication networks with reconfigurable intelligent surfaces,” in _Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , Jun 2021, pp. 8082–8086.
  * [177] Y. Zou, Y. Liu, K. Han, X. Liu, and K. K. Chai, “Meta-learning for RIS-assisted NOMA networks,” in _Proc. IEEE Global Commu. Conf.(GLOBECOM)_ , Dec. 2021, pp. 1–6.
  * [178] J. Vanschoren, “Meta-learning: A survey,” pp. 1–29, Oct. 2018. [Online]. Available: arXiv preprint arXiv:1810.03548
  * [179] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _IEEE Trans. pattern analysis and machine intelligence_ , vol. 44, no. 9, pp. 5149–5169, Sep 2021.
  * [180] Z. Chu, Z. Zhu, F. Zhou, M. Zhang, and N. Al-Dhahir, “Intelligent reflecting surface assisted wireless powered sensor networks for internet of things,” _IEEE Trans. Commun._ , vol. 69, no. 7, pp. 4877–4889, Jul. 2021.
  * [181] C. Pan, H. Ren, K. Wang, M. Elkashlan, A. Nallanathan, J. Wang, and L. Hanzo, “Intelligent reflecting surface aided MIMO broadcasting for simultaneous wireless information and power transfer,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1719–1734, Aug. 2020.
  * [182] A. Liu, Z. Huang, M. Li, Y. Wan, W. Li, T. X. Han, C. Liu, R. Du, D. K. P. Tan, J. Lu _et al._ , “A survey on fundamental limits of integrated sensing and communication,” _IEEE Commun. Surveys Tuts._ , vol. 24, no. 2, pp. 994–1034, 2nd quarter 2022.
  * [183] Y. Yao, H. Zhou, and M. Erol-Kantarci, “Joint sensing and communications for deep reinforcement learning-based beam management in 6g,” in _Proc. IEEE Global Commu. Conf.(GLOBECOM)_. IEEE, Dec. 2022, pp. 5019–5024.
  * [184] Z. Wang, X. Mu, and Y. Liu, “STARS enabled integrated sensing and communications,” _IEEE Trans. on Wireless Commu. (Early access)_ , pp. 1–16, Feb. 2023.

|  Hao Zhou received his Ph.D. degree from University of Ottawa, Canada, in
2023. He is an incoming Postdoctoral Research Fellow at Department of Computer
Science, McGill University. Before that, he got his B.Eng. and M.Eng degrees
from Huazhong University of Science and Technology in 2016, and Tianjin
University in 2019, respectively, in China. His research interests include
microgrid energy trading, 5G network slicing, mmWave, reconfigurable
intelligent surfaces, and so on. He is devoted to developing machine learning
algorithms for Smart Grid and 5G/6G networks. He won the best paper reward at
the 2023 IEEE ICC conference, and the 2023 Outstanding Self-financed Abroad
Chinese Students Award.
---|---
|  Melike Erol-Kantarci is Chief Cloud RAN AI$\backslash$ML Data Scientist at
Ericsson and Canada Research Chair in AI-enabled Next-Generation Wireless
Networks and Full Professor at the School of Electrical Engineering and
Computer Science at the University of Ottawa. She is the founding director of
the Networked Systems and Communications Research (NETCORE) laboratory. She
has received numerous awards and recognitions. Dr. Erol-Kantarci is the co-
editor of three books on smart grids, smart cities and intelligent
transportation. She has over 200 peer-reviewed publications. She has delivered
70+ keynotes, plenary talks and tutorials around the globe. Dr. Erol-Kantarci
is on the editorial board of the IEEE Transactions on Communications, IEEE
Transactions on Cognitive Communications and Networking and IEEE Networking
Letters. She has acted as the general chair and technical program chair for
many international conferences and workshops. Her main research interests are
AI-enabled wireless networks, 5G and 6G wireless communications, smart grid
and Internet of Things. Dr. Erol-Kantarci is an IEEE ComSoc Distinguished
Lecturer, IEEE Senior member and ACM Senior Member.
---|---
|  Yuanwei Liu (S’13-M’16-SM’19-F’24),
(http://www.eecs.qmul.ac.uk/$\sim$yuanwei) received the PhD degree in
electrical engineering from the Queen Mary University of London, U.K., in
2016. He was with the Department of Informatics, King’s College London, from
2016 to 2017, where he was a Post-Doctoral Research Fellow. He has been a
Senior Lecturer (Associate Professor) with the School of Electronic
Engineering and Computer Science, Queen Mary University of London, since Aug.
2021, where he was a Lecturer (Assistant Professor) from 2017 to 2021. His
research interests include non-orthogonal multiple access, reconfigurable
intelligent surface, integrated sensing and communications, and machine
learning.
Yuanwei Liu is a Fellow of the IEEE, a Web of Science Highly Cited Researcher,
an IEEE Communication Society Distinguished Lecturer, an IEEE Vehicular
Technology Society Distinguished Lecturer, the academic Chair for the Next
Generation Multiple Access Emerging Technology Initiative, the rapporteur of
ETSI Industry Specification Group on Reconfigurable Intelligent Surfaces on
work item of “Multi-functional Reconfigurable Intelligent Surfaces (RIS):
Modelling, Optimisation, and Operation”, and the UK representative for the
URSI Commission C on “Radio communication Systems and Signal Processing”. He
was listed as one of 35 Innovators Under 35 China in 2022 by MIT Technology
Review. He received IEEE ComSoc Outstanding Young Researcher Award for EMEA in
2020. He received the 2020 IEEE Signal Processing and Computing for
Communications (SPCC) Technical Committee Early Achievement Award, IEEE
Communication Theory Technical Committee (CTTC) 2021 Early Achievement Award.
He received IEEE ComSoc Outstanding Nominee for Best Young Professionals Award
in 2021. He is the co-recipient of the Best Student Paper Award in IEEE
VTC2022-Fall, the Best Paper Award in ISWCS 2022, the 2022 IEEE SPCC-TC Best
Paper Award, and the 2024 IEEE ICCT Best Paper Award. He serves as the Co-
Editor-in-Chief of IEEE ComSoc TC Newsletter, an Area Editor of IEEE
Communications Letters, an Editor of IEEE Communications Surveys & Tutorials,
IEEE Transactions on Wireless Communications, IEEE Transactions on Vehicular
Technology, IEEE Transactions on Network Science and Engineering, and IEEE
Transactions on Communications (2018-2023). He serves as the (leading) Guest
Editor for Proceedings of the IEEE on Next Generation Multiple Access, IEEE
JSAC on Next Generation Multiple Access, IEEE JSTSP on Intelligent Signal
Processing and Learning for Next Generation Multiple Access, and IEEE Network
on Next Generation Multiple Access for 6G. He serves as the Publicity Co-Chair
for IEEE VTC 2019-Fall, the Panel Co-Chair for IEEE WCNC 2024, Symposium Co-
Chair for Cognitive Radio & AI-Enabled Networks for IEEE GLOBECOM 2022 and
Communication Theory for IEEE GLOBECOM 2023. He serves as the chair of Special
Interest Group (SIG) in SPCC Technical Committee on Signal Processing
Techniques for Next Generation Multiple Access, the vice-chair of SIG in SPCC
Technical Committee on Near Field Communications for Next Generation Mobile
Networks, and the vice-chair of SIG WTC on Reconfigurable Intelligent Surfaces
for Smart Radio Environments.
---|---
|  H. Vincent Poor (S’72, M’77, SM’82, F’87) received the Ph.D. degree in EECS
from Princeton University in 1977. From 1977 until 1990, he was on the faculty
of the University of Illinois at Urbana-Champaign. Since 1990 he has been on
the faculty at Princeton, where he is currently the Michael Henry Strater
University Professor. During 2006 to 2016, he served as the dean of
Princeton’s School of Engineering and Applied Science. He has also held
visiting appointments at several other universities, including most recently
at Berkeley and Cambridge. His research interests are in the areas of
information theory, machine learning and network science, and their
applications in wireless networks, energy systems and related fields. Among
his publications in these areas is the recent book Machine Learning and
Wireless Communications. (Cambridge University Press, 2022). Dr. Poor is a
member of the National Academy of Engineering and the National Academy of
Sciences and is a foreign member of the Chinese Academy of Sciences, the Royal
Society, and other national and international academies. He received the IEEE
Alexander Graham Bell Medal in 2017.
---|---