# Model Predictive Actor-Critic: Accelerating Robot
Skill Acquisition with Deep Reinforcement Learning

Andrew S. Morgan1*, Daljeet Nandha2*, Georgia Chalvatzaki2,
Carlo D’Eramo2, Aaron M. Dollar1, and Jan Peters2 Computations performed on
the Lichtenberg cluster of TU Darmstadt and the Yale HPC Grace cluster. This
work has been partially funded by the RoboTrust, Skill4Robots projects and NSF
Grants IIS-1752134 & IIS-1900681. Dr. Chalvatzaki is funded by the DFG EN
Program (CH 2676/1-1).1Department of Mechanical Engineering & Materials
Science, Yale University, USA. ({andrew.morgan,
aaron.dollar}@yale.edu).2Intelligent Autonomous Systems, Technische
Universität Darmstadt, Germany<EMAIL_ADDRESS>{georgia,
<EMAIL_ADDRESS>mail@jan-peters.net).∗Authors contributed equally.

###### Abstract

Substantial advancements to model-based reinforcement learning algorithms have
been impeded by the model-bias induced by the collected data, which generally
hurts performance. Meanwhile, their inherent sample efficiency warrants
utility for most robot applications, limiting potential damage to the robot
and its environment during training. Inspired by information theoretic model
predictive control and advances in deep reinforcement learning, we introduce
Model Predictive Actor-Critic (MoPAC)222Code available:
https://github.com/dnandha/mopac., a hybrid model-based/model-free method that
combines model predictive rollouts with policy optimization as to mitigate
model bias. MoPAC leverages optimal trajectories to guide policy learning, but
explores via its model-free method, allowing the algorithm to learn more
expressive dynamics models. This combination guarantees optimal skill learning
up to an approximation error and reduces necessary physical interaction with
the environment, making it suitable for real-robot training. We provide
extensive results showcasing how our proposed method generally outperforms
current state-of-the-art and conclude by evaluating MoPAC for learning on a
physical robotic hand performing valve rotation and finger gaiting–a task that
requires grasping, manipulation, and then regrasping of an object.

##  I Introduction

Figure 1:  (a) Complex robot skills, like in-hand manipulation, are difficult
to acquire and perform with conventional methods. (b) MoPAC is suitable for
training an underactuated hand to efficiently learn such tasks, like finger
gaiting, compared to other state-of-the-art techniques.

Robotic systems are expected to operate in increasingly unstructured and
dynamical environments. Aside from the difficulties evident in perception,
decision making, and planning, precisely controlling robots still remains
difficult, as they must operate under non-linear, contact-rich conditions.
Traditional approaches to this problem include methods of optimal control [1,
2], but require that a model of the robot and its environment is known a
priori, which generally cannot be guaranteed.

In this direction, model-based reinforcement learning (MBRL) approximates
iteratively the dynamics model of the environment while planning actions
through trajectory optimization [3, 4, 5, 6, 7, 8], commonly using Model
Predictive Control (MPC) [9, 10, 11]. Though sample-efficient, MBRL has been
heavily impeded by the bias in the learned model–as optimal control methods
will generally continue to exploit known regions of the learned dynamics
without exploring outwards to new, unrevealed states [12, 13, 14]. Model-free
reinforcement learning (MFRL), on the other hand, has achieved impressive
results in learning complex skills through deep neural networks [15, 16, 17,
18, 19], offering high performance. However, unlike their MB counterpart, the
lack of an internal model makes MF algorithms data-hungry, suffering from poor
use of samples; as the complexity of the task increases, so does the number of
samples required to learn the optimal policy [20, 21, 22]. This disadvantage
makes modern deep MFRL methods inappropriate for learning tasks on physical
robots [23, 24, 25], as the increased amount of necessary interactions is
likely to damage the system.

We believe that the effective combination of these RL paradigms will enable
the learning of complex skills on real robots [26, 27]. In this paper, we
address this challenge by investigating optimal methods of guiding policy
learning, while also learning the model dynamics. Intuitively, having
interweaving MB-MF components can potentially allow scheduling of separate
solutions according to different phases of learning, i.e. MB when action
planning is required, and MF when additional exploration is needed.

Various works try to mix the benefits of MB and MF methods. For example, [28]
proposes the use of trajectory optimization for guiding the policy search by
exploring high-reward regions. In [29], the authors propose a MB method which
learns an ensemble of models and optimizes a meta-policy objective over all
models. [30] proposes the initialization of MFRL algorithms from learned model
dynamics, for combining the sample efficiency of MB methods with the task-
specific performance of MF approaches. Many works study the use of MB methods
for accelerating the learning of MFRL. The latter can be extended into MB
methods by sampling from a model (or model ensemble to increase stochasticity)
[31, 11, 32]. MBPO [14] uses branched rollouts in an actor-critic setting to
exploit the learned dynamics for model-based policy optimization. [33]
proposes an extension to MBPO, by performing model rollouts of specific
horizons, while optimizing the policy objective with back-propagation through
time.

In this work, we introduce Model Predictive Actor-Critic (MoPAC)–an algorithm
that seamlessly combines the sample efficiency of MBRL, with MF actor-critic
methods for improved exploration. To ensure exploitation of the learned
policy, we propose the use of model predictive rollouts, a method that
inherits properties of model predictive path integrals [10], capitalizing on
the free-energy of the learned system and leveraging the information theoretic
constraint in the MB simulations. Our novel method is theoretically sound, as
we provide a bound on the performance of trajectory optimization through MPC,
when approximating the dynamics model and the value-function, that potentially
allows planning for longer time-horizons. Moreover, the maximum entropy
objective [24] in the policy optimization counterbalances the exploitation of
model predictive rollouts with exploration on the real environment, enabling
the learning of more expressive model dynamics, together with approximating
the optimal policy. Our empirical results, both in simulated control tasks and
on a physical robotic hand that performs in-hand manipulation, showcase the
accelerated learning that MoPAC offers against representative baselines,
evincing its effectiveness for learning complex skills on real-robot
platforms.

##  II Preliminaries

We consider Reinforcement Learning [34] for solving control problems modeled
as finite-horizon Markov Decision Processes (MDPs)
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma\rangle$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}$ is
the reward function, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$
is the transition kernel, and $\gamma\in[0,1)$ is the discount factor. We
define a policy $\pi\in\Pi:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ as the
probability distribution of the event of executing an action $a$ in a state
$s$. A policy $\pi$ induces a value function (VF) corresponding to the
expected cumulative discounted reward collected by the agent when executing
action $a$ in state $s$, and following the policy $\pi$ thereafter:
$Q^{\pi}(s,a)\triangleq\mathbb{E}\left[\sum_{k=0}^{\infty}\gamma^{k}r_{i+k+1}|s_{i}=s,a_{i}=a,\pi\right]$,
where $r_{i+1}$ is the reward obtained after the $i$-th transition. Solving an
MDP consists of finding the optimal policy $\pi^{*}$, i.e. the one maximizing
the expected cumulative discounted reward.

Trajectory Optimization designs a trajectory that minimizes some measure of
performance. MPC is such a technique that optimizes a cost function over a
finite time-horizon, while taking into account the system dynamics. The cost
function is optimized w.r.t. a control variable, yielding the optimal control
value for a given state with consideration of the predicted future states. In
essence, MPC provides a locally optimal policy or sequence of actions (up to
horizon $H$), based on the following optimization problem:

$\displaystyle\pi_{MPC}(s)=\arg\max_{\pi_{0:H-1}}\mathbb{E}[\
\sum\limits_{t=0}^{H-1}\gamma^{t}r(s_{t},a_{t})+\gamma^{H}r_{f}(s_{H})]$ (1)
$\displaystyle\mbox{ }a_{t}=\pi_{t}(s_{t}),s_{0}=s$

where the states evolve according to the transition dynamics of the MDP, i.e.
$s_{t+1}=f(s_{t},a_{t})$. From each optimized sequence resulting from the
optimization process of MPC (of length equal to horizon $H$), the first action
is applied to the agent, and the procedure is repeated again at the next time
step. The term $r_{f}(s_{H})$ denotes the terminal reward.

In the context of MBRL, [35] combines MB trajectory optimization with VF
estimation. [36] uses model predictive path integrals [37] in a $Q$-learning
setting, and [38] shows that combining MPC with VF approximation yields
optimal policies, however considering the true dynamics.

##  III Performance bound under imperfect model

Coupling MPC with a VF, that propagates global information, optimizes action
sequences for longer time horizons, while being less prominent to
approximation errors than greedy action selection [38]. In this work, we
design an actor-critic algorithm that will benefit from the integration of
trajectory optimization for acquiring optimal control policies, together with
the learning of stable VF approximations provided by modern deep actor-critic
algorithms, e.g. soft actor-critic (SAC) [24]. To do so, we extend the bound
of [38] by incorporating the approximation error when learning the dynamics
model.

######  Theorem 1.

Let the approximation error of the dynamics model be
$\epsilon_{f}=|\hat{f}(s,a)-f^{*}(s,a)|$, the approximation error of the VF be
$\epsilon_{V}=max_{s}|\hat{V}(s)-V^{*}(s)|$, and the terminal reward of (1) be
$r_{f}(s_{H})=\hat{V}(s_{H})$, then the performance of the MPC policy in (1)
with the learned dynamics model is bounded by:

$J(\pi^{*})-J(\pi^{MPC})\leq\frac{2\gamma^{H}\epsilon_{V}}{1-\gamma^{H}}+r_{max}\frac{1-\gamma^{H}}{1-\gamma}\epsilon_{f}.$
(2)

Proof is provided in the Appendix. As the contribution of the model error
increases with the horizon, $H\to\infty$ leads to the upper bound

$J(\pi^{*})-J(\pi^{MPC})\leq\frac{r_{max}\epsilon_{f}}{1-\gamma}.$ (3)

Algorithm 1 Model Predictive Actor-Critic

1:Initialize parameters ${\theta}$, $\rho$, $\psi$, $\bar{\psi}$, $\phi$,
$\bar{\phi}$

2:Initialize $D$, $D_{env}$, $D_{model}$ $\triangleright$ initialize
experience buffers

3:$\bar{\phi}\leftarrow\phi$, $\bar{\psi}\leftarrow\psi$ $\triangleright$
initialize target parameters

4:for each iteration do

5: $D_{env}\leftarrow
D_{env}\cup\\{s_{t+1},a_{t},s_{t}\\},a_{t}\sim\pi_{\theta}(s_{t})$

6: for N epochs do

7: Train model $f_{\rho}$ on $D_{env}$:
$\rho\leftarrow\rho-\lambda_{f}\nabla_{\rho}J_{f_{\rho}}$

8: end for

9: for M model predictive rollouts do

10: Sample $s_{t}$ uniformly from $D_{env}$

11: Perform MPR (Alg. 2) from $s_{t}$

12: Add transitions to $D_{model}$

13: end for

14: for G gradient steps do

15: Update parameters using data $D\leftarrow D_{env}\cup D_{model}$

16: $\psi\leftarrow\psi-\lambda_{\psi}\nabla_{\psi}J_{V_{\psi}}$
$\triangleright$ VF update

17: $\phi\leftarrow\phi-\lambda_{\phi}\nabla_{\phi}J_{Q_{\phi}}$
$\triangleright$ Q update

18: $\theta\leftarrow\theta-\lambda_{\theta}\nabla_{\theta}J_{\pi_{\theta}}$
$\triangleright$ policy update

19: $\bar{\psi}\leftarrow\tau\psi+(1-\tau)\psi$ $\triangleright$ target VF
update

20: $\bar{\phi}\leftarrow\tau\phi+(1-\tau)\phi$ $\triangleright$ target Q
update

21: end for

22:end for

As expected, the performance error between the optimal policy and the MPC
policy is affected by the model approximation error $\epsilon_{f}$, given a
prediction horizon $H$. As we will show in the following, the maximum entropy
exploration of SAC [24] can acquire more expressive dynamics models by
visiting unmodeled transitions in the environment; together with the
approximation of the VF, we can leverage the bound of (2) to acquire near-
optimal trajectories for policy learning.

Model-based monotonic improvement, as proven by [14], can be achieved when
learning the dynamics model together with the policy. Namely, the authors give
an upper-bound in the performance gain obtained when applying the learned
policy to the learned dynamics model, compared to applying it on the real MDP
(i.e. the true dynamics). Their finding is summarized in the following lemma.

######  Lemma 2.

Let the expected TV-distance error of the transition probability distributions
be bounded by $\varepsilon_{f}$ and the policy divergence be bounded by
$\varepsilon_{\pi}$. Then the following bound holds:

$\displaystyle J(\pi)-\hat{J}(\pi)\geq-\left[\frac{2\gamma
r_{max}(\varepsilon_{f}+2\varepsilon_{\pi})}{1-\gamma^{2}}+\frac{4r_{max}\varepsilon_{\pi}}{1-\gamma}\right].$
(4)

This lemma is directly applicable to our proposed algorithm. Its combination
with our Theorem 1 suggests that sufficiently low errors in model learning and
policy approximation can yield near-optimal performance.

Algorithm 2 Model Predictive Rollouts (MPR)

1:$s_{0}$, $\rho$, $\bar{\psi}$, $\pi_{\theta}$, $n$, $\zeta$, $\gamma$,
$\lambda$ $\triangleright$ start state, model parameters, target VF
parameters, current policy, actuation noise, annealing parameter, discount,
control hyperparameter

2:$a_{MPR},s_{MPR}$ $\triangleright$ MPR trajectory

3:$H\leftarrow anneal(\zeta)$ $\triangleright$ horizon length

4:$\mathcal{R}\leftarrow zeros(\cdot)$ $\triangleright$ trajectory reward

5:for $t=0,...,H-1$ do

6: Sample an initial action from policy $a_{t}\sim\pi_{\theta}(s_{t})$

7: Sample exploration noise $\\{n_{0},\dots,n_{H-1}\\}\sim n$

8: $\mathcal{R}\leftarrow\mathcal{R}+\gamma^{t}r(s_{t},a_{t}+n_{t})$
$\triangleright$ rewards

9: $s_{t+1}\leftarrow f_{\rho}(s_{t},a_{t}+n_{t})$ $\triangleright$ system
dynamics

10:end for

11:$\mathcal{R}\leftarrow\mathcal{R}+\gamma^{H}V_{\bar{\psi}}(s_{H})$
$\triangleright$ add terminal state reward

12:$\mathcal{C}=-\mathcal{R}$ $\triangleright$ convert to cost minimization

13:$\beta\leftarrow min[\mathcal{C}]$

14:$\eta\leftarrow\sum_{i=1}^{N}\exp(-\frac{1}{\lambda}[\mathcal{C}_{i}-\beta])$

15:$w(n)\leftarrow\frac{1}{\eta}\exp(-\frac{1}{\lambda}[\mathcal{C}-\beta])$
$\triangleright$ import. sampling weight

16:$a_{MPR}\leftarrow a+\sum_{i=1}^{N}w(n_{i})n_{i}$ $\triangleright$ adjust
action sequence

17:$s_{MPR}\leftarrow f_{\rho}(s_{0},a_{MPR})$ $\triangleright$ state
following optimal action

##  IV Model Predictive Actor-Critic

We introduce Model Predictive Actor-Critic (MoPAC), an algorithm that
leverages the theoretical guarantees provided by Theorem 1 and Lemma 2. MoPAC
has three main interacting components: (i) model learning from environment
transitions, (ii) model predictive rollouts for acquiring samples from optimal
trajectories, and (iii) soft updates using a maximum entropy objective for
policy learning over a mixture of model and environment data. Algorithm 1
summarizes our approach.

Model learning. We use an ensemble of $N$ functions to approximate the model
of the environment $\\{f_{\rho_{1}},...,f_{\rho_{N}}\\}$. Each of these
functions is a probabilistic deep neural network whose purpose is to
approximate the system dynamics, namely the next state of the agent given the
current state and action. Probabilistic model ensembles have been studied as a
way of realizing Bayesian neural networks, for capturing the epistemic
uncertainty when learning complex dynamics, in order to mitigate overfitting
when using a single model [11]. Specifically, an ensemble of models is
randomly initialized within an observable space; hence, each model learns a
different mapping of the dynamics. For generating predictions from the
ensembles, we sample transitions from the elite networks, i.e. those with the
lowest L2-loss in a validation set, during the model rollouts by sampling
uniformly a model for every simulation.

Model predictive rollouts. Solving the MPC optimization problem of (1) is
prohibitively expensive and hard to obtain online. This limitation is overcome
by the information theoretic model predictive path integral (i-MPPI) method
[10], a sampling-based algorithm that uses an approximation of the true
dynamics, and is able to optimize both convex and non-convex cost criteria,
thus being applicable to large classes of stochastic systems and
representations. i-MPPI uses the free energy of the system and relative
entropy (KL-divergence) for providing generalized path integral expressions.
Crucially, it uses importance sampling for acquiring the optimal control
paths. In MoPAC, we use a similar setting for generating model-based rollouts
that will profit from the optimal performance guarantees of Sec. III. In our
model predictive rollouts (MPR), we initialize the control sequence employing
the learned policy $\pi_{\theta}$ to sample initial actions. Then, we design a
similar setup as in i-MPPI for creating trajectory simulations in a specific
time horizon $H$, starting from an initial true state (Alg. 2). We evaluate
the rewards of the rollouts, using also a VF for the final state (2), and we
perform importance sampling of the optimal transitions over all simulations,
along with an information-theoretic update of the actions’ exploration noise.
Finally, we collect the resulting optimal transitions in the model-based
replay memory $D_{model}$.

When the approximation error of the learned model is low, the trajectory
optimization on the learned model performed by MoPAC yields comparable
performance as acting on the real environment (Lemma 2). This remark, combined
with the use of the VF, allows us to obtain near-optimal performance–up to an
approximation error (Theorem 1). Moreover, using the learned policy
distribution for sampling the initial control sequence, together with the
relative entropy objective, guarantees that we are simulating transitions in a
reasonable area around the learned policy, providing good trajectories for
exploitation, while the underlying actor-critic explores new transitions.

Soft policy optimization. As the underlying actor-critic algorithm in MoPAC,
we adopt SAC [24] to benefit from the exploration induced by the soft policy
updates based on the maximum entropy principle, counterbalancing the effect of
the exploitation induced by MPRs. Nevertheless, MoPAC can be applied to any
known off-policy actor-critic algorithm. In SAC, the training of the policy
alternates between a soft policy evaluation step based on the soft Bellman
backup operator [25], and a soft policy improvement step that minimizes the
expected KL divergence: $J_{\pi}(\theta,D)=\mathbb{E}_{s_{t}\sim
D}[D_{KL}(\pi||\exp{\\{Q^{\pi}-V^{\pi}\\}})]$, where $Q^{{\pi}}\mbox{,
}V^{\pi}$ are the soft Q-function and soft VF of policy $\pi$ respectively.

##  V Experimental results

Figure 2: Experimental results comparing MoPAC to other baselines in both
MuJoCo environments (a-d) and robotic manipulation tasks (e-g). Results on
MuJoCo tasks are averaged over $5$ separate experiments, while for the robotic
manipulation tasks we learn a single policy and average its performance on $5$
evaluation runs. All plots show the $95\%$ confidence intervals. The automated
object reset system is depicted in the bottom right.

###  V-A Simulated tasks

We evaluate MoPAC in the simulated control tasks of MuJoCo [39] included in
the OpenAI-gym library [40]: HalfCheetah-v2, Ant-v2, Hopper-v2, and
Walker2d-v2. We compare the average return of MoPAC over $5$ trials,
consisting of $1,000$ true environment interactions per epoch, against the
baselines SAC [25], MBPO [14], and MBRL [10]. We decided not to compare with
[33] as the absence of code and a detailed algorithm, makes the reproduction
of their results prohibitive. Here, SAC and MBPO are trained according to the
hyperparameter settings provided by their respective works, while for MBRL we
use the same settings as MoPAC, namely using horizons $5-15$ with linear
annealing for all tasks. The number of environment interactions per episode is
constant across all algorithms. A batch size of $10,000$ is chosen for the
model rollouts in both MoPAC and MBPO.

Fig. 2 shows the average return of each algorithm per environment, plotted
w.r.t. the number of epochs. In all environments MoPAC and MBPO outperform
both SAC, evincing the advantage of using model rollouts, and MBRL, that
strongly suffers from poor exploration in the use of the learned model.
Notably, MoPAC learns faster than MBPO. Specifically, we observe a significant
speed-up in the learning of HalfCheetah-v2 and Walker2d-v2. The Ant-v2 and
Hopper-v2 are more challenging tasks, as they require more interactions with
the environment to learn their dynamics; in Hopper-v2 we observe a speedup in
learning and convergence, while in Ant-v2 MoPAC learns faster in the first
epochs and ends up with slightly better performance than MBPO. We expect that
by further tuning the prediction horizons of our MPR, as now we use the same
horizons across all tasks regardless of task-complexity, will result in
increased performance.

###  V-B Robotic tasks

We further underscore the efficacy of MoPAC by comparing our algorithm with
SAC and MBPO on a Yale Openhand Model Q [41, 42] through two different
manipulation tasks–valve rotation and finger gaiting. The Model Q is an open
source underactuated hand with four two-link fingers and four total actuators
(Dynamixel XM-430). Within the hand, a single motor actuates two opposing
fingers that are coupled by a differential, allowing passive reconfigurability
between the fingers. Two additional motors actuate the remaining two fingers
individually. The fourth and final motor serves to rotate the palm of the
hand, allowing the two coupled fingers to reorient perpendicular to the palm
axis (Fig. 1). Being underactuated, the Model Q’s joint configuration cannot
be accurately determined, as with many soft, compliant, or underactuated
hands, this hand is not equipped with joint encoders or tactile sensors [43].
This inherent compliance suits the spirit of our evaluations well, as the
reconfigurability of the fingers presents added challenges in sufficiently
learning the dynamics of the environmental interactions.

Valve Rotation. We develop two different valves, the penta-valve and the
round-valve (resembling the geometry of a door knob), for evaluation (Fig. 1).
Each of the valves were connected to a current-disabled Dynamixel actuator
placed directly below the hand as to measure rotation via the encoder. Albeit
disabled, valve rotation was resisted by the $353.5:1$ gearbox inside of the
actuator, generally limiting rotation to only large actions from the hand (see
twisting of flexure joints in Fig. 1). Policies were trained for both valves
with each of the three aforementioned algorithms on the physical system. The
goal of this task was to continually rotate the valve counterclockwise until
episode completion ($50$ actions). We define a standardized reward function,
$r(\cdot)=\theta_{s^{\prime}}-\theta_{s}$, and run each of the algorithms with
similar hyperparameters and epoch lengths ($50$ interactions per episode, $5$
episodes per epoch, MPR horizons $2-5$ annealed).

Experimental results to these tasks, depicted in Fig. 2(e-g), illustrate the
benefit of our MoPAC algorithm compared to baselines. Notably, the reward
convergence of the round-valve was less than that of the penta-valve,
generally due to the increased difficulty in estimating the task’s dynamical
hand-object nature, since the round-valve requires grasping before rotation.
Due to the design of the hand’s flexure joints, the motors are decreasingly
able to provide rotational torque the more a finger is actuated, i.e. off-axis
torsion limits the amount of force the finger-contacts can transmit to rotate
the object. This further leads to a grasping and rotational action that, if
enacted with rigid joints instead of compliant ones, would result in a reward
from the valve rotation, but does not necessarily happen with flexure joints.
Generally, the function of mapping grasping force and palm orientation to
valve rotation must first be inferred by the dynamics model in order to
sufficiently learn the task.

Albeit in some ways easier to estimate, hand-object dynamics of the penta-
valve is also not deterministic, as the fingers interact with both the
fingerpad (rubber) and side of the finger (plastic), where slipping can occur.
This task noted a higher converging reward comparatively, as it was not
required that the valve be grasped for rotation, dismissing off-axis torsion
constraints. Towards the beginning of training, the hand utilized the two
independent, non-rotational fingers to achieve a greedy reward, i.e. small
movement given the valve configuration, but soon thereafter noted that
retreating the fingers into a non-actuated position generally benefited the
cumulative reward as to limit interference with the valve.

Finger Gaiting. In addition to the valve rotation assessment, we further
challenged our algorithm with the task of finger gaiting–a task requiring
coordinated movement between the opposing finger pairs in the hand. Here, the
hand is not only expected to maintain a stable grasp on the object during the
entire episode, but to rotate the object about the palm axis of the hand (Fig.
1). We incorporated a similar reward function as in the previous section,
$r(\cdot)=\theta_{s^{\prime}_{palm}}-\theta_{s_{palm}}$, but now the object is
not constrained to only reorient along the reward axis, being free to move in
$SE(3)$ space. In this evaluation, we utilize the apple (Obj. #13) from the
YCB Object and Model Set [44] as to maintain standardization of the task.

The episode starts with a stable grasp acquired by the two individually driven
fingers, with the coupled fingers close to, but not touching, the object. We
monitor the pose of the object during manipulation via an AprilTag [45]
attached to the bottom of the apple, and detected by an external camera.
Subsequently, the reward of an action can be calculated through this setup, in
addition to detecting when the hand drops the object. When a drop is detected,
the episode is ended and the object is systematically reset via an automated
object reset system (Fig. 2). This system consists of an object crane that
controls a tether routed through the center of the hand. During reset, the
object lifts the object into the palm of the hand as to stabilize, and then
slowly lowers the object into the starting position.

In this evaluation, we note similar curves to those in the valve rotation task
(Fig. 2). Specifically, we see MoPAC outperform the two other algorithms,
reaching convergence faster with fewer environment interactions. Intuitively,
this quality is increasingly advantageous for tasks where system reset cannot
be completed quickly or autonomously–although it was possible in this
experiment, object reset required about $12$ seconds and slowed the learning
process. Training for the valve rotation tasks took $\sim 2$ hours for each
valve and each learning algorithm, whereas training for finger gaiting took
$\sim 5$ hours. Notably, if we were to stop training upon MoPAC convergence,
significant time can be saved, which is especially desirable when increasing
task complexities.

##  VI Conclusions

Transferring the advances of deep RL into real-world robotic problems is
challenging. Deep model-free RL (MFRL) methods, though able to learn complex
skills, typically require an excessive amount of interactions with the
environment, making their applicability prohibitive for robotics. On the other
hand, model-based RL (MBRL) approaches learn the dynamics model and select
actions through trajectory optimization techniques; albeit sample-efficient,
they suffer from local optima. In this work, we proposed Model Predictive
Actor-Critic (MoPAC), a method seeking to combine the advantages of deep MF
actor-critic methods with the data-efficiency of MB approaches. In particular,
MoPAC introduces model predictive rollouts, inspired by information-theoretic
model predictive path integrals, based on the principles of the dynamics free-
energy and on an information-theoretic constraint for collecting samples
through trajectory optimization. MoPAC uses a MFRL actor-critic algorithm for
policy improvement and model learning, and it uses the model for performing
the model predictive rollouts to collect additional samples for guiding policy
learning. The core advantage of MoPAC is that, though MB rollouts favor policy
exploitation through planning using the model, the MFRL actor-critic
encourages efficient exploration for policy optimization and model learning.

Our model predictive rollouts are backed up by a performance bound, which
guarantees that sufficiently low errors in the value function, and model
approximation, yields near-optimal performance. We empirically demonstrate the
efficiency of MoPAC in providing accelerated policy learning for simulated
control tasks against representative baselines. Furthermore, we showcase the
applicability of MoPAC for learning challenging in-hand manipulation tasks
with a four-fingered robotic hand. In the future, we will study ways of
principally adjusting the mixing of the MB-MF samples across the training
process, but also ways of scheduling the MB over the MF method according to
the learning progress, and vice versa.

##  VII Appendix

Proof of Theorem 1. Let the performance of applying policy $\hat{\pi}$ from
MPC using the approximated model $\hat{f}$ be denoted as $\hat{V}$ and the
performance gain of applying the optimal policy $\pi^{*}$ on the perfect model
$f^{*}$ be $V^{*}$ over a planning horizon $H$. The performance error for any
given starting state $s$ is

$\displaystyle V^{*}(s)-\hat{V}(s)$ $\displaystyle=\sum_{s\sim
f^{*}}[\sum_{t=0}^{H-1}f^{*}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}V^{*}(s_{H})]$
$\displaystyle-\sum_{s\sim\hat{f}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}\hat{V}(s_{H})].$
(5)

Adding and subtracting
$\sum_{s\sim\hat{f}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}V^{*}(s_{H})]$in
(VII) yields

$\displaystyle V^{*}(s)-\hat{V}(s)$
$\displaystyle=\gamma^{H}\sum_{s\sim\hat{f}}[V^{*}(s_{H})-\hat{V}(s_{H})]$
$\displaystyle\quad+\sum_{s\sim
f^{*}}[\sum_{t=0}^{H-1}f^{*}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}V^{*}(s_{H})]$
$\displaystyle-\sum_{s\sim\hat{f}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}V^{*}(s_{H})].$
(6)

Since the value function error is upper-bounded by
$max_{s}|V^{*}(s)-\hat{V}(s)|=\epsilon_{V}$, we can bound the following
relations as

$\displaystyle\sum_{s\sim f^{*}}$
$\displaystyle[\sum_{t=0}^{H-1}f^{*}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}V^{*}(s_{H})]\leq$
$\displaystyle\sum_{s\sim f^{*}}$
$\displaystyle[\sum_{t=0}^{H-1}f^{*}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}\hat{V}(s_{H})]+\gamma^{H}\epsilon_{V}$
(7) $\displaystyle\sum_{s\sim\hat{f}}$
$\displaystyle[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}V^{*}(s_{H})]\geq$
$\displaystyle\sum_{s\sim\hat{f}}$
$\displaystyle[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}+\gamma^{H}\hat{V}^{(}s_{H})-\gamma^{H}\epsilon_{V}.$
(8)

Substituting (VII) into (6), we have

$\displaystyle
V^{*}(s)-\hat{V}(s)\leq\gamma^{H}\sum_{s\sim\hat{f}}[V^{*}(s_{H})-\hat{V}(s_{H})]+2\gamma^{H}\epsilon_{V}$
$\displaystyle+\sum_{s\sim
f^{*}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}-\sum_{s\sim\hat{f}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}r_{t}$
$\displaystyle\leq\gamma^{H}\sum_{s\sim\hat{f}}[V^{*}(s_{H})-\hat{V}(s_{H})]+2\gamma^{H}\epsilon_{V}$
$\displaystyle+r_{max}\\{\sum_{s\sim
f^{*}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}]-\sum_{s\sim\hat{f}}[\sum_{t=0}^{H-1}\hat{f}(s_{t},a_{t})\gamma^{t}]\\}$
$\displaystyle=\gamma^{H}\sum_{s\sim\hat{f}}[V^{*}(s_{H})-\hat{V}(s_{H})]+2\gamma^{H}\epsilon_{V}+r_{max}\sum_{t=0}^{H-1}\gamma^{t}\epsilon_{f}$
$\displaystyle\leq
2\gamma^{H}\epsilon_{V}(1+\gamma^{H}+\gamma^{2H}+...)+r_{max}\sum_{t=0}^{H-1}\gamma^{t}\epsilon_{f}$
$\displaystyle\leq\frac{2\gamma^{H}\epsilon_{V}}{1-\gamma^{H}}+r_{max}\sum_{t=0}^{H-1}\gamma^{t}\epsilon_{f}=\frac{2\gamma^{H}\epsilon_{V}}{1-\gamma^{H}}+r_{max}\frac{1-\gamma^{H}}{1-\gamma}\epsilon_{f}$
(9) $\displaystyle\leq\frac{r_{max}\epsilon_{f}}{1-\gamma}.$ (10)

For MPC with imperfect dynamics and prediction horizon $H$, the bound (9)
holds, showcasing the inevitable error due to the model approximation.
However, this relation can be upper-bounded by the quantity in (10)
considering an infinite-horizon, $H\to\infty$, prediction. The bound of (9)
shows that sufficiently low approximation errors in the VF and the model can
yield near-optimal performance, which is related to the prediction horizon and
the discounted factor.

## References

  * [1] Y. Tassa, T. Erez, and W. D. Smart, “Receding horizon differential dynamic programming,” in _Advances in neural information processing systems_ , 2008, pp. 1465–1472.
  * [2] C. G. Atkeson, “Using local trajectory optimizers to speed up global optimization in dynamic programming,” in _Advances in neural information processing systems_ , 1994, pp. 663–670.
  * [3] M. Deisenroth and C. E. Rasmussen, “Pilco: A model-based and data-efficient approach to policy search,” in _Proceedings of the 28th International Conference on machine learning (ICML-11)_ , 2011, pp. 465–472.
  * [4] V. Kumar, E. Todorov, and S. Levine, “Optimal control with learned local models: Application to dexterous manipulation,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 378–383.
  * [5] Y. Luo, H. Xu, Y. Li, Y. Tian, T. Darrell, and T. Ma, “Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees,” _arXiv preprint arXiv:1807.03858_ , 2018.
  * [6] G. Chalvatzaki, X. S. Papageorgiou, P. Maragos, and C. S. Tzafestas, “Learn to adapt to human walking: A model-based reinforcement learning approach for a robotic assistant rollator,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 3774–3781, 2019.
  * [7] T. Wang and J. Ba, “Exploring model-based planning with policy networks,” _arXiv preprint arXiv:1906.08649_ , 2019.
  * [8] A. S. Morgan, K. Hang, and A. M. Dollar, “Object-agnostic dexterous manipulation of partially constrained trajectories,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5494–5501, 2020.
  * [9] S. Levine and P. Abbeel, “Learning neural network policies with guided policy search under unknown dynamics,” in _Advances in Neural Information Processing Systems_ , 2014, pp. 1071–1079.
  * [10] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou, “Information theoretic mpc for model-based reinforcement learning,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 1714–1721.
  * [11] K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 4754–4765.
  * [12] B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter, “Differentiable mpc for end-to-end planning and control,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 8289–8300.
  * [13] M. Pereira, D. D. Fan, G. N. An, and E. Theodorou, “Mpc-inspired neural network policies for sequential decision making,” _arXiv preprint arXiv:1802.05803_ , 2018.
  * [14] M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 12 519–12 530.
  * [15] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 229–256, 1992.
  * [16] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International conference on machine learning_ , 2015, pp. 1889–1897.
  * [17] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [18] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [19] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [20] S. Fujimoto, H. Van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” _arXiv preprint arXiv:1802.09477_ , 2018\.
  * [21] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1334–1373, 2016.
  * [22] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine, “Composable deep reinforcement learning for robotic manipulation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 6244–6251.
  * [23] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 3389–3396.
  * [24] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” _arXiv preprint arXiv:1801.01290_ , 2018.
  * [25] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [26] Y. Chebotar, K. Hausman, M. Zhang, G. Sukhatme, S. Schaal, and S. Levine, “Combining model-based and model-free updates for trajectory-centric reinforcement learning,” _arXiv preprint arXiv:1703.03078_ , 2017.
  * [27] V. Pong, S. Gu, M. Dalal, and S. Levine, “Temporal difference models: Model-free deep rl for model-based control,” _arXiv preprint arXiv:1802.09081_ , 2018.
  * [28] S. Levine and V. Koltun, “Guided policy search,” in _International Conference on Machine Learning_ , 2013, pp. 1–9.
  * [29] I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel, “Model-based reinforcement learning via meta-policy optimization,” _arXiv preprint arXiv:1809.05214_ , 2018.
  * [30] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine, “Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 7559–7566.
  * [31] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel, “Model-ensemble trust-region policy optimization,” _arXiv preprint arXiv:1802.10592_ , 2018\.
  * [32] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar, “Deep dynamics models for learning dexterous manipulation,” in _Conference on Robot Learning_ , 2020, pp. 1101–1112.
  * [33] I. Clavera, V. Fu, and P. Abbeel, “Model-augmented actor-critic: Backpropagating through paths,” _arXiv preprint arXiv:2005.08068_ , 2020\.
  * [34] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [35] F. Farshidian, D. Hoeller, and M. Hutter, “Deep value model predictive control,” _arXiv preprint arXiv:1910.03358_ , 2019.
  * [36] M. Bhardwaj, A. Handa, D. Fox, and B. Boots, “Information theoretic model predictive q-learning,” in _Learning for Dynamics and Control_ , 2020, pp. 840–850.
  * [37] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Aggressive driving with model predictive path integral control,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 1433–1440.
  * [38] K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch, “Plan online, learn offline: Efficient learning and exploration via model-based control,” _arXiv preprint arXiv:1811.01848_ , 2018.
  * [39] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2012, pp. 5026–5033.
  * [40] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [41] R. R. Ma and A. M. Dollar, “An underactuated hand for efficient finger-gaiting-based dexterous manipulation,” in _2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014)_ , 2014, pp. 2214–2219.
  * [42] “Yale openhand project.” [Online]. Available: https://www.eng.yale.edu/grablab/openhand/
  * [43] A. S. Morgan, W. G. Bircher, and A. M. Dollar, “Towards generalized manipulation learning through grasp mechanics-based features and self-supervision,” _IEEE Transactions on Robotics_ , pp. 1–17, 2021.
  * [44] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set,” _IEEE Robotics & Automation Magazine_, vol. 22, no. 3, pp. 36–52, 2015.
  * [45] J. Wang and E. Olson, “Apriltag 2: Efficient and robust fiducial detection,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 4193–4198.


# Present and Future of Reconfigurable Intelligent Surface-Empowered
Communications

Ertugrul Basar, and H. Vincent Poor E. Basar is with CoreLab, Department of
Electrical and Electronics Engineering, Koç University, Istanbul, Turkey
(e-mail: ebasar@ku.edu.tr).H. Vincent Poor is with the Department of
Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544
USA (e-mail: poor@princeton.edu).

## Introduction

Signal processing and communication communities have witnessed the rise of
many exciting communication technologies in recent years. Notable examples
include alternative waveforms, massive multiple-input multiple-output (MIMO)
signaling, non-orthogonal multiple access (NOMA), joint communications and
sensing, sparse vector coding, index modulation, and so on. It is inevitable
that 6G wireless networks will require a rethinking of wireless communication
systems and technologies, particularly at the physical layer (PHY),
considering the fact that the cellular industry reached another important
milestone with the development of 5G wireless networks with diverse
applications [1]. Within this perspective, this article aims to shed light on
the rising concept of reconfigurable intelligent surface (RIS)-empowered
communications towards 6G wireless networks [2, 3]. Software-defined RISs can
manipulate their impinging signals in an effective way to boost certain key
performance indicators. We discuss the recent developments in the field and
put forward promising candidates for future research and development.
Specifically, we put our emphasis on active, transmitter-type, transmissive-
reflective, and standalone RISs, by discussing their advantages and
disadvantages compared to reflective RIS designs. Finally, we also envision an
ultimate RIS architecture, which is able to adjust its operation modes
dynamically, and introduce the new concept of PHY slicing over RISs towards 6G
wireless networks.

## Preliminaries

An RIS is composed of many sub-wavelength and conductive elements, and it can
be viewed as a programmable cluster with many scatterers. As a whole, an RIS
can be perceived as a signal processing system with multiple inputs and
outputs, including several parallel subsystems with reconfigurable transfer
functions.

One of the fundamental advantages of RISs compared to existing solutions such
as relaying and beamforming is their nearly passive nature. In other words,
ideally, a passive RIS does not consume power for RF signal processing and
signal manipulation. From a signals and systems perspective, the input-output
relationship of the $n$th element of an RIS can be written as
$y_{n}=p_{n}e^{j\phi_{n}}x_{n}$, where $x_{n}$ is the incoming signal, $y_{n}$
is the outgoing signal, and $\phi_{n}\in\left[0,2\pi\right)$ and
$p_{n}\in\left[0,1\right]$ stand for the adjustable phase and amplitude of the
$n$th RIS element, respectively. This phase adjustment can be implemented by
means of variable loads, delay lines, or phase shifters. An RIS with $N$
elements can be thought of as $N$ parallel systems, whose adjustable
parameters can be exploited for signal manipulation.

Signals incoming to an RIS are scattered to the propagation medium by the RIS
elements, which results in a product path loss for the end-to-end link between
the source and the destination. Specifically, the signals impinging on the RIS
might suffer a considerable path loss and they further attenuate along their
routes to their destination. Considering unit-gain (passive) RIS elements and
line-of-sight (LOS) dominated links, one can easily show that the maximized
received signal-to-noise ratio (SNR) for a passive RIS-empowered system
follows [4]

$\gamma_{\text{p}}\propto N^{2}\frac{P_{T}}{d_{SR}^{2}d_{RD}^{2}\sigma^{2}}.$
(1)

Here, $P_{T}$ is the transmit power, $\sigma^{2}$ is the receiver noise power,
and $d_{SR}$ and $d_{RD}$ respectively stand for source-RIS/RIS-destination
distances, where the source-destination link is assumed to be unavailable. Two
important conclusions can be drawn from (1). First, the received SNR increases
with a factor of $N^{2}$, which has been known since the early RIS studies
[2], thanks to the coherent combining of forward and backward links of the
RISs with intelligent phase adjustment. However, the received SNR decays with
a factor of $d_{SR}^{2}d_{RD}^{2}$, which considerably hurts the RIS-assisted
systems in longer distances111For clarity of presentation, we dropped higher-
order $\pi$ and wavelength $(\lambda$) terms in (1), while they cannot be
ignored in a complete link-budget analysis.. This is the reason why several
research teams in our community, including ours, have been trying to find
optimum RIS locations to boost their effect on the received signal. One
conclusion of this work is that, if there exists a strong direct link,
possibly with a LOS component that decays only with $d_{SD}^{2}$, where
$d_{SD}$ is the source-destination distance, it becomes very challenging to
find sweet spots to operate the RIS since its effect on the received SNR
dwindles remarkably. In other words, considering the fact that the direct link
SNR is given by $\gamma_{\text{d}}\propto P_{T}/(d_{SD}^{2}\sigma^{2})$, it
would be challenging to compete with it if the RIS size is not extremely large
and/or $d_{SR}$ and $d_{RD}$ are more than several tens of meters.
Consequently, most recent practical campaigns generally position RISs either
close to the transmitter or the receiver, to compensate this double path loss
effect [5]. Unfortunately, we cannot foresee an easy solution to this low end-
to-end path gain problem due to the underlying physics of RISs unless we
consider active ones.

In this article, we first introduce active RISs to overcome the double path
loss problem of fully passive RISs. Then we put our emphasis on more
sophisticated RIS architectures such as transmitter-type RISs, transmissive-
reflective RISs, and standalone RIS, which might exploit the active RIS
concept as well.

## Passive vs. Active RISs

The concept of active RISs has been recently introduced to overcome the
aforementioned received power limitation of RISs [6]. In this RIS
architecture, RIS elements, equipped with active RF electronics, induce a
power amplification factor $(p_{n}>1)$ for the impinging signals to boost the
outgoing signals. From a signals and systems perspective, we have a new input-
output relationship for this case:
$y_{n}=p_{n}e^{j\phi_{n}}x_{n}+p_{n}e^{j\phi_{n}}v_{n}$, where
$v_{n}\sim\mathcal{CN}(0,\sigma_{v}^{2})$ is the input noise amplified by the
$n$th active RIS element and thanks to the amplification, we have $p_{n}>1$.
Here, $\mathcal{CN}$ stands for a circularly symmetrical complex Gaussian
distribution.

Assuming a maximum RIS-reflect power of $P_{A}$ and RIS-induced noise power of
$\sigma_{v}^{2}$, the maximized received SNR can be obtained after simple
derivations as

$\gamma_{\text{a}}\propto
N\frac{P_{T}P_{A}}{P_{A}\sigma_{v}^{2}d_{SR}^{2}+\sigma^{2}d_{RD}^{2}+\sigma^{2}\sigma_{v}^{2}d_{SR}^{2}d_{RD}^{2}/N}.$
(2)

Here, thanks to an amplification that is dependent on source-RIS channels [6],
one can escape from the fundamental limitation of RISs, multiplicative path-
loss effect. This can be verified by a close inspection of (2) since the last
term in the denominator can be readily dropped due to multiplicative noise
variances. In other words, an active RIS can benefit from a superior path gain
at the level of specular reflection that only experiences additive path loss.

In light of this information, in Fig. 1, we compare the received SNR of
passive and active RIS-aided communication systems with the following
parameters: $P_{T}=P_{A}=20$ dBm, $\sigma^{2}=\sigma_{v}^{2}=-90$ dBm,
$N=256$, and for a carrier frequency of $5$ GHz. $xy$-coordinates of the
source and RIS are respectively selected as $(0,0)$, and $(10,20)$, and the
shown $12$ test points are given accordingly to assess the received SNR. For
comparison purposes only, the received SNR of specular reflection
$\gamma_{\text{s}}\propto P_{T}/((d_{SR}+d_{RD})^{2}\sigma^{2})$ is also
calculated and shown in Fig. 1 without stressing too much on its practicality
for the considered setup. We have the following observations: i) the received
SNR for passive RIS diminishes very quickly when we get away from the RIS, ii)
the active RIS not only provides less variation in received power but also
significantly better (around $40$ dB higher) SNR thanks to an optimized
amplification factor, which is calculated as $19.7$ dB for this specific
setup222Following [6], we calculated the optimum amplification factor as
$p_{1}=\cdots=p_{N}=\sqrt{\frac{P_{A}}{\sum_{n=1}^{N}\left|h_{n}\right|^{2}+\sigma_{v}^{2}}}$.
Here, $\left|h_{n}\right|$ is the magnitude of the $n$th source-RIS channel,
which is equal to $\lambda/(4\pi d_{SR})$ for all $n$ under the far-field and
LOS assumption. and system parameters, and iii) in accordance with the
amplification factor of the active RIS, it also provides approximately $20.6$
dB better SNR compared to the specular reflection333This can be also verified
by the ratio
$\gamma_{\text{a}}/\gamma_{\text{s}}\simeq\frac{NP_{A}(d_{SR}+d_{RD})^{2}}{(P_{A}d_{SR}^{2}+d_{RD}^{2})}\sim
N$ assuming $\sigma_{v}^{2}=\sigma^{2}$ and $N\gg 1$..

Figure 1: Comparison of passive RIS, specular reflection, and active RIS in
terms of received SNR in decibels.

In conclusion, the active RIS not only transforms the multiplicative path loss
of passive RISs into an additive form, but also benefits from an additional
amplification gain on top it, making active RIS systems even more more
powerful than specular reflection alone. A hybrid architecture comprising both
active and passive RIS elements might further improve the analysis and
optimization, and can provide a compromise between two solutions. The power
consumption associated with active RIS elements might be a concern for RISs
not connected to the power grid, and a detailed investigation of these issues
is beyond the scope of this article. We envision that an active RIS might
outperform an amplify-and-forward relay in terms of cost and efficiency, and
leave this interesting comparison to future studies.

## Relay- vs. Transmitter-Type RISs

The conceptual similarity between relaying and RIS-assisted systems has
attracted the attention of many researchers recently. While we believe that an
RIS provides many distinguishing features compared to a half- or full-duplex
relay, depending on its functionality, it can operate in either relay or
transmitter mode. As shown in Fig. 1, in relay mode, the RIS (whether active
or passive) is placed somewhere between the source and the destination to
assist the communication in between, most probably, when the direct link is
very weak or blocked. However, if we decide to follow the path of passive
RISs, a very promising direction for RISs might be transmitter-type operation,
where the RIS becomes a part of the transmitter and plays an active role in
signaling and modulation [7].

In a setup where the RIS is used in the transmitter mode, it can manipulate
the incoming unmodulated carrier signal to encode information bits. In other
words, by feeding the RIS with an unmodulated carrier, it is possible to
create virtual signal constellations over-the-air to convey information. This
can be verified from the RIS element’s basic signal model
$y_{n}=p_{n}e^{j\phi_{n}}x_{n}$, where with proper adjustments of $p_{n}$ and
$\phi_{n}$, virtual amplitude-phase modulation constellations can be created.
Following the popular terminology of modulations, notable examples include
spatial modulation, media-based modulation, and the more general index
modulation, transmitter-type RISs can be regarded as reflection modulation,
where we embed information into reflection states of an RIS. In principle,
this is conceptually a similar architecture to media-based modulation [8],
where different radiation patterns of a reconfigurable antenna is exploited to
transmit information, that is, to create a virtual signal constellation.

Two major advantages of transmitter-type RISs are summarized as follows.
First, the hardware architecture of an RIS-based transmitter would be much
simpler compared to a traditional one with upconverters and filters, while an
unmodulated cosine carrier signal can be generated very easily using an RF
digital-to-analog converter with an internal memory and a power amplifier.
Second, since the RIS is relatively close to the signal source, the
multiplicative path loss effect is restrained. One can even position the RIS
to lie in the near-field of the transmitter to further boost the received
signal power, however, taking into account near-field effects carefully.

To sum up, transmitter-type RISs might be a remedy to implement RF chain-free
transmitters and to mimic MIMO systems by transmitting multiple streams
simultaneously over the RIS with a relatively simple architecture. We refer
interested readers to the recent study of [9], and the references therein, for
practical transmitter-type RIS architectures developed by researchers from
Southeast University.

## Reflective Only vs. Transmissive-Reflective RISs

Is it possible to provide coverage on the back side of an RIS? The answer is
yes, thanks to the simultaneously transmitting and reflecting RISs [10, 11].
Our discussion so far has generally assumed that both the source and the
destination are on the same side of the RIS, that is, within the same half
space of the smart environment. This is a constraint stemming from the
physical architecture of the RIS, which is composed of a conductive substrate
that acts as a blocker for the signals hitting the surface, not allowing them
to pass through the surface. The concept of transmissive-reflective RISs
challenges this status quo by carefully reengineering the RIS structure to
allow it to simultaneously reflect and transmit (in principle, refract) the
incoming signals to provide a full $360^{\text{o}}$ coverage. This is achieved
by magnetic RIS elements that can support not only surface electric
polarization currents but also magnetic currents to simultaneously control the
refracted and reflected signals.

Early research on RISs envisioned them as objects hanging on walls or facades
of buildings. But, an RIS could also be positioned in the middle of a
communication environment, or embedded in a wall between two different
environments, to receive signals and forward them in different directions.
From a signal processing perspective, this type of RIS can be considered as a
single-input dual-output system, however, a number of problems need to be
solved in terms of system protocol design, such as mode, energy, and time
splitting. Other concerns include the complicated hardware architecture and
potentially unaesthetic appearance of such RISs, which might be the price to
be paid for a full $360^{\text{o}}$ coverage. Nevertheless, such surfaces can
provide effective solutions for outdoor-to-indoor coverage extension, multi-
room coverage, and realizing more effective NOMA 2.0 systems. It is worth
noting that the multiplicative path loss effect discussed earlier would still
be a concern with this type of design and might potentially limit the overall
coverage on both sides of the RIS. In this context, the design of an active
transmissive-reflective surface would be a notable leap forward for full-
coverage.

We conclude this section by quoting the authors of [10] for the comparison of
reflective only and transmissive-reflective RISs: biscuits placed on a metal
plate and ice cubes in a glass of water, which correspond to reflective RIS
elements on an opaque RIS substrate and magnetic RIS elements on a transparent
RIS substrate, respectively. Which will prevail in the following years? The
answer will be given in the following years considering the interesting trade-
offs that they offer in terms of coverage and complexity in smart
environments.

## Interconnected vs. Standalone RISs

Some key questions that arise is research on RISs include the following: How
might an RIS obtain knowledge of channel phases? How can the RIS adjust itself
in real-time? Is it possible to perform channel estimation at the RIS? How
realistic is it to assume fully-passive RISs? Where does the intelligence of
the RIS come from? We have to admit that these questions are not easy to
answer, but thanks to recent developments in the field, a number of
interesting directions might be followed to realize smart RISs for 6G wireless
networks.

A fully-passive but interconnected RIS has a dedicated communication link,
that is, a fully-functional RF chain, that receives critical information to be
forwarded to the microprocessor controlling the RIS. This link might be
between the RIS and the source, destination, or a central control unit and
acts as a guide for the RIS when adjusting its reflection state. On the other
hand, a standalone RIS is equipped with sparse sensors embedded among passive
RIS elements and has a number of RF chains for background baseband signal
processing to acquire knowledge about the wireless environment. A notable
example is given in [12], where the use of sparse sensors is introduced in the
first place by considering a hybrid RIS architecture with a number of elements
connected to the baseband. The use of deep learning tools at the RIS to train
it on how to interact with incoming signals is another very promising
direction for standalone RISs. Here, the considered deep learning model learns
how to map the observed environment descriptors (channels) to the optimal RIS
configurations.

We finally note that a detailed investigation of potential artificial
intelligence techniques for RIS-empowered systems might be the topic of
another article. It would not be surprising to see that future RISs will
leverage tools from artificial intelligence to adapt themselves in real time
without any user intervention.

## The Ultimate RIS Architecture and PHY Slicing

Our discussion so far has provided interesting perspectives by focusing on a
number of promising RIS architectures and making relative comparisons among
them. To provide the bigger picture with further insights, the considered RIS
architectures in this article are summarized in Fig. 2 with the pros and cons
of each of them, where we focus on passive vs. active RISs, relay- vs.
transmitter-type RISs, reflective-only vs. transmissive-reflective RISs, and
interconnected vs. standalone RISs, respectively. We believe that the future
research and development in this field will possibly focus on these
interesting RIS architectures, particularly the ones at the right-hand side of
Fig. 2, trying to unlock their true potential by exploring new applications
and low-cost designs. We also note that all four architectures at the left-
hand side of Fig. 2 are envisioned to have a passive nature initially, while
it is also possible to use the active RIS concept for transmitter-type and
standalone RISs, and potentially transmissive-reflective RISs by carefully
reengineering RIS elements.

Figure 2: Comparison of different RIS architectures with their pros and cons.

Against this background, we propose an ultimate RIS architecture shown in Fig.
3. In this RIS design, we envision a combination of the aforementioned RIS
architectures while assuming that these type of sophisticated RISs might be
constructed physically in the future. Here, the ultimate RIS has a hybrid
architecture with a mix of passive and active RIS elements, embedded with
sparse sensors for channel acquisition, and has a dedicated subsurface for
transmissive-reflective operation. To eliminate interference and/or improve
secrecy, this RIS might also switch to the absorption mode without reflecting
or transmitting any signal. In particular, the ultimate RIS can simultaneously
work in the following modes: active-reflect, active-transmit, passive-reflect,
passive-transmit, and absorb. The ultimate RIS might also collect sensory
information from its neighborhood and convey this data with transmitter-type
operation (modulation on-the-fly) if necessary. Advanced functionalities of
this ultimate RIS might include over-the-air Doppler/multipath mitigation,
channel equalization, and interference cancellation. Although the ultimate RIS
architecture might be perceived as a straightforward combination of existing
RIS technologies, the underlying signal processing tasks and communication
engineering problems would necessitate novel approaches. For instance, an
interesting optimization problem needs to be solved to determine the optimal
sizes of different subsurface slices. Furthermore, this RIS should be
carefully positioned considering the distribution of near, far, and back users
to achieve the maximum sum-rate.

Figure 3: The ultimate RIS architecture composed of different subsurfaces for
improved flexibility.

The ultimate RIS architecture shown in Fig. 3 might be exciting in its own
right, but it may further pave for a new concept towards 6G: PHY slicing.
Inspired by the 5G network slicing concept, RIS-empowered PHY slicing would
enable the multiplexing of different RIS architectures and/or applications on
the same physical RIS by dividing the RIS into subsurfaces. This RIS with PHY
slicing functionality might be fully reconfigurable, meaning that the sizes of
passive, active, and transmissive subsurfaces could be adjusted in real-time
according to the needs of the users in the network. Different applications,
such as data transfer, wireless charging, PHY security etc., could be
simultaneously supported by this ultimate RIS as well by slicing the RIS.

## Concluding Remarks

In this article, we have provided a general perspective on the future of RIS
technologies towards 6G wireless networks and put forward an ultimate RIS
architecture. We have also shown that an active RIS might be a feasible
solution to the low path gain problem of passive RISs by a quantitative
analysis in terms of received SNR. At this point, we can raise several
questions for future research: Can we ignore the power consumption of active
RISs, and more importantly, might the increasing electromagnetic exposure
stemming from the active components be a concern? Will an active RIS be more
cost effective than relays or small base stations? Who will control and
monitor RISs (users, vendors, operators?), particularly the standalone ones?
Do we need a single and sophisticated RIS or simpler and smaller ones employed
in large numbers? How many RISs are required then to cover a certain area? We
hope to find answers to these interesting questions in the near future.

We conclude that passive RISs might provide cost-effective solutions to
increase indoor coverage and overcome blockage problems, while active ones
would be more suitable for outdoor environments in which distances between
terminals are much larger and RIS power consumption might not be a major
concern. Transmitter-type RISs might be a remedy for future Internet-of-things
applications with their relatively simple transmitter architectures without
any RF chains. Finally, transmissive-reflective RISs may unlock the potential
of future NOMA systems by also providing outdoor-to-indoor coverage extension.

Clearly, the concept of RIS-empowered communication has brought a great deal
of excitement to our community recently and we will see whether it will be a
strong candidate towards 6G in the following years.

## Open Codes

The MATLAB script used to obtain the SNR values in Fig. 1 is given at
https://corelab.ku.edu.tr/tools/.

## Acknowledgment

The work of E. Basar was supported by the Scientific and Technological
Research Council of Turkey (TUBITAK)-COST project 120E401.

The work of H. V. Poor was supported by the U.S. National Science Foundation
under Grant CCD-1908308.

## References

  * [1] R. W. Heath, “Going toward 6G [From the editor],” _IEEE Signal Process. Mag._ , vol. 36, no. 3, pp. 3–4, May 2019.
  * [2] E. Basar, M. D. Renzo, J. de Rosny, M. Debbah, M.-S. Alouini, and R. Zhang, “Wireless communications through reconfigurable intelligent surfaces,” _IEEE Access_ , vol. 7, p. 116753–116773, Sep. 2019.
  * [3] C. Pan et al., “Reconfigurable intelligent surfaces for 6G systems: Principles, applications, and research directions,” _IEEE Commun. Mag._ , vol. 59, no. 6, pp. 14–20, May 2021.
  * [4] E. Basar and I. Yildirim, “Reconfigurable intelligent surfaces for future wireless networks: A channel modeling perspective,” _IEEE Wireless Commun._ , vol. 28, no. 3, pp. 108–114, June 2021.
  * [5] M. Dunna, C. Zhang, D. Sievenpiper, and D. Bharadia, “ScatterMIMO: Enabling virtual MIMO with smart surfaces,” in _Proc. 26th Ann. Int. Conf. Mobile Comput. Netw._ , London, UK, Apr. 2020.
  * [6] Z. Zhang, L. Dai, X. Chen, C. Liu, F. Yang, R. Schober, and H. V. Poor, “Active RIS vs. passive RIS: Which will prevail in 6G?” Mar. 2021. [Online]. Available: https://arxiv.org/abs/2103.15154v1
  * [7] E. Basar, “Transmission through large intelligent surfaces: A new frontier in wireless communications,” in _Proc. European Conf. Netw. Commun. (EuCNC 2019)_ , Valencia, Spain, June 2019.
  * [8] E. Basar, “Media-based modulation for future wireless systems: A tutorial,” _IEEE Wireless Commun._ , vol. 26, no. 5, pp. 160–166, Oct. 2019.
  * [9] J. Y. Dai, W. Tang, M. Z. Chen, C. H. Chan, Q. Cheng, S. Jin, and T. J. Cui, “Wireless communication based on information metasurfaces,” _IEEE Trans. Microw. Theory Tech._ , vol. 69, no. 3, pp. 1493–1510, Mar. 2021\.
  * [10] Y. Liu, X. Mu, J. Xu, R. Schober, Y. Hao, H. V. Poor, and L. Hanzo, “STAR: Simultaneous transmission and reflection for 360 coverage by intelligent surfaces,” Mar. 2021. [Online]. Available: https://arxiv.org/abs/2103.09104v2
  * [11] S. Zhang, H. Zhang, B. Di, Y. Tan, M. D. Renzo, Z. Han, H. V. Poor, and L. Song, “Intelligent omni-surface: Ubiquitous wireless transmission by reflective-transmissive metasurface,” _IEEE Trans. Wireless Commun. (to appear)_ , June 2021.
  * [12] A. Taha, M. Alrabeiah, and A. Alkhateeb, “Enabling large intelligent surfaces with compressive sensing and deep learning,” _IEEE Access_ , vol. 9, pp. 44 304–44 321, Mar. 2021.

Ertugrul Basar received his Ph.D. degree from Istanbul Technical University in
2013. He is currently an Associate Professor with the Department of Electrical
and Electronics Engineering, Koç University, Istanbul, Turkey and the director
of Communications Research and Innovation Laboratory (CoreLab). His primary
research interests include beyond 5G systems, index modulation, intelligent
surfaces, waveform design, and signal processing for communications. Dr. Basar
currently serves as a Senior Editor of IEEE Communications Letters and an
Editor of IEEE Transactions on Communications and Frontiers in Communications
and Networks. He is a Young Member of Turkish Academy of Sciences and a Senior
Member of IEEE.
---
H. Vincent Poor is the Michael Henry Strater University Professor at Princeton
University. His interests include information theory, machine learning and
networks science, and their applications in wireless networks, energy systems,
and related fields. Dr. Poor is a member of the National Academy of
Engineering and the National Academy of Sciences, and a foreign member of the
Chinese Academy of Sciences and the Royal Society. He received the Technical
Achievement and Society Awards of the IEEE Signal Processing Society in 2007
and 2011, respectively, and the IEEE Alexander Graham Bell Medal in 2017.
---