A Tutorial on 5G Positioning
Lorenzo Italiano, Graduate Student Member, IEEE,
Bernardo Camajori Tedeschini, Graduate Student Member, IEEE,
Mattia Brambilla, Member, IEEE,
Huiping Huang, Member, IEEE,

Monica Nicoli, Senior Member, IEEE,
Henk Wymeersch, Senior Member, IEEE

This work was partially funded by the European Union—NextGenerationEU under the National Sustainable Mobility Center CN00000023, Italian Ministry of University and Research (MUR) Decree n. 1033—17/06/2022 (Spokes 6 and 9), and under the MUR Decree n. 352—09/04/2022, by the Vinnova B5GPOS Project under Grant 2022-01640, and by the Swedish Research Council (VR grant 2022-03007).
L. Italiano, B. Camajori Tedeschini, and M. Brambilla are with Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, Milan, Italy.
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
M. Nicoli is with Dipartimento di Ingegneria Gestionale (DIG), Politecnico di Milano, Milan, Italy.
(e-mail: monica.nicoli@polimi.it).

H. Huang and H. Wymeersch are with the Department of Electrical Engineering, Chalmers University of Technology, Göteborg, Sweden. (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. V, NO. N, MONTH YEAR

The widespread adoption of the fifth generation (5G) of cellular networks has brought new opportunities for localization-based services.
High-precision positioning use cases and functionalities defined by the standards are drawing the interest of vertical industries. In the transition to the deployment, this paper aims to provide an in-depth tutorial on 5G positioning, summarizing the historical events that led to the standardization of cellular-based positioning, describing current and forthcoming releases of the Third Generation Partnership Project (3GPP) standard, and discussing about the major research trends.

This paper is intended to represent an exhaustive guide for researchers and practitioners by providing fundamental notions on wireless localization, comprehensive definitions of measurements and architectures, examples of algorithms, and details on simulation approaches.
Our approach aims to merge practical aspects of enabled use cases and related requirements with theoretical methodologies and fundamental bounds, allowing to understand the trade-off between system complexity and achievable, i.e., tangible, benefits of 5G positioning services.
We also discuss about current limitations to be resolved for delivering accurate positioning solutions.
We evaluate the performances of 3GPP Rel-16 positioning in outdoor and indoor environments, providing thorough analyses of the effect of changing the system configuration.

5G, positioning, localization, tutorial, simulation, beyond 5G, Release 16, architecture, PRS, SRS, algorithm, ray tracing.

§ CURRENT LIMITATIONS OF  5G  POSITIONING

In this paper, we highlighted the importance of cellular positioning, starting with a historical overview, outlining the major trends of
the research (Section <ref>), providing examples of measurements and algorithms (Section <ref> and detailing the latest standard for cellular positioning (Section <ref>)
with associated simulations and performance analyses (Section <ref>).
However, if technical concepts and architectures are well defined from a theoretical point of view, the practical implementation into commercial systems is still restrained.
The discussion in the following  sections is thus focused on current impairments that still limit a pervasive adoption of cellular positioning technologies.

§.§ Antenna position and orientation

Accurate cellular positioning strictly needs a precise knowledge of the true location of each antenna panel of a BS in terms of latitude, longitude, and altitude. At present, the information about the BS location is approximately known, e.g., with GNSS surveys, and typically, there are no indications on the exact positions of distributed panels: there is one location information for each BS. Considering that there are sites with non-co-located panels (possible distances of tens of meters between different panels), a lack of this information unavoidably introduces errors in time-based positioning measurements. It follows that precise surveys for each BS are needed to construct a reliable database of exact antenna positions, and this operation can be tedious, time-consuming, and complex due to (not so rare) impervious sites.

Similar to the antenna positions, it is also required to have precise information about its tilting to guarantee reliable angular information. Manual tilting measurements are subject to errors, and also, in this case, the operations can be risky and complex, even more than measuring the position. Of course, the antenna supports should not allow for rotations over the years, i.e., they should be resistant with respect to severe weather conditions.
Furthermore, accurate calibration procedures are requested to guarantee optimal performance of the antenna arrays at BS.

Lastly, an exact knowledge of the length of cabling from the antenna to the signal source generator (typically at the baseband unit) and the cabling material is required to precisely measure the signal TOF.

§.§ Synchronization error

While the recommendation for communication of ITU indicates a tolerable synchronization error of $\pm 1.5$ $\mu$s <cit.>, the requirements for positioning are more stringent.
As a matter of fact, a synchronization error of $\pm 3$ ns results into a positioning error of $\approx$1 m, and the upper bound for synchronization of $\pm1.5$ $\mu$s results into $\approx$450 m of undesired ranging error, which is clearly incompatible with most of positioning requirements (see Section <ref>) and would prevent precise cellular positioning services.
At present, 5G networks use GNSS-based synchronization or packet-based synchronization with IEEE 1588v2 PTP <cit.>, but these standards cannot provide an accuracy close to 1 ns.

Reaching a near-zero nanosecond error is challenging, but research demonstrates that fiber-based solutions such as the White Rabbit protocol <cit.> can reach synchronization error values of 1 ns or even less <cit.>.
Having a precisely synchronized 5G  network will ensure a common scanning of the time domain for all BS, which would exactly transmit in the allocated time slot, limiting the interference and avoiding introducing degrading effects on time-domain measurements due to clock drifts.

§.§ BS density

The foreseen density of 5G BS in urban scenarios is one BS every 200 m <cit.>. If having such a high number of BS increases the investment costs of operators, on the other side, it  brings a significant improvement on the cellular positioning use case, boosting the roll-out of commercial services. We demonstrated that it is possible to localize a UE with a single BS in LOS; thus, a high density of BS would minimize blind areas and NLOS conditions, allowing for a precise cellular positioning service to the users. Clearly, the coverage of a single BS would be limited to a few tens of meters, thus demanding the network to perform handover procedures quickly. The advantage of having close BS is that it facilitates the indoor/outdoor transition, guaranteeing a seamless positioning service.

§.§ Hardware availability

As of today, experimental activities on 5G positioning are slowed down by a lack of commercial-ready hardware allowing the extraction of physical level parameters.
As a matter of fact, current practical works mainly adopt modified commercial devices <cit.> or ad-hoc hardware <cit.> which rarely permit the exploitation of raw measurements.
So far, the only research paper that measures raw 5G TOF is <cit.>.
However, the expensive cost of the hardware and non-compact size, together with the not-so-easy accessibility and usability, produce an inevitable slowdown of the research and testing procedures.
The above limitations are valid for both FR1 and FR2 bands and are further exacerbated for the latter. This lack, which is going to be resolved soon due to the high push from industries, prevents a pervasive assessment of 5G positioning potentialities at mmWave and large bandwidths, which would unleash the rollout of advanced and precise cellular-based location services. The last desired feature is also limited by a restricted deployment of public mmWave BS.

§.§ Deployment of private networks

An additional notable issue pertains to the indoor 5G positioning domain and

it revolves around the current state of private networks. As of today, it is observed that private networks have not been widely integrated into industrial settings despite the positioning opportunities they hold (see Section <ref>). This deficiency in the deployment has prompted industries to seek alternative technologies to fulfill their specific connectivity and positioning requirements. One such alternative that has gained considerable attention is UWB technology, particularly in industrial facilities where precise positioning is requested for the automation of workflows <cit.>.

§ CONCLUSION

This tutorial paper on 5G positioning aims to serve as a trusted reference for understanding the potentialities and limitations of the latest cellular localization technology.
We covered a journey to explore the fundamental concepts, techniques, and challenges associated with 5G positioning, delving into the technical underpinnings of 5G networks and how they can enable accurate positioning.
After summarizing the transition from 1G to 4G, we detailed the 5G evolution across the releases of the 3GPP standard, and we explored the major research trends towards 6G.
We delved into an explanation of the 5G positioning system and its associated capabilities, as defined by current industry standards, and highlighted how the latest technological enhancements could bring new possibilities for the roll-out of commercial cellular positioning services.

This tutorial is designed to be a valuable resource not only for academic audiences but also for professionals and businesses operating in or considering entry into the market of positioning services. To this extent, we presented results from extensive simulations designed to assess the positioning performance in diverse settings, including outdoor and indoor environments. Several analyses have been conducted to motivate the adoption of 5G technology for industrial positioning, revealing its appeal for indoor applications while simultaneously highlighting the inherent current limitations in outdoor contexts.

The findings revealed the superior accuracy of ranging measurements compared to angle-based methods. Specifically, UL-AOA positioning can be susceptible to the multipath effect, although it is worth noting that the angle accuracy is significantly linked to the dimensions of the antenna array.

Moreover, integrating multipath detection techniques offers the potential to mitigate this influence by eliminating anomalous positioning estimations, yielding refined results.
The simultaneous utilization of angle and ranging measurements proves advantages for achieving precise positioning, particularly in areas characterized by a low density of BS. Additionally, we illustrated the methodology for conducting position estimation using a single BS, obtaining promising results.
Furthermore, tracking filters demonstrate their efficacy in environments characterized by multipath interference and limited measurement data, such as indoor and urban scenarios.
Compared to urban settings, more reliable outcomes are observed in restricted environments, such as industrial areas. This discrepancy may be attributed to several factors, including the proximity of BS to the user, the consistent presence of at least three BS in LOS, as well as the availability of larger bandwidth (100 vs 400 MHz).

Given the increasing demand for precise and reliable positioning in various applications, we can envision a promising future for 5G positioning technologies. The progress made in this field, as outlined in this tutorial, underscores the potential for transformative changes in various sectors. We hope that this tutorial serves as a valuable resource for researchers, engineers, and innovators, contributing to the continued evolution and widespread adoption of 5G positioning solutions, ultimately enhancing our daily lives and driving innovation across industries.

§ ACKNOWLEDGMENT

We acknowledge the MADE Competence Center Industry 4.0 for providing the 3D lidar scan of its environment, enabling us to conduct indoor simulations.

[]Lorenzo Italiano (Graduate Student Member, IEEE) obtained the B.Sc. degree in Computer Engineering (2020) from the University of Florence, Italy, where he maturated an interest in Internet of Things and cybersecurity topics. He proceeded with his studies at the Politecnico di Milano, Italy, in Computer Science and Engineering, with a focus on cybersecurity and signal processing, obtaining his M.Sc. degree (cum Laude) in October 2022.
In November 2022, Lorenzo started his Ph.D. career at the Politecnico di Milano. His research interests include 5G positioning, V2X communications, and blockchain technologies. He was a recipient of the Distinguished Paper Award at ICSOC 2022.

[]Bernardo Camajori Tedeschini (Graduate Student Member, IEEE) received the B.Sc. (Hons.) in Computer Science and M.Sc. (Hons.) degrees in Telecommunications Engineering from the Politecnico di Milano, Italy, in 2019 and 2021, respectively. From November 2021 he started as PhD fellow in Information Technology at Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano.
He is currently a visiting researcher with the Laboratory for Information & Decision Systems (LIDS) at the Massachusetts Institute of Technology (MIT), Cambridge, MA. His research interests include federated learning, machine learning and localization methods.
He was a recipient of the Ph.D. grant from the ministry of the Italian government Ministero dell'Istruzione, dell'Università e della Ricerca (MIUR) and the Roberto Rocca Doctoral Fellowship granted by MIT and Politecnico di Milano.

[]Mattia Brambilla (Member, IEEE) received the B.Sc. and M.Sc. degrees in telecommunication engineering and the Ph.D. degree (cum laude) in information technology from the Politecnico di Milano, in 2015, 2017, and 2021, respectively. He was a visiting researcher with the NATO Centre for Maritime Research and Experimentation (CMRE), La Spezia, Italy, in 2019. In 2021 he joined the faculty of Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB) at the Politecnico di Milano as Research Fellow. His research interests include signal processing, statistical learning, and data fusion for cooperative localization and communication. He was the recipient of the Best Student Paper Award at the 2018 IEEE Statistical Signal Processing Workshop.

[]Huiping Huang (Member, IEEE) is a postdoctoral researcher of the Communication Systems Group at Chalmers University of Technology, Gothenburg, Sweden. He received the Ph.D. degree from the Darmstadt University of Technology, Darmstadt, Germany, in 2023. Prior to that, he received his B.Eng. and M.Sc. degrees from Shenzhen University, Shenzhen, China, in 2015 and 2018, respectively, all in Electrical and Electronic Engineering.
His research interests lie in signal processing and wireless communications, with main focuses on direction-of-arrival estimation, source localization, channel estimation, robust adaptive beamforming, sparse array design, compressed sensing, and optimization theory with applications to radar, sonar, navigation, microphone array processing, and so on.

[]Monica Nicoli (Senior Member, IEEE) received the M.Sc. (Hons.) and Ph.D. degrees in communication engineering from Politecnico di Milano, Milan, Italy, in 1998 and 2002, respectively. She was a Visiting Researcher with ENI Agip, from 1998 to 1999, and Uppsala University, in 2001. In 2002, she joined Politecnico di Milano as a Faculty Member. She is currently an Associate Professor in telecommunications with the Department of Management, Economics and Industrial Engineering. Her research interests include signal processing, machine learning, and wireless communications, with emphasis on smart mobility and Internet of Things (IoT) applications. She was a recipient of the Marisa Bellisario Award, in 1999, and a co-recipient of the best paper awards of the IEEE Symposium on Joint Communications and Sensing, in 2021, the IEEE Statistical Signal Processing Workshop, in 2018, and the IET Intelligent Transport Systems journal, in 2014. She is an Associate Editor of the IEEE Transactions on Intelligent Transportation Systems. She has also served as an Associate Editor for the EURASIP Journal on Wireless Communications and Networking, from 2010 to 2017, and a Lead Guest Editor for the Special Issue on Localization in Mobile Wireless and Sensor Networks, in 2011.

[]Henk Wymeersch (Senior Member, IEEE) (S’01, M’05, SM’19) obtained
the Ph.D. degree in Electrical Engineering/Applied
Sciences in 2005 from Ghent University, Belgium.
He is currently a Professor of Communication Systems with the Department of Electrical Engineering at Chalmers University of Technology, Sweden. He
is also a Distinguished Research Associate with Eindhoven University of Technology. Before joining Chalmers, he was a postdoctoral researcher from
2005 until 2009 with the Laboratory for Information and Decision Systems at the Massachusetts Institute of Technology. Prof. Wymeersch served as Associate Editor for IEEE Communication Letters (2009-2013), IEEE Transactions on Wireless
Communications (since 2013), and IEEE Transactions on Communications (2016-2018). During 2019-2021, he was an IEEE Distinguished Lecturer with the Vehicular Technology Society. His current research interests include the convergence of communication and sensing, in a 5G and Beyond 5G context.


# Recurrent Off-policy Baselines for Memory-based Continuous Control

Zhihan Yang
Carleton College
Northfield, MN, USA
<EMAIL_ADDRESS>
&Hai Nguyen11footnotemark: 1
Northeastern University
Boston, MA, USA
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

When the environment is partially observable (PO), a deep reinforcement
learning (RL) agent must learn a suitable temporal representation of the
entire history in addition to a strategy to control. This problem is not
novel, and there have been model-free and model-based algorithms proposed for
this problem. However, inspired by recent success in model-free image-based
RL, we noticed the absence of a model-free baseline for history-based RL that
(1) uses full history and (2) incorporates recent advances in off-policy
continuous control. Therefore, we implement recurrent versions of DDPG, TD3,
and SAC (RDPG, RTD3, and RSAC) in this work, evaluate them on short-term and
long-term PO domains, and investigate key design choices. Our experiments show
that RDPG and RTD3 can surprisingly fail on some domains and that RSAC is the
most reliable, reaching near-optimal performance on nearly all domains.
However, one task that requires systematic exploration still proved to be
difficult, even for RSAC. These results show that model-free RL can learn good
temporal representation using only reward signals; the primary difficulty
seems to be computational cost and exploration. To facilitate future research,
we have made our PyTorch implementation publicly available111Code:
https://github.com/zhihanyang2022/off-policy-continuous-control.

##  1 Introduction

In recent years, deep off-policy reinforcement learning (RL) algorithms based
on learning the optimal $Q$-function is enjoying great success in fully
observable continuous control domains [24, 5, 9, 10]. These algorithms usually
achieve significantly better performance than deep on-policy RL methods given
the same number of environment interactions and can learn highly competent
policies on robotic locomotion tasks (e.g., MuJoCo [31] domains in [1]) where
the state and action spaces are large. Their popularity also comes from their
easy-to-implement update rules derived from the Bellman equation, and their
robustness to hyper-parameters.

In this paper, we attempt to extend their success to partially observable (PO)
domains. Continuous control under partial observability is an interesting
problem for two reasons. First, continuous action and partial observability
are common in real-world applications and often occur together, e.g., in
robotics. Second, PO domains are more challenging than fully observable
domains because, to succeed, an RL agent must learn a suitable temporal
representation of the entire observation-action history, in addition to a
strategy to control.

Continuous control under partial observability is not a novel problem, and
there have been model-free and model-based algorithms proposed for this
problem (see Section 2). However, we recognize the need for an open-source
implementation of baseline algorithms that (1) are easy to implement, (2) can
learn from the entire history, and (3) can learn stably and achieve good
performance on a diverse set of PO tasks. To fill in the first and second gap,
this paper describes a neural network architecture (see the rightmost of
Figure 1) that can be used to easily implement recurrent versions of DDPG,
TD3, and SAC (RDPG, RTD3, and RSAC), and draws connection to a state-of-the-
art image-based off-policy model-free algorithm DrQ [21] (see the middle of
Figure 1). This architecture allows the recurrent agents to take into account
the entire history $h_{t}=\\{o_{1:t},a_{1:t-1}\\}$ at each timestep $t$,
making it possible to learn arbitrary time dependencies. In addition, we adapt
the standard experience replay procedure for these recurrent agents. To our
knowledge, these agents are not available as is in popular RL repositories
[16, 30, 6, 33].

To fulfill the third gap, we evaluate RDPG, RTD3, and RSAC on short-term and
long-term PO domains. In addition, we investigate important design choices,
including the sharing of recurrent representation across actors and critics,
and the types of recurrent neural networks used. Our experiments show that
RDPG and RTD3 can surprisingly fail on some domains and that RSAC is the most
reliable, often reaching near-optimal performance on nearly all domains.
However, one task that requires systematic exploration still proved to be
difficult, even for RSAC. These results show that model-free RL can learn good
temporal representation using only reward signals; the primary difficulty
seems to be computational cost and exploration. In the end, we suggest future
research directions. Note that we do not intend to compare our algorithms to
other algorithms but show that one of our algorithms can achieve near-optimal
performance in several domains.

Figure 1: Comparison between architectures for standard, image-based, and
recurrent off-policy model-free RL. The convolutional neural networks (CNNs)
and recurrent neural networks (RNNs) can be trained end-to-end using gradients
from loss terms, just like the actor and critic networks.

##  2 Related work

There have been several recent works on learning memory-based policies in deep
RL for continuous control. Recurrent Deterministic Policy Gradient (RDPG) [15]
prepends recurrent layers to both the actor and critic networks of Deep
Deterministic Policy Gradient (DDPG) [24], and was able to solve a variety of
simple PO domains, including sensor integration and memory tasks. However,
since it is based on DDPG, it inevitably suffers from value over-estimation
and premature policy convergence, two problems addressed later by algorithms
such as Twin-delayed DDPG (TD3) [5] and Soft Actor-critic (SAC) [9]. Moreover,
it has not been publicly reproduced. To address value over-estimation and
investigate high-dimensional tasks, [26] combines TD3 and LSTM [17] into
TD3-LSTM. While TD3-LSTM shows good performance on high-dimensional sensor
integration tasks, its network architecture is designed for POMDPs that are
solvable with very short-term memory (e.g., 3-5 timesteps) and cannot be
efficiently extended. A concurrent work [29] also combines TD3 and LSTM, but
can be used with arbitrary memory length and is hence most similar to our
work.

While the aforementioned works learn the temporal representation and the
policy jointly, Variational Recurrent Model (VRM) [13] was proposed to learn
these two separately and showed better performance than its SAC-LSTM
implementation (see Appendix A.2). However, learning a separate model of the
environment is a complex task, and the investigated domains are exclusively
sensor integration tasks with either positional or velocity information and
dense rewards. In this sense, VRM is related to several model-based algorithms
proposed for image-based control tasks [23, 12, 11], since each image only
offers positional information and an optimal policy must integrate such
information across frames. However, learning from images entails a different
set of challenges and is beyond our scope.

While previous approaches all fall under off-policy RL, various other
approaches have been proposed for memory-based control in general. Several
approaches build on the concept of belief states [20], a distribution over the
state space given the entire history. Since the precise belief state is
computationally expensive to track for continuous domains, algorithms mostly
either work with small and discrete observation and action spaces (e.g., [28])
or only seek to approximate the belief (e.g., with particle filters [19],
[25]). While both approaches require the use of RNNs, [34] does not and
instead lets the policy read and set part of the state space as a form of
memory and train the policy via directed policy search. However, it uses a
linear policy and requires a well-defined latent space during training. Apart
from continuous control, memory-based control in discrete action spaces has
been explored mainly by adding recurrent layers to the $Q$-network [14, 35,
22] in DQN [27].

##  3 Background

###  3.1 Partially observable Markov Decision Processes (POMDP)

In real-world settings, an agent often can only have access to a reflection of
the underlying state of the environment due to noise, occlusions, limited
measurement resolutions, and so on. Under partial observability, a decision-
making problem can be effectively modeled as a POMDP [20]. We can formally
specify a finite-horizon POMDP by the tuple
$(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\Omega,\mathcal{O},H,\gamma)$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{T}(s_{t+1}|s_{t},a_{t})$ is the transition probability,
$\mathcal{R}(s_{t},a_{t},s_{t+1})$ is the reward function, $\Omega$ is the
observation space, $\mathcal{O}(o_{t}|s_{t+1},a_{t})$ is the observation
probability, $H$ is the horizon, and $\gamma\in[0,1]$ is the discount factor.
The goal is to learn a policy $\pi$ that maximizes the expected discounted
return for a finite horizon $H$ defined as
$\mathbf{E}_{\pi}[\sum_{t=0}^{H}\gamma^{t}r_{t}]$. To perform optimally in a
POMDP, an agent must condition its policy on the entire history of
observations and actions that it has seen so far. However, the size of the
history space increases exponentially with the length of the history.
Therefore, a recurrent neural network (RNN) is often used to summarize the
history, and the policy is then condition on the RNN’s fixed-size hidden
states.

###  3.2 Off-policy RL methods for fully-observable continuous control

#### Deep deterministic policy gradient (DDPG)

DDPG [24] is an extension of the tabular Q-learning to (1) continuous state
spaces and (2) continuous action spaces. To deal with (1), it uses a deep
neural network $Q_{\phi}$ to represent the $Q$-function, commonly known as the
critic. It replaces the exact maximization problem in target computation
(which is costly due to (2)) with approximation using an actor neural network
$\mu_{\theta}$, commonly known as the actor. During training, given a batch of
$M$ transitions $\\{(s_{i},a_{i},r_{i},s^{\prime}_{i},d_{i})\\}_{i=1}^{M}$
uniformly sampled from a replay buffer, DDPG performs

$\displaystyle\phi$
$\displaystyle\leftarrow\phi-\eta\nabla_{\phi}\frac{1}{M}\sum_{i=1}^{M}\left(\underbrace{Q_{\phi}(s_{i},a_{i})}_{\text{prediction}}-\underbrace{(r_{i}+\gamma(1-d_{i})Q_{\phi_{\text{targ}}}(s^{\prime}_{i},\mu_{\theta_{\text{targ}}}(s^{\prime}_{i})))}_{\text{target}}\right)^{2}$
(train critic) $\displaystyle\theta$
$\displaystyle\leftarrow\theta+\eta\nabla_{\theta}\frac{1}{M}\sum_{i=1}^{M}Q_{\phi}\left(s_{i},\mu_{\theta}\left(s_{i}\right)\right),$
(train actor)

where $\gamma$ is the discount factor, $\eta$ is the learning rate, and
$\phi_{\text{targ}}$ and $\theta_{\text{targ}}$ are slowly changing versions
of $\phi$ and $\theta$ obtained through polyak averaging.

#### Twin-delayed DDPG (TD3)

TD3 [5] addresses the issue of value over-estimation in DDPG. Specifically, in
DDPG, $\mu_{\theta_{\text{targ}}}$ may exploit222This is because
$\mu_{\theta}(s)$ is trained to maximize $Q_{\phi}(s,\cdot)$ and may exploit
its erroneously high values; $\mu_{\theta_{\text{targ}}}(s)$ and
$Q_{\phi_{\text{targ}}}(s,\cdot)$ are slowly changing versions of
$\mu_{\theta}(s)$ and $Q_{\phi}(s,\cdot)$, and hence may inherit this
property. erroneously high values of $Q_{\phi_{\text{targ}}}(s,\cdot)$ and
hence make the targets too large. As a result, bad actions at certain states
may end up having over-estimated $Q$-values. In response, TD3 introduces three
tricks. First, random noise is added to the output of
$\mu_{\theta_{\text{targ}}}(s)$; this is called target smoothing. Second,
$\mu_{\theta}$ is trained less frequently than $Q_{\phi}$; this is called
delayed policy update. Finally, an additional $Q$-network is added, and both
$Q$-networks are updated using the same target, computed using the minimum of
the target versions of themselves; this is called clipped double-$Q$ learning.
The actor is updated with respect to only one $Q$-network. TD3 significantly
out-performs DDPG on several dense-reward high-dimensional continuous-control
domains.

#### Soft Actor-Critic (SAC)

Unlike DDPG and TD3, SAC [9] was derived from Soft Policy Iteration instead of
the standard policy iteration. Nevertheless, SAC is similar to TD3 in two
important ways. First, SAC learns a stochastic actor network, so its next-
state actions used for target computation are sampled from a distribution,
similar to target smoothing in TD3. Second, SAC also maintains two
$Q$-networks, but uses both of them for training the actor network. However,
SAC does not use target actor network or delayed policy update. In practice,
TD3 and SAC have similar performance.

###  3.3 Recurrent neural networks

Recurrent neural networks (RNNs) can be best thought as a function that, at
each timestep $t$, takes in (1) the previous summary ($\text{summary}_{t-1}$)
and (2) some new information ($x_{t}$), and produces (1) some output
($\text{output}_{t}$) for subsequently layers and (2) some summary for the
next timestep ($\text{summary}_{t}$). Despite this simple setup, several
parametrizations have been proposed for learning this function effectively,
with the most popular ones being Vanilla RNN (VRNN) [4], Long Short-Term
Memory (LSTM) [17], and Gated Recurrent Unit (GRU) [3]. For more details,
please refer to Section A.1. Since RNNs are differentiable with respect to
their parameters, they can be trained using gradient descent in a procedure
known as Backpropagation Through Time (BPTT).

##  4 Architecture & algorithm

In this section, we describe the network architecture and algorithms that we
used to train RDPG stably. We outline RDPG instead of RTD3 or RSAC because
DDPG uses the least number of tricks for improving stability. Then, we
describe how it can be easily extended for RTD3 and RSAC.

#### Architecture

DDPG involves the interaction between an actor $\mu_{\theta}(s)$ and a critic
$Q_{\phi}(s,a)$. To adapt for POMDPs, recall that $h_{t}=(o_{1:t},a_{1:t-1})$
approximates $s_{t}$ in POMDPs. Therefore, we modify the actor to be
$\mu_{\theta}(h_{t})$ and the critic to be $Q_{\phi}(h_{t},a_{t})$. However,
it is in general not possible to pass $h_{t}$ as is since fully-connected
neural networks do not accept variable-sized inputs. To that end, we applied
RNNs to distill and summarize $h_{t}$ into a fixed-size hidden vector
$\hat{h}_{t}$. We also modified the standard experience replay to store
individual episodes instead of individual transitions.

#### Algorithm

Formally, each update step consists of the following three sub-steps. First,
one episode (a larger batch of episodes is used in practice) is sampled
uniformly from the replay buffer:

$\displaystyle\left(o_{1},\ldots,o_{T+1}\right),\left(a_{0},\ldots,a_{T}\right),\left(r_{1},\ldots,r_{T}\right),\left(d_{1},\ldots
d_{T}\right),$

where $d_{t}$ represents the episode termination flag and $a_{0}$ is the first
dummy action.

Second, using this episode, the hidden vectors can be computed by unrolling an
RNN over the sequence of observations and actions as follows ($\|$ denotes
concatenation):

$\displaystyle\hat{h}_{1}^{\phi},\ldots,\hat{h}_{T+1}^{\phi}$
$\displaystyle=\operatorname{RNN}_{\phi}\left(o_{1}\left\|a_{0},\ldots,o_{T+1}\right\|a_{T}\right)$
$\displaystyle\hat{h}_{1}^{\phi_{\text{targ
}}},\ldots,\hat{h}_{T+1}^{\phi_{\text{targ }}}$
$\displaystyle=\operatorname{RNN}_{\phi_{\text{targ
}}}\left(o_{1}||a_{0},\ldots,o_{T+1}\|a_{T}\right)$
$\displaystyle\hat{h}_{1}^{\theta},\ldots,\hat{h}_{T+1}^{\theta}$
$\displaystyle=\operatorname{RNN}_{\theta}\left(o_{1}||a_{0},\ldots,o_{T+1}||a_{T}\right)$
$\displaystyle\hat{h}_{1}^{\theta_{\text{targ
}}},\ldots,\hat{h}_{T+1}^{\theta_{\text{targ }}}$
$\displaystyle=\operatorname{RNN}_{\theta_{\text{targ
}}}\left(o_{1}||a_{0},\ldots,o_{T+1}\|a_{T}\right),$

where we used a separate RNN for each of the non-recurrent actor, the non-
recurrent critic, and their target networks.

Third, parameter updates to the recurrent actor and critic can be computed as
follows:

$\displaystyle\phi$
$\displaystyle\leftarrow\phi-\eta\nabla_{\phi}\left\\{\frac{1}{T}\sum_{t=1}^{T}\left(Q_{\phi}\left(\hat{h}_{t}^{\phi},a_{t}\right)-\left(r_{t}+\gamma\left(1-d_{t}\right)Q_{\phi_{\text{targ
}}}\left(\hat{h}_{t+1}^{\phi_{\text{targ }}},\mu_{\theta_{\text{targ
}}}\left(\hat{h}_{t+1}^{\theta_{\text{targ
}}}\right)\right)\right)\right)^{2}\right\\}$ $\displaystyle\theta$
$\displaystyle\leftarrow\theta+\eta\nabla_{\theta}Q_{\phi}\left(\hat{h}_{t}^{\phi},\mu_{\theta}\left(\hat{h}_{t}^{\theta}\right)\right)$

and $\phi_{\text{targ}}$ and $\theta_{\text{targ}}$ are updated using polyak
averaging. Unlike in Section 3.2, here $\phi$ denotes parameters of the entire
recurrent critic (the non-recurrent critic plus its corresponding RNN); the
same logic applies to $\theta$, $\phi_{\text{targ}}$, and
$\theta_{\text{targ}}$. Gradients flow through the hidden vectors and reach
the RNN parameters, allowing the entire architecture to be trained end-to-end.
For some environments, episodes have variable lengths, so masking is used.

#### Extending to RTD3 and RSAC

Unlike DDPG, TD3 and SAC use two critics instead of one. This difference is
preserved in their recurrent versions, and hence they both use three RNNs (not
counting the ones in target networks): one for the actor, two for the two
critics. Tricks such as clipped double-$Q$ learning and entropy bonus rewards
are directly ported from their non-recurrent versions.

##  5 Domains

(a) pendulum

(b) cartpole

(c) reacher

(d) watermaze

(e) push-r-bump

Figure 2: Domains to perform experiments. For watermaze (d), the agent is
illustrated as the green dot and the platform as the red circle.

###  5.1 Sensor integration tasks

Optimal control of physical systems might require integrating sensor inputs
over a few past timesteps due to noise or the difficulty of measuring
correctly all relevant physical quantities. Here, we use the
position/velocity-only versions of three fully-observable physical-control
domains (see a, b of Figure 2), similar to domains investigated in [15], [13],
and [26]. These domains are pendulum-swingup
($\dim(\mathcal{S})=3,\dim(\mathcal{A})=1$), cartpole-balance
($\dim(\mathcal{S})=5,\dim(\mathcal{A})=1$), and cartpole-swingup
($\dim(\mathcal{S})=5,\dim(\mathcal{A})=1$); they are generally considered to
be increasingly more difficult. They all have a maximum episode length $T$ of
$200$ and use a dense reward function. Their position-only versions remove all
entries of the observation related to velocity, and their velocity-only
versions remove all entries related to position. We also append the previous
action to the velocity-only observations, which we observed to improve the
performance of recurrent agents.

###  5.2 Memory tasks

Another important type of PO tasks are memory tasks, where the agent needs to
memorize (instead of simply integrate) some information over timesteps. We
evaluate our agents on two memory tasks: reacher-pomdp and watermaze. reacher-
pomdp (see Figure 2c) ($T=50$) modifies the original 2-joint reacher task from
OpenAI Gym [1] so that the location of the goal only appears for the first
timestep. This task provides dense rewards. watermaze (see Figure 2d)
($T=200$) is borrowed from [15]. Since the paper does not open-source its
domains, we re-implemented it with minor modifications. watermaze simulates a
task that is used to study spatial learning for rodents. In this task, the
agent (e.g., a mouse) is released at the center of a circular world. It is
rewarded by being on top of an invisible circular platform, which is randomly
positioned per episode. The agent can only observe the platform when being on
top of the platform, and is taken back to the center afterwards. To maximize
return, the agent must memorize the location of the platform to return to it
faster.

###  5.3 Active exploration task

In a PO world, it is very common for an agent to perform non-rewarding but
informative actions to be more certain about the world before performing
rewarding actions. As an example, we created a domain called push-r-bump (see
Figure 2e) ($T=50$), which is a continuous version of a similar domain in
[28]. In this domain, a robot controls the movement and the stiffness of a
finger. When the finger is soft, it can glide over bumps without moving them,
causing reactive angles on the finger. In contrast, when the finger is stiff
enough, it can move bumps. Any bump movement will terminate the current
episode and will bring a reward of $1$ if the right bump is pushed right for a
short distance and $-1$ otherwise. The robot can observe the finger’s position
and angle, but it does not observe the bumps’ locations. Starting each
episode, the position of the finger is initialized around the center, and the
two bumps are randomized to one of three possible cases (see Figure 3).
Therefore, an optimal agent must first use a soft finger to scan around and
localize the right bump, and finally push this bump to the right for a short
distance with a stiff finger to get a positive reward.

Figure 3: Three cases in push-r-bump; colors and numbers are for illustration
and are not observed.

##  6 Experiments & results

Unless otherwise noted, (1) RDPG, RTD3, and RSAC take in observations instead
of states, and (2) the default RNN used is LSTM. For return, we plot the mean
and min/max (1 standard deviation for other statistics) w.r.t. 5 seeds as in
[9]. Statistics are recorded once per 1k steps (1k environment steps and 1k
update steps). Each return recorded is the average return over 10 episodes
with deterministic actors. Training time is w.r.t. Nvidia’s RTX 2080 (16G).
Hyper-parameters are in Appendix A.3.

###  6.1 Performance on sensor integration tasks

From the 3rd and 4th column of Table 1, we see that RSAC is the only algorithm
to achieve similar performance at convergence as SAC-state on all PO domains.
Interestingly, RDPG has 2 failed seeds in pendulum-p and failed completely in
cartpole-swingup-p/v. It is perhaps surprising that RDPG can have failed seeds
on pendulum-p, because the MDP version of pendulum-p is arguably one of the
simplest continuous control domains.

It is well know that, in DDPG, the actor can exploit erroneously high values
in the critics and destabilize learning [5]. Since the actor in RDPG is
augmented with a two-layer RNN, the actor is more expressive and may hence be
a stronger exploiter of such errors. However, since RDPG can perform
reasonably well (despite one failed seed in cartpole-swingup) on the MDP
versions of the domains (see the second column of Table 1), we argue that this
source of exploitation, if exists, is not the primary cause for the algorithm
to diverge on PO variants. Instead, we hypothesize that partial observability
itself may open up more room for such exploitation, and is worthy of further
investigation. A supporting evidence is that RTD3 and RSAC have successful
seeds on cartpole-swingup-p/v, presumably because of their mechanisms designed
to address such exploitation.

RTD3 performs better than RDPG, and has similar performance at convergence as
RSAC on pendulum-p/v and cartpole-balance-p/v. However, RTD3 learns slower
than RSAC on pendulum-p/v, and has one failed seed in both cartpole-
swingup-p/v. Interestingly, RTD3-state also has two failed seeds in the MDP
version of cartpole-swingup, suggesting that this failing behavior may be due
to RTD3’s brittleness with more expressive actor and critics instead of
partial observability. Empirically, through visualizing the learned policies
in cartpole-swingup-p/v, we noticed that RTD3 can get stuck in a suboptimal
policy of quickly moving the cart to one side to briefly drag the pole to its
horizontal position (away from the downward vertical position) to gain some
reward.

As aforementioned, RSAC is the only algorithm that matches its non-recurrent
MDP performance on all PO tasks. Algorithmically, RSAC is highly similar to
RTD3 in terms of many design choices, except that its actor maximizes its
entropy in addition to the return. Entropy maximization can prevent premature
convergence of the actor, and has been shown to explore equally rewarding
avenues of behavior much better than simply adding random noise to a
deterministic policy [8]. In fact, exploration may be crucial to learning an
expressive temporal representation from scratch since better exploration leads
to a wider distribution of observations and rewards, which prevents over-
fitting due to limited experience. In practice, this may help RSAC to be less
affected by a large network and learn more consistently than RTD3 (e.g., by
escaping local optima).

|  |  |
---|---|---|---
|  |  |
|  |  |
Table 1: Performance on sensor integration tasks. DDPG, TD3, and SAC all learn
competent policies given the state. However, the performance of DDPG and TD3
degrade with LSTMs - wide error bands are due to splits between successful and
failed seeds.

###  6.2 Performance and exemplary trajectories on memory and active
exploration tasks

RDPG, RTD3, and RSAC all learn competent policies on reacher-pomdp (see Table
2), where dense rewards are available. In contrast, RSAC is the only algorithm
that performs well and consistently on watermaze, which offers only sparse
rewards and requires active exploration at the beginning of training. RSAC is
trained under the entropy maximization framework, potentially making the
policy explore better throughout training. We illustrate one RSAC seed in
Figure 4a and 4b; it learns an exploratory policy and can memorize the
platform position to return quickly within fewer timesteps.

In push-r-bump, no algorithm can consistently solve the task to 100% success
rate. Still, RDPG, RTD3, and RSAC all have successful seeds, and RTD3 and RSAC
have seeds that solve 1 or 2 cases (among the 3 mentioned in Section 5.3).
This is likely an issue of insufficient exploration, since systematic
exploration is required for the algorithms to solve all 3 cases correctly by
chance to obtain sparse positive rewards to start learning. We visualize 1
successful RTD3 seed on 1 case in Figure 4c, and the same seed on 2 other
cases in Appendix A.5 along with how RNN cell states evolve.

|  |  |  |
---|---|---|---|---
Table 2: Performance on memory and active exploration tasks. For push-r-bump,
we plot individual seeds’ success rates to show that neither RDPG, RTD3, nor
RSAC can consistently solve the task.

(a) Platform at $90^{\circ}$

(b) Platform at $180^{\circ}$

(c) Case 1 in Figure 3; events happen from left to right.

Figure 4: Visualization of learned policies. (a) (b) One RSAC seed trained on
watermaze. (c) One RTD3 seed trained on push-r-bump executing in one among
three cases.

###  6.3 Sharing a single recurrent representation across the actor and the
critics

Figure 5: RSAC-Share reduces training time.

In RSAC, there are 5 recurrent modules: 1 within the actor, 2 within the two
critics, and 2 within the two critic targets. In this setup, each update step
involves 5 rounds of RNN forward passes and 3 rounds of BPTTs. Since each
round of computation scales with $T$, it is natural to ask whether sharing a
single recurrent module across them (1 for all non-target networks, and 1 for
all target networks) could have similar performance as without sharing while
using less computation. We call this approach RSAC-Share. While it is
straightforward to optimize parameters of the single shared representation
w.r.t. both the actor and the two critic losses, these two kinds of losses
might have different scales and could lead to instability [15]. Additionally,
while this approach only involves 2 rounds of RNN forward passes, it still
requires 3 rounds of BPTTs.

Seeking further improvement, we borrow the insights from DrQ [21], which
trains SAC on image-based domains and shares a single convolutional neural
network (CNN) across an actor and two critics. In DrQ, the shared CNN
accumulates gradients from both critic losses; the gradients of the actor loss
w.r.t. the CNN’s parameters are ignored. Similarly, for RSAC-Share, we
optimize the parameters of the shared recurrent module w.r.t. the two critic
losses only. Intuitively, using gradients from the critic losses (instead of
the actor loss) helps the shared recurrent module learn a richer
representation, since, given a state, the critics must learn values for all
actions but the actor only needs to learn the maximizing action. Doing so also
reduces computation: the final version of RSAC-Share uses 2 rounds of RNN
forward passes and BPTTs per update step, halving the RNN computation in RSAC.

Our experiments show that RSAC-Share is on par with RSAC (see the first row in
Table 3), and significantly reduces training time (see Figure 5). RSAC-Share
is less stable on cartpole-swingup-v, but reaches better performance at
convergence on reacher-pomdp.

We have two interesting observations. Firstly, the Pearson correlation (see
Appendix A.4) of $Q$-values produced by the two critics is higher for RSAC-
Share (see the second row in Table 3), presumably because they share the same
recurrent representation. Such high correlation potentially makes clipped
double $Q$-learning in RSAC-Share ineffective, since the minimum of two highly
correlated random variables is almost equal to either one of the two.
Secondly, RSAC-Share learns higher $Q$-values than RSAC does (see the third
row in Table 3); this is likely due to the first observation, and that the
actor in RSAC-Share may be a stronger maximizer of the critics’ outputs, since
it directly acts on the critics’ recurrent representation. While these
observations suggest risks of value over-estimation, RSAC-Share has strong
empirical performance, and out-performs both RDPG and RTD3. Note that, for
both observations, $Q$-values are calculated from 10 episodes sampled from the
replay buffer.

|  |  |
---|---|---|---
|  |  |
|  |  |
Table 3: Performance of RSAC-Share against RSAC. The first row shows that
RSAC-Share is on par with RSAC in terms of return. The second row shows that
RSAC-Share has higher correlation between critics’ outputs. The third row
shows that RSAC-Share learn higher values.

###  6.4 Comparison among VRNN, LSTM, and GRU

In earlier experiments, we used LSTM as the recurrent representation since it
has shown to be hard to beat on a variety of tasks [7], and was used as the
recurrent component of several deep RL algorithms [14, 15, 13, 26]. In this
section, we compare RSAC-LSTM, RSAC-VRNN, and RSAC-GRU against each other. Our
experiments (see Table 4) show that RSAC-LSTM significantly out-performs RSAC-
VRNN and is on par with RSAC-GRU, indicating that using either LSTM or GRU as
the recurrent representation is an equally reasonable choice. Interestingly,
RSAC-GRU and RSAC-LSTM have different speeds of learning, but seem to achieve
similar asymptotic performance. RSAC-VRNN performed the worst, presumably due
to the well-known vanishing/exploding-gradient problem [18] when training VRNN
using long sequences.

|  |  |
---|---|---|---
Table 4: Performance of RSAC-LSTM, RSAC-VRNN, and RSAC-GRU. RSAC-LSTM and
RSAC-GRU are on par with each other, and both greatly out-performs RSAC-VRNN.

##  7 Conclusions

We present several recurrent off-policy baselines, tested on a diverse set of
continuous control tasks. Our experiments show that RSAC is the most reliable
algorithm out of the three baselines investigated RSAC, RDPG, and RTD3.
However, they still face difficulties when solving tasks which provide only
sparse rewards and require active exploration. Unfortunately, these two
characteristics are often common in interesting real-world POMDPs. Another
limitation is the inability to deal with too long episodes, inheriting from
known weakness of RNNs (e.g., long training time, difficulties learning long-
term dependencies). This suggests a potential research direction of using
transformers [32], which improve upon these weaknesses, as an alternative to
RNNs for solving POMDPs.

## References

  * Brockman et al. [2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Chihara and Hesterberg [2018] Laura M Chihara and Tim C Hesterberg.  _Mathematical statistics with resampling and R_.  John Wiley & Sons, 2018.
  * Cho et al. [2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Elman [1990] Jeffrey L Elman.  Finding structure in time.  _Cognitive science_ , 14(2):179–211, 1990.
  * Fujimoto et al. [2018] Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  _International Conference on Machine Learning_ , 2018.
  * Fujita et al. [2021] Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa.  Chainerrl: A deep reinforcement learning library.  _Journal of Machine Learning Research_ , 22(77):1–14, 2021.  URL http://jmlr.org/papers/v22/20-376.html.
  * Greff et al. [2016] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber.  Lstm: A search space odyssey.  _IEEE transactions on neural networks and learning systems_ , 28(10):2222–2232, 2016.
  * Haarnoja et al. [2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _International Conference on Machine Learning_ , pages 1352–1361. PMLR, 2017.
  * Haarnoja et al. [2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _International Conference on Machine Learning_ , 2018a.
  * Haarnoja et al. [2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018b.
  * Hafner et al. [2019a] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  _arXiv preprint arXiv:1912.01603_ , 2019a.
  * Hafner et al. [2019b] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In _International Conference on Machine Learning_ , pages 2555–2565. PMLR, 2019b.
  * Han et al. [2020] Dongqi Han, Kenji Doya, and Jun Tani.  Variational recurrent models for solving partially observable control tasks.  _International Conference on Learning Representations_ , 2020.
  * Hausknecht and Stone [2015] Matthew Hausknecht and Peter Stone.  Deep recurrent q-learning for partially observable mdps.  In _2015 aaai fall symposium series_ , 2015.
  * Heess et al. [2015] Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver.  Memory-based control with recurrent neural networks.  _Deep Reinforcement Learning Workshop, NeurIPS_ , 2015.
  * Hill et al. [2018] Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu.  Stable baselines.  https://github.com/hill-a/stable-baselines, 2018.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 1997.
  * Hochreiter et al. [2001] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al.  Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
  * Igl et al. [2018] Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson.  Deep variational reinforcement learning for pomdps.  In _International Conference on Machine Learning_ , pages 2117–2126. PMLR, 2018.
  * Kaelbling et al. [1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra.  Planning and acting in partially observable stochastic domains.  _Artificial Intelligence_ , 101(1-2):99–134, 1998\.
  * Kostrikov et al. [2020] Ilya Kostrikov, Denis Yarats, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  _arXiv preprint arXiv:2004.13649_ , 2020.
  * Lample and Chaplot [2017] Guillaume Lample and Devendra Singh Chaplot.  Playing fps games with deep reinforcement learning.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Lee et al. [2019] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  _arXiv preprint arXiv:1907.00953_ , 2019.
  * Lillicrap et al. [2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _International Conference on Machine Learning_ , 2015.
  * Ma et al. [2020] Xiao Ma, Peter Karkus, David Hsu, Wee Sun Lee, and Nan Ye.  Discriminative particle filter reinforcement learning for complex partial observations.  _arXiv preprint arXiv:2002.09884_ , 2020.
  * Meng et al. [2021] Lingheng Meng, Rob Gorbet, and Dana Kulić.  Memory-based deep reinforcement learning for pomdp.  _International Conference on Intelligent Robots and Systems_ , 2021\.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 2015.
  * Nguyen et al. [2020] Hai Nguyen, Brett Daley, Xinchao Song, Christopher Amato, and Robert Platt.  Belief-grounded networks for accelerated robot learning under partial observability.  _arXiv preprint arXiv:2010.09170_ , 2020.
  * Ni et al. [2021] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov.  Recurrent model-free rl is a strong baseline for many pomdps.  _arXiv preprint arXiv:2110.05038_ , 2021.
  * Raffin et al. [2020] Antonin Raffin, Ashley Hill, Maximilian Enerstus, Adam Gleave, Anssi Kanervisto, and Noah Dormann.  Stable Baselines3, 5 2020.  URL https://github.com/DLR-RM/stable-baselines3.
  * Todorov et al. [2012] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Weng et al. [2021] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Hang Su, and Jun Zhu.  Tianshou: A highly modularized deep reinforcement learning library.  _arXiv preprint arXiv:2107.14171_ , 2021.
  * Zhang et al. [2016] Marvin Zhang, Zoe McCarthy, Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Learning deep neural network policies with continuous memory states.  In _2016 IEEE international conference on robotics and automation (ICRA)_ , pages 520–527. IEEE, 2016.
  * Zhu et al. [2017] Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao.  On improving deep reinforcement learning for pomdps.  _arXiv preprint arXiv:1704.07978_ , 2017.

##  Appendix A Appendix

###  A.1 Popular instantiations of RNNs

#### Elman Network (EN)

EN [4], also known as the Vanilla RNN (VRNN), is one of the simplest forms of
RNN. EN uses a single vector $h$ to represent the summary (of all previous
inputs) and the output, and the parametrization used is

$\displaystyle h_{t}=\tanh(W_{ih}x_{t}+b_{ih}+W_{hh}h_{t-1}+b_{hh}),$

where $W$ and $b$ denote learnable weights and biases. EN suffers from the
vanishing-gradient problem [18], which hinders its ability to learn long-term
dependencies.

#### Long Short-term Memory (LSTM)

LSTM [17] was invented to tackle the vanishing-gradient problem that plagues
EN. Even it seems to be handcrafted and is complicated, it has been shown to
be hard to beat on a variety of tasks [7]. Unlike EN, it uses two vectors $h$
(the hidden state) and $c$ (the cell state) to represent the summary (of all
previous inputs), and only $h$ to represent the output; its parametrization is

$\displaystyle f_{t}$
$\displaystyle=\sigma\left(W_{if}x_{t}+b_{if}+W_{hf}h_{t-1}+b_{hf}\right)$
(forget gate) $\displaystyle i_{t}$
$\displaystyle=\sigma\left(W_{ii}x_{t}+b_{ii}+W_{hi}h_{t-1}+b_{hi}\right)$
(input gate) $\displaystyle\tilde{c}_{t}$
$\displaystyle=\tanh\left(W_{ig}x_{t}+b_{ig}+W_{hg}h_{t-1}+b_{hg}\right)$
(candidate cell state) $\displaystyle c_{t}$ $\displaystyle=f_{t}\odot
c_{t-1}+i_{t}\odot\tilde{c}_{t}$ (new cell state) $\displaystyle o_{t}$
$\displaystyle=\sigma\left(W_{io}x_{t}+b_{io}+W_{ho}h_{t-1}+b_{ho}\right)$
(output gate) $\displaystyle h_{t}$
$\displaystyle=o_{t}\odot\tanh\left(c_{t}\right),$ (output)

where $W$ and $b$ denote learnable weights and biases. The parametrization has
an intuitive appeal: the new cell state is the sum of the results of applying
the forget gate to the new cell state and applying the input gate to the
input.

#### Gated-recurrent Unit (GRU)

GRU [3] was motivated by LSTM but is simpler. Like EN, it uses a single vector
$h$ (the hidden state) to represent the summary (of all previous inputs) and
the output. Like LSTM, it computes the new hidden state by a weighted sum of
the old hidden state and a candidate hidden state. The key difference is two-
fold: how the candidate is computed, and the fact that GRU uses a single gate
to replace two separate input and forget gates. GRU’s parametrization is

$\displaystyle z_{t}$
$\displaystyle=\sigma\left(W_{iz}x_{t}+b_{iz}+W_{hz}h_{t-1}+b_{hz}\right)$
(forget/input gate) $\displaystyle r_{t}$
$\displaystyle=\sigma\left(W_{ir}x_{t}+b_{ir}+W_{hr}h_{t-1}+b_{hr}\right)$
$\displaystyle\tilde{h}_{t}$
$\displaystyle=\tanh\left(W_{ig}x_{t}+b_{ig}+r_{t}*\left(W_{hg}h_{t-1}+b_{hg}\right)\right)$
(candidate hidden state) $\displaystyle h_{t}$
$\displaystyle=\left(1-z_{t}\right)\tilde{h}_{t}+z_{t}*h_{t-1},$ (hidden state
& output)

where $W$ and $b$ denote learnable weights and biases.

###  A.2 Comparison to VRM and its baselines

In this section, we compare our RSAC agent to VRM [13] and its baselines: SAC-
LSTM and SLAC. SAC-LSTM is similar to RSAC, but uses several tricks to allow
for truncated BPTT so that it is more amendable to CPU training. While these
tricks are useful, they have not been investigated thoroughly in literature.
The version of SLAC used is a modification of the original SLAC algorithm [23]
to non-pixel observations. On a set of four low-dimensional domains open-
sourced by VRM authors, RSAC out-performs or is on par with all other
algorithms (see Table 5). To promote fair comparison, we used the original
code without any modifications, and plot the mean and standard error over 5
seeds as in the original paper.

|  |  |
---|---|---|---
Table 5: Performance of RSAC against VRM and its baselines.

###  A.3 Hyper-parameters

All our agents used hyper-parameters similar to a popular repository [30].

For non-recurrent agents, the actors and critics are 2-layer perceptrons with
hidden dimensions $(256,256)$ with ReLU as the intermediate activation. For
recurrent agents, two recurrent layers with hidden dimension $256$ are
prepended to the non-recurrent actors and critics; note that we consider the
recurrent layers as parts of the actors and critics.

For DDPG, TD3, and their recurrent versions, the action noise during training
is $0.1$. For TD3 and its recurrent version, the target noise is $0.5$ and the
noise clip is $0.2$. Note that these values are w.r.t. to a normalized action
space such that each dimension lies within $[-1,1]$. For SAC and its recurrent
version, the entropy regularizer $\alpha$ is initialized to $1$ and auto-tuned
as in [30], which implements [10].

For non-recurrent agents, the replay buffer has a capacity of 1 million
transitions and, for each network update, a batch of 100 transitions is
sampled. For recurrent agents, the replay buffer has a capacity of 5000
episodes and, for each network update, a batch of 10 episodes is sampled. The
discount factor $\gamma$ is $0.99$. The polyak-averaging $\rho$ coefficient is
$0.995$. The learning rate for both actors and critics is $3e-4$.

Non-recurrent algorithms take actions uniformly sampled from the action space
for the first 1k steps, and recurrent algorithms do the same for the first 10k
steps. During this period, returns are recorded but other statistics are
unavailable, so we do not plot them.

###  A.4 Pearson correlation coefficient

The correlation coefficient [2], or the Pearson correlation coefficient, is a
popular measure for the strength of a linear relationship between two random
variables. If only pairs of samples
$\mathcal{D}=\left\\{(x_{i},y_{i})\right\\}_{i=1}^{M}$ are available, we can
compute the sample-version of this measure by the formula

$\displaystyle
r_{\mathcal{D}}=\frac{\sum_{i=1}^{M}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{M}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i=1}^{M}(y_{i}-\bar{y})^{2}}},$

where $\bar{x}$ and $\bar{y}$ are the empirical means.

We illustrate the result of applying this formula to four example datasets
with decreasing relationship strength (see Figure 6). In this work, we used
$r$ under the assumption that $Q$-values produced by the two critics follow a
linear relationship. We believe that this assumption is reasonable since both
critics are updated using the same target, and hence have no reason to follow
any other relationship.

Figure 6: Pearson correlation for different example datasets.

###  A.5 Additional trajectories for push-r-bump

In this section, we show trajectories of the same RTD3 seed used in Figure 4c
on two other cases of push-r-bump; the trajectories for Case 2 and 3 are shown
in Figure 7 and 8 respectively. In all 3 cases, we observe that the agent
first checks for bump at position 2, and then checks for bump at position 3.
Intuitively, this behavior allows the agent to collect enough information to
differentiate among the 3 cases: Case 1 has a bump at position 2 but not 3,
Case 2 has a bump at both position 2 and 3, and Case 3 has a bump at position
3 but not 2.

We also visualize the evolution of cell state of the 2nd LSTM layer of the
same RTD3 seed for all 3 cases (see Figure 9). Cell states have 256 dimensions
and we chose to project them to 3 dimensions using Principal Component
Analysis (PCA). In this latent space, all trajectories share the same starting
location, since the cell state is initialized to zeros. Case 3 splits with
Case 1 and 2 at region A because the agent does not detect a bump at position
2. In region B, Case 2 terminates earlier than Case 1 because, in Case 2, the
agent can directly push the bump at position 3 while, in Case 1, the agent
does not detect a bump at position 3 and must return and push the bump at
position 2.

Figure 7: One RTD3 seed on Case 2 of push-r-bump. Figure 8: One RTD3 seed on
Case 3 of push-r-bump. Figure 9: Evolution of the cell state of the 2nd LSTM
layer (projected using PCA) of one RTD3 seed for all 3 cases in push-r-bump.
Event A and B correspond to events when the finger makes contact with bumps.
Axis scales are arbitrary.