# Co-Adaptation of Algorithmic and
Implementational Innovations in
Inference-based Deep Reinforcement Learning

Hiroki Furuta
The University of Tokyo
<EMAIL_ADDRESS>
&Tadashi Kozuno
University of Alberta
Tatsuya Matsushima
The University of Tokyo
&Yutaka Matsuo
The University of Tokyo
&Shixiang Shane Gu
Google Research

###### Abstract

Recently many algorithms were devised for reinforcement learning (RL) with
function approximation. While they have clear algorithmic distinctions, they
also have many implementation differences that are algorithm-independent and
sometimes under-emphasized. Such mixing of algorithmic novelty and
implementation craftsmanship makes rigorous analyses of the sources of
performance improvements across algorithms difficult. In this work, we focus
on a series of off-policy inference-based actor-critic algorithms – MPO, AWR,
and SAC – to decouple their algorithmic innovations and implementation
decisions. We present unified derivations through a single control-as-
inference objective, where we can categorize each algorithm as based on either
Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence
minimization and treat the rest of specifications as implementation details.
We performed extensive ablation studies, and identified substantial
performance drops whenever implementation details are mismatched for
algorithmic choices. These results show which implementation or code details
are co-adapted and co-evolved with algorithms, and which are transferable
across algorithms: as examples, we identified that tanh Gaussian policy and
network sizes are highly adapted to algorithmic types, while layer
normalization and ELU are critical for MPO’s performances but also transfer to
noticeable gains in SAC. We hope our work can inspire future work to further
demystify sources of performance improvements across multiple algorithms and
allow researchers to build on one another’s both algorithmic and
implementational innovations.111The implementation is available at
https://github.com/frt03/inference-based-rl.

##  1 Introduction

Deep reinforcement learning (RL) has achieved huge empirical successes in both
continuous [36, 18] and discrete [39, 26] problem settings with on-policy [52,
54] and off-policy [13, 22, 23] algorithms. Especially in the continuous
control domain, interpreting RL as probabilistic inference [60, 61] has
yielded many kinds of algorithms with strong empirical performances [34, 50,
51, 12, 29, 21, 16, 24].

Recently, there has been a series of off-policy algorithms derived from this
perspective for learning policies with function approximations [2, 22, 48].
Notably, Soft Actor Critic (SAC) [22, 23], based on a maximum entropy
objective and soft Q-function, significantly outperforms on-policy [52, 54]
and off-policy [36, 18, 13] methods. Maximum a posteriori Policy Optimisation
(MPO) [2], inspired by REPS [50], employs a pseudo-likelihood objective, and
achieves high sample-efficiency and fast convergence compared to the variety
of policy gradient methods [36, 54, 7]. Similar to MPO, Advantage Weighted
Regression (AWR) [48], and its variant, Advantage Weighted Actor-Critic (AWAC)
[42], also employ the pseudo-likelihood objective weighted by the exponential
of advantages, and reports more stable performance than baselines both in
online and offline [35] settings.

While these inference-based algorithms have similar derivations to each other,
their empirical performances have large gaps among them when evaluated on the
standard continuous control benchmarks, such as OpenAI Gym [6] or DeepMind
Control Suite [58]. Critically, as each algorithm has unique low-level
implementation or code design decisions – such as value estimation techniques,
action distribution for the policy, and network architectures – aside from
high-level algorithmic choices, it is difficult to exactly identify the causes
of these performance gaps as algorithmic or implementational.

In this paper, we first derive MPO, AWR, and SAC from a single objective
function, through either Expectation-Maximization (EM) or KL minimization,
mathematically clarifying the algorithmic connections among the recent state-
of-the-art off-policy actor-critic algorithms. This unified derivation allows
us to precisely identify implementation techniques and code details for each
algorithm, which are residual design choices in each method that are generally
transferable to other algorithms. To reveal the sources of the performance
gaps, we experiment with carefully-selected ablations of these identified
implementation techniques and code details, such as tanh-squashed Gaussian
policy, clipped double Q-learning [13], and network architectures.
Specifically, we keep the high-level algorithmic designs while normalizing the
implementation designs, enabling proper algorithmic comparisons. Our empirical
results successfully distinguish between highly co-adapted design choices and
no-co-adapted ones222We here regard as co-adaptation the indispensable
implementation or code decisions that do not stem directly from the conceptual
algorithmic development, but from empirical considerations.. We identified
that clipped double Q-learning and tanh-squashed policies, the sources of SoTA
performance of SAC, are highly co-adapted, specific to KL-minimization-based
method, SAC, and difficult to transfer and benefit in EM-based methods, MPO or
AWR. In contrast, we discover that ELU [8] and layer normalization [4], the
sources of SoTA performance of MPO, are transferable choices from MPO that
also significantly benefit SAC. We hope our work can inspire more future works
to precisely decouple algorithmic innovations from implementation or code
details, which allows exact sources of performance gains to be identified and
algorithmic researches to better build on one another.

##  2 Related Work

##### Inference-based RL algorithms

RL as probabilistic inference has been studied in several prior contexts [60,
61, 34, 11, 57, 45], but many recently-proposed algorithms [2, 22, 48] are
derived separately and their exact relationships are difficult to get out
directly, due to mixing of algorithmic and implementational details,
inconsistent implementation choices, environment-specific tunings, and
benchmark differences. Our work organizes them as a unified policy iteration
method, to clarify their exact mathematical algorithmic connections and tease
out subtle, but important, implementation differences. We center our analyses
around MPO, AWR, and SAC, because they are representative algorithms that span
both EM-based [49, 50, 43, 44, 1, 56, 42] and KL-control-based [59, 51, 12,
29, 21, 33, 32] RL and achieve some of the most competitive performances on
popular benchmarks [6, 58]. REPS [50], an EM approach, inspired MPO, AWR, and
our unified objective in Eq. 1, while Soft Q-learning [21], a practical
extension of KL control to continuous action space through Liu and Wang [38],
directly led to the development of SAC.

##### Meta analyses of RL algorithms

While many papers propose novel algorithms, some recent works focused on meta
analyses of some of the popular algorithms, which attracted significant
attention due to these algorithms’ high-variance evaluation performances,
reproducibility difficulty [9, 28, 65], and frequent code-level optimizations
[25, 63, 10, 3]. Henderson et al. [25] empirically showed how these RL
algorithms have inconsistent results across different official implementations
and high variances even across runs with the same hyper-parameters, and
recommended a concrete action item for the community – use more random seeds.
Tucker et al. [63] show that high performances of action-dependent baselines
[19, 17, 37, 20, 67] were more directly due to different subtle implementation
choices. Engstrom et al. [10] focus solely on PPO and TRPO, two on-policy
algorithms, and discuss how code-level optimizations, instead of the claimed
algorithmic differences, actually led more to PPO’s superior performances.
Andrychowicz et al. [3] describes low-level (e.g. hyper-parameter choice, and
regularization) and high-level (e.g. policy loss) design choices in on-policy
algorithms affects the performance of PPO by showing results of large-scale
evaluations.

In contrast to those prior works that mainly focus on a single family of on-
policy algorithms, PPO and TRPO, and evaluating their implementation details
alone, our work focuses on two distinct families of off-policy algorithms, and
more importantly, presents unifying mathematical connections among
independently-proposed state-of-the-art algorithms. Our experiments
demonstrate how some implementation choices in Table 1 and code details are
co-evolved with algorithmic innovations and/or have non-trivial effects on the
performance of off-policy inference-based methods.

---
Method | Algorithm | Implementation
$\pi_{q}$ update |  $\pi_{p}$ update | ${\mathcal{G}}$ |  ${\mathcal{G}}$ estimate | $\pi_{\theta}$
MPO | EM | Analytic + TR | SG + TR | $Q^{\pi_{p}}$ | Retrace(1) | $\pi_{p}=\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta}(s))$
AWR | EM | Analytic | Mixture + SG | $A^{\bm{\tilde{\pi}_{p}}}$ |  TD($\lambda$) | $\pi_{p}=\mathcal{N}(\mu_{\theta}(s),\Sigma)$
AWAC | EM | Analytic | Mixture + SG | $Q^{\pi_{p}}$ | TD(0) | $\pi_{p}=\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta})$
SAC | KL | SG | (Fixed to Unif.) | $Q^{\bm{\pi_{q}}}_{\textbf{soft}}$ | TD(0) + TD3 | $\pi_{q}=\text{Tanh}(\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta}(s)))$
PoWER | EM | Analytic | Analytic | $\eta\log Q^{\pi_{p}}$ | TD(1) | $\pi_{p}=\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta}(s))$
RWR | EM | Analytic | SG | $\eta\log r$ | – | $\pi_{p}=\mathcal{N}(\mu_{\theta}(s),\Sigma)$
REPS | EM | Analytic | $\pi_{q}$ | $A^{\pi_{p}}$ | TD(0) | $\pi_{p}=\text{Softmax}$
UREX | EM | Analytic | SG | $Q^{\pi_{p}}$ | TD(1) | $\pi_{p}=\text{Softmax}$
V-MPO | EM | Analytic + TR | SG + TR | $A^{\pi_{p}}$ |  $n$-step TD | $\pi_{p}=\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta}(s))$
TRPO | KL | TR | $\pi_{q}$ | $A^{\pi_{p}}$ | TD(1) | $\pi_{q}=\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta})$
PPO | KL | SG + TR | $\pi_{q}$ | $A^{\pi_{p}}$ | GAE | $\pi_{q}=\mathcal{N}(\mu_{\theta}(s),\Sigma_{\theta})$
DDPG∗ | KL | SG | (Fixed) | $Q^{\pi_{q}}$ | TD(0) | $\pi_{q}=\mu_{\theta}(s)$
TD3∗ | KL | SG | (Fixed) | $Q^{\pi_{q}}$ | TD(0) + TD3 | $\pi_{q}=\mu_{\theta}(s)$

Table 1:  Taxonomy based on the components of inference-based off-policy
algorithms: MPO [2], AWR [48], AWAC [42], SAC [22], and other algorithms (see
Appendix B). We follow the notation of Sec. 3 & 4. We characterize them with
the algorithm family (EM or KL), how policies are updated ($\pi_{q}$ and
$\pi_{p}$; SG stands for stochastic-gradient-based, and TR stands for trust-
region-based), the choice of ${\mathcal{G}}(\cdot)$, how ${\mathcal{G}}$ is
estimated, and the parameterization of the policy. While the advantage
function just can be interpreted as Q-function with baseline subtraction, we
explicitly write $A^{\pi}$ when the state-value function is parameterized, not
Q-function. (∗Note that DDPG and TD3 are not “inference-based”, but can be
classified as KL control variants.)

##  3 Preliminaries

We consider a Markov Decision Process (MDP) defined by state space
$\mathcal{S}$, action space $\mathcal{A}$, state transition probability
function
$p:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,\infty)$,
initial state distribution $p_{1}:\mathcal{S}\rightarrow[0,\infty)$, reward
function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and discount
factor $\gamma\in[0,1)$. Let $R_{t}$ denote a discounted return
$\sum_{u=t}^{\infty}\gamma^{u-t}r(s_{u},a_{u})$. We assume the standard RL
setting, where the agent chooses actions based on a parametric policy
$\pi_{\theta}$ and seeks for parameters that maximize the expected return
$\mathbb{E}_{\pi_{\theta}}[R_{1}]$. Value functions for a policy $\pi$ are the
expected return conditioned by a state-action pair or a state, that is,
$Q^{\pi}(s_{t},a_{t}):=\mathbb{E}_{\pi}[R_{t}|s_{t},a_{t}]$, and
$V^{\pi}(s_{t}):=\mathbb{E}_{\pi}[R_{t}|s_{t}]$. They are called the state-
action-value function (Q-function), and state-value function, respectively.
The advantage function [5] is defined as
$A^{\pi}(s,a):=Q^{\pi}(s,a)-V^{\pi}(s)$. The Q-function for a policy $\pi$ is
the unique fixed point of the Bellman operator ${\mathcal{T}}^{\pi}$ defined
by
${\mathcal{T}}^{\pi}Q(s,a)=r(s,a)+\int\pi(a^{\prime}|s^{\prime})p(s^{\prime}|s,a)Q(s^{\prime},a^{\prime})~{}ds^{\prime}da^{\prime}$
for any function $Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. We
denote a trajectory or successive state-action sequence as
$\tau:=(s_{1},a_{1},s_{2},a_{2},\dots)$. We also define an unnormalized state
distribution under the policy $\pi$ by
$d_{\pi}(s)=\sum_{t=1}^{\infty}\gamma^{t}p(s_{t}=s|\pi)$.

##### Inference-based Methods

For simplicity, we consider a finite-horizon MDP with a time horizon $T$ for
the time being. As a result, a trajectory $\tau$ becomes a finite length:
$\tau:=(s_{1},a_{1},\ldots,s_{T-1},a_{T-1},s_{T})$.

As in previous works motivated by probabilistic inference [34, 2], we
introduce to the standard graphical model of an MDP a binary event variable
${\mathcal{O}}_{t}\in\\{0,1\\}$, which represents whether the action in time
step $t$ is optimal or not. To derive the RL objective, we consider the
marginal log-likelihood $\log\Pr({\mathcal{O}}=1|\pi_{p})$, where $\pi_{p}$ is
a policy. Note that ${\mathcal{O}}=1$ means ${\mathcal{O}}_{t}=1$ for every
time step. As is well known, we can decompose this using a variational
distribution $q$ of a trajectory as follows:

$\displaystyle\log\Pr\left({\mathcal{O}}=1|\pi_{p}\right)$
$\displaystyle=\mathbb{E}_{q}\left[\log\Pr({\mathcal{O}}=1|\tau)-\log\frac{q(\tau)}{p(\tau)}+\log\frac{q(\tau)}{p(\tau|{\mathcal{O}}=1)}\right]$
$\displaystyle={\mathcal{J}}(p,q)+D_{KL}(q(\tau)\;||\;p(\tau|{\mathcal{O}}=1)),$

where
${\mathcal{J}}(p,q):=\mathbb{E}_{q}\left[\log\Pr({\mathcal{O}}=1|\tau)\right]-D_{KL}(q(\tau)\;||\;p(\tau))$
is the evidence lower bound (ELBO). Inference-based methods aim to find the
parametric policy which maximizes the ELBO ${\mathcal{J}}(p,q)$.

There are several algorithmic design choices for $q$ and
$\Pr({\mathcal{O}}=1|\tau)$. Since any $q$ is valid, a popular choice is the
one that factorizes in the same way as $p$, that is,

$p(\tau)=p(s_{1})\prod_{t}p(s_{t+1}|s_{t},a_{t})\pi_{p}(a_{t}|s_{t}),\qquad
q(\tau)=p(s_{1})\prod_{t}p(s_{t+1}|s_{t},a_{t})\pi_{q}(a_{t}|s_{t}),$

where $\pi_{p}$ is a prior policy, and $\pi_{q}$ is a variational posterior
policy. We may also choose which policy ($\pi_{p}$ or $\pi_{q}$) to
parameterize. As for $\Pr({\mathcal{O}}=1|\tau)$, the most popular choice is
the following one [21, 22, 2, 34, 48]:

$\Pr({\mathcal{O}}=1|\tau)\propto\exp\left(\textstyle\sum_{t=1}^{T}\eta^{-1}{\mathcal{G}}(s_{t},a_{t})\right),$

where $\eta>0$ is a temperature, and ${\mathcal{G}}$ is a function over
$\mathcal{S}\times\mathcal{A}$, such as an immediate reward function $r$,
Q-function $Q^{\pi}$, and advantage function $A^{\pi}$. While we employ
$\exp(\cdot)$ in the present paper, there are alternatives [46, 47, 55, 66]:
for example, Siegel et al. [55] and Wang et al. [66] consider an indicator
function $f:x\in\mathbb{R}\mapsto\mathbbm{1}[x\geq 0]$, whereas Oh et al. [46]
consider the rectified linear unit $f:x\in\mathbb{R}\mapsto\max\\{x,0\\}$.

Incorporating these design choices, we can rewrite the ELBO
${\mathcal{J}}(p,q)$ in a more explicit form as;

${\mathcal{J}}(\pi_{p},\pi_{q})=\sum_{t=1}^{T}\mathbb{E}_{q}\left[\eta^{-1}{\mathcal{G}}(s_{t},a_{t})-D_{KL}(\pi_{q}(\cdot|s_{t})\;||\;\pi_{p}(\cdot|s_{t}))\right].$

In the following section, we adopt this ELBO and consider its relaxation to
the infinite-horizon setting. Then, starting from it, we derive MPO, AWR, and
SAC.

##  4 A Unified View of Inference-based Off-Policy Actor-Critic Algorithms

In Sec. 3, we provided the explicit form of the ELBO
${\mathcal{J}}(\pi_{p},\pi_{q})$. However, in practice, it is difficult to
maximize it as the expectation $\mathbb{E}_{q}$ depends on $\pi_{q}$.
Furthermore, since we are interested in a finite-horizon setting, we replace
$\sum_{t\in[T]}\mathbb{E}_{q}$ with $\mathbb{E}_{d_{\pi}(s)}$. Note that the
latter expectation $\mathbb{E}_{d_{\pi}(s)}$ is taken with the unnormalized
state distribution $d_{\pi}$ under an arbitrary policy $\pi$. This is commonly
assumed in previous works [2, 22]. This relaxation leads to the following
optimization:

$\underset{\pi_{p},\pi_{q}}{\max}~{}{\mathcal{J}}(\pi_{p},\pi_{q})~{}\text{
s.t. }\int d_{\pi}(s)\int\pi_{p}(a|s)~{}dads=1~{}\text{and }\int
d_{\pi}(s)\int\pi_{q}(a|s)~{}dads=1\,,$ (1)

where
${\mathcal{J}}(\pi_{p},\pi_{q})=\mathbb{E}_{d_{\pi}(s)}[\eta^{-1}{\mathcal{G}}(s,a)-D_{KL}(\pi_{q}\;||\;\pi_{p})]$
. With this objective, we can regard recent popular SoTA off-policy
algorithms, MPO [2], AWR [48], and SAC [22] as variants of a unified policy
iteration method. The components are summarized in Table 1. We first explain
how these algorithms can be grouped into two categories of approaches for
solving Eq. 1, and then we elaborate additional implementation details each
algorithm makes.

###  4.1 Unified Policy Iteration: Algorithmic Perspective

Eq. 1 allows the following algorithmic choices: how or if to parameterize
$\pi_{p}$ and $\pi_{q}$; what optimizer to use for them; and if to optimize
them jointly, or individually while holding the other fixed. We show that the
algorithms in Table 1 can be classified into two categories: Expectation-
Maximization control (EM control) and direct Kullback-Leibler divergence
minimization control (KL control).

####  4.1.1 Expectation-Maximization (EM) Control

This category subsumes MPO [2] (similarly REPS [50]), AWR [48], and AWAC [42],
and we term it EM control. At high level, the algorithm non-parametrically
solves for the variational posterior $\pi_{q}$ while holding the parametric
prior $\pi_{p}=\pi_{\theta_{p}}$ fixed (E-Step), and then optimize $\pi_{p}$
holding the new $\pi_{q}$ fixed (M-Step). This can be viewed as either a
generic EM algorithm and as performing coordinate ascent on Eq. 1. We denote
$\theta_{p}$ and $\pi_{q}$ _after_ iteration $k$ of EM steps by
$\theta_{p}^{(k)}$ and $\pi_{q}^{(k)}$, respectively.

In E-step at iteration $k$, we force the variational posterior policy
$\pi_{q}^{(k)}$ to be close to the optimal posterior policy, i.e., the
maximizer of the ELBO ${\mathcal{J}}(\pi_{\theta_{p}^{(k-1)}},\pi_{q})$ with
respect to $\pi_{q}$. EM control converts the hard-constraint optimization
problem in Eq. 1 to solving the following Lagrangian,

$\displaystyle{\mathcal{J}}(\pi_{q},\beta$ $\displaystyle)=\int
d_{\pi}(s)\int\pi_{q}(a|s)\eta^{-1}{\mathcal{G}}(s,a)~{}dads$
$\displaystyle-\int
d_{\pi}(s)\int\pi_{q}(a|s)\log\frac{\pi_{q}(a|s)}{\pi_{\theta_{p}^{(k-1)}}(a|s)}~{}dads+\beta\left(1-\int
d_{\pi}(s)\int\pi_{q}(a|s)~{}dads\right).$ (2)

We analytically obtain the solution of Eq. 2,

$\pi_{q}^{(k)}(a|s)=Z(s)^{-1}\pi_{\theta_{p}^{(k-1)}}(a|s)\exp\left(\eta^{-1}{\mathcal{G}}(s,a)\right),$

where $Z(s)$ is the partition function.

In M-Step at iteration $k$, we maximize the ELBO
${\mathcal{J}}(\pi_{q}^{(k)},\pi_{p})$ with respect to $\pi_{p}$. Considering
the optimization with respect to $\pi_{p}$ in Eq. 1 results in forward KL
minimization, which is often referred to as a pseudo-likelihood or policy
projection objective,

$\underset{\theta_{p}}{\max}~{}\mathbb{E}_{d_{\pi}(s)\pi_{q}^{(k)}(a|s)}\left[\log\pi_{\theta_{p}}(a|s)\right]=\underset{\theta_{p}}{\max}~{}\mathbb{E}_{d_{\pi}(s)\pi_{\theta_{p}^{(k-1)}}(a|s)}\left[\frac{\log\pi_{\theta_{p}}(a|s)}{Z(s)}\exp\left(\eta^{-1}{\mathcal{G}}(s,a)\right)\right],$
(3)

where we may approximate
$Z(s)\approx\frac{1}{M}\sum^{M}_{j=1}\exp(\eta^{-1}{\mathcal{G}}(s,a_{j}))$
with $a_{j}\sim\pi_{\theta_{p}^{(k-1)}}(\cdot|s)$ in practice.

####  4.1.2 Direct Kullback-Leibler (KL) Divergence Minimization Control

In contrast to the EM control in Sec. 4.1.1, we only optimize the variational
posterior $\pi_{q}$ while holding the prior $\pi_{p}$ fixed. In this scheme,
$\pi_{q}$ is parameterized, so we denote it as $\pi_{\theta_{q}}$. This leads
to KL control [59, 62, 51, 12, 29, 21].

Equivalent to E-step in Sec. 4.1.1, we force the variational posterior policy
$\pi_{\theta_{q}}$ to be close to the optimal posterior policy, i.e, the
maximizer of ${\mathcal{J}}(\pi_{p},\pi_{q})$ with respect to $\pi_{q}$. The
difference is that instead of analytically solving
${\mathcal{J}}(\pi_{p},\pi_{q})$ for $\pi_{q}$, we optimize
${\mathcal{J}}(\pi_{p},\pi_{\theta_{q}})$ with respect to $\theta_{q}$, which
results in the following objective,

$\underset{\theta_{q}}{\max}~{}\mathbb{E}_{d_{\pi}(s)\pi_{\theta_{q}}(a|s)}\left[\eta^{-1}{\mathcal{G}}(s,a)-\log\frac{\pi_{\theta_{q}}(a|s)}{\pi_{p}(a|s)}\right].$

####  4.1.3 “Optimal” Policies of EM and KL Control

While we formulate EM and KL control in a unified framework, we note that they
have a fundamental difference in their definition of “optimal” policies. KL
control fixes $\pi_{p}$ and converges to a regularized-optimal policy in an
exact case [15, 64]. In contrast, EM control continues updating both $\pi_{p}$
and $\pi_{q}$, resulting in convergence to the standard optimal policy in an
exact case [51]. We also note that EM control solves KL control as a sub-
problem; for example, the first E-step exactly corresponds to a KL control
problem, except for the policy parameterization.

###  4.2 Unified Policy Iteration: Implementation Details

In this section, we explain the missing pieces of MPO, AWR, and SAC in Sec.
4.1. Additionally, we also describe the details of other algorithms [30, 49,
50, 41, 56, 36, 13, 52, 54] from the EM and KL control perspective. See
Appendix B for the details.

####  4.2.1 MPO

##### Algorithm

MPO closely follows the EM control scheme explained in Sec. 4.1.1, wherein
$\pi_{p}=\pi_{\theta_{p}}$ is parametric, and $\pi_{q}$ is non-parametric.

##### Implementation [$\pi_{q}$ Update]

This corresponds to the E-step. MPO uses a trust-region (TR) method.
Concretely, it replaces the reverse KL penalty (second term) in the Lagrangian
(Eq. 2) with a constraint and analytically solves it for $\pi_{q}$. As a
result, $\eta$ is also optimized during the training by minimizing the dual of
Eq. 2, which resembles REPS [50]:

$g(\eta)=\eta\epsilon+\eta\log\mathbb{E}_{d_{\pi}(s)\pi_{\theta_{p}^{(k-1)}}(a|s)}\left[\exp\left(\eta^{-1}Q^{\pi_{\theta_{p}^{(k-1)}}}(s,a)\right)\right].$

##### [$\pi_{p}$ Update]

This corresponds to the M-step. MPO uses a combination of Stochastic Gradient
(SG) ascent and trust-region method based on a forward KL divergence similarly
to TRPO [52]. Concretely, it obtains $\theta_{p}^{(k)}$ by maximizing the
objective (Eq. 3) with respect to $\theta_{p}$ subject to
$\mathbb{E}_{d_{\pi}(s)}[D_{KL}(\pi_{\theta_{p}^{(k-1)}}(\cdot|s)\;||\;\pi_{\theta_{p}}(\cdot|s))]\leq\varepsilon$.
MPO further decompose this KL divergence to two terms, assuming Gaussian
policies; one term includes only the mean vector of
$\pi_{\theta_{p}}(\cdot|s)$, and the other includes only its covariance
matrix. Abdolmaleki et al. [2] justify this as a log-prior of MAP estimation,
which is assumed as a second-order approximation of KL divergence.

##### [${\mathcal{G}}$ and ${\mathcal{G}}$ Estimate]

MPO uses the Q-function of $\pi_{\theta_{p}^{(k-1)}}$ as ${\mathcal{G}}$ in
the $k$-th E-step. It originally uses the Retrace update [40], while its
practical implementation [27] uses a single-step Bellman update.

##### [$\pi_{\theta}$]

For a parameterized policy, MPO uses a Gaussian distribution with state-
dependent mean vector and diagonal covariance matrix. The trust-region method
in MPO’s M-step heavily relies on this Gaussian assumption. Since a Gaussian
distribution has an infinite support, MPO has a penalty term in its policy
loss function that forces the mean of the policy to stay within the range of
action space.

####  4.2.2 AWR and AWAC

##### Algorithm

AWR slightly deviates from the EM control. In the M-step, AWR simply set
$Z(s)$ to $1$.

##### Implementation [$\pi_{q}$ Update]

This corresponds the E-step. AWR and AWAC analytically solves the Lagrangian
(Eq. 2). In contrast to MPO, they don’t use trust-region method.

##### [$\pi_{p}$ Update]

This corresponds to the M-step. At iteration $k$, AWR uses an average of all
previous policies
$\widetilde{\pi}_{p^{k}}:=\frac{1}{k}\sum_{j=0}^{k-1}\pi_{\theta_{p}^{(j)}}$
instead of $\pi_{\theta_{p}^{(k-1)}}$ (cf. Eq. 3). In practice, the average
policy $\widetilde{\pi}_{p^{k}}$ is replaced by samples of actions from a
replay buffer, which stores action samples of previous policies.

##### [${\mathcal{G}}$ and ${\mathcal{G}}$ Estimate]

AWR uses the advantage function of $\widetilde{\pi}_{p^{k}}$ as
${\mathcal{G}}$ in the $k$-th E-step, and learns the state-value function of
$\pi_{\theta_{p}^{(k-1)}}$ with TD($\lambda$) [53]. Due to its choice of
$\tilde{\pi}_{p^{k}}$, this avoids importance corrections [40]. In contrast,
AWAC estimates the advantage via the Q-function with TD($0$).

##### [$\pi_{\theta}$]

For a parameterized policy, both AWR and AWAC use a Gaussian distribution with
state-dependent mean vector and state-independent diagonal covariance matrix
(a constant one for AWR). As in MPO, they uses a penalty term to keep the mean
of the policy within the range of action space.

####  4.2.3 SAC

##### Algorithm

Contrary to MPO and AWR, SAC follows the KL control scheme explained in Sec.
4.1.2, wherein the variational posterior policy $\pi_{q}=\pi_{\theta_{q}}$ is
parameterized, and the prior policy $\pi_{p}$ is fixed to the uniform
distribution over the action space $\mathcal{A}$. SAC uses as ${\mathcal{G}}$
a soft Q-function:

$\displaystyle Q^{\pi_{\theta_{q}}}_{\text{soft}}(s_{t},a_{t})$
$\displaystyle:=r(s_{t},a_{t})+\gamma\mathbb{E}_{\pi_{\theta_{q}}}\left[V^{\pi_{\theta_{q}}}_{\text{soft}}(s_{t+1})\right],$
$\displaystyle V^{\pi_{\theta_{q}}}_{\text{soft}}(s_{t})$
$\displaystyle:=V^{\pi_{\theta_{q}}}(s_{t})+\mathbb{E}_{\pi_{\theta_{q}}}\left[\textstyle\sum_{u=t}^{\infty}\gamma^{u-t}\eta{\mathcal{H}}(\pi_{\theta_{q}}(\cdot|s_{t}))\middle|s_{t}\right],$

with ${\mathcal{H}}(\pi_{\theta_{q}}(\cdot|s_{t}))$ being
$-\mathbb{E}_{\pi_{\theta_{q}}}[\log\pi_{\theta_{q}}(a_{t}|s_{t})|s_{t}]$.

##### Implementation [$\pi_{q}$ and $\pi_{p}$ Update]

SAC performs stochastic gradient (SG) ascent updates of only $\pi_{q}$, where
the objective function is shown in Eq. 1. SAC has no operation equivalent to
M-step in MPO and AWR, since it keeps the prior policy $\pi_{p}$ to the
uniform distribution.

Similar to the temperature tuning in MPO, Haarnoja et al. [23] consider the
dual function of entropy-constrained Lagrangian and treat the temperature
$\eta$ as a Lagrange multiplier, which results in,

$g(\eta)=-\eta\bar{{\mathcal{H}}}-\eta\mathbb{E}_{d_{\pi}(s)\pi_{\theta_{q}}(a|s)}\left[\log\pi_{\theta_{q}}(a|s)\right],$

where $\bar{{\mathcal{H}}}$ is the target entropy to ensure that the entropy
of the policy should be larger than it.

##### [${\mathcal{G}}$ and ${\mathcal{G}}$ Estimate]

SAC uses $Q^{\pi_{\theta_{q}}}_{\text{soft}}$ as ${\mathcal{G}}$, and
TD(0)-like algorithm based on this soft Q-function. In policy evaluation,
clipped double Q-learning [13], which retains multiple Q-functions (typically
two) and takes the minimum, is employed to suppress the overestimation of Q
value.

##### [$\pi_{\theta}$]

The policy of SAC is a squashed Gaussian distribution: SAC first samples a
random variable $u$ from a Gaussian distribution
${\mathcal{N}}(\mu_{\theta}(s),\Sigma_{\theta}(s))$ with state-dependent mean
vector and diagonal covariance; then, it applies the tanh function to $u$ and
obtain the action $a\in[-1,1]^{|\mathcal{A}|}$. (If a dimension of the action
space is not $[-1,1]$, an appropriate scaling and shift by an affine function
is applied after the squashing.) Tanh squashing prevents out-of-bounds action.

###  4.3 Empirical Comparison

To evaluate the empirical performance of these off-policy algorithms (MPO,
AWR, AWAC, and SAC), we compare their performances on Open AI Gym MuJoCo
environments, namely, Hopper, Walker2d, HalfCheetah, Ant, Humanoid, and
Swimmer, following Haarnoja et al. [23]. We reproduce all algorithms based on
pytorch RL library [14], referring their original implementations [22, 48, 42,
27]. Figure 1 shows that SAC outperforms others in four environments (Hopper,
Walker2d, HalfCheetah, and Humanoid), while MPO in Ant and AWR in Swimmer
achieves the best performance. Generally, SAC seems to perform consistently
better. We extensively evaluate MPO, AWR, and SAC on the 28 tasks on DeepMind
Control Suite and 3 MuJoCo manipulation tasks. See Appendix C and D for the
details.

Figure 1: Benchmarking results on OpenAI Gym MuJoCo locomotion environments.
All methods are run with 10 random seeds. SAC seems to perform consistently
better.

---
| SAC (D) | SAC (S) | AWAC (D) | AWAC (S) | MPO (D) | MPO (S)
Hopper-v2 | $\bf{3013\pm 602}$ | $1601\pm 733$ | $2329\pm 1020$ | $2540\pm 755$ | $2352\pm 959$ | $2136\pm 1047$
Walker2d-v2 | $\bf{5820\pm 411}$ | $1888\pm 922$ | $3307\pm 780$ | $3662\pm 712$ | $4471\pm 281$ | $3972\pm 849$
HalfCheetah-v2 | $15254\pm 751$ | $\bf{15701\pm 630}$ | $7396\pm 677$ | $7226\pm 449$ | $12028\pm 191$ | $11769\pm 321$
Ant-v2 | $5532\pm 1266$ | $1163\pm 1326$ | $3659\pm 523$ | $3008\pm 375$ | $\bf{7179\pm 190}$ | $6584\pm 455$
Humanoid-v2 | $\bf{8081\pm 1149}$ | $768\pm 215$ | $5243\pm 200$ | $2738\pm 982$ | $6858\pm 373$ | $5709\pm 1081$
Swimmer-v2 | $114\pm 21$ | $\bf{143\pm 3}$ | $35\pm 8$ | $38\pm 7$ | $69\pm 29$ | $70\pm 40$

Table 2:  Ablation of Clipped Double Q-Learning. (D) denotes algorithms with
clipped double Q-learning, and (S) denotes without it. We test original SAC
(D), AWAC (D), MPO (S) and some variants; SAC without clipped double Q (S),
AWAC (S), and MPO with clipped double Q (D). SAC (S) beats SAC (D) in
HalfCheetah and Swimmer, while it fails in Hopper, Walker, Ant and Humanoid,
which implies that SAC (S) obtains a highly exploratory policy, since it fails
in termination environments. The learning curves are shown in Appendix F.

##  5 Evaluation on Implementational Choices

In Sec. 4.3, SAC shows notable results in most environments, while we revealed
that the derivation and formulation of those methods resemble each other. To
specify the effect of each implementational or code detail on the performance,
we experiment with extensive and careful one-by-one ablations on; (1) clipped
double Q-learning, (2) action distribution for the policy, (3) activation and
normalization, and (4) network size. The former two correspond to
implementation details (the choice of $\mathcal{G}$ estimate and the
parameterization of the policy), and the latter two correspond to code details
(see Appendix G for further experiments on other implementation details, such
as $\pi_{p}$ update or the choice of $\mathcal{G}$). We also conclude several
recommendations for the practitioners (Table 6).

###  5.1 Clipped Double Q-Learning

We hypothesize that clipped double Q-learning has a large effect on the
notable performance of SAC, and could be transferable in EM control methods.
To verify this, we test the effect of clipped double Q-learning. Instead of
AWR, here we evaluate AWAC since it uses Q-function. Table 2 shows that
single-Q SAC outperforms original one in HalfCheetah and Swimmer that do not
have the termination of the episode, while struggles to learn in Hopper,
Walker, Ant and Humanoid that have the episodic termination conditions. A
hypothesis is that single-Q SAC obtains a more exploratory policy due to
larger overestimation bias, which can help in environments where explorations
are safe, but hurt in environments where reckless explorations lead to
immediate terminations. In contrast, clipped double Q-learning does not help
MPO or AWAC as significantly as SAC; most results do not change or lead to
slight improvements over the originals. This suggests that clipped double
Q-learning might be a co-dependent and indispensable choice to KL control
methods.

##### Recommendation

Use clipped double Q-learning as a default choice, but you can omit it in EM
control methods or non-terminal environments, such as HalfCheetah or Swimmer.

###  5.2 Action Distribution for the Policy

Another hypothesis is that the tanh-squashed policy (last column in Table 1)
is an important and transferable design choice. We compare SAC without tanh
transform (with MPO action penalty instead) to the original one, which results
in drastic degradation (Table 3). This can be caused by the maximum entropy
objective that encourages maximizing the covariance. These observations
suggest that the high performance of SAC seems to highly depend on the
implementational choice of the policy distribution. In contrast, MPO and AWR
with tanh squashing don’t enjoy such large performance gain, or achieve worse
cumulative rewards. This also suggests that tanh-squashed policy might be
highly co-adapted in SAC and less transferable to EM control methods. Note
that for EM control, we must clip the actions to keep them within the supports
of distributions; $a\in[-1+\epsilon,1-\epsilon]^{|\mathcal{A}|}$. We found
that no clipping cause significant numerical instability. See Appendix H for
the details.

##### Recommendation

Use the original distributions for each algorithm. If you use the tanh-
squashed Gaussian in EM control methods, clip the action to be careful for the
numerical instability.

---
| SAC (w/) | SAC (w/o) | AWR (w/) | AWR (w/o) | MPO (w/) | MPO (w/o)
Hopper-v2 | $3013\pm 602$ | $6\pm 10$ | $2709\pm 905$ | $\bf{3085\pm 593}$ | $2149\pm 849$ | $2136\pm 1047$
Walker2d-v2 | $\bf{5820\pm 411}$ | $-\infty$ | $3295\pm 335$ | $4717\pm 678$ | $3167\pm 815$ | $3972\pm 849$
HalfCheetah-v2 | $\bf{15254\pm 751}$ | $-\infty$ | $3653\pm 652$ | $5742\pm 667$ | $9523\pm 312$ | $11769\pm 321$
Ant-v2 | $5532\pm 1266$ | $-\infty$ | $445\pm 106$ | $1127\pm 224$ | $2880\pm 306$ | $\bf{6584\pm 455}$
Humanoid-v2 | $\bf{8081\pm 1149}$ | $108\pm 82^{\dagger}$ | $2304\pm 1629^{\dagger}$ | $5573\pm 1020$ | $6688\pm 192$ | $5709\pm 1081$
Swimmer-v2 | $114\pm 21$ | $28\pm 11$ | $121\pm 3$ | $\bf{128\pm 4}$ | $110\pm 42$ | $70\pm 40$

Table 3:  Ablation of Tanh transformation (†numerical error happens during
training). We test SAC without tanh squashing, AWR with tanh, and MPO with
tanh. SAC without tanh transform results in drastic degradation of the
performance, which can be caused by the maximum entropy objective that
encourages the maximization of the covariance. In contrast, EM Control methods
don’t enjoy the performance gain from the tanh-squashed policy, which seems a
less transferable choice. The learning curves are shown in Appendix F.

###  5.3 Activation and Normalization

The implementation of MPO [27] has some code-level detailed choices; ELU
activation [8] and layer normalization [4]. We hypothesize that these,
sometimes unfamiliar, code details in practical implementation stabilize the
learning process, and contribute to the performance of MPO most. Especially in
Ant (Sec. 4.3), MPO significantly outperforms SAC. To investigate this much
deeper, we add them into SAC or AWR and remove them from MPO, while
maintaining the rest of the implementations. Table 4 shows that layer
normalization can contribute to a significantly higher performance of SAC in
Ant, and replacing ReLU with ELU also improves performance a lot in Swimmer,
where AWR is the best in Figure 1. In contrast, the performance of MPO
drastically collapsed when we just removed ELU and layer normalization. This
observation suggests that these code-level choices are not only indispensable
for MPO, but transferable and beneficial to both KL and EM control methods.
Additionally, we tested incorporating ELU and layer normalization to SAC in 12
DM Control tasks where MPO outperformed SAC, and observed that they again
often benefit SAC performances substantially. See Appendix C for the details.

##### Recommendation

It is worth considering to replace the activation function from ReLU to ELU
and incorporate the layer normalization, to achieve the best performance in
several tasks. Both of them are transferable between EM and KL control
methods.

---
| Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
SAC | $3013\pm 602$ | $\bf{5820\pm 411}$ | $15254\pm 751$ | $5532\pm 1266$ | $8081\pm 1149$ | $114\pm 21$
SAC-E+ | $2337\pm 903$ | $5504\pm 431$ | $\bf{15350\pm 594}$ | $6457\pm 828$ | $\bf{8196\pm 892}$ | $\bf{146\pm 7}$
SAC-L+ | $2368\pm 179$ | $5613\pm 762$ | $13074\pm 2218$ | $\bf{7349\pm 176}$ | $8146\pm 470$ | $99\pm 18$
SAC-E+L+ | $1926\pm 417$ | $5751\pm 400$ | $12555\pm 1259$ | $7017\pm 132$ | $7687\pm 1385$ | $143\pm 9$
MPO | $2136\pm 1047$ | $3972\pm 849$ | $11769\pm 321$ | $6584\pm 455$ | $5709\pm 1081$ | $70\pm 40$
MPO-E- | $2700\pm 879$ | $3553\pm 1145$ | $11638\pm 664$ | $5917\pm 702$ | $4870\pm 1917$ | $108\pm 28$
MPO-L- | $824\pm 250$ | $2413\pm 1352$ | $6064\pm 4596$ | $2135\pm 2988$ | $5039\pm 838$ | $-\infty$
MPO-E-L- | $843\pm 168$ | $1708\pm 663$ | $-1363\pm 20965$ | $807\pm 2351$ | $5566\pm 787$ | $-\infty$
AWR | $3085\pm 593$ | $4717\pm 678$ | $5742\pm 667$ | $1127\pm 224$ | $5573\pm 1020$ | $128\pm 4$
AWR-E+ | $1793\pm 1305$ | $4418\pm 319$ | $5910\pm 754$ | $2288\pm 715$ | $6708\pm 226$ | $128\pm 4$
AWR-L+ | $2525\pm 1130$ | $4900\pm 671$ | $5391\pm 232$ | $639\pm 68$ | $5962\pm 376$ | $129\pm 2$
AWR-E+L+ | $\bf{3234\pm 118}$ | $4906\pm 304$ | $6081\pm 753$ | $2283\pm 927$ | $6041\pm 270$ | $130\pm 1$

Table 4:  Incorporating ELU/layer normalization into SAC and AWR. E+/L+
indicates adding, and E-/L- indicates removing ELU/layer normalization.
Introducing layer normalization or ELU into SAC improves the performances in
Ant (beating MPO), Swimmer (beating AWR), HalfCheetah, and Humanoid. AWR also
shows the improvement in several tasks. MPO removing layer normalization
largely drops its performance. Both code-level details seems transferable
between EM and KL control methods.

###  5.4 Network Size

While on-policy algorithms such as PPO and TRPO can use the common network
architecture as in prior works [10, 25], the architecture that works well in
all the off-policy inference-based methods is still not obvious, and the RL
community does not have agreed upon default choice. To validate the dependency
on the networks among the inference-based methods, we exchange the size and
number of hidden layers in the policy and value networks. We denote the
network size of MPO ((256, 256, 256) for policy and (512, 512, 256) for value)
as large (L), of SAC ((256, 256) for policy and value) as middle (M), and of
AWR ((128, 64) for policy and value) as small (S) (see Appendix A for the
details). Table 5 illustrates that SAC with large network seems to have better
performance. However, as shown in Appendix F (Figure 8), the learning curve
sometimes becomes unstable. While the certain degree of robustness to the
network size is observed in SAC, EM control methods, MPO and AWR, seem fragile
and more dependent on the specific network size. This trend is remarkable in
MPO. AWR (M) or (L) also struggles to learn in high-dimensional state tasks,
such as Ant (111 dim), or Humanoid (376 dim). The results also imply that, in
contrast to on-policy methods [3], the network size in the off-policy
inference-based methods seems a less transferable choice.

##### Recommendation

For SAC, use medium size network. Large size will also work, but the learning
curve might be unstable. For MPO, we strongly recommend to stick to large
size, because it is very sensitive to the network size. For AWR, using small
size is a better choice, especially in high-dimensional state tasks, such as
Ant, or Humanoid.

---
| Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
SAC (L) | $2486\pm 746$ | $3188\pm 2115$ | $\bf{16528\pm 183}$ | $\bf{7495\pm 405}$ | $\bf{8255\pm 578}$ | $118\pm 26$
SAC (M) | $3013\pm 602$ | $\bf{5820\pm 411}$ | $15254\pm 751$ | $5532\pm 1266$ | $8081\pm 1149$ | $114\pm 21$
SAC (S) | $\bf{3456\pm 81}$ | $4939\pm 284$ | $12241\pm 400$ | $3290\pm 691$ | $7724\pm 497$ | $59\pm 11$
MPO (L) | $2136\pm 1047$ | $3972\pm 849$ | $11769\pm 321$ | $6584\pm 455$ | $5709\pm 1081$ | $70\pm 40$
MPO (M) | $661\pm 79$ | $1965\pm 1426$ | $-\infty$ | $5192\pm 538$ | $6015\pm 771$ | $81\pm 28$
MPO (S) | $430\pm 99$ | $2055\pm 990$ | $5003\pm 1567$ | $3587\pm 957$ | $4745\pm 1428$ | $59\pm 28$
AWR (L) | $3221\pm 193$ | $4688\pm 648$ | $4360\pm 542$ | $35\pm 43$ | $665\pm 54$ | $\bf{133\pm 3}$
AWR (M) | $2816\pm 910$ | $4826\pm 547$ | $5538\pm 720$ | $413\pm 117$ | $3849\pm 1647$ | $\bf{133\pm 2}$
AWR (S) | $3085\pm 593$ | $4717\pm 678$ | $5742\pm 667$ | $1127\pm 224$ | $5573\pm 1020$ | $128\pm 4$

Table 5:  The performance of each algorithm with different network size. (S)
stands for the small network size from AWR, (M) for the middle network size
from SAC, and (L) for the large network size from MPO. Generally, SAC, a KL
control method seems more robust to the network size than EM control methods;
MPO and AWR.

---
| MPO | AWR | SAC
Clipped Double Q [5.1] | $\triangle$ | $\triangle$ | $\bigcirc$
Tanh Gaussian [5.2] | $\triangle$ | $\triangle$ | $\bigcirc$
ELU & LayerNorm [5.3] | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Large Network [5.4] | $\bigcirc$ | $\triangle$ | $\bigcirc$
Medium Network [5.4] | $\times$ | $\triangle$ | $\bigcirc$
Small Network [5.4] | $\times$ | $\bigcirc$ | $\triangle$

Table 6:  Intuitive summary of the ablations. $\bigcirc$ stands for
indispensable choice, $\bigcirc$ stands for recommended choice, $\triangle$
stands for not much different or worse choice than expected, and $\times$
stands for un-recommended choice. See Recommendation for the details.

##  6 Conclusion

In this work, we present a taxonomy of inference-based algorithms, and
successfully identify algorithm-specific as well as algorithm-independent
implementation details that cause substantial performance improvements. We
first reformulated recent inference-based off-policy algorithms – such as MPO,
AWR and SAC – into a unified mathematical objective and exhaustively clarified
the algorithmic and implementational differences. Through precise ablation
studies, we empirically show that implementation choices like tanh-squashed
distribution and clipped double Q-learning are highly co-adapted to KL control
methods (e.g. SAC), and difficult to benefit in EM control methods (e.g. MPO
or AWR). As an example, the network architectures of inference-based off-
policy algorithms, especially EM controls, seem more co-dependent than on-
policy methods like PPO or TRPO, which therefore have significant impacts on
the overall algorithm performances and need to be carefully tuned per
algorithm. Such dependence of each algorithmic innovation on specific hand-
tuned implementation details makes accurate performance gain attributions and
cumulative build-up of research insights difficult. In contrast, we also find
that some code-level implementation details, such as ELU and layer
normalization, are not only indispensable choice to MPO, but also transferable
and beneficial to SAC substantially. We hope our work can encourage more works
that study precisely the impacts of algorithmic properties and empirical
design choices, not only for one type of algorithms, but also across a broader
spectrum of deep RL algorithms.

### Acknowledgements

We thank Yusuke Iwasawa, Masahiro Suzuki, Marc G. Bellemare, Ofir Nachum, and
Sergey Levine for many fruitful discussions and comments. This work has been
supported by the Mohammed bin Salman Center for Future Science and Technology
for Saudi-Japan Vision 2030 at The University of Tokyo (MbSC2030).

## References

  * Abdolmaleki et al. [2018a] Abbas Abdolmaleki, Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Dan Belov, Nicolas Heess, and Martin Riedmiller.  Relative entropy regularized policy iteration.  _arXiv preprint arXiv:1812.02256_ , 2018a.
  * Abdolmaleki et al. [2018b] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller.  Maximum a posteriori policy optimisation.  In _International Conference on Learning Representations_ , 2018b.
  * Andrychowicz et al. [2021] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem.  What matters for on-policy deep actor-critic methods? a large-scale study.  In _International Conference on Learning Representations_ , 2021.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Baird III [1993] Leemon C Baird III.  Advantage updating.  Technical report, WRIGHT LAB WRIGHT-PATTERSON AFB OH, 1993.
  * Brockman et al. [2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Ciosek and Whiteson [2020] Kamil Ciosek and Shimon Whiteson.  Expected policy gradients for reinforcement learning.  _Journal of Machine Learning Research_ , 2020.
  * Clevert et al. [2016] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  In _International Conference on Learning Representations_ , 2016.
  * Duan et al. [2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel.  Benchmarking deep reinforcement learning for continuous control.  In _International Conference on Machine Learning_ , 2016.
  * Engstrom et al. [2019] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry.  Implementation matters in deep rl: A case study on ppo and trpo.  In _International Conference on Learning Representations_ , 2019.
  * Fellows et al. [2019] Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson.  Virel: A variational inference framework for reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Fox et al. [2016] Roy Fox, Ari Pakman, and Naftali Tishby.  Taming the noise in reinforcement learning via soft updates.  In _Conference on Uncertainty in Artificial Intelligence_ , 2016.
  * Fujimoto et al. [2018] Scott Fujimoto, Herke van Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International Conference on Machine Learning_ , 2018.
  * Fujita et al. [2021] Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa.  Chainerrl: A deep reinforcement learning library.  _Journal of Machine Learning Research_ , 2021.
  * Geist et al. [2019] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin.  A theory of regularized Markov decision processes.  In _International Conference on Machine Learning_ , 2019.
  * Ghasemipour et al. [2020] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu.  A divergence minimization perspective on imitation learning methods.  In _Conference on Robot Learning_ , 2020.
  * Grathwohl et al. [2017] Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud.  Backpropagation through the void: Optimizing control variates for black-box gradient estimation.  _arXiv preprint arXiv:1711.00123_ , 2017.
  * Gu et al. [2016] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine.  Continuous deep q-learning with model-based acceleration.  In _International Conference on Machine Learning_ , 2016.
  * Gu et al. [2017a] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, and Sergey Levine.  Q-Prop: Sample-efficient policy gradient with an off-policy critic.  In _International Conference on Learning Representations_ , 2017a.
  * Gu et al. [2017b] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, Bernhard Schölkopf, and Sergey Levine.  Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2017b.
  * Haarnoja et al. [2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _International Conference on Machine Learning_ , 2017.
  * Haarnoja et al. [2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , 2018a.
  * Haarnoja et al. [2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018b.
  * Hafner et al. [2020] Danijar Hafner, Pedro A Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess.  Action and perception as divergence minimization.  _arXiv preprint arXiv:2009.01791_ , 2020.
  * Henderson et al. [2017] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters.  In _AAAI Conference on Artificial Intelligence_ , 2017.
  * Hessel et al. [2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In _AAAI Conference on Artificial Intelligence_ , 2018.
  * Hoffman et al. [2020] Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas.  Acme: A research framework for distributed reinforcement learning.  _arXiv preprint arXiv:2006.00979_ , 2020.
  * Islam et al. [2017] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup.  Reproducibility of benchmarked deep reinforcement learning tasks for continuous control.  _arXiv preprint arXiv:1708.04133_ , 2017.
  * Jaques et al. [2017] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck.  Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control.  In _International Conference on Machine Learning_ , 2017.
  * Kober and Peters [2008] Jens Kober and Jan Peters.  Policy search for motor primitives in robotics.  In _Advances in neural information processing systems_ , 2008.
  * Kumar et al. [2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Kuznetsov et al. [2020] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov.  Controlling overestimation bias with truncated mixture of continuous distributional quantile critics.  In _International Conference on Machine Learning_ , 2020.
  * Lee et al. [2020] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Levine [2018] Sergey Levine.  Reinforcement Learning and Control as Probabilistic Inference: Tutorial and review.  _arXiv preprint arXiv:1805.00909_ , 2018.
  * Levine et al. [2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Lillicrap et al. [2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  In _International Conference on Learning Representations_ , 2016.
  * Liu et al. [2017] Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu.  Action-depedent control variates for policy optimization via stein’s identity.  _arXiv preprint arXiv:1710.11198_ , 2017.
  * Liu and Wang [2016] Qiang Liu and Dilin Wang.  Stein variational gradient descent: A general purpose bayesian inference algorithm.  _arXiv preprint arXiv:1608.04471_ , 2016.
  * Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Munos et al. [2016] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare.  Safe and efficient off-policy reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2016.
  * Nachum et al. [2017] Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans.  Improving policy gradient by exploring under-appreciated rewards.  In _International Conference on Learning Representations_ , 2017.
  * Nair et al. [2020] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine.  Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Neumann et al. [2011] Gerhard Neumann et al.  Variational inference for policy search in changing situations.  In _International Conference on Machine Learning_ , 2011.
  * Norouzi et al. [2016] Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans, et al.  Reward augmented maximum likelihood for neural structured prediction.  In _Advances In Neural Information Processing Systems_ , 2016.
  * O’Donoghue et al. [2020] Brendan O’Donoghue, Ian Osband, and Catalin Ionescu.  Making sense of reinforcement learning and probabilistic inference.  In _International Conference on Learning Representations_ , 2020.
  * Oh et al. [2018] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee.  Self-imitation learning.  In _International Conference on Machine Learning_ , 2018.
  * Okada and Taniguchi [2019] Masashi Okada and Tadahiro Taniguchi.  Variational inference mpc for bayesian model-based reinforcement learning.  In _Conference on Robot Learning_ , 2019.
  * Peng et al. [2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.  Advantage-Weighted Regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * Peters and Schaal [2007] Jan Peters and Stefan Schaal.  Reinforcement learning by reward-weighted regression for operational space control.  In _Proceedings of the 24th international conference on Machine learning_ , 2007.
  * Peters et al. [2010] Jan Peters, Katharina Mülling, and Yasemin Altün.  Relative entropy policy search.  In _AAAI Conference on Artificial Intelligence_ , 2010.
  * Rawlik et al. [2012] Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar.  On stochastic optimal control and reinforcement learning by approximate inference.  In _International Joint Conference on Artificial Intelligence_ , 2012\.
  * Schulman et al. [2015] John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel.  Trust region policy optimization.  In _International Conference on Machine Learning_ , 2015.
  * Schulman et al. [2016] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  In _International Conference on Learning Representations_ , 2016.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Siegel et al. [2020] Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin A. Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  In _International Conference on Learning Representations_ , 2020.
  * Song et al. [2020] H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Riedmiller, and Matthew M. Botvinick.  V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control.  In _International Conference on Learning Representations_ , 2020.
  * Tang and Kucukelbir [2020] Yunhao Tang and Alp Kucukelbir.  Hindsight expectation maximization for goal-conditioned reinforcement learning.  _arXiv preprint arXiv:2006.07549_ , 2020.
  * Tassa et al. [2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Todorov [2006] Emanuel Todorov.  Linearly-solvable markov decision problems.  In _Advances in Neural Information Processing Systems_ , 2006.
  * Todorov [2008] Emanuel Todorov.  General duality between optimal control and estimation.  In _IEEE Conference on Decision and Control_ , 2008.
  * Toussaint [2009] Marc Toussaint.  Robot trajectory optimization using approximate inference.  In _International Conference on Machine Learning_ , 2009.
  * Toussaint and Storkey [2006] Marc Toussaint and Amos Storkey.  Probabilistic inference for solving discrete and continuous state markov decision processes.  In _International conference on Machine learning_ , 2006.
  * Tucker et al. [2018] George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine.  The mirage of action-dependent baselines in reinforcement learning.  In _International Conference on Machine Learning_ , 2018.
  * Vieillard et al. [2021] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist.  Leverage the average: an analysis of kl regularization in rl.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Wang et al. [2019] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba.  Benchmarking model-based reinforcement learning.  _arXiv preprint arXiv:1907.02057_ , 2019.
  * Wang et al. [2020] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas.  Critic regularized regression.  _arXiv preprint arXiv:2006.15134_ , 2020.
  * Wu et al. [2018] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel.  Variance reduction for policy gradient with action-dependent factorized baselines.  _arXiv preprint arXiv:1803.07246_ , 2018.
  * Wu et al. [2019] Yifan Wu, George Tucker, and Ofir Nachum.  Behavior Regularized Offline Reinforcement Learning.  _arXiv preprint arXiv:1911.11361_ , 2019.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] We added the failure
cases of ablations that end up the insufficient coverage and the unclear
insights in Appendix H.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
Actually, we conducted exhaustive evaluations through the enormous
experiments, which might lead to force the future research to spend much
computing resources. We hope our empirical observations and recommendations
help the practitioners to explore the explosive configuration space.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We open-source the codebase at https://github.com/frt03/inference-based-rl.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Appendix A.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] Our experimental results were averaged
among 10 random seeds.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
A.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] We cited
the authors of the codebase in Sec. 4.3.

    2. (b)

Did you mention the license of the assets?

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

## Appendix

##  Appendix A Network Architectures

In this section, we describe the details of the network architectures used in
Sec. 4 and 5.

We mainly used 4 GPUs (NVIDIA V100; 16GB) for the experiments in Sec. 4 and 5
and it took about 4 hours per seed (in the case of 3M steps). Actually, we
conducted exhaustive evaluations through the enormous experiments, and we hope
our empirical observations and recommendations help the practitioners to
explore the explosive configuration space.

---
Architecture | MPO | AWR | AWAC | SAC
Policy network | (256, 256, 256) | (128, 64) | (256, 256) | (256, 256)
Value network | (512, 512, 256) | (128, 64) | (256, 256) | (256, 256)
Activation function | ELU | ReLU | ReLU | ReLU
Layer normalization | ✓ | – | – | –
Input normalization | – | ✓ | – | –
Optimizer | Adam |  | SGD
---
(momentum=0.9)
Adam | Adam
Learning rate (policy) | 1e-4 | 5e-5 | 3e-4 | 3e-4
Learning rate (value) | 1e-4 | 1e-2 | 3e-4 | 3e-4
Weight initialization | Uniform | Xavier Uniform | Xavier Uniform | Xavier Uniform
Initial output scale (policy) | 1.0 | 1e-4 | 1e-2 | 1e-2
Target update | Hard | – | Soft (5e-3) | Soft (5e-3)
Clipped Double Q | False | – | True | True
Table 7: Details of each network architecture. We refer the original
implementations of each algorithm which is available online [23, 14, 48, 27,
42]. Note that AWR uses different learning rates of the policy per
environment.

---
MPO | Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
Learning rate ($\eta$) | 1e-2
Dual constraint | 1e-1
Mean constraint | 3.34e-4 | 1.67e-4 | 1e-3 | 1e-3 | 5.88e-5 | 1e-3
Stddev constraint | 3.34e-7 | 1.67e-7 | 1e-6 | 1e-6 | 5.88e-8 | 1e-6
Action penalty constraint | 1e-3
Inital stddev scale | 0.7 | 0.3 | 0.5 | 0.5 | 0.3 | 0.5
Discount factor $\gamma$ | 0.99

Table 8:  Hyper-parameters of MPO. We follow the implementation by Hoffman et
al. [27]. Some of mean & stddev constraint are divided by the number of
dimensions in the action space as suggested by Hoffman et al. [27], which is
empirically better.

---
AWR | Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
Learning rate (policy) | 1e-4 | 2.5e-5 | 5e-5 | 5e-5 | 1e-5 | 5e-5
Stddev scale | 0.4 | 0.4 | 0.4 | 0.2 | 0.4 | 0.4
Exp-Advantage Weight clip | 20.0
Action penalty coefficient | 10.0
Discount factor $\gamma$ | 0.99
$\lambda$ for TD($\lambda$) | 0.95

Table 9:  Hyper-parameters of AWR. We follow the implementation by Peng et al.
[48].

##### Small Network (AWR)

We denote the policy and value network used in AWR as a small (S) network,
described as follows (in Sec. 5.4, we didn’t change the activation and
distribution):

⬇

from torch import nn

\paractivation = nn.ReLU()

distribution = GaussianHeadWithFixedCovariance()

\parpolicy = nn.Sequential(

nn.Linear(obs_size, 128),

activation,

nn.Linear(128, 64),

activation,

nn.Linear(64, action_size),

distribution,

)

vf = nn.Sequential(

nn.Linear(obs_size, 128),

activation,

nn.Linear(128, 64),

activation,

nn.Linear(64, 1),

)

##### Medium Network (SAC)

We denote the policy and value network used in SAC as a medium (M) network,
described as follows (in Sec. 5.4, we didn’t change the activation and
distribution):

⬇

from torch import nn

\paractivation = nn.ReLU()

distribution = TanhSquashedDiagonalGaussian()

\parpolicy = nn.Sequential(

nn.Linear(obs_size, 256),

activation,

nn.Linear(256, 256),

activation,

nn.Linear(256, action_size * 2),

distribution

)

q_func = nn.Sequential(

ConcatObsAndAction(),

nn.Linear(obs_size + action_size, 256),

activation,

nn.Linear(256, 256),

activation,

nn.Linear(256, 1)

)

##### Large Network (MPO)

We denote the policy and value network used in MPO as a large (L) network,
described as follows (in Sec. 5.4, we didn’t change the activation and
distribution):

⬇

from torch import nn

\paractivation = nn.ELU()

distribution = GaussianHeadWithDiagonalCovariance()

\parpolicy = nn.Sequential(

nn.Linear(obs_size, 256),

nn.LayerNorm(256),

nn.Tanh(),

activation,

nn.Linear(256, 256),

activation,

nn.Linear(256, 256),

activation,

nn.Linear(256, action_size * 2),

distribution

)

q_func = nn.Sequential(

ConcatObsAndAction(),

nn.Linear(obs_size + action_size, 512),

nn.LayerNorm(512),

nn.Tanh(),

activation,

nn.Linear(512, 512),

activation,

nn.Linear(512, 256),

activation,

nn.Linear(256, 1)

)

##  Appendix B Relations to Other Algorithms

We here explain the relation of the unified policy iteration scheme covers
other algorithms. While we mainly focused on AWR, MPO, and SAC in the this
paper, our unified scheme covers other algorithms too, as summarized in Table
1:

##### EM control algorithms:

  * •

PoWER [30]: $\pi_{p}~{}(=\pi_{\theta})$ update is analytic.
${\mathcal{G}}=\eta\log Q^{\pi_{p}}$ and $Q^{\pi_{p}}$ is estimated by
TD($1$). $\pi_{\theta}={\mathcal{N}}(\mu_{\theta}(s),\Sigma_{\theta}(s))$ .

  * •

RWR [49]: $\pi_{p}=\pi_{\theta}$ is updated by SG. ${\mathcal{G}}=\eta\log r$
, and $\pi_{\theta}={\mathcal{N}}(\mu_{\theta}(s),\Sigma)$ . When the reward
is unbounded, RWR requires adaptive reward transformation (e.g.
$u_{\beta}(r(s,a))=\beta\exp(-\beta r(s,a))$; $\beta$ is a learnable
parameter).

  * •

REPS [50]: $\pi_{p}$ is $\pi_{q}$ of the previous EM step (on-policy) or a
mixture of all previous $\pi_{q}$ (off-policy), which is approximated by
samples. ${\mathcal{G}}=A^{\pi_{p}}$ and estimated by a single-step TD error
with a state-value function computed by solving a dual function. $\pi_{q}$ is
assumed as a softmax policy for discrete control in the original paper.

  * •

UREX [41]: $\pi_{p}=\pi_{\theta}$ is updated by SG.
${\mathcal{G}}=Q^{\pi_{p}}$ and estimated by TD($1$). $\pi_{\theta}$ is
assumed as a softmax policy for discrete control in the original paper.

  * •

V-MPO [56]: Almost the same as MPO, but a state-value function is trained by
$n$-step bootstrap instead of Q-function. Top-K advantages are used in E-step.

##### KL control algorithms:

  * •

TRPO [52]:
$\pi_{q}=\pi_{\theta}={\mathcal{N}}(\mu_{\theta}(s),\Sigma_{\theta})$. The KL
penalty is converted to a constraint, and the direction of the KL is reversed.
${\mathcal{G}}=A^{\pi_{p}}$ and estimated by TD($1$). $\pi_{p}$ is
continuously updated to $\pi_{q}$.

  * •

PPO with a KL penalty [54]:
$\pi_{q}=\pi_{\theta}={\mathcal{N}}(\mu_{\theta}(s),\Sigma_{\theta})$. The
direction of the KL penalty is reversed. ${\mathcal{G}}=A^{\pi_{p}}$ and
estimated by GAE [53]. An adaptive $\eta$ is used so that
$D_{KL}(\pi_{p}\;||\;\pi_{q})$ approximately matches to a target value.

  * •

DDPG333Note that DDPG and TD3 are not the “inference-based” algorithms, but we
can classify these two as KL control variants. [36]: $\pi_{q}=\pi_{\theta}$ is
the delta distribution and updated by SG. ${\mathcal{G}}=Q^{\pi_{q}}$ and
estimated by TD($0$). $\eta=0$ (i.e., the KL divergence and $\pi_{p}$ update
are ignored).

  * •

TD322footnotemark: 2 [13]: It is a variant of DDPG and leverages three
implementational techniques, clipped double Q-learning, delayed policy
updates, and target policy smoothing.

  * •

BRAC [68] and BEAR [31] (Offline RL): When we assume $\pi_{p}$ = $\pi_{b}$
(any behavior policy), and omitting its update, some of the offline RL
methods, such as BRAC or BEAR, can be interpreted as one of the KL control
methods. Both algorithms utilize the variants of clipped double Q-learning
($\lambda$-interpolation between max and min).

##  Appendix C Benchmarks on DeepMind Control Suite

Figure 2:  Benchmarking results on DeepMind Control Suite 28 environments. The
performances are averaged among 10 random seeds. We use an action repeat of 1
throughout all experiments for simplicity.

In this section, we show the benchmarking results on 28 tasks in DeepMind
Control Suite (Figure 2). Each algorithm is run with 2.5M steps (except for
humanoid domain; 14M steps), following Abdolmaleki et al. [2]. While the
previous work mentioned that tuning the number of action repeats was effective
[33], we used an action repeat of 1 throughout all experiments for simplicity.
We also use the hyper-parameters of each algorithm presented in Appendix A. As
discussed in Sec. 5.3, we incorporate ELU and layer normalization into SAC in
several domains where SAC is behind MPO or AWR. ELU and layer normalization
significantly improve performances, especially in pendulum_swingup and
cartpole_swingup_sparse. Some of MPO results don’t seem to match the original
paper, but we appropriately confirmed that these results are equivalent to
those of its public implementation [27].

---
| SAC | SAC-E+ | SAC-L+ | SAC-E+L+ | MPO | AWR
cartpole_balance | $\bm{975\pm 12}$ | – | – | – | $824\pm 118$ | $905\pm 146$
manipulator_bring_ball | $0.27\pm 0.0$ | $0.80\pm 0.1$ | $\bm{0.96\pm 0.3}$ | $0.72\pm 0.1$ | $0.79\pm 0.1$ | $0.85\pm 0.2$
ball_in_cup_catch | $\bm{980\pm 0.5}$ | – | – | – | $976\pm 6$ | $538\pm 328$
point_mass_easy | $\bm{850\pm 75}$ | – | – | – | $632\pm 254$ | $597\pm 329$
reacher_easy | $\bm{949\pm 14}$ | – | – | – | $920\pm 18$ | $934\pm 25$
reacher_hard | $\bm{962\pm 18}$ | – | – | – | $941\pm 22$ | $868\pm 70$
hopper_hop | $\bm{151\pm 50}$ | – | – | – | $41\pm 22$ | $0.1\pm 0.2$
walker_run | $\bm{615\pm 56}$ | – | – | – | $416\pm 99$ | $133\pm 40$
finger_spin | $\bm{962\pm 39}$ | – | – | – | $888\pm 69$ | $317\pm 185$
hopper_stand | $\bm{720\pm 207}$ | – | – | – | $640\pm 199$ | $5\pm 1$
walker_stand | $\bm{972\pm 6}$ | – | – | – | $945\pm 18$ | $633\pm 221$
fish_swim | $152\pm 23$ | $317\pm 32$ | $108\pm 9$ | $130\pm 6$ | $\bm{434\pm 66}$ | $97\pm 13$
swimmer_swimmer15 | $\bm{199\pm 15}$ | – | – | – | $139\pm 14$ | $52\pm 4$
swimmer_swimmer6 | $229\pm 12$ | $223\pm 11$ | $138\pm 13$ | $189\pm 17$ | $\bm{238\pm 29}$ | $170\pm 4$
acrobot_swingup | $10\pm 10$ | $21\pm 10$ | $15\pm 7$ | $34\pm 27$ | $\bm{127\pm 36}$ | $4\pm 2$
cartpole_swingup | $\bm{822\pm 45}$ | – | – | – | $776\pm 109$ | $767\pm 106$
pendulum_swingup | $542\pm 279$ | $550\pm 232$ | $718\pm 62$ | $\bm{830\pm 4}$ | $819\pm 11$ | $1\pm 4$
acrobot_swingup_sparse | $0.40\pm 0.1$ | $0.43\pm 0.2$ | $0.46\pm 0.0$ | $0.42\pm 0.1$ | $\bf{4\pm 4}$ | $0.0\pm 0.0$
finger_turn_easy | $\bm{922\pm 34}$ | – | – | – | $556\pm 116$ | $374\pm 117$
finger_turn_hard | $\bm{904\pm 21}$ | – | – | – | $410\pm 156$ | $109\pm 113$
fish_upright | $\bm{876\pm 25}$ | – | – | – | $631\pm 166$ | $478\pm 143$
cheetah_run | $\bm{682\pm 44}$ | – | – | – | $331\pm 60$ | $285\pm 73$
walker_walk | $916\pm 77$ | $\bm{960\pm 7}$ | $866\pm 93$ | $875\pm 97$ | $931\pm 25$ | $482\pm 115$
humanoid_run | $17\pm 47$ | $3\pm 2$ | $52\pm 62$ | $\bm{71\pm 39}$ | $22\pm 42$ | $0.8\pm 0.0$
cartpole_balance_sparse | $987\pm 27$ | $892\pm 119$ | $982\pm 30$ | $982\pm 10$ | $949\pm 76$ | $\bm{1000\pm 0.0}$
cartpole_swingup_sparse | $370\pm 370$ | $582\pm 261$ | $648\pm 324$ | $\bm{745\pm 36}$ | $585\pm 294$ | $3\pm 10$
humanoid_stand | $221\pm 231$ | $469\pm 245$ | $448\pm 359$ | $630\pm 129$ | $\bm{651\pm 183}$ | $6\pm 0.0$
humanoid_walk | $182\pm 255$ | $166\pm 148$ | $\bm{250\pm 209}$ | $219\pm 177$ | $224\pm 197$ | $1\pm 0.0$

Table 10: Raw scores of Figure 2. The performances are averaged among 10
random seeds. Each algorithm is run with 2.5M steps (except for humanoid
domain; 14M steps), following Abdolmaleki et al. [2]. We use an action repeat
of 1 throughout all experiments for simplicity.

##  Appendix D Benchmarks on MuJoCo Manipulation Tasks

We extensively evaluate their performance in the manipulation tasks (Figure
3). The trend seems the same as the locomotion tasks, while AWR beats SAC and
MPO in Striker, which means they fall into sub-optimal.

Figure 3: Benchmarking results on OpenAI Gym MuJoCo manipulation environments.
All experiments are run with 10 random seeds. SAC and MPO completely solve
Reacher and Pusher, while in Striker they fall into sub-optimal.

##  Appendix E Reproduction Results of AWR on MuJoCo Locomotion Environments

We re-implemented AWR based on PFRL, a pytorch-based RL library [14],
referring its original implementation [48]. Figure 4 shows the performance of
our implementation in the original experimental settings, also following
hyper-parameters. We recovered the original results in Peng et al. [48]
properly.

Figure 4: Reproduction of Advantage Weighted Regression (AWR). We obtained
comparable results to the original paper.

##  Appendix F Learning Curves

In this section, we present learning curves of the experiments in Sec. 5.

###  F.1 Clipped Double Q-Learning

Figure 5:  The learning curves of Table 2; ablation of Clipped Double
Q-Learning. We test original SAC (double), AWAC (double), MPO (single), and
some variants; SAC without clipped double Q-learning (single), AWAC (single),
and MPO with clipped double Q-learning (double).

###  F.2 Action Distribution for the Policy

Figure 6:  The learning curves of Table 3; ablation of Tanh transformation. A
line that stopped in the middle means that its training has stopped at that
step due to numerical error. We test SAC without tanh squashing, AWR with
tanh, and MPO with tanh.

###  F.3 Activation and Normalization

Figure 7:  The learning curves of Table 4; incorporating ELU/layer
normalization into SAC and AWR. E+/L+ indicates adding, and E-/L- indicates
removing ELU/layer normalization.

###  F.4 Network Size

Figure 8:  The learning curves of Table 5; experiment for finer network sizes.
(S) stands for the small network size from AWR, (M) for the middle network
size from SAC, and (L) for the large network size from MPO.

##  Appendix G Additional Experiments for Deeper Analysis of Implementation
Details

We share the additional experimental results for deeper analysis of
implementation details and co-adaptation nature. We report the final
cumulative return after 3M steps for Ant/HalfCheetah/Walker2d/Swimmer, 1M
steps for Hopper, and 10M steps for Humanoid. All results below are averaged
among 10 random seeds. These extensive experimental observations below suggest
not only the co-adaptive nature and transferability of each implementation and
code detail (discussed in Sec. 6), but also the properties of each kind of
algorithm (KL-based and EM-based); KL-based methods, such as SAC, shows the
co-dependent nature in implementation details (clipped double Q-learning,
Tanh-Gaussian Policy) but robustness to the code details related to neural
networks. In contrast, EM-based methods, such as MPO and AWR, show the co-
dependent nature in code details but robustness to the implementation details.
We hope these empirical observations from our experiments are valuable
contributions to the RL community.

#####  $\pi_{p}$ Update

We test different types of $\pi_{p}$ Update as summarized in Table 1 to
investigate the effectiveness of implementation choices. We prepare 4
variants: (1) MPO or AWAC with a uniform prior, (2) SAC with target policy
instead of a fixed uniform prior, (3) MPO without trust-region (only SG
update). The details of (1) - (3) are described below:

  1. (1)

We use the actions sampled from uniform distribution as well as the samples
from the policy at the past iteration $\pi_{\theta_{p}^{(k-1)}}$ for the
M-step in EM-controls;
$a_{j}\sim\alpha\text{Unif.}+(1-\alpha)\pi_{\theta_{p}^{(k-1)}},\alpha\in(0,1]$.
These variants are much closer to SAC (using uniform distribution as
$\pi_{p}$). We test $\alpha=0.25,0.5,0.75$ for both MPO and AWAC.

  2. (2)

We copy the parameter of $\pi_{q}$ at a certain interval and use it as
$\pi_{p}$ in the objective of KL Control, similar to MPO/PPO/TRPO. It seems
“KL-regularized” actor-critic, rather than “soft” (entropy-regularized). We
test both Lagrangian constraint (${\epsilon}=0.1,0.01,0.001$) and
regularization coefficient ($\eta=1.0,0.1,0.01$).

  3. (3)

Original MPO stabilizes the $\pi_{p}$ Update incorporating TR (trust-region)
into SG. We test the effect of TR, just removing TR term in the M-step of MPO.

However, the variants listed above have shown drastic degradation compared to
the original choice (we omit the performance table since most of them failed).
For example, the larger $\alpha$ (1) we chose, the lower scores the algorithm
achieved. Also, KL-SAC (2) did not learn meaningful behaviors, and removing TR
from MPO (3) induced significant performance drops. These failures suggest
that the implementation choice of $\pi_{p}$ Update might be the most important
one and should be designed carefully for both KL and EM control families.

#####  $\mathcal{G}$: Soft Q-function

We investigate the effect of the soft Q-function, instead of standard Q
function as MPO or AWAC use. We prepare MPO with soft Q, AWAC with soft Q, and
SAC without soft Q-function, just modifying Bellman equation and keep the
policy objectives as they are.

Table 11 shows that SAC without soft Q degrades its performance over 5 tasks
except for Ant, while it is not so drastic compared to clipped double Q or
Tanh-Gaussian policy. In contrast, MPO with soft Q slightly improves the
performance (over 4 tasks), and AWAC with soft Q slightly also does (over 3
tasks). These trends are similar to the clipped double Q or Tanh-Gaussian
policy. We think these experiments support our empirical observation: KL-based
methods, such as SAC, show the robustness to the code details, while EM-based
methods, such as MPO and AWR, show the co-dependent nature in code details but
robustness to the implementation details.

---
| Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
SAC | $\bf{3013\pm 602}$ | $\bf{5820\pm 411}$ | $\bf{15254\pm 751}$ | $5532\pm 1266$ | $\bf{8081\pm 1149}$ | $\bf{114\pm 21}$
SAC (w/o Soft Q) | $2487\pm 870$ | $5674\pm 202$ | $12319\pm 2731$ | $6496\pm 305$ | $6772\pm 3060$ | $\bf{114\pm 33}$
MPO | $2136\pm 1047$ | $3972\pm 849$ | $11769\pm 321$ | $\bf{6584\pm 455}$ | $5709\pm 1081$ | $70\pm 40$
MPO (w/ Soft Q) | $2271\pm 1267$ | $3817\pm 794$ | $11911\pm 274$ | $6312\pm 332$ | $6571\pm 461$ | $80\pm 32$
AWAC | $2329\pm 1020$ | $3307\pm 780$ | $7396\pm 677$ | $3659\pm 523$ | $5243\pm 200$ | $35\pm 8$
AWAC (w/ Soft Q) | $2545\pm 1062$ | $3671\pm 575$ | $7199\pm 628$ | $3862\pm 483$ | $5152\pm 162$ | $35\pm 10$

Table 11:  Ablation of Soft Q-function (the choice of $\mathcal{G}$ in Table
1), adding to MPO and AWAC while removing from SAC.

##### Network Size for AWAC

To investigate the co-dependent nature between implementation and code details
more precisely, we add the network size ablation of AWAC, whose
implementations stand between MPO and AWR. AWAC differs $\pi_{p}$ Update and
network size (the default choice of AWAC is (M)) from MPO (in fact, MPO uses
TD(0) in open-source implementation [27] and we assume the difference of
$\pi_{\theta}$ might be minor). Also, AWAC differs $\mathcal{G}$ and
$\mathcal{G}$ estimate from AWR.

The results of AWAC (Table 12) show a similar trend to AWR in high-dimensional
tasks (Ant, Humanoid); a larger network did not help. We may hypothesize that
$\pi_{p}$ Update of AWR/AWAC, mixture+SG, is not good at optimizing larger
networks, compared to SG + TR of MPO. In contrast, especially, Hopper and
Walker2d show a similar trend to MPO; the larger, the better. Totally, AWAC
with different network sizes shows the mixture trend of AWR and MPO, which is
the same as implementation details. We think these observations might
highlight the co-adaptation nature between implementation and code details.

---
| Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
AWAC (L) | $\bm{2764\pm 919}$ | $\bm{4350\pm 542}$ | $6433\pm 832$ | $2342\pm 269$ | $4164\pm 1707$ | $\bm{40\pm 5}$
AWAC (M) | $2329\pm 1020$ | $3307\pm 780$ | $\bm{7396\pm 677}$ | $3659\pm 523$ | $5243\pm 200$ | $35\pm 8$
AWAC (S) | $2038\pm 1152$ | $2022\pm 971$ | $5864\pm 768$ | $\bm{3705\pm 659}$ | $\bm{5331\pm 125}$ | $34\pm 11$

Table 12:  Ablation of network size for AWAC.

##### Combination of Clipped Double Q-Learning/Tanh-Gaussian and Soft
Q-function

We observe that both clipped double Q-learning/Tanh-Gaussian policy and soft
Q-function are the important implementation choices to KL control, SAC, which
lead to significant performance gains. To test the co-adaptation nature more
in detail, we implement these two choices into MPO and AWAC at the same time.

The results (Table 13 and Table 14) show that incorporating such multiple
combinations does not show any notable improvement in EM Controls, MPO and
AWAC. They also suggest the co-adaptation nature of those two implementations
to KL Controls, especially SAC.

---
| Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
MPO (S) | $2136\pm 1047$ | $3972\pm 849$ | $11769\pm 321$ | $6584\pm 455$ | $5709\pm 1081$ | $70\pm 40$
MPO (D) | $2352\pm 959$ | $\bm{4471\pm 281}$ | $12028\pm 191$ | $\bf{7179\pm 190}$ | $6858\pm 373$ | $69\pm 29$
MPO (Soft Q, S) | $2271\pm 1267$ | $3817\pm 794$ | $11911\pm 274$ | $6312\pm 332$ | $6571\pm 461$ | $\bm{80\pm 32}$
MPO (Soft Q, D) | $1283\pm 632$ | $4378\pm 252$ | $\bm{12117\pm 126}$ | $6822\pm 94$ | $\bm{6895\pm 433}$ | $45\pm 4$
AWAC (S) | $2540\pm 755$ | $3662\pm 712$ | $7226\pm 449$ | $3008\pm 375$ | $2738\pm 982$ | $38\pm 7$
AWAC (D) | $2329\pm 1020$ | $3307\pm 780$ | $7396\pm 677$ | $3659\pm 523$ | $5243\pm 200$ | $35\pm 8$
AWAC (Soft Q, S) | $\bm{2732\pm 660}$ | $3658\pm 416$ | $7270\pm 185$ | $3494\pm 330$ | $2926\pm 1134$ | $36\pm 10$
AWAC (Soft Q, D) | $2545\pm 1062$ | $3671\pm 575$ | $7199\pm 628$ | $3862\pm 483$ | $5152\pm 162$ | $35\pm 10$

Table 13:  Ablation of combination in implementation components; Soft
Q-function (the choice of $\mathcal{G}$) and Clipped Double Q-Learning (the
choice of $\mathcal{G}$ estimate), adding to MPO and AWAC. (D) denotes
algorithms with clipped double Q-learning, and (S) denotes without it.

---
| Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 | Ant-v2 | Humanoid-v2 | Swimmer-v2
MPO | $2136\pm 1047$ | $\bm{3972\pm 849}$ | $11769\pm 321$ | $\bf{6584\pm 455}$ | $5709\pm 1081$ | $70\pm 40$
MPO (Soft Q) | $2271\pm 1267$ | $3817\pm 794$ | $\bm{11911\pm 274}$ | $6312\pm 332$ | $\bm{6571\pm 461}$ | $\bm{80\pm 32}$
MPO (Soft Q, Tanh) | $314\pm 8^{\dagger}$ | $368\pm 47^{\dagger}$ | $3427\pm 207^{\dagger}$ | $628\pm 221^{\dagger}$ | $5919\pm 202^{\dagger}$ | $35\pm 8^{\dagger}$
AWAC | $2329\pm 1020$ | $3307\pm 780$ | $7396\pm 677$ | $3659\pm 523$ | $5243\pm 200$ | $35\pm 8$
AWAC (Soft Q) | $2545\pm 1062$ | $3671\pm 575$ | $7199\pm 628$ | $3862\pm 483$ | $5152\pm 162$ | $35\pm 10$
AWAC (Soft Q, Tanh) | $\bm{2989\pm 484}$ | $2794\pm 1692$ | $6263\pm 247$ | $3507\pm 458$ | $66\pm 4$ | $32\pm 5$

Table 14:  Ablation of combination in implementation components; Soft
Q-function (the choice of $\mathcal{G}$) and Tanh-squashed Gaussian policy
(the parameterization of the policy), adding to MPO and AWAC (†numerical error
happens during training).

##  Appendix H Failed Ablations

This section provides the failure case of ablations on tanh-squashed
distributions and exchanging network architectures, which shows the
catastrophic failure during training, and unclear insights.

###  H.1 Action Distribution for the Policy: Without Action Clipping

We observe that naive application of tanh-squashing to MPO and AWR
significantly suffers from numerical instability, which ends up with NaN
outputs (Table 15 and Figure 9). As we point out in Sec. 5.2, the practical
solution is to clip the action within the supports of distribution surely;
$a\in[-1+\epsilon,1-\epsilon]^{|\mathcal{A}|}$.

⬇

eps = 1e-6

actions = torch.clamp(actions, min=-1.+eps, max=1.-eps)

---
| SAC (w/) | SAC (w/o) | AWR (w/) | AWR (w/o) | MPO (w/) | MPO (w/o)
Hopper-v2 | $3013\pm 602$ | $6\pm 10$ | $\bf{3267\pm 383}$ | $3085\pm 593$ | $301\pm 12^{\dagger}$ | $2136\pm 1047$
Walker2d-v2 | $\bf{5820\pm 411}$ | $-\infty$ | $3281\pm 1084^{\dagger}$ | $4717\pm 678$ | $328\pm 95^{\dagger}$ | $3972\pm 849$
HalfCheetah-v2 | $\bf{15254\pm 751}$ | $-\infty$ | $1159\pm 599^{\dagger}$ | $5742\pm 667$ | $831\pm 242^{\dagger}$ | $11769\pm 321$
Ant-v2 | $5532\pm 1266$ | $-\infty$ | $152\pm 101^{\dagger}$ | $1127\pm 224$ | $202\pm 102^{\dagger}$ | $\bf{6584\pm 455}$
Humanoid-v2 | $\bf{8081\pm 1149}$ | $108\pm 82^{\dagger}$ | $538\pm 49^{\dagger}$ | $5573\pm 1020$ | $5642\pm 77^{\dagger}$ | $5709\pm 1081$
Swimmer-v2 | $114\pm 21$ | $28\pm 11$ | $117\pm 16$ | $\bf{128\pm 4}$ | $37\pm 6^{\dagger}$ | $70\pm 40$

Table 15:  Ablation of Tanh transformation (†numerical error happens during
training). We test SAC without tanh squashing, AWR with tanh, and MPO with
tanh. SAC without tanh transform results in drastic degradation of the
performance, which can be caused by the maximum entropy objective that
encourages the maximization of the covariance.  Figure 9:  The learning curves
of Table 15. We test SAC without tanh-squashed distribution, AWR with tanh,
and MPO with tanh. SAC without tanh transform (using MPO action penalty
instead) results in drastic degradation of the performance, which can be
caused by the maximum entropy objective that encourages the maximization of
the covariance. AWR and MPO with tanh squashing become numerically unstable. A
line that stopped in the middle means that its training has stopped at that
step due to numerical error.

###  H.2 Network Architecture: Whole Swapping

In contrast to prior works on TRPO and PPO, the network architecture that
works well in all the off-policy inference-based methods is not obvious, and
the RL community doesn’t have an agreeable default choice. Since the solution
space is too broad without any prior knowledge, one possible ablation is that
we test 3 different architectures that work well on at least one algorithm.

To validate the dependency of the performance on the network architecture, we
exchange the configuration of the policy and value networks, namely, the size
and number of hidden layers, the type of activation function, network
optimizer and learning rate, weight-initialization, and the normalization of
input state (See Appendix A). All other components remain the original
implementations.

However, this ablation study might end up the insufficient coverage and the
unclear insights. We broke down the network architecture comparison into the
one-by-one ablations of activation and normalization, and experimented with
finer network sizes.

Figure 10: Swapping network architectures between each methods. These results
suggest that these off-policy inference-based algorithms might be fragile with
other network architectures and more co-dependent with architectures than on-
policy algorithms. A line that stops in the middle means that training has
stopped at that step with numerical error due to NaN outputs.
---
Algorithm | Architecture
MPO | AWR | SAC
MPO | $2136\pm 1047$ | $623\pm 316$ | $1108\pm 828$
AWR | $2509\pm 1117$ | $\bf{3085\pm 593}$ | $2352\pm 960$
SAC | $2239\pm 669$ | $651\pm 381^{\dagger}$ | $3013\pm 602$
Table 16:  Results in Hopper-v2 environment (†numerical error happens during
training). All results are averaged over 10 seeds and we also show their
standard deviations.
---
Algorithm | Architecture
MPO | AWR | SAC
MPO | $3972\pm 849$ | $481\pm 210$ | $1548\pm 1390$
AWR | $1312\pm 680^{\dagger}$ | $4717\pm 678$ | $428\pm 89$
SAC | $5598\pm 795$ | $117\pm 164$ | $\bm{5820\pm 556}$
Table 17:  Results in Walker2d-v2 environment (†numerical error happens during
training). All results are averaged over 10 seeds and we also show their
standard deviations.
---
Algorithm | Architecture
MPO | AWR | SAC
MPO | $11769\pm 321$ | $339\pm 517$ | $-\infty$
AWR | $485\pm 57$ | $5742\pm 667$ | $1060\pm 146$
SAC | $\bf{16541\pm 341}$ | $589\pm 367^{\dagger}$ | $15254\pm 751$
Table 18:  Results in HalfCheetah-v2 environment (†numerical error happens
during training). All results are averaged over 10 seeds and we also show
their standard deviations.
---
Algorithm | Architecture
MPO | AWR | SAC
MPO | $6584\pm 455$ | $967\pm 202$ | $-\infty$
AWR | $-30\pm 12$ | $1127\pm 224$ | $243\pm 167$
SAC | $\bf{7159\pm 1577}$ | $479\pm 463^{\dagger}$ | $5532\pm 1266$
Table 19:  Results in Ant-v2 environment (†numerical error happens during
training). All results are averaged over 10 seeds and we also show their
standard deviations.
---
Algorithm | Architecture
MPO | AWR | SAC
MPO | $5709\pm 1081$ | $288\pm 126$ | $371\pm 72$
AWR | $420\pm 30$ | $5573\pm 1020$ | $507\pm 48$
SAC | $\bf{9225\pm 1010}$ | $205\pm 0$ | $8081\pm 1149$
Table 20:  Results in Humanoid-v2 environment. All results are averaged over
10 seeds and we also show their standard deviations.
---
Algorithm | Architecture
MPO | AWR | SAC
MPO | $70\pm 40$ | $41\pm 15$ | $-\infty$
AWR | $124\pm 3$ | $128\pm 4$ | $\bf{130\pm 8}$
SAC | $53\pm 6^{\dagger}$ | $47\pm 3$ | $114\pm 21$
Table 21:  Results in Swimmer-v2 environment (†numerical error happens during
training). All results are averaged over 10 seeds and we also show their
standard deviations.