# Efficient Reinforcemen Learning via Decoupling Exploration and Utilization

Jingpu Yang1, Qirui Zhao1, Helin Wang1, Yuxiao Huang1, Zirui Song2, Miao
Fang1,3*
1Northeastern University, Shenyang, China
2University of Technology Sydney, Sydney, Australia
3Olimei Company, Guangzhou, China * Corresponding author
<EMAIL_ADDRESS>
###### Abstract

Reinforcement Learning (RL), recognized as an efficient learning approach, has
achieved remarkable success across multiple fields and applications, including
gaming, robotics, and autonomous vehicles. Classical single-agent
reinforcement learning grapples with the imbalance of exploration and
exploitation as well as limited generalization abilities. This methodology
frequently leads to algorithms settling for suboptimal solutions that are
tailored only to specific datasets. In this work, our aim is to train agent
with efficient learning by decoupling exploration and utilization, so that
agent can escaping the conundrum of suboptimal Solutions. In reinforcement
learning, the previously imposed pessimistic punitive measures have deprived
the model of its exploratory potential, resulting in diminished exploration
capabilities. To address this, we have introduced an additional optimistic
Actor to enhance the model’s exploration ability, while employing a more
constrained pessimistic Actor for performance evaluation. The above idea is
implemented in the proposed OPARL (Optimistic and Pessimistic Actor
Reinforcement Learning) algorithm. This unique amalgamation within the
reinforcement learning paradigm fosters a more balanced and efficient
approach. It facilitates the optimization of policies that concentrate on
high-reward actions via pessimistic exploitation strategies while concurrently
ensuring extensive state coverage through optimistic exploration. Empirical
and theoretical investigations demonstrate that OPARL enhances agent
capabilities in both utilization and exploration. In the most tasks of
DMControl benchmark and Mujoco environment, OPARL performed better than state-
of-the-art methods.

###### Index Terms:

Reinforcement Learning, Sample Efficiency, Exploration, Ensemble Q-Learning

##  I Introduction

Reinforcement Learning(RL)[1], either in single-agent[2, 3, 4] or multi-
agent[5, 6, 7, 8, 9] contexts, has garnered considerable attention due to its
potential for a variety of real-world applications. The latest research
indicates that, Deep RL has achieved considerable success in various fields
such as robotics[10], recommendation systems[11], and strategy games[12]. In
single-agent training, DQN[13] surpassed human-level performance in Atari
video games, while Alpha[14, 15, 16] defeated numerous professional players in
the game of Go, achieving unprecedented heights. However, there are some
problems in the generalization of single-agent. Due to the divergence in data
distribution between the training and testing environments, conventional RL
algorithms are significantly impacted by extrapolation error, resulting in the
overestimation of Q-values for Out-Of-Distribution (OOD) state-action
pairs[17]. Overestimation can make certain state-action pairs appear more
attractive than their actual value warrants. This leads to the agent
erroneously prioritizing these overestimated actions, while neglecting other
potentially superior but underestimated actions. Furthermore, the imbalance
between exploration and exploitation strategies can lead to diminished model
generalization capabilities.

Figure 1: Performance comparison.

To address these issues, previous approaches introduced conservative ideas to
alleviate these problems, such as Behavior cloning[18] or forced pessimism.
Behavior Cloning learns policies directly from expert behavior data, rather
than through interaction with the environment. This results in the model being
reliant on a large volume of high-quality expert data and potentially
incapable of making correct decisions when encountering novel states not
present in the training set, thus leading to limited generalization
capabilities of the model. Trust Region Deep Deterministic Policy
Gradient(TD3) trains two independent Q-networks to independently estimate the
Q-values for the same state-action pairs[19], employing a pessimistic approach
by selecting the lower of the two estimates for the same state-action pair.
This method of reducing overestimation enhances the algorithm’s stability and
performance. However, enforced pessimism may lead to the underestimation of
certain advantageous state-action pairs, thereby constraining the policy’s
potential for exploration. Consequently, prior research has focused on the
pessimistic constraint of optimistic exploration under uncertainty, which
implies that overestimation of expected rewards could initiate exploration of
states and actions that would otherwise remain unexplored. Thus, we engage in
exploring the general form of the principle of optimistic
uncertainty—overestimation of expected rewards—to stimulate the investigation
of states and actions typically not explored while simultaneously ensuring the
model’s performance and stability.

Figure 2: Visualization of the OPARL framework. Blue represents pessimistic
thoughts and orange represents optimistic thoughts.

In the realm of Reinforcement Learning for continuous control problems, two
roughly antithetical methodologies have traditionally prevailed. On one hand,
several researchers endeavor to mitigate overestimation bias, for instance, by
adopting the minimum of the estimated values or by employing multiple
estimates to formulate an approximate lower bound, thereby manifesting a form
of pessimism towards the current value function estimation. Conversely, there
exists a school of thought advocating that an inherent optimism in
approximation estimations effectively fosters exploration within the
environment and action space, thereby augmenting the potential for higher
reward acquisition. Both approaches have underpinned the development of state-
of-the-art algorithms, as evidenced by the literature [20, 21, 22], suggesting
a non-mutually exclusive relationship between these methodologies.
Consequently, our study aims to amalgamate these seemingly disparate
strategies, thereby harnessing the synergistic benefits of both tactical
optimism and tactical pessimism. This integrated perspective leads to the
conceptualization of tactical optimism and tactical pessimism in the context
of RL. Empirical evidence suggests notable variations in the efficacy of these
methods across diverse environments. Our objective is to harmonize these
ostensibly conflicting paradigms, postulating that the relative contributions
of each component may fluctuate in accordance with the specificities of the
task at hand. To this end, we bifurcate the role of the actor in Actor-Critic
models, introducing two distinct actors: the ’optimism actor’ for exploration
and the ’pessimism actor’ for exploitation. An optimist actor plays a crucial
role in facilitating exploration; however, in scenarios characterized by
significant estimation errors, the pessimist actor becomes instrumental in
stabilizing the learning process. This leads to the formulation of our
proposed OPARL(Optimistic and Pessimistic Actor Reinforcement Learning)
algorithm, wherein our model accommodates a highly optimistic state overlay
during the exploration phase, enhancing the likelihood of discovering high-
reward regions. Despite this, such a technique may not always converge to the
optimal strategy. Conversely, we employ tactical pessimism during the policy
recuperation phase to optimize the exploitation of previously identified high-
reward behaviors. The overall concept is summarized in Figure 2.

In our approach, we engage with the environment using a tactically optimistic
exploration strategy, while simultaneously employing a tactically pessimistic
approach for evaluating all data observed thus far. This tactically
pessimistic utilization strategy is designed to maximize long-term rewards,
effectively mitigating biases introduced by short-term localized rewards.
Therefore, even if the exploration strategy leads to suboptimal behavior in
task rewards, the utilization strategy can still recalibrate the exploration
strategy to support further exploration. More subtly, this allows the
exploration strategy to seek better states and rewards. Ultimately, the
exploration strategy can generate superior data, rendering the final
performance less sensitive to the balance between intrinsic and extrinsic
rewards, which is advantageous for the model to explore more effectively in
diverse environments. Our series of experiments have demonstrated that OPARL
can balance both pessimistic and optimistic algorithms, offering a wider range
of exploration (optimistic exploration methods) and more stable performance
(pessimistic utilization methods), achieving the best of both worlds and
elevating the technological level for single-agent tasks in challenging
environments. Our research validates that conducting exploration and training
in an environment inclined towards highly optimistic estimations, particularly
characterized by an overestimation of expected rewards, is not only feasible
but also effective. Simultaneously, by decoupling and rebalancing exploration
and exploitation, superior results are achieved. The experiments indicate that
OPARL effectively enhances the model’s generalization ability. Our model
demonstrates improved states and rewards in four Mujoco environments and 14
out of 18 DMControl environments, while still maintaining the stability of the
original model, The effect is as shown in the Figure 1.

##  II RELATED WORK

###  II-A Reinforcement Learning

In the realm of reinforcement learning[23], a machine learning approach
characterized by interaction with the environment, significant advantages have
been demonstrated in diverse domains such as robotics control[10][24], game
theory[4][12], and autonomous vehicles[25]. Within these applications, agents
engage in a trial-and-error methodology to learn optimal action strategies,
thereby achieving peak performance in specific settings. Reinforcement
learning can be categorized into online and offline modalities. In online
reinforcement learning, agents interact with the environment in real-time to
learn strategies. However, this method’s primary drawback lies in its reliance
on immediate environmental feedback, which can slow the learning process and
pose safety risks in real-world applications. Conversely, offline
reinforcement learning utilizes pre-collected data for agent training,
reducing dependence on real-time environmental interactions but potentially
limiting the strategy’s generalization capabilities in the face of novel data.

In single-agent reinforcement learning, key challenges encountered during
training include balancing exploration and exploitation, reward delays, and
environmental uncertainty. Previous researchers have addressed these issues
through various methods, such as employing advanced exploration
strategies[26], designing more intricate reward functions[27], and
implementing Model Predictive Control (MPC)[28] to enhance the learning
efficiency and adaptability of the agents. However, advanced exploration
strategies do not fully ensure a balance between exploration and exploitation.
Consequently, we propose the OPARL algorithm, which decouples exploration and
exploitation strategies in single-agent scenarios. By training with distinct
parameters for each strategy, OPARL reduces interference between exploration
and exploitation, leading to improved learning outcomes.

###  II-B Sample Efficient Reinforcement Learning

In the domain of RL, sample efficiency emerges as a critical metric for
evaluating algorithmic performance[29], especially in scenarios necessitating
physical interactions, such as robotics[10][24] and autonomous driving[25].
The primary goal of sample-efficient reinforcement learning is to expedite the
learning process with minimal environmental interactions, thereby facilitating
rapid and accurate decision-making. The quintessential challenge in sample-
efficient RL lies in extracting the maximal amount of useful information from
each environmental interaction, while concurrently maintaining the stability
and generalizability of the algorithm.

To address this challenge, researchers have pioneered various methodologies. A
prominent approach is Model-Based Reinforcement Learning (MBRL), which
constructs an internal model of the environment to predict future states and
rewards, thereby reducing the dependency on actual environmental samples. This
approach has been effectively implemented in the research conducted by
Kurutach et al[30]. Additionally, Transfer Learning and Meta-Learning play
pivotal roles in enhancing sample efficiency. These techniques expedite the
learning process in new tasks by leveraging knowledge acquired from previous
tasks, thereby optimizing sample utilization. The potential of Meta-Learning
in swiftly adapting to new environments was demonstrated in the studies by
Finn et al[31].

Moreover, Data Augmentation and regularization techniques have proven valuable
in enhancing the learning efficiency of existing sample data. These techniques
generate additional training data from limited interactions or employ
regularization to prevent overfitting. However, the implementation of these
techniques often necessitates complex algorithmic design and fine-tuning.
Furthermore, their efficacy is typically confined to specific tasks and may
not generalize across diverse tasks or environments. In contrast, our proposed
algorithm is comparatively straightforward, achieving equivalent or superior
performance with fewer interactions. It rapidly learns from limited
experiments, circumventing the high costs associated with data collection and
potential safety risks.

###  II-C Optimistic and Pessimistic in Reinforcement Learning

In the realm of exploratory learning research, the strategies of optimistic
and pessimistic exploration have long been contentious[32, 33, 34]. Scholars
argue that in uncertain environments, it is necessary to employ algorithms
based on optimistic principles (reward maximization) for exploring state-
action pairs with high cognitive uncertainty[35, 34, 36, 37, 38], and such
overestimation bias may not always be detrimental. In certain cases, errors
due to underestimation bias can be harmful, as overestimation bias aids in
encouraging exploration[20][39] of actions that are overvalued, while
underestimation bias may impede exploration. When these highly random regions
correspond to areas of high value, promoting exploration is beneficial, yet
underestimation bias could prevent agents from learning the high values of
such regions. Consequently, our algorithm retains a highly optimistic approach
in exploration strategies to probe more high-value areas.

However, if highly random regions also possess lower values, overestimation
bias might lead to agents excessively exploring low-value areas. Thus, another
group of scholars emphasizes the necessity to standardize highly uncertain
state-action pairs. Based on the DDPG[40] algorithm, the TD3[19] method
selects the smaller of two $Q$ values[41] as an approximate lower bound and
explores the environment pessimistically. By reducing pessimistic exploration
in value estimation, agents can achieve better long-term rewards. Therefore,
our algorithm maintains this pessimistic approach in utilization strategies to
minimize extrapolation errors, ensuring model stability and performance.

##  III PROBLEM SETUP

We will formulate the problem as a Markov Decision Process (MDP)
$M\equiv(S,A,R,p,\gamma$), where:

  * •

$S$ represents the state space,

  * •

$A$ represents the action space,

  * •

$p$ represents the transition dynamics,

  * •

$R$ represents the reward function,

  * •

$\gamma\in[0,1)$ represents the discount factor.

For a given state $s\in S$, the policy $\pi$ maps the state to an action
(deterministic policy), and the agent selects the action $a\in A$ based on
policy $\pi$, receiving a reward $r$ and a new state $s^{\prime}$ from the
environment. Our goal is to learn behavior that maximizes rewards, where the
benefit is defined as the total discounted reward
$R_{t}=\sum_{i=t}^{T}\gamma^{i-t}r(s_{i},a_{i})$, and the state-action value
function
$Q_{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid
s_{0}=s,a_{0}=a\right]$.

To learn behavior that maximizes rewards, we first need to obtain an
experience pool buffer containing high-reward regions. The buffer is
$\\{(s,a,s^{\prime},r,d\\_b$}, which is collected using the tactically
optimistic behavior strategy $\pi_{\text{opt}}$. The optimistic action
$a_{\text{opt}}$ is obtained through $\pi_{\text{opt}}$, and a new state
$s^{\prime}$ is obtained by interacting with the environment. The reward $r$
and whether the action is completed $d\\_b$ are added to the experience pool
to obtain more high-reward regions. When using data from the experience pool,
we extract states $s$, actions $a$, new states $s^{\prime}$, rewards $r$, and
whether the action is completed $d\\_b$ from the experience pool and use
tactical pessimism to obtain more reasonable actions, thereby maximizing the
reward. The specific algorithms of both will be presented in detail in the
fourth part.

##  IV Optimistic and Pessimistic Actors in RL

The aim of this section is to establish a novel algorithm to decouple two
distinct functions of actors. The first function involves optimistic
experience collection and strategizing to explore areas with higher returns.
The second function is designed to penalize outlier state-action pairs,
employing tactical pessimism to secure more reasonable actions in high-return
areas, thus maximizing rewards. This decoupling allows us to explore the
environment optimistically while mitigating biases in overestimating expected
rewards in our evaluation strategy. Most online reinforcement learning methods
implicitly integrate some level of noise into their exploration strategies to
expand their exploration scope, as exemplified by the $\epsilon$-greedy noise
in DQN or $OU$ noise in TD3. However, these methods often fall short of
enabling comprehensive exploration. Our framework, OPARL, amalgamates the $Q$
values of multiple state-action pairs to access the broadest possible
exploration area.

In Section A, we introduce the general OPARL framework, which can be
effectively combined with any reinforcement learning algorithm for
enhancement. In Section B, we delve into a mathematical exploration of merging
optimistic exploration with pessimistic exploitation, illustrating how this
strategy enables an agent to attain high rewards in the face of challenging
exploration tasks. Section C explores the implementation of combining
optimistic exploration and pessimistic exploitation through parameter
resetting, demonstrating the synergy of these approaches.

###  IV-A The OPARL Framework

Algorithm 1 OPTIMISTIC AND PESSIMISTIC ACTOR IN RL (OPARL)

0: Initialize critic networks $Q_{\theta 1}$, $Q_{\theta 2}$, …, $Q_{\theta
n}$, and actor network $\pi_{\phi opt}$, $\pi_{\phi pes}$; with random
parameters $\theta_{1}$, $\theta_{2}$, …, $\theta_{n}$, $\phi_{opt}$,
$\phi_{pes}$;

0: Initialize target networks $\theta_{1}^{\prime}\leftarrow\theta_{1}$,
$\theta_{2}^{\prime}\leftarrow\theta_{2}$, …,
$\theta_{N}^{\prime}\leftarrow\theta_{N}$,
$\phi_{opt}^{\prime}\leftarrow\phi_{opt}$,
$\phi_{pes}^{\prime}\longleftarrow\phi_{pes}$;

0: Initialize replay buffer $\mathcal{B}$;

0:

1: for $t=1$ to $T$ do

2: if $t$ mod $k$ then

3: $a_{tu}=\pi_{\phi opt}(s_{t})+\epsilon,\epsilon\sim N(0,s^{2})$;

4: Obtain new state $s_{t+1}$ and reward $r$;

5: $Q_{max}=\max_{i\in N}(Q_{\theta i}(s_{t+1},a_{tu}))$;

6: $a_{opt}=a_{tu}$ where $Q_{u}=Q_{max}$;

7: else

8: $a_{t}=\pi_{\phi pes}(s_{t})+\epsilon,\epsilon\sim N(0,s^{2})$;

9: Obtain new state $s_{t+1}$ and reward $r$;

10: $Q_{min}=\min_{i\in N}(Q_{\theta i}(s_{t+1},a_{t}))$;

11: $a_{pes}=a_{t}$ where $Q_{\theta i}=Q_{min}$;

12: end if

13: Sample mini-batch: $(s,a,r,s_{t+1})$ $\sim$ $\mathcal{B}$;

13:

14: $\tilde{a_{t}}=\pi_{\phi pes}(s_{t+1})+\epsilon,\epsilon\sim N(0,s^{2})$;

15: Obtain new state $s_{t+1}$ and reward $r$;

16: $Q_{min}=\min_{i\in N}(Q_{\theta i}(s_{t+1},\tilde{a_{t}}))$;

17: $a_{pes}=\tilde{a_{t}}$ where $Q_{\theta i}=Q_{min}$;

18: Update critics:$y$ = $r+\gamma\min_{i\in N}Q_{\theta
i}(s^{\prime},a_{pes})$;$\theta_{i}\longleftarrow\operatorname*{arg\,min}_{\theta_{i}}N^{-1}\sum(y-Q_{\theta_{i}}(s,a))^{2}$;

19: Update target
networks:$\theta^{\prime}_{i}\leftarrow\tau\theta_{i}+(1-\tau)\theta^{\prime}_{i}$
, where $i\in
N$;$\phi^{\prime}_{pes}\leftarrow\tau\phi_{pes}+(1-\tau)\phi^{\prime}_{pes}$;

20: if $t$ mod $w$ then

21: $\phi_{pes}\leftarrow\phi_{opt}$;

22: end if

23: end for

Our algorithm is bifurcated into two distinct phases: optimistic exploration
and pessimistic utilization. In the exploration phase, optimistic strategic
exploration is conducted at every $k$ step, determining the action with the
maximal Q-value through a post-optimistic strategy state. The action
associated with the highest Q-value is selected, signifying the culmination of
the optimistic exploration process. Rewards $r$ and next states $s_{t+1}$ are
procured from environmental interactions and are systematically stored in a
buffer. In contrast, during the pessimistic phase, actions correlated with the
minimal Q-value, as dictated by the pessimistic strategy, are chosen to accrue
rewards and next states, which are then amalgamated into the buffer. This
facilitates agents in exploring regions with augmented rewards, attributable
to an elevated exploration ratio. In the utilization phase, a pessimistic
approach is adopted. The state, as ascertained through the pessimistic
strategy function, identifies the action with the lowest Q-value for the
purposes of loss computation and parameter updating. Parameters from the
optimistic strategy are periodically transferred to the pessimistic strategy
function at every $w$ step, fostering a more profound integration of both
optimistic and pessimistic methodologies.The pseudocode for OPARL is shown in
Algorithm 1.

###  IV-B The Algorithm of OPARL

We assume $Q_{\theta}^{\prime}$ stands for optimistic exploration, while
$Q_{\theta}^{\prime\prime}$ stands for pessimistic exploitation. Formulas1, 2,
represent the exploration ranges that we consider to be pessimistic,
optimistic.

$y1=r+\gamma\min_{i=1,2,\ldots,N}Q_{\theta_{i}}^{\prime}(s^{\prime},\widetilde{a}).$
(1)

$y2=r+\gamma\max_{i=1,2,\ldots,N}Q_{\theta_{i}}^{\prime\prime}(s^{\prime},\widetilde{a}).$
(2)

And Formulas 3 stands for random Walkely determined $Q^{*}_{\theta i}$ values
from 1 to $N$, respectively. From this, we can discern that Formula 3, which
involves random selection of state-action pairs among N $Q$-values,
encapsulates exploration bounds that correspond to optimistic and pessimistic
exploration at its upper and lower limits, respectively. This delineates the
exploratory scope within which the random strategy operates, framed by the
optimistic exploration’s propensity for high-reward regions and the cautious
approach characteristic of pessimistic exploration.

$y^{\prime}=r+\gamma Q^{*}_{\theta i}(s^{\prime},\widetilde{a}),\>where\>{i\in
1,2,\ldots,N}.$ (3)

Formula 4 suggests that the variance induced by pessimistic exploration is
less than or equal to the variance from randomly selecting $Q$-values,
implying a narrower exploratory range for pessimistic exploration. Conversely,
Formula 5 indicates that the variance caused by optimistic exploration is
greater than or equal to the variance from random $Q$-value selection,
suggesting a broader exploratory range for optimistic exploration. This
academic interpretation aligns with the inherent nature of these exploration
strategies within their respective scopes.

$\min_{\theta_{i}}N^{-1}\sum(y1-Q_{\theta_{i}}^{\prime}(s,a))^{2}\leq\min_{\theta_{i}}N^{-1}\sum(y^{\prime}-Q_{\theta_{i}}^{\prime}(s,a))^{2}.$
(4)

$\max_{\theta_{i}}N^{-1}\sum(y2-Q_{\theta_{i}}^{\prime}(s,a))^{2}\leq\max_{\theta_{i}}N^{-1}\sum(y^{\prime}-Q_{\theta_{i}}^{\prime}(s,a))^{2}.$
(5)

Formula 6 shows an estimate of future expectations. Therefore, as deduced from
formulas 5 and 6, optimistic exploration has the potential to explore more
high-reward areas as it can engage in a greater number of activities while
ensuring the expectations of those actions remain constant. Similarly,
formulas 4 and 6 illustrate that despite unchanged expectations, the range of
actions in pessimistic exploration undergoes a certain reduction. This implies
that pessimistic exploration is inclined towards selecting reasonable actions
from high-reward areas over those with the highest short-term returns. The
OPARL algorithm effectively amalgamates optimism with pessimism, enabling
broader exploration and achieving longer-term returns compared to other
models.

$\displaystyle Q_{\theta_{i}}(s_{t},a_{t})$
$\displaystyle=r_{t}+\gamma\mathbb{E}[Q_{\theta}(s_{t+1},a_{t+1})].$ (6)

###  IV-C Parameter Reset

OPARLTD3PPOSACEBDRND

((a)) Ant

((b)) Hopper

((c)) Humanoid

((d)) Walker

Figure 3:  The performance curves presented in Mujoco environment, including
our OPARL and other benchmark algorithms within Mujoco framework. The shaded
region represents half a standard deviation of the average evaluation over 5
distinct seeds.Curves are smoothed with a window of size 10 for visual
clarity.

In our investigation, we aim to delineate the parameters of a synergistic
approach that integrates the characteristics of a pessimistic exploration
strategy, denoted as $\phi_{pes}$, into the framework of an optimistic
exploration approach, symbolized as $\phi_{opt}$. The principal objective of
this integration is to empower the optimistic exploration strategy with the
capacity to assimilate insights and knowledge gleaned from the pessimistic
exploration paradigm, thereby augmenting its exploratory efficiency and
effectiveness[42]. This methodological integration is operationalized by
transferring the state dictionary from the pessimistic exploration strategy to
the optimistic exploration framework. Such a transfer enables the initiation
of the optimistic strategy with parameter configurations that bear resemblance
to those employed by the pessimistic strategy. This strategic alignment is
instrumental in mitigating the risk of excessive deviation from established
exploration protocols and ensuring a grounded approach to exploratory
behavior.

Concurrently, as the training process evolves, the optimistic exploration
strategy is designed to dynamically refine its behavior policies through
continuous parameter updates[43]. This progressive adaptation fosters the
development of more sophisticated and effective exploration techniques,
thereby enhancing the strategy’s capacity to proficiently execute its
designated tasks. Additionally, this approach underscores the importance of
maintaining congruence between the target policy network and the current
policy network throughout the training phase. Such an alignment is pivotal for
ensuring consistency and coherence in the training process[44]. The
versatility of this integrated approach renders it applicable for refining any
exploration model that encapsulates elements of both optimism and pessimism in
its operational paradigm.

##  V EXPERIMENTS

Our experiments demonstrate the efficacy of decoupling exploration and
exploitation strategies across a spectrum of tasks with varying demands for
exploration. In Section A, we elucidate the baseline and environmental
prerequisites. Section B articulates a comparative analysis of fundamental
online reinforcement learning algorithms such as TD3, Soft Actor-Critic(SAC),
and Proximal Policy Optimization(PPO)[45] within the Mujoco
environment[46][47], juxtaposed against our proposed model. Section C
escalates the comparison to contemporary advanced exploration models like
Random Network Distillation(RND)[48] and exploration-by-disagreement(EBD)[49],
underscoring the superior exploratory prowess of our model, OPARL. In Section
D, the complexity is furthered by evaluating OPARL in the more challenging
DMControl environment[50], benchmarking against baseline models including TD3
and SAC, as well as the avant-garde model REDQ, accompanied by a thorough
analysis. Section E delves into a series of ablation studies designed to
substantiate the merits of decoupled exploration and exploitation. These
studies dissect the performance of OPARL under scenarios of unmitigated
optimism, unrelenting pessimism, and in the absence of an ensemble Q function.

TABLE I: After undergoing training for one million time steps within the
DMControl suite, the OPARL algorithm demonstrated superior performance in the
majority of tasks, excelling in 14 out of 18 scenarios. OPARL consistently
outperformed its underlying algorithm TD3 across all tasks. Specifically,
OPARL exhibited performance improvements of 14.30%, 4.08%, and 7.36% over the
REDQ, SAC, and TD3 algorithms, respectively. The highest scores are denoted in
bold black typeface. The symbol ± represents the standard deviation across 10
trials. Detailed performance trajectories are available in Appendix A.

Domain |  Task |  OPARL |  REDQ* |  SAC* |  TD3
---|---|---|---|---|---
BallInCup |  Catch |  983.8 $\pm$ 3.8 |  978.8 $\pm$ 3.7 |  980.3 $\pm$ 3.4 |  979.2 $\pm$ 1.0
Cartpole |  Balance |  999.8 $\pm$ 0.0 |  984.0 $\pm$ 6.0 |  997.7 $\pm$ 1.5 |  999.8 $\pm$ 0.0
Cartpole |  BalanceSparse |  1000.0 $\pm$ 0.0 |  872.1 $\pm$ 262.7 |  997.6 $\pm$ 5.7 |  1000.0 $\pm$ 0.0
Cartpole |  Swingup |  874.6 $\pm$ 7.5 |  828.1 $\pm$ 17.2 |  865.1 $\pm$ 1.6 |  865.9 $\pm$ 0.8
Cheetah |  Run |  861.6 $\pm$ 21.4 |  614.2 $\pm$ 58.2 |  873.4 $\pm$ 21.5 |  788.7 $\pm$ 50.9
Finger |  Spin |  974.6 $\pm$ 15.5 |  940.1 $\pm$ 33.5 |  966.3 $\pm$ 27.1 |  949.0 $\pm$ 16.2
Fish |  Swim |  356.0 $\pm$ 65.1 |  159.3 $\pm$ 100.1 |  342.4 $\pm$ 134.5 |  367.4 $\pm$ 59.8
Fish |  Upright |  926.0 $\pm$ 18.6 |  799.6 $\pm$ 113.8 |  898.6 $\pm$ 50.4 |  913.0 $\pm$ 14.6
Hopper |  Stand |  599.5 $\pm$ 202.2 |  393.5 $\pm$ 225.8 |  597.8 $\pm$ 308.8 |  354.4 $\pm$ 149.9
Pendulum |  Swingup |  424.4 $\pm$ 155.0 |  382.6 $\pm$ 297.0 |  226.2 $\pm$ 228.9 |  382.6 $\pm$ 209.3
PointMass |  Easy |  909.4 $\pm$ 8.5 |  880.9 $\pm$ 16.7 |  889.9 $\pm$ 33.1 |  892.5 $\pm$ 5.2
Reacher |  Easy |  983.6 $\pm$ 1.2 |  970.9 $\pm$ 24.4 |  983.5 $\pm$ 4.2 |  983.8 $\pm$ 1.2
Reacher |  Hard |  977.5 $\pm$ 2.9 |  964.1 $\pm$ 24.0 |  958.6 $\pm$ 40.9 |  934.1 $\pm$ 71.2
Swimmer |  Swimmer6 |  471.4 $\pm$ 60.8 |  215.8 $\pm$ 119.0 |  359.3 $\pm$ 130.9 |  275.6 $\pm$ 84.9
Swimmer |  Swimmer15 |  442.1 $\pm$ 187.5 |  178.6 $\pm$ 116.6 |  264.6 $\pm$ 136.9 |  220.4 $\pm$ 82.3
Walker |  Run |  752.1 $\pm$ 23.2 |  590.9 $\pm$ 51.6 |  773.0 $\pm$ 32.9 |  649.3 $\pm$ 82.9
Walker |  Stand |  989.0 $\pm$ 2.0 |  974.0 $\pm$ 12.6 |  986.8 $\pm$ 2.7 |  984.5 $\pm$ 1.3
Walker |  Walk |  975.3 $\pm$ 2.9 |  957.3 $\pm$ 10.6 |  973.5 $\pm$ 4.4 |  969.2 $\pm$ 2.1
Average |  Scores |  805.7 |  704.7 |  774.1 |  750.5
Improvements |  Percentage |  / |  14.30% |  4.08% |  7.36%

  * *

The data marked with [*] is cited from the paper[50]

###  V-A Evaluation Setting

Baseline: In our research, we subjected the newly proposed OPARL algorithm to
benchmark testing, assessing its efficacy relative to a cohort of acknowledged
algorithms. Owing to their demonstrated reliability in various tasks, TD3[19],
SAC[51], and Proximal Policy Optimization (PPO) [45]were chosen as our
principal benchmarks. In exploration comparisons, we selected state-of-the-art
models such as RND[48] and EBD[49] for juxtaposition. Additionally, within the
DMControl suite, we integrated comparisons with Randomized Ensembled Double
Q-Learning (REDQ)[52], which is distinguished by its ensemble Q-technology
that augments exploratory capabilities and stability. To guarantee equitable
and uniform comparisons, we utilized implementations from CleanRL with default
hyperparameters as stipulated by the original authors, and the algorithms of
the comparative models are delineated in the appendix C.
Environments: Our study utilized the state-based DMControl suite[50] under the
Mujoco[46][47] framework provided by OpenAI’s Gym[53], to assess performance
across four continuous control tasks. The choice of OpenAI Gym[53] was due to
its array of environments conducive to benchmarking reinforcement learning
algorithms, enabling efficient algorithm-environment interactions. In the
Mujoco environment, each algorithm was evaluated over a million time steps,
with five seeds randomly selected every 5000 steps to calculate the average
rewards and their variance, indicating the range of performance variability.
Consistent with related studies, in the DMControl environment, we randomly
selected ten seeds every 10000 steps for calculating average rewards, using
the variance from these ten seeds to represent the fluctuation range of the
model’s performance. For more environment details, please refer to the
appendix B.
Setup: We modulate the levels of optimism and pessimism in our model via the
ensemble $Q$, initially set at five. The state-action pairs are determined by
selecting the one with the greatest variance from these five $Q$ values. To
balance optimistic and pessimistic explorations, we set their ratio at 1:1.
Additionally, we revise the pessimistic exploration parameters $\phi_{pes}$
every 20000 time steps and incorporate them into the optimistic exploration
parameters $\phi_{opt}$. The other parameter settings of this algorithm are in
line with the original TD3 model[19]. We believe that adhering to these
baseline parameters facilitates a fair comparison with existing methodologies
while allowing the nuances of our approach to be highlighted. Further details
about the implementation are available in the appendix D.

###  V-B Evaluation of OPARL in Mujoco Environment

((a)) Ant

((b)) Hopper

((c)) Humanoid

((d)) Walker

Figure 4: The ablation performance curves presented in the Mujoco environment.
The shaded region represents half a standard deviation of the average
evaluation over 5 distinct seeds.Curves are smoothed with a window of size 10
for visual clarity.

In the Mujoco environment, our results, as depicted in the accompanying figure
4, reveal that the OPARL model exhibits exceptional performance, particularly
in the Hopper and Ant scenarios. Notably, OPARL consistently surpasses its
foundational algorithm, TD3, in the majority of test cases. A trend observed
is the progressive stabilization of the model with increasing time steps,
marked by OPARL exhibiting the least fluctuation amplitude among all models
compared. This attribute underscores OPARL’s capacity to enhance learning
stability. In a comprehensive assessment, particularly in the Ant environment,
our model significantly outperforms competing models. Quantitatively, OPARL
achieves performance improvements of 44.04%, 51.25% and 773.02% over the SAC,
TD3, and PPO[45] models, respectively. At the same time, our algorithm
achieves performance enhancements of 50.93% and 497.62% over the RND[48] and
EBD[49]. This significant improvement underscores the efficacy of integrating
multiple Q-values and adopting a maximal value approach (optimistic
exploration strategy) in the decision-making process, which markedly augments
the algorithm’s capacity for exploration. Consequently, this approach
substantially broadens its adaptability and learning scope within complex
environments. The findings affirm the premise that the integration of diverse
predictive models can facilitate more extensive and effective environmental
exploration, which is crucial for learning in high-dimensional and intricate
tasks.

###  V-C Evaluation of OPARL in DMControl Environment

Table I delineates the mean performance metrics of the algorithm after a
training regimen spanning one million time steps. The empirical outcomes
significantly indicate that the OPARL algorithm surpasses the comparative
algorithms in the majority of the environments tested (14 out of 18).
Specifically, when juxtaposed with the REDQ, SAC, and TD3 algorithms, OPARL
demonstrated performance enhancements of 14.30%, 4.08%, and 7.36%,
respectively. Notably, in the ”swimmer-swimmer6” and ”swimmer-swimmer15”
scenarios, OPARL’s performance increments were particularly pronounced,
amounting to 71.04% and 100.59%, respectively. A more granular analysis of the
learning curves, commencing from a foundational level, is provided in the
appendix A. These findings not only underscore the adaptability and
superiority of OPARL across a diverse array of environments but also lay a
solid groundwork for further exploration of its potential in complex task
applications.

###  V-D Ablation Study

A series of ablation studies were conducted to assess the impact of novel
components integrated into our algorithm’s structure. The experimental design
involved the independent application of an optimistic exploration
strategy—selecting the maximal value from an ensemble of five $Q$ values—and a
pessimistic exploration strategy—selecting the minimal value from the
ensemble. Additionally, we examined the effect of not introducing extra
ensemble $Q$ values, thus maintaining the original configuration with only two
$Q$ values. The evaluation results are presented in the figure. While Yarats
et al.’s study[54] suggests that adequate exploration and state coverage can
enable standard reinforcement learning to achieve equivalent outcomes without
a pessimistic strategy, both prior work on decoupled strategy learning[55] and
our experimental findings consistently demonstrate that to enhance
performance, optimistic exploration must be coupled with pessimistic
constraints.

##  VI CONCLUSION

We propose Optimistic and Pessimistic Actor in reinforcement learning(OPARL),
a straightforward and effective reinforcement learning algorithm that
decouples the exploration and exploitation strategies of the model. This
approach, characterized by optimistic exploration and pessimistic
exploitation, achieves superior outcomes in most environments. Our research
has demonstrated that high optimism in exploration is crucial in the learning
process of complex environments. However, previous Actor-Critic algorithms,
relying solely on a fixed degree of optimism, were unable to select the most
rational actions from a long-term perspective, while traditional pessimistic
trimming severely limited the model’s exploratory capabilities. Our
experiments show that by decoupling the original algorithm into distinct
exploration and exploitation functions, OPARL significantly enhances
performance in the Mujoco environment, especially achieving rapid learning in
challenging settings such as Ant and Humanoid. Moreover, the model effectively
adapts to the more complex DMControl environment, improving performance in 14
out of 18 settings, thereby reaching the state-of-the-art level.

## References

  * [1] Ali Dorri, Salil S. Kanhere, and Raja Jurdak.  Multi-agent systems: A survey.  IEEE Access, 6:28573–28593, 2018.
  * [2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [3] AdriàPuigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell.  Agent57: Outperforming the atari human benchmark.  preprint arXiv: Learning,arXiv: Learning (2020).
  * [4] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [5] Lei Yuan, Lihe Li, Ziqian Zhang, Feng Chen, Tianyi Zhang, Cong Guan, Yang Yu, and Zhi-Hua Zhou.  Learning to coordinate with anyone.  In The Fifth International Conference on Distributed Artificial Intelligence, pages 1–9, 2023.
  * [6] Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang.  Multi-agent incentive communication via decentralized teammate modeling.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9466–9474, 2022.
  * [7] Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang.  Discovering generalizable multi-agent coordination skills from multi-task offline data.  In The Eleventh International Conference on Learning Representations, 2022.
  * [8] Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu.  A survey of progress on cooperative multi-agent reinforcement learning in open environment.  preprint arXiv:2312.01058 (2023).
  * [9] Lei Yuan, Tao Jiang, Lihe Li, Feng Chen, Zongzhang Zhang, and Yang Yu.  Robust multi-agent communication via multi-view message certification.  arXiv preprint arXiv:2305.13936, 2023.
  * [10] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [11] Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song.  Generative adversarial user model for reinforcement learning based recommendation system.  In International Conference on Machine Learning, pages 1052–1061, 2019.
  * [12] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver.  Grandmaster level in starcraft II using multi-agent reinforcement learning.  Nature, 575(7782):350–354, 2019.
  * [13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [14] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [15] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis.  Mastering the game of go without human knowledge.  Nature, 550(7676):354–359, 2017.
  * [16] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [17] Sebastian Thrun and Anton Schwartz.  Issues in using function approximation for reinforcement learning.  Jan 1999.
  * [18] Scott Fujimoto and Shixiang Shane Gu.  A minimalist approach to offline reinforcement learning.  In Advances in Neural Information Processing Systems , NeurIPS 2021, pages 20132–20145, 2021.
  * [19] Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In International conference on machine learning, pages 1587–1596, 2018.
  * [20] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann.  Better exploration with optimistic actor-critic.  preprint arXiv: Machine Learning (2019).
  * [21] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann.  Better exploration with optimistic actor-critic.  preprint arXiv:1910.12807(2019).
  * [22] Scott Fujimoto, Herkevan Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  preprint arXiv:1802.09477(2018).
  * [23] Richard S Sutton and Andrew G Barto.  Reinforcement Learning: An Introduction.  2018\.
  * [24] Yafei Hu, Junyi Geng, Chen Wang, John Keller, and Sebastian A. Scherer.  Off-policy evaluation with online adaptation for robot exploration in challenging environments.  IEEE Robotics Autom. Letters, pages 3780–3787, 2023.
  * [25] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua.  Safe, multi-agent, reinforcement learning for autonomous driving.  preprint arXiv:1610.03295 (2016).
  * [26] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy.  Deep exploration via bootstrapped dqn.  In Advances in neural information processing systems, pages 4026–4034, 2016.
  * [27] Daniel Dewey.  Reinforcement learning and the reward engineering principle.  2014\.
  * [28] Carlos E Garcia, David M Prett, and Manfred Morari.  Model predictive control: Theory and practice—a survey.  Automatica, 25(3):335–348, 1989.
  * [29] Firstname Lastname and Firstname Lastname.  Towards sample efficient reinforcement learning.  In Proceedings of the Conference Name, pages start page–end page. Organizer, Year.
  * [30] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel.  Model-ensemble trust-region policy optimization.  preprint arXiv:1802.10592 (2018).
  * [31] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135, 2017.
  * [32] Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael I. Jordan.  Tactical optimism and pessimism for deep reinforcement learning.  In Advances in Neural Information Processing Systems 34, December 6-14, 2021, virtual, pages 12849–12863, 2021.
  * [33] Fan Li, Mingsheng Fu, Wenyu Chen, Fan Zhang, Haixian Zhang, Hong Qu, and Zhang Yi.  Improving exploration in actor–critic with weakly pessimistic value estimation and optimistic policy optimization.  IEEE Transactions on Neural Networks and Learning Systems, 2022.
  * [34] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine.  How to train your robot with deep reinforcement learning: lessons we have learned.  The International Journal of Robotics Research, 40(4-5):698–721, 2021.
  * [35] Scott Fujimoto, Herkevan Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  preprint arXiv: Artificial Intelligence (2018).
  * [36] Yi Zhang, Xiaohan Shi, Hengxu Zhang, Yongji Cao, and Vladimir Terzija.  Review on deep learning applications in frequency analysis and control of modern power system.  International Journal of Electrical Power & Energy Systems, 136:107744, 2022.
  * [37] Martina Stadler, Jacopo Banfi, and Nicholas Roy.  Approximating the value of collaborative team actions for efficient multiagent navigation in uncertain graphs.  2023\.
  * [38] Rasmus Bruckner, Hauke R Heekeren, and Dirk Ostwald.  Belief states and categorical-choice biases determine reward-based learning under perceptual uncertainty.  bioRxiv, pages 2020–09, 2020.
  * [39] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann.  Better exploration with optimistic actor-critic.  preprint arXiv: Machine Learning (2019).
  * [40] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  2016\.
  * [41] Hadovan Hasselt.  Double q-learning.  2010\.
  * [42] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel.  Vime: Variational information maximizing exploration.  In Advances in Neural Information Processing Systems, pages 1109–1117, 2016.
  * [43] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In International Conference on Machine Learning, pages 1889–1897, 2015.
  * [44] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  preprint arXiv:1509.02971 (2015).
  * [45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  preprint arXiv: Learning (2017).
  * [46] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  2012\.
  * [47] Yuval Tassa, Yotam Doron, Muldal Alistair, Tom Erez, Yazhe Li, DiegodeLas Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, TimothyP. Lillicrap, and Martin Riedmiller.  Deepmind control suite.  preprint arXiv: (2018).
  * [48] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  In International Conference on Learning Representations, 2018.
  * [49] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta.  Self-supervised exploration via disagreement.  In International Conference on Machine Learning, 2019.
  * [50] Yuval Tassa, Yotam Doron, A. Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Tim Lillicrap, and Martin Riedmiller.  Deepmind control suite.  In Conference/Workshop where it was published, 2018.
  * [51] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In International Conference on Machine Learning, pages 1856–1865, 2018.
  * [52] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross.  Randomized ensembled double q-learning: Learning fast without a model.  2021\.
  * [53] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  preprint arXiv: Learning (2016).
  * [54] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto.  Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning.  preprint arXiv:2201.13425.
  * [55] Lukas Schafer, Filippos Christianos, Josiah P. Hanna, and Stefano V. Albrecht.  Decoupled reinforcement learning to stabilise intrinsically-motivated exploration.  Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2:1146 – 1154, 2022.
  * [56] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  preprint arXiv:1506.02438 (2018).

##  Appendix A Performance Comparison Diagram of Each Model in DMControl
Environment.

Our algorithm demonstrates superior performance compared to the original TD3
model across all test environments, with particularly notable achievements in
the Swimmer swimmer6 and Swimmer swimmer15 scenarios. In these specific
environments, our model exhibits significantly enhanced performance when
contrasted with the original TD3 model.

(a) Walker walk

(b) Cheetah run

(c) Cartpole balance sparse

(d) Cartpole swingup

(e) Finger spin.

(f) Fish swim

(g) Fish upright

(h) Hopper stand

(i) Pendulum swingup

(j) Point mass easy

(k) Reacher easy

(l) Reacher hard

(m) Swimmer swimmer6

(n) Swimmer swimmer15

(o) Walker run

(p) Walker stand

Figure 5: Performance curves for OpenAI gym continuous control tasks on Deep-
Mind Control Suite. The shaded region represents half of the standard
deviation of the average evaluation over 10 seeds. The curves are smoothed
with a moving average window of size ten visual clarity.

##  Appendix B Detailed Description of The Environment

(a) Ant

(b) Hopper

(c) Walker2d

(d) Humanoid

Ant: The environment is modeled after the one introduced by Schulman, Moritz,
Levine, Jordan, and Abbeel in their work[56]. The ant model is a quadruped
robot, featuring a freely rotating torso with four attached legs, each leg
comprising two links. The primary objective is to achieve coordinated forward
(rightward) movement by applying torques to eight hinges that connect the leg
links and the torso, encompassing nine parts and eight hinges in total. This
environment allows versatile movement simulations within its framework. The
observation space includes positional and velocity information for various
parts of the ant’s body. Rewards in this environment are structured around
maintaining a healthy state, moving forward, and minimizing control costs.

Hopper:The environment is predicated on the foundational work of Erez, Tassa,
and Todorov in ”Infinite Horizon Model Predictive Control for Nonlinear
Periodic Tasks.” The aim of this environment is to augment the quantity of
independent state and control variables in contrast to classical control
settings. The hopper robot is depicted as a two-dimensional monopodal entity,
comprising four primary bodily segments: an upper torso, a thigh situated
midsection, a lower leg beneath the thigh, and a solitary foot that serves as
the foundational support for the entire structure. The objective is to execute
forward (rightward) hopping maneuvers by exerting torques upon the three
hinges that interconnect the quadruple body segments. The model’s overarching
goal is to facilitate effective hopping and balance, whilst simultaneously
optimizing the distance progressed forward and forestalling any potential
collapse.

Walker2d: The environment extends the hopping robot scenario introduced by
Erez, Tassa, and Todorov in ”Infinite Horizon Model Predictive Control for
Nonlinear Periodic Tasks” by incorporating an additional set of limbs, thereby
enabling the robot to achieve forward locomotion rather than mere hopping.
This adaptation aligns with the broader aim of MuJoCo-based environments to
expand the repertoire of independent state and control variables relative to
classical control environments. The Walker robot is conceptualized as a bi-
dimensional bipedal entity, structurally segmented into four principal
components: a singular torso serving as the upper segment from which two legs
diverge; two thighs situated below the torso; two lower legs beneath the
thighs; and a pair of feet affixed to the lower legs, forming the base for the
entire structure. The primary objective is the calibrated manipulation of both
pairs of feet, legs, and thighs, employing torques across six hinges
interlinking the sextet of body segments, to facilitate a concerted forward
(rightward) progression. The Walker model shares similarities with the Hopper
in terms of its action space, which comprises torques applied across joints,
with the overarching goal of accomplishing steady bipedal traversal and
maximizing the distance covered in the forward trajectory.

Humanoid:The environment is predicated on the groundwork laid by Tassa, Erez,
and Todorov in ”Synthesis and Stabilization of Complex Behaviors through
Online Trajectory Optimization.” The Humanoid model is an intricate 3D bipedal
robot, devised to emulate human locomotion and kinetics. It comprises a torso
(abdomen) with appendages articulated as a pair of legs and arms. Each limb
bifurcates into two segments, akin to human knees and elbows. The principal
aim of the environment is to navigate forward at the highest possible velocity
while maintaining equilibrium. The action space encompasses the modulation of
torques across multiple bodily articulations, while the observation space
encompasses the positional and velocity data of the respective body segments.
The model presents the challenge of executing efficient bipedal ambulation and
a repertoire of other complex maneuvers.

##  Appendix C Detailed Description of The Comparison Algorithm

Algorithm 2 TD3

0: Initialize critic networks $Q_{\theta_{1}}$, $Q_{\theta_{2}}$, and actor
network $\pi_{\phi}$ with random parameters $\theta_{1}$, $\theta_{2}$, $\phi$

0: Initialize target networks $\theta^{0}_{1}\leftarrow\theta_{1}$,
$\theta^{0}_{2}\leftarrow\theta_{2}$, $\phi^{0}\leftarrow\phi$

0: Initialize replay buffer $\mathcal{B}$

1: for $t=1$ to $T$ do

2: Select action with exploration noise $a\sim\pi_{\phi}(s)+\epsilon$,
$\epsilon\sim\mathcal{N}(0,\sigma)$

3: Observe reward $r$ and new state $s^{\prime}$

4: Store transition tuple $(s,a,r,s^{\prime})$ in $\mathcal{B}$

5: Sample mini-batch of $N$ transitions $(s,a,r,s^{\prime})$ from
$\mathcal{B}$

6: $\tilde{a}\leftarrow\pi_{\phi^{0}}(s^{\prime})+\epsilon$,
$\epsilon\sim\text{clip}(\mathcal{N}(0,\tilde{\sigma}),-c,c)$

7: $y\leftarrow r+\gamma\min_{i=1,2}Q_{\theta^{0}_{i}}(s^{\prime},\tilde{a})$

8: Update critics
$\theta_{i}\leftarrow\arg\min_{\theta_{i}}N^{-1}\sum(y-Q_{\theta_{i}}(s,a))^{2}$

9: if $t\mod d=0$ then

10: Update $\phi$ by the deterministic policy gradient:

11:
$\nabla_{\phi}J(\phi)=N^{-1}\sum\nabla_{a}Q_{\theta_{1}}(s,a)|_{a=\pi_{\phi}(s)}\nabla_{\phi}\pi_{\phi}(s)$

12: Update target networks:

13: $\theta^{0}_{i}\leftarrow\tau\theta_{i}+(1-\tau)\theta^{0}_{i}$

14: $\phi^{0}\leftarrow\tau\phi+(1-\tau)\phi^{0}$

15: end if

16: end for

Algorithm 3 Soft Actor-Critic

0: Initialize parameter vectors $\theta_{\mathrm{v}}$,
$\theta^{\prime}_{\mathrm{v}}$, $\theta$, $\phi$.

1: for each iteration do

2: for each environment step do

3: $\mathbf{a}_{t}\sim\pi_{\phi}(\mathbf{a}_{t}|\mathbf{s}_{t})$

4: $\mathbf{s}_{t+1}\sim p(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t})$

5:
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(\mathbf{s}_{t},\mathbf{a}_{t},r(\mathbf{s}_{t},\mathbf{a}_{t}),\mathbf{s}_{t+1})\\}$

6: end for

7: for each gradient step do

8:
$\theta_{\mathrm{v}}\leftarrow\theta_{\mathrm{v}}-\lambda_{V}\hat{\nabla}_{\theta_{\mathrm{v}}}J_{V}(\theta_{\mathrm{v}})$

9:
$\theta_{i}\leftarrow\theta_{i}-\lambda_{Q}\hat{\nabla}_{\theta_{i}}J_{Q}(\theta_{i})$
for $i\in\\{1,2\\}$

10: $\phi\leftarrow\phi-\lambda_{\pi}\hat{\nabla}_{\phi}J_{\pi}(\phi)$

11:
$\theta^{\prime}_{\mathrm{v}}\leftarrow\tau\theta_{\mathrm{v}}+(1-\tau)\theta^{\prime}_{\mathrm{v}}$

12: end for

13: end for

Algorithm 4 PPO, Actor-Critic Style

1: for $iteration=1,2,\ldots$ do

2: for $actor=1,2,\ldots,N$ do

3: Run policy $\pi_{\theta_{\text{old}}}$ in environment for $T$ timesteps

4: Compute advantage estimates $\hat{A}_{1},\ldots,\hat{A}_{T}$

5: end for

6: Optimize surrogate $L$ wrt $\theta$, with $K$ epochs and minibatch size
$M\leq NT$

7: $\theta_{\text{old}}\leftarrow\theta$

8: end for

Algorithm 5 Randomized Ensembled Double Q-learning (REDQ)

0: Initialize policy parameters $\theta$,  $N$ Q-function parameters
$\phi_{i}$, $i=1,\ldots,N$, empty replay buffer $\mathcal{D}$. Set target
parameters $\phi_{\text{targ},i}\leftarrow\phi_{i},\text{ for }i=1,2,\ldots,N$

1: Repeat

2: Take one action $a_{t}\sim\pi_{\theta}(\cdot|s_{t})$. Observe reward
$r_{t}$, new state $s_{t+1}$.

3: Add data to buffer:
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{(s_{t},a_{t},r_{t},s_{t+1})\\}$

4: for $G$ updates do

5: Sample a mini-batch $B=\left\\{\left(s,a,r,s^{\prime}\right)\right\\}$ from
$\mathcal{D}$

6: Sample a set $\cal{M}$ of  $M$ distinct indices from $\\{1,2,\ldots,N\\}$

7: Compute the Q target $y$ (same for all of the $N$ Q-functions):
$y=r+\gamma\left(\min_{i\in\cal{M}}Q_{\phi_{\text{targ},i}}\left(s^{\prime},\tilde{a}^{\prime}\right)-\alpha\log\pi_{\theta}\left(\tilde{a}^{\prime}\mid
s^{\prime}\right)\right),\quad\tilde{a}^{\prime}\sim\pi_{\theta}\left(\cdot\mid
s^{\prime}\right)$

8: for $i=1,\ldots,N$ do

9: Update $\phi_{i}$ with gradient descent using
$\nabla_{\phi}\frac{1}{|B|}\sum_{\left(s,a,r,s^{\prime}\right)\in
B}\left(Q_{\phi_{i}}(s,a)-y\right)^{2}$

10: Update target networks with
$\phi_{\operatorname{targ},i}\leftarrow\rho\phi_{\operatorname{targ},i}+(1-\rho)\phi_{i}$

11: end for

12: end for

13: Update policy parameters $\theta$ with gradient ascent using
$\nabla_{\theta}\frac{1}{|B|}\sum_{s\in
B}\left(\frac{1}{N}\sum_{i=1}^{N}Q_{\phi_{i}}\left(s,\tilde{a}_{\theta}(s)\right)-\alpha\log\pi_{\theta}\left(\tilde{a}_{\theta}(s)|s\right)\right),\quad\tilde{a}_{\theta}(s)\sim\pi_{\theta}(\cdot\mid
s)$

##  Appendix D The Specific Parameter Setting of OPARL Algorithm

TABLE II: Hyperparameters in experiment. Hyperparameter | Value
---|---
Qnum(The number of ensemble Q’s) | 5
start-timesteps(Time steps initial random policy is used) | 25000
max-timesteps(Max time steps to run environment) | 1000000
expl-noise(Std of Gaussian exploration noise) | 0.1
discount(Discount factor) | 0.99
tau(Target network update rate) | 0.005
policy-noise(Noise added to target policy during critic update) | 0.2
noise-clip(Range to clip target policy noise) | 0.5
policy-freq(Frequency of delayed policy updates) | 2
policy-utd(The ratio of the number of training sessions to the number of interactions with the environment) | 1
policy-interval(The number of interactions with the environment per exploration) | 1
N(Enter the number of actions generated by a state) | 1
bili(The ratio of optimistic exploration to pessimistic exploration) | 2