# LLQL: Logistic Likelihood Q-Learning for Reinforcement Learning

Outongyi Lv
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Bingxin Zhou
Shanghai Jiao Tong University
<EMAIL_ADDRESS>

###### Abstract

Modern Reinforcement learning (RL) can be categorized into online and offline
variants. As a pivotal aspect of both online and offline RL, current research
on the Bellman equation revolves primarily around optimization techniques and
performance enhancement rather than exploring the inherent structural
properties of the Bellman error, such as its distribution characteristics.
This study investigates the distribution of the Bellman approximation error in
both online and offline settings through iterative exploration of the Bellman
equation. We observed that both in online RL and offline RL, the Bellman error
conforms to a _Logistic distribution_. Building upon this discovery, this
study employed the Logistic maximum likelihood function ($\rm LLoss$) as an
alternative to the commonly used MSE Loss, assuming that Bellman errors adhere
to a normal distribution. We validated our hypotheses through extensive
numerical experiments across diverse online and offline environments. In
particular, we applied corrections to the loss function across various
baseline algorithms and consistently observed that the loss function with
Logistic corrections outperformed the MSE counterpart significantly.
Additionally, we conducted Kolmogorov–Smirnov tests to confirm the reliability
of the Logistic distribution. This study’s theoretical and empirical insights
provide valuable groundwork for future investigations and enhancements
centered on the distribution of Bellman errors.

##  1 Introduction

Modern Deep Reinforcement Learning (RL) has witnessed remarkable advancements
in diverse applications, encompassing strategy games (Mnih et al., 2013;
Kaiser et al., 2019) to Capacitated Vehicle Routing Problem (CVRP) problems
(Kwon et al., 2020; Hottung et al., 2021; Bi et al., 2022). RL operates by
guiding an agent to actively interact with an environment through a series of
actions, aiming to maximize the expectation of rewards over time. The
cumulative reward concerning the current state is captured by the _Bellman
equation_ (Bellman, 1954). Although the Bellman equation’s recursive nature
theoretically guides conventional RL towards optimal or near-optimal
solutions, its computational demands raise concerns, especially when dealing
with extensive state and action spaces (Patterson et al., 2022). In the realm
of online RL, _Soft Actor Critic_ (SAC), introduced by Haarnoja et al. (2018a;
b), incorporates the soft Bellman operator to enhance the overall reward and
improve model performance and stability, catalyzing significant advancements
in RL techniques (Christodoulou, 2019; Ward et al., 2019). On a parallel
front, Kumar et al. (2020) found that offline RL has underscored concerns
regarding substantial overestimations in action (Q-value) estimations. This
insight prompted subsequent developments of the Conservative Q-Learning (CQL)
framework, sparking renewed interest in refining offline RL methodologies
(Bayramoğlu et al., 2021; Lyu et al., 2022; Kostrikov et al., 2021; Garg et
al., 2023).

The conventional practice of employing Bellman equations for Q-iterations has
gradually waned in modern RL discourse. Instead, a shift of preference has
been observed in updating the iterative Q-function with the maximum-entropy
policy to ensure robust modeling and mitigate estimation errors using the
Bellman operator (Ziebart, 2010). SAC deploys an auxiliary policy network to
circumvent intractable estimations over log-partitioned Q-values. More
recently, Extreme Q-Learning (XQL) (Garg et al., 2023) defines a novel sample-
free objective towards optimal soft-value functions in the maximum entropy RL
setting, thereby obviating the necessity for conventional network iterations.
These frameworks mark a significant departure from established practices and
offer exciting prospects for advancements in RL optimization techniques
(Hansen-Estruch et al., 2023; Hejna & Sadigh, 2023).

Figure 1: The evolving distributions of Bellman error and $V$ loss computed by
equation 8 and 13 that using all data from the Reply-Buffer at different
epochs (100, 200, 400, 700 and final) of online RL training on
LunarLanderContinuous-v2.

In parallel with the evolution of optimization techniques, researchers have
exhibited a substantial interest in minimizing the _Bellman error_ (Baird,
1995), a metric denoting the disparity between the current state-action value
estimation and the value outlined by the Bellman equation. The objective is to
precisely represent the value function of state-action pairs under the current
policy (Geist et al., 2017). Following these efforts, researchers have strived
to modify the objective function (Feng et al., 2019; Dai et al., 2018;
Fujimoto et al., 2022) or optimize the update rules (Bas-Serrano et al., 2021;
Gong et al., 2020) to minimize the Bellman error. However, despite various
attempts to achieve an adequate policy by indirectly addressing the
distribution of the Bellman error, there lacks a straightforward analysis of
the main properties of the Bellman error, particularly in terms of exploring
more suitable error distributions beyond the normal distribution.

To the best of our knowledge, this study presents the first comprehensive
exploration rooted in the Logistic distribution of the Bellman error. Drawing
inspiration from Garg et al. (2023), we define the _Gumbel error_ to depict
the gap between the estimation and true values of the $Q$ function, which
yields a Logistic distribution character for the Bellman error within the
realm of online RL. We rigorously derived that under certain conditions, the
Bellman error is inherently a biased estimate, additionally, there exists an
irreducible, artificially uncontrollable error. In such circumstances, the
selection of an appropriate distribution type becomes crucial, rendering the
traditional assumption of mean square error no longer suitable. This
proposition is in line with Bas-Serrano et al. (2021). We further extend our
examination to the loss of the $V$ function under the online condition and
found that it also conforms to a Logistic distribution. Empirical validation
of our theoretical propositions spans $8$ distinct online environments and $9$
offline environments, encompassing both the empirical distribution of the
Bellman error and the performance of trained networks. The results unveil a
robust preference for the Logistic distribution of Bellman errors within
online RL, surpassing both Gumbel and Gaussian distributions.

##  2 Preliminaries

RL explores the expected cumulative reward by a Markov decision process
defined by a tuple $({\mathcal{S}},{\mathcal{A}},{\mathbb{P}},r,\gamma)$,
where ${\mathcal{S}}$ and ${\mathcal{A}}$ respectively denote the state and
action spaces, ${\mathbb{P}}(\bm{s^{\prime}}|\bm{s,a})$ is the state
transition probability from state $\bm{s}$ that drives toward the next state
$\bm{s^{\prime}}$, $r$ defines the reward of taking an action ${\bm{a}}$ at
the current state ${\bm{s}}$, in state ${\bm{s}}$, the reward obtained by
performing action ${\bm{a}}$ is defined as $r({\bm{s}},{\bm{a}})$,
$\gamma\in(0,1)$ is the discount factor on future rewards. In online RL, an
agent constantly interacts with the environment to enrich the state-action
pair $(\bm{s,a,s^{\prime}},r)$ upon a behavior policy $\pi({\bm{a}}|{\bm{s}})$
with respect to its Reply-Buffer, which is a container for the state-action
pairs and facilitating network training. In contrast, in offline RL, the agent
can not interact with the environment. As a result, the agent relies solely on
an elaborate dataset (comprising a multitude of
$({\bm{s}},{\bm{a}},{\bm{s}}^{\prime},r)$ tuples) for learning, and it is not
able to add any new $({\bm{s}},{\bm{a}},{\bm{s}}^{\prime},r)$ pairs to the
Reply-Buffer.

###  2.1 Objectives in Reinforcement Learning

The target of RL, as defined by _Actor-Critic_ (AC) algorithm (Konda &
Tsitsiklis, 1999), is to find the optimal policy $\pi({\bm{a}}|{\bm{s}})$ that
maximizes the cumulative discounted reward at a fixed time step $T$, i.e.,

$\mathbb{E}_{{\bm{a}}_{t}\sim\pi({\bm{a}}_{t}|{\bm{s}}_{t})}\left[\sum_{t=0}^{T}\gamma^{t}r({\bm{s}}_{t},{\bm{a}}_{t})\right].$

Alternatively, _Soft AC_ (SAC) Haarnoja et al. (2018a; b) encompass soft
conditions in future rewards to facilitate the learning of policy $\pi$ with
the regularization strength $\beta$, i.e.,

$\mathbb{E}_{{\bm{a}}_{t}\sim\pi({\bm{a}}_{t}|{\bm{s}}_{t})}\left[\sum_{t=0}^{T}\gamma^{t}(r({\bm{s}}_{t},{\bm{a}}_{t})-\beta\log(\pi({\bm{a}}_{t}|{\bm{s}}_{t})))\right].$

Later on, Garg et al. (2023) introduces the KL divergence between the policy
$\pi$ and the prior distribution of a reference distribution $\mu$ to augment
the reward function in the objective:

$\mathbb{E}_{{\bm{a}}_{t}\sim\pi({\bm{a}}_{t}|{\bm{s}}_{t})}\left[\sum_{t=0}^{T}\gamma^{t}(r({\bm{s}}_{t},{\bm{a}}_{t})-\beta\log\frac{\pi({\bm{a}}_{t}|{\bm{s}}_{t})}{\mu({\bm{a}}_{t}|{\bm{s}}_{t})})\right].$
(1)

In online RL, $\mu({\bm{a}}|{\bm{s}})$ is typically sampled from a uniform
distribution, while in offline RL, $\mu({\bm{a}}|{\bm{s}})$ is usually sampled
from the empirical distribution of offline data. This approach aims better to
approximate the behavioral policy (Neu et al., 2017).

Equation 1 serves as a special case of the remaining two, which will be
elaborated in Section 2.2.

###  2.2 (Soft) Bellman Equation in RL

In Section 2.1, we noticed that equation 1 is the most general one. Therefore,
we will consider it as the baseline for further analysis. In fact, when the
value of $\mu$ is 1, equation 1 will degenerate back to SAC, and when the
value of $\beta$ is 0, equation 1 will degenerate back to AC.

We thus focus on studying the general form in equation 1. The goal is to
obtain the optimal Bellman iterative equation, which has been widely used in
$Q$-learning (Watkins & Dayan, 1992):

$\displaystyle Q^{t+1}({\bm{s}},{\bm{a}})$
$\displaystyle=r({\bm{s}},{\bm{a}})+\gamma\max_{{\bm{a}}^{\prime}}(Q^{t}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$
(2)

This objective is obtained through the Bellman iterative operator,
specifically the soft Bellman iteration has a general version corresponding to
equation 1 is:

$Q^{k+1}({\bm{s}},{\bm{a}})\leftarrow\operatorname*{arg\,min}_{Q}\biggl{[}r({\bm{s}},{\bm{a}})+\mathbb{E}_{{\bm{s}}^{\prime},{\bm{a}}^{\prime}\sim\pi}\left[Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})-\beta\log\frac{\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}{\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}\right]-Q^{k}({\bm{s}},{\bm{a}})\biggr{]}^{2},$

The soft Bellman iteration can be solved from equation 2.2:

$Q^{k+1}({\bm{s}},{\bm{a}})=r({\bm{s}},{\bm{a}})+\mathbb{E}_{{\bm{s}}^{\prime},{\bm{a}}^{\prime}\sim\pi}\biggl{[}Q^{k}({\bm{s}}^{\prime},{\bm{a}}^{\prime})-\beta\log\frac{\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}{\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}\biggr{]}.$
(3)

If we want to take the optimal strategy (corresponding to the $\max(\cdot)$
operator in equation 2, we need to find an optimal $\pi^{*}$ which satisfies:

$\pi^{*}({\bm{a}}^{\prime}|{\bm{s}}^{\prime})=\operatorname*{arg\,max}_{\pi}(\mathbb{E}_{{\bm{s}}^{\prime},{\bm{a}}^{\prime}\sim\pi}\left[Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})-\beta\log\frac{\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}{\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}\right]),$
(4)

where
$\sum_{{\bm{a}}^{\prime}}\pi^{*}({\bm{a}}^{\prime}|{\bm{s}}^{\prime})=1$.
Using the Lagrange multiplier method, we can easily get the following:

$\pi^{*}({\bm{a}}|{\bm{s}})=\frac{\mu({\bm{a}}|{\bm{s}})e^{Q({\bm{s}},{\bm{a}})/\beta}}{\sum_{\bm{a}}\mu({\bm{a}}|{\bm{s}})e^{Q({\bm{s}},{\bm{a}})/\beta}}.$
(5)

The details for deriving can be found in Appendix A.5, it is a general
conclusion. If we take equation 5 to equation 4, then equation 4 can be
further simplified as:

$\mathbb{E}_{{\bm{s}}^{\prime},{\bm{a}}^{\prime}\sim\pi}\left[Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})-\beta\log\frac{\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}{\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}\right]\rightarrow\mathbb{E}_{{\bm{s}}^{\prime}}\biggl{[}\beta\log\sum_{{\bm{a}}^{\prime}}\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})e^{Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})/\beta}\biggr{]}.$
(6)

Hence, the max component:
$\max_{{\bm{a}}^{\prime}}Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})$ of equation 2
is reflected in equation 6, this means if we take the best policy $\pi^{*}$ ,
then we will have:

$\max_{{\bm{a}}^{\prime}}Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})=\beta\log\sum_{{\bm{a}}^{\prime}}\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})e^{Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})/\beta}.$
(7)

Garg et al. (2023) found that estimating the log sum part in equation 7
through sampling is challenging. Thus, they employed a regression-based
approach to circumvent the need for sampling estimation.

##  3 Characterization of Bellman Error with Logistic Distribution

###  3.1 Bellman Error Analysis

Firstly, we establish the probability density functions (PDF) of the Gumbel
distribution and the Logistic distribution, where ${\rm
Gumbel}(x;\mu,\beta)=\frac{1}{\beta}\exp[{-[\frac{x-\mu}{\beta}+\exp[{-\frac{x-\mu}{\beta}}]]}]$,
and $\rm
Logistic(x;\mu,\beta)=\frac{1}{\beta}\frac{exp[{-(x-\mu)}/{\beta}]}{(1+\exp[{-(x-\mu)}/{\beta}])^{2}}$,
and $X\sim\rm Gumbel(\mu,\beta)$ (or $\rm Logistic(\mu,\beta)$) indicates $X$
follows a Gumbel (or Logistic) distribution with parameters $\mu$ and $\beta$.

In this paper, we define the online Bellman error as $\varepsilon^{\theta}$
under the $Q$ network parameter $\theta$ and policy $\pi$ :

$\varepsilon^{\theta}({\bm{s}},{\bm{a}})=\biggl{[}r({\bm{s}},{\bm{a}})+\gamma_{{\bm{a}}^{\prime}\sim\pi}\hat{Q}_{\theta}({\bm{s}}^{\prime},{\bm{a}}^{\prime})\biggr{]}-\hat{Q}_{\theta}({\bm{s}},{\bm{a}}).$
(8)

where $\hat{Q}_{\theta}$ means the estimated $Q$ network with parameter
$\theta$.

For the purpose of theoretical analysis, we consider a deterministic
environment, meaning that each $({\bm{s}},{\bm{a}})$ pair uniquely determines
the subsequent state ${\bm{s}}^{\prime}$, while disregarding the influence of
state transition probabilities. This simplification enhances the analysis of
error propagation and is more practical for real-world applications.

First, let’s consider the Bellman error up to the $t$-th iteration. Similar to
Garg et al. (2023), we also think the cause for the Bellman error is the
deviation between the estimated $Q$ value and the real $Q$ value in each
iteration :

$\hat{Q}^{t}({\bm{s}},{\bm{a}})=\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{t}({\bm{s}},{\bm{a}}).$
(9)

$\overline{Q}$ represents the one without any gaps. We believe the real $Q$
value will not appear any error during the max iterative process, which means:

$\displaystyle\overline{Q}({\bm{s}},{\bm{a}})$
$\displaystyle=r({\bm{s}},{\bm{a}})+\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

While it is generally impossible to access the real $\overline{Q}$, we only
have the estimated $\hat{Q}$. In general, we initialize a
$\hat{Q}^{0}({\bm{s}},{\bm{a}})$ for iterating, whether through tabular
methods or neural networks, with a random initialization. The approximated
$\hat{Q}^{t}({\bm{s}},{\bm{a}})$ value could be approached by this iteration:

$\displaystyle\hat{Q}^{t+1}({\bm{s}},{\bm{a}})$
$\displaystyle=r({\bm{s}},{\bm{a}})+\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{t}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

In order to further explore their relationship with Bellman error , we derived
these Lemmas for Theorem 1.

######  Lemma 1.

(Fisher & Tippett, 1928) For i.i.d. random variables $X_{1},\dots,X_{n}\sim
f(X)$, $f(X)$ has the exponential tails, then
$\lim_{n\to\infty}max_{i}(X_{i})$ follows the Gumbel distribution.

######  Lemma 2.

$X$ follow the Gumbel distribution ${\rm Gumbel}(A,B)$, $C,D$ are both
constant, $D>0$. $Y=X+C$ will follow ${\rm Gumbel}(C+A,B)$, $Z=DX$ will follow
${\rm Gumbel}(DA,DB)$.

######  Lemma 3.

For mutually independent random variables $X_{1},\dots,X_{n}$, which satisfies
$X_{i}\sim{\rm Gumbel}(C_{i},\beta)$. Then $\max_{i}(X_{i})\sim{\rm
Gumbel}({\beta}\ln\sum_{i}e^{\frac{1}{\beta}C_{i}},\beta)$.

Lemma 1, Lemma 2 and Lemma 3 describe the basic properties of the Gumbel
distribution, which are important and necessary. Lemma 4 gives the
distribution analysis for $\epsilon({\bm{s}},{\bm{a}})$ which has been defined
in equation 8. The proof for these facts can be found in Appendix A.1,A.2.

Next, we will propose the Lemma 4. The following presents the relationship
between $\overline{Q}$ and $\hat{Q}$.

######  Lemma 4.

For $\epsilon^{t}({\bm{s}},{\bm{a}})$, which has been defined in equation 9,
if the following three assumptions can be satisfied:

1\. The action space is infinity, and it is a countable set.

2\. There is a injection mapping from pair $({\bm{s}},{\bm{a}})$ to the next
state ${\bm{s}}^{\prime}$, which means the next state is only determined by
the current state ${\bm{s}}$ and action ${\bm{a}}$.

3\. for any $({\bm{s}},{\bm{a}})$ pair, $\hat{Q}^{0}({\bm{s}},{\bm{a}})$ come
from the same exponential tail distribution $P$ ( for example $N(0,1)$ is
feasible ).

Under these hypothesis,
$\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))$
will follow the Gumbel distribution, if we suppose:

$\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))\sim
Gumbel(C_{1},\beta_{1}).$

Then for each $t$, $\epsilon^{t}({\bm{s}},{\bm{a}})$ will follow the $\rm
Gumbel$ distribution, actually, we will have:

$\epsilon^{t}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{t}({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{t}).$

Where:

$C_{1}({\bm{s}},{\bm{a}})=C_{1}$
$C_{2}({\bm{s}},{\bm{a}})=\gamma(C_{1}({\bm{s}},{\bm{a}})+\beta_{1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{1}}}).$
$C_{t}({\bm{s}},{\bm{a}})=\gamma(\beta_{t-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{t-1}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{t-1}}})(t\geq
3).$ $\beta_{t}=\gamma^{t-1}\beta_{1}.$

Additional, Special Situation Discussion: If for $\forall s_{1},s_{2}$, we
have two set:

$S_{1}=[r({\bm{s}}_{1},{\bm{a}}_{1}),r({\bm{s}}_{1},{\bm{a}}_{2}),...,r({\bm{s}}_{1},{\bm{a}}_{n})...]$
$S_{2}=[r({\bm{s}}_{2},{\bm{a}}_{1}),r({\bm{s}}_{2},{\bm{a}}_{2}),...,r({\bm{s}}_{2},{\bm{a}}_{n})...]$

and

$S_{1}\backslash S_{2}=\emptyset.$

Then in this case, we have:

$\epsilon^{t}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{t}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{t}).$

Where

$C_{t}=\gamma(C_{t-1}+\beta_{t-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{t-1}}})(t\geq
2).$

This Lemma reveals that the error between the true $Q$ and estimated $Q$ is
complex. This Lemma also give a good foundation for proving Theorem 1. You can
find the proof in Appendix A.3.

######  Lemma 5.

For i.i.d. random variables $X\sim{\rm Gumbel}(C_{X},\beta)$, $Y\sim{\rm
Gumbel}(C_{Y},\beta)$, then $(X-Y)\sim{\rm Logistic}(C_{X}-C_{Y},\beta)$.

The following Theorem 1 gives a discussion about the Bellman error
distribution, it may come as a surprise that this error is not unbiased, but a
biased estimation:

######  Theorem 1.

(Logistic distribution for Bellman error): During network training,
$\varepsilon^{\theta}({\bm{s}},{\bm{a}})$, which defined by equation 8, assume
that it also represents that the network has been updated $(\theta)$ times. If
the assumptions of Lemma 4 are all satisfied, then
$\varepsilon^{\theta}({\bm{s}},{\bm{a}})$ will follow the Logistic
distribution, actually:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(C_{\theta}({\bm{s}},{\bm{a}})-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

Besides, if we consider the Special Situation in Lemma 2, then:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

The detailed proof of Theorem 1 is in Appendix A.4. According to Theorem 1,
when we want to optimize this Bellman error, we certainly hope that the
expectation of Bellman error is as close to zero as possible, i.e.,

$\mathbb{E}\biggl{[}{\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\biggr{]}\to
0,$

Theorem 1 told us that this distribution will continue to change from
”flatted” to ”elongated” because the variance will continue shrinking during
updating, and the center of the distribution will be offset to $0$, but this
does not mean the error is unbiased. From Theorem 1, we can see that the
center of the initial error distribution will deviate greatly from 0, and the
variance will be vary large, as the iteration proceeds, it will continue to
shift towards $0$ and become taller, This is consistent with what is shown in
Figure 1.

But during network updating, we always use the $\rm MSELoss$, which is used
based on the assumption of a Normal distribution $N(0,\sigma)$. This would be
very unreasonable and theoretically unfounded. In fact, in most deep RL
networks, such a $\rm MSELoss$ function is commonly used when updating a batch
of tuple $({\bm{s}},{\bm{a}},r,{\bm{s}}^{\prime})$ from the Reply-Buffer. $\rm
MSELoss$ is derived from the likelihood function:

$\log[\prod_{i=1}^{n}p({\rm\varepsilon}_{i})]=-n\log(\sqrt{2\pi}\sigma)-\sum_{i=1}^{n}\frac{{\rm\varepsilon}^{2}_{i}}{2\sigma^{2}}\\\
\propto\sum_{i=1}^{n}-\frac{1}{2}({\rm\varepsilon}_{i})^{2}.$ (10)

Using a similar approach, we will discuss the replacement of $\rm MSELoss$
with regard to likelihood functions in Section 3.3.

###  3.2 Distribution of the V Loss in Online RL

As mentioned in Section 2.2, we start the analysis from the perspective of the
policy $\pi$ by employing the optimal soft Bellman operator in SAC (Haarnoja
et al., 2018b), the parameter $\mu$ is chosen to follow a uniform
distribution, and $\beta$ is set to $1$. It can be observed from equation 5
that:

$\pi^{*}({\bm{a}}|{\bm{s}})=\frac{e^{Q({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{Q({\bm{s}},{\bm{a}})}}.$
(11)

When training the policy network with parameter $\delta$: $\pi^{\delta}$, our
objective is to allow the policy to learn this distribution as closely as
possible, hence employing the Kullback-Leibler (KL) divergence as the loss
function for policy loss:

$Loss(\pi^{\delta})={\rm
KL}\left(\pi^{\delta}(a|s)||\frac{e^{Q({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{Q({\bm{s}},{\bm{a}})}}\right).$

######  Theorem 2.

(The relationship between the estimated policy and the true policy): There
exists a gap between the true policy and the estimated policy under the
parameter $\theta$ , which related on whether equation 9 is accurately
estimated, in fact:

$\hat{\pi}(a|s)=\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}e^{\epsilon^{\theta}({\bm{s}},{\bm{a}})}\overline{\pi}(a|s).$
(12)

From this, we can derive the distribution for the $V$ function in SAC,
according to the theory proposed by Haarnoja et al. (2018b), we often
approximate the loss function for calculating the value function $V$ in the
network as follows:

${\rm
Loss}^{V}({\bm{s}})=\biggl{[}\hat{V}({\bm{s}})-\left[\hat{Q}_{\theta}({\bm{s}},{\bm{a}})-\log(\hat{\pi}({\bm{a}}|{\bm{s}}))\right]\biggr{]}.$
(13)

This implies employing the $V$ network to approximate $Q(s,a)-\log(\pi(a|s))$.
However, due to $Q(s,a)$ and $\pi$ not being the ground truth, this introduces
an error term into the estimation. Let us assume this error term to be
$\epsilon^{V}(s)$ and we call it $V$ Loss.

######  Theorem 3.

(Logistic distribution for V Loss function): $V$ Loss will asymptotically
follow a Logistic distribution related to
$\epsilon^{\theta}({\bm{s}},{\bm{a}})$, in fact:

$\displaystyle\epsilon^{V}(s)\sim{\rm
Logistic}\left(-\log\left(\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}\right),\beta_{1}\right).$
(14)

and we will have:

$\displaystyle\epsilon^{V}(s)\stackrel{{\scriptstyle\epsilon^{\theta}({\bm{s}},{\bm{a}})\to
0}}{{\longrightarrow}}{\rm Logistic}\left(0,\beta_{1}\right).$ (15)

Therefore, for the $V$ network, Logistic regression should still be used
instead of Normal regression in the iteration process. In Section 3.3, we will
use the likelihood function to provide an alternative for $\rm MSELoss$
function for $Q$ and $V$ iteration, which is called $\rm LLoss$.

Figure 2: The distribution for Bellman error for two online and offline
environments, and $V$ Loss for only online parts during the half of the
training epochs.

In Figure 1, we depict the evolving trajectories and fitted curves of Bellman
error and $V$ Loss distributions over online training time in
LunarLanderContinuous-v2 (for scenarios at 100, 200, 400, 700 time steps, and
the final training). From this, it is evident that the distributions gradually
become sharper and exhibit a strong fit to the Logistic distribution. The
graphical representation of the evolving offline Bellman error distribution
over training time can be found in Appendix C.2.

In Figures 2, we present the graphical depiction of optimal parameter values
for partial Normal, Gumbel, and Logistic distributions under the evaluation of
goodness-of-fit tests in terms of Bellman error and $V$ Loss for online RL and
only Bellman error for offline RL. Our experimental methodology is identical
to that of XQL Garg et al. (2023). Additionally, we provide the corresponding
Kolmogorov–Smirnov (KS) statistic magnitudes (An, 1933) in Table 3-4 to
provide support for our theory. We conducted testing using 8 distinct online
environments and 9 distinct offline environments(the names of which are
specified in Section 4). Further details can be found in Section 4 and
Appendix C.3.

###  3.3 Logistic Likelihood Q-Learning

In order to show the fairness of Logistic, we use the unbiased Logistic
distribution instead of the Normal distribution, because it is simple and
feasible. Although the theory tells us that it is not biased for Bellman
error. But the bias is not a easy thing to make some additional estimates.
Perhaps this may become a possibility for future research.

Here, we pretend that the error service is assigned from the $\rm
Logistic(0,\sigma)$, not $N(0,\sigma)$. Next, we will provide specific
likelihood functions to seek corresponding loss functions as replacements.

We know that the PDF of the Logistic distribution is:

$p({\rm\varepsilon}_{i})=\frac{1}{\sigma}\frac{e^{\frac{-{\rm\varepsilon}_{i}}{\sigma}}}{(1+e^{\frac{-{\rm\varepsilon}_{i}}{\sigma}})^{2}}.$
(16)

With the log-likelihood function, we can get:

$\log\left[\prod_{i=1}^{n}p({\rm\varepsilon}_{i})\right]=-n\log(\sigma)+\sum_{i=1}^{n}\left[-\frac{{\rm\varepsilon}_{i}}{\sigma}-2\log\left(1+e^{\frac{-{\rm\varepsilon}_{i}}{\sigma}}\right)\right].$
(17)

Finally, we can derive the Logistic Loss function ($\rm LLoss$) as:

${\rm
LLoss}=\frac{1}{N}\sum_{i=1}^{N}\left[\frac{{\rm\varepsilon}_{i}}{\sigma}+2\log\left(1+e^{\frac{-{\rm\varepsilon}_{i}}{\sigma}}\right)\right].$
(18)

Theorem 4 reveals the relationship between $\rm MSELoss$ and $\rm LLoss$.

######  Theorem 4.

(Relationship between $\rm LLoss$ and $\rm MSELoss)$ The $\rm MSELoss$ can be
utilized as an approximate estimation of $\rm LLoss$.

$\rm LLoss=ln4+\frac{1}{2}\rm MSELoss+o(\rm\varepsilon^{3}).$

Theorem 4 demonstrates that the $\rm MSELoss$ can be regarded as an
approximate form of $\rm LLoss$ when higher-order terms are neglected.

##  4 Experiment

Our experiments are all carried out using the control variable method in both
online and offline RL, which means we only change the Loss function from $\rm
MSELoss$ to $\rm LLoss$ to prove the effectiveness of $\rm LLoss$. For online
RL, we employ SAC, and CQL as the baseline algorithm for improvement.
Conversely, for offline RL, because of the inferior performance of SAC in the
offline environment, we opted not to utilize it and instead turned to
employing IQL (Kostrikov et al., 2021), which demonstrated comparatively
superior performance. We also conducted a stability analysis of the parameter
$\sigma$ and conducted respective KS tests on the Bellman error distribution
and $V$ Loss distribution for each environment. We provided detailed
information about the hyperparameters settings used for both online and
offline training in Appendix B.

###  4.1 Experiment Protocol

Our online model was trained for 160,000 iterations in 8 gym environments
while our offline model was trained for 500 iterations in 9 D4RL environments
(Fu et al., 2020). They both have a stopping criterion set to the variance of
Reward remaining below a threshold, where the online threshold is 5 for 1000
epochs, and the offline threshold is 5 for 50 epochs. Our program was executed
under gym (ver.0.23.1), mujoco (ver.2.3.7), and D4RL (ver.1.1).

Figure 3: The average reward variation of SAC, LSAC, and XQL across four
environments in the context of online training. Figure 4: The average reward
variation of IQL, LIQL across four environments in the context of offline
training.

###  4.2 Results Analysis

#### Online RL

We made improvements based on the SAC and CQL algorithms code from their Git
Hub. Our methods are respectively referred to as LSAC and LCQL. We only
replaced $\rm MSELoss$ with $\rm LLoss$ to observe the improvement in
performance. Besides any other settings in the program are the same. This
means that the difference in performance is entirely attributed to the
variation in loss functions. In the comparison with XQL, we fine-tuned XQL
based on the $\beta$ range proposed by its authors. The details regarding the
fine-tuning process and model settings are provided in Appendix B. In our
setting, Most of the environments were run under the condition of 200 max
steps. We trained each environment for 160,000 iterations, we also
incorporated a variance threshold of 5 to determine convergence for 1000
epochs. Based on the rolling epoch timeline, we plotted and stored the Average
Reward for each 2000 epoch. It can be observed from Figure 3 that compared to
$\rm MSELoss$, $\rm LLoss$ demonstrates more prominent performance in both SAC
and CQL. More details for the result and enhancement can be seen in Table 1
and Appendix C.4.

Table 1: The average reward and enhanced ratio after the online training, the
second line indicates the enhancement ratios relative to the baseline
algorithms (LSAC compared to SAC, LCQL compared to CQL).

| SAC | CQL | XQL | LSAC | LCQL
---|---|---|---|---|---
|  |  |  | (Ours) | (Ours)
LunarLander-Continuous (v2) | 19.99 | 104.15 | -489.19 | 133.95 | 112.57
|  |  |  | 570.09 | 8.08
HalfCheetah (v2) | 696.96 | 653.62 | 684.96 | 714.54 | 675.33
|  |  |  | 2.52 | 3.32
Hopper (v4) | 509.47 | 495.34 | 487.08 | 544.72 | 515.30
|  |  |  | 6.92 | 4.03
Walker2d (v2) | 221.46 | 194.59 | 3.42 | 251.63 | 219.27
|  |  |  | 13.62 | 12.68
HumanoidStandup (v4) | 14,157.95 | 14,166.01 | 8,030.26 | 16,781.59 | 14,258.06
|  |  |  | 18.53 | 0.65
InvertedPendulum (v4) | 1001 | 1001 | 1001 | 1001 | 1001
|  |  |  | 0.00 | 0.00
InvertedDouble-Pendulum (v2) | 8466.48 | 4295.46 | 3290.36 | 8941.93 | 4647.64
|  |  |  | 5.62 | 8.20
BipedalWalker (v3) | 68.69 | 43.91 | 64.56 | 77.59 | 71.96
|  |  |  | 12.96 | 63.88
avg. improvement |  |  |  | 78.78 | 12.61

Table 2: The average reward and enhanced ratio after the offline training, all
algorithms have calculated the enhancement ratios relative to the IQL, and it
can be observed that LIQL demonstrates the most significant improvement.

|  |  |  | enhancement over IQL (%)
---|---|---|---|---
|  | IQL | LIQL | LIQL | XQL | CQL | TD3+BC | one-step RL
|  |  |  | (ours) |  |  |  |
medium | hopper-v2 | 228.19 | 240.71 | 5.49 | 7.24 | -11.77 | -10.56 | 2.11
walker2d-v2 | 138.42 | 161.26 | 16.50 | 4.09 | -7.41 | 6.89 | -10.11
halfcheetah-v2 | 319.41 | 335.44 | 5.02 | 0.63 | -7.18 | 1.89 | 4.47
replay | hopper-v2 | 243.92 | 264.30 | 8.35 | 1.36 | 2.94 | 0.90 | -13.80
walker2d-v2 | 153.09 | 176.44 | 15.25 | 2.71 | 4.46 | 10.69 | 2.96
halfcheetah-v2 | 215.28 | 221.31 | 2.80 | 2.75 | 0.31 | -35.70 | -33.02
expert | hopper-v2 | 224.49 | 237.38 | 5.74 | 17.05 | 15.19 | 7.10 | 7.73
walker2d-v2 | 261.75 | 270.15 | 3.21 | 0.46 | -0.73 | 0.45 | 4.61
halfcheetah-v2 | 305.14 | 347.30 | 13.82 | 3.58 | 5.65 | 4.61 | 12.90
avg. improvement |  |  | 8.46 | 4.43 | 0.16 | -1.53 | 3.10

#### Offline RL

We conducted improved experiments based on the IQL components integrated from
rlkit’s Git Hub. We set the maximum step length as 100, and the maximum
iteration count as 500, and incorporated a variance threshold of 5 to
determine convergence for 50 epochs. The method of controlling variables is
the same as in the online setting. Our algorithm is referred to as LIQL. Due
to some dimensional discrepancies between the IQL algorithm provided by rilkit
and the IQL algorithm, we use the improvement ratio relative to the IQL
baseline as the measure of algorithm performance. The change in the average
reward during training is depicted in Figure 4, and relevant details are
presented in Table 2 and Appendix C.4. The results also indicate that our
model exhibits the highest enhancement ratio.

#### KS Tests

We conducted KS tests on the Bellman error for each environment, and for the
online section, we included KS tests for $V$ Loss. The test results are
presented in Table 3 and Table 4. If you want the complete version and
additional details of the validation tables, we provided them and tests for
$V$ Loss in Appendix C.5. The KS tests results indicate that our assumption of
the Logistic distribution is more accurate than the other two distributions.

Table 3: The fitness and KS tests for Bellman error (online).

| R2 $\uparrow$ | KS statistic $\downarrow$
---|---|---
| Logistic | Gumbel | Normal | Logistic | Gumbel | Normal
LunarLanderContinuous-v2 | $0.985$ | $0.971$ | $0.975$ | $0.052$ | $0.070$ | $0.071$
HalfCheetah-v2 | $0.991$ | $0.990$ | $0.989$ | $0.026$ | $0.047$ | $0.033$
Hopper-v4 | $0.989$ | $0.985$ | $0.981$ | $0.067$ | $0.073$ | $0.085$
Walker2d-v2 | $0.988$ | $0.967$ | $0.975$ | $0.054$ | $0.084$ | $0.072$
HumanoidStandup-v4 | $0.667$ | $0.641$ | $0.628$ | $0.269$ | $0.322$ | $0.291$
InvertedPendulum-v4 | $0.983$ | $0.963$ | $0.971$ | $0.115$ | $0.175$ | $0.117$
InvertedDoublePendulum-v4 | $0.999$ | $0.981$ | $0.998$ | $0.021$ | $0.079$ | $0.023$
BipedalWalker-v3 | $0.997$ | $0.979$ | $0.990$ | $0.039$ | $0.101$ | $0.057$

Table 4: The fitness and KS tests for Bellman error (offline).

|  | R2 $\uparrow$ | KS statistic $\downarrow$
---|---|---|---
|  | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal
medium | hopper-v2 | $0.981$ | $0.975$ | $0.976$ | $0.040$ | $0.094$ | $0.053$
walker2d-v2 | $0.927$ | $0.923$ | $0.913$ | $0.062$ | $0.072$ | $0.076$
halfcheetah-v2 | $0.836$ | $0.831$ | $0.833$ | $0.050$ | $0.052$ | $0.069$
replay | hopper-v2 | $0.954$ | $0.927$ | $0.948$ | $0.038$ | $0.104$ | $0.049$
walker2d-v2 | $0.950$ | $0.908$ | $0.937$ | $0.075$ | $0.149$ | $0.093$
halfcheetah-v2 | $0.852$ | $0.813$ | $0.836$ | $0.031$ | $0.105$ | $0.048$
expert | hopper-v2 | $0.985$ | $0.970$ | $0.982$ | $0.056$ | $0.105$ | $0.059$
walker2d-v2 | $0.981$ | $0.959$ | $0.973$ | $0.067$ | $0.138$ | $0.075$
halfcheetah-v2 | $0.919$ | $0.869$ | $0.913$ | $0.036$ | $0.098$ | $0.045$

#### Sensitivity Analysis

We conducted some sensitivity analysis on the variation of $\sigma$ across
different environments in both online and offline settings. We conducted
proportional $\sigma$ variations for each environment to observe the changes
in the final average reward and maximum average reward. More details can be
found in Appendix C.6. The results indicate that the performance of our
approach within a certain range of $\sigma$ variations outperforms the $\rm
MSELoss$ and exhibits a certain level of robustness.

##  5 Conclusion

Our research findings indicate that modifying $\rm MSELoss$ to $\rm LLoss$ is
significantly more effective and straightforward in both online and offline
scenarios, resulting in a noteworthy enhancement of algorithmic performance.
One of our strengths lies in the ease of implementing loss function
modifications within the code. The validation of correctness is accomplished
through the amalgamation of KS tests, clearly underscoring the substantial
potential for future advancements and optimizations by exploring the
fundamental properties of loss functions from a distributional perspective.
However, despite our conducted sensitivity analysis, we maintained the
experimental $\sigma$ value as a constant in our study. We believe that in the
future, similar to the approach of SAC, a more in-depth investigation into the
automatic update of $\sigma$ could greatly enhance the model’s performance.

In addition, we hope to conduct more research on bias estimation to explore
more mysterious properties. A dynamic biased estimator may be a future
direction.

## References

  * An (1933) Kolmogorov An.  Sulla determinazione empirica di una legge didistribuzione.  _Giorn Dell’inst Ital Degli Att_ , 4:89–91, 1933.
  * Baird (1995) Leemon Baird.  Residual algorithms: Reinforcement learning with function approximation.  In _Machine Learning Proceedings 1995_ , pp. 30–37. Elsevier, 1995.
  * Bas-Serrano et al. (2021) Joan Bas-Serrano, Sebastian Curi, Andreas Krause, and Gergely Neu.  Logistic q-learning.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 3610–3618. PMLR, 2021.
  * Bayramoğlu et al. (2021) Öykü Zeynep Bayramoğlu, Engin Erzin, Tevfik Metin Sezgin, and Yücel Yemez.  Engagement rewarded actor-critic with conservative q-learning for speech-driven laughter backchannel generation.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , pp. 613–618, 2021.
  * Bellman (1954) Richard Bellman.  The theory of dynamic programming.  _Bulletin of the American Mathematical Society_ , 60(6):503–515, 1954.
  * Bi et al. (2022) Jieyi Bi, Yining Ma, Jiahai Wang, Zhiguang Cao, Jinbiao Chen, Yuan Sun, and Yeow Meng Chee.  Learning generalizable models for vehicle routing problems via knowledge distillation.  _arXiv:2210.07686_ , 2022.
  * Christodoulou (2019) Petros Christodoulou.  Soft actor-critic for discrete action settings.  _arXiv:1910.07207_ , 2019.
  * Dai et al. (2018) Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song.  Sbeed: Convergent reinforcement learning with nonlinear function approximation.  In _International Conference on Machine Learning_ , pp. 1125–1134. PMLR, 2018.
  * Feng et al. (2019) Yihao Feng, Lihong Li, and Qiang Liu.  A kernel loss for solving the bellman equation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Fisher & Tippett (1928) Ronald Aylmer Fisher and Leonard Henry Caleb Tippett.  Limiting forms of the frequency distribution of the largest or smallest member of a sample.  In _Mathematical proceedings of the Cambridge philosophical society_ , volume 24, pp. 180–190. Cambridge University Press, 1928.
  * Fu et al. (2020) Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Fujimoto et al. (2022) Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu.  Why should i trust you, bellman? the bellman error is a poor replacement for value error.  _arXiv:2201.12417_ , 2022.
  * Garg et al. (2023) Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon.  Extreme q-learning: Maxent rl without entropy.  _arXiv:2301.02328_ , 2023.
  * Geist et al. (2017) Matthieu Geist, Bilal Piot, and Olivier Pietquin.  Is the bellman residual a bad proxy?  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Gong et al. (2020) Chen Gong, Yunpeng Bai, Xinwen Hou, and Xiaohui Ji.  Stable training of bellman error in reinforcement learning.  In _Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18–22, 2020, Proceedings, Part V 27_ , pp. 439–448. Springer, 2020.
  * Haarnoja et al. (2018a) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pp. 1861–1870. PMLR, 2018a.
  * Haarnoja et al. (2018b) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  _arXiv:1812.05905_ , 2018b.
  * Hansen-Estruch et al. (2023) Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine.  Idql: Implicit q-learning as an actor-critic method with diffusion policies.  _arXiv:2304.10573_ , 2023.
  * Hejna & Sadigh (2023) Joey Hejna and Dorsa Sadigh.  Inverse preference learning: Preference-based rl without a reward function.  _arXiv:2305.15363_ , 2023.
  * Hottung et al. (2021) André Hottung, Yeong-Dae Kwon, and Kevin Tierney.  Efficient active search for combinatorial optimization problems.  _arXiv:2106.05126_ , 2021.
  * Kaiser et al. (2019) Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al.  Model-based reinforcement learning for atari.  _arXiv:1903.00374_ , 2019.
  * Konda & Tsitsiklis (1999) Vijay Konda and John Tsitsiklis.  Actor-critic algorithms.  _Advances in neural information processing systems_ , 12, 1999.
  * Kostrikov et al. (2021) Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv:2110.06169_ , 2021.
  * Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Kwon et al. (2020) Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min.  Pomo: Policy optimization with multiple optima for reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:21188–21198, 2020.
  * Lyu et al. (2022) Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu.  Mildly conservative q-learning for offline reinforcement learning.  _arXiv:2206.04745_ , 2022.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv:1312.5602_ , 2013.
  * Neu et al. (2017) Gergely Neu, Anders Jonsson, and Vicenç Gómez.  A unified view of entropy-regularized markov decision processes.  _arXiv:1705.07798_ , 2017.
  * Patterson et al. (2022) Andrew Patterson, Adam White, and Martha White.  A generalized projected bellman error for off-policy value estimation in reinforcement learning.  _The Journal of Machine Learning Research_ , 23(1):6463–6523, 2022.
  * Ward et al. (2019) Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose.  Improving exploration in soft-actor-critic with normalizing flows policies.  _arXiv:1906.02771_ , 2019.
  * Watkins & Dayan (1992) Christopher JCH Watkins and Peter Dayan.  Q-learning.  _Machine learning_ , 8:279–292, 1992.
  * Ziebart (2010) Brian D Ziebart.  _Modeling purposeful adaptive behavior with the principle of maximum causal entropy_.  Carnegie Mellon University, 2010.

##  Appendix A Proof for Lemmas and Theorems.

###  A.1 Proof for Lemma2:

#### Lemma 2 :

$X$ follow the Gumbel distribution ${\rm Gumbel}(A,B)$, $C,D$ are both
constant, $D>0$. $Y=X+C$ will follow ${\rm Gumbel}(C+A,B)$, $Z=DX$ will follow
${\rm Gumbel}(DA,DB)$.

###### Proof.

The Cumulative Probability Density Function (CDF) for ${\rm Gumbel(x,A,B)}$
is:

${\rm Gumbel(x,A,B)}=e^{-e^{-\frac{(x-A)}{B}}}.$

So:

$P(Y<\alpha)=P(X<\alpha-C)=e^{-e^{-\frac{(\alpha-(C+A))}{B}}}.$
$P(Z<\alpha)=P(X<\frac{\alpha}{D})=e^{-e^{-\frac{(\frac{\alpha}{D}-A)}{B}}}=e^{-e^{-\frac{({\alpha}-DA)}{DB}}}.$

So:

$Y\sim{\rm Gumbel}(C+A,B).$ $Z\sim{\rm Gumbel}(DA,DB).$

∎

###  A.2 Proof for Lemma 3

#### Lemma 3:

For mutually independent random variables $X_{1},\dots,X_{n}$, which satisfies
$X_{i}\sim{\rm Gumbel}(C_{i},\beta)$. Then $\max_{i}(X_{i})\sim{\rm
Gumbel}({\beta}\ln\sum_{i}e^{\frac{1}{\beta}C_{i}},\beta)$.

###### Proof.

As mentioned in Section 3.1, the Probability Density Function (PDF) for ${\rm
Gumbel(C_{i},\beta)}$ is: ${\rm
Gumbel}(x;C_{i},\beta)=\frac{1}{\beta}\exp[{-[\frac{x-C_{i}}{\beta}+\exp[{-\frac{x-C_{i}}{\beta}}]]}]$.
The Cumulative Probability Density Function (CDF) for ${\rm
Gumbel(C_{i},\beta)}$ is : $e^{-e^{-\frac{(x-C_{i})}{\beta}}}$. So, we have:

$P(\max_{i}(X_{i})<A)=P(X_{1}<A,X_{2}<A,X_{3}<A,...,X_{n}<A).$

and

$P(X_{i}<A)=e^{-e^{-\frac{(A-C_{i})}{\beta}}}.$

Then

$P(X_{1}<A,X_{2}<A,X_{3}<A,...,X_{n}<A)=e^{-e^{-\frac{(A-C_{1})}{\beta}}}\cdot
e^{-e^{-\frac{(A-C_{2})}{\beta}}}...\cdot
e^{-e^{-\frac{(A-C_{n})}{\beta}}}=e^{-\sum_{i}e^{-\frac{(A-C_{i})}{\beta}}}.$
$P(X_{1}<A,X_{2}<A,X_{3}<A,...,X_{n}<A)=e^{-e^{-\frac{A}{\beta}}\sum_{i}e^{\frac{C_{i}}{\beta}}}=e^{-e^{-\frac{A}{\beta}}e^{\ln(\sum_{i}e^{\frac{C_{i}}{\beta}}})}=e^{-e^{-\frac{A}{\beta}+\ln(\sum_{i}e^{\frac{C_{i}}{\beta}}})}.$
$P(\max_{i}(X_{i})<A)=e^{-e^{-\frac{1}{\beta}[A-{\beta}\ln(\sum_{i}e^{\frac{C_{i}}{\beta}})]}}.$

So this means

$\max_{i}(X_{i})\sim{\rm
Gumbel}({\beta}\ln\sum_{i}e^{\frac{1}{\beta}C_{i}},\beta).$

∎

###  A.3 Proof for Lemma4:

#### Lemma 4 :

For $\epsilon^{t}({\bm{s}},{\bm{a}})$, which has been defined in equation 9,
if the following three assumptions can be satisfied:

1\. The action space is infinity, and it is a countable set.

2\. There is a injection mapping from pair $({\bm{s}},{\bm{a}})$ to the next
state ${\bm{s}}^{\prime}$, which means the next state is only determined by
the current state ${\bm{s}}$ and action ${\bm{a}}$.

3\. for any $({\bm{s}},{\bm{a}})$ pair, $\hat{Q}^{0}({\bm{s}},{\bm{a}})$ come
from the same exponential tail distribution $P$ ( for example $N(0,1)$ is
feasible ).

Under these hypothesis,
$\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))$
will follow the Gumbel distribution, if we suppose:

$\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))\sim
Gumbel(C_{1},\beta_{1}).$

Then for each $t$, $\epsilon^{t}({\bm{s}},{\bm{a}})$ will follow the $\rm
Gumbel$ distribution, actually, we will have:

$\epsilon^{t}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{t}({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{t}).$

Where:

$C_{1}({\bm{s}},{\bm{a}})=C_{1}$
$C_{2}({\bm{s}},{\bm{a}})=\gamma(C_{1}({\bm{s}},{\bm{a}})+\beta_{1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{1}}}).$
$C_{t}({\bm{s}},{\bm{a}})=\gamma(\beta_{t-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{t-1}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{t-1}}})(t\geq
3).$ $\beta_{t}=\gamma^{t-1}\beta_{1}.$

Additional, Special Situation Discussion: If for $\forall s_{1},s_{2}$, we
have two set:

$S_{1}=[r({\bm{s}}_{1},{\bm{a}}_{1}),r({\bm{s}}_{1},{\bm{a}}_{2}),...,r({\bm{s}}_{1},{\bm{a}}_{n})...]$
$S_{2}=[r({\bm{s}}_{2},{\bm{a}}_{1}),r({\bm{s}}_{2},{\bm{a}}_{2}),...,r({\bm{s}}_{2},{\bm{a}}_{n})...]$

and

$S_{1}\backslash S_{2}=\emptyset.$

Then in this case, we have:

$\epsilon^{t}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{t}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{t}).$

Where

$C_{t}=\gamma(C_{t-1}+\beta_{t-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{t-1}}})(t\geq
2).$

###### Proof.

If we use the Bellman operator during updating at the $t$th iteration for
estimating , we will have:

$\hat{Q}^{t}({\bm{s}},{\bm{a}})=r({\bm{s}},{\bm{a}})+\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{t-1}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

In Section 3.1, we have known that the true, unbiased $\overline{Q}$ should
be:

$\overline{Q}({\bm{s}},{\bm{a}})=r({\bm{s}},{\bm{a}})+\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

By subtracting the two equations, it can be deduced that the error
$\epsilon^{t}({\bm{s}},{\bm{a}})$ at the $t$th step is:

$\epsilon^{t}({\bm{s}},{\bm{a}})=\gamma[\max_{{\bm{a}}^{\prime}}(\hat{Q}^{t-1}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))-\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))].$

Let’s see what’s going on for more details. When $t=1$, we will have:

$\epsilon^{1}({\bm{s}},{\bm{a}})=\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

According to the assumption, under the condition of any fixed state
${\bm{s}}^{\prime}$, $\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime})$ is
exponent-tailed i.i.d countable random variables with respect to
${\bm{a}}^{\prime}$, this will satisfy the conditions described in Lemma 1. So
using Lemma 1, we can know that
$\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))$
will follow a $\rm Gumbel$ distribution. By assumption, since
$\hat{Q}^{0}({\bm{s}},{\bm{a}})$ is all randomly initialized, so they will
follow the same $\rm Gumbel$ distribution at time $0$, let us suppose:

$\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}^{0}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))\sim
Gumbel(C_{1},\beta_{1}).$

$\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))$
is a constant that does not affect this distribution type, but affect the
location of the $\rm Gumbel$ distribution. According to Lemma 2, we will have:

$\epsilon^{1}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{1}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{1}).$

Let’s see how ${\bm{s}}^{\prime}$ will continue in this trajectory. By
assumption, the action space is countable, so we can enumerate all actions to
a list:
$[({\bm{s}}^{\prime},{\bm{a}}_{1},r_{1},{\bm{s}}^{{}^{\prime\prime}}_{1}),({\bm{s}}^{\prime},{\bm{a}}_{2},r_{2},{\bm{s}}^{{}^{\prime\prime}}_{2}),...({\bm{s}}^{\prime},{\bm{a}}_{n},r_{n},{\bm{s}}^{{}^{\prime\prime}}_{n}),...]$.
According to the above discussion, we have:

$\epsilon^{1}({\bm{s}}^{\prime},{\bm{a}}_{i})\sim
Gumbel(C_{1}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{{}^{\prime\prime}}_{i},{\bm{a}}^{\prime})),\beta_{1})$

When $t=2$, we will have:

$\epsilon^{2}({\bm{s}},{\bm{a}})=\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})+\epsilon^{1}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

According to assumption, actions can be countable, which means we have a
action space $[{\bm{a}}_{1},{\bm{a}}_{2},........]$. Let
$\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{1}({\bm{s}}^{\prime},{\bm{a}}_{i})$
be $L_{i}$, obviously there is no connection between $a_{i}$ and $a_{j}$ when
$i\neq j$. So $L_{i}$ is a sequence of mutually independent countable random
variables, using Lemma 3, we can have:

$L_{i}\sim
Gumbel(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{1}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{{}^{\prime\prime}}_{i},{\bm{a}}^{\prime})),\beta_{1}).$

Noticed that:

$\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})=r({\bm{s}}^{\prime},{\bm{a}}_{i})+\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{{}^{\prime\prime}}_{i},{\bm{a}}^{\prime})).$

So:

$L_{i}\sim Gumbel(r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{1},\beta_{1}).$

According to Lemma 3, then

$max_{{\bm{a}}_{i}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{1}({\bm{s}}^{\prime},{\bm{a}}_{i}))=max_{i}(L_{i})\sim
Gumbel(C_{1}+\beta_{1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{1}}},\beta_{1}).$

Let $\gamma max_{i}(L_{i})\sim Gumbel(C_{2}({\bm{s}},{\bm{a}}),\beta_{2})$.
Because the discounted factor $\gamma$ is positive number, according to Lemma
3:

$C_{2}({\bm{s}},{\bm{a}})=\gamma(C_{1}+\beta_{1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{1}}})$
$\beta_{2}=\gamma(\beta_{1})$

So:

$\epsilon^{2}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{2}({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{2}).$

At this point, we can already discern some patterns. However, to ensure
thoroughness, we will conduct one more iteration here:

$\epsilon^{3}({\bm{s}},{\bm{a}})=\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})+\epsilon^{2}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

In this case, Let
$\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{2}({\bm{s}}^{\prime},{\bm{a}}_{i})$
be $M_{i}$, obviously there is no connection between $a_{i}$ and $a_{j}$ when
$i\neq j$. So $M_{i}$ is a sequence of mutually independent countable random
variables, using Lemma 3, we can have:

$M_{i}\sim
Gumbel(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{2}({\bm{s}}^{\prime},{\bm{a}}_{i})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{{}^{\prime\prime}}_{i},{\bm{a}}^{\prime})),\beta_{2}).$
$M_{i}\sim
Gumbel(r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{2}({\bm{s}}^{\prime},{\bm{a}}_{i}),\beta_{2}).$

According to Lemma 3, then

$max_{{\bm{a}}_{i}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{2}({\bm{s}}^{\prime},{\bm{a}}_{i}))=max_{i}(M_{i})\sim
Gumbel(\beta_{2}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{2}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{2}}},\beta_{2}).$

Let:

$C_{3}(s,a)=\gamma(\beta_{2}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{2}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{2}}})$
$\beta_{3}=\gamma\beta_{2}$

We will have:

$\epsilon^{3}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{3}({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{3}).$

Continuing in this manner, we will find that when $(t\geq 3)$, the approach
becomes identical. We will have a general iteration format:

$\epsilon^{t}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{t}(s,a)-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{t}).$

Where

$C_{2}({\bm{s}},{\bm{a}})=\gamma(C_{1}+\beta_{1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{1}}}).$
$C_{t}({\bm{s}},{\bm{a}})=\gamma(\beta_{t-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{t-1}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{t-1}}})(t\geq
3).$

Additional, Special Situation Discussion If for $\forall s_{1},s_{2}$, we have
two set:

$S_{1}=[r({\bm{s}}_{1},{\bm{a}}_{1}),r({\bm{s}}_{1},{\bm{a}}_{2}),...,r({\bm{s}}_{1},{\bm{a}}_{n})...]$
$S_{2}=[r({\bm{s}}_{2},{\bm{a}}_{1}),r({\bm{s}}_{2},{\bm{a}}_{2}),...,r({\bm{s}}_{2},{\bm{a}}_{n})...]$

and

$S_{1}\backslash S_{2}=\emptyset.$

Then when $t=2$:

$C_{2}=\gamma(C_{1}+\beta_{1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{1}}})$
$\beta_{2}=\gamma(\beta_{1})$

So:

$\epsilon^{2}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{2}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{2}).$

Which means this condition removes the correlation between $C_{i}$ and
$({\bm{s}},{\bm{a}})$. So in this case:

$\epsilon^{t}({\bm{s}},{\bm{a}})\sim
Gumbel(C_{t}-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{t}).$

Where

$C_{t}=\gamma(C_{t-1}+\beta_{t-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{t-1}}})(t\geq
2).$

∎

###  A.4 Proof for Theorem 1

#### Theorem 1

(Logistic distribution for Bellman error): During network training,
$\varepsilon^{\theta}({\bm{s}},{\bm{a}})$, which defined by equation 8, assume
that it also represents that the network has been updated $(\theta)$ times. If
the assumptions of Lemma 4 are all satisfied, then
$\varepsilon^{\theta}({\bm{s}},{\bm{a}})$ will follow the Logistic
distribution, actually:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(C_{\theta}({\bm{s}},{\bm{a}})-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

Besides, if we consider the Special Situation in Lemma 2, then:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

###### Proof.

While doing $Q$ updating in neural network, let’s analyse the convergence of
the $Q$ network, which can measure the Bellman error, according to equation 8,
we will have the definition for the Bellman error under the setting of
parameter $\theta$ :

${\varepsilon}^{\theta}({\bm{s}},{\bm{a}})=\hat{Q}_{\theta}({\bm{s}},{\bm{a}})-r({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}_{\theta}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

It is known that all conditions of Lemma 4 are satisfied. So according to
Lemma 4, if we consider:

$\hat{Q}_{\theta}({\bm{s}},{\bm{a}})=\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}{({\bm{s}},{\bm{a}})}.$

From the proof of Lemma 4, we can know that:

$\epsilon^{\theta}{({\bm{s}},{\bm{a}})}\sim
Gumbel(C_{\theta}({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})),\beta_{\theta}).$

where

$C_{\theta}({\bm{s}},{\bm{a}})=\gamma(\beta_{\theta-1}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta-1}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta-1}}})(\theta\geq
2).$ $\beta_{\theta}=\gamma^{\theta}\beta_{1}.$

So:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})=\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})-r({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}(\hat{Q}_{\theta}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

Because of:

$\overline{Q}({\bm{s}},{\bm{a}})=r({\bm{s}},{\bm{a}})+\gamma
max_{{\bm{a}}^{\prime}}\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})$

So:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})=\gamma\max_{{\bm{a}}^{\prime}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})]+\epsilon^{\theta}({\bm{s}},{\bm{a}})-\gamma\max_{{\bm{a}}^{\prime}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})+\epsilon^{\theta}({\bm{s}}^{\prime},{\bm{a}}^{\prime})].$

Notice that this equation has two parts, the front one:
$\gamma\max_{{\bm{a}}^{\prime}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})]+\epsilon^{\theta}({\bm{s}},{\bm{a}})$
and the back one
$\gamma\max_{{\bm{a}}^{\prime}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})+\epsilon^{\theta}({\bm{s}}^{\prime},{\bm{a}}^{\prime})]$.
Let us discuss them separately.

Let us first analyse the front part, according to Lemma 2, it is easy to have:

$\gamma\max_{{\bm{a}}^{\prime}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})]+\epsilon^{\theta}({\bm{s}},{\bm{a}})\sim{Gumbel}(C_{\theta}({\bm{s}},{\bm{a}}),\beta_{\theta}).$

Let us see another part, according to Lemma 2, it is easy to have:

$\epsilon^{\theta}{({\bm{s}}^{\prime},{\bm{a}}_{i})}\sim
Gumbel(C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}_{i}{{}^{\prime\prime}},{\bm{a}}^{\prime})),\beta_{\theta}).$

Thus

$[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})]\sim{Gumbel}(C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}_{i}{{}^{\prime\prime}},{\bm{a}}^{\prime}))+\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i}),\beta_{\theta}).$

Because of

$-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}_{i}{{}^{\prime\prime}},{\bm{a}}^{\prime}))+\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})=r({\bm{s}}^{\prime},{\bm{a}}_{i})$

So using Lemma 2:

$[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})]\sim
Gumbel(C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})+r({\bm{s}}^{\prime},{\bm{a}}_{i}),\beta_{\theta}).$

Using Lemma 3, we can know that:

$max_{a_{i}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})]\sim
Gumbel(\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

Using Lemma 2: We assume that the $\gamma$ is a constant that is very close to
1, this typically aligns with practical requirements, thus it will not
significantly affect the distribution. Apparently,
$\gamma\max_{{\bm{a}}^{\prime}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})]+\epsilon^{\theta}({\bm{s}},{\bm{a}})$
and $\gamma
max_{a_{i}}[\overline{Q}({\bm{s}}^{\prime},{\bm{a}}_{i})+\epsilon^{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})]$
are independent. As evidenced in the proof of Lemma 2, there only exists a
correlation between $\epsilon^{\theta}({\bm{s}},{\bm{a}})$ and
$\epsilon^{\theta-1}({\bm{s}},{\bm{a}})$ for any action ${\bm{a}}$:

$\epsilon^{\theta}({\bm{s}},{\bm{a}})=\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})+\epsilon^{\theta-1}({\bm{s}}^{\prime},{\bm{a}}^{\prime}))-\gamma\max_{{\bm{a}}^{\prime}}(\overline{Q}({\bm{s}}^{\prime},{\bm{a}}^{\prime})).$

Then, According to Lemma 5:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim{Gumbel}(C_{\theta}({\bm{s}},{\bm{a}}),\beta_{\theta})-Gumbel(\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$
${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(C_{\theta}({\bm{s}},{\bm{a}})-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta}({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

This implies us that this error is not unbiased, but rather, it exhibits bias.

The discussion of additional cases is consistent with Lemma 2. The additional
conditions removed the connection between $C_{\theta}$ and pair
$({\bm{s}},{\bm{a}})$, so in this case:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(C_{\theta}-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})+C_{\theta}}{\beta_{\theta}}},\beta_{\theta}).$

So, that is:

${\rm\varepsilon}^{\theta}({\bm{s}},{\bm{a}})\sim
Logistic(-\beta_{\theta}ln\sum_{i}e^{\frac{r({\bm{s}}^{\prime},{\bm{a}}_{i})}{\beta_{\theta}}},\beta_{\theta}).$

∎

###  A.5 Sovling for equation (5)

$\pi^{*}=arg\max_{\pi}[\sum{\mathbb{P}}({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})\left[Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})-\beta\log\frac{\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}{\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}\right].$

This can be solved by the Lagrange multiplier method with:

$\sum_{{\bm{a}}^{\prime}}\pi^{*}({\bm{a}}^{\prime}|{\bm{s}}^{\prime})=1.$

We can construct the Lagrange function as:

$f(\pi,L)=\sum_{{\bm{s}}^{\prime},{\bm{a}}^{\prime}}{\mathbb{P}}({\bm{s}}^{\prime}|{\bm{s}},{\bm{a}})\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})\left[Q({\bm{s}}^{\prime},{\bm{a}}^{\prime})-\beta\log\frac{\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}{\mu({\bm{a}}^{\prime}|{\bm{s}}^{\prime})}\right]+L\left[\sum_{{\bm{a}}^{\prime}}\pi({\bm{a}}^{\prime}|{\bm{s}}^{\prime})-1\right].$

Take

$\frac{\partial f}{\partial L}=0,\frac{\partial f}{\partial\pi}=0.$

We can have:

$\mu({\bm{a}}|{\bm{s}})e^{\frac{Q({\bm{s}},{\bm{a}})+L-1}{\beta}}=\pi({\bm{a}}|{\bm{s}}).$
$log(e^{\frac{L-1}{\beta}}\sum\mu({\bm{a}}|{\bm{s}})e^{\frac{Q({\bm{s}},{\bm{a}})}{\beta}})=log(1)=0\to
L=1-log\sum\mu({\bm{a}}|{\bm{s}})e^{\frac{Q({\bm{s}},{\bm{a}})}{\beta}}.$

Then take $L$ back, we can have:

$\pi^{*}({\bm{a}}|{\bm{s}})=\frac{\mu({\bm{a}}|{\bm{s}})e^{Q({\bm{s}},{\bm{a}})/\beta}}{\sum_{\bm{a}}\mu({\bm{a}}|{\bm{s}})e^{Q({\bm{s}},{\bm{a}})/\beta}}.$

###  A.6 Proof for Theorem2

#### Theorem 2

(The relationship between the estimated policy and the true policy): There
exists a gap between the true policy and the estimated policy under the
parameter $\theta$ , which related on whether equation 9 is accurately
estimated, in fact:

$\hat{\pi}(a|s)=\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}e^{\epsilon^{\theta}({\bm{s}},{\bm{a}})}\overline{\pi}(a|s).$

###### Proof.

The true, unbiased policy $\overline{\pi}$ should satisfy:

$\overline{\pi}(a|s)=\frac{e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}.$

But when we do this, we only have the estimated $Q$ function:
$\hat{Q}_{\theta}$, so in reality:

$\hat{\pi}(a|s)=\frac{e^{\hat{Q}_{\theta}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\hat{Q}_{\theta}({\bm{s}},{\bm{a}})}}.$

Taking the ratio between them:

$\frac{\hat{\pi}(a|s)}{\overline{\pi}(a|s)}=e^{(\hat{Q}_{\theta}({\bm{s}},{\bm{a}})-\overline{Q}({\bm{s}},{\bm{a}}))}\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\hat{Q}_{\theta}({\bm{s}},{\bm{a}})}}=e^{\epsilon^{\theta}({\bm{s}},{\bm{a}})}\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\hat{Q}_{\theta}({\bm{s}},{\bm{a}})}}.$

So:

$\hat{\pi}(a|s)=\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}e^{\epsilon^{\theta}({\bm{s}},{\bm{a}})}\overline{\pi}(a|s).$

∎

###  A.7 Proof for Theorem3

#### Theorem 3

(Logistic distribution for V Loss function): $V$ Loss will asymptotically
follow a Logistic distribution related to
$\epsilon^{\theta}({\bm{s}},{\bm{a}})$, in fact:

$\displaystyle\epsilon^{V}(s)\sim{\rm
Logistic}\left(-\log\left(\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}\right),\beta_{1}\right).$
(19)

and we will have:

$\displaystyle\epsilon^{V}(s)\stackrel{{\scriptstyle\epsilon^{\theta}({\bm{s}},{\bm{a}})\to
0}}{{\longrightarrow}}{\rm Logistic}\left(0,\beta_{1}\right).$ (20)

###### Proof.

If we think the true $\overline{V}$ satisfy:

$\overline{V}=\left[\overline{Q}({\bm{s}},{\bm{a}})-log(\overline{\pi}({\bm{a}}|{\bm{s}}))\right].$

But actually, we only have the estimated $V$, which means:

$\overline{V}+\epsilon^{V}=\left[\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})-\log(\hat{\pi}({\bm{a}}|{\bm{s}}))\right].$

According to Theorem 2, we have:

$\overline{V}+\epsilon^{V}=\left[\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})-log(\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}e^{\epsilon^{\theta}({\bm{s}},{\bm{a}})}\overline{\pi}({\bm{a}}|{\bm{s}}))\right].$

Here is needed for special explanation, the
$\epsilon^{\theta}({\bm{s}},{\bm{a}})$ in
$\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})$ is
independent with the $\epsilon^{\theta}({\bm{s}},{\bm{a}})$ in
$\log(\hat{\pi}({\bm{a}}|{\bm{s}}))$, this is because the two parts
$\hat{Q}_{\theta}({\bm{s}},{\bm{a}})$ and $\log(\hat{\pi}({\bm{a}}|{\bm{s}}))$
are estimated and calculated separately. Although their random items
$\epsilon^{\theta}({\bm{s}},{\bm{a}})$ come from the same distribution, each
estimation is equivalent to independent random sampling. Then:

$\epsilon^{V}({\bm{s}})=\epsilon({\bm{s}},{\bm{a}})-\log\left(\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}\right)-{\epsilon({\bm{s}},{\bm{a}})}\\\
\;\sim{\rm
Logistic}\left(-\log\left(\frac{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})}}{\sum_{\bm{a}}e^{\overline{Q}({\bm{s}},{\bm{a}})+\epsilon^{\theta}({\bm{s}},{\bm{a}})}}\right),\beta_{1}\right).$

If we want $\epsilon({\bm{s}},{\bm{a}})\to 0$,then we will have

$E[\epsilon^{V}(s)]\to 0.$

∎

###  A.8 Proof for Theorem 5

#### Theorem 5 :

(Relationship between $\rm LLoss$ and $\rm MSELoss)$ The $\rm MSELoss$ can be
utilized as an approximate estimation of $\rm LLoss$.

$\rm LLoss=ln4+\frac{1}{2}\rm MSELoss+o(\rm\varepsilon^{3}).$

###### Proof.

Let $\frac{\varepsilon}{\sigma}=t$, then:

$\rm MSELoss=\frac{1}{2}t^{2}.$ $\rm LLoss=t+2log(1+e^{-t}).$

If we performed a Taylor expansion on $\rm LLoss$, we can have:

$\rm
LLoss=t+2[ln2-\frac{1}{2}t+\frac{1}{8}t^{2}+o(t^{3})]=ln4+\frac{1}{4}t^{2}+o(t^{3})=ln4+\frac{1}{2}\rm
MSELoss+o(t^{3}).$

∎

##  Appendix B Model hyperparameters detail and training specifics section.

Here, we provide the model configuration and hyperparameters settings for both
online and offline parts. We utilize code based on SAC for the replacement of
the loss function. Parameter specifics are provided in Table 5 and Table 6.
Table 5 provides the beta configuration for $V$ Loss in SAC under $\rm LLoss$,
$\beta$ configuration for Bellman error in SAC and CQL, as well as other model
parameter settings, where all remaining parameters follow the default values
in the SAC code by (Haarnoja et al., 2018b). Table 6 provides the $\beta$
configuration for Bellman error in IQL (Kostrikov et al., 2021) under $\rm
LLoss$, as well as other model parameter settings, where all remaining
parameters follow the default values in the IQL code. The source code is taken
from the rlkit GitHub repository.

Table 5: The detailed parameter settings for online training, with beta as the
scaling parameter in $\rm LLoss$.

| V ($\beta$) | Bellman error ($\beta$) | CQL ($\beta$) | batch | TAU | learning rate ($lr$) | GAMMA ($\gamma$) | Scaling | max. step
---|---|---|---|---|---|---|---|---|---
LunarLanderContinuous-v2 | 10 | 10 | 10 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 200
HalfCheetah-v2 | 10 | 10 | 10 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 200
Hopper-v4 | 10 | 10 | 10 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 200
Walker2d-v2 | 10 | 10 | 10 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 200
HumanoidStandup-v4 | 90 | 90 | 60 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 100
InvertedPendulum-v4 | 10 | 10 | 10 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 1000
InvertedDoublePendulum-v2 | 10 | 10 | 10 | 256 | 0.005 | 0.0003 | 0.99 | 1 | 1000
BipedalWalker-v3 | 20 | 20 | 20 | 256 | 0.005 | 0.0003 | 0.99 | 50 | 200

Table 6: The detailed parameter settings for offline training, with $\beta$ be
the scaling parameter in $\rm LLoss$.

|  eval steps / epoch |  train steps / train loop |  txpl. steps / train loop | # steps before training | batch |  Bellman error ($\beta$) | max. step
---|---|---|---|---|---|---|---
hopper-medium-v2 | 100 | 100 | 100 | 100 | 256 | 10 | 100
walker2d-medium-v2 | 100 | 100 | 100 | 100 | 256 | 3 | 100
halfcheetah-medium-v2 | 100 | 100 | 100 | 100 | 256 | 3 | 100
hopper-medium-replay-v2 | 100 | 100 | 100 | 100 | 256 | 3 | 100
walker2d-medium-replay-v2 | 100 | 100 | 100 | 100 | 256 | 20 | 100
halfcheetah-medium-replay-v2 | 100 | 100 | 100 | 100 | 256 | 20 | 100
hopper-medium-expert-v2 | 100 | 100 | 100 | 100 | 256 | 10 | 100
halfcheetah-medium-expert-v2 | 100 | 100 | 100 | 100 | 256 | 3 | 100
walker2d-medium-expert-v2 | 100 | 100 | 100 | 100 | 256 | 5 | 100

##  Appendix C KS tests and experiment result

###  C.1 KS Tests ((An, 1933))

#### Step1

: Collect the data as $[x_{1},x_{2},...x_{n}]$ and compute its cumulative
distribution function as $F^{*}(x)(x\in[x_{1},x_{2},...,x_{n}])$.

#### Step2

: Plot the cumulative distribution function $F(x)$ of a specific distribution
(Gumbel/Logistic/Normal) under the optimal parameters.

#### Step3

: Calculate the size of the KS statistic, using the following formula:

$KS=max_{i}|F^{*}(x_{i})-F(x_{i})|$

#### Step4

: A smaller KS statistic indicates a closer similarity between the cumulative
distribution function of the data and the specified distribution function.

###  C.2 The evolving Bellman error distribution over training time

In this section, we present the evolution of the Bellman error and $V$ Loss
distributions in both online training and offline training, figure 5 to 13
represent the images of the Bellman error distribution as it changes with
epochs in the offline environment, while figures 14 to 20 represent the images
of the Bellman error and $V$ Loss distributions as they change with epochs in
the online environment.

Figure 5:  The evolving distributions of Bellman error computed by equation 8
that using all data from the Reply-Buffer at different epochs (50, 100, 200,
300 and final) of offline RL training on halfcheetah-medium. Figure 6:  The
evolving distributions of Bellman error computed by equation 8 that using all
data from the Reply-Buffer at different epochs (50, 100, 200, 300 and final)
of offline RL training on walker2d-medium. Figure 7:  The evolving
distributions of Bellman error computed by equation 8 that using all data from
the Reply-Buffer at different epochs (50, 100, 200, 300 and final) of offline
RL training on hopper-medium. Figure 8:  The evolving distributions of Bellman
error computed by equation 8 that using all data from the Reply-Buffer at
different epochs (50, 100, 200, 300 and final) of offline RL training on
halfcheetah-medium-replay. Figure 9:  The evolving distributions of Bellman
error computed by equation 8 that using all data from the Reply-Buffer at
different epochs (50, 100, 200, 300 and final) of offline RL training on
walker2d-medium-replay. Figure 10:  The evolving distributions of Bellman
error computed by equation 8 that using all data from the Reply-Buffer at
different epochs (50, 100, 200, 300 and final) of offline RL training on
hopper-medium-replay. Figure 11:  The evolving distributions of Bellman error
computed by equation 8 that using all data from the Reply-Buffer at different
epochs (50, 100, 200, 300 and final) of offline RL training on hopper-medium-
expert. Figure 12:  The evolving distributions of Bellman error computed by
equation 8 that using all data from the Reply-Buffer at different epochs (50,
100, 200, 300 and final) of offline RL training on walker2d-medium-expert.
Figure 13:  The evolving distributions of Bellman error computed by equation 8
that using all data from the Reply-Buffer at different epochs (50, 100, 200,
300 and final) of offline RL training on halfcheetah-medium-expert. Figure 14:
The evolving distributions of Bellman error computed by equation 8 and 13 that
using all data from the Reply-Buffer at different epochs (100, 200, 400, 700
and final) of online RL training on HalfCheetah-v2. Figure 15:  The evolving
distributions of Bellman error computed by equation 8 and 13 that using all
data from the Reply-Buffer at different epochs (100, 200, 400, 700 and final)
of online RL training on Hopper-v4. Figure 16:  The evolving distributions of
Bellman error computed by equation 8 and 13 that using all data from the
Reply-Buffer at different epochs (100, 200, 400, 700 and final) of online RL
training on Walker2d-v2. Figure 17:  The evolving distributions of Bellman
error computed by equation 8 and 13 that using all data from the Reply-Buffer
at different epochs (100, 200, 400, 700 and final) of online RL training on
HumanoidStandup-v4. Figure 18:  The evolving distributions of Bellman error
computed by equation 8 and 13 that using all data from the Reply-Buffer at
different epochs (100, 200, 400, 700 and final) of online RL training on
InvertedPendulum-v4. Figure 19:  The evolving distributions of Bellman error
computed by equation 8 and 13 that using all data from the Reply-Buffer at
different epochs (100, 200, 400, 700 and final) of online RL training on
BipedalWalker-v3. Figure 20:  The evolving distributions of Bellman error
computed by equation 8 and 13 that using all data from the Reply-Buffer at
different epochs (100, 200, 400, 700 and final) of online RL training on
InvertedDoublePendulum-v4.

###  C.3 The variations of Bellman error and $V$ Loss distribution in online
and offline environments.

In this section, we will present all distribution details to confirm the
characteristics of the Logistic distribution, which is slightly shorter in the
tail and slightly longer in the head compared to the Gumbel distribution. In
the head region of the distribution, the Logistic distribution fits much
better than the Normal and Gumbel distributions, while in the tail region, the
Logistic distribution is superior to the Normal distribution and generally
outperforms the Gumbel distribution in most environments. These phenomena can
be observed from Figures 22 and 21. Figure 22 provides a detailed view of the
Bellman error distribution in the offline environment, while Figure 21
displays the detailed Bellman error and $V$ Loss distributions in the online
environment.

Figure 21: The distribution for Bellman error and $V$ Loss for the other
online environment during the half of the training epochs. Figure 22: The
distribution for Bellman error for the other offline environments during the
half of the training epochs.

###  C.4 The variation of Average Reward in online and offline environments.

In this section, we provide additional graphs depicting the average reward
variations in other online and offline environments that were not presented in
the main text. Details refer to figures 23 and 24. Figure 23 corresponds to
the offline setting, while figure 24 corresponds to the online setting.
Additionally, we include a comparison graph between CQL and LCQL, results are
in figure 25.

Figure 23: The average reward variation of IQL, LIQL across the other five
environments in the context of offline training. Figure 24: The average reward
variation of SAC, LSAC, and XQL across the other four environments in the
context of online training. Figure 25: The average reward variation of CQL and
LCQL across the eight environments in the context of online training.

###  C.5 The complete version of the containing goodness-of-fit tests and KS
tests for Bellman error and $V$ Loss.

In this section, we provide the full versions of Table 3 and Table 4 as
presented in the main text, along with the addition of a table related to
$V$’s fitness and KS tests in Tables 7, 8, and 9 for the results.

Table 7: The complete version of the fitness and KS tests for Bellman-error
(Online)

| R2 $\uparrow$ | SSE ($\times 10^{-4}$) $\downarrow$ | RMSE ($\times 10^{-4}$) $\downarrow$ | KS statistic $\downarrow$
---|---|---|---|---
| Logistic | Gumbel | Normal | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal
LunarLanderContinuous-v2 | $0.985$ | $0.971$ | $0.975$ | $1.119$ | $2.224$ | $1.902$ | $7.555$ | $10.653$ | $9.850$ | $0.052$ | $0.070$ | $0.071$
HalfCheetah-v2 | $0.991$ | $0.990$ | $0.989$ | $1.344$ | $1.425$ | $1.549$ | $8.282$ | $8.405$ | $8.888$ | $0.026$ | $0.047$ | $0.033$
Hopper-v4 | $0.989$ | $0.985$ | $0.981$ | $2.697$ | $3.793$ | $4.807$ | $11.734$ | $13.912$ | $15.661$ | $0.067$ | $0.073$ | $0.085$
Walker2d-v2 | $0.988$ | $0.967$ | $0.975$ | $0.900$ | $2.549$ | $1.903$ | $6.778$ | $11.404$ | $9.854$ | $0.054$ | $0.084$ | $0.072$
HumanoidStandup-v4 | $0.667$ | $0.641$ | $0.628$ | $27.164$ | $29.279$ | $30.318$ | $37.228$ | $38.652$ | $39.331$ | $0.269$ | $0.322$ | $0.291$
InvertedPendulum-v4 | $0.983$ | $0.963$ | $0.971$ | $20.961$ | $46.307$ | $35.772$ | $32.702$ | $48.606$ | $42.721$ | $0.115$ | $0.175$ | $0.117$
InvertedDoublePendulum-v4 | $0.999$ | $0.981$ | $0.998$ | $0.249$ | $5.623$ | $0.324$ | $3.959$ | $16.938$ | $4.063$ | $0.021$ | $0.079$ | $0.023$
BipedalWalker-v3 | $0.997$ | $0.979$ | $0.990$ | $1.206$ | $7.888$ | $3.563$ | $7.843$ | $20.061$ | $13.482$ | $0.039$ | $0.101$ | $0.057$

Table 8: The complete version of the fitness and KS tests for Bellman-error
(Offline)

| R2 $\uparrow$ | SSE ($\times 10^{-4}$) $\downarrow$ | RMSE ($\times 10^{-4}$) $\downarrow$ | KS statistic $\downarrow$
---|---|---|---|---
| Logistic | Gumbel | Normal | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal
hopper-medium-v2 | $0.981$ | $0.975$ | $0.976$ | $10.175$ | $13.975$ | $13.191$ | $29.617$ | $34.709$ | $33.722$ | $0.040$ | $0.094$ | $0.053$
walker2d-medium-v2 | $0.927$ | $0.923$ | $0.913$ | $15.714$ | $15.915$ | $18.563$ | $36.801$ | $37.129$ | $40.003$ | $0.062$ | $0.072$ | $0.076$
halfcheetah-medium-v2 | $0.836$ | $0.831$ | $0.833$ | $9.087$ | $9.394$ | $9.223$ | $12.314$ | $12.941$ | $12.444$ | $0.050$ | $0.052$ | $0.069$
halfcheetah-medium-replay-v2 | $0.852$ | $0.813$ | $0.836$ | $12.864$ | $16.242$ | $14.221$ | $33.301$ | $37.416$ | $35.012$ | $0.031$ | $0.105$ | $0.048$
walker2d-medium-replay-v2 | $0.950$ | $0.908$ | $0.937$ | $10.514$ | $19.256$ | $13.262$ | $30.101$ | $40.743$ | $33.812$ | $0.075$ | $0.149$ | $0.093$
hopper-medium-replay-v2 | $0.954$ | $0.927$ | $0.948$ | $10.773$ | $17.278$ | $12.265$ | $30.475$ | $38.594$ | $32.516$ | $0.038$ | $0.104$ | $0.049$
hopper-medium-expert-v2 | $0.985$ | $0.970$ | $0.982$ | $8.054$ | $17.097$ | $10.162$ | $26.351$ | $38.391$ | $29.598$ | $0.056$ | $0.105$ | $0.059$
walker2d-medium-expert-v2 | $0.981$ | $0.959$ | $0.973$ | $11.186$ | $23.807$ | $15.584$ | $31.053$ | $45.302$ | $36.653$ | $0.067$ | $0.138$ | $0.075$
halfcheetah-medium-expert-v2 | $0.919$ | $0.869$ | $0.913$ | $12.722$ | $20.568$ | $13.719$ | $33.117$ | $42.108$ | $34.391$ | $0.036$ | $0.098$ | $0.045$

Table 9: The complete version of the fitness and KS tests for $V$ Loss
(Online)

| R2 $\uparrow$ | SSE ($\times 10^{-4}$) $\downarrow$ | RMSE ($\times 10^{-4}$) $\downarrow$ | KS statistic $\downarrow$
---|---|---|---|---
| Logistic | Gumbel | Normal | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal | Logistic | Gumbel | Normal
LunarLanderContinuous-v2 | $0.995$ | $0.972$ | $0.987$ | $1.700$ | $9.037$ | $4.199$ | $9.314$ | $21.472$ | $14.638$ | $0.044$ | $0.099$ | $0.061$
HalfCheetah-v2 | $0.999$ | $0.969$ | $0.998$ | $0.176$ | $7.206$ | $0.428$ | $2.997$ | $19.174$ | $4.676$ | $0.025$ | $0.089$ | $0.027$
Hopper-v4 | $0.992$ | $0.985$ | $0.985$ | $2.338$ | $4.737$ | $4.624$ | $10.922$ | $15.547$ | $15.361$ | $0.058$ | $0.071$ | $0.075$
Walker2d-v2 | $0.994$ | $0.957$ | $0.987$ | $0.620$ | $4.867$ | $1.529$ | $5.625$ | $15.763$ | $8.834$ | $0.053$ | $0.125$ | $0.073$
HumanoidStandup-v4 | $0.938$ | $0.887$ | $0.909$ | $2.647$ | $4.827$ | $3.863$ | $11.622$ | $15.694$ | $14.041$ | $0.049$ | $0.109$ | $0.064$
InvertedPendulum-v4 | $0.984$ | $0.957$ | $0.972$ | $14.406$ | $36.099$ | $23.981$ | $27.111$ | $42.916$ | $34.978$ | $0.052$ | $0.118$ | $0.061$
InvertedDoublePendulum-v4 | $0.999$ | $0.974$ | $0.999$ | $0.270$ | $9.239$ | $0.791$ | $3.715$ | $21.711$ | $6.352$ | $0.032$ | $0.091$ | $0.033$
BipedalWalker-v3 | $0.993$ | $0.950$ | $0.988$ | $3.338$ | $23.354$ | $5.233$ | $13.051$ | $34.518$ | $16.342$ | $0.090$ | $0.161$ | $0.102$

###  C.6 Sensitivity analysis.

In this section, based on the empirical results obtained by adjusting
$\sigma$, we provide a range of $\sigma$ variations and their relationship
with maximum average reward and final training average reward. From the
graphs, it is evident that within a certain range of $\sigma$ variations, $\rm
LLoss$ outperforms $\rm MSELoss$ in both maximum average reward and the final
training average reward. This indicates that even within a small range of
$\sigma$ adjustments, $\rm LLoss$ consistently yields superior results
compared to $\rm MSELoss$ and exhibits a degree of robustness. Figures 26 and
27 depict the variations in maximum average reward and final average reward
for different $\sigma$ values in the offline setting, with the dashed line
representing the $\rm MSELoss$ standard for IQL. Figures 29 and 29 similarly
show the variations in maximum average reward and final average reward for
different $\sigma$ values in the online setting, with the dashed line
representing the $\rm MSELoss$ standard for SAC.

Figure 26: The relationship between the variation of $\sigma$ and the maximum
average reward in offline training. Figure 27: The relationship between the
variation of $\sigma$ and the average reward in offline training.

Figure 28: The relationship between the variation of $\sigma$ and the maximum
average reward in online training.

Figure 29: The relationship between the variation of $\sigma$ and the average
reward in online training.