# A Connection between One-Step RL and
Critic Regularization in Reinforcement Learning

Benjamin Eysenbach  Matthieu Geist  Sergey Levine  Ruslan Salakhutdinov

###### Abstract

As with any machine learning problem with limited data, effective offline RL
algorithms require careful regularization to avoid overfitting. One class of
methods, known as one-step RL, perform just one step of policy improvement.
These methods, which include advantage-weighted regression and conditional
behavioral cloning, are thus simple and stable, but can have limited
asymptotic performance. A second class of methods, known as critic
regularization, perform many steps of policy improvement with a regularized
objective. These methods typically require more compute but have appealing
lower-bound guarantees. In this paper, we draw a connection between these
methods: applying a multi-step critic regularization method with a
regularization coefficient of 1 yields the same policy as one-step RL. While
our theoretical results require assumptions (e.g., deterministic dynamics),
our experiments nevertheless show that our analysis makes accurate, testable
predictions about practical offline RL methods (CQL and one-step RL) with
commonly-used hyperparameters.

reinforcement learning, offline RL, regularization

##  1 Introduction

Figure 1:  Both $n$-step RL and critic regularization can interpolate between
behavioral cloning (left) and un-regularized RL (right) by varying the
regularization parameter. Endpoints of these regularization paths are the
same. We prove that these methods also obtain the same policy for an
intermediate degree of regularization.

Reinforcement learning (RL) algorithms tend to perform better when
regularized, especially when given access to only limited data, and especially
in batch (i.e., offline) settings where the agent is unable to collect new
experience. While RL algorithms can be regularized using the same tools as in
supervised learning (e.g., weight decay, dropout), we will use
“regularization” to refer to those techniques unique to the RL setting. Such
regularization methods include policy regularization (penalizing the policy
for sampling out-of-distribution action) and value regularization (penalizing
the critic for making large predictions). Research on these sorts of
regularization has grown significantly in recent years, yet theoretical work
studying the tradeoffs between regularization methods remains limited
(Vieillard et al., 2020).

Many RL methods perform regularization and can be classified by whether they
perform one or many steps of policy improvement. _One-step RL_ methods
(Brandfonbrener et al., 2021; Peng et al., 2019; Peters & Schaal, 2007; Peters
et al., 2010) perform one step of policy iteration, updating the policy to
choose actions the are best according to the Q-function of the behavioral
policy. The policy is often regularized to not deviate far from the behavioral
policy. In theory, policy iteration can take a large number of iterations
($\tilde{{\mathcal{O}}}(|{\mathcal{S}}||{\mathcal{A}}|/(1-\gamma))$ (Scherrer,
2013)) to converge, so one-step RL (one step of policy iteration) fails to
find the optimal policy on most tasks. Empirically, policy iteration often
converges in a smaller number of iterations (Sutton & Barto, 2018, Sec. 4.3),
and the policy after just a single iteration can sometimes achieve performance
comparable to multi-step RL methods (Brandfonbrener et al., 2021). _Critic
regularization_ methods modify the training of the value function such that it
predicts smaller returns for unseen actions (Kumar et al., 2020; Chebotar et
al., 2021; Yu et al., 2021; Hatch et al., 2022; Nachum et al., 2019; An et
al., 2021; Bai et al., 2022; Buckman et al., 2021). Intuitively, such critic
regularization causes the policy to avoid sampling unseen actions. In this
paper, we will use “critic regularization” to specifically refer to multi-step
methods that use critic regularization; multi-step methods that do not utilize
critic regularization (e.g., KL control (Ziebart, 2010), TD3+BC (Fujimoto &
Gu, 2021)) are outside the scope of our analysis.

These RL regularization methods appear distinct. Critic regularization
typically involves solving a two-player game, whereby a policy predicts
actions with high values while the critic decreases the values predicted for
those actions. Prior work (Kumar et al., 2020) has argued that this complexity
is worthwhile because the regularization effect is being propagated across
time via Bellman backups: decreasing the value at one state will also decrease
the value at states that lead there.

In this paper, we show that a certain type of one-step RL is equivalent to a
certain type of critic regularization, under some assumptions (see Fig. 1).
The key idea is that, when using a certain TD loss, the regularized critic
updates converge not to the true Q-values, but rather the Q-values multiplied
by an importance weight. For the critic, these importance weights mean that
the Q-values end up estimating the expected returns of the behavioral policy
($Q^{\beta}$, as in many one-step methods (Peters et al., 2010; Peters &
Schaal, 2007; Peng et al., 2019; Brandfonbrener et al., 2021)), rather than
the expected returns of the optimal policy ($Q^{\pi}$). For the actor, these
importance weights mean that the logarithm of the Q-values includes a term
that looks like a KL divergence. This connection allows us to make precise how
critic regularization methods implicitly regularize the policy.

The key contribution of this paper is a construction showing that one-step RL
produces the same policy as a multi-step critic regularization method, for a
certain regularization coefficient. We also discuss similar connections in
settings with varying degrees of regularization, in goal-conditioned settings,
and in RL settings with success classifiers. The main assumption behind these
results is that the critic is updated using an update rule based on the cross
entropy loss, rather than the MSE loss. Our result is potentially surprising
because, algorithmically and mechanistically, one-step RL and critic
regularization are very different. Nonetheless, our analysis may help explain
why prior work has found that one-step RL and critic regularization methods
can perform similarly on some (Brandfonbrener et al., 2021; Emmons et al.,
2021) (but not all (Kostrikov et al., 2021)) problems. Our results hint that
one-step RL methods may be a simpler approach to achieving the theoretical
guarantees typically associated with critic regularization methods. Our
analysis does not say that practical implementations (which violate our
assumptions) will always behave the same in practice; they do not
(Brandfonbrener et al., 2021, Sec. 7). Nonetheless, our experiments show that
our analysis makes accurate testable predictions about practical methods.
While our results do not say whether users should regularize the actor or
critic in practice, they hint that one-step RL methods may be a simpler way of
achieving the theoretical and empirical properties of critic regularization on
RL tasks that require strong regularization.

##  2 Related Work

Regularization has been applied to RL in many different ways (Neu et al.,
2017; Geist et al., 2019), and features prominently in offline RL methods
(Lange et al., 2012; Levine et al., 2020). While RL algorithms can be
regularized using the same techniques as in supervised learning (e.g., weight
decay, dropout), our focus will be on regularization methods unique to the RL
setting. Such RL-specific regularization methods can be roughly categorized
into one-step RL methods and critic regularization methods.

One-step RL methods (Brandfonbrener et al., 2021; Gülçehre et al., 2020;
Peters & Schaal, 2007; Peng et al., 2019; Peters et al., 2010; Wang et al.,
2018) apply a single step of policy improvement to the behavioral policy.
These methods first estimate the Q-values of the behavioral policy, either via
regression or iterative Bellman updates. Then, these methods optimize the
policy to maximize these Q-values minus an actor regularizer. Many goal-
conditioned or task-conditioned imitation learning methods (Savinov et al.,
2018; Ding et al., 2019; Sun et al., 2019; Ghosh et al., 2020; Paster et al.,
2021; Yang et al., 2021; Srivastava et al., 2019; Kumar et al., 2019b; Chen et
al., 2021; Lynch & Sermanet, 2021; Li et al., 2020; Eysenbach et al., 2020a)
also fits into this mold (Eysenbach et al., 2022), yielding policies that
maximize the Q-values of the behavioral policy while avoiding unseen actions.
Note that non-conditional imitation learning methods do not perform policy
improvement, and do not fit into this mold. One-step methods are typically
simple to implement and computationally efficient.

Critic regularization methods instead modify the objective for the Q-function
so that it predicts lower returns for unseen actions (Kumar et al., 2020;
Chebotar et al., 2021; Yu et al., 2021; Hatch et al., 2022; Nachum et al.,
2019; An et al., 2021; Bai et al., 2022; Buckman et al., 2021). Critic
regularization methods are typically more challenging to implement correctly
and more computationally demanding (Kumar et al., 2020; Nachum et al., 2019;
Bai et al., 2022; An et al., 2021), but can lead to better results on some
challenging problems (Kostrikov et al., 2021) Our analysis will show that one-
step RL is equivalent to a certain type of critic regularization.

Some regularization methods do not fit exactly into these two categories.
Methods like KL control regularize both the actor and the reward function
(Geist et al., 2019; Ziebart, 2010; Haarnoja et al., 2018; Abdolmaleki et al.,
2018; Wu et al., 2019; Jaques et al., 2019; Rezaeifar et al., 2022). Other
methods only regularize the policy used in the critic updates (Fujimoto et
al., 2019; Kumar et al., 2019a).

Our results are conceptually similar to prior work on regularization in the
supervised learning setting. There, regularization methods like weight decay,
spectral regularization, and early stopping all appear quite distinct, but are
actually mathematical equivalent under some assumptions (Bakushinskii, 1967;
Wahba, 1987; Fleming, 1990; Santos, 1996; Bauer et al., 2007). This result is
surprising because the methods are so different: weight decay modifies the
loss function, early stopping modifies the update procedure, and spectral
normalization is a post-hoc correction.

##  3 Preliminaries

We start by defining the single-task RL problem, and then introduce
prototypical examples of one-step RL and critic regularization. We then define
an actor critic algorithm for use in our analysis.

###  3.1 Notation

We assume an MDP with states $s$, actions $a$, initial state distribution
$p_{0}(s_{0})$, dynamics $p(s^{\prime}\mid s,a)$, and reward function
$r(s,a)$. We assume episodes always have infinite length (i.e., there are no
terminal states). Without loss of generality, we assume rewards are positive;
adding a positive constant to all rewards can make them all positive without
changing the optimal policy. We will learn a Markovian policy $\pi(a\mid s)$
to maximize the expected discounted sum of rewards:

$\max_{\pi}\mathbb{E}_{\pi(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid
s_{0}\sim p_{0}(s_{0})\right],$

where $\pi(\tau)=p(s_{0})\prod_{t=0}^{\infty}\pi(a_{t}\mid s_{t})p(s_{t+1}\mid
s_{t},a_{t})$ is the probability of policy $\pi$ sampling an infinite-length
trajectory $\tau=(s_{0},a_{0},\cdots)$. We define Q-values for policy
$\pi(a\mid s)$ as

$Q^{\pi}(s,a)=\mathbb{E}_{\pi(\tau)}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid
s_{0}=s,a_{0}=a\right].$

Because the rewards are positive, these Q-values are also positive,
$Q^{\pi}(s,a)>0$. Since we focus on the offline setting, we will consider two
policies: $\beta(a\mid s)$ is the _behavioral_ policy that collected the
dataset, and $\pi(a\mid s)$ is the _online_ policy output by the algorithm
that attempts to maximize the rewards. We will use $p(s,a,s^{\prime})$ to
denote the empirical distribution of transitions in an offline dataset, and
$p(s,a)$ and $p(s)$ denote the corresponding marginal distributions. The
behavioral policy is defined as $\beta(a\mid s)=p(a\mid s)$.

###  3.2 Examples of Regularization in RL

While actor and critic regularization methods can be implemented in many ways,
we introduce two prototypical examples below to make our discussion more
concrete.

#### Example of one-step RL: Brandfonbrener et al. (2021).

One-step RL first estimates the Q-values of the behavioral policy
($Q^{\beta}(s,a)$), and then optimizes the policy to maximize the Q-values
minus a actor regularizer. While the actor regularizer can take different
forms and the Q-values can be learned via regression, we will use a reverse KL
regularizer and TD-style critic update so that the objective is similar to
critic regularization:

$\displaystyle\max_{\pi}\mathbb{E}_{p(s)\pi(a\mid
s)}\left[Q^{\beta}(s,a)+\lambda\log\frac{\beta(a\mid s)}{\pi(a\mid s)}\right]$
(1)

where $Q^{\beta}(s,a)=\lim_{t\rightarrow\infty}Q_{t}(s,a)$ and

$\displaystyle
Q_{t+1}\leftarrow\operatorname*{arg\,min}_{Q}\mathbb{E}_{p(s,a)}\left[\left(Q(s,a)-y^{\beta,Q_{t}}(s,a)\right)^{2}\right]$
$\displaystyle y^{\beta,Q_{t}}(s,a)\triangleq
r(s,a)+\gamma\mathbb{E}_{\begin{subarray}{c}p(s^{\prime}\mid s,a)\\\
\beta(a^{\prime}\mid
s^{\prime})\end{subarray}}\left[Q_{t}(s^{\prime},a^{\prime})\right].$

The scalar $\lambda$ is the regularization coefficient and $\beta(a\mid s)$ is
an estimate of the behavioral policy, typically learned via behavioral
cloning. Like most TD methods (Haarnoja et al., 2018; Mnih et al., 2013;
Fujimoto et al., 2018), the TD targets $y$ are not considered learnable. In
practice, most methods do not solve optimize the critic to convergence at each
step, instead taking just a few gradient steps before updating the TD targets.
This one-step critic loss is different from the multi-step critic losses used
in other RL methods (e.g., TD3, SVG(0)) because it uses the TD target
$y^{\beta,Q}(s,a)$ (corresponds to a fixed policy) rather than
$y^{\pi,Q}(s,a)$ (corresponding to a sequence of learned policies). One-step
RL amounts to performing one step of policy iteration, rather than full policy
optimization. While truncating the iterations of policy iteration can be
suboptimal, it can also be interpreted as a form of early stopping
regularization.

#### Example of critic regularization: Kumar et al. (2020).

CQL (Kumar et al., 2020) modifies the standard Bellman loss to include an
additional term that decreases the values predicted for unseen actions. The
actor objective is to maximize Q values; some CQL implementations also
regularize the actor loss (Hoffman et al., 2020; Kumar et al., 2020)). The
objectives can then be written as

$\displaystyle\max_{\pi}\mathbb{E}_{p(s)\pi(a\mid
s)}\left[Q^{\pi}(s,a)\right]$ (2)

where $Q^{\pi}(s,a)=\lim_{t\rightarrow\infty}Q_{t}(s,a)$ and

$\displaystyle Q_{t+1}=$
$\displaystyle\operatorname*{arg\,min}_{Q}\mathbb{E}_{p(s,a)}\left[\left(Q(s,a)-y^{\pi,Q_{t}}(s,a)\right)^{2}\right]$
$\displaystyle+\lambda\left(\mathbb{E}_{p(s)\pi(a\mid
s)}\left[Q(s,a)\right]-\mathbb{E}_{p(s)\beta(a\mid
s)}\left[Q(s,a)\right]\right).$

The second term decreases the Q-values for unseen actions (those sampled from
$\pi(a\mid s)$) while the third term increases the values predicted for seen
actions (those sampled from the behavioral policy $\beta(a\mid s)$). Unlike
standard temporal difference methods, the CQL updates resemble a competitive
game between the actor and the critic. In practice, this cyclic dependency can
create unstable learning (Kumar et al., 2020; Hoffman et al., 2020).

###  3.3 How are these methods connected?

Prior work has observed that one-step methods and critic regularization
methods perform similarly on many (Fujimoto & Gu, 2021; Emmons et al., 2021)
(but not all (Kostrikov et al., 2021)) tasks. Despite the differences in
objectives and implementations of these two methods (and, more broadly, the
actor/critic regularization methods for which they are prototypes), are there
deeper, unifying connections between the methods?

In the next section, we introduce a different actor-critic method that will
allow us to draw a connection between one-step RL and critic regularization.
We experimentally validate this equivalence in Sec. 5.1. Despite its
difference from practically-used methods, such as one-step RL and CQL, we will
show that it makes accurate predictions about the behavior of these practical
methods (Sec. 5.2 and 5.3).

###  3.4 Classifier Actor Critic

To support our analysis, we will introduce a new actor-critic algorithm. This
algorithm is similar to prior work, but trains the critic using a cross
entropy loss instead of an MSE loss. We introduce this algorithm not because
we expect it to perform better than existing actor-critic methods, but rather
because it allows us to make precise a connection between actor and critic
regularization. This method treats the value function like a classifier, so we
will call it _classifier actor critic_. We will then introduce actor-
regularized and critic-regularized versions of this method. The subsequent
section (Sec. 4) will show that these two regularized methods learn the same
policy.

The key to our analysis will be to treat Q-values like probabilities, so we
define the critic loss in terms of a cross-entropy loss, similar to prior work
(Kalashnikov et al., 2018; Eysenbach et al., 2021). Recalling that Q-values
are positive (Sec. 3.1), we transform the Q-values to have the correct range
by using $\frac{Q}{Q+1}\in[0,1)$. We will minimize the cross-entropy loss
applied to the transformed Q-values:

$\displaystyle\mathbb{E}_{\begin{subarray}{c}p(s,a)\end{subarray}}\left[\mathcal{CE}\left(\frac{Q(s,a)}{Q(s,a)+1};\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}\right)\right]$
$\displaystyle=-\mathbb{E}_{p(s,a)}\left[\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}\log\frac{Q(s,a)}{Q(s,a)+1}\right.$
$\displaystyle\hskip
80.00012pt\left.+\frac{1}{y^{\pi,Q_{t}}(s,a)+1}\log\frac{1}{Q(s,a)+1}\right]$
$\displaystyle\stackrel{{\scriptstyle\text{const.}}}{{=}}-\mathbb{E}_{p(s,a)}\bigg{[}y^{\pi,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}$
$\displaystyle\hskip 40.00006pt\underbrace{\hskip
80.00012pt+\log\frac{1}{Q(s,a)+1}}_{\triangleq{\mathcal{L}}_{\text{critic}}(Q,y^{\pi,Q_{t}})}\bigg{]}.$
(3)

In the last line we scale both the positive and negative term by
$y^{\pi,Q_{t}}(s,a)+1$, a choice that does not change the optimal classifier
but reduces notational clutter. When the TD target can be computed exactly,
solving this optimization problem results in performing one SARSA update:
$Q(s,a)\leftarrow r(s,a)+\gamma Q(s^{\prime},a^{\prime})$ (see Lemma 4.1).
Thus, by solving this optimization problem many times, each time using the
previous Q-value to compute the TD targets, we will converge to the correct
Q-values (see Lemma 4.1). The actor objective is to maximize the expected
_log_ of the Q-values:

$\displaystyle\max_{\pi}{\mathcal{L}}_{\text{actor}}(\pi)\triangleq\mathbb{E}_{p(s)\pi(a\mid
s)}\left[\log(Q^{\pi}(s,a))\right]$ (4)

where $Q^{\pi}(s,a)=\lim_{t\rightarrow\infty}Q_{t}(s,a)$ and

$\displaystyle
Q_{t+1}=\operatorname*{arg\,min}_{Q}{\mathcal{L}}_{\text{critic}}(Q,y^{\pi,Q_{t}}).$

While most actor-critic methods do not use the logarithm transformation, prior
work on conditional behavioral cloning (e.g., (Savinov et al., 2018; Ding et
al., 2019; Sun et al., 2019; Ghosh et al., 2020; Srivastava et al., 2019))
implicitly includes this transformation (Eysenbach et al., 2022). In the
absence of additional regularization, the optimal policy $\pi(a\mid
s)=\mathbbm{1}(a=\operatorname*{arg\,max}_{a^{\prime}}Q(s,a^{\prime}))$ is the
same as the optimal policy for the standard actor objective (without the
logarithm). We next introduce a one-step version of this method, as well as a
critic regularization variant that resembles CQL. While we will implicitly use
a regularization coefficient of $1$ below, Appendix B.1 discusses versions of
classifier actor critic with varying degrees of regularization.

#### One-step RL.

To make classifier actor critic resemble one-step RL (Brandfonbrener et al.,
2021), we make two changes: estimating the value of the behavioral policy and
adding a regularization term to the actor objective. To estimate the value of
the behavioral policy, we modify the critic loss to sample the next action
$a^{\prime}$ from the behavioral policy (i.e., we use $y^{\beta,Q_{t}}(s,a)$
rather than $y^{\pi,Q_{t}}(s,a)$). We also regularize the policy by adding a
relative entropy term to the actor loss, analogous to the reverse KL penalty
used in one-step RL:

$\displaystyle\max_{\pi}\mathbb{E}_{p(s)\pi(a\mid s)}\left[\log
Q^{\beta}(s,a)+\log\beta(a\mid s)-\log\pi(a\mid s)\right]$ (5)

where $Q^{\beta}(s,a)=\lim_{t\rightarrow\infty}Q_{t}(s,a)$ and

$\displaystyle
Q_{t+1}=\operatorname*{arg\,min}_{Q}{\mathcal{L}}_{\text{critic}}(Q,y^{\beta,Q_{t}}).$

In tabular settings, this critic objective estimates the Q-values for
$\beta(a\mid s)$ (Appendix Lemma 4.1).

#### Critic regularization.

To emulate CQL, we modify the critic loss (Eq. 3) by adding a penalty term
that decreases the values for unseen actions. Whereas CQL applies this penalty
to the Q-values directly, we will apply it to the logarithm of the
Q-values:111From a dimensional analysis perspective (Huntley, 1967), this
choice makes sense because it allows the penalty term to have the same “units”
as the critic loss: log Q-values. A second motivation for regularizing the
logarithm is that the actor loss uses a logarithm.

$\displaystyle\max_{\pi}\;\mathbb{E}_{p(s)\pi(a\mid s)}\left[\log
Q_{r}^{\pi}(s,a)\right]$ (6)

where $Q_{r}^{\pi}(s,a)=\lim_{t\rightarrow\infty}Q_{t}(s,a)$ and
$Q_{t+1}(s,a)=\operatorname*{arg\,min}_{Q}{\mathcal{L}}_{\text{critic}}^{r}(Q,y^{\pi,Q_{t}})$:

$\displaystyle{\mathcal{L}}_{\text{critic}}^{r}(Q,y^{\pi,Q_{t}})\triangleq{\mathcal{L}}_{\text{critic}}(Q,y^{\pi,Q_{t}})$
$\displaystyle+\lambda\Big{(}\mathbb{E}_{\\!\begin{subarray}{c}p(s)\\\
\pi(a\mid
s)\end{subarray}\\!\\!}\left[\log(Q(s,a)+1)\right]-\mathbb{E}_{\begin{subarray}{c}\\!p(s)\\\
\beta(a\mid s)\end{subarray}\\!\\!}\left[\log(Q(s,a)+1)\right]\Big{)}.$

##  4 A Connection between One-Step RL and Critic Regularization

This section provides our main result, which is that actor and critic
regularization yield the same policy under some settings. The key to proving
this connection will be to analyze the Q-values learned by critic
regularization. While we mainly focus on the single-task setting, Sec. 4.1
describes how similar results also apply to other settings, including goal-
conditioned RL, example-based control, and settings with smaller degrees of
regularization. All proofs are in Appendix A.

To relate one-step RL to critic regularization, we start by analyzing the
Q-values learned by both methods. We first show that the classifier critic
converges to the correct Q-values:

######  Lemma 4.1.

Assume that states and actions are tabular (discrete and finite), that rewards
are positive, and that TD targets can be computed exactly (without sampling).
Incrementally update the critic by solving a sequence of optimization
problems:

$Q_{t+1}\leftarrow\operatorname*{arg\,min}_{Q}{\mathcal{L}}_{\text{critic}}(Q,y^{\pi,Q_{t}}).$

This sequence of Q-functions will converge to $Q^{\pi}$:

$\lim_{t\rightarrow\infty}Q_{t}(s,a)=Q^{\pi}(s,a)\text{ for all states $s$ and
actions $a$}.$

Because one-step RL trains the critic using
${\mathcal{L}}_{\text{critic}}(Q,y^{\beta,Q})$, it learns Q-values
corresponding to $Q^{\beta}(s,a)$. When regularization is added to the critic
updates, it learns different Q-values. Perhaps surprisingly, this
regularization means that our estimates for the value of policy $\pi(a\mid s)$
look like the value of the original behavioral policy:

######  Lemma 4.2.

Assume that states and actions are tabular (discrete and finite), that rewards
are positive, and that TD targets can be computed exactly (without sampling).
Incrementally update the critic by minimizing a sequence of regularized critic
losses using policy $\pi$ and hyperparameter $\lambda=1$:

$Q_{t+1}\leftarrow\operatorname*{arg\,min}_{Q}{\mathcal{L}}_{\text{critic}}^{r}(Q,y^{\pi,Q_{t}}).$

In the limit, this sequence of Q-functions will converge to the Q-values for
the behavioral policy ($\beta(a\mid s)$), weighted by the ratio of the
behavioral and online policies:

$\lim_{t\rightarrow\infty}Q_{t}(s,a)=\frac{Q^{\beta}(s,a)\beta(a\mid
s)}{\pi(a\mid s)}$

for all states $s$ and actions $a$.

###### Proof sketch.

The ratio $\frac{\beta(a\mid s)}{\pi(a\mid s)}$ above is an importance weight.
Ordinarily, a TD backup for policy $\pi(a\mid s)$ would entail sampling an
action $a\sim\pi(a\mid s)$. However, this importance weight means that TD
backup is effectively performed by sampling an action $a\sim\beta(a\mid s)$.
Such a TD backup resembles the TD backup for $\beta(a\mid s)$. The full proof
is in Appendix A. ∎

Intuitively, this result says that critic regularization reweights the
Q-values to assign higher values to in-distribution actions, where
$\beta(a\mid s)$ is large. An unexpected part of this result is that the
Q-values correspond to the behavioral policy. In other words, critic
regularization added to a multi-step RL method (one using
$y^{\pi,Q_{t}}(s,a)$) yields the same critic as a one-step RL method (one
using $y^{\beta,Q_{t}}(s,a)$). Our main result is a direct corollary of this
Lemma:

######  Theorem 4.3.

Let a behavioral policy $\beta(a\mid s)$ be given and let $Q^{\beta}(s,a)$ be
the corresponding value function. Let $\pi(a\mid s)$ be an arbitrary policy
(typically learned) with support constrained to $\beta(a\mid s)$ (i.e.,
$\pi(a\mid s)>0\implies\beta(a\mid s)>0$). Let $Q_{r}^{\pi}(s,a)$ be the
critic obtained by the regularized critic update (Eq. 6) to this policy with
$\lambda=1$. Then critic regularization results in the same policy as one-step
RL:

$\mathbb{E}_{\pi(a\mid s)}\left[\log
Q_{r}^{\pi}(s,a)\right]=\mathbb{E}_{\pi(a\mid s)}\left[\log
Q^{\beta}(s,a)+\log\frac{\beta(a\mid s)}{\log\pi(a\mid s)}\right]$

for all states $s$.

MDPs where regularization decreases returns.

MDP where regularization increases returns.

Figure 2:  Actor and critic regularization produce identical policies. Across
three tabular settings, we plot the action probabilities $\pi(a\mid s)$ for
the policies produced by one-step RL and critic-regularized classifier actor-
critic ($R^{2}\geq 0.999$). We also plot the action probabilities for a policy
learned by an unregularized policy to confirm that the equivalence between
one-step RL and critic regularization is not a coincidence.

Since both forms of regularization result in the same objective for the actor,
they must produce the same policy in the end. While prior work has mentioned
that critic regularization implicitly regularizes the policy (Yu et al.,
2021), this result shows that under the assumptions stated above, the implicit
regularization of critic regularization results in the exact same policy
learning objective as one-step RL. This equivalence holds when $\lambda=1$,
and not necessarily for other regularization coefficients. Appendix B.1 shows
how a variant of this result that includes an additional regularization
mechanism does apply to different regularization coefficients. This connection
between one step RL and critic regularization concerns their _objective
functions_ , not the _procedures_ used to optimize those objective functions.
Indeed, because practical offline RL algorithms sometimes use different
optimization procedures (e.g., TD vs. MC estimates of $Q^{\beta}(s,a)$), they
will incur errors in estimating $Q^{\beta}(s,a)$, violating Theorem 4.3’s
assumption that these Q-values are estimated exactly.

#### Limitations.

Our theoretical analysis makes assumptions that may not always hold in
practice. For example, our results use a critic loss based on the cross
entropy loss, while most (but not all (Kalashnikov et al., 2018; Eysenbach et
al., 2020b)) practical methods use the MSE. Our analysis assumes that critic
regularization arrives at an equilibrium, and ignores errors introduced by
function approximation and sampling. Nonetheless, our theoretical results will
make accurate predictions about prior offline RL methods.

###  4.1 Extensions of the Analysis

We extend this analysis in three ways. _First_ , we also show that a similar
connection can be established for lesser degrees of regularization
($\lambda<1$) (see Appendix B.1). _Second_ , we show that a similar connection
holds for RL problems defined via success examples (Pinto & Gupta, 2016; Tung
et al., 2018; Kalashnikov et al., 2021; Singh et al., 2019; Zolna et al.,
2020; Calandra et al., 2017; Eysenbach et al., 2021) These results use
existing actor-critic method, rather than classifier actor critic (see
Appendix C). _Third_ , we extend our analysis to multi-task settings by
looking at goal-conditioned RL problems. Taken together, these extensions show
that the connection between actor and critic regularization extends to other
commonly-studied problem settings.

##  5 Numerical Simulations

Our numerical simulations study whether the theoretical connection between
actor and critic regularization holds empirically. The first experiments (Sec.
5.1) will use classifier actor-critic, and we will expect the equivalence to
hold exactly in this setting. We then study whether this connection still
holds for practical prior methods (one-step RL and CQL), which violate our
assumptions. We study these commonly-used methods in both tabular settings
(Sec. 5.2) and on a benchmark offline RL task with continuous states and
actions (Sec. 5.3). We do not expect these methods to always be the same (see,
e.g., Kostrikov et al. (2021, Table 1)), and we will focus our experiments on
critic regularization with moderate regularization coefficients. See Appendix
F for details and hyperparameters for the experiments. Code for the tabular
experiments is available online.222Code: https://github.com/ben-eysenbach/ac-
connection

###  5.1 Exact Equivalence for Classifier Actor Critic

(a) reward function

(b) Q-learning

(c) One-step RL

(d) CQL($\lambda=10$)

(e) CQL($\lambda=0.1$)

Figure 3:  CQL can behave like one-step RL. We design a gridworld _(a)_ so
that one-step RL _(c)_ learns a suboptimal policy. For the three cells
highlighted in blue, the optimal policy _(b)_ navigates towards the high-
reward state (green) while the one-step RL policy _(c)_ navigates away from
the high-reward state. _(d)_ CQL with a large regularization coefficient
exhibits the same suboptimal behavior as one-step RL, taking actions that lead
away from the high-reward states. _(e)_ CQL with a small regularization
coefficient behaves like Q-learning. For clarity, we only show the argmax
action in each state; we omit the arrow when the argmax action is “do
nothing”.

Our first experiment aims to validate our theoretical result under the
required assumptions: when using classifier actor-critic as the RL algorithm,
and when using a tabular environment. We use a $5\times 5$ deterministic
gridworld with 5 actions (up/down/left/right/nothing). We describe the reward
function and other details in Appendix F. To ensure that critic regularization
converges to a fixed point and to avoid oscillatory learning dynamics, we
update the policy using an exponential moving average. We also include
(unregularized) classifier actor-critic to confirm that regularization is
important in some settings.

We compare these three methods in three environments. The first setting (Fig.
2 _(left)_) checks our theory that one-step RL and critic regularization
should obtain the same policy. The second setting (Fig. 2 _(center)_) shows
that one-step RL and critic regularization learn the same (suboptimal) policy
in settings where using the Q-values for the behavioral policy lead to a
suboptimal policy. The final setting is designed so that regularization
increases the expected returns. The dataset is a single trajectory from the
initial state to the goal. With such limited data, unregularized classifier
actor critic overestimates the Q-values at unseen actions, learning a policy
that mistakenly takes these actions. In contrast, the regularized approaches
learn to imitate the expert trajectory. Fig. 2 _(right)_ shows that both forms
of regularization produce the optimal policy. In summary, these tabular
experiments validate our theoretical results, including in settings where
regularization is useful and harmful. These experiments also demonstrate that
the actor-critic method introduced in Sec. 3.4 does converge (Lemma 4.1).

###  5.2 Predictions about Prior Methods: Tabular Setting

Based on our theoretical analysis, we predict that practical implementations
of one-step RL and critic regularization will exhibit similar behavior, for a
certain critic regularization coefficient. This section studies the tabular
setting, and the following section will use a continuous control benchmark.
For critic regularization, we used CQL (Kumar et al., 2020) together with soft
value iteration; following (Brandfonbrener et al., 2021), we implement one-
step RL (reverse KL) using Q-learning.

We designed a deterministic gridworld so one-step RL would fail to learn the
optimal policy (see Fig. 3 _(left)_). If CQL interpolates between the
behavioral policy (random) and the optimal policy, then the argmax action
would always be the same as the action for $\pi^{*}$. Based on our analysis,
we make a different prediction: that CQL will learn a policy similar to the
one-step RL policy. We show results in Fig. 3, just showing the argmax action
for visual clarity. The CQL policy takes actions away from both the high-
reward state and the low reward state, similar to the behavioral policy but
different from both the behavioral policy and the optimal policy. This
experiment suggests that CQL can exhibit behavior similar to one-step RL. Of
course, this effect is mediated by the regularization strength: a larger
regularization coefficient would cause CQL to learn a random policy, and a
coefficient of 0 would make CQL identical to Q-learning. We extend Fig. 3 to
include classifier actor critic and regularized variants in Appendix Fig. 9.

Figure 4:  CQL and one-step RL take similar actions on most MDPs that resemble
Fig. 3

#### How often does one-step RL approximate CQL?

To show that the results in Fig. 3 are not cherry-picked, we repeated this
experiment using 100 MDPs that are structurally similar to that in Fig. 3, but
where the locations of the high-reward and low reward state are randomized. In
each randomly generated MDP, we determine whether CQL exhibits behavior
similar to one-step RL by looking at the states where CQL takes actions that
differ from the reward-maximizing actions (as determined by running Q-learning
with unlimited data). Since there are five total actions, a random policy
would have a similarity score of 20%. As shown in Fig. 4, the similarity score
is significantly higher than chance for the vast majority of MDPs, showing
that one-step RL and CQL($\lambda=10$) produce similar policies on most such
gridworlds.

#### When does one-step RL approximate CQL?

Because one-step RL is highly regularized (policy iteration is truncated after
just one step), one might imagine that it would be most similar to CQL with a
very large regularization coefficient. To study this, we use the same
environment (Fig. 3) and measure the fraction of states where one-step RL and
CQL choose the same argmax action. As shown in Fig. 5, one-step RL is most
similar to CQL with _moderate_ regularization ($\lambda=10$), and is less
similar to CQL with a very strong regularization.

Figure 5:  One-step RL is most similar to CQL with a moderate regularization
coefficient.

###  5.3 Predictions about Prior Methods: Continuous Control Setting

Our final set of experiments studies whether our theoretical results can make
accurate testable predictions about practically-used regularization methods in
a setting where they are commonly used: offline RL benchmarks with continuous
states and actions. For these experiments, we will use well-tuned
implementations of CQL and one-step RL from Hoffman et al. (2020), using the
default hyperparameters without modification. While our theoretical results do
not apply directly to these practical methods, which violate the assumptions
in our analysis, they nonetheless raise the questions of whether these methods
perform similarly in practice. We made one change to the one-step RL
implementation to makethe comparison more fair: because CQL learns two Q
functions and takes the minimum (a trick introduced in Fujimoto et al.
(2018)), we applied this same parametrization to the one-step RL
implementation. Since offline RL methods can perform differently on datasets
of varying quality (Wang et al., 2020; Fujimoto & Gu, 2021; Paine et al.,
2020; Wang et al., 2021; Fujimoto et al., 2019), we will repeat our
experiments on four datasets from the D4RL benchmark (Fu et al., 2020).

#### Lower bounds on Q-values.

One oft-cited benefit of critic regularization is that it has guarantees about
value-estimation (Kumar et al., 2020): under appropriate assumptions, the
learned value function will underestimate the discounted expected returns of
the policy. Because our analysis shows a connection between one-step RL and
critic regularization, it raises the question of whether one-step RL methods
have similar value-estimation properties. Taken at face value, this hypothesis
seems obvious: the behavioral critic estimates the value of the behavioral
policy, so it should underestimate the value of any policy that is better than
the behavioral policy. Despite this, the lower bound property of methods like
one-step RL are rarely discussed, suggesting that it has yet to be widely
appreciated.

Figure 6:  Q-value under/over-estimation. (Top) Experiments on benchmark
datasets of varying quality show that one-step RL underestimates the Q-values.
(Bottom) Despite the theoretical guarantees about critic regularization (CQL)
yielding underestimates, in practice we observe that the values learned via
critic regularization can sometimes overestimate the actual returns. We plot
the mean and standard deviation across five random seeds. Note that the
Q-values are equivalent to the value function $V^{\pi}(s)$.

Fig. 6 shows both these predicted and actual (discounted) returns throughout
the course of training. The results for one-step RL confirm our theoretical
prediction on $\nicefrac{{4}}{{4}}$ datasets: the Q-values from one-step RL
underestimate the actual returns. In contrast, we observe that critic
regularization overestimates the true returns on $\nicefrac{{2}}{{4}}$
environments, perhaps because the regularization coefficients used to achieve
good returns in practice are too weak to guarantee the lower bound property,
and perhaps because the theoretical guarantees are only guaranteed to hold at
convergence. In total, these experiments confirm our theoretical predictions
that one-step RL will result in Q-values that are underestimations, while also
questioning the claim that critic regularization methods are always preferable
for ensuring underestimation.

#### Critic regularization causes actor regularization.

Our analysis in Sec. 4 not only suggests that one-step RL methods might
inherit properties of critic regularization (as studied in the previous
section), but also suggests that critic regularization methods may behave like
one-step methods. In particular, while critic regularization methods such as
CQL do not explicitly regularize their actor, we hypothesize that they
_implicitly_ regularize the actor (Lemma 4.2), similar to how one-step RL
methods explicitly regularize the actor.

Figure 7:  Critic regularization causes actor regularization. Performing
critic regularization via CQL implicitly results in actor regularization,
similar to one-step RL: the MSE between the predicted actions and the dataset
actions decreases. We plot the mean and standard deviation across five random
seeds.

We measure the MSE between the action in the dataset and the action predicted
by the learned policy. Fig. 7 shows the results. Some CQL implementations,
including ours, “warm-start” the actor by applying a behavioral cloning loss
for 50,000 iterations; we omit these initial gradient steps from our plots so
that any effect is caused solely by the critic regularization. On
$\nicefrac{{4}}{{4}}$ datasets, we observe that the MSE between the CQL
policy’s actions and the actions in the datasets decreases throughout
training. Perhaps the one exception is on the medium-replay dataset, where the
MSE eventually starts to increase after 5e5 gradient steps. While directly
regularizing the actor leads to MSE errors that are $\sim 3\times$ smaller,
this plot nevertheless provides evidence that critic regularization indirectly
regularizes the actor.

##  6 Conclusion

In this paper, we drew a connection between two seemingly-distinct RL
regularization methods: one-step RL and critic regularization. While our
analysis made assumptions that are typically violated in practice, it
nonetheless made accurate, testable predictions about practical methods with
commonly-used hyperparameters: critic regularization methods can behave like
one-step methods, and vice versa.

#### Acknowledgements.

We thank Aviral Kumar, George Tucker, David Brandfonbrener, and Scott Fujimoto
for discussions and feedback on drafts of the paper. We thank Young Geng and
Ilya Kostrikov for sharing code. We thank Chongyi Zheng for spotting a bug
that prompted this project. This material is supported by the Fannie and John
Hertz Foundation, NSF GRFP (DGE1745016), ONR N000142312368, DARPA/AFRL
FA87502321015, and the Office of Naval Research

## References

  * Abdolmaleki et al. (2018) Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M.  Maximum a posteriori policy optimisation.  In _International Conference on Learning Representations_ , 2018.
  * Abramovich & Persson (2016) Abramovich, S. and Persson, L.-E.  Some new estimates of the ’Jensen gap’.  _Journal of Inequalities and Applications_ , 2016(1):1–9, 2016.
  * Agarwal et al. (2019) Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W.  Reinforcement learning: Theory and algorithms.  _CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep_ , 2019.
  * An et al. (2021) An, G., Moon, S., Kim, J.-H., and Song, H. O.  Uncertainty-based offline reinforcement learning with diversified Q-ensemble.  _Advances in neural information processing systems_ , 34:7436–7447, 2021.
  * Bai et al. (2022) Bai, C., Wang, L., Yang, Z., Deng, Z.-H., Garg, A., Liu, P., and Wang, Z.  Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning.  In _International Conference on Learning Representations_ , 2022.
  * Bakushinskii (1967) Bakushinskii, A. B.  A general method of constructing regularizing algorithms for a linear incorrect equation in hilbert space.  _Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki_ , 7(3):672–677, 1967.
  * Bauer et al. (2007) Bauer, F., Pereverzev, S., and Rosasco, L.  On regularization algorithms in learning theory.  _Journal of complexity_ , 23(1):52–72, 2007.
  * Brandfonbrener et al. (2021) Brandfonbrener, D., Whitney, W., Ranganath, R., and Bruna, J.  Offline RL without off-policy evaluation.  _Advances in Neural Information Processing Systems_ , 34:4933–4946, 2021.
  * Buckman et al. (2021) Buckman, J., Gelada, C., and Bellemare, M. G.  The importance of pessimism in fixed-dataset policy optimization.  In _International Conference on Learning Representations_ , 2021.
  * Calandra et al. (2017) Calandra, R., Owens, A., Upadhyaya, M., Yuan, W., Lin, J., Adelson, E. H., and Levine, S.  The feeling of success: Does touch sensing help predict grasp outcomes?  In _Conference on Robot Learning_ , pp. 314–323. PMLR, 2017.
  * Chebotar et al. (2021) Chebotar, Y., Hausman, K., Lu, Y., Xiao, T., Kalashnikov, D., Varley, J., Irpan, A., Eysenbach, B., Julian, R. C., Finn, C., et al.  Actionable models: Unsupervised offline reinforcement learning of robotic skills.  In _International Conference on Machine Learning_ , pp. 1518–1528. PMLR, 2021.
  * Chen et al. (2021) Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Ding et al. (2019) Ding, Y., Florensa, C., Abbeel, P., and Phielipp, M.  Goal-conditioned imitation learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Emmons et al. (2021) Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S.  Rvs: What is essential for offline rl via supervised learning?  In _International Conference on Learning Representations_ , 2021.
  * Eysenbach et al. (2020a) Eysenbach, B., Geng, X., Levine, S., and Salakhutdinov, R.  Rewriting history with inverse rl: Hindsight inference for policy improvement.  In _Advances in Neural Information Processing Systems_ , 2020a.
  * Eysenbach et al. (2020b) Eysenbach, B., Salakhutdinov, R., and Levine, S.  C-learning: Learning to achieve goals via recursive classification.  In _International Conference on Learning Representations_ , 2020b.
  * Eysenbach et al. (2021) Eysenbach, B., Levine, S., and Salakhutdinov, R.  Replacing rewards with examples: Example-based policy search via recursive classification.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Eysenbach et al. (2022) Eysenbach, B., Udatha, S., Salakhutdinov, R. R., and Levine, S.  Imitating past successes can be very suboptimal.  _Advances in Neural Information Processing Systems_ , 35:6047–6059, 2022.
  * Finn et al. (2016) Finn, C., Levine, S., and Abbeel, P.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International conference on machine learning_ , pp. 49–58. PMLR, 2016.
  * Fleming (1990) Fleming, H. E.  Equivalence of regularization and truncated iteration in the solution of ill-posed image reconstruction problems.  _Linear Algebra and its applications_ , 130:133–150, 1990\.
  * Fu et al. (2018) Fu, J., Singh, A., Ghosh, D., Yang, L., and Levine, S.  Variational inverse control with events: A general framework for data-driven reward definition.  _Advances in neural information processing systems_ , 31, 2018.
  * Fu et al. (2020) Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Fujimoto & Gu (2021) Fujimoto, S. and Gu, S. S.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ , 34:20132–20145, 2021.
  * Fujimoto et al. (2018) Fujimoto, S., Hoof, H., and Meger, D.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pp. 1587–1596. PMLR, 2018.
  * Fujimoto et al. (2019) Fujimoto, S., Meger, D., and Precup, D.  Off-policy deep reinforcement learning without exploration.  In _International conference on machine learning_ , pp. 2052–2062. PMLR, 2019.
  * Gao et al. (2017) Gao, X., Sitharam, M., and Roitberg, A. E.  Bounds on the Jensen gap, and implications for mean-concentrated distributions.  _arXiv preprint arXiv:1712.05267_ , 2017.
  * Geist et al. (2019) Geist, M., Scherrer, B., and Pietquin, O.  A theory of regularized Markov decision processes.  In _International Conference on Machine Learning_ , pp. 2160–2169. PMLR, 2019.
  * Ghosh et al. (2020) Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C. M., Eysenbach, B., and Levine, S.  Learning to reach goals via iterated supervised learning.  In _International Conference on Learning Representations_ , 2020.
  * Gülçehre et al. (2020) Gülçehre, Ç., Wang, Z., Novikov, A., Le Paine, T., Colmenarejo, S. G., Zolna, K., Agarwal, R., Merel, J., Mankowitz, D. J., Paduraru, C., et al.  RL unplugged: Benchmarks for offline reinforcement learning.  2020\.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pp. 1861–1870. PMLR, 2018.
  * Hatch et al. (2022) Hatch, K., Yu, T., Rafailov, R., and Finn, C.  Example-based offline reinforcement learning without rewards.  _Proceedings of Machine Learning Research vol_ , 144:1–17, 2022.
  * Hazan et al. (2019) Hazan, E., Kakade, S., Singh, K., and Van Soest, A.  Provably efficient maximum entropy exploration.  In _International Conference on Machine Learning_ , pp. 2681–2691. PMLR, 2019.
  * Hoffman et al. (2020) Hoffman, M., Shahriari, B., Aslanides, J., Barth-Maron, G., Behbahani, F., Norman, T., Abdolmaleki, A., Cassirer, A., Yang, F., Baumli, K., Henderson, S., Novikov, A., Colmenarejo, S. G., Cabi, S., Gulcehre, C., Paine, T. L., Cowie, A., Wang, Z., Piot, B., and de Freitas, N.  Acme: A research framework for distributed reinforcement learning.  _arXiv preprint arXiv:2006.00979_ , 2020.
  * Huntley (1967) Huntley, H. E.  _Dimensional analysis_.  Dover publications, 1967.
  * Jaques et al. (2019) Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  _arXiv preprint arXiv:1907.00456_ , 2019.
  * Kalashnikov et al. (2018) Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Conference on Robot Learning_ , pp. 651–673. PMLR, 2018.
  * Kalashnikov et al. (2021) Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jonschkowski, R., Finn, C., Levine, S., and Hausman, K.  Scaling up multi-task robotic reinforcement learning.  In _5th Annual Conference on Robot Learning_ , 2021.
  * Kostrikov et al. (2021) Kostrikov, I., Nair, A., and Levine, S.  Offline reinforcement learning with implicit Q-learning.  In _International Conference on Learning Representations_ , 2021.
  * Kumar et al. (2019a) Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S.  Stabilizing off-policy Q-learning via bootstrapping error reduction.  _Advances in Neural Information Processing Systems_ , 32, 2019a.
  * Kumar et al. (2019b) Kumar, A., Peng, X. B., and Levine, S.  Reward-conditioned policies.  _arXiv preprint arXiv:1912.13465_ , 2019b.
  * Kumar et al. (2020) Kumar, A., Zhou, A., Tucker, G., and Levine, S.  Conservative Q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Lange et al. (2012) Lange, S., Gabel, T., and Riedmiller, M.  Batch reinforcement learning.  In _Reinforcement learning_ , pp. 45–73. Springer, 2012.
  * Levine et al. (2020) Levine, S., Kumar, A., Tucker, G., and Fu, J.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Li et al. (2020) Li, A., Pinto, L., and Abbeel, P.  Generalized hindsight for reinforcement learning.  _Advances in neural information processing systems_ , 33:7754–7767, 2020.
  * Lu et al. (2021) Lu, C., Ball, P., Parker-Holder, J., Osborne, M., and Roberts, S. J.  Revisiting design choices in offline model based reinforcement learning.  In _International Conference on Learning Representations_ , 2021.
  * Lynch & Sermanet (2021) Lynch, C. and Sermanet, P.  Language conditioned imitation learning over unstructured data.  _Proceedings of Robotics: Science and Systems_ , 2021.
  * Lyu et al. (2022) Lyu, J., Ma, X., Yan, J., and Li, X.  Efficient continuous control with double actors and regularized critics.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 7655–7663, 2022.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A.  Playing atari with deep reinforcement learning.  _ArXiv_ , abs/1312.5602, 2013.
  * Nachum et al. (2019) Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., and Schuurmans, D.  Algaedice: Policy gradient from arbitrary experience.  _arXiv preprint arXiv:1912.02074_ , 2019.
  * Nash (1951) Nash, J.  Non-cooperative games.  _Annals of mathematics_ , pp. 286–295, 1951.
  * Neu et al. (2017) Neu, G., Jonsson, A., and Gómez, V.  A unified view of entropy-regularized Markov decision processes.  _arXiv preprint arXiv:1705.07798_ , 2017.
  * Paine et al. (2020) Paine, T. L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N.  Hyperparameter selection for offline reinforcement learning.  _arXiv preprint arXiv:2007.09055_ , 2020.
  * Paster et al. (2021) Paster, K., McIlraith, S. A., and Ba, J.  Planning from pixels using inverse dynamics models.  In _International Conference on Learning Representations_ , 2021.
  * Peng et al. (2019) Peng, X. B., Kumar, A., Zhang, G., and Levine, S.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * Peters & Schaal (2007) Peters, J. and Schaal, S.  Reinforcement learning by reward-weighted regression for operational space control.  In _Proceedings of the 24th international conference on Machine learning_ , pp. 745–750, 2007.
  * Peters et al. (2010) Peters, J., Mulling, K., and Altun, Y.  Relative entropy policy search.  In _Twenty-Fourth AAAI Conference on Artificial Intelligence_ , 2010\.
  * Pinto & Gupta (2016) Pinto, L. and Gupta, A.  Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.  In _2016 IEEE international conference on robotics and automation (ICRA)_ , pp. 3406–3413. IEEE, 2016.
  * Rezaeifar et al. (2022) Rezaeifar, S., Dadashi, R., Vieillard, N., Hussenot, L., Bachem, O., Pietquin, O., and Geist, M.  Offline reinforcement learning as anti-exploration.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 8106–8114, 2022.
  * Santos (1996) Santos, R. J.  Equivalence of regularization and truncated iteration for general ill-posed problems.  _Linear algebra and its applications_ , 236:25–33, 1996\.
  * Savinov et al. (2018) Savinov, N., Dosovitskiy, A., and Koltun, V.  Semi-parametric topological memory for navigation.  In _International Conference on Learning Representations_ , 2018.
  * Scherrer (2013) Scherrer, B.  Improved and generalized upper bounds on the complexity of policy iteration.  _Advances in Neural Information Processing Systems_ , 26, 2013.
  * (62) Siegel, N., Springenberg, J. T., Berkenkamp, F., Abdolmaleki, A., Neunert, M., Lampe, T., Hafner, R., Heess, N., and Riedmiller, M.  Keep doing what worked: Behavior modelling priors for offline reinforcement learning.  In _International Conference on Learning Representations_.
  * Singh et al. (2019) Singh, A., Yang, L., Hartikainen, K., Finn, C., and Levine, S.  End-to-end robotic reinforcement learning without reward engineering.  _arXiv preprint arXiv:1904.07854_ , 2019.
  * Srivastava et al. (2019) Srivastava, R. K., Shyam, P., Mutz, F., Jaśkowski, W., and Schmidhuber, J.  Training agents using upside-down reinforcement learning.  _arXiv preprint arXiv:1912.02877_ , 2019.
  * Sun et al. (2019) Sun, H., Li, Z., Liu, X., Zhou, B., and Lin, D.  Policy continuation with hindsight inverse dynamics.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tung et al. (2018) Tung, H.-Y., Harley, A. W., Huang, L.-K., and Fragkiadaki, K.  Reward learning from narrated demonstrations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 7004–7013, 2018.
  * Vieillard et al. (2020) Vieillard, N., Kozuno, T., Scherrer, B., Pietquin, O., Munos, R., and Geist, M.  Leverage the average: an analysis of kl regularization in reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:12163–12174, 2020.
  * Villaflor et al. (2020) Villaflor, A., Dolan, J., and Schneider, J.  Fine-tuning offline reinforcement learning with model-based policy optimization.  _Offline Reinforcement Learning Workshop at Neural Information Processing Systems_ , 2020.
  * Wahba (1987) Wahba, G.  Three topics in ill-posed problems.  In _Inverse and ill-posed problems_ , pp. 37–51. Elsevier, 1987\.
  * Wang et al. (2018) Wang, Q., Xiong, J., Han, L., Liu, H., Zhang, T., et al.  Exponentially weighted imitation learning for batched historical data.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Wang et al. (2021) Wang, R., Wu, Y., Salakhutdinov, R., and Kakade, S.  Instabilities of offline rl with pre-trained neural representation.  In _International Conference on Machine Learning_ , pp. 10948–10960. PMLR, 2021.
  * Wang et al. (2020) Wang, Z., Novikov, A., Zolna, K., Merel, J. S., Springenberg, J. T., Reed, S. E., Shahriari, B., Siegel, N., Gulcehre, C., Heess, N., et al.  Critic regularized regression.  _Advances in Neural Information Processing Systems_ , 33:7768–7778, 2020.
  * Wen et al. (2021) Wen, J., Kumar, S., Gummadi, R., and Schuurmans, D.  Characterizing the gap between actor-critic and policy gradient.  In _International Conference on Machine Learning_ , pp. 11101–11111. PMLR, 2021.
  * Wu et al. (2019) Wu, Y., Tucker, G., and Nachum, O.  Behavior regularized offline reinforcement learning.  _arXiv preprint arXiv:1911.11361_ , 2019.
  * Yang et al. (2021) Yang, R., Fang, M., Han, L., Du, Y., Luo, F., and Li, X.  MHER: Model-based hindsight experience replay.  _ArXiv_ , abs/2107.00306, 2021.
  * Yu et al. (2021) Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C.  Combo: Conservative offline model-based policy optimization.  _Advances in neural information processing systems_ , 34:28954–28967, 2021.
  * Zhou et al. (2021) Zhou, W., Bajracharya, S., and Held, D.  Plas: Latent action space for offline reinforcement learning.  In _Conference on Robot Learning_ , pp. 1719–1735. PMLR, 2021.
  * Ziebart (2010) Ziebart, B. D.  _Modeling purposeful adaptive behavior with the principle of maximum causal entropy_.  Carnegie Mellon University, 2010.
  * Zolna et al. (2020) Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C., Wang, Z., Aytar, Y., Denil, M., de Freitas, N., and Reed, S.  Offline learning from demonstrations and unlabeled experience.  _arXiv preprint arXiv:2011.13885_ , 2020.

## Appendices

In the Appendices, we provide proofs of the theoretical results (Appendix C),
extend the analysis to other RL settings (Appendices C–D), and then provide
details of the experiments (Appendix F).

##  Appendix A Proofs

###  A.1 Proof of Lemma 4.1

_Proof sketch._ Lemma 4.1 shows that classifier actor critic converges. The
key idea of the proof will be to show that the incremental updates for
classifier actor critic are exactly the same as the incremental updates for
Q-learning. Q-learning converges, so an algorithm that performs the same
incremental updates as Q-learning must also converge.

###### Proof.

As the cross entropy loss is minimized when the predictions equal the labels,
updates for ${\mathcal{L}}_{\text{critic}}(Q,\pi)$ can be written as
$\frac{Q(s,a)}{Q(s,a)+1}\leftarrow\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}$.
If the updates are performed by averaging over all possible next states (e.g.,
in the tabular setting), these updates are equivalent to directly updating
$Q(s,a)\leftarrow y^{\pi,Q_{t}}(s,a)=r(s,a)+\gamma\mathbb{E}_{p(s^{\prime}\mid
s,a)\pi(a^{\prime}\mid s^{\prime})}\left[Q_{t}(s^{\prime},a^{\prime})\right]$,
which is the standard policy evaluation update for policy $\pi(a\mid s)$.
Thus, we can invoke the standard result that policy evaluation converges to
$Q^{\pi}$ (Agarwal et al., 2019, Theorem 1.14.) to argue that updates for
${\mathcal{L}}_{\text{critic}}$ likewise converge to $Q^{\pi}$. ∎

In this proof, the TD targets were the expectation over the next state and
next action. If Eq. 3 were optimized using a single-sample estimate of this
expectation, ${\mathbf{y}}=r(s,a)+\gamma Q_{t}(s^{\prime},a^{\prime})$, then
the updates would be biased:

$\frac{Q(s,a)}{Q(s,a)+1}\leftarrow\mathbb{E}\left[\frac{{\mathbf{y}}}{{\mathbf{y}}+1}\right]\leq\frac{\mathbb{E}[{\mathbf{y}}]}{\mathbb{E}[{\mathbf{y}}]+1}=\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}.$

In settings with stochastic transitions or policies, these updates would
result in estimating a lower bound on $Q^{\pi}(s,a)$.

###  A.2 Proof of Lemma 4.2 and Theorem 4.3

###### Proof.

Our proof proceeds in three steps. First, we derive the update equations for
the regularized critic update. That is, if we maintained a table of Q-values,
what would the new value for $Q(s,a)$ be? Second, we show that these updates
are equivalent to performing policy evaluation on a _re-parametrized_ critic
$\tilde{Q}(s,a)=Q(s,a)\frac{\pi(a\mid s)}{\beta(a\mid s)}$. We invoke the
standard results for policy evaluation to prove convergence that
$\tilde{Q}(s,a)$ convergences. Finally, we undo the reparametrization to
obtain convergence results for $Q(s,a)$.

Step 0. We start by rearranging the regularized critic objective:

$\displaystyle{\mathcal{L}}_{\text{critic}}^{r}(Q,y^{\pi,Q_{t}})$
$\displaystyle\triangleq{\mathcal{L}}_{\text{critic}}(Q,y^{\pi,Q_{t}})+\bigg{(}\mathbb{E}_{p(s)\pi(a\mid
s)}\left[\log(Q(s,a)+1)\right]-\mathbb{E}_{p(s)\beta(a\mid
s)}\left[\log(Q(s,a)+1)\right]\bigg{)}$
$\displaystyle=-\mathbb{E}_{p(s,a)}\left[y^{\pi,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}+\log\frac{1}{Q(s,a)+1}\right]$
$\displaystyle\qquad+\bigg{(}\mathbb{E}_{p(s)\pi(a\mid
s)}\left[\log(Q(s,a)+1)\right]-\mathbb{E}_{p(s)\beta(a\mid
s)}\left[\log(Q(s,a)+1)\right]\bigg{)}$
$\displaystyle=-\mathbb{E}_{p(s,a)}\left[y^{\pi,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}+\cancel{\log\frac{1}{Q(s,a)+1}}\right]$
$\displaystyle\qquad-\bigg{(}\cancel{\mathbb{E}_{p(s)\pi(a\mid
s)}\left[\log\frac{1}{Q(s,a)+1}\right]}+\mathbb{E}_{p(s)\beta(a\mid
s)}\left[\log\frac{1}{Q(s,a)+1}\right]\bigg{)}$
$\displaystyle=-\mathbb{E}_{p(s,a)}\left[y^{\pi,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}\right]+\mathbb{E}_{p(s)\beta(a\mid
s)}\left[\log\frac{1}{Q(s,a)+1}\right].$

For the cancellation on the third line, we used the fact that
$p(s,a)=p(s)\beta(a\mid s)$.

Step 1. To start, note that the regularized critic update is equivalent to a
weighted classification loss: positive examples are sampled $(s,a)\sim
p(s)\beta(a\mid s)$ and receive weight
$\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}$, and negative examples are
sampled $(s,a)\sim p(s)\pi(a\mid s)$ and receive weight
$\frac{1}{y^{\pi,Q_{t}}(s,a)+1}$. The Bayes’ optimal classifier is given by

$\displaystyle\frac{Q(s,a)}{Q(s,a)+1}=\frac{\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}p(s)\beta(a\mid
s)}{\frac{y^{\pi,Q_{t}}(s,a)}{y^{\pi,Q_{t}}(s,a)+1}p(s)\beta(a\mid
s)+\frac{1}{y^{\pi,Q_{t}}(s,a)+1}p(s)\pi(a\mid
s)}=\frac{y^{\pi,Q_{t}}(s,a)\beta(a\mid s)}{y^{\pi,Q_{t}}(s,a)\beta(a\mid
s)+\pi(a\mid s)}.$

Solving for $Q(s,a)$ on the left hand side, the optimal value for $Q(s,a)$ is
given by

$Q(s,a)=y^{\pi,Q_{t}}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid
s)}=(r(s,a)+\mathbb{E}_{p(s^{\prime}\mid s,a)\pi(a^{\prime}\mid
s^{\prime})}[Q_{t}(s^{\prime},a^{\prime})])\frac{\beta(a\mid s)}{\pi(a\mid
s)}.$ (7)

This equation tells us what each update for the regularized critic loss does.

Step 2. To analyze these updates, we define $\tilde{Q}(s,a)\triangleq
Q_{t}(s,a)\frac{\pi(a\mid s)}{\beta(a\mid s)}$. Then these updates can be
written using $\tilde{Q}(s,a)$ as

$\tilde{Q}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid
s)}=\left(r(s,a)+\mathbb{E}_{p(s^{\prime}\mid s,a)\pi(a^{\prime}\mid
s^{\prime})}\left[\tilde{Q}(s^{\prime},a^{\prime})\frac{\beta(a^{\prime}\mid
s^{\prime})}{\pi(a^{\prime}\mid s^{\prime})}\right]\right)\frac{\beta(a\mid
s)}{\pi(a\mid s)},$ (8)

which can be simplified to

$\tilde{Q}(s,a)=r(s,a)+\mathbb{E}_{p(s^{\prime}\mid s,a)\beta(a^{\prime}\mid
s^{\prime})}\left[\tilde{Q}(s^{\prime},a^{\prime})\right].$ (9)

Note that the ratio $\frac{\beta(a^{\prime}\mid
s^{\prime})}{\pi(a^{\prime}\mid s^{\prime})}$ inside the expectation acts like
an importance weight, so that the expectation over $\pi(a^{\prime}\mid
s^{\prime})$ becomes an expectation over $\beta(a^{\prime}\mid s^{\prime})$.
Thus, the regularized critic updates are equivalent to perform policy
evaluation on $\tilde{Q}(s,a)$. An immediately consequence is that the
regularized critic updates converge, and they converge to
$\tilde{Q}^{*}(s,a)=Q^{\beta}(s,a)$.

Step 3. Finally, we translate these convergence results for $\tilde{Q}(s,a)$
into convergence results for $Q(s,a)$. Written in terms of the original
Q-values, we see that the optimal critic for the regularized critic update is

$Q^{*}(s,a)=\tilde{Q}^{*}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid
s)}=Q^{\beta}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid s)}.$ (10)

This completes the proof of Lemma 4.2.

∎

We now prove Theorem 4.3 by applying a logarithm:

###### Proof.

$\log Q^{*}(s,a)=\log\left(Q^{\beta}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid
s)}\right)=\log Q^{\beta}(s,a)+\log\beta(a\mid s)-\log\pi(a\mid s).$

∎

We note that our proof does not account for stochastic and function
approximation errors. However, if we assume that the TD updates are
deterministic (e.g., as they are in deterministic MDPs), then the updates for
classifier actor-critic are identical to those of Q-learning (Lemma 4.1).
Thus, it immediately inherits any theoretical results regarding the
propagation of errors for Q-learning.

While this Theorem 4.3 shows that one-step RL and critic regularization have
the same fixed point, it does not say how many transitions or gradient updates
are required to reach those fixed points.

###  A.3 Why use the cross-entropy loss?

Our proof of Theorem 4.3 helps explain why classifier actor-critic use the
cross entropy loss for the critic loss, rather than the MSE loss. Precisely,
our analysis requires that the optimal Q function be a _ratio_ ,
$\tilde{Q}(s,a)=\frac{Q(s,a)\pi(a\mid s)}{\beta(a\mid s)}$. The cross entropy
loss can readily estimate ratios. For example, the optimal classifier for data
drawn from $p(x)$ and $q(x)$ is $C(x)=\frac{p(x)}{p(x)+q(x)}$, so the ratio
can be expressed as $\frac{C(x)}{1-C(x)}=\frac{p(x)}{q(x)}$. However, fitting
a function $C(x)$ to data drawn from (say) a 1:1 mixture of $p(x)$ and $q(x)$
would result in $C(x)=\frac{1}{2}p(x)+\frac{1}{2}q(x)$, which we cannot
transform to express the ratio $\frac{p(x)}{q(x)}$ as a function of $C(x)$.

Figure 8: Under the assumptions of Theorem 4.3, one-step RL is most similar to
critic regularization with a coefficient of $\lambda=1$.

###  A.4 Validating the Theory

Our theoretical results suggest that one-step RL and critic regularization
should be most similar with critic regularization is applied with a
regularization coefficient of $\lambda=1$. To test this hypothesis, we took
the task from Fig. 2 _(Left)_ and measured the similarity between one-step RL
and critic-regularized classifier actor critic, for varying values of the
critic regularization parameter. We measured the similarity of the policies
obtained by the two methods by counting the fraction of states where the two
methods choose the same (argmax) action. The results, shown in Fig. 8,
validate our theoretical prediction that these methods should be most similar
with $\lambda=1$.

###  A.5 What about using the policy gradient?

Our analysis fundamentally requires using TD learning: the key step is that
doing TD backups using one policy is equivalent to doing (modified) TD backups
with a different policy. However, the actor updates for both methods could be
implemented using a policy gradient or natural gradient, rather than a
straight-through gradient estimator. Indeed, much of the work on one-step RL
methods (Peng et al., 2019; Siegel et al., ) uses an actor update that
resembles a policy gradient or natural policy gradient (e.g., 1-step RL with a
reverse KL penalty (Brandfonbrener et al., 2021)).

##  Appendix B Varying the regularization coefficient

While our main analysis (Theorem 4.3)showed that regularization and critic
regularization yield the same policy when these regularizers are applied with
a certain strength, in practice the strength of regularization is controlled
by a hyperparameter. This hyperparameter raises a question: _does the
connection between one-step RL and critic regularization hold for different
values of this hyperparameter?_

In this section, we show that there remains a precise connection between actor
and critic regularization, even for different values of this hyperparameter.
This result not only suggests that the connection is stronger than initially
suggested by the main result. Proving this connection also helps highlight how
many regularization methods can be cast from a similar mold.

###  B.1 A Regularization Coefficient.

We start by modifying the actor regularizer and critic regularizer introduced
in Sec. 3.4 to include an additional hyperparameter.

#### Mixture policy.

Both the actor and critic losses will make use of a mixture policy,
$(1-\lambda)\pi(a\mid s)+\lambda\beta(a\mid s)$, where $\lambda\in[0,1]$ will
be a hyperparameter. Larger values of $\lambda$ yield a mixture policy that is
closer to the behavioral policy; this will correspond to higher degrees of
regularization. Mixtures of policies are commonly used in practice (Kumar et
al., 2020, Appendix F),(Villaflor et al., 2020, Eq. 11), (Finn et al., 2016,
Sec. 4.3) (Lyu et al., 2022) (Hazan et al., 2019, Eq. 2.5), even though it
rarely appears in theoretical offline RL literature. Indeed, because critic
regularization resembles a two-player zero-sum game, mixture policies might
even be _required_ to find a (Nash) equilibrium of the critic regularizer
(Nash, 1951).

####  $\lambda$-weighted critic loss.

With this concept of a mixture policy, we define the $\lambda$-weighted actor
and critic regularizers. For the $\lambda$-weighted critic loss, we will
change how the TD targets are computed. Instead of sampling the next action
from $\pi$ or $\beta$, we will sample the next action from a
$\lambda_{\text{TD}}$-weighted combination of these two policies, reminiscent
of how prior work has regularized the actions sampled for the TD backup
(Fujimoto et al., 2019; Zhou et al., 2021):

$y^{\lambda_{\text{TD}}}\triangleq
y^{(1-\lambda)\pi+\lambda\beta}(s,a)=r(s,a)+\gamma\mathbb{E}_{\begin{subarray}{c}p(s^{\prime}\mid
s,a)\\\ (1-\lambda_{\text{TD}})\pi(a\mid s)+\lambda_{\text{TD}}\beta(a\mid
s)\end{subarray}}[Q(s^{\prime},a^{\prime})].$

When introducing one-step RL in Sec. 3.4, we used $\lambda_{\text{TD}}=1$.

Using this TD target, the $\lambda$-weighted critic loss can now be written as
a combination of the un-regularized objective (Eq. 3) plus the regularized
objective (Eq. 6):

$\displaystyle{\mathcal{L}}_{\text{critic}}^{r}(Q,\lambda_{\text{critic}})$
$\displaystyle\triangleq(1-\lambda_{\text{critic}})\left(-\mathbb{E}_{p(s,a)}\left[\frac{y^{\lambda_{\text{TD}}}(s,a)}{y^{\lambda_{\text{TD}}}(s,a)+1}\log\frac{Q(s,a)}{Q(s,a)+1}+\frac{1}{y^{\lambda_{\text{TD}}}(s,a)+1}\log\frac{1}{Q(s,a)+1}\right]\right)$
$\displaystyle\qquad+\lambda\left(-\mathbb{E}_{\begin{subarray}{c}p(s,a)\\\
a^{-}\sim\pi(\cdot\mid
s)\end{subarray}}\left[\frac{y^{\lambda_{\text{TD}}}(s,a)}{y^{\lambda_{\text{TD}}}(s,a)+1}\log\frac{Q(s,a)}{Q(s,a)+1}+\frac{1}{y^{\lambda_{\text{TD}}}(s,a)+1}\log\frac{1}{Q(s,a)+1}\right]\right)$
$\displaystyle=-\mathbb{E}_{\begin{subarray}{c}p(s,a)\\\
a^{-}\sim(1-\lambda_{\text{critic}})\pi(\cdot\mid
s)+\lambda_{\text{critic}}\beta(\cdot\mid
s)\end{subarray}}\left[\frac{y^{\lambda_{\text{TD}}}(s,a)}{y^{\lambda_{\text{TD}}}(s,a)+1}\log\frac{Q(s,a)}{Q(s,a)+1}+\frac{1}{y^{\lambda_{\text{TD}}}(s,a)+1}\log\frac{1}{Q(s,a^{-})+1}\right].$
(11)

The second line rewrites this objective: the first term looks the same as the
original “positive” term in the critic objective, while the “negative” term
uses actions sampled from a mixture of the current policy and the behavioral
policy. When $\lambda_{\text{critic}}=1$, we recover the regularized critic
loss introduced in Sec. 3.4.

####  $\lambda$-weighted actor loss.

Finally, the strength of the actor regularizer can be controlled by changing
the reverse KL penalty. While it may seem like changing the reward scale would
varying the strength of the actor loss, this is not the case for classifier
actor critic because of the $\log(\cdot)$ in the actor loss. Instead, we will
relax the reverse KL penalty between the learned policy $\pi(a\mid s)$ and the
behavioral policy $\beta(a\mid s)$ so that only the mixture policy only needs
to be close to behavioral policy:

$\displaystyle{\mathcal{L}}_{\text{actor}}^{r}(\pi,\lambda_{\text{KL}})$
$\displaystyle\triangleq\mathbb{E}_{p(s)\pi(a\mid s)}\left[\log
Q(s,a)+\log\beta(a\mid s)-\log\left((1-\lambda_{\text{KL}})\pi(a\mid
s)+\lambda_{\text{KL}}\beta(a\mid s)\right)\right].$ (12)

As indicated on the second line, replacing $\beta(a\mid s)$ with the mixture
policy has an effect similar to that of decreasing the weight applied to the
KL penalty. The approximation on the second line is determined by the Jensen
Gap (Abramovich & Persson, 2016; Gao et al., 2017). When introducing one-step
RL in Sec. 3.4, we used $\lambda_{\text{KL}}=1$, together with
$\lambda_{\text{TD}}=1$.

In summary, the strength of the actor and critic regularizers can be
controlled through additional hyperparameters
($\lambda_{\text{critic}},\lambda_{\text{TD}},\lambda_{\text{KL}}$). Indeed,
it is typical for offline RL methods to require many hyperparameters
(Brandfonbrener et al., 2021; Lu et al., 2021; Paine et al., 2020; Wu et al.,
2019), and performance is sensitive to their settings. However, the close
connection that we have shown between actor and critic regularizers allows us
to decrease the number of hyperparameters.

###  B.2 Analysis

In our main result (Thm. 4.3), we showed that one-stel RL and critic
regularization are equivalent when
$\lambda_{\text{critic}}=\lambda_{\text{TD}}=\lambda_{\text{KL}}=1$. This is a
large value for the regularization strength, and we now consider what happens
for smaller degrees of regularization: is there still a connection between
one-step RL and critic regularization?

The following theorem will prove that this is the case. In particular,
applying critic regularization with coefficient $\lambda_{\text{critic}}$
yields the same policy as applying one-step RL with
$\lambda_{\text{TD}}=\lambda_{\text{KL}}=\lambda_{\text{critic}}$. That is,
there is a very simple recipe for converting the hyperparameters for critic
regularization into the hyperparameters for one-step RL.

######  Theorem B.1.

Let policy $\pi(a\mid s)$ be given, let $Q^{\beta}(s,a)$ be the Q-function of
the behavioral policy, and let
$Q_{r}^{\lambda_{\text{TD}}}(s,a,\lambda_{\text{critic}})$ be the critic
obtained by the $\lambda_{\text{critic}}$-weighted regularized critic update
(Eq. 11) using TD targets $y^{\lambda_{\text{TD}}}(s,a)$. If
$\lambda_{\text{critic}}=\lambda_{\text{TD}}=\lambda_{\text{KL}}$, then the
$\lambda_{\text{KL}}$-weighted actor loss (Eq. 12) is equivalent to the un-
regularized policy objective using the regularized critic:

$\displaystyle\mathbb{E}_{p(s)\pi(a\mid s)}\left[\log Q(s,a)+\log\beta(a\mid
s)-\log\left((1-\lambda_{\text{KL}})\pi(a\mid
s)+\lambda_{\text{KL}}\beta(a\mid s)\right)\right]$
$\displaystyle=\mathbb{E}_{\pi(a\mid s)}\left[\log
Q_{r}^{\lambda_{\text{TD}}}(s,a,\lambda_{\text{critic}})\right]\qquad\text{for
all states }s.$

While we used the cross entropy loss for this result, it turns out that the
result also holds for the more standard MSE loss (we omit the proof for
brevity).

#### Limitations.

Before presenting the proof in Sec. B.3, we discuss a few limitations of this
result. Like the rest of the analysis in this paper, the form of the critic
regularizer is different from that often used in practice. Additionally, our
analysis assumes ignores many sources of errors (e.g., sampling, function
approximation), and assumes that each objective is optimized exactly.

###  B.3 Proof of Theorem B.1

###### Proof.

We start by defining the fixed point of the $\lambda$-weighted regularized
critic loss. Like in the single-task setting, this loss resembles a weighted
classification problem, so we can write down the Bayes’ optimal classifier as

$\displaystyle\frac{Q(s,a)}{Q(s,a)+1}$
$\displaystyle=\frac{\frac{y^{\lambda_{\text{TD}}}(s,a)}{y^{\lambda_{\text{TD}}}(s,a)+1}p(s)\beta(a\mid
s)}{\frac{y^{\lambda_{\text{TD}}}(s,a)}{y^{\lambda_{\text{TD}}}(s,a)+1}p(s)\beta(a\mid
s)+\frac{1}{y^{\lambda_{\text{TD}}}(s,a)+1}p(s)((1-\lambda_{\text{critic}})\pi(a\mid
s)+\lambda_{\text{critic}}\beta(a\mid s))}$
$\displaystyle=\frac{y^{\lambda_{\text{TD}}}(s,a)\beta(a\mid
s)}{y^{\lambda_{\text{TD}}}(s,a)\beta(a\mid
s)+(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)}.$

Solving for $Q(s,a)$ on the left hand side, the optimal value for $Q(s,a)$ is
given by

$\displaystyle Q(s,a)$
$\displaystyle=y^{\lambda_{\text{TD}}}(s,a)\frac{\beta(a\mid
s)}{(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)}$ $\displaystyle=(r(s,a)+\mathbb{E}_{p(s^{\prime}\mid
s,a),a^{\prime}\sim(1-\lambda_{\text{TD}}),\pi(\cdot\mid
s^{\prime})+\lambda_{\text{TD}}\beta(\cdot\mid
s)}[Q(s^{\prime},a^{\prime})])\frac{\beta(a\mid
s)}{(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)}.$ (13)

Note that the next action $a^{\prime}$ is sampled from a mixture policy
defined by $\lambda_{\text{TD}}$. This equation tells us what each update for
the $\lambda$-weighted regularized critic loss does.

To analyze these updates, we define

$\tilde{Q}(s,a)\triangleq Q(s,a)\frac{(1-\lambda_{\text{critic}})\pi(a\mid
s)+\lambda_{\text{critic}}\beta(a\mid s)}{\beta(a\mid s)}.$

Like before, the ratio $\frac{\beta(a^{\prime}\mid
s^{\prime})}{(1-\lambda_{\text{TD}})\pi(a^{\prime}\mid
s^{\prime})+\lambda_{\text{TD}}\beta(a^{\prime}\mid s^{\prime})}$ can act like
an importance weight. When $\lambda_{\text{TD}}=\lambda_{\text{critic}}$, then
this importance weight cancels with the sampling distribution, providing the
following identity:

$\displaystyle\mathbb{E}_{p(s^{\prime}\mid
s,a),a^{\prime}\sim(1-\lambda_{\text{TD}}),\pi(\cdot\mid
s^{\prime})+\lambda_{\text{TD}}\beta(\cdot\mid s)}[Q(s^{\prime},a^{\prime})]$
$\displaystyle=\mathbb{E}_{p(s^{\prime}\mid
s,a),a^{\prime}\sim(1-\lambda_{\text{TD}}),\pi(\cdot\mid
s^{\prime})+\lambda_{\text{TD}}\beta(\cdot\mid
s)}\left[\tilde{Q}(s,a)\frac{\beta(a\mid
s)}{(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)}\right]$ $\displaystyle=\mathbb{E}_{p(s^{\prime}\mid
s,a),a^{\prime}\sim\beta(\cdot\mid s^{\prime})}[\tilde{Q}(s,a)].$

Substituting this identity in Eq. 13, we can write the updates using
$\tilde{Q}(s,a)$:

$\displaystyle\tilde{Q}(s,a)\frac{\beta(a\mid
s)}{(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)}$ $\displaystyle=\left(r(s,a)+\mathbb{E}_{p(s^{\prime}\mid
s,a),a^{\prime}\sim\beta(\cdot\mid
s^{\prime})}[\tilde{Q}(s,a)]\right)\frac{\beta(a\mid
s)}{(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)},$

which can be simplified to

$\displaystyle\tilde{Q}(s,a)=r(s,a)+\mathbb{E}_{p(s^{\prime}\mid
s,a),a^{\prime}\sim\beta(\cdot\mid s^{\prime})}[\tilde{Q}(s,a)].$

We then translate these convergence results for $\tilde{Q}(s,a)$ into
convergence results for $Q(s,a)$. Written in terms of the original Q-values,
we see that the optimal critic for the regularized critic update is

$Q^{*}(s,a)=Q^{\beta}(s,a)\frac{\beta(a\mid
s)}{(1-\lambda_{\text{critic}})\pi(a\mid s)+\lambda_{\text{critic}}\beta(a\mid
s)}.$ (14)

Note that this holds for any value of
$\lambda_{\text{critic}}=\lambda_{\text{TD}}\in[0,1]$. This result suggests
that two common forms of regularization, decreasing the values predicted at
unseen actions and regularizing the actions used in the TD backup, can produce
the same effect: a critic that estimates the Q-values of the behavioral policy
(multiplied by some importance weight).

Finally, substitute this Q-function into the un-regularized actor loss, we see
that the result is equivalent to the $\lambda$-weighted actor loss:

$\displaystyle\mathbb{E}_{p(s)\pi(a\mid s)}\left[\log Q^{*}(s,a)\right]=$
$\displaystyle\mathbb{E}_{p(s)\pi(a\mid s)}\bigg{[}\log
Q^{\beta}(s,a)+\underbrace{\log\beta(a\mid
s)-\log\left((1-\lambda_{\text{KL}})\pi(a\mid
s)+\lambda_{\text{KL}}\beta(a\mid s)\right)}_{\text{$\lambda$-weighted actor
regularizer}}\bigg{]}$

∎

##  Appendix C Regularization for Goal-Conditioned Problems

Like single-task RL problems, goal-conditioned RL problems have also been
approached with both one-step methods (Ghosh et al., 2020; Ding et al., 2019;
Sun et al., 2019) and critic regularization (Chebotar et al., 2021). In these
problems, the aim is to learn a goal-conditioned policy $\pi(a\mid s,s_{g})$
that maximizes the expected discounted sum of goal-conditioned rewards
$r_{g}(s,a)$, where goals are sampled $s_{g}\sim p_{g}(s_{g})$:

$\max_{\pi}\mathbb{E}_{p_{g}(s_{g})}\mathbb{E}_{\pi(\tau\mid
s_{g})}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{g}(s_{t},a_{t})\right].$

We will use the goal-conditioned reward function
$r_{g}(s,a)=p(s^{\prime}=s_{g}\mid s,a)$, which is defined in terms of the
environment dynamics. In settings with discrete states, maximizing this reward
function is equivalent to maximizing the sparse indicator reward function
($r_{g}(s,a)=\mathbbm{1}(s_{g}=s)$).

In this section, we show that one-step RL and critic regularization are
equivalent for a certain goal-conditioned actor-critic method. Unlike our
analysis in the single-task setting, this analysis here uses an existing
method, C-learning (Eysenbach et al., 2020b). C-learning is a TD method that
already makes use of the cross entropy loss for training the critic:

$\displaystyle\max_{Q}\;$
$\displaystyle(1-\gamma)\mathbb{E}_{p(s,a,s^{\prime})}\left[\log\frac{Q(s,a,s_{g}=s^{\prime})}{Q(s,a,s_{g}=s^{\prime})+1}\right]+\gamma\mathbb{E}_{p(s,a)p_{g}(s_{g})}\left[y^{\pi,Q_{t}}(s,a,s_{)}\log\frac{Q(s,a,s_{g})}{Q(s,a,s_{g})+1}\right]$
$\displaystyle+\mathbb{E}_{p(s,a)p_{g}(s_{g})}\left[\log\frac{1}{Q(s,a,s_{g}=s^{\prime})+1}\right],$

where $y^{\pi,Q_{t}}(s,a,s_{g})=\mathbb{E}_{p(s^{\prime}\mid
s,a)\pi(a^{\prime}\mid
s^{\prime},s_{g})}\left[Q(s^{\prime},a^{\prime},s_{g})\right]$ serves the role
of the TD target.

The first two terms increase the Q-values while the last term decreases the
Q-values. The actor is updated to maximize the Q-values. While this objective
for the actor can be written in many ways, we will write it as maximizing a
log ratio because it will allow us to draw a precise equivalence between actor
and critic regularization:

$\max_{\pi}\mathbb{E}_{p_{g}(s_{g})p(s)\pi(a\mid s,s_{g})}\left[\log
Q(s,a,s_{g})\right]$

We will now consider variants of C-learning that incorporate actor and critic
regularization.

#### One-step RL.

We will consider a variant of C-learning that resembles one-step RL
(Brandfonbrener et al., 2021). The critic update will be similar to before,
but the next-actions sampled for the TD updates will be sampled from the
_marginal_ behavioral policy:

$\displaystyle\max_{Q}\;$
$\displaystyle(1-\gamma)\mathbb{E}_{p(s,a,s^{\prime})}\left[\log\frac{Q(s,a,s_{g}=s^{\prime})}{Q(s,a,s_{g}=s^{\prime})+1}\right]+\gamma\mathbb{E}_{p(s,a)p_{g}(s_{g})}\left[y^{\beta,Q_{t}}(s,a,s_{)}\log\frac{Q(s,a,s_{g})}{Q(s,a,s_{g})+1}\right]$
$\displaystyle+\mathbb{E}_{p(s,a)p_{g}(s_{g})}\left[\log\frac{1}{Q(s,a,s_{g}=s^{\prime})+1}\right],$

where $y^{\beta,Q_{t}}(s,a,s_{g})=\mathbb{E}_{p(s^{\prime}\mid
s,a)\beta(a^{\prime}\mid s^{\prime})}[Q_{t}(s^{\prime},a^{\prime},s_{g})]$.
The actor update will be modified to include a reverse KL divergence:

$\max_{\pi}\mathbb{E}_{p(s)p_{g}(s_{g})\pi(a\mid s,s_{g})}\left[\log
Q(s,a,s_{g})+\log\beta(a\mid s)-\pi(a\mid s,s_{g})\right].$ (15)

Note that we are regularizing the policy to be similar to the average
behavioral policy, $\beta(a\mid s)$. Compared to regularization towards a
goal-conditioned behavioral policy $\beta(a\mid s,s_{g})$, this choice gives
the policy additional flexibility: when trying to reach goal $s_{g}$, it is
allowed to take actions that were not taken by $\beta(a\mid s,s_{g})$, as long
as they were taken by the behavioral policy when trying to reach some other
goal $s_{g}^{\prime}$.

#### Critic regularization.

To regularize the critic, we will modify the “negative” term in the C-learning
objective to use actions sampled from the policy:

$\displaystyle\max_{Q}\;$
$\displaystyle(1-\gamma)\mathbb{E}_{p(s,a,s^{\prime})}\left[\log\frac{Q(s,a,s_{g}=s^{\prime})}{Q(s,a,s_{g}=s^{\prime})+1}\right]$
(16)
$\displaystyle+\gamma\mathbb{E}_{p(s,a)p_{g}(s_{g})}\left[y^{\pi,Q_{t}}(s,a,s_{g})\log\frac{Q(s,a,s_{g})}{Q(s,a,s_{g})+1}\right]$
(17) $\displaystyle+\mathbb{E}_{p(s)p_{g}(s_{g}),a\sim\pi(\cdot\mid
s,s_{g})}\left[\log\frac{1}{Q(s,a,s_{g})+1}\right].$ (18)

###  C.1 Analysis for Goal-Conditioned Problems

Like in the single-task setting, these two forms of regularization yield the
same fixed points:

######  Theorem C.1.

Let policy $\pi(a\mid s,s_{g})$ be given, let $Q^{\beta}(s,a,s_{g})$ be the
Q-values for the marginal behavioral policy $\beta(a\mid s)$ and let
$Q_{r}^{\pi}(s,a,s_{g})$ be the critic obtained by the regularized critic
update (Eq. 18). Then performing regularized policy updates (Eq. 15) using the
behavioral critic is equivalent to the un-regularized policy objective using
the regularized critic:

$\mathbb{E}_{\pi(a\mid s,s_{g})}\left[\log
Q^{\beta}(s,a,s_{g})+\log\beta(a\mid s)-\log\pi(a\mid
s,s_{g})\right]=\mathbb{E}_{\pi(a\mid s,s_{g})}\left[\log
Q_{r}^{\pi}(s,a,s_{g})\right]$

for all states $s$ and goals $s_{g}$.

###### Proof.

We start by determining the fixed point of critic-regularized C-learning. Like
in the single-task setting, the C-learning objective resembles a weighted-
classification problem, so we can write down the Bayes’ optimal classifier as

$\frac{Q(s,a,s_{g})}{Q(s,a,s_{g})+1}=\frac{((1-\gamma)p(s^{\prime}=s_{g}\mid
s,a)+\gamma p(s=s_{g})y(s^{\prime},s_{g}))\beta(a\mid
s)}{((1-\gamma)p(s^{\prime}=s_{g}\mid s,a)+\gamma
p(s=s_{g})y(s^{\prime},s_{g}))\beta(a\mid s)+p(s_{g})\pi(a\mid s,s_{g})}.$

Solving for $Q(s,a,s_{g})$ on the left hand side, the optimal value for
$Q(s,a,s_{g})$ is given by

$Q(s,a,s_{g})=((1-\gamma)p(s^{\prime}=s_{g}\mid s,a)+\gamma
p(s=s_{g})y(s^{\prime},s_{g}))\frac{\beta(a\mid s)}{\pi(a\mid s,s_{g})}$

This tells us what each critic-regularized C-learning update does.

To analyze these updates, we define $\tilde{Q}(s,a,s_{g})\triangleq
Q(s,a,s_{g})\frac{\pi(a\mid s,s_{g})}{\beta(a\mid s)}$. Then these updates can
be written using $\tilde{Q}(s,a,s_{g})$ as

$\tilde{Q}(s,a,s_{g})\frac{\beta(a\mid s)}{\pi(a\mid
s,s_{g})}=\left((1-\gamma)p(s^{\prime}=s_{g}\mid
s,a)+\gamma\mathbb{E}_{p(s^{\prime}\mid s,a)\pi(a^{\prime}\mid
s^{\prime},s_{g})}\left[\tilde{Q}(s^{\prime},a^{\prime},s_{g})\frac{\beta(a^{\prime}\mid
s^{\prime})}{\pi(a^{\prime}\mid
s^{\prime},s_{g})}\right]\right)\frac{\beta(a\mid s)}{\pi(a\mid s,s_{g})}.$

These updates can be simplified to

$\tilde{Q}(s,a,s_{g})=(1-\gamma)p(s^{\prime}=s_{g}\mid
s,a)+\gamma\mathbb{E}_{p(s^{\prime}\mid s,a)\beta(a^{\prime}\mid
s^{\prime})}\left[\tilde{Q}(s^{\prime},a^{\prime},s_{g})\right].$

Like before, the ratio $\frac{\beta(a^{\prime}\mid
s^{\prime})}{\pi(a^{\prime}\mid s^{\prime},s_{g})}$ inside the expectation
acts like an importance weight. Thus, the regularized critic updates are
equivalent to perform policy evaluation on $\tilde{Q}(s,a,s_{g})$. Note that
this is estimating the probability that the _average_ behavioral policy
$\beta(a\mid s)$ reaches goal $s_{g}$; this is _not_ the probability that a
goal-directed behavioral policy $\beta(a\mid s,s_{g})$ reaches the goal.

Finally, we translate these convergence results for $\tilde{Q}(s,a,s_{g})$
into convergence results for $Q(s,a,s_{g})$. Written in terms of the original
Q-values, we see that the optimal critic for the regularized critic update is

$Q^{*}(s,a,s_{g})=\tilde{Q}^{*}(s,a,s_{g})\frac{\beta(a\mid s)}{\pi(a\mid
s,s_{g})}=Q^{\beta(\cdot\mid\cdot)}(s,a,s_{g})\frac{\beta(a\mid s)}{\pi(a\mid
s,s_{g})}.$

Thus, critic regularization implicitly regularizes the actor objective so that
it is the same objective as one-step RL:

$\displaystyle\mathbb{E}_{p(s),s_{g}\sim p(s),\pi(a\mid s,s_{g})}\left[\log
Q^{*}(s,a,s_{g})\right]$ $\displaystyle=\mathbb{E}_{p(s),s_{g}\sim
p(s),\pi(a\mid s,s_{g})}\left[\log
Q^{\beta(\cdot\mid\cdot)}(s,a,s_{g})+\log\beta(a\mid s)-\log\pi(a\mid
s,s_{g})\right].$

∎

##  Appendix D Regularization for Example-based Control Problems

While specifying tasks in terms of reward functions is standard for MDPs, it
can be difficult for real-world applications of RL. So, prior work has looked
at specifying tasks by goal states (as in the previous section) or sets of
states representing good outcomes (Pinto & Gupta, 2016; Tung et al., 2018; Fu
et al., 2018). In addition to requiring more flexible and user-friend forms of
task specification, these algorithms targeted at real-world applications often
demand regularization. In the same way that prior goal-conditioned RL
algorithms have employed critic regularization, so too have prior example-
based control algorithms (Singh et al., 2019; Hatch et al., 2022). In this
section, we extend our analysis to regularization of an example-based control
algorithm. Again, we will show that a certain form of critic regularization is
equivalent to regularizing the actor.

We first define the problem of example-based control (Fu et al., 2018). In
these problems, the agent is given a small collection of states $s\sim
p_{e}(s)$, which are examples of successful outcomes. The aim is to learn a
policy $\pi(a\mid s)$ that maximizes the probability of reaching a success
state:

$\max_{\pi}\mathbb{E}_{p(s_{g})}\mathbb{E}_{\pi(\tau\mid
s_{g})}\left[\sum_{t=0}^{\infty}\gamma^{t}p_{e}(s_{t})\right].$

Note that this objective function is exactly equivalent to a reward-
maximization problem, with a reward function $r(s,a)=p_{e}(s_{t})$.

In this section, we show that one-step RL and critic regularization are
equivalent for a certain example-based control algorithm. Unlike our analysis
in the single-task setting, this analysis here uses an existing method, RCE
(Eysenbach et al., 2021). RCE is a TD method that already makes use of the
cross entropy loss for training the critic:

$\displaystyle\max_{Q}\;$
$\displaystyle(1-\gamma)\mathbb{E}_{p_{e}(s)\beta(a\mid
s)}\left[\log\frac{Q(s,a)}{Q(s,a)+1}\right]+\mathbb{E}_{p(s,a)}\left[\gamma
y^{\pi,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}+\log\frac{1}{Q(s,a)+1}\right],$

where $y^{\pi,Q_{t}}(s,a)=\mathbb{E}_{p(s^{\prime}\mid s,a)\pi(a^{\prime}\mid
s^{\prime})}[Q(s^{\prime},a^{\prime})]$ serves the role of the TD target. The
first two terms increase the Q-values while the last term decreases the
Q-values. The actor is updated to maximize the Q-values. While this objective
for the actor can be written in many ways, we will write it as maximizing a
log ratio because it will allow us to draw a precise equivalence between actor
and critic regularization:

$\max_{\pi}\mathbb{E}_{p(s)\pi(a\mid s)}\left[\log Q(s,a)\right]$

We will now consider variants of RCE that incorporate actor and critic
regularization.

#### One-step RL.

We will consider a variant of RCE that resembles one-step RL (Brandfonbrener
et al., 2021). The critic update will be similar to before, but the next-
actions sampled for the TD updates will be sampled from the behavioral policy:

$\displaystyle\max_{Q}\;$
$\displaystyle(1-\gamma)\mathbb{E}_{p_{e}(s)\beta(a\mid
s)}\left[\log\frac{Q(s,a)}{Q(s,a)+1}\right]+\mathbb{E}_{p(s,a)}\left[\gamma
y^{\beta,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}+\log\frac{1}{Q(s,a)+1}\right],$

where $y^{\beta,Q_{t}}(s,a)=\mathbb{E}_{p(s^{\prime}\mid
s,a)\beta(a^{\prime}\mid s^{\prime})}[Q(s^{\prime},a^{\prime})]$. The actor
update will be modified to include a reverse KL divergence:

$\max_{\pi}\mathbb{E}_{p(s),\pi(a\mid s)}\left[\log Q(s,a)+\log\beta(a\mid
s)-\pi(a\mid s)\right].$ (19)

#### Critic regularization.

To regularize the critic, we will modify the “negative” term in the RCE
objective to use actions sampled from the policy:

$\displaystyle(1-\gamma)\mathbb{E}_{p_{e}(s)\beta(a\mid
s)}\left[\log\frac{Q(s,a)}{Q(s,a)+1}\right]+\mathbb{E}_{p(s,a),a^{-}\sim\pi(\cdot\mid
s)}\left[\gamma
y^{\pi,Q_{t}}(s,a)\log\frac{Q(s,a)}{Q(s,a)+1}+\log\frac{1}{Q(s,a^{-})+1}\right],$
(20)

###  D.1 Analysis for Example-based Control Problems

Like in the single-task setting, these two forms of regularization yield the
same fixed points:

######  Theorem D.1.

Let policy $\pi(a\mid s)$ be given, let $Q^{\beta}(s,a)$ be the Q-values for
the behavioral policy $\beta(a\mid s)$ and let $Q_{r}^{\pi}(s,a)$ be the
critic obtained by the regularized critic update (Eq. 20). Then performing
regularized policy updates (Eq. 19) using the behavioral critic is equivalent
to the un-regularized policy objective using the regularized critic:

$\mathbb{E}_{\pi(a\mid s)}\left[\log Q^{\beta}(s,a)+\log\beta(a\mid
s)-\log\pi(a\mid s)\right]=\mathbb{E}_{\pi(a\mid s)}\left[\log
Q_{r}^{\pi}(s,a)\right]$

for all states $s$.

###### Proof.

We start by determining the fixed point of critic-regularized RCE. Like in the
single-task setting, The RCE objective resembles a weighted-classification
problem, so we can write down the Bayes’ optimal classifier as

$\frac{Q(s,a)}{Q(s,a)+1}=\frac{((1-\gamma)p_{e}(s)+\gamma
y^{\pi,Q_{t}}(s,a))\beta(a\mid s)}{((1-\gamma)p_{e}(s)+\gamma
y^{\pi,Q_{t}}(s,a))\beta(a\mid s)+\pi(a\mid s)}.$

Solving for $Q(s,a)$ on the left hand side, the optimal value for $Q(s,a)$ is
given by

$Q(s,a)=((1-\gamma)p_{e}(s)+\gamma y^{\pi,Q_{t}}(s,a))\frac{\beta(a\mid
s)}{\pi(a\mid s)}$

This tells us what each critic-regularized RCE update does.

To analyze these updates, we define $\tilde{Q}(s,a)\triangleq
Q(s,a)\frac{\pi(a\mid s)}{\beta(a\mid s)}$. Then these updates can be written
using $\tilde{Q}(s,a)$ as

$\tilde{Q}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid
s)}=\left((1-\gamma)p_{e}(s)+\gamma\mathbb{E}_{p(s^{\prime}\mid
s,a)\pi(a^{\prime}\mid
s^{\prime})}\left[\tilde{Q}(s^{\prime},a^{\prime})\frac{\beta(a^{\prime}\mid
s^{\prime})}{\pi(a^{\prime}\mid s^{\prime})}\right]\right)\frac{\beta(a\mid
s)}{\pi(a\mid s)}.$

These updates can be simplified to

$\tilde{Q}(s,a)=(1-\gamma)p_{e}(s)+\gamma\mathbb{E}_{p(s^{\prime}\mid
s,a)\beta(a^{\prime}\mid
s^{\prime})}\left[\tilde{Q}(s^{\prime},a^{\prime})\right].$

Like before, the ratio $\frac{\beta(a^{\prime}\mid
s^{\prime})}{\pi(a^{\prime}\mid s^{\prime})}$ inside the expectation acts like
an importance weight. Thus, the regularized critic updates are equivalent to
perform policy evaluation on $\tilde{Q}(s,a)$.

Finally, we translate these convergence results for $\tilde{Q}(s,a)$ into
convergence results for $Q(s,a)$. Written in terms of the original Q-values,
we see that the optimal critic for the regularized critic update is

$Q^{*}(s,a)=\tilde{Q}^{*}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid
s)}=Q^{\beta}(s,a)\frac{\beta(a\mid s)}{\pi(a\mid s)}.$

Thus, critic regularization implicitly regularizes the actor objective so that
it is the same objective as one-step RL:

$\displaystyle\mathbb{E}_{p(s),\pi(a\mid s)}\left[\log
Q^{*}(s,a)\right]=\mathbb{E}_{p(s),\pi(a\mid s)}\left[\log
Q^{\beta}(s,a)+\log\beta(a\mid s)-\log\pi(a\mid s)\right].$

∎

##  Appendix E Additional Experiments

###  E.1 Is classifier actor critic a good model of practically-used RL
methods?

To study whether classifier actor critic is an accurate model for practically-
used RL methods, we extended Fig. 3 by adding three additional methods:
classifier actor critic, classifier actor critic with actor regularization
(Equations 4 and 3). Comparing Q-learning with classifier actor critic, we see
that both yield reward-maximizing policies (in line with Lemma 4.1), though
these policies are different (i.e., they perform symmetry breaking in
different ways). Comparing one-step RL with actor-regularized classifier actor
critic, we observe that both methods take the same three actions at the states
within the blue box, states where we expect regularization to have a large
effect. Outside this blue box, these two methods occasionally take different
actions. Similarly, comparing CQL to critic-regularized classifier actor-
critic, we observe that the methods take the same actions within the blue box,
but occasionally take different actions outside the blue box. In line with
Theorem 4.3, classifier actor-critic with critic regularization produces the
exact same policy as classifier actor-critic with actor regularization. Taken
together, these results provide empirical backing for our theoretical results,
while also showing that classifier actor critic is only an approximate model
of practical algorithms, not a perfect model.

(a) Q-learning

(b) One-step RL

(c) CQL($\lambda=10$)

(d) …. classifier actor critic ………. (no regularization)

(e) … classifier actor critic …….. (actor regularization)

(f) .. classifier actor critic …….. (critic regularization)

Figure 9: Extension of Fig. 3 with additional figures for classifier actor
critic and regularized variants. As expected, classifier actor critic produces
a reward-maximizing policy (Lemma 4.1), as does Q-learning. In line with
Theorem 4.3, critic-regularized and actor-regularized classifier actor critic
product the same policies. Comparing (b) one-step RL (e) actor-regularized
classifier actor critic, we see that the methods produce the same three
actions within the blue box (states where we expect regularization to be
important), but can produce different actions in states outside the blue box.
The comparisons between (c) CQL and (d) critic-regularized classifier actor
critic are similar, supporting the claim that classifier actor critic is only
an approximate (no a perfect) model of practically-used offline RL methods.

##  Appendix F Experimental Details

###  F.1 Tabular experiments

#### Implementing critic regularization for classifier actor critic.

The objective for critic regularization in contrastive actor critic (Eq. 6) is
nontrivial to optimize because of the cyclic dependency between the policy and
the critic: simply alternating between optimizing the actor and the critic
does not converge. In our experiments, we update the critic using an
exponential moving average of the policy, as proposed in Wen et al. (2021). We
found that this decision was sufficient for ensuring convergence. When
applying CQL in the tabular setting (Figures 3 and 4), we did not do this
because soft value iteration represents the policy implicitly in terms of the
value function.

#### Fig. 2 _(left)_

The initial state and goal state are located in opposite corners. The reward
function is $+1$ for reaching the goal and 0 otherwise. We use a dataset of 20
trajectories, 50 steps each, collected by a random policy. We use
$\gamma=0.95$ and train for 20k full-batch updates, using a learning rate of
1e-2. The Q table is randomly initialized using a standard normal
distribution.

#### Fig. 2 _(center)_

The initial state and goal state are located in adjacent corners. The goal
state has a reward of +3.5, the states between the initial state and goal
state have a reward +1, and all other states (including the initial state)
have a reward of +2. We use a dataset of 20 trajectories, 50 steps each,
collected by a random policy. We use $\gamma=0.95$ and train for 20k full-
batch updates, using a learning rate of 1e-2. The Q table is randomly
initialized using a standard normal distribution.

#### Fig. 2 _(right)_

The initial state and goal state are located in adjacent corners. The reward
is +0.01 at the goal state and 0 otherwise. We use a dataset of 1 trajectories
with 10 steps, which traces the following path:

$[(0,0),(1,0),(1,1),(1,2),(1,3),(1,4),(0,4),(0,4),(0,4),(0,4)].$

We use $\gamma=0.95$ and train for 10k full-batch updates, using a learning
rate of 1e-2. The Q table is randomly initialized using a standard normal
distribution.

#### Fig. 3

There is a bad state (reward of $-10$) next to the optimal state (reward of
$+1$), so the behavioral policy navigates away from the optimal state. We
generate 10 trajectories of length 100 from a uniform random policy. We use
$\gamma=0.95$ and train each method for 10k full-batch updates. The Q table is
randomly initialized using a standard normal distribution. One-step RL
performs SARSA updates while CQL performs soft value iteration (as suggested
in the CQL paper).

#### Fig. 4

We generate 100 random variants of Fig. 3 by randomly sampling the high-reward
state and low-reward state (without replacement). The datasets are generated
in the same way.

#### Fig. 5

We use the same environment and dataset as in Fig. 3, but train the CQL agent
with varying values of $\lambda$, each with 5 random seeds. We train the one-
step RL agent for 5 random seeds. For each point on the X axis of Fig. 5, we
compare compute $5\times 5$ pairwise comparisons and report the mean and
standard deviation.

###  F.2 Continuous control experiments

For the experiments in Figures 6 and 7, we used the implementation of one-step
RL (reverse KL) and CQL provided by Hoffman et al. (2020). We choose this
implementation because it is well tuned and uses similar hyperparameters for
the two methods. As mentioned in the main text, the only change we made to the
implementation was adding the twin-Q trick to one-step RL, such that it
matched the critic architecture used by CQL. We did not change any of the
other hyperparameters, including hyperparameters controlling the
regularization strength.