In deep Reinforcement Learning (RL), value functions are typically approximated using deep neural networks and trained via mean squared error regression objectives to fit the true value functions. Recent research has proposed an alternative approach, utilizing the cross-entropy classification objective, which has demonstrated improved performance and scalability of RL algorithms. However, existing study have not extensively benchmarked the effects of this replacement across various domains, as the primary objective was to demonstrate the efficacy of the concept across a broad spectrum of tasks, without delving into in-depth analysis. Our work seeks to empirically investigate the impact of such a replacement in an offline RL setup and analyze the effects of different aspects on performance. Through large-scale experiments conducted across a diverse range of tasks using different algorithms, we aim to gain deeper insights into the implications of this approach. Our results reveal that incorporating this change can lead to superior performance over state-of-the-art solutions for some algorithms in certain tasks, while maintaining comparable performance levels in other tasks, however for other algorithms this modification might lead to the dramatic performance drop. This findings are crucial for further application of classification approach in research and practical tasks.[Our code is available at <https://github.com/DT6A/ClORL>]

§ INTRODUCTION

In the realm of deep Reinforcement Learning (RL), the conventional approach to approximating value functions has long relied on employing the Bellman optimality operator alongside mean squared error (MSE) regression objectives, owing to the continuous nature of the task at hand. However, insights from other domains of machine learning have illuminated the potential benefits of employing classification objectives even in scenarios where regression seems as a natural choice <cit.>. This shift has been attributed to various hypotheses, including the stability of gradients <cit.>, improved feature representation <cit.>, and implicit biases <cit.>.

While the use of regression loss has yielded remarkable results in value-based RL <cit.>, it also presents certain challenges and limitations <cit.>. Notably, recent research <cit.> has demonstrated that replacing regression with classification for training value functions offers several advantages, including enhanced scalability, feature representation, performance, and robustness to noisy targets and non-stationarity. While this study broadly explores the effects of this replacement, it does not delve deeply into the ease of implementation or the impact of newly introduced hyperparameters.

Offline RL <cit.> represents a rapidly developing RL subfield, wherein the objective is to train an agent using a pre-collected dataset without direct interaction with the environment. In recent years, numerous algorithms have been developed to address this setup <cit.>. The majority of these algorithms fall under the category of off-policy value-based approaches, owing to their alignment with the problem setup's requirements.

Given the prevalence of off-policy value-based approaches in offline RL, it becomes imperative to delve deeper into the potential impact of employing classification for value function training within this domain. While <cit.> offer some experimental insights into offline RL tasks, we contend that there remain significant gaps to be addressed. Therefore, our study seeks to contribute to this area by conducting a thorough investigation into the utilization of classification objectives for value function training in offline RL scenarios.

This study aims to address the following questions through large-scale experiments conducted on a large range of tasks from the standard D4RL benchmark <cit.>:

* Is classification a "plug-and-play replacement" for offline RL algorithms, and how does it impact performance?

* Does the use of classification objectives facilitate a more robust hyperparameter search?

* What is the impact of the hyperparameters introduced with classification?

* Does classification enable more efficient scaling of dense neural networks compared to regression?

§ PRELIMINARIES

§.§ Offline Reinforcement Learning

The RL problem is conventionally framed as a Markov Decision Process characterized by a tuple $S, A, P, R, \gamma$, where: $S \subset \mathbb{R}^n$ denotes the state space, $A \subset \mathbb{R}^m$ represents the action space, $P: S \times A \to S$ is the transition function, $R: S \times A \to \mathbb{R}$ is the reward function, and $\gamma \in (0, 1)$ is the discount factor. The objective RL of is to determine a policy (agent) $\pi: S \to A$ that maximizes the expected sum of discounted rewards: $\sum_{t=0}^{\infty}\gamma^tR(s_t, a_t)$. This entails the policy learning process, where the agent interacts with its environment by observing environmental states, taking actions in response, and receiving corresponding rewards.

Offline RL presents a departure from the traditional RL setup in that the agent relies solely on a pre-collected dataset $D$ collected   by external agents. This paradigm introduces novel challenges, such as the estimation of value functions for out-of-distribution state-action pairs, thus giving rise to an entire subfield dedicated to addressing these challenges.

§.§ Q-function objective and classification

The Q-function stands as a pivotal concept in value-based RL, representing the expected return of a policy starting from state $s_t$ and taking action $a_t$: $Q(s_t, a_t) = \mathbb{E}_\pi [\sum_{k=0}^{\infty} \gamma^kR(s_{t + k}, a_{t + k}) | s_t, a_t]$. Typically, value functions are trained using the Bellman optimality operator:

\[
(\hat{\mathcal{T}}Q)(s, a, \hat{\theta}) = R(s, a) + \gamma \max_{a'} Q(s', a', \hat{\theta})
\]

Here, $s'$ denotes the state observed after executing action $a$ at state $s$, and $\hat{\theta}$ (also known as target network)  represents a copy with delayed updates of the parameter $\theta$ which parametrize the value function. During each training step, parameters $\theta$ are adjusted using temporal difference (TD) error with mean squared error (MSE):

\[
\text{TD}_\text{{MSE}}(\theta) = \mathbb{E}_D((\hat{\mathcal{T}}Q)(s_t, a_t, \hat{\theta}) - Q(s_t, a_t, \theta))^2
\]

We adopt the framework proposed by <cit.> for replacing MSE with cross-entropy loss. The approach involves parameterizing the Q-function as a distribution over returns, segmented into $m$ bins with widths $\zeta$, where the first bin corresponds to the predefined $v_{min}$ value, and the last to $v_{max}$. The scalar value of the Q-function is then computed as:

\[
Q(s, a, \theta) = \mathbb{E}[Z(s, a, \theta)],\ \ Z(s, a, \theta) = \sum_{i=1}^mp_i(s, a, \theta) \delta_i,
\]

where $p_i$ represents the probability of the $i$-th bin, and $\delta_i$ denotes the corresponding bin value. This formulation allows us to express the TD error using cross-entropy and utilize it for updating the value function parameters:

\[
\text{TD}_\text{{CE}}(\theta) = \mathbb{E}_D\sum_{i=1}^mp^{\mathcal{T}}_i(s, a, \hat{\theta}) \log p_i(s, a, \theta)
\]

<cit.> demonstrated that the HL-Gauss method <cit.> is particularly effective for mapping continuous values into bins within this framework. HL-Gauss employs a normal distribution analog for mapping values into neighboring bins, with a hyperparameter $\sigma$ determining the distribution's breadth. The authors recommend tuning the ratio $\sigma / \zeta$ as a more interpretable hyperparameter, with a default value of $0.75$ chosen based on statistical  considerations and it has shown good empirical result.

§ METHODOLOGY

Our methodology is straightforward: we select several offline RL algorithms and adapt them for cross-entropy loss, as outlined in Section <ref>. By default, we determine the values of $v_{min}$ and $v_{max}$ by computing all possible discounted returns within a given dataset and identifying the minimum and maximum values, which aligns naturally with the offline RL setup. Herein, we provide brief descriptions of the algorithms employed in our study.

Revisited BRAC (ReBRAC). ReBRAC <cit.> stands as a minimalist, state-of-the-art ensemble-free algorithm for both offline and offline-to-online RL. It employs MSE to penalize deviations from actions present in the dataset. Built upon TD3+BC <cit.>, ReBRAC incorporates several modifications that significantly enhance its performance. The Q-loss function takes the form:

\[
\mathbb{E}_{(s, a, s', \hat{a'}) \sim D, a' \sim \pi(s')}(Q(s, a, \theta) - [R(s, a) + \gamma (Q(s', a', \hat{\theta}) - \beta(a' - \hat{a'})^2)])^2
\]

where $\beta$ denotes a penalty weight. Although only the target network part differs, the bin mapping is conducted in the same manner, with the penalty subtracted beforehand. We select ReBRAC as an exemplary algorithm with policy regularization due to its high performance, simplicity, and its encounter with the Q-function divergence problem while solving certain D4RL AntMaze tasks.

Implicit Q-Learning (IQL). IQL <cit.> represents another competitive offline and offline-to-online RL algorithm. Its key advantage lies in its exclusion of out-of-distribution examples during training, a departure from most other offline RL approaches. This is achieved through the training of the V-function: $V(s_t) = \mathbb{E}_\pi [\sum_{k=0}^{\infty} \gamma^kR(s_{t + k}, a_{t + k}) | s_t]$. The Q-function loss takes the form:

\[
\mathbb{E}_{(s, a, s') \sim D}(Q(s, a, \theta) - [R(s, a) + \gamma V_{\phi}(s')])^2
\]

Once again, the target component allows for classification without additional manipulations. Regularization is achieved through a specially designed V-function loss, without explicit penalties for the policy or value functions. We opt for IQL as it the best example of algorithms within this family.

Large-Batch SAC (LB-SAC). LB-SAC <cit.> is an instance of the ensemble-based SAC-N algorithm <cit.>, utilizing large-batch optimization to reduce ensemble size. SAC-N comprises $N$ Q-functions, each with the following loss:

\[
\mathbb{E}_{(s, a, s') \sim D, a' \sim \pi(s')}(Q(s, a, \theta_i) - [R(s, a) + \gamma \min_{j=1..N} Q(s', a', \hat{\theta}_j)])^2
\]

No additional adjustments are necessary to replace MSE with cross-entropy. The rationale behind this objective lies in the assumption that if Q-values follow a normal distribution $\mathcal{N}(\mu, \sigma)$ then:

\[
\min_{j=1..N} Q(s, a, \hat{\theta_j}) \approx \mu(s, a) - \Phi^{-1}\left(\frac{N - \frac{\pi}{8}
}{N - \frac{\pi}{4} + 1}\right)\sigma(s, a)
\]

Ensemble-based approaches are known for their efficacy across many offline RL tasks. According to the above statement, SAC-N can be also considered as an example of offline RL algorithm with Q-function regularization. We opt for LB-SAC instead of the original SAC-N due to computational constraints.

§ EXPERIMENTAL RESULTS

§.§ Experimental setup

We conducted our experiments using three sets of tasks from the D4RL benchmark <cit.>: Gym-MuJoCo, AntMaze, and Adroit. We utilized all datasets within each set.

For ReBRAC in <ref>, we employed the best hyperparameters as outlined in <cit.>. When exploring parameter search in, we utilized the same hyperparameter grids for ReBRAC and IQL and used a custom grid for LB-SAC due to computational constraints. For a comprehensive overview of the experimental details and hyperparameters, refer to <ref> and <ref>.

The evaluation protocol is taken from <cit.>, where hyperparameters search is done using four random seeds and separate set of seeds is used for the evaluation with the exception that we utilized four random seeds for the final evaluation instead of ten, due to computational constraints. Note, that because of the chosen evaluation protocol tuned hyperparameters might perform worse than non-tuned which characterizes the sensitivity to the random initialization.

§.§ Is classification plug-and-play?

Our initial goal was to substitute MSE with cross-entropy without modifying any other aspects of the algorithms. We fixed the number of bins $m$ to 101, representing a reasonable number of classes. We set $\sigma/\zeta = 0.75$ by default as was proposed by <cit.>. Results for this modification are presented in the CE columns of <ref> for Gym-MuJoCo, <ref> for AntMaze, and <ref> for Adroit.

The original ReBRAC algorithm exhibited minimal performance variation with the introduction of cross-entropy in Gym-MuJoCo tasks, except for the random dataset where a notable performance drop was observed. However, on average, the score remained relatively stable. In the case of AntMaze, where ReBRAC faced Q-function divergence issues, the introduction of classification mitigated this problem, resulting in significantly reduced standard deviations and a slight improvement in average performance. However, in Adroit, where ReBRAC struggled with overfitting due to small dataset sizes, classification did not alleviate this issue and led to decreased performance across most scenarios.

In contrast, both IQL and LB-SAC experienced a significant performance drop in Gym-MuJoCo tasks. IQL's performance in AntMaze dramatically decreased, rendering the algorithm unable to solve medium and large tasks with this modification. Meanwhile, LB-SAC's performance did not change significantly for AntMaze but improved for the umaze task. Notably, in the Adroit task, IQL's performance was significantly boosted in the pen environment, with relatively minor changes observed in other scenarios. However, LB-SAC's performance dropped across most tasks.

Our primary hypothesis for explaining the successful application of classification with ReBRAC and its failure with IQL and LB-SAC is that ReBRAC heavily relies on policy regularization. Consequently, plugging classification into Q networks does not strongly affect its offline RL component compared to the impact observed in IQL or LB-SAC.

Average normalized score over the final evaluation and four unseen training seeds on Gym-MuJoCo tasks. "hc" stands for "halfcheetah", "hp" stands for "hopper", "w l" stands for "walker2d", "r" stands for "random", "m" stands for "medium", "e" stands for "expert", "me" stands for "medium-expert", "mr" stands for "medium-replay", "fr" stands for "full-replay". MSE denotes original algorithm implementation, CE denotes the replacement of MSE with cross-entropy, CE+AT denotes cross-entropy with tuned algorithm parameters, CE+CT denotes cross-entropy with tuned classification parametersThese notations are reused further.. ReBRAC original scores are taken from <cit.>.

max width=

1c     4cReBRAC     4cIQL     4cLB-SAC

Task     MSE     CE     CE+AT     CE+CT     MSE     CE     CE+AT     CE+CT     MSE     CE     CE+AT     CE+CT

hc-r      29.5 $\pm$  1.5     13.6 $\pm$  0.7     13.6 $\pm$  0.7     12.4 $\pm$  0.5     18.8 $\pm$  1.5     1.9 $\pm$  0.0     6.3 $\pm$  1.3     4.4 $\pm$  2.9     28.2 $\pm$  1.4     10.0 $\pm$  0.3     11.6 $\pm$  0.2      11.1 $\pm$  0.7

hc-m     65.6 $\pm$  1.0      59.4 $\pm$  0.3     63.1 $\pm$  0.7     62.1 $\pm$  0.5     49.8 $\pm$  0.2     42.3 $\pm$  1.7     42.9 $\pm$  0.1     44.1  $\pm$  0.2     64.5 $\pm$  1.3     56.7 $\pm$  2.1     63.6 $\pm$  9.7     55.5 $\pm$  2.7

hc-e     105.9 $\pm$  1.7      106.2 $\pm$  2.9     106.5 $\pm$  3.1     105.5 $\pm$  0.8     96.4 $\pm$  0.6     92.7 $\pm$  0.1     91.1 $\pm$  2.2     93.8 $\pm$  0.2     103.0 $\pm$  1.5     103.9 $\pm$  1.0     105.0 $\pm$  1.3     104.5 $\pm$  2.2

hc-me     101.1 $\pm$  5.2     101.8 $\pm$  3.3     101.0 $\pm$  4.9     104.5 $\pm$  4.8     92.1 $\pm$  3.2     88.1 $\pm$  3.0     86.3 $\pm$  3.0     92.5 $\pm$  0.3     104.5 $\pm$  2.4     105.4 $\pm$  2.0     105.2 $\pm$  0.0     107.2 $\pm$  1.3

hc-mr     51.0 $\pm$  0.8     50.3 $\pm$  0.2     50.7 $\pm$  3.1     49.6 $\pm$  4.0     45.6 $\pm$  0.3     38.5 $\pm$  1.4     38.9 $\pm$  1.9     41.4 $\pm$  2.9     52.8 $\pm$  0.7     55.4 $\pm$  0.9     54.4 $\pm$  1.6     58.1 $\pm$  1.3

hc-fr     82.1 $\pm$  1.1     83.9 $\pm$  1.2     82.3 $\pm$  1.3     82.9 $\pm$  0.4     74.5 $\pm$  1.8     63.3 $\pm$  0.6     64.8 $\pm$  0.5     74.4 $\pm$  0.5     79.0 $\pm$  2.0     80.7 $\pm$  0.3     81.3 $\pm$  1.0     82.3 $\pm$  1.1

hp-r     8.1 $\pm$  2.4     7.5 $\pm$  0.3     7.4 $\pm$  1.6     9.6 $\pm$  1.5     5.7 $\pm$  2.3     16.1  $\pm$  3.4     15.3 $\pm$  5.0     11.8 $\pm$  7.8     14.5 $\pm$  11.5     8.2 $\pm$  2.2     9.6 $\pm$  0.4     10.9 $\pm$  2.3

hp-m     102.0 $\pm$  1.0     102.4 $\pm$  0.2     101.5 $\pm$  1.9     101.5 $\pm$  1.5     59.8 $\pm$  5.0     53.7 $\pm$  2.0     51.9 $\pm$  1.5     53.6 $\pm$  2.1     90.0 $\pm$  27.5     7.9 $\pm$  0.8     10.5 $\pm$  6.2     7.8 $\pm$  1.0

hp-e     100.1 $\pm$  8.3     107.0 $\pm$  5.7     107.8 $\pm$  6.1     111.1 $\pm$  0.4     110.8 $\pm$  1.8     110.3 $\pm$  0.7     110.7 $\pm$  0.3     110.2 $\pm$  0.7     1.3 $\pm$  0.0     21.6 $\pm$  39.7     65.9 $\pm$  51.9     12.3 $\pm$  13.1

hp-me     107.0 $\pm$  6.4     110.9 $\pm$  0.8     111.0 $\pm$  1.0     111.5 $\pm$  0.2     95.3 $\pm$  31.8     65.3 $\pm$  8.7     73.5 $\pm$  32.6     94.7 $\pm$  7.9     111.3 $\pm$  0.3     14.9 $\pm$  7.0     12.5 $\pm$  7.2     54.7 $\pm$  37.6

hp-mr     98.1 $\pm$  5.3     99.5 $\pm$  1.3     99.8 $\pm$  1.6     97.6 $\pm$  3.3     91.4 $\pm$  5.5     21.8 $\pm$  3.8     27.4 $\pm$  17.8     28.6 $\pm$  8.2     63.0 $\pm$  48.1     66.9 $\pm$  42.8     100.6 $\pm$  5.5     87.4 $\pm$  29.6

hp-fr     107.1 $\pm$  0.4     108.1 $\pm$  0.6     108.6 $\pm$  0.2     108.4 $\pm$  0.6     106.9 $\pm$  1.0     42.7 $\pm$  2.7     49.9 $\pm$  4.3     104.0 $\pm$  0.5     107.0 $\pm$  0.7     65.6 $\pm$  50.2     109.9 $\pm$  0.4     109.1 $\pm$  1.8

wl-r     18.4 $\pm$  4.5     4.8 $\pm$  1.0     8.4 $\pm$  6.6     10.2 $\pm$  7.5     4.5 $\pm$  6.2     4.2 $\pm$  2.1     3.7 $\pm$  1.5     7.3 $\pm$  9.5     21.7 $\pm$  0.0     20.1 $\pm$  2.9     21.8 $\pm$  0.0     21.5 $\pm$  0.2

wl-m     82.5 $\pm$  3.6     84.9 $\pm$  3.7     84.9 $\pm$  3.7     86.2 $\pm$  0.7     83.7 $\pm$  0.2     78.8 $\pm$  1.5     82.1 $\pm$  1.3     80.2 $\pm$  1.4     89.3 $\pm$  5.3     89.6 $\pm$  10.7     93.7 $\pm$  7.3     98.0 $\pm$  1.3

wl-e     112.3 $\pm$  0.2     112.6 $\pm$  0.2     112.7 $\pm$  0.1     112.6 $\pm$  0.1     113.9 $\pm$  0.0     108.5 $\pm$  0.3     108.6 $\pm$  0.0     109.3 $\pm$  0.1     114.2 $\pm$  0.4     59.8 $\pm$  45.0     107.6 $\pm$  1.2     112.5 $\pm$  0.2

wl-me     111.6 $\pm$  0.3     112.0 $\pm$  0.1     111.9 $\pm$  0.1     112.0 $\pm$  0.2     112.8 $\pm$  0.4     96.0 $\pm$  7.8     108.2 $\pm$  1.7     109.4 $\pm$  0.2     110.6 $\pm$  0.4     73.1 $\pm$  18.1     94.0 $\pm$  33.2     109.6 $\pm$  8.8

wl-mr     77.3 $\pm$  7.9     82.9 $\pm$  4.2     85.3 $\pm$  3.1     86.3 $\pm$  5.1     81.9 $\pm$  6.3     55.8 $\pm$  10.1     65.7 $\pm$  6.3     85.2 $\pm$  1.0     92.6 $\pm$  2.7     90.9 $\pm$  5.3     95.3 $\pm$  6.0     99.8 $\pm$  1.9

wl-fr     102.2 $\pm$  1.7     101.1 $\pm$  2.6     111.0 $\pm$  1.4     103.4 $\pm$  1.3     98.2 $\pm$  0.9     85.7 $\pm$  3.4     84.5 $\pm$  2.5     93.2 $\pm$  0.7     102.1 $\pm$  1.0     110.4 $\pm$  1.9     109.4 $\pm$  1.3     110.5 $\pm$  2.4

Avg     81.2     80.4     81.5     81.5     74.5     59.2     61.7     68.7     74.9     57.8     69.5     69.6

Average normalized score over the final evaluation and ten unseen training seeds on AntMaze tasks. "um" stands for "umaze", "med" stands for "medium", "lrg" stands for "large", "p" stands for "play", "d" stands for "diverse". ReBRAC original scores are taken from <cit.>.

max width=

1c     4cReBRAC     4cIQL     4cLB-SAC

Task     MSE     CE     CE+AT     CE+CT     MSE     CE     CE+AT     CE+CT     MSE     CE     CE+AT     CE+CT

um      97.8 $\pm$  1.0     95.0 $\pm$  6.0     98.5 $\pm$  1.0     98.5 $\pm$  1.7     72.5 $\pm$  20.6     48.0 $\pm$  2.4     50.2 $\pm$  5.4     52.2 $\pm$  4.1     18.25 $\pm$  35.8     41.0 $\pm$  33.2     36.0 $\pm$  33.4     57.5 $\pm$  12.7

um-d     88.3 $\pm$  13.0     91.7 $\pm$  4.2     90.0 $\pm$  4.7     94.5 $\pm$  1.0     72.5 $\pm$  9.5     52.0 $\pm$  6.2     49.7 $\pm$  3.4     49.5  $\pm$  5.9     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.2 $\pm$  0.5     0.0 $\pm$  0.0

med-p      84.0 $\pm$  4.2     81.2 $\pm$  2.0     86.7 $\pm$  8.1     88.5 $\pm$  2.3     80.0 $\pm$  8.1     0.5 $\pm$  1.0     0.2 $\pm$  0.5     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

med-d     76.3  $\pm$  13.5     73.5 $\pm$  7.0     82.2 $\pm$  5.1     88.0 $\pm$  5.2     72.5 $\pm$  12.5     0.0 $\pm$  0.0     0.5 $\pm$  0.5     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

lrg-p     60.4 $\pm$  26.1     68.2 $\pm$  6.0     86.5 $\pm$  3.6     85.5 $\pm$  9.1     45.0 $\pm$  19.1     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

lrg-d     54.4 $\pm$  25.1     55.0 $\pm$  11.7     82.2 $\pm$  5.1     76.7 $\pm$  10.7     25.0 $\pm$  19.1     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

Avg     76.8     77.4     87.6     88.6     61.2     16.7     16.7     16.9     3.0     6.8     6.0     9.5

Average normalized score over the final evaluation and ten unseen training seeds on Adroit tasks. "ham" stands for "hammer", "rel" stands for "relocate", "h" stands for "human", "c" stands for "cloned", "e" stands for "expert". ReBRAC original scores are taken from <cit.>.

max width=

1c     4cReBRAC     4cIQL     4cLB-SAC

Task     MSE     CE     CE+AT     CE+CT     MSE     CE     CE+AT     CE+CT     MSE     CE     CE+AT     CE+CT

pen-h     103.5 $\pm$  14.1     91.9 $\pm$  9.3     97.9 $\pm$  13.7     93.8 $\pm$  16.3     12.1 $\pm$  13.9     110.0 $\pm$  9.2     104.1 $\pm$  17.5     93.8 $\pm$  8.3     4.5 $\pm$  2.6     7.1 $\pm$  5.6     20.1 $\pm$  19.6     5.8 $\pm$  8.8

pen-c     91.8 $\pm$  21.7     81.3 $\pm$  14.4     86.3 $\pm$  8.8     107.6 $\pm$  12.2     23.3 $\pm$  25.0     20.5 $\pm$  18.8     100.6 $\pm$  11.2     29.0 $\pm$  42.0     26.1 $\pm$  5.3     20.0 $\pm$  4.5     20.8 $\pm$  5.0     22.0 $\pm$  8.0

pen-e     154.1 $\pm$  5.4     152.4 $\pm$  7.2     154.9 $\pm$  3.0     156.1 $\pm$  5.4     49.6 $\pm$  41.5     136.5 $\pm$  6.5     142.3 $\pm$  7.7     142.2 $\pm$  5.0     130.5 $\pm$  16.8     38.0 $\pm$  13.2      62.0 $\pm$  9.3     43.6 $\pm$  29.8

door-h      0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     6.6 $\pm$  5.5     3.5 $\pm$  0.9     3.5 $\pm$  2.5     3.2 $\pm$  2.5     -0.2 $\pm$  0.1     -0.2 $\pm$  0.1     -0.2 $\pm$  0.1     -0.2 $\pm$  0.1

door-c     1.1 $\pm$  2.6     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.1 $\pm$  0.0     0.1 $\pm$  0.0     2.9 $\pm$  1.6     1.4 $\pm$  1.0     0.0 $\pm$  0.0     0.2 $\pm$  0.5     0.5 $\pm$  1.0     0.0 $\pm$  0.0

door-e     104.6 $\pm$  2.4     105.7 $\pm$  1.0     106.2 $\pm$  0.1     104.6 $\pm$  3.3     103.3 $\pm$  3.0     104.0 $\pm$  2.6     100.8 $\pm$  2.0     105.1 $\pm$  2.5     95.0 $\pm$  8.6     70.6 $\pm$  33.8     76.5 $\pm$  5.5     68.7 $\pm$  19.4

ham-h     0.2 $\pm$  0.2     0.1 $\pm$  0.0     0.2 $\pm$  0.2     0.1 $\pm$  0.0     2.7 $\pm$  2.7     1.2 $\pm$  0.2     3.0 $\pm$  3.1     3.4 $\pm$  2.4     0.1 $\pm$  0.0     0.0 $\pm$  0.0     0.1 $\pm$  0.0     0.1 $\pm$  0.0

ham-c     6.7 $\pm$  3.7     11.9 $\pm$  10.0     6.7 $\pm$  7.5     6.7 $\pm$  9.7     0.2 $\pm$  0.0     0.7 $\pm$  0.5     1.3 $\pm$  0.5     0.9 $\pm$  0.7     20.2 $\pm$  16.8     13.6 $\pm$  15.4     0.0 $\pm$  0.0     19.9 $\pm$  21.7

ham-e     133.8 $\pm$  0.7     112.7 $\pm$  35.2     131.2 $\pm$  6.3     133.7 $\pm$  1.6     129.3 $\pm$  0.5     129.5 $\pm$  0.1     129.5 $\pm$  0.0     130.6 $\pm$  0.0     76.6 $\pm$  59.5     91.1 $\pm$  10.5     96.0 $\pm$  9.9     89.2 $\pm$  11.2

rel-h     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.1 $\pm$  0.0     0.1 $\pm$  0.1     0.0 $\pm$  0.0     0.0 $\pm$  0.0     -0.1 $\pm$  0.0     0.0 $\pm$  0.0     -0.1 $\pm$  0.0

rel-c    0.9 $\pm$  1.6     0.6 $\pm$  1.3     0.2 $\pm$  0.1     0.4 $\pm$  0.2     0.0 $\pm$  0.0     0.1 $\pm$  0.0     0.0 $\pm$  0.0     0.1 $\pm$  0.0     0.0 $\pm$  0.0     -0.1 $\pm$  0.0     -0.1 $\pm$  0.0     -0.1 $\pm$  0.0

rel-e     106.6 $\pm$  3.2     108.6 $\pm$  0.6     107.6 $\pm$  2.8     107.6 $\pm$  2.6     107.5 $\pm$  2.7     104.6 $\pm$  1.9     108.8 $\pm$  0.7     106.5 $\pm$  2.1     26.7 $\pm$  18.8     5.3 $\pm$  3.5     5.1 $\pm$  3.5     2.5 $\pm$  4.3

Avg w/o e     25.5     23.2     23.9     26.0     5.6     17.0     26.9     16.4     6.3     5.0     5.1     5.9

Avg     58.6     55.4     57.6     59.2     36.2     50.9     58.0     51.3     31.6     20.4     23.4     20.9

§.§ Algorithms hyperparameters search with classification

The subsequent step in our investigation involved determining whether better performance could be achieved by tuning the hyperparameters of the original algorithms. The evaluation of the best hyperparameter sets can be found under the CE+AT columns in <ref>, <ref>, and <ref>. It is worth noting that, for ReBRAC in the AntMaze tasks, we used large-batch optimization, similar to that used in Gym-MuJoCo, which was previously hindered by Q-function divergence.

In Gym-MuJoCo tasks, these hyperparameter adjustments yielded slight improvements in ReBRAC's average performance, although random datasets remained problematic. Similarly, in Adroit tasks, performance saw some improvement but still fell short of the original algorithm's performance. Notably, for AntMaze tasks, the hyperparameter adjustments significantly enhanced ReBRAC's performance, establishing a new state-of-the-art performance in this domain.

For IQL and LB-SAC, tuning the algorithms' hyperparameters with a cross-entropy objective helped alleviate underperformance in Gym-MuJoCo tasks, although random datasets remained a common issue. However, this approach did not yield significant improvements for AntMaze tasks, and only marginally improved average performance in Adroit.

Following the methodology outlined by <cit.>, we investigated whether algorithms utilizing classification offered superior hyperparameter search under uniform policy selection on D4RL tasks, using Expected Online Performance (EOP). The results are presented in <ref> under the CE+AT rows. It is evident that for ReBRAC, classification facilitated much better hyperparameter search in the AntMaze domain, marginally improved search for Gym-MuJoCo, and showed slight degradation for Adroit tasks. IQL benefited only in the Adroit domain, while LB-SAC showed a slight improvement in AntMaze tasks.

Expected Online Performance <cit.> under uniform policy selection aggregated over D4RL domains across four training seeds. This demonstrates the sensitivity to the choice of hyperparameters given a certain budget for online evaluation. +CE+AT denotes cross-entropy with tuned algorithm parameters, +CE+CT denotes cross-entropy with tuned classification parameters, +CE+MT denotes cross-entropy with mixed parameters tuning.

max width=

Domain          Algorithm                1 policy            2 policies           3 policies       5 policies                    10 policies 9 policies for LB-SAC+CE+CT.                          15 policies     18 policies     20 policies
12*Gym-MuJoCo     ReBRAC     62.0 $\pm$ 17.1     70.6 $\pm$ 9.9     73.3 $\pm$ 5.5     74.8 $\pm$ 2.1     75.6 $\pm$ 0.8     75.8 $\pm$ 0.6     75.9 $\pm$ 0.6     76.0 $\pm$ 0.5

ReBRAC+CE+AT     64.1 $\pm$ 15.4     71.7 $\pm$ 8.7     74.0 $\pm$ 4.8     75.4 $\pm$ 1.9     76.2 $\pm$ 1.1     76.6 $\pm$ 1.0     76.7 $\pm$ 0.9     76.8 $\pm$ 0.8

ReBRAC+CE+CT     78.4 $\pm$ 1.4     79.2 $\pm$ 1.2     79.7 $\pm$ 1.0     80.1 $\pm$ 0.7     80.5 $\pm$ 0.5     80.6 $\pm$ 0.4     80.7 $\pm$ 0.4     80.7 $\pm$ 0.3

ReBRAC+CE+MT     62.0 $\pm$ 15.5     70.1 $\pm$ 10.5     73.3 $\pm$ 6.6     75.3 $\pm$ 2.9     76.4 $\pm$ 0.9     76.6 $\pm$ 0.5     76.7 $\pm$ 0.3     -

IQL     62.3 $\pm$ 9.8     67.6 $\pm$ 6.0     69.5 $\pm$ 3.8     70.9 $\pm$ 1.9     71.6 $\pm$ 0.7     71.8 $\pm$ 0.4     71.9 $\pm$ 0.3     71.9 $\pm$ 0.3

IQL+CE+AT     55.7 $\pm$ 4.2     58.1 $\pm$ 3.6     59.3 $\pm$ 2.8     60.5 $\pm$ 1.8     61.4 $\pm$ 0.9     61.7 $\pm$ 0.7     61.8 $\pm$ 0.6     61.8 $\pm$ 0.6

IQL+CE+CT     58.4 $\pm$ 7.2     62.5 $\pm$ 5.6     64.4 $\pm$ 4.3     66.1 $\pm$ 2.6     67.2 $\pm$ 1.3     67.6 $\pm$ 1.0     67.8 $\pm$ 0.9     67.9 $\pm$ 0.9

IQL+CE+MT     65.2 $\pm$ 3.6     67.2 $\pm$ 2.7     68.1 $\pm$ 2.0     68.9 $\pm$ 1.2     69.5 $\pm$ 0.6     69.7 $\pm$ 0.4     69.7 $\pm$ 0.3     -

LB-SAC     52.5 $\pm$ 13.8     59.5 $\pm$ 8.3     62.0 $\pm$ 5.5     64.1 $\pm$ 3.8     -     -     -     -

LB-SAC+CE+AT     48.9 $\pm$ 11.3     54.4 $\pm$ 7.0     56.4 $\pm$ 5.2     58.3 $\pm$ 4.1     -     -     -     -

LB-SAC+CE+CT     63.4 $\pm$ 3.7     65.4 $\pm$ 2.4     66.2 $\pm$ 1.7     66.9 $\pm$ 1.0     67.3 $\pm$ 0.6     -     -     -

LB-SAC+CE+MT     44.4 $\pm$ 11.1     50.5 $\pm$ 7.9     53.0 $\pm$ 5.4     55.0 $\pm$ 2.9     56.3 $\pm$ 1.5     56.8 $\pm$ 1.2     57.0 $\pm$ 1.0     -

12*AntMaze     ReBRAC     67.9 $\pm$ 10.0     73.6 $\pm$ 7.4     76.1 $\pm$ 5.5     78.3 $\pm$ 3.4     79.9 $\pm$ 1.7     80.4 $\pm$ 1.1     -     -

ReBRAC+CE+AT     84.8 $\pm$ 4.9     87.5 $\pm$ 3.5     88.7 $\pm$ 2.5     89.7 $\pm$ 1.5     90.4 $\pm$ 0.8     90.7 $\pm$ 0.5     -     -

ReBRAC+CE+CT     83.8 $\pm$ 10.3     88.3 $\pm$ 4.7     89.4 $\pm$ 2.2     90.0 $\pm$ 0.9     90.4 $\pm$ 0.6     90.6 $\pm$ 0.4     90.7 $\pm$ 0.4     90.7 $\pm$ 0.3

ReBRAC+CE+MT     63.5 $\pm$ 33.0     79.8 $\pm$ 22.3     85.8 $\pm$ 13.7     89.2 $\pm$ 5.2     90.8 $\pm$ 1.4     91.2 $\pm$ 0.9     91.3 $\pm$ 0.7     -

IQL     21.1 $\pm$ 5.4     24.1 $\pm$ 4.7     25.7 $\pm$ 4.2     27.5 $\pm$ 3.6     29.5 $\pm$ 2.7     30.4 $\pm$ 2.0     30.7 $\pm$ 1.7     30.9 $\pm$ 1.5

IQL+CE+AT     15.6 $\pm$ 0.9     16.1 $\pm$ 0.8     16.4 $\pm$ 0.7     16.7 $\pm$ 0.7     17.0 $\pm$ 0.6     17.2 $\pm$ 0.5     17.3 $\pm$ 0.4     17.3 $\pm$ 0.4

IQL+CE+CT     16.2 $\pm$ 0.7     16.6 $\pm$ 0.7     16.8 $\pm$ 0.7     17.1 $\pm$ 0.7     17.5 $\pm$ 0.6     17.7 $\pm$ 0.6     17.8 $\pm$ 0.5     17.8 $\pm$ 0.5

IQL+CE+MT     23.3 $\pm$ 9.3     28.1 $\pm$ 9.9     31.2 $\pm$ 9.6     35.1 $\pm$ 8.3     39.2 $\pm$ 5.1     40.6 $\pm$ 3.0     41.0 $\pm$ 2.3     -

LB-SAC     1.0 $\pm$ 1.4     1.6 $\pm$ 1.6     2.1 $\pm$ 1.6     2.8 $\pm$ 1.4     -     -     -     -

LB-SAC+CE+AT     3.2 $\pm$ 3.9     5.1 $\pm$ 3.9     6.3 $\pm$ 3.4     7.6 $\pm$ 2.2     -     -     -     -

LB-SAC+CE+CT     7.9 $\pm$ 1.5     8.7 $\pm$ 1.1     9.1 $\pm$ 0.9     9.4 $\pm$ 0.5     9.6 $\pm$ 0.3     -     -     -

LB-SAC+CE+MT     2.8 $\pm$ 3.7     4.6 $\pm$ 4.0     6.0 $\pm$ 3.8     7.6 $\pm$ 3.1     9.2 $\pm$ 1.7     9.7 $\pm$ 1.0     9.8 $\pm$ 0.8     -

12*Adroit     ReBRAC     44.1 $\pm$ 18.4     53.2 $\pm$ 10.9     56.1 $\pm$ 6.1     57.8 $\pm$ 2.3     58.6 $\pm$ 0.9     58.9 $\pm$ 0.7     59.0 $\pm$ 0.7     59.1 $\pm$ 0.6

ReBRAC+CE+AT     43.9 $\pm$ 17.1     52.7 $\pm$ 10.3     55.6 $\pm$ 6.0     57.5 $\pm$ 2.5     58.5 $\pm$ 0.9     58.7 $\pm$ 0.6     58.8 $\pm$ 0.6     58.9 $\pm$ 0.5

ReBRAC+CE+CT     56.9 $\pm$ 1.7     57.9 $\pm$ 1.3     58.4 $\pm$ 1.0     58.8 $\pm$ 0.7     59.2 $\pm$ 0.4     59.3 $\pm$ 0.3     59.4 $\pm$ 0.3     59.4 $\pm$ 0.2

ReBRAC+CE+MT     40.8 $\pm$ 17.0     49.8 $\pm$ 12.0     53.5 $\pm$ 8.1     56.3 $\pm$ 4.1     57.8 $\pm$ 1.4     58.0 $\pm$ 0.6     58.1 $\pm$ 0.4     -

IQL     33.9 $\pm$ 2.3     35.2 $\pm$ 1.7     35.8 $\pm$ 1.5     36.4 $\pm$ 1.2     37.1 $\pm$ 0.9     37.4 $\pm$ 0.6     37.5 $\pm$ 0.6     37.5 $\pm$ 0.5

IQL+CE+AT     53.0 $\pm$ 3.5     55.0 $\pm$ 3.1     56.1 $\pm$ 2.7     57.2 $\pm$ 1.9     58.1 $\pm$ 0.8     58.3 $\pm$ 0.4     58.4 $\pm$ 0.3     58.4 $\pm$ 0.3

IQL+CE+CT     49.6 $\pm$ 1.3     50.4 $\pm$ 0.9     50.7 $\pm$ 0.7     51.0 $\pm$ 0.6     51.3 $\pm$ 0.4     51.4 $\pm$ 0.4     51.5 $\pm$ 0.3     51.5 $\pm$ 0.3

IQL+CE+MT     54.2 $\pm$ 3.1     55.9 $\pm$ 2.3     56.7 $\pm$ 1.7     57.4 $\pm$ 1.0     57.8 $\pm$ 0.3     57.8 $\pm$ 0.1     57.8 $\pm$ 0.1     -

LB-SAC     15.7 $\pm$ 14.1     23.4 $\pm$ 12.8     27.7 $\pm$ 10.5     31.7 $\pm$ 6.6     -     -     -     -

LB-SAC+CE+AT     12.7 $\pm$ 8.0     17.1 $\pm$ 5.9     19.1 $\pm$ 4.3     20.6 $\pm$ 2.3     -     -     -     -

LB-SAC+CE+CT     22.0 $\pm$ 2.7     23.4 $\pm$ 2.6     24.2 $\pm$ 2.7     25.2 $\pm$ 2.6     26.4 $\pm$ 2.2     -     -     -

LB-SAC+CE+MT     12.5 $\pm$ 8.8     17.3 $\pm$ 6.5     19.4 $\pm$ 4.6     21.1 $\pm$ 2.6     22.3 $\pm$ 1.5     22.8 $\pm$ 1.2     23.0 $\pm$ 1.1     -

§.§ What is the impact of classification parameters?

In <cit.>, the authors conducted a limited study on the influence of specific classification hyperparameters, only examining if $\sigma/\zeta=0.75$ was suitable across varying numbers of bins $m$, using online RL tasks. Drawing inspiration from their work, we selected a set of $m$ values: ${21, 51, 101, 201, 401}$, and a set of $\sigma/\zeta$ values: ${0.55, 0.65, 0.75, 0.85}$. Subsequently, we conducted experiments using all possible pairs of these parameters, while maintaining the parameters of the original algorithms.

Evaluation results, presented in <ref>, <ref>, and <ref> under the CE+CT columns, showcased notable differences in the impact of tuning classification parameters across algorithms. Specifically, ReBRAC exhibited improved performance when classification hyperparameters were fine-tuned, surpassing the original version's efficacy. In contrast, IQL and LB-SAC did not consistently benefit from classification parameter tuning compared to algorithm-specific adjustments.

EOP for this hyperparameter search is also provided in <ref> under the +CE+CT rows. It's evident that when having good hyperparameters for original algorithm tuning classification parameters yields greater benefits than tuning algorithm-specific hyperparameters, as expected, with the only exception of IQL on Adroit tasks. Additionally, this search converges quickly, with training approximately three different policies often sufficing to achieve near-optimal performance, a highly advantageous property for real-world application of offline RL.

We present algorithms' performance heatmaps for the parameter grid averaged over domains in <ref>, with detailed scores per dataset available in <ref>. Additionally, in <ref>, we provide further insights by fixing one parameter and averaging over the second. Our analysis indicates that setting $m = 101$ and $\sigma/\zeta = 0.75$ may serve as a reasonable starting point and produce results better than the average. Notably, a higher number of classes often yields improved performance, suggesting a preference for larger $m$ values. Regarding $\sigma/\zeta$, our findings suggest that $0.75$ remains a favorable choice, especially when faced with limited tuning resources. However, the optimal selection of classification parameters heavily depends on the specific algorithm, environment, and dataset characteristics.

Heatmaps for the impact of the classification parameters averaged over the domains. See <ref> and <ref> for more results.

<cit.> did not investigate the impact of the $v_{min}$ and $v_{max}$ choices, nor did they propose a method for selecting these when the reward function is unknown. In our previous experiments, we computed these values by taking the minimum and maximum return across all possible sub-trajectories in the dataset. However, this approach may not be optimal because offline RL algorithms often introduce pessimism into the Q function. Additionally, setting these limits based on extreme values may cause the values on the edges of the support to differ from other values from the model's perspective, potentially impacting the final results.

To evaluate the effect of these parameters, we computed the support size as $v_{max} - v_{min}$ from the dataset and multiplied this size by a parameter $v_{expand}$. We then extended the support in two ways: Subtracting $v_{expand}(v_{max} - v_{min})$ from $v_{min}$ (referred to as $min$), and simultaneously subtracting from $v_{min}$ and adding to $v_{max}$ the term $v_{expand}(v_{max} - v_{min})/2$ (referred to as $both$)[The division by 2 ensures equal bin sizes across the two variants.]. The experimental results, shown in <ref>, demonstrate that our initial choice of support parameters was sub-optimal and increasing the support range may strongly benefit. Surprisingly, in some cases, reducing the support was beneficial. The $both$ strategy generally yielded better performance.

Our observations support the assumption about efficiency of cross-entropy for offline RL algorithms with policy regularization compared to other types. Notably, ReBRAC exhibited less sensitivity to the classification parameters.

Dependency of the algorithms performance on $v_{expand}$ values averaged over domains. See <ref> for tabular representation.

§.§ Do MLPs scale better with classification?

In <cit.>, authors emphasized the enhanced scalability of models when regression is replaced with classification, particularly when using Transformers or ResNets. In this subsection, we delve into whether similar scalability benefits apply to Multilayer Perceptrons (MLPs), which are commonly utilized in the development of novel RL approaches. Experimental results are presented in <ref>. Surprisingly, our findings suggest that there is no consistent improvement in terms of scaling when MSE is replaced with cross-entropy.

This discrepancy from the results reported by <cit.> could be attributed to the fundamental architectural differences between MLPs and models like Transformers or ResNets. Unlike these latter architectures, vanilla MLPs typically lack residual connections <cit.>, which have been identified as a crucial factor in enabling effective scaling with depth. Consequently, the absence of such connections in MLPs may limit their ability to capitalize on the benefits offered by classification over regression.

Dependency of the algorithms performance on the number of additional layers averaged over domains. See <ref> for tabular representation.

§.§ Combining the findings

We investigated whether a mixed tuning strategy for hyperparameters could enhance classification performance, building on our previous findings. For each algorithm, we tuned the following parameters:
$m$ from the set $\{201, 401\}$, $v_{expand}$ from the set of $\{-0.05, 0.05, 0.1\}$ using the $both$ strategy, and three values for one of algorithm-specific parameter: $\beta_1$ for ReBRAC, IQL $\tau$ for IQL, and N critics for LB-SAC. For ReBRAC and IQL, the second parameter was held constant across all tasks (see <ref>). $\sigma/\zeta$ was set to $0.75$. To enhance readability, we provide detailed per-dataset results in a separate tables in <ref> and EOP results in <ref> under the +CE+MT rows.

Per-dataset results indicate that the proposed strategy yields better performance in most cases, performing on par in others, with the exception of LB-SAC on Gym-MuJoCo. It makes this approach the best choice when no prior knowledge on optimal algorithm-specific parameters is available.

EOP results reveal that this tuning strategy is particularly effective for IQL, especially under a low fine-tuning budget. For ReBRAC, benefits are apparent only under a high fine-tuning budget. Conversely, this strategy did not perform well for LB-SAC. The suboptimal performance of LB-SAC can be attributed to the strong dependence on ensemble size; due to computational constraints, we were unable to use an ensemble size of 50, which is optimal for many datasets.

To conclude, our experiments show that the best performance with classification can be achieved by tuning algorithm-specific hyperparameters in conjunction with classification parameters, particularly the support range and the number of bins.

§ RELATED WORK

Prior research beyond the realm of RL has demonstrated the potential performance enhancements associated with replacing regression with classification objectives <cit.>. Within the RL domain, some studies have experimented with employing classification objectives as a workaround, albeit without conducting comprehensive analyses of this modification <cit.>. Categorical distributional RL <cit.> works are also relevant for the considered topic, where the classification is also used, however usage of classification instead of regression is not a central topic in this reseach direction. Additionally, several works in offline RL have demonstrated the benefits of utilizing classification objectives for various tasks, albeit lacking in-depth analyses of this specific component and its elements <cit.>.

To the best of our knowledge, <cit.> represent the first and only study to make regression replacement with classification a central research question. The authors compared different methods of converting regression targets into classification targets, providing experimental results across a diverse array of tasks encompassing Atari games, robotics, and natural language processing problems in online and offline RL setups. Their study revealed that HL-Gauss <cit.> represents the optimal approach for representing RL regression targets with categorical distributions while this was not the case for the supervised regression tasks <cit.>. The authors assert that cross-entropy serves as a "drop-in" replacement for MSE in RL, leading to a more stable training process and enhanced scalability with deep neural network architectures such as Transformers <cit.> or ResNets <cit.>. Our work draws primary inspiration from <cit.> and aims to provide a more in-depth analysis of this phenomenon in offline RL, which we believe holds significant potential benefits for both offline RL researchers and practitioners.

§ CONCLUSION

In this study, we explored the impact of integrating classification objectives into offline RL algorithms.  Our findings provide nuanced insights into the efficacy of classification in improving offline RL algorithm performance. Initially, we examined whether classification objectives could be seamlessly integrated into existing algorithms without altering other aspects. While some algorithm with policy regularization (ReBRAC) demonstrated promising results across tasks like Gym-MuJoCo and AntMaze, other algorithms with implicit regularization (IQL) and algorithm with Q function regularization (LB-SAC) faced challenges. And the only case when classification have high chances to bring improvement wihout much effort is the divergence of Q function with MSE loss.

Next, we explored the impact of different hyperparameter tuning with classification objectives. Notably, we observed performance improvements for ReBRAC, when tuning classification hyperparameters over algorithm-specific ones. Other results underscore the importance of carefully selecting both types of hyperparameters: algorithm-specific and classification-specific, when employing classification.

Furthermore, our investigation into the scalability of MLPs with classification revealed mixed results. Contrary to previous findings with architectures like Transformers and ResNets, we did not observe consistent improvements in scaling when using classification objectives with MLPs. This highlights the importance of considering the architectural nuances of different models when assessing the potential benefits of classification objectives.

Looking ahead, several avenues for future research emerge. One promising direction involves examining how classification affects the performance of offline algorithms in offline-to-online RL setup. Understanding how classification objectives impact the transferability of learned policies to online settings could provide valuable insights into the practical applicability of classification usage. Another theoretically promising direction is leveraging classification-specifics for uncertainty estimation in offline RL. For example, the entropy provided by the Q function could be used to incorporate pessimism into offline RL algorithms.

In conclusion, our study underscores the need for a nuanced understanding of the interplay between algorithm design, task characteristics, and the integration of classification objectives in RL. While classification holds promise in certain contexts, its efficacy is highly dependent on factors such as algorithm design and hyperparameter selection. Future research could further explore these nuances and develop approaches that leverage classification objectives optimally across a diverse range of RL tasks and algorithms.

§ ACKNOWLEDGMENTS

We thank Niao He and Ali Batuhan Yardim for providing computational resources and offering valuable feedback on the project.

§ EXPERIMENTAL DETAILS

For results with original algorithms and algorithms hyperparameters search in <ref> <ref> and <ref>, we conducted a hyperparameter search and selected the best results from the final evaluations for each dataset. We used the JAX implementation of ReBRAC from the Clean Offline RL (CORL) library <cit.> and used the same code template for IQL and LB-SAC. Algorithmic part of IQL implementation is based on the original codebase from <cit.> and in case of LB-SAC we have adapted SAC-N implementation from <https://github.com/Howuhh/sac-n-jax>.

The experiments were conducted on RTX Titan, Quadro RTX 6000 and Quadro RTX 8000.

Our study utilized the v2 version of datasets for Gym-MuJoCo and AntMaze, and v1 for Adroit. The agents were trained for one million steps in all domains and evaluated over ten episodes for Gym-MuJoCo and Adroit and over one hundred episodes for AntMaze. Following <cit.>, AntMaze reward function is multiplied by 100.

For ReBRAC, we fine-tuned the $\beta_1$ parameter with ${0.001, 0.01, 0.05, 0.1}$ values and the $\beta_2$ parameter with ${0, 0.001, 0.01, 0.1, 0.5}$ values for Gym-MuJoCo and Adroit. For AntMaze the corresponding ranges are ${0.0005, 0.001, 0.002, 0.003}$ and ${0, 0.0001, 0.0005, 0.001}$. When replacing regression with classification for AntMaze we also set batch size to 1024 and learning rates to $0.001$ as it is was done for Gym-MuJoCo in original algorithm which slightly improves the performance.

For IQL in all domains, we selected $\beta$ value from ${0.5, 1, 3, 6, 10}$ and IQL $\tau$ from ${0.5, 0.7, 0.9, 0.95}$.

For LB-SAC we selected the number of critics in the range of ${2, 5, 10, 25, 50}$. Note, that for LB-SAC we used sub-optimal parameters of the batch size and learning rate due to the computational constraints.

§.§ Mixed tuning hyperparameters choice

For ReBRAC, we fine-tuned the $\beta_1$ parameter with ${0.001, 0.01, 0.05}$ values and the $\beta_2$ parameter was set to $0.001$.

For IQL, we selected IQL $\tau$ from ${0.5, 0.7, 0.9}$ and set $\beta$ value to $3$.

For LB-SAC we selected the number of critics in the range of ${2, 10, 25}$.

§ HYPERPARAMETERS

§.§ ReBRAC

ReBRAC's general hyperparameters.

Parameter     Value

optimizer                             Adam <cit.>

batch size                            1024 on Gym-MuJoCo and AntMaze (for classification),  256 otherwise

learning rate (all networks)           1e-3 on Gym-MuJoCo and AntMaze (for classification), 3e-4 otherwise

tau ($\tau$)                          5e-3

hidden dim (all networks)             256

num hidden layers (all networks)      3

gamma ($\gamma$)                      0.999 on AntMaze, 0.99 otherwise

nonlinearity                          ReLU

ReBRAC's best hyperparameters.

max width=

Task Name     $\beta_1$ (MSE)     $\beta_2$ (MSE)     $\beta_1$ (CE+AT)     $\beta_2$ (CE+AT)     $m$ (CE+CT)     $\sigma/\zeta$ (CE+CT)     $\beta_1$ (CE+MT)     $m$ (CE+MT)     $v_{expand}$ (CE+MT)

halfcheetah-random     0.001     0.1      0.001     0.1     101     0.85     0.001      201      0.05

halfcheetah-medium     0.001     0.01     0.001     0.5     21     0.75      0.001     201      0.1

halfcheetah-expert     0.01     0.01     0.01     0.01     201     0.85     0.01      201      0.05

halfcheetah-medium-expert     0.01     0.1      0.01     0.001     101     0.85      0.01     201      0.1

halfcheetah-medium-replay     0.01     0.001      0.001     0.001     201     0.65     0.001      201      0.1

halfcheetah-full-replay     0.001     0.1     0.001     0.001     21     0.75      0.001      201     -0.05

hopper-random     0.001     0.01     0.001     0.5     201     0.85     0.001      401      -0.05

hopper-medium     0.01     0.001     0.01     0.001     201     0.75     0.01       201     -0.05

hopper-expert     0.1     0.001     0.1     0.1     401      0.65     0.05      401       0.1

hopper-medium-expert     0.1     0.01     0.1     0.0     101     0.75      0.05     401      0.05

hopper-medium-replay     0.05     0.5     0.05     0.0     101     0.75     0.01      401      -0.05

hopper-full-replay     0.01     0.01     0.01     0.0     101     0.65     0.01       201     0.05

walker2d-random     0.01     0.0     0.05     0.1     101     0.65     0.05       201     0.05

walker2d-medium     0.05     0.1     0.05     0.1     101     0.55      0.05     201      0.1

walker2d-expert     0.01     0.5     0.01     0.5     101     0.75     0.01      401      0.05

walker2d-medium-expert     0.01     0.01     0.01     0.5     401     0.85      0.05     401      0.05

walker2d-medium-replay     0.05     0.01     0.05     0.0     201     0.65     0.01      401      -0.05

walker2d-full-replay     0.01     0.01     0.001     0.001     201     0.55      0.001     201       0.1

antmaze-umaze              0.003     0.002     0.003     0.0     401     0.75     0.05      401      0.1

antmaze-umaze-diverse      0.003     0.001     0.003     0.002     201     0.75      0.01     201      0.1

antmaze-medium-play        0.001     0.0005     0.003     0.0005     201     0.65     0.001      401      0.05

antmaze-medium-diverse     0.001     0.0     0.003     0.002     201     0.85      0.01     401       0.1

antmaze-large-play         0.002     0.001     0.003     0.0005     101     0.75     0.01      401      0.1

antmaze-large-diverse      0.002     0.002     0.003     0.0005     401     0.75      0.01      401     0.1

pen-human     0.1     0.5     0.1     0.001     51     0.85      0.05      201     -0.05

pen-cloned      0.05     0.5     0.1     0.001     401     0.75     0.05      401      0.05

pen-expert     0.01     0.01     0.01     0.0     401     0.85     0.01     401     -0.05

door-human     0.1     0.1     0.1     0.1     401     0.85      0.001      401     -0.05

door-cloned     0.01     0.1      0.1     0.001     101     0.55     0.05       201     0.1

door-expert      0.05     0.01     0.05     0.001     401     0.75     0.05      401     -0.05

hammer-human     0.01     0.5     0.1     0.001     51     0.85     0.05       201     -0.05

hammer-cloned     0.1     0.5     0.05     0.1     21     0.85      0.05      401      0.05

hammer-expert      0.01     0.01     0.01     0.5     401     0.55      0.01     401      0.1

relocate-human     0.1     0.01     0.1     0.01     201     0.55     0.05      201      0.05

relocate-cloned     0.1     0.01     0.1     0.0     401     0.75      0.05      201     0.1

relocate-expert     0.05     0.01     0.05     0.5     21     0.55     0.05      401     0.05

§.§ IQL

IQL's general hyperparameters.

Parameter     Value

optimizer                             Adam <cit.>

batch size                            256

learning rate (all networks)          3e-4

tau ($\tau$)                          5e-3

hidden dim (all networks)             256

num hidden layers (all networks)      2

gamma ($\gamma$)                      0.99

nonlinearity                          ReLU

learning rate decay                   Cosine

dropout rate                          0.1 for Adroit, 0 otherwise

IQL's best hyperparameters.

max width=

Task Name     IQL $\tau$ (MSE)     $\beta$ (MSE)     IQL $\tau$ (CE+AT)     $\beta$ (CE+AT)     $m$ (CE+CT)     $\sigma/\zeta$ (CE+CT)     IQL $\tau$ (CE+MT)     $m$ (CE+MT)     $v_{expand}$ (CE+MT)

halfcheetah-random     0.95     10.0     0.5     3.0     401     0.55     0.7      401      -0.05

halfcheetah-medium     0.95     3.0     0.5     10.0     401     0.55     0.7      401      0.1

halfcheetah-expert     0.7     6.0     0.7     1.0     401     0.55     0.5      401      0.1

halfcheetah-medium-expert     0.5     0.5     0.5     3.0     401     0.75      0.7      401     0.1

halfcheetah-medium-replay     0.9     6.0     0.5     6.0     401     0.55      0.7       401     0.1

halfcheetah-full-replay     0.5     0.5     0.5     6.0     401     0.55     0.5      401      0.1

hopper-random     0.95     10.0     0.95     6.0     51     0.85      0.9      201     -0.05

hopper-medium     0.7     0.5     0.7     10.0     51     0.85      0.5     201       0.05

hopper-expert     0.9     0.5     0.7     3.0     201     0.75     0.9      201      -0.05

hopper-medium-expert     0.7     10.0     0.5     3.0     201     0.85      0.5     401      0.05

hopper-medium-replay     0.7     0.5     0.5     6.0     401     0.85     0.7      401      0.05

hopper-full-replay     0.7     3.0     0.5     3.0     401     0.55      0.5     401      -0.05

walker2d-random     0.9     0.5     0.95     3.0     21     0.65      0.9     401      0.1

walker2d-medium     0.5     1.0     0.5     0.5     101     0.55      0.7      401     0.1

walker2d-expert     0.7     6.0     0.7     10.0     401     0.55     0.7      401      0.05

walker2d-medium-expert     0.7     3.0     0.5     6.0     401     0.85     0.5      401      0.1

walker2d-medium-replay     0.7     1.0     0.5     6.0     401     0.75      0.7     401      0.05

walker2d-full-replay     10.0     0.7     0.5     1.0     401     0.55      0.7     401      0.1

antmaze-umaze              0.7     10.0     0.5     3.0     101     0.65      0.9      201     0.05

antmaze-umaze-diverse      0.9     10.0     0.9     6.0     51     0.75      0.5     201       0.05

antmaze-medium-play        0.9     6.0     0.9     1.0     51     0.75      0.9     401       0.1

antmaze-medium-diverse     0.9     6.0     0.95     6.0     51     0.75      0.9     401      0.1

antmaze-large-play         0.9     10.0     0.9     10.0     101     0.75      0.9     401      0.1

antmaze-large-diverse      0.9     6.0     0.95     3.0     201     0.55     0.9      201      0.05

pen-human     0.7     0.5     0.7     1.0     201     0.85     0.9      201      -0.05

pen-cloned      0.9     10.0     0.5     0.5     401     0.65     0.5      401      0.1

pen-expert     0.9     0.5     0.5     10.0     401     0.75      0.5      401     -0.05

door-human     0.95     1.0     0.9     6.0     401     0.75     0.9      201      0.1

door-cloned     0.7     1.0     0.9     0.5     21     0.75      0.7     201      0.1

door-expert     0.9     10.0     0.9     6.0     101     0.55      0.7      201     0.05

hammer-human     0.9     10.0     0.7     3.0     21     0.55      0.7     401      -0.05

hammer-cloned     0.7     6.0     0.7     10.0     21     0.65      0.7     201      0.1

hammer-expert     0.95     0.5     0.95     0.5     401     0.65     0.9      401      -0.05

relocate-human     0.7     10.0     0.7     1.0     101     0.75     0.9      401      0.05

relocate-cloned     0.7     1.0     0.7     1.0     201     0.65      0.9     401       0.05

relocate-expert     0.5     0.5     0.95     0.5     401     0.75      0.9      401     0.1

§.§ LB-SAC

LB-SAC's general hyperparameters.

Parameter     Value

optimizer                             Adam <cit.>

batch size                            1024

learning rate (all networks)           6e-4

tau ($\tau$)                          5e-3

hidden dim (all networks)             256

num hidden layers (all networks)      3

gamma ($\gamma$)                      0.99

nonlinearity                          ReLU

LB-SAC's best hyperparameters.

max width=

Task Name     N critics (MSE)     N critics (CE+AT)     $m$ (CE+CT)     $\sigma/\zeta$ (CE+CT)     N critics (CE+MT)     $m$ (CE+MT)     $v_{expand}$ (CE+MT)

halfcheetah-random     2     50     51      0.75       2     201      0.1

halfcheetah-medium     2     5     101      0.75       10     201      -0.05

halfcheetah-expert     5     5     51      0.85       10      201     0.05

halfcheetah-medium-expert     5     5     201     0.85     10      201      -0.05

halfcheetah-medium-replay     5     5     201     0.85     10      201      0.1

halfcheetah-full-replay     2     2     201     0.75     2      401      -0.05

hopper-random     5     25     201     0.85     25      401      -0.05

hopper-medium     25     25     201     0.75       25     401      0.1

hopper-expert     50     50     201      0.65      25      401      0.05

hopper-medium-expert     50     50     51     0.75      25     201      0.1

hopper-medium-replay     5     5     101     0.85      10     201      0.05

hopper-full-replay     5     5     51     0.75     10      201      0.05

walker2d-random     50     5     51     0.75     10      201      0.05

walker2d-medium     10     10     101     0.85      10     201      -0.05

walker2d-expert     25     50     201     0.65      25     401      -0.05

walker2d-medium-expert     10     25     201     0.85      10     401      0.05

walker2d-medium-replay     5     5     101     0.85     10      201      0.1

walker2d-full-replay     5     5     201     0.85     10      201      0.1

antmaze-umaze              5     5     201     0.65      2     201      0.1

antmaze-umaze-diverse      25     2     101     0.75      2     401      0.05

antmaze-medium-play        25     5     101     0.75      2      401      0.05

antmaze-medium-diverse     25     5     101     0.75      2     401      0.05

antmaze-large-play         25     25     101     0.75      25     201      0.1

antmaze-large-diverse      25     25     101     0.75     25      201      0.1

pen-human     5     10     51      0.75       10     201      0.05

pen-cloned     50     50     51     0.65      25      201     -0.05

pen-expert     25     25     201     0.85     25      201      0.1

door-human     2     50     51     0.75     25      201     0.05

door-cloned     50     50     201     0.85      25     201      0.05

door-expert     50     50     201     0.75      25     401      0.05

hammer-human     5     50     101     0.75      2      201     -0.05

hammer-cloned     50     25     51     0.85      10     201      -0.05

hammer-expert     25     25     201     0.85     10      401      0.05

relocate-human      2     50     101     0.65     25      401      0.1

relocate-cloned      50     50     101     0.85      25     201      -0.05

relocate-expert      50     25     101     0.75     25       401     -0.05

§ COMPUTATIONAL COSTS

Computational costs for all experiments.
Note, ReBRAC computational costs are taken from <cit.>. The total amount of compute is approximately 23497 hours (979 days).

max width=

Algorithm     Number of runs     Approximate hours per run

ReBRAC+MSE, tuning     2784     0.39

ReBRAC+CE+AT, tuning     2784     0.34

ReBRAC+CE+CT, tuning     2880     0.48

ReBRAC+CE+MT, tuning     2592     0.50

IQL+MSE, tuning     2880     0.3

IQL+CE+AT, tuning     2880     0.26

IQL+CE+CT, tuning     2880     0.27

IQL+CE+MT, tuning     2592     0.30

LB-SAC+MSE, tuning     720     1.29

LB-SAC+CE+AT, tuning     720     1.61

LB-SAC+CE+CT, tuning     1296     2.32

LB-SAC+CE+MT, tuning     2592     1.56

ReBRAC+MSE, eval     360     0.36

ReBRAC+CE, eval     144     0.45

ReBRAC+CE+AT, eval     144     0.47

ReBRAC+CE+CT, eval     144     0.47

ReBRAC+CE+MT, eval     144     0.34

IQL+MSE, eval     144     0.35

IQL+CE, eval     144     0.33

IQL+CE+AT, eval     144     0.24

IQL+CE+CT, eval     144     0.34

IQL+CE+MT, eval     144     0.34

LB-SAC+MSE, eval     144     1.25

LB-SAC+CE, eval     144     1.34

LB-SAC+CE+AT, eval     144     1.69

LB-SAC+CE+CT, eval     144     2.21

LB-SAC+CE+MT, eval     144     1.89

ReBRAC+MSE, depth scale     720     0.61

ReBRAC+CE, depth scale     720     0.38

IQL+MSE, depth scale     720     0.35

IQL+CE, depth scale     720     0.38

ReBRAC, $v_{expand}$     1440     0.45

IQL+CE, $v_{expand}$     1440     0.34

LB-SAC+CE, $v_{expand}$     1440     2.53

Sum (w/o ReBRAC MSE)     34032     0.69

§ MLPS SCALE

Dependency of the algorithms performance on the number of additional layers averaged over domains.

max width=

Domain          Algorithm                +0 layers            +1 layer           +2 layers       +3 layers                    +4 layers
4*Gym-MuJoCo     ReBRAC

4*AntMaze     ReBRAC     76.4     75.8     61.2     55.8     52.2

ReBRAC+CE     89.2

4*Adroit     ReBRAC

§ IMPACT OF $M$ AND $\SIGMA/\ZETA$

Average performance on different domains for a fixed $m$ value and averaged over $\sigma/\zeta$.

max width=

Domain          Algorithm                21            51           101       201                    401

ReBRAC+CE     77.6     78.0     80.2     79.5     76.7

IQL+CE     48.4     52.6     58.9     65.4     66.6

LB-SAC+CE     -     59.0     65.9     65.2     -

ReBRAC+CE     63.7     87.1     89.4     89.1     89.2

IQL+CE     15.5     16.4     16.8     15.9     16.1

LB-SAC+CE     -     6.0     8.2     9.3     -

ReBRAC+CE     56.2     56.9     55.5     57.9     58.0

IQL+CE     49.3     47.5     49.8     50.3     50.9

LB-SAC+CE     -     19.8     22.5     23.5     -

Average (w/o LB-SAC)         51.7     56.4     58.4     59.6     59.5

Average         -     47.0     49.6     50.6     -

Average performance on different domains for a fixed $\sigma/\zeta$ value and averaged over $m$.

max width=

Domain          Algorithm                0.55            0.65           0.75       0.85

ReBRAC+CE     78.7     78.4     78.4     78.0

IQL+CE     59.2     58.1     58.3     57.8

LB-SAC+CE     -     59.0     65.9     65.2

ReBRAC+CE     85.3     83.0     83.8     82.7

IQL+CE     16.1     16.5     16.3     15.7

LB-SAC+CE     -     6.0     8.2     9.3

ReBRAC+CE     56.8     56.8     57.3     56.6

IQL+CE     49.4     49.7     49.7     49.4

LB-SAC+CE     -     19.8     22.5     23.5

Average (w/o LB-SAC)         57.5     57.0     57.3     56.7

Average         -     47.4     48.9      48.6

§ IMPACT OF $V_{MIN}$ AND $V_{MAX}$

Dependency of the algorithm's performance on the $v_{expand}$ parameter.

max width=

Domain     Algorithm     -0.05     0.0     0.05     0.1     0.2
6*Gym-MuJoCo     ReBRAC, $both$     81.0     79.9     80.4     79.8     80.2

ReBRAC, $min$     71.2     79.9     80.6     80.7     79.0
IQL, $both$     61.3     62.7     61.1     60.2     61.1

IQL, $min$     61.4     62.7     58.2     57.7     56.7

LB-SAC, $both$
59.9     64.0     66.1     67.5     65.6

LB-SAC, $min$
65.8     64.0     64.7     66.1     64.5

6*AntMaze     ReBRAC, $both$     90.1     86.3     89.0     89.3     83.6

ReBRAC, $min$     90.2     86.3     87.5     89.8     86.4
IQL, $both$     16.7     16.6     34.8     33.0     22.8

IQL, $min$     16.9     16.6     16.2     16.2     16.2

LB-SAC, $both$
3.5     8.1     6.0     6.0     4.25

LB-SAC, $min$
10.3     8.1     7.5     7.2     3.5

6*Adroit     ReBRAC, $both$     56.5     54.3     55.5     59.9     55.2

ReBRAC, $min$     56.7     54.3     56.6     57.7     56.8
IQL, $both$     49.7     49.2     51.8     51.1     51.0

IQL, $min$     49.4     49.2     50.9     50.0     47.6

LB-SAC, $both$
23.5     22.3     24.5     22.8     19.1

LB-SAC, $min$
21.3     22.3     19.3     18.8     21.0

§ MIXED TUNING RESULTS

Average normalized score over the final evaluation and four unseen training seeds on Gym-MuJoCo tasks. CE+MT denotes cross-entropy with mixed hyperparameters tuning.

max width=

1c     3cReBRAC     3cIQL     3cLB-SAC

Task     MSE     CE     CE+MT     MSE     CE     CE+MT     MSE     CE     CE+MT

hc-r      29.5 $\pm$  1.5     13.6 $\pm$  0.7     9.7 $\pm$  0.3
18.8 $\pm$  1.5     1.9 $\pm$  0.0     9.2 $\pm$  4.1
28.2 $\pm$  1.4     10.0 $\pm$  0.3     9.9 $\pm$  0.2

hc-m     65.6 $\pm$  1.0      59.4 $\pm$  0.3     62.1 $\pm$ 0.6
49.8 $\pm$  0.2     42.3 $\pm$  1.7     46.7 $\pm$  0.2
64.5 $\pm$  1.3     56.7 $\pm$  2.1     65.5 $\pm$  0.5

hc-e     105.9 $\pm$  1.7      106.2 $\pm$  2.9     106.3 $\pm$ 1.0
96.4 $\pm$  0.6     92.7 $\pm$  0.1     94.5 $\pm$  0.1
103.0 $\pm$  1.5     103.9 $\pm$  1.0     98.2 $\pm$  1.6

hc-me     101.1 $\pm$  5.2     101.8 $\pm$  3.3     104.1 $\pm$  3.1
92.1 $\pm$  3.2     88.1 $\pm$  3.0     90.5 $\pm$  4.1
104.5 $\pm$  2.4     105.4 $\pm$  2.0     103.2 $\pm$  2.0

hc-mr     51.0 $\pm$  0.8     50.3 $\pm$  0.2     49.8 $\pm$  6.9
45.6 $\pm$  0.3     38.5 $\pm$  1.4     43.1 $\pm$  0.1
52.8 $\pm$  0.7     55.4 $\pm$  0.9     54.5 $\pm$  1.1

hc-fr     82.1 $\pm$  1.1     83.9 $\pm$  1.2     80.4 $\pm$  1.7
74.5 $\pm$  1.8     63.3 $\pm$  0.6     73.1 $\pm$  0.4
79.0 $\pm$  2.0     80.7 $\pm$  0.3     76.6 $\pm$  9.7

hp-r     8.1 $\pm$  2.4     7.5 $\pm$  0.3     8.2 $\pm$  5.3
5.7 $\pm$  2.3     16.1  $\pm$  3.4     9.5 $\pm$  0.0
14.5 $\pm$  11.5     8.2 $\pm$  2.2     14.8 $\pm$  11.0

hp-m     102.0 $\pm$  1.0     102.4 $\pm$  0.2     100.7 $\pm$  3.0
59.8 $\pm$  5.0     53.7 $\pm$  2.0     47.6 $\pm$   4.2
90.0 $\pm$  27.5     7.9 $\pm$  0.8     15.6 $\pm$  13.5

hp-e     100.1 $\pm$  8.3     107.0 $\pm$  5.7     110.6 $\pm$  0.3
110.8 $\pm$  1.8     110.3 $\pm$  0.7     110.7 $\pm$  0.3
1.3 $\pm$  0.0     21.6 $\pm$  39.7     12.8 $\pm$  12.8

hp-me     107.0 $\pm$  6.4     110.9 $\pm$  0.8     108.2 $\pm$  7.3
95.3 $\pm$  31.8     65.3 $\pm$  8.7     72.9 $\pm$  39.1
111.3 $\pm$  0.3     14.9 $\pm$  7.0     9.1 $\pm$  5.5

hp-mr     98.1 $\pm$  5.3     99.5 $\pm$  1.3     102.3 $\pm$  0.9
91.4 $\pm$  5.5     21.8 $\pm$  3.8     83.4 $\pm$  15.7
63.0 $\pm$  48.1     66.9 $\pm$  42.8     66.7 $\pm$  32.7

hp-fr     107.1 $\pm$  0.4     108.1 $\pm$  0.6     108.6 $\pm$  0.4
106.9 $\pm$  1.0     42.7 $\pm$  2.7      105.6 $\pm$  0.3
107.0 $\pm$  0.7     65.6 $\pm$  50.2     100.3 $\pm$  2.6

wl-r     18.4 $\pm$  4.5     4.8 $\pm$  1.0      4.2 $\pm$  1.4
4.5 $\pm$  6.2     4.2 $\pm$  2.1     7.7 $\pm$  3.8
21.7 $\pm$  0.0     20.1 $\pm$  2.9     21.7 $\pm$  0.1

wl-m     82.5 $\pm$  3.6     84.9 $\pm$  3.7      85.2 $\pm$  0.5
83.7 $\pm$  0.2     78.8 $\pm$  1.5     80.7 $\pm$  3.8
89.3 $\pm$  5.3     89.6 $\pm$  10.7     95.9 $\pm$  3.3

wl-e     112.3 $\pm$  0.2     112.6 $\pm$  0.2      112.3 $\pm$  0.0
113.9 $\pm$  0.0     108.5 $\pm$  0.3     109.4 $\pm$ 0.1
114.2 $\pm$  0.4     59.8 $\pm$  45.0     109.8 $\pm$  1.2

wl-me     111.6 $\pm$  0.3     112.0 $\pm$  0.1      110.1 $\pm$  0.4
112.8 $\pm$  0.4     96.0 $\pm$  7.8     110.8 $\pm$  0.4
110.6 $\pm$  0.4     73.1 $\pm$  18.1     111.3 $\pm$  2.5

wl-mr     77.3 $\pm$  7.9     82.9 $\pm$  4.2     82.4 $\pm$  13.5
81.9 $\pm$  6.3     55.8 $\pm$  10.1     81.9 $\pm$  3.8
92.6 $\pm$  2.7     90.9 $\pm$  5.3     84.3 $\pm$  8.9

wl-fr     102.2 $\pm$  1.7     101.1 $\pm$  2.6     100.7 $\pm$  21.1
98.2 $\pm$  0.9     85.7 $\pm$  3.4      94.0 $\pm$  1.2
102.1 $\pm$  1.0     110.4 $\pm$  1.9      95.7 $\pm$  4.0

Avg     81.2     80.4     80.3
74.5     59.2     70.6
74.9     57.8     63.6

Average normalized score over the final evaluation and ten unseen training seeds on AntMaze tasks.

max width=

1c     3cReBRAC     3cIQL     3cLB-SAC

Task     MSE     CE     CE+MT      MSE     CE     CE+MT     MSE     CE     CE+MT

um      97.8 $\pm$  1.0     95.0 $\pm$  6.0     98.2 $\pm$  1.7
72.5 $\pm$  20.6     48.0 $\pm$  2.4     80.0 $\pm$  4.6
18.25 $\pm$  35.8     41.0 $\pm$  33.2     61.0 $\pm$  13.7

um-d     88.3 $\pm$  13.0     91.7 $\pm$  4.2      86.7 $\pm$  7.2
72.5 $\pm$  9.5     52.0 $\pm$  6.2     45.2 $\pm$  4.5
0.0 $\pm$  0.0     0.0 $\pm$  0.0     1.0 $\pm$  2.0

med-p      84.0 $\pm$  4.2     81.2 $\pm$  2.0     89.2 $\pm$  4.9
80.0 $\pm$  8.1     0.5 $\pm$  1.0     47.7 $\pm$  6.5
0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

med-d     76.3  $\pm$  13.5     73.5 $\pm$  7.0     92.5 $\pm$ 1.9
72.5 $\pm$  12.5     0.0 $\pm$  0.0     45.7 $\pm$  6.9
0.0 $\pm$  0.0     0.0 $\pm$  0.0     8.0 $\pm$  16.0

lrg-p     60.4 $\pm$  26.1     68.2 $\pm$  6.0     86.0 $\pm$ 1.1
45.0 $\pm$  19.1     0.0 $\pm$  0.0     11.2 $\pm$  4.7
0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

lrg-d     54.4 $\pm$  25.1     55.0 $\pm$  11.7      80.2 $\pm$  2.9
25.0 $\pm$  19.1     0.0 $\pm$  0.0     4.0 $\pm$ 1.8
0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.0

Avg     76.8     77.4     88.8
61.2     16.7     38.9
3.0     6.8     11.6

Average normalized score over the final evaluation and ten unseen training seeds on Adroit tasks.

max width=

1c     3cReBRAC     3cIQL     3cLB-SAC

Task     MSE     CE     CE+MT      MSE     CE     CE+MT     MSE     CE     CE+MT

pen-h     103.5 $\pm$  14.1     91.9 $\pm$  9.3      93.8 $\pm$  15.4
12.1 $\pm$  13.9     110.0 $\pm$  9.2     98.0 $\pm$  3.7
4.5 $\pm$  2.6     7.1 $\pm$  5.6     6.2 $\pm$  8.5

pen-c     91.8 $\pm$  21.7     81.3 $\pm$  14.4      106.1 $\pm$   28.6
23.3 $\pm$  25.0     20.5 $\pm$  18.8     102.4 $\pm$  9.3
26.1 $\pm$  5.3     20.0 $\pm$  4.5     16.6 $\pm$  8.5

pen-e     154.1 $\pm$  5.4     152.4 $\pm$  7.2      158.2 $\pm$  0.8
49.6 $\pm$  41.5     136.5 $\pm$  6.5     150.3 $\pm$  1.0
130.5 $\pm$  16.8     38.0 $\pm$  13.2      55.3 $\pm$  50.6

door-h      0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.0 $\pm$  0.1
6.6 $\pm$  5.5     3.5 $\pm$  0.9     7.8 $\pm$  0.9
-0.2 $\pm$  0.1     -0.2 $\pm$  0.1     -0.2 $\pm$  0.1

door-c     1.1 $\pm$  2.6     0.0 $\pm$  0.0     0.0 $\pm$  0.0
0.1 $\pm$  0.0     0.1 $\pm$  0.0     1.3 $\pm$  1.4
0.0 $\pm$  0.0     0.2 $\pm$  0.5     0.0 $\pm$  0.0

door-e     104.6 $\pm$  2.4     105.7 $\pm$  1.0      106.2 $\pm$ 0.2
103.3 $\pm$  3.0     104.0 $\pm$  2.6     105.8 $\pm$  0.1
95.0 $\pm$  8.6     70.6 $\pm$  33.8     80.6 $\pm$  6.2

ham-h     0.2 $\pm$  0.2     0.1 $\pm$  0.0     2.8 $\pm$  4.1
2.7 $\pm$  2.7     1.2 $\pm$  0.2     3.0 $\pm$  2.8
0.1 $\pm$  0.0     0.0 $\pm$  0.0     0.1 $\pm$  0.0

ham-c     6.7 $\pm$  3.7     11.9 $\pm$  10.0     4.0 $\pm$  3.8
0.2 $\pm$  0.0     0.7 $\pm$  0.5     1.1 $\pm$  0.9
20.2 $\pm$  16.8     13.6 $\pm$  15.4     0.3 $\pm$  0.5

ham-e     133.8 $\pm$  0.7     112.7 $\pm$  35.2      134.3 $\pm$  0.3
129.3 $\pm$  0.5     129.5 $\pm$  0.1      128.9 $\pm$  0.2
76.6 $\pm$  59.5     91.1 $\pm$  10.5     129.3 $\pm$  14.7

rel-h     0.0 $\pm$  0.0     0.0 $\pm$  0.0     0.1 $\pm$  0.3
0.0 $\pm$  0.0     0.1 $\pm$  0.0     0.1 $\pm$  0.1
0.0 $\pm$  0.0     -0.1 $\pm$  0.0     0.0 $\pm$  0.0

rel-c    0.9 $\pm$  1.6     0.6 $\pm$  1.3     0.3 $\pm$  0.3
0.0 $\pm$  0.0     0.1 $\pm$  0.0      0.1 $\pm$  0.0
0.0 $\pm$  0.0     -0.1 $\pm$  0.0      0.0 $\pm$  0.0

rel-e     106.6 $\pm$  3.2     108.6 $\pm$  0.6      106.7 $\pm$  4.3
107.5 $\pm$  2.7     104.6 $\pm$  1.9     109.8 $\pm$  0.5
26.7 $\pm$  18.8     5.3 $\pm$  3.5     0.6 $\pm$  0.2

Avg w/o e     25.5     23.2     25.8
5.6     17.0     26.7
6.3     5.0     2.8

Avg     58.6     55.4     59.3
36.2     50.9     59.0
31.6     20.4     24.0

§ CLASSIFICATION PARAMETERS PERFORMANCE HEATMAPS

§.§ ReBRAC, Gym-MuJoCo

Heatmaps for the impact of the classification parameters on Gym-MuJoCo datasets for ReBRAC.

§.§ ReBRAC, AntMaze

Heatmaps for the impact of the classification parameters on AntMaze datasets for ReBRAC.

§.§ ReBRAC, Adroit

Heatmaps for the impact of the classification parameters on Adroit datasets for ReBRAC.

§.§ IQL, Gym-MuJoCo

Heatmaps for the impact of the classification parameters on Gym-MuJoCo datasets for IQL.

§.§ IQL, AntMaze

Heatmaps for the impact of the classification parameters on AntMaze datasets for IQL.

§.§ IQL, Adroit

Heatmaps for the impact of the classification parameters on Adroit datasets for IQL.

§.§ LB-SAC, Gym-MuJoCo

Heatmaps for the impact of the classification parameters on Gym-MuJoCo datasets for LB-SAC.

§.§ LB-SAC, AntMaze

Heatmaps for the impact of the classification parameters on AntMaze datasets for LB-SAC.

§.§ LB-SAC, Adroit

Heatmaps for the impact of the classification parameters on Adroit datasets for LB-SAC.