# CORL: Research-oriented Deep Offline Reinforcement Learning Library

Denis Tarasov
Tinkoff
<EMAIL_ADDRESS>
&Alexander Nikulin
Tinkoff
<EMAIL_ADDRESS>
&Dmitry Akimov
Tinkoff
<EMAIL_ADDRESS>
&Vladislav Kurenkov
Tinkoff
<EMAIL_ADDRESS>
&Sergey Kolesnikov
Tinkoff
<EMAIL_ADDRESS>

###### Abstract

CORL111CORL Repository: https://github.com/tinkoff-ai/CORL is an open-source
library that provides thoroughly benchmarked single-file implementations of
both deep offline and offline-to-online reinforcement learning algorithms. It
emphasizes a simple developing experience with a straightforward codebase and
a modern analysis tracking tool. In CORL, we isolate methods implementation
into separate single files, making performance-relevant details easier to
recognize. Additionally, an experiment tracking feature is available to help
log metrics, hyperparameters, dependencies, and more to the cloud. Finally, we
have ensured the reliability of the implementations by benchmarking commonly
employed D4RL datasets providing a transparent source of results that can be
reused for robust evaluation tools such as performance profiles, probability
of improvement, or expected online performance.

##  1 Introduction

Deep Offline Reinforcement Learning (Levine et al., 2020) has been showing
significant advancements in numerous domains such as robotics (Smith et al.,
2022; Kumar et al., 2021), autonomous driving (Diehl et al., 2021) and
recommender systems (Chen et al., 2022). Due to such rapid development, many
open-source offline RL solutions222https://github.com/hanjuku-kaso/awesome-
offline-rl##oss emerged to help RL practitioners understand and improve well-
known offline RL techniques in different fields. On the one hand, they
introduce offline RL algorithms standard interfaces and user-friendly APIs,
simplifying offline RL methods incorporation into existing projects. On the
other hand, introduced abstractions may hinder the learning curve for
newcomers and the ease of adoption for researchers interested in developing
new algorithms. One needs to understand the modularity design (several files
on average), which (1) can be comprised of thousands of lines of code or (2)
can hardly fit for a novel
method333https://github.com/takuseno/d3rlpy/issues/141 .

In this technical report, we take a different perspective on an offline RL
library and also incorporate emerging interest in the offline-to-online setup.
We propose CORL (Clean Offline Reinforcement Learning) – minimalistic and
isolated single-file implementations of deep offline and offline-to-online RL
algorithms, supported by open-sourced D4RL (Fu et al., 2020) benchmark
results. The uncomplicated design allows practitioners to read and understand
the implementations of the algorithms straightforwardly. Moreover, CORL
supports optional integration with experiments tracking tools such as
Weighs&Biases (Biewald, 2020), providing practitioners with a convenient way
to analyze the results and behavior of all algorithms, not merely relying on a
final performance commonly reported in papers.

We hope that the CORL library will help offline RL newcomers study implemented
algorithms and aid the researchers in quickly modifying existing methods
without fighting through different levels of abstraction. Finally, the
obtained results may serve as a reference point for D4RL benchmarks avoiding
the need to re-implement and tune existing algorithms’ hyperparameters.

Figure 1: The illustration of the CORL library design. Single-file
implementation takes a yaml configuration file with both environment and
algorithm parameters to run the experiment, which logs all required statistics
to Weights&Biases (Biewald, 2020).

##  2 Related Work

Since the Atari breakthrough (Mnih et al., 2015), numerous open-source RL
frameworks and libraries have been developed over the last years: (Dhariwal et
al., 2017; Hill et al., 2018; Castro et al., 2018; Gauci et al., 2018; Keng &
Graesser, 2017; garage contributors, 2019; Duan et al., 2016; Kolesnikov &
Hrinchuk, 2019; Fujita et al., 2021; Liang et al., 2018; Fujita et al., 2021;
Liu et al., 2021; Huang et al., 2021; Weng et al., 2021; Stooke & Abbeel,
2019), focusing on different perspectives of the RL. For example, stable-
baselines (Hill et al., 2018) provides many deep RL implementations that
carefully reproduce results to back up RL practitioners with reliable
baselines during methods comparison. On the other hand, Ray (Liang et al.,
2018) focuses on implementations scalability and production-friendly usage.
Finally, more nuanced solutions exist, such as Dopamine (Castro et al., 2018),
which emphasizes different DQN variants, or ReAgent (Gauci et al., 2018),
which applies RL to the RecSys domain.

At the same time, the offline RL branch and especially offline-to-online,
which we are interested in this paper, are not yet covered as much: the only
library that precisely focuses on offline RL setting is d3rlpy (Takuma Seno,
2021). While CORL also focuses on offline RL methods (Nair et al., 2020; Kumar
et al., 2020; Kostrikov et al., 2021; Fujimoto & Gu, 2021; An et al., 2021;
Chen et al., 2021), similar to d3rlpy, it takes a different perspective on
library design and provides non-modular independent algorithms
implementations. More precisely, CORL does not introduce additional
abstractions to make offline RL more general but instead gives an "easy-to-
hack" starter kit for research needs. Finally, CORL also provides recent
offline-to-online solutions (Nair et al., 2020; Kumar et al., 2020; Kostrikov
et al., 2021; Wu et al., 2022; Nakamoto et al., 2023) that are gaining
interest among researchers and practitioners.

Although CORL does not represent the first non-modular RL library, which is
more likely the CleanRL (Huang et al., 2021) case, it has two significant
differences from its predecessor. First, CORL is focused on offline and
offline-to-online RL, while CleanRL implements online RL algorithms. Second,
CORL intends to minimize the complexity of the requirements and external
dependencies. To be more concrete, CORL does not have additional requirements
with abstractions such as $stable$-$baselines$ (Hill et al., 2018) or
$envpool$ (Weng et al., 2022) but instead implements everything from scratch
in the codebase.

##  3 CORL Design

### Single-File Implementations

Implementational subtleties significantly impact agent performance in deep RL
(Henderson et al., 2018; Engstrom et al., 2020; Fujimoto & Gu, 2021).
Unfortunately, user-friendly abstractions and general interfaces, the core
idea behind modular libraries, encapsulate and often hide these important
nuances from the practitioners. For such a reason, CORL unwraps these details
by adopting single-file implementations. To be more concrete, we put
environment details, algorithms hyperparameters, and evaluation parameters
into a single file444We follow the PEP8 style guide with a maximum line length
of 89, which increases LOC a bit.. For example, we provide

  * •

$any\\_percent\\_bc.py$ (404 LOC555Lines Of Code) as a baseline algorithm for
offline RL methods comparison,

  * •

$td3\\_bc.py$ (511 LOC) as a competitive minimalistic offline RL algorithm
(Fujimoto & Gu, 2021),

  * •

$dt.py$ (540 LOC) as an example of the recently proposed trajectory
optimization approach (Chen et al., 2021)

Figure 1 depicts an overall library design. To avoid over-complicated offline
implementations, we treat offline and offline-to-online versions of the same
algorithms separately. While such design produces code duplications among
realization, it has several essential benefits from the both educational and
research perspective:

  * •

Smooth learning curve. Having the entire code in one place makes understanding
all its aspects more straightforward. In other words, one may find it easier
to dive into 540 LOC of single-file Decision Transformer (Chen et al., 2021)
implementation rather than 10+ files of the original implementation666Original
Decision Transformer implementation: https://github.com/kzl/decision-
transformer.

  * •

Simple prototyping. As we are not interested in the code’s general
applicability, we could make it implementation-specific. Such a design also
removes the need for inheritance from general primitives or their refactoring,
reducing abstraction overhead to zero. At the same time, this idea gives us
complete freedom during code modification.

  * •

Faster debugging. Without additional abstractions, implementation simplifies
to a single for-loop with a global Python name scope. Furthermore, such flat
architecture makes accessing and inspecting any created variable easier during
training, which is crucial in the presence of modifications and debugging.

### Configuration files

Although it is a typical pattern to use a command line interface (CLI) for
single-file experiments in the research community, CORL slightly improves it
with predefined configuration files. Utilizing YAML parsing through CLI, for
each experiment, we gather all environment and algorithm hyperparameters into
such files so that one can use them as an initial setup. We found that such
setup (1) simplifies experiments, eliminating the need to keep all algorithm-
environment-specific parameters in mind, and (2) keeps it convenient with the
familiar CLI approach.

### Experiment Tracking

Offline RL evaluation is another challenging aspect of the current offline RL
state (Kurenkov & Kolesnikov, 2022). To face this uncertainty, CORL supports
integration with Weights&Biases (Biewald, 2020), a modern experiment tracking
tool. With each experiment, CORL automatically saves (1) source code, (2)
dependencies (requirements.txt), (3) hardware setup, (4) OS environment
variables, (5) hyperparameters, (6) training, and system metrics, (7) logs
(stdout, stderr). See Appendix B for an example.

Although, Weights&Biases is a proprietary solution, other alternatives, such
as Tensorboard (Abadi et al., 2015) or Aim (Arakelyan et al., 2020), could be
used within a few lines of code change. It is also important to note that with
Weights&Biases tracking, one could easily use CORL with sweeps or public
reports.

We found full metrics tracking during the training process necessary for two
reasons. First, it removes the possible bias of the final or best performance
commonly reported in papers. For example, one could evaluate offline RL
performance as max archived score, while another uses the average scores over
$N$ (last) evaluations (Takuma Seno, 2021). Second, it provides an opportunity
for advanced performance analysis such as EOP (Kurenkov & Kolesnikov, 2022) or
RLiable (Agarwal et al., 2021). In short, when provided with all metrics logs,
one can utilize all performance statistics, not merely relying on commonly
used alternatives.

##  4 Benchmarking D4RL

###  4.1 Offline

In our library, we implemented the following offline algorithms: $N\%$777$N$
is a percentage of best trajectories with the highest return used for
training. We omit the percentage when it is equal to $100$. Behavioral Cloning
(BC), TD3 + BC (Fujimoto & Gu, 2021), CQL (Kumar et al., 2020), IQL (Kostrikov
et al., 2021), AWAC (Nair et al., 2020), SAC-N, EDAC (An et al., 2021), and
Decision Transformer (DT) (Chen et al., 2021). We evaluated every algorithm on
the D4RL benchmark (Fu et al., 2020), focusing on Gym-MuJoCo, Maze2D, AntMaze,
and Adroit tasks. Each algorithm was run for one million gradient
steps888Except SAC-$N$, EDAC, and DT due to their original hyperparameters.
See Appendix D for details.  and evaluated using ten episodes for Gym-MuJoCo
and Adroit tasks. For Maze2d, we use 100 evaluation episodes. In our
experiments, we tried to rely on the hyperparameters proposed in the original
works (see Appendix D for details) as much as possible.

The final performance is reported in Table 1 and the maximal performance in
Table 2. The scores are normalized to the range between 0 and 100 (Fu et al.,
2020). Following the recent work by Takuma Seno (2021), we report the last and
best-obtained scores to illustrate each algorithm’s potential performance and
overfitting properties. Figure 2 shows the performance profiles and
probability of improvement of IQL over other algorithms (Agarwal et al.,
2021). See Appendix A for complete training performance graphs.

(a)

(b)

Figure 2: (a) Performance profiles after offline training (b) Probability of
improvement of IQL to other algorithms after offline training. The curves
(Agarwal et al., 2021) are for D4RL benchmark spanning Gym-MuJoCo, Maze2d,
AntMaze, and Adroit datasets. Table 1: Normalized performance of the last
trained policy on D4RL averaged over 4 random seeds.

Task Name BC BC-10% TD3+BC AWAC CQL IQL SAC-$N$ EDAC DT halfcheetah-medium-v2
42.40 $\pm$ 0.19 42.46 $\pm$ 0.70 48.10 $\pm$ 0.18 49.46 $\pm$ 0.62 47.04
$\pm$ 0.22 48.31 $\pm$ 0.22 68.20 $\pm$ 1.28 67.70 $\pm$ 1.04 42.20 $\pm$ 0.26
halfcheetah-medium-replay-v2 35.66 $\pm$ 2.33 23.59 $\pm$ 6.95 44.84 $\pm$
0.59 44.70 $\pm$ 0.69 45.04 $\pm$ 0.27 44.46 $\pm$ 0.22 60.70 $\pm$ 1.01 62.06
$\pm$ 1.10 38.91 $\pm$ 0.50 halfcheetah-medium-expert-v2 55.95 $\pm$ 7.35
90.10 $\pm$ 2.45 90.78 $\pm$ 6.04 93.62 $\pm$ 0.41 95.63 $\pm$ 0.42 94.74
$\pm$ 0.52 98.96 $\pm$ 9.31 104.76 $\pm$ 0.64 91.55 $\pm$ 0.95 hopper-
medium-v2 53.51 $\pm$ 1.76 55.48 $\pm$ 7.30 60.37 $\pm$ 3.49 74.45 $\pm$ 9.14
59.08 $\pm$ 3.77 67.53 $\pm$ 3.78 40.82 $\pm$ 9.91 101.70 $\pm$ 0.28 65.10
$\pm$ 1.61 hopper-medium-replay-v2 29.81 $\pm$ 2.07 70.42 $\pm$ 8.66 64.42
$\pm$ 21.52 96.39 $\pm$ 5.28 95.11 $\pm$ 5.27 97.43 $\pm$ 6.39 100.33 $\pm$
0.78 99.66 $\pm$ 0.81 81.77 $\pm$ 6.87 hopper-medium-expert-v2 52.30 $\pm$
4.01 111.16 $\pm$ 1.03 101.17 $\pm$ 9.07 52.73 $\pm$ 37.47 99.26 $\pm$ 10.91
107.42 $\pm$ 7.80 101.31 $\pm$ 11.63 105.19 $\pm$ 10.08 110.44 $\pm$ 0.33
walker2d-medium-v2 63.23 $\pm$ 16.24 67.34 $\pm$ 5.17 82.71 $\pm$ 4.78 66.53
$\pm$ 26.04 80.75 $\pm$ 3.28 80.91 $\pm$ 3.17 87.47 $\pm$ 0.66 93.36 $\pm$
1.38 67.63 $\pm$ 2.54 walker2d-medium-replay-v2 21.80 $\pm$ 10.15 54.35 $\pm$
6.34 85.62 $\pm$ 4.01 82.20 $\pm$ 1.05 73.09 $\pm$ 13.22 82.15 $\pm$ 3.03
78.99 $\pm$ 0.50 87.10 $\pm$ 2.78 59.86 $\pm$ 2.73 walker2d-medium-expert-v2
98.96 $\pm$ 15.98 108.70 $\pm$ 0.25 110.03 $\pm$ 0.36 49.41 $\pm$ 38.16 109.56
$\pm$ 0.39 111.72 $\pm$ 0.86 114.93 $\pm$ 0.41 114.75 $\pm$ 0.74 107.11 $\pm$
0.96 Gym-MuJoCo avg 50.40 69.29 76.45 67.72 78.28 81.63 83.52 92.92 73.84
maze2d-umaze-v1 0.36 $\pm$ 8.69 12.18 $\pm$ 4.29 29.41 $\pm$ 12.31 82.67 $\pm$
28.30 -8.90 $\pm$ 6.11 42.11 $\pm$ 0.58 130.59 $\pm$ 16.52 95.26 $\pm$ 6.39
18.08 $\pm$ 25.42 maze2d-medium-v1 0.79 $\pm$ 3.25 14.25 $\pm$ 2.33 59.45
$\pm$ 36.25 52.88 $\pm$ 55.12 86.11 $\pm$ 9.68 34.85 $\pm$ 2.72 88.61 $\pm$
18.72 57.04 $\pm$ 3.45 31.71 $\pm$ 26.33 maze2d-large-v1 2.26 $\pm$ 4.39 11.32
$\pm$ 5.10 97.10 $\pm$ 25.41 209.13 $\pm$ 8.19 23.75 $\pm$ 36.70 61.72 $\pm$
3.50 204.76 $\pm$ 1.19 95.60 $\pm$ 22.92 35.66 $\pm$ 28.20 Maze2d avg 1.13
12.58 61.99 114.89 33.65 46.23 141.32 82.64 28.48 antmaze-umaze-v2 55.25 $\pm$
4.15 65.75 $\pm$ 5.26 70.75 $\pm$ 39.18 57.75 $\pm$ 10.28 92.75 $\pm$ 1.92
77.00 $\pm$ 5.52 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 57.00 $\pm$ 9.82 antmaze-
umaze-diverse-v2 47.25 $\pm$ 4.09 44.00 $\pm$ 1.00 44.75 $\pm$ 11.61 58.00
$\pm$ 7.68 37.25 $\pm$ 3.70 54.25 $\pm$ 5.54 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00
51.75 $\pm$ 0.43 antmaze-medium-play-v2 0.00 $\pm$ 0.00 2.00 $\pm$ 0.71 0.25
$\pm$ 0.43 0.00 $\pm$ 0.00 65.75 $\pm$ 11.61 65.75 $\pm$ 11.71 0.00 $\pm$ 0.00
0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 antmaze-medium-diverse-v2 0.75 $\pm$ 0.83 5.75
$\pm$ 9.39 0.25 $\pm$ 0.43 0.00 $\pm$ 0.00 67.25 $\pm$ 3.56 73.75 $\pm$ 5.45
0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 antmaze-large-play-v2 0.00
$\pm$ 0.00 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 20.75 $\pm$ 7.26
42.00 $\pm$ 4.53 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 antmaze-
large-diverse-v2 0.00 $\pm$ 0.00 0.75 $\pm$ 0.83 0.00 $\pm$ 0.00 0.00 $\pm$
0.00 20.50 $\pm$ 13.24 30.25 $\pm$ 3.63 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.00
$\pm$ 0.00 AntMaze avg 17.21 19.71 19.33 19.29 50.71 57.17 0.00 0.00 18.12
pen-human-v1 71.03 $\pm$ 6.26 26.99 $\pm$ 9.60 -3.88 $\pm$ 0.21 81.12 $\pm$
13.47 13.71 $\pm$ 16.98 78.49 $\pm$ 8.21 6.86 $\pm$ 5.93 5.07 $\pm$ 6.16 67.68
$\pm$ 5.48 pen-cloned-v1 51.92 $\pm$ 15.15 46.67 $\pm$ 14.25 5.13 $\pm$ 5.28
89.56 $\pm$ 15.57 1.04 $\pm$ 6.62 83.42 $\pm$ 8.19 31.35 $\pm$ 2.14 12.02
$\pm$ 1.75 64.43 $\pm$ 1.43 pen-expert-v1 109.65 $\pm$ 7.28 114.96 $\pm$ 2.96
122.53 $\pm$ 21.27 160.37 $\pm$ 1.21 -1.41 $\pm$ 2.34 128.05 $\pm$ 9.21 87.11
$\pm$ 48.95 -1.55 $\pm$ 0.81 116.38 $\pm$ 1.27 door-human-v1 2.34 $\pm$ 4.00
-0.13 $\pm$ 0.07 -0.33 $\pm$ 0.01 4.60 $\pm$ 1.90 5.53 $\pm$ 1.31 3.26 $\pm$
1.83 -0.38 $\pm$ 0.00 -0.12 $\pm$ 0.13 4.44 $\pm$ 0.87 door-cloned-v1 -0.09
$\pm$ 0.03 0.29 $\pm$ 0.59 -0.34 $\pm$ 0.01 0.93 $\pm$ 1.66 -0.33 $\pm$ 0.01
3.07 $\pm$ 1.75 -0.33 $\pm$ 0.00 2.66 $\pm$ 2.31 7.64 $\pm$ 3.26 door-
expert-v1 105.35 $\pm$ 0.09 104.04 $\pm$ 1.46 -0.33 $\pm$ 0.01 104.85 $\pm$
0.24 -0.32 $\pm$ 0.02 106.65 $\pm$ 0.25 -0.33 $\pm$ 0.00 106.29 $\pm$ 1.73
104.87 $\pm$ 0.39 hammer-human-v1 3.03 $\pm$ 3.39 -0.19 $\pm$ 0.02 1.02 $\pm$
0.24 3.37 $\pm$ 1.93 0.14 $\pm$ 0.11 1.79 $\pm$ 0.80 0.24 $\pm$ 0.00 0.28
$\pm$ 0.18 1.28 $\pm$ 0.15 hammer-cloned-v1 0.55 $\pm$ 0.16 0.12 $\pm$ 0.08
0.25 $\pm$ 0.01 0.21 $\pm$ 0.24 0.30 $\pm$ 0.01 1.50 $\pm$ 0.69 0.14 $\pm$
0.09 0.19 $\pm$ 0.07 1.82 $\pm$ 0.55 hammer-expert-v1 126.78 $\pm$ 0.64 121.75
$\pm$ 7.67 3.11 $\pm$ 0.03 127.06 $\pm$ 0.29 0.26 $\pm$ 0.01 128.68 $\pm$ 0.33
25.13 $\pm$ 43.25 28.52 $\pm$ 49.00 117.45 $\pm$ 6.65 relocate-human-v1 0.04
$\pm$ 0.03 -0.14 $\pm$ 0.08 -0.29 $\pm$ 0.01 0.05 $\pm$ 0.03 0.06 $\pm$ 0.03
0.12 $\pm$ 0.04 -0.31 $\pm$ 0.01 -0.09 $\pm$ 0.15 0.05 $\pm$ 0.01 relocate-
cloned-v1 -0.06 $\pm$ 0.01 -0.00 $\pm$ 0.02 -0.30 $\pm$ 0.01 -0.04 $\pm$ 0.04
-0.29 $\pm$ 0.01 0.04 $\pm$ 0.01 -0.01 $\pm$ 0.10 0.17 $\pm$ 0.35 0.16 $\pm$
0.09 relocate-expert-v1 107.58 $\pm$ 1.20 97.90 $\pm$ 5.21 -1.73 $\pm$ 0.96
108.87 $\pm$ 0.85 -0.30 $\pm$ 0.02 106.11 $\pm$ 4.02 -0.36 $\pm$ 0.00 71.94
$\pm$ 18.37 104.28 $\pm$ 0.42 Adroit avg 48.18 42.69 10.40 56.75 1.53 53.43
12.43 18.78 49.21 Total avg 37.95 43.06 37.16 58.36 37.61 61.92 44.16 43.65
48.31

Table 2: Normalized performance of the best trained policy on D4RL averaged
over 4 random seeds.

Task Name BC BC-10% TD3+BC AWAC CQL IQL SAC-$N$ EDAC DT halfcheetah-medium-v2
43.60 $\pm$ 0.14 43.90 $\pm$ 0.13 48.93 $\pm$ 0.11 50.06 $\pm$ 0.50 47.62
$\pm$ 0.03 48.84 $\pm$ 0.07 72.21 $\pm$ 0.31 69.72 $\pm$ 0.92 42.73 $\pm$ 0.10
halfcheetah-medium-replay-v2 40.52 $\pm$ 0.19 42.27 $\pm$ 0.46 45.84 $\pm$
0.26 46.35 $\pm$ 0.29 46.43 $\pm$ 0.19 45.35 $\pm$ 0.08 67.29 $\pm$ 0.34 66.55
$\pm$ 1.05 40.31 $\pm$ 0.28 halfcheetah-medium-expert-v2 79.69 $\pm$ 3.10
94.11 $\pm$ 0.22 96.59 $\pm$ 0.87 96.11 $\pm$ 0.37 97.04 $\pm$ 0.17 95.38
$\pm$ 0.17 111.73 $\pm$ 0.47 110.62 $\pm$ 1.04 93.40 $\pm$ 0.21 hopper-
medium-v2 69.04 $\pm$ 2.90 73.84 $\pm$ 0.37 70.44 $\pm$ 1.18 97.90 $\pm$ 0.56
70.80 $\pm$ 1.98 80.46 $\pm$ 3.09 101.79 $\pm$ 0.20 103.26 $\pm$ 0.14 69.42
$\pm$ 3.64 hopper-medium-replay-v2 68.88 $\pm$ 10.33 90.57 $\pm$ 2.07 98.12
$\pm$ 1.16 100.91 $\pm$ 1.50 101.63 $\pm$ 0.55 102.69 $\pm$ 0.96 103.83 $\pm$
0.53 103.28 $\pm$ 0.49 88.74 $\pm$ 3.02 hopper-medium-expert-v2 90.63 $\pm$
10.98 113.13 $\pm$ 0.16 113.22 $\pm$ 0.43 103.82 $\pm$ 12.81 112.84 $\pm$ 0.66
113.18 $\pm$ 0.38 111.24 $\pm$ 0.15 111.80 $\pm$ 0.11 111.18 $\pm$ 0.21
walker2d-medium-v2 80.64 $\pm$ 0.91 82.05 $\pm$ 0.93 86.91 $\pm$ 0.28 83.37
$\pm$ 2.82 84.77 $\pm$ 0.20 87.58 $\pm$ 0.48 90.17 $\pm$ 0.54 95.78 $\pm$ 1.07
74.70 $\pm$ 0.56 walker2d-medium-replay-v2 48.41 $\pm$ 7.61 76.09 $\pm$ 0.40
91.17 $\pm$ 0.72 86.51 $\pm$ 1.15 89.39 $\pm$ 0.88 89.94 $\pm$ 0.93 85.18
$\pm$ 1.63 89.69 $\pm$ 1.39 68.22 $\pm$ 1.20 walker2d-medium-expert-v2 109.95
$\pm$ 0.62 109.90 $\pm$ 0.09 112.21 $\pm$ 0.06 108.28 $\pm$ 9.45 111.63 $\pm$
0.38 113.06 $\pm$ 0.53 116.93 $\pm$ 0.42 116.52 $\pm$ 0.75 108.71 $\pm$ 0.34
Gym-MuJoCo avg 70.15 80.65 84.83 85.92 84.68 86.28 95.60 96.36 77.49
maze2d-umaze-v1 16.09 $\pm$ 0.87 22.49 $\pm$ 1.52 99.33 $\pm$ 16.16 136.61
$\pm$ 11.65 92.05 $\pm$ 13.66 50.92 $\pm$ 4.23 153.12 $\pm$ 6.49 149.88 $\pm$
1.97 63.83 $\pm$ 17.35 maze2d-medium-v1 19.16 $\pm$ 1.24 27.64 $\pm$ 1.87
150.93 $\pm$ 3.89 131.50 $\pm$ 25.38 128.66 $\pm$ 5.44 122.69 $\pm$ 30.00
93.80 $\pm$ 14.66 154.41 $\pm$ 1.58 68.14 $\pm$ 12.25 maze2d-large-v1 20.75
$\pm$ 6.66 41.83 $\pm$ 3.64 197.64 $\pm$ 5.26 227.93 $\pm$ 1.90 157.51 $\pm$
7.32 162.25 $\pm$ 44.18 207.51 $\pm$ 0.96 182.52 $\pm$ 2.68 50.25 $\pm$ 19.34
Maze2d avg 18.67 30.65 149.30 165.35 126.07 111.95 151.48 162.27 60.74
antmaze-umaze-v2 68.50 $\pm$ 2.29 77.50 $\pm$ 1.50 98.50 $\pm$ 0.87 78.75
$\pm$ 6.76 94.75 $\pm$ 0.83 84.00 $\pm$ 4.06 0.00 $\pm$ 0.00 42.50 $\pm$ 28.61
64.50 $\pm$ 2.06 antmaze-umaze-diverse-v2 64.75 $\pm$ 4.32 63.50 $\pm$ 2.18
71.25 $\pm$ 5.76 88.25 $\pm$ 2.17 53.75 $\pm$ 2.05 79.50 $\pm$ 3.35 0.00 $\pm$
0.00 0.00 $\pm$ 0.00 60.50 $\pm$ 2.29 antmaze-medium-play-v2 4.50 $\pm$ 1.12
6.25 $\pm$ 2.38 3.75 $\pm$ 1.30 27.50 $\pm$ 9.39 80.50 $\pm$ 3.35 78.50 $\pm$
3.84 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.75 $\pm$ 0.43 antmaze-medium-diverse-v2
4.75 $\pm$ 1.09 16.50 $\pm$ 5.59 5.50 $\pm$ 1.50 33.25 $\pm$ 16.81 71.00 $\pm$
4.53 83.50 $\pm$ 1.80 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.50 $\pm$ 0.50 antmaze-
large-play-v2 0.50 $\pm$ 0.50 13.50 $\pm$ 9.76 1.25 $\pm$ 0.43 1.00 $\pm$ 0.71
34.75 $\pm$ 5.85 53.50 $\pm$ 2.50 0.00 $\pm$ 0.00 0.00 $\pm$ 0.00 0.00 $\pm$
0.00 antmaze-large-diverse-v2 0.75 $\pm$ 0.43 6.25 $\pm$ 1.79 0.25 $\pm$ 0.43
0.50 $\pm$ 0.50 36.25 $\pm$ 3.34 53.00 $\pm$ 3.00 0.00 $\pm$ 0.00 0.00 $\pm$
0.00 0.00 $\pm$ 0.00 AntMaze avg 23.96 30.58 30.08 38.21 61.83 72.00 0.00 7.08
21.04 pen-human-v1 99.69 $\pm$ 7.45 59.89 $\pm$ 8.03 9.95 $\pm$ 8.19 121.05
$\pm$ 5.47 58.91 $\pm$ 1.81 106.15 $\pm$ 10.28 56.48 $\pm$ 7.17 35.84 $\pm$
10.57 77.83 $\pm$ 2.30 pen-cloned-v1 99.14 $\pm$ 12.27 83.62 $\pm$ 11.75 52.66
$\pm$ 6.33 129.66 $\pm$ 1.27 14.74 $\pm$ 2.31 114.05 $\pm$ 4.78 52.69 $\pm$
5.30 26.90 $\pm$ 7.85 71.17 $\pm$ 2.70 pen-expert-v1 128.77 $\pm$ 5.88 134.36
$\pm$ 3.16 142.83 $\pm$ 7.72 162.69 $\pm$ 0.23 14.86 $\pm$ 4.07 140.01 $\pm$
6.36 116.43 $\pm$ 40.26 36.04 $\pm$ 4.60 119.49 $\pm$ 2.31 door-human-v1 9.41
$\pm$ 4.55 7.00 $\pm$ 6.77 -0.11 $\pm$ 0.06 19.28 $\pm$ 1.46 13.28 $\pm$ 2.77
13.52 $\pm$ 1.22 -0.10 $\pm$ 0.06 2.51 $\pm$ 2.26 7.36 $\pm$ 1.24 door-
cloned-v1 3.40 $\pm$ 0.95 10.37 $\pm$ 4.09 -0.20 $\pm$ 0.11 12.61 $\pm$ 0.60
-0.08 $\pm$ 0.13 9.02 $\pm$ 1.47 -0.21 $\pm$ 0.10 20.36 $\pm$ 1.11 11.18 $\pm$
0.96 door-expert-v1 105.84 $\pm$ 0.23 105.92 $\pm$ 0.24 4.49 $\pm$ 7.39 106.77
$\pm$ 0.24 59.47 $\pm$ 25.04 107.29 $\pm$ 0.37 0.05 $\pm$ 0.02 109.22 $\pm$
0.24 105.49 $\pm$ 0.09 hammer-human-v1 12.61 $\pm$ 4.87 6.23 $\pm$ 4.79 2.38
$\pm$ 0.14 22.03 $\pm$ 8.13 0.30 $\pm$ 0.05 6.86 $\pm$ 2.38 0.25 $\pm$ 0.00
3.49 $\pm$ 2.17 1.68 $\pm$ 0.11 hammer-cloned-v1 8.90 $\pm$ 4.04 8.72 $\pm$
3.28 0.96 $\pm$ 0.30 14.67 $\pm$ 1.94 0.32 $\pm$ 0.03 11.63 $\pm$ 1.70 12.67
$\pm$ 15.02 0.27 $\pm$ 0.01 2.74 $\pm$ 0.22 hammer-expert-v1 127.89 $\pm$ 0.57
128.15 $\pm$ 0.66 33.31 $\pm$ 47.65 129.66 $\pm$ 0.33 0.93 $\pm$ 1.12 129.76
$\pm$ 0.37 91.74 $\pm$ 47.77 69.44 $\pm$ 47.00 127.39 $\pm$ 0.10 relocate-
human-v1 0.59 $\pm$ 0.27 0.16 $\pm$ 0.14 -0.29 $\pm$ 0.01 2.09 $\pm$ 0.76 1.03
$\pm$ 0.20 1.22 $\pm$ 0.28 -0.18 $\pm$ 0.14 0.05 $\pm$ 0.02 0.08 $\pm$ 0.02
relocate-cloned-v1 0.45 $\pm$ 0.31 0.74 $\pm$ 0.45 -0.02 $\pm$ 0.04 0.94 $\pm$
0.68 -0.07 $\pm$ 0.02 1.78 $\pm$ 0.70 0.10 $\pm$ 0.04 4.11 $\pm$ 1.39 0.34
$\pm$ 0.09 relocate-expert-v1 110.31 $\pm$ 0.36 109.77 $\pm$ 0.60 0.23 $\pm$
0.27 111.56 $\pm$ 0.17 0.03 $\pm$ 0.10 110.12 $\pm$ 0.82 -0.07 $\pm$ 0.08
98.32 $\pm$ 3.75 106.49 $\pm$ 0.30 Adroit avg 58.92 54.58 20.51 69.42 13.65
62.62 27.49 33.88 52.60 Total avg 51.27 55.21 54.60 77.72 55.84 76.53 54.82
60.10 54.57

Based on these results, we make several valuable observations. First, IQL and
AWAC are the most competitive baselines in offline setup on average. Note that
AWAC is often omitted in recent works.

Observation 1: IQL and AWAC are the strongest offline baselines on average.

Second, EDAC outperforms all other algorithms on Gym-MuJoCo by a significant
margin, and to our prior knowledge, there are still no algorithms that perform
much better on these tasks. SAC-N shows the best performance on Maze2d tasks.
However, simultaneously, SAC-N and EDAC cannot solve AntMaze tasks and perform
poorly in the Adroit domain.

Observation 2: SAC-N and EDAC are the strongest baselines for Gym-MuJoCo and
Maze2d, but they perform poorly on both AntMaze and Adroit domains.

Third, during our experiments, we observed that the hyperparameters proposed
for CQL in Kumar et al. (2020) do not perform as well as claimed on most
tasks. CQL is extremely sensitive to the choice of hyperparameters, and we had
to tune them a lot to make it work on each domain (see Table 7). For example,
AntMaze requires five hidden layers for the critic networks, while other
tasks’ performance suffers with this number of layers. The issue of
sensitivity999See also https://github.com/aviralkumar2907/CQL/issues/9,
https://github.com/tinkoff-ai/CORL/issues/14 and https://github.com/young-
geng/CQL/issues/5 was already mentioned in prior works as well (An et al.,
2021; Ghasemipour et al., 2022).

Observation 3: CQL is extremely sensitive to the choice of hyperparameters and
implementation details.

Fourth, we also observe that the hyperparameters do not always work the same
way when transferring between Deep Learning frameworks
101010https://github.com/tinkoff-ai/CORL/issues/33. Our implementations of IQL
and CQL use PyTorch, but the parameters from reference JAX implementations
sometimes strongly underperform (e.g., IQL on Hopper tasks and CQL on Adroit).

Observation 4: Hyperparameters are not always transferable between Deep
Learning frameworks.

###  4.2 Offline-to-Online

Table 3: Normalized performance of algorithms after offline pretraining and
online finetuning on D4RL averaged over 4 random seeds.

Task Name AWAC CQL IQL SPOT Cal-QL antmaze-umaze-v2 52.75 $\pm$ 8.67 $\to$
98.75 $\pm$ 1.09 94.00 $\pm$ 1.58 $\to$ 99.50 $\pm$ 0.87 77.00 $\pm$ 0.71
$\to$ 96.50 $\pm$ 1.12 91.00 $\pm$ 2.55 $\to$ 99.50 $\pm$ 0.50 65.75 $\pm$
3.49 $\to$ 99.75 $\pm$ 0.43 antmaze-umaze-diverse-v2 56.00 $\pm$ 2.74 $\to$
0.00 $\pm$ 0.00 9.50 $\pm$ 9.91 $\to$ 99.00 $\pm$ 1.22 59.50 $\pm$ 9.55 $\to$
63.75 $\pm$ 25.02 36.25 $\pm$ 2.17 $\to$ 95.00 $\pm$ 3.67 48.75 $\pm$ 3.83
$\to$ 97.50 $\pm$ 3.77 antmaze-medium-play-v2 0.00 $\pm$ 0.00 $\to$ 0.00 $\pm$
0.00 59.00 $\pm$ 11.18 $\to$ 97.75 $\pm$ 1.30 71.75 $\pm$ 2.95 $\to$ 89.75
$\pm$ 1.09 67.25 $\pm$ 10.47 $\to$ 97.25 $\pm$ 1.30 0.00 $\pm$ 0.00 $\to$
98.50 $\pm$ 0.87 antmaze-medium-diverse-v2 0.00 $\pm$ 0.00 $\to$ 0.00 $\pm$
0.00 63.50 $\pm$ 6.84 $\to$ 97.25 $\pm$ 1.92 64.25 $\pm$ 1.92 $\to$ 92.25
$\pm$ 2.86 73.75 $\pm$ 7.29 $\to$ 94.50 $\pm$ 1.66 1.25 $\pm$ 0.83 $\to$ 96.25
$\pm$ 3.63 antmaze-large-play-v2 0.00 $\pm$ 0.00 $\to$ 0.00 $\pm$ 0.00 28.75
$\pm$ 7.76 $\to$ 88.25 $\pm$ 2.28 38.50 $\pm$ 8.73 $\to$ 64.50 $\pm$ 17.04
31.50 $\pm$ 12.58 $\to$ 87.00 $\pm$ 3.24 0.25 $\pm$ 0.43 $\to$ 92.25 $\pm$
3.70 antmaze-large-diverse-v2 0.00 $\pm$ 0.00 $\to$ 0.00 $\pm$ 0.00 35.50
$\pm$ 3.64 $\to$ 91.75 $\pm$ 3.96 26.75 $\pm$ 3.77 $\to$ 64.25 $\pm$ 4.15
17.50 $\pm$ 7.26 $\to$ 81.00 $\pm$ 14.14 0.00 $\pm$ 0.00 $\to$ 89.75 $\pm$
2.59 AntMaze avg 18.12 $\to$ 16.46 (-1.66) 48.38 $\to$ 95.58 (+47.20) 56.29
$\to$ 78.50 (+22.21) 52.88 $\to$ 92.38 (+39.50) 19.33 $\to$ 95.67 (+76.34)
pen-cloned-v1 88.66 $\pm$ 15.10 $\to$ 86.82 $\pm$ 11.12 -2.76 $\pm$ 0.08 $\to$
-1.28 $\pm$ 2.16 84.19 $\pm$ 3.96 $\to$ 102.02 $\pm$ 20.75 6.19 $\pm$ 5.21
$\to$ 43.63 $\pm$ 20.09 -2.64 $\pm$ 0.14 $\to$ 0.04 $\pm$ 3.95 door-cloned-v1
0.93 $\pm$ 1.66 $\to$ 0.01 $\pm$ 0.00 -0.33 $\pm$ 0.01 $\to$ -0.33 $\pm$ 0.01
1.19 $\pm$ 0.93 $\to$ 20.34 $\pm$ 9.32 -0.21 $\pm$ 0.14 $\to$ 0.02 $\pm$ 0.31
-0.33 $\pm$ 0.01 $\to$ -0.33 $\pm$ 0.01 hammer-cloned-v1 1.80 $\pm$ 3.01 $\to$
0.24 $\pm$ 0.04 0.56 $\pm$ 0.55 $\to$ 2.85 $\pm$ 4.81 1.35 $\pm$ 0.32 $\to$
57.27 $\pm$ 28.49 3.97 $\pm$ 6.39 $\to$ 3.73 $\pm$ 4.99 0.27 $\pm$ 0.01 $\to$
0.14 $\pm$ 0.15 relocate-cloned-v1 -0.04 $\pm$ 0.04 $\to$ -0.04 $\pm$ 0.01
-0.33 $\pm$ 0.01 $\to$ -0.33 $\pm$ 0.01 0.04 $\pm$ 0.04 $\to$ 0.32 $\pm$ 0.38
-0.24 $\pm$ 0.01 $\to$ -0.15 $\pm$ 0.05 -0.33 $\pm$ 0.01 $\to$ -0.33 $\pm$
0.00 Adroit Avg 22.84 $\to$ 21.76 (-1.08) -0.72 $\to$ 0.22 (+0.94) 21.69 $\to$
44.99 (+23.3) 2.43 $\to$ 11.81 (+9.38) -0.76 $\to$ -0.12 (+0.64) Total avg
20.01 $\to$ 18.58 (-1.43) 28.74 $\to$ 57.44 (+28.7) 42.45 $\to$ 65.10 (+22.65)
32.70 $\to$ 60.15 (+27.45) 11.30 $\to$ 57.35 (+46.05)

Table 4: Cumulative regret of online finetuning calculated as
$1-\text{{average success rate}}$ averaged over 4 random seeds.

Task Name AWAC CQL IQL SPOT Cal-QL antmaze-umaze-v2 0.04 $\pm$ 0.01 0.02 $\pm$
0.00 0.07 $\pm$ 0.00 0.02 $\pm$ 0.00 0.02 $\pm$ 0.00 antmaze-umaze-diverse-v2
0.88 $\pm$ 0.01 0.09 $\pm$ 0.01 0.43 $\pm$ 0.11 0.22 $\pm$ 0.07 0.04 $\pm$
0.01 antmaze-medium-play-v2 1.00 $\pm$ 0.00 0.08 $\pm$ 0.01 0.09 $\pm$ 0.01
0.06 $\pm$ 0.00 0.08 $\pm$ 0.01 antmaze-medium-diverse-v2 1.00 $\pm$ 0.00 0.08
$\pm$ 0.00 0.10 $\pm$ 0.01 0.05 $\pm$ 0.01 0.08 $\pm$ 0.01 antmaze-large-
play-v2 1.00 $\pm$ 0.00 0.21 $\pm$ 0.02 0.34 $\pm$ 0.05 0.29 $\pm$ 0.07 0.29
$\pm$ 0.04 antmaze-large-diverse-v2 1.00 $\pm$ 0.00 0.21 $\pm$ 0.03 0.41 $\pm$
0.03 0.23 $\pm$ 0.08 0.29 $\pm$ 0.06 AntMaze avg 0.82 0.11 0.24 0.15 0.13 pen-
cloned-v1 0.46 $\pm$ 0.02 0.97 $\pm$ 0.00 0.37 $\pm$ 0.01 0.58 $\pm$ 0.02 0.97
$\pm$ 0.01 door-cloned-v1 1.00 $\pm$ 0.00 1.00 $\pm$ 0.00 0.83 $\pm$ 0.03 0.99
$\pm$ 0.01 1.00 $\pm$ 0.00 hammer-cloned-v1 1.00 $\pm$ 0.00 1.00 $\pm$ 0.00
0.65 $\pm$ 0.10 0.98 $\pm$ 0.01 1.00 $\pm$ 0.00 relocate-cloned-v1 1.00 $\pm$
0.00 1.00 $\pm$ 0.00 1.00 $\pm$ 0.00 1.00 $\pm$ 0.00 1.00 $\pm$ 0.00 Adroit
avg 0.86 0.99 0.71 0.89 0.99 Total avg 0.84 0.47 0.43 0.44 0.48

We also implement the following algorithms in offline-to-online setup: CQL
(Kumar et al., 2020), IQL (Kostrikov et al., 2021), AWAC (Nair et al., 2020),
SPOT (Wu et al., 2022) and Cal-QL (Nakamoto et al., 2023). Inspired by
Nakamoto et al. (2023), we evaluate algorithms on AntMaze and Adroit Cloned
datasets111111Note, Nakamoto et al. (2023) used modified Cloned datasets while
we employ original data from D4RL because these datasets are more common to
for benchmarking.. Each algorithm is trained offline over 1 million steps and
tuned using online transitions over another 1 million steps. The AntMaze tasks
are evaluated using 100 episodes, while the Adroit tasks are tested with ten
episodes.

The scores, normalized after the offline stage and online tuning, are reported
in Table 3. We also provide finetuning cumulative regret proposed by Nakamoto
et al. (2023) in Table 4. Cumulative regret is calculated as
$(1-\text{{average success rate})}$121212As specified by the authors:
https://github.com/nakamotoo/Cal-QL/issues/1. It is bounded between 0 and 1,
indicating the range of possible values. Lower values of cumulative regret
indicate better algorithm efficiency. The performance profiles and probability
of improvement of Cal-QL over other algorithms after online finetuning are
presented in Figure 3.

(a)

(b)

Figure 3: (a) Performance profiles after online tuning (b) Probability of
improvement of Cal-QL to other algorithms after online tuning. The curves
(Agarwal et al., 2021) are for D4RL benchmark spanning AntMaze and Adroit
cloned datasets.

AWAC, initially proposed for finetuning purposes, appeared to be the worst of
the considered algorithms, where the score is improved only on the most
straightforward antmaze-umaze-v2 dataset. At the same time, on other datasets,
performances either stay the same or even drop.

Observation 5: AWAC does not benefit from online tuning on the considered
tasks.

Cal-QL was proposed as a modification of CQL, which is expected to work better
in offline-to-online setting. However, in our experiments, CQL obtained the
same score (which is the best among all algorithms) after finetuning, and it
even has a better regret than Cal-QL on AntMaze. At the same time, we could
not make both algorithms solve Adroit tasks131313The issues are Observations 3
and 4. Additional hyperparameters search is needed..

Observation 6: There is no big difference between CQL and Cal-QL. On AntMaze,
these algorithms perform the best but work poorly on Adroit.

IQL starts with better offline scores on AntMaze, but it is less efficient in
finetuning than other algorithms except for AWAC. At the same time, IQL is the
only algorithm that notably improves its scores after tuning on Adroit tasks,
making it a very competitive offline-to-online baseline considering the
average score.

Observation 7: Considering offline and offline-to-online results, IQL appears
to be the strongest baseline on average.

##  5 Conclusion

This paper introduced CORL, a single-file implementation library for offline
and offline-to-online reinforcement learning algorithms with configuration
files and advanced metrics tracking support. In total, we provided
implementations of nine offline and five offline-to-online algorithms. All
implemented approaches were benchmarked on D4RL datasets, closely matching
(sometimes overperforming) the reference results, if available. Focusing on
implementation clarity and reproducibility, we hope that CORL will help RL
practitioners in their research and applications.

This study’s benchmarking results and observations are intended to serve as
references for future offline reinforcement learning research and its
practical applications. By sharing comprehensive logs, researchers can readily
access and utilize our results without having to re-run any of our
experiments, ensuring that the results are replicable.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Agarwal et al. (2021) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare.  Deep reinforcement learning at the edge of the statistical precipice.  _Advances in Neural Information Processing Systems_ , 2021.
  * An et al. (2021) Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song.  Uncertainty-based offline reinforcement learning with diversified q-ensemble.  _Advances in neural information processing systems_ , 34:7436–7447, 2021.
  * Arakelyan et al. (2020) Gor Arakelyan, Gevorg Soghomonyan, and The Aim team.  Aim, 6 2020.  URL https://github.com/aimhubio/aim.
  * Biewald (2020) Lukas Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Castro et al. (2018) Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare.  Dopamine: A research framework for deep reinforcement learning.  _arXiv preprint arXiv:1812.06110_ , 2018.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Chen et al. (2022) Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed H. Chi.  Off-policy actor-critic for recommender systems.  _Proceedings of the 16th ACM Conference on Recommender Systems_ , 2022\.
  * Dhariwal et al. (2017) Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov.  Openai baselines.  https://github.com/openai/baselines, 2017.
  * Diehl et al. (2021) Christopher P. Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram.  Umbrella: Uncertainty-aware model-based offline reinforcement learning leveraging planning.  _ArXiv_ , abs/2111.11097, 2021.
  * Duan et al. (2016) Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel.  Benchmarking deep reinforcement learning for continuous control.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, pp. 1329–1338. JMLR.org, 2016.
  * Engstrom et al. (2020) Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, L. Rudolph, and Aleksander Madry.  Implementation matters in deep rl: A case study on ppo and trpo.  In _ICLR_ , 2020.
  * Fu et al. (2020) Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Fujimoto & Gu (2021) Scott Fujimoto and Shixiang Shane Gu.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ , 34:20132–20145, 2021.
  * Fujita et al. (2021) Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa.  Chainerrl: A deep reinforcement learning library.  _Journal of Machine Learning Research_ , 22(77):1–14, 2021.  URL http://jmlr.org/papers/v22/20-376.html.
  * garage contributors (2019) The garage contributors.  Garage: A toolkit for reproducible reinforcement learning research.  https://github.com/rlworkgroup/garage, 2019.
  * Gauci et al. (2018) Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Zhengxing Chen, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye.  Horizon: Facebook’s open source applied reinforcement learning platform.  _arXiv preprint arXiv:1811.00260_ , 2018.
  * Ghasemipour et al. (2022) Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum.  Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters.  _Advances in Neural Information Processing Systems_ , 35:18267–18281, 2022.
  * Henderson et al. (2018) Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence_ , AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018.  ISBN 978-1-57735-800-8.
  * Hill et al. (2018) Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu.  Stable baselines.  https://github.com/hill-a/stable-baselines, 2018.
  * Huang et al. (2021) Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga.  Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.  _arXiv preprint arXiv:2111.08819_ , 2021.
  * Keng & Graesser (2017) Wah Loon Keng and Laura Graesser.  Slm lab.  https://github.com/kengz/SLM-Lab, 2017.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kolesnikov & Hrinchuk (2019) Sergey Kolesnikov and Oleksii Hrinchuk.  Catalyst.rl: A distributed framework for reproducible rl research, 2019\.  URL https://arxiv.org/abs/1903.00027.
  * Kostrikov et al. (2021) Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021.
  * Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Kumar et al. (2021) Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine.  A workflow for offline model-free robotic reinforcement learning.  In _5th Annual Conference on Robot Learning_ , 2021.  URL https://openreview.net/forum?id=fy4ZBWxYbIo.
  * Kurenkov & Kolesnikov (2022) Vladislav Kurenkov and Sergey Kolesnikov.  Showing your offline reinforcement learning work: Online evaluation budget matters.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 11729–11752. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/kurenkov22a.html.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Liang et al. (2018) Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica.  RLlib: Abstractions for distributed reinforcement learning.  In _International Conference on Machine Learning (ICML)_ , 2018\.
  * Liu et al. (2021) Xiao-Yang Liu, Zechu Li, Zhaoran Wang, and Jiahao Zheng.  ElegantRL: Massively parallel framework for cloud-native deep reinforcement learning.  https://github.com/AI4Finance-Foundation/ElegantRL, 2021.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, February 2015.  ISSN 1476-4687.  doi: 10.1038/nature14236.
  * Nair et al. (2020) Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine.  Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Nakamoto et al. (2023) Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine.  Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.  _arXiv preprint arXiv:2303.05479_ , 2023.
  * Smith et al. (2022) Laura Smith, Ilya Kostrikov, and Sergey Levine.  A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning, August 2022.
  * Stooke & Abbeel (2019) Adam Stooke and Pieter Abbeel.  rlpyt: A research code base for deep reinforcement learning in pytorch, 2019.  URL https://arxiv.org/abs/1909.01500.
  * Takuma Seno (2021) Michita Imai Takuma Seno.  d3rlpy: An offline deep reinforcement library.  In _NeurIPS 2021 Offline Reinforcement Learning Workshop_ , December 2021.
  * Weng et al. (2021) Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu.  Tianshou: A highly modularized deep reinforcement learning library.  _arXiv preprint arXiv:2107.14171_ , 2021.
  * Weng et al. (2022) Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan.  EnvPool: A highly parallel reinforcement learning environment execution engine.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 22409–22421. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/8caaf08e49ddbad6694fae067442ee21-Paper-Datasets_and_Benchmarks.pdf.
  * Wu et al. (2022) Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long.  Supported policy optimization for offline reinforcement learning.  _arXiv preprint arXiv:2202.06239_ , 2022.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] See section 3

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We release our codebase, configs, and in-depth reports at
https://github.com/tinkoff-ai/CORL

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Appendix D

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
D

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [Yes] See Appendix C

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [N/A]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Additional Benchmark Information

###  A.1 Offline

(a)

(b)

(c)

(d)

Figure 4: Graphical representation of the normalized performance of the last
trained policy on D4RL averaged over 4 random seeds. (a) Gym-MuJoCo datasets.
(b) Maze2d datasets (c) AntMaze datasets (d) Adroit datasets

(a)

(b)

(c)

(d)

Figure 5: Graphical representation of the normalized performance of the best
trained policy on D4RL averaged over 4 random seeds. (a) Gym-MuJoCo datasets.
(b) Maze2d datasets (c) AntMaze datasets (d) Adroit datasets

(a)

(b)

(c)

Figure 6: Training curves for HalfCheetah task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset

(a)

(b)

(c)

Figure 7: Training curves for Hopper task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset

(a)

(b)

(c)

Figure 8: Training curves for Walker2d task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset

(a)

(b)

(c)

Figure 9: Training curves for Maze2d task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset

(a)

(b)

(c)

(d)

(e)

(f)

Figure 10: Training curves for AntMaze task.
(a) Umaze dataset, (b) Medium-play dataset, (c) Large-play dataset, (d) Umaze-
diverse dataset, (e) Medium-diverse dataset, (f) Large-diverse dataset

(a)

(b)

(c)

Figure 11: Training curves for Pen task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset

(a)

(b)

(c)

Figure 12: Training curves for Door task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset

(a)

(b)

(c)

Figure 13: Training curves for Hammer task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset

(a)

(b)

(c)

Figure 14: Training curves for Relocate task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset

###  A.2 Offline-to-online

(a)

(b)

Figure 15: Graphical representation of the normalized performance of the last
trained policy on D4RL after online tuning averaged over 4 random seeds.
(a) AntMaze datasets (b) Adroit datasets

(a)

(b)

(c)

(d)

(e)

(f)

Figure 16: Training curves for AntMaze task during online tuning.
(a) Umaze dataset, (b) Medium-play dataset, (c) Large-play dataset, (d) Umaze-
diverse dataset, (e) Medium-diverse dataset, (f) Large-diverse dataset

(a)

(b)

(c)

(d)

Figure 17: Training curves for Adroit Cloned task during online tuning.
(a) Pen, (b) Door, (c) Hammer, (d) Relocate

##  Appendix B Weights&Biases Tracking

Figure 18: Screenshots of Weights&Biases experiment tracking interface.

##  Appendix C License

Our codebase is released under Apache License 2.0. The D4RL datasets (Fu et
al., 2020) are released under Apache License 2.0.

##  Appendix D Experimental Details

We modify reward on AntMaze task by subtracting $1$ from reward as it is done
in previous works except CQL and Cal-QL, where (0, 1) are mapped into (-5, 5).

We used original implementation of TD3 +
BC141414https://github.com/sfujim/TD3_BC,
SAC-$N$/EDAC151515https://github.com/snu-mllab/EDAC,
SPOT161616https://github.com/thuml/SPOT and custom implementations of
IQL171717https://github.com/gwthomas/IQL-PyTorch and CQL/Cal-
QL181818https://github.com/young-geng/CQL as the basis for ours.

For most of the algorithms and datasets, we use default hyperparameters if
available. Configuration files for every algorithm and environment are
presented in our GitHub repository. Hyperparameters are also provided in
subsection D.2.

All the experiments ran using V100 and A100 GPUs, which took approximately
5000 hours of compute in total.

###  D.1 Number of update steps and evaluation rate

Following original work, SAC-$N$ and EDAC are trained for 3 million steps
(except AntMaze, which is trained for 1 million steps) in order to obtain
state-of-the-art performance and tested every 10000 steps. Decision
Transformer (DT) training is splitted into datasets pass epochs. We train DT
for 50 epochs on each dataset and evaluate every 5 epochs. All other
algorithms are trained for 1 million steps and evaluated every 5000 steps
(50000 for AntMaze). We evaluate every policy for 10 episodes on Gym-MuJoCo
and Adroit tasks and for 100 for Maze2d and AntMaze tasks.

###  D.2 Hyperparameters

Table 5: BC and BC-$N\%$ hyperparameters. $\dagger$ used for the best
trajectories choice.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
BC hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Learning Rate | 3e-4
Mini-batch size | 256
|  |
Architecture | Policy hidden dim | 256
Policy hidden layers | 2
Policy activation function | ReLU
|  |
BC-$N\%$ hyperparameters | Ratio of best trajectories used | 0.1
Discount factor† | 1.0
Max trajectory length† | 1000
|  |
|  |

Table 6: TD3+BC hyperparameters.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
TD3 hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Critic learning rate | 3e-4
Actor learning rate | 3e-4
Mini-batch size | 256
Discount factor | 0.99
Target update rate | 5e-3
Policy noise | 0.2
Policy noise clipping | (-0.5, 0.5)
Policy update frequency | 2
|  |
Architecture | Critic hidden dim | 256
Critic hidden layers | 2
Critic activation function | ReLU
Actor hidden dim | 256
Actor hidden layers | 2
Actor activation function | ReLU
|  |
TD3+BC hyperparameters | $\alpha$ | 2.5
|  |
|  |

Table 7: CQL and Cal-QL hyperparameters. Note: used hyperparameters are
suboptimal on Adroit for the implementation we provide.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
SAC hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Critic learning rate | 3e-4
Actor learning rate | 1e-4
Mini-batch size | 256
Discount factor | 0.99
Target update rate | 5e-3
Target entropy | -1 $\cdot$ Action Dim
| Entropy in Q target | False
|  |
Architecture | Critic hidden dim | 256
Critic hidden layers | 5, AntMaze
| 3, otherwise
Critic activation function | ReLU
Actor hidden dim | 256
Actor hidden layers | 3
Actor activation function | ReLU
|  |
CQL hyperparameters | Lagrange | True, Maze2d and AntMaze
| False, otherwise
Offline $\alpha$ | 1.0, Adroit
| 5.0, AntMaze
| 10.0, otherwise
Lagrange gap | 5, Maze2d
| 0.8, AntMaze
Pre-training steps | 0
Num sampled actions (during eval) | 10
Num sampled actions (logsumexp) | 10
|  |
Cal-QL hyperparameters | Mixing ratio | 0.5
Online $\alpha$ | 1.0, Adroit
| 5.0, AntMaze
|  |
|  |

Table 8: IQL hyperparameters.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
IQL hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Critic learning rate | 3e-4
Actor learning rate | 3e-4
Value learning rate | 3e-4
Mini-batch size | 256
Discount factor | 0.99
Target update rate | 5e-3
Learning rate decay | Cosine
Deterministic policy | True, Hopper Medium and Medium-replay
| False, otherwise
$\beta$ | 6.0, Hopper Medium-expert
| 10.0, AntMaze
| 3.0, otherwise
$\tau$ | 0.9, AntMaze
| 0.5, Hopper Medium-expert
| 0.7, otherwise
|  |
Architecture | Critic hidden dim | 256
Critic hidden layers | 2
Critic activation function | ReLU
Actor hidden dim | 256
Actor hidden layers | 2
Actor activation function | ReLU
Value hidden dim | 256
Value hidden layers | 2
Value activation function | ReLU
|  |
|  |

Table 9: AWAC hyperparameters.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
AWAC hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Critic learning rate | 3e-4
Actor learning rate | 3e-4
Mini-batch size | 256
Discount factor | 0.99
Target update rate | 5e-3
$\lambda$ | 0.1, Maze2d, AntMaze
| 0.3333, otherwise
|  |
Architecture | Critic hidden dim | 256
Critic hidden layers | 2
Critic activation function | ReLU
Actor hidden dim | 256
Actor hidden layers | 2
Actor activation function | ReLU
|  |
|  |

Table 10: SAC-$N$ and EDAC hyperparameters.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
SAC hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Critic learning rate | 3e-4
Actor learning rate | 3e-4
$\alpha$ learning rate | 3e-4
Mini-batch size | 256
Discount factor | 0.99
Target update rate | 5e-3
Target entropy | -1 $\cdot$ Action Dim
|  |
Architecture | Critic hidden dim | 256
Critic hidden layers | 3
Critic activation function | ReLU
Actor hidden dim | 256
Actor hidden layers | 3
Actor activation function | ReLU
|  |
SAC-N hyperparameters | Number of critics | 10, HalfCheetah
| 20, Walker2d
| 25, AntMaze
| 200, Hopper Medium-expert, Medium-replay
| 500, Hopper Medium
|  |
EDAC hyperparameters | Number of critics | 10, HalfCheetah
| 10, Walker2d, AntMaze
| 50, Hopper
$\mu$ | 5.0, HalfCheetah Medium-expert, Walker2d Medium-expert
| 1.0, otherwise
|  |
|  |

Table 11: DT hyperparameters.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
DT hyperparameters | Optimizer | AdamW (Loshchilov & Hutter, 2017)
Batch size |  $256$, AntMaze
|  $4096$, otherwise
Return-to-go conditioning | (12000, 6000), HalfCheetah
| (3600, 1800), Hopper
| (5000, 2500), Walker2d
| (160, 80), Maze2d umaze
| (280, 140), Maze2d medium and large
| (1, 0.5), AntMaze
| (3100, 1550), Pen
| (2900, 1450), Door
| (12800, 6400), Hammer
| (4300, 2150), Relocate
Reward scale | 1.0, AntMaze
| 0.001, otherwise
Dropout | 0.1
Learning rate | 0.0008
Adam betas | (0.9, 0.999)
Clip grad norm | 0.25
Weight decay | 0.0003
| Total gradient steps | 100000
| Linear warmup steps | 10000
|  |
Architecture | Number of layers | 3
Number of attention heads | 1
Embedding dimension | 128
Activation function | GELU
|  |
|  |

Table 12: SPOT hyperparameters.

|  |
---|---|---
|  |
| Hyperparameter | Value
|  |
VAE hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Learning rate | 1e-3
Mini-batch size | 256
Number of iterations | $10^{5}$
KL term weight | 0.5
VAE architecture | Encoder hidden dim | 750
Encoder layers | 3
Latent dim | 2 $\times$ action dim
Decoder hidden dim | 750
Decoder layers | 3
TD3 hyperparameters | Optimizer | Adam (Kingma & Ba, 2014)
Critic learning rate | 3e-4
Actor learning rate | 1e-4
Mini-batch size | 256
Discount factor | 0.99
Target update rate | 5e-3
Policy noise | 0.2
Policy noise clipping | (-0.5, 0.5)
Policy update frequency | 2
|  |
Architecture | Critic hidden dim | 256
Critic hidden layers | 2
Critic activation function | ReLU
Actor hidden dim | 256
Actor hidden layers | 2
Actor activation function | ReLU
|  |
SPOT hyperparameters | $\lambda$ | 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, AntMaze
| 1.0, Adroit
|  |
|  |


# Structure-Aware Residual-Center Representation for
Self-Supervised Open-Set 3D Cross-Modal Retrieval

###### Abstract

Existing methods of 3D cross-modal retrieval heavily lean on category
distribution priors within the training set, which diminishes their efficacy
when tasked with unseen categories under open-set environments. To tackle this
problem, we propose the Structure-Aware Residual-Center Representation (SRCR)
framework for self-supervised open-set 3D cross-modal retrieval. To address
the center deviation due to category distribution differences, we utilize the
Residual-Center Embedding (RCE) for each object by nested auto-encoders,
rather than directly mapping them to the modality or category centers.
Besides, we perform the Hierarchical Structure Learning (HSL) approach to
leverage the high-order correlations among objects for generalization, by
constructing a heterogeneous hypergraph structure based on hierarchical inter-
modality, intra-object, and implicit-category correlations. Extensive
experiments and ablation studies on four benchmarks demonstrate the
superiority of our proposed framework compared to state-of-the-art methods.

Index Terms—  3D Object Retrieval, Cross-Modal Retrieval, Open-Set Learning,
Self-Supervised Learning, Hypergraph

##  1 Introduction

The proliferation of multimedia data on the Internet, including videos,
images, text, and more, has sparked growing interest within the community in
the field of cross-modal retrieval tasks. Among them, 3D cross-modal retrieval
(3DCMR) has garnered growing attention due to the inherent diverse modalities
of 3D data [1] and its relevance across crucial domains such as robotics,
medicine, and other significant fields.

Typical 3D cross-modal retrieval task aims to retrieve 3D data from one
modality given queries from different modalities. To address the heterogeneity
gap [2] from different modalities, a widely adopted strategy of 3D cross-modal
retrieval methods is to seek a function that maps data samples from diverse
modalities into a unified global representation space [1, 3], which is called
the center.

Fig. 1: Illustration of the proposed SRCR. Given 3D objects of unseen
categories represented by different modalities, our method generates the
residual-center embeddings for each modality of each object. Then unified
center representations are generated via hierarchical structure learning for
cross-modal retrieval with unseen categories generalization.

Current methods of constructing such mapping can be broadly categorized into
two approaches. One straightforward solution for this task is to construct
complex nonlinear transformations [4, 5] that map two types of pre-trained
features into a shared space. The alternative approach employs adversarial
loss to learn category-related central embedding [1, 3] through end-to-end
training. However, both methods exhibit a pronounced dependence on the prior
distribution of category spaces within the training sets, which leads to
substantial representational biases when confronted with objects of unseen
categories. Furthermore, dependency on training labels in adversarial loss
also complicates the deployment of 3D cross-modal retrieval.

To overcome the aforementioned challenges, we propose the Structure-Aware
Residual-Center Representation (SRCR) framework for the self-supervised open-
set 3D cross-modal retrieval task, as shown in Fig. 1. On one hand, to
overcome the center deviation due to category distribution differences, we
utilize the residual-center embedding for each object by nested auto-encoders,
rather than directly mapping them to the modality or category center. On the
other hand, we perform a hierarchical structure learning approach to utilize
the high-order correlations among objects for generalization, by constructing
a heterogeneous hypergraph structure based on hierarchical intra-modality,
inter-object, and implicit-category correlations. Our contributions are
summarized as follows:

  * •

We introduce a practical open-set setting for 3D cross-modal retrieval and
generate four datasets for benchmarking of downstream 3D cross-modal tasks.

  * •

We propose the Structure-Aware Residual-Center Representation (SRCR) framework
for the open-set 3D cross-modal retrieval, including the Residual-Center
Embedding (RCE) and Hierarchical Structure Learning (HSL) modules, which are
designed to overcome the modality diversion accentuated by unseen categories
distribution.

  * •

We propose a hierarchical hypergraph structure to capture the high-order
correlations among objects, under the guidance of hierarchical inter-modality,
intra-object, and implicit-category correlations.

  * •

The proposed framework significantly outperforms the state-of-the-art 3D
cross-modal retrieval methods under the open-set setting.

##  2 Related Work

###  2.1 Cross-Modal Retrieval

Existing methods usually construct a mapping function into a unified common
space to overcome the heterogeneity gap [2] from different modalities. These
approaches could be roughly classified into projection-based [4] and
discrimination-based [1, 3] methods. While such methods excel under the
closed-set assumption, their reliance on training category distribution limits
their generalization in real-world, open-set environments.

###  2.2 Open-Environment Learning

Most current methods of open-environment learning are designed for open-set
recognition [6, 7], which is usually used to detect whether the sample belongs
to the seen categories or not. While some methods have succeeded in open-set
3D multi-modal retrieval [8], the complexity and inherent disparities between
different modalities still present considerable challenges in open-set cross-
modal retrieval.

##  3 Methodology

###  3.1 Problem Setup

Given $N$ 3D objects $\\{o_{i}\\}=\\{o^{r}_{i}\\}_{r=1}^{M}$ represented by
$M$ modalities, the goal of 3D cross-modal retrieval (3DCMR) is to develop a
model using the training set
$\mathcal{D}_{trn}=\\{(o_{i},y_{i})\\}^{L}_{i=1}$, and then employ it to
identify similar objects from the query set
$\mathcal{D}_{q}=\\{(o^{q}_{i},\hat{y}_{i})\\}^{Q}_{i=1}$ to the target set
$\mathcal{D}_{t}=\\{(o^{t}_{i},\hat{y}_{i})\\}^{T}_{i=1}$, where the query and
target objects are represented in different modalities ($t\neq q$). Here, $L$,
$Q$, and $T$ denote the number of samples in the training, query, and target
set, respectively. The query set and target set are from testing set
$\mathcal{D}_{tes}=\\{\mathcal{D}_{q},\mathcal{D}_{t}\\}$,
$y_{i}\in\mathcal{Y}=\\{c_{j}\\}^{Y}_{j=1}$ and
$\hat{y}_{i}\in\mathcal{\hat{Y}}=\\{\hat{c}_{j}\\}^{\hat{Y}}_{j=1}$ denote the
category space of the training set and target set, where $Y$ and $\hat{Y}$ are
the numbers of categories in the training and testing sets, respectively.

Traditional 3DCMR task is based on the close-set assumption, which means that
in the testing set $\mathcal{D}_{tes}=\\{\mathcal{D}_{q},\mathcal{D}_{t}\\}$,
all categories of objects in the testing set have been seen in the training
set $\mathcal{D}_{trn}$. The category spaces of the training set and testing
set are the same indicating $\mathcal{Y}=\mathcal{\hat{Y}}$.

Different from the traditional closed-set assumption, we consider a more
practical condition that the testing set consists entirely of categories not
encountered in the training set. We term this task as Open-Set 3D Cross-Modal
Retrieval. Under this circumstance, $\mathcal{D}_{trn}$ and
$\mathcal{D}_{tes}$ have their individual distributions, which means
$\mathcal{Y}\neq\mathcal{\hat{Y}}$. This task seeks to minimize the expected
risk:

$\begin{split}f^{*}=\mathop{argmin}\limits_{f\in\mathcal{H}}&\mathbb{E}_{(D_{i},D_{j})\sim(\mathcal{D}_{q},\mathcal{D}_{t})}\left[\mathbb{I}(\hat{y}_{i}\neq\hat{y}_{j})e^{-\mathbb{D}(f(o^{q}_{i}),f(o^{t}_{j}))}\right.\\\
&\left.+\mathbb{I}(\hat{y}_{i}=\hat{y}_{j})(1-e^{-\mathbb{D}(f(o^{q}_{i}),f(o^{t}_{j}))})\right]\end{split},$
(1)

where $D_{i}=(o^{q}_{i},\hat{y}_{i})$ and $D_{j}=(o^{t}_{j},\hat{y}_{j})$ are
samples drawn from the query set $\mathcal{D}_{q}$ and target set
$\mathcal{D}_{t}$. $\mathbb{I}(\cdot)$ is the indicator function, which
returns $1$ if the expression is true and $0$ otherwise.
$f:=o^{r}_{i}\rightarrow z_{i}$ is the function that maps the 3D object
$o^{r}_{i}$ represented in different modalities into the same embedding
$z_{i}\in\mathbb{R}^{d}$. $\mathcal{H}$ is the hypothesis space of function
$f(\cdot)$ and $\mathbb{D}(z_{i},z_{j})$ is a distance metric function.

Fig. 2: An overview of the proposed structure-aware residual-center
representation framework (SRCR). Our framework comprises two main modules:
Residual-Center Embedding (RCE) and Hierarchical Structure Learning (HSL),
which are used for residual embedding generation and structure-aware feature
alignment, respectively.

###  3.2 Framework Architecture

The architecture of SRCR, as illustrated in Fig. 2, is composed of two
modules: Residual-Center Embedding (RCE) and Hierarchical Structure Learning
(HSL). Given basic features of different modalities extracted by common-used
networks. The Residual-Center Embedding module is designed to generate the
residual center embeddings for each object, rather than directly mapping them
to the modality or category center. Then, in the Hierarchical Structure
Learning stage, the hierarchical hypergraph structure is constructed based on
the inter-modality, intra-object, and implicit-category correlations. Guided
by this structure, the combination of hypergraph convolution and memory bank
effectively leverages the high-order correlations between seen and unseen
categories and different modalities. Finally, the aligned embedding of each
modality is generated for the cross-modal retrieval or other downstream tasks.

###  3.3 Residual-Center Embedding

In order to improve category generalization while projecting into the unified
space, the residual-center embedding module is developed. Specifically, the
RCE consists of two nested auto-encoders and takes the basic features of
different modalities as input. The outer auto-encoder $\mathcal{A}^{r}_{out}$
encodes the basic features into a latent space and pulls them together into a
unified embedding. The inner auto-encoder $\mathcal{A}^{r}_{in}$ encodes the
modality embeddings from the outer auto-encoder to the residual space, which
transforms the embedding between the modality space and the unified space.

####  3.3.1 Residual Learning

Given $N$ 3D objects $\\{o_{i}\\}^{N}_{i=1}$ and basic features
$\\{f^{r}_{i}\\}^{M}_{r=1}$ of each object. As shown in Fig. 2, the outer
auto-encoder
$\mathcal{A}^{r}_{out}=\\{\Psi^{r}_{out},\Phi^{r}_{out}\\}_{r=1}^{M}$
compresses the basic features into a unified space $\mathbb{S}_{u}$ and does
the reverse reconstruction, for better representation, which can be defined as
follows:

$\left\\{\begin{aligned} u^{r}_{i}=&\Psi^{r}_{out}(f^{r}_{i})\\\
\hat{f}^{r}_{i}=&\Phi^{r}_{out}(u^{r}_{i})\\\ \end{aligned}\right.,$ (2)

where $\Psi^{r}_{out}:=\mathbb{S}_{r}\rightarrow\mathbb{S}_{u}$ is the encoder
that maps the $r$-th modality space $\mathbb{S}_{r}$ into the unified space
$\mathbb{S}_{u}$, and
$\Phi^{r}_{out}:=\mathbb{S}_{u}\rightarrow\mathbb{S}_{r}$ is the decoder that
maps the features from unified space $\mathbb{S}_{u}$ back to
$\mathbb{S}_{r}$. $u^{r}_{i}\in\mathbb{R}^{d_{u}}$ and
$\hat{f}^{n}_{i}\in\mathbb{R}^{d_{0}}$ denotes the compressed features and
reconstruction features of each modality.

An aggregation function $\mathcal{U}$ are adopted to generate the unified
embedding
$u_{i}=\mathcal{U}(\\{u^{r}_{i}\\}_{r=1}^{M}),u_{i}\in\mathbb{R}^{d_{u}}$ of
object $o_{i}$, which are treated as the semantic center. The inner auto-
encoder $\mathcal{A}^{r}_{in}=\\{\Psi^{r}_{in},\Phi^{r}_{in}\\}_{r=1}^{M}$
aims to generate the semantic center $u_{i}$ of each object and the residual-
center embedding between each modality $\hat{f}^{n}_{i}$. Specifically, we
construct learnable parameter encodings $e^{r}\in\mathbb{R}^{d_{u}}$ for each
modality, and $\mathcal{A}^{r}_{in}$ takes them aligned with $\hat{f}^{r}_{i}$
to get middle embedding:

$\left\\{\begin{aligned}
&\delta^{r}_{i}=\Psi^{r}_{in}(\hat{f}^{r}_{i}+e^{r})\\\
&c^{r}_{i}=\Phi^{r}_{in}(\hat{f}^{r}_{i}+\delta^{r}_{i})\end{aligned}\right.,$
(3)

where $\Psi^{r}_{in}$ and $\Phi^{r}_{in}$ denote the encoder and decoder map
function between modality and residual space, $\delta^{r}_{i}$ denotes the
residual-center embedding of $r$-th modality of object $o_{i}$.

####  3.3.2 Loss Function for RCE

To get a better representation of modality embedding and residual-center
embedding, the Residual-Center Loss $\mathcal{L}_{rc}$ and Cross-
Reconstruction Loss $\mathcal{L}_{cr}$ are adopted here. The constraints for
each loss are derived from different modalities data of the same object,
rather than class labels.

Residual-Center Loss. The loss is designed to pull the distance among the
estimated embeddings $\\{u^{r}_{i}\\}_{r=1}^{M}$ from different modalities
closer, which is defined as follows:

$\mathcal{L}_{rc}=\frac{1}{M}\sum\nolimits_{r=1}^{M}(\lVert
u^{r}_{i}-u_{i}\rVert_{2}+\lVert c^{r}_{i}-u_{i}\rVert_{2}),$ (4)

where $\lVert\cdot\rVert_{2}$ is the $\mathcal{L}_{2}$ norm function.

Cross-Reconstrution Loss. To promote the generalization ability of the RCE, we
propose the cross-reconstruction Loss. Motivated by [9], the
$\mathcal{L}_{cr}$ are defined as the distance of results by exchanging
decoder in the inner auto-encoder.

$\mathcal{L}_{cr}=\frac{1}{M(M-1)}\sum_{k=1}^{M}\sum_{l\neq
k}(\lVert\Phi^{l}_{in}(\Psi^{k}_{in}(\hat{f}^{k}_{i}+\delta^{k}_{i}))-c^{k}_{i}\rVert_{2}),$
(5)

where $\lVert\cdot\rVert_{2}$ is the $\mathcal{L}_{2}$ norm function.

Joint Optimization. In the residual-center embedding stage, the overall loss
function is given combined Eq. 4 and Eq. 5:

$\mathcal{L}_{RCE}=\alpha\mathcal{L}_{rc}+(1-\alpha)\mathcal{L}_{cr},$ (6)

where $\alpha$ is the hyper-parameter for trade-off.

###  3.4 Hierarchical Structure Learning

Although the RCE module generates the residual-center embedding of different
modalities, the distribution gaps between seen and unseen categories still
affect the retrieval under the open-set setting. As shown in Fig. 2, we
proposed the hierarchical structure learning module for generalization across
modalities and categories. Specifically, the hierarchical hypergraph is
constructed to capture the hierarchical correlations. Then, the hypergraph
convolution and memory bank are adopted for embedding smoothing and
distilling.

####  3.4.1 Hierarchical Hypergraph Construction

We adopt a hierarchical hypergraph to take the most advantage of high-order
correlations between modalities, objects, and categories. A hypergraph can be
represented as $\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$, where
$\mathcal{V}$ and $\mathcal{E}$ are the vertex set and the hyperedge set,
respectively.

Heterogeneous Vertices. For the vertices, we first construct the centralized
embedding of each modality by aligning $\hat{f}^{r}_{i}$ with the residual
feature $\delta^{r}_{i}$, then we treat centralized embeddings of each object
as the heterogeneous vertices.

$\left.\begin{aligned}
&v^{r}_{i}=\tau\hat{f}^{r}_{i}+(1-\tau)\delta^{r}_{i}\\\
&\mathcal{V}=\bigcup\nolimits_{r=1}^{M}\\{v^{r}_{i}\\}_{i=1}^{N}\end{aligned}\right.,$
(7)

where $\tau$ denotes the hyper-parameters for centralized fusion,
$\hat{f}^{r}_{i}$ and $\delta^{r}_{i}$ denote the modality embedding and
residual-center embedding of object $o_{i}$ in $r$-th modality, $M$ and $N$
denote the number of modalities and object samples.

Hierarchical Hyperedges. The hierarchical hypergraph is composed of three
types of hyperedges, including inter-modality, intra-object, and implicit-
category, which can be defined as follows:

$\left.\begin{aligned} &\mathcal{E}_{m}=\\{\mathcal{M}_{v}(r)\mid
r\in\\{1,\cdots,M\\}\\}\\\ &\mathcal{E}_{o}=\\{\mathcal{N}_{v}(i)\mid
i\in\\{1,\cdots,N\\}\\}\\\
&\mathcal{E}_{c}=\\{\mathcal{N}_{\mathrm{KNN}_{k}}(v)\mid
v\in\mathcal{V}\\}\end{aligned}\right.,$ (8)

where $\mathcal{M}_{v}(r)$ denotes the vertex subset that belong to the same
modality $r$, $\mathcal{N}_{v}(i)$ denotes the vertex subset that belong to
the same object $o_{i}$, and $\mathcal{N}_{\mathrm{KNN}_{k}}(v)$ denotes the
k-nearest neighbors of vertex $v$.

In this way, $M$ inter-modality hyperedges, $N$ intra-object hyperedges and
$M\times N$ implicit-category hyperedges are constructed. Finally, we combine
these three hyperedge groups to get the total hyperedges:
$\mathcal{E}=\mathcal{E}_{m}\cup\mathcal{E}_{o}\cup\mathcal{E}_{c}$.

####  3.4.2 Hypergraph Convolution and Alignment

To leverage the high-order correlation between objects and modalities, we
utilize the hypergraph convolution [10] to smooth the embedding under the
hierarchical structure, which is formulated as:

$\tilde{\mathbf{V}}=\sigma\left(\mathbf{D}^{-\frac{1}{2}}_{v}\mathbf{H}\mathbf{W}\mathbf{D}^{-1}_{e}\mathbf{H}^{\top}\mathbf{D}^{-\frac{1}{2}}_{v}\mathbf{V}\mathbf{\Theta}\right),$
(9)

where $\mathbf{H}$ denotes the incidence matrix of the hypergraph.
$\mathbf{D}_{v}$ and $\mathbf{D}_{e}$ are the diagonal degree matrices for
vertex and hyperedge, respectively.

After obtaining the structure-aware embedding $\tilde{v}^{r}_{i}$ of the 3D
object $o^{r}_{i}$, we construct a memory bank $\mathcal{B}$ that contains $L$
invariant memory anchors. Following [8], we compute the activation score for
each memory anchor in the memory bank by
$s^{r}_{ij}=\mathcal{D}_{m}(\tilde{v}^{r}_{i},a^{r}_{j})$, where $a^{r}_{j}$
denotes the anchor and $D_{m}(\cdot,\cdot)$ denotes the distance metric
function. We rebuild the aligned embedding of each object by
$z_{i}=\sum\nolimits_{j=1}^{L}\hat{s}^{r}_{ij}a^{r}_{j},z^{r}_{i}\in\mathbb{R}^{d_{z}}$,
where $\hat{s}^{r}_{ij}$ denotes the normlization of activation score.

####  3.4.3 Loss Function for HSL

To train the hypergraph convolution and learnable memory anchors under
hierarchical structure, we adopt the self-supervised Memory Reconstruction
Loss $\mathcal{L}_{mr}$ for HSL:

$\mathcal{L}_{HSL}=\mathcal{L}_{mr}=\big{\lVert}\tilde{v}^{r}_{i}-z^{r}_{i}\big{\rVert}_{2},$
(10)

where $\lVert\cdot\rVert_{2}$ is the $\mathcal{L}_{2}$ norm function.

Table 1: Experimental results of Image2Point retrieval on the OCAB, OCNT,
OCES, and OCMN datasets.

Image2Point | OCAB | OCNT | OCES | OCMN
---|---|---|---|---
mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$ | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$ | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$ | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$
SDML | 0.1489 | 0.1061 | 0.8824 | 0.0465 | 0.0316 | 0.9657 | 0.0942 | 0.0442 | 0.9486 | 0.0578 | 0.0248 | 0.9735
CMCL | 0.1702 | 0.1520 | 0.8565 | 0.0623 | 0.0332 | 0.9665 | 0.0991 | 0.0477 | 0.9444 | 0.1175 | 0.0917 | 0.9001
MMSAE | 0.1218 | 0.0802 | 0.9093 | 0.0410 | 0.0191 | 0.9817 | 0.0810 | 0.0362 | 0.9567 | 0.0571 | 0.0235 | 0.9746
PROSER | 0.1119 | 0.0446 | 0.9386 | 0.0426 | 0.0171 | 0.9752 | 0.0968 | 0.0402 | 0.9641 | 0.0523 | 0.0133 | 0.9806
HGM2R | 0.1367 | 0.0925 | 0.8978 | 0.1812 | 0.1072 | 0.8184 | 0.2184 | 0.1126 | 0.8215 | 0.0988 | 0.0789 | 0.9282
Ours | 0.2220 | 0.1714 | 0.7947 | 0.2861 | 0.1585 | 0.7292 | 0.4004 | 0.1835 | 0.6378 | 0.1549 | 0.1488 | 0.8625

Table 2: Experimental results of Point2Image retrieval on the OCAB, OCNT,
OCES, and OCMN datasets.

Point2Image | OCAB | OCNT | OCES | OCMN
---|---|---|---|---
mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$ | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$ | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$ | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$
SDML | 0.1636 | 0.1367 | 0.8629 | 0.0393 | 0.0191 | 0.9820 | 0.0811 | 0.0413 | 0.9654 | 0.0682 | 0.0475 | 0.9512
CMCL | 0.1628 | 0.1343 | 0.8594 | 0.0394 | 0.0203 | 0.9786 | 0.0815 | 0.0420 | 0.9598 | 0.1219 | 0.1116 | 0.9074
MMSAE | 0.0821 | 0.0657 | 0.9211 | 0.0347 | 0.0191 | 0.9846 | 0.0891 | 0.0413 | 0.9496 | 0.0460 | 0.0278 | 0.9699
PROSER | 0.0708 | 0.0555 | 0.9579 | 0.0387 | 0.0182 | 0.9834 | 0.0885 | 0.0406 | 0.9492 | 0.0693 | 0.0336 | 0.9653
HGM2R | 0.1553 | 0.1613 | 0.8593 | 0.1452 | 0.0742 | 0.8996 | 0.2260 | 0.1265 | 0.8430 | 0.1006 | 0.0747 | 0.9305
Ours | 0.3013 | 0.3202 | 0.7048 | 0.2811 | 0.1377 | 0.7316 | 0.4471 | 0.1914 | 0.5936 | 0.1277 | 0.1117 | 0.8994

##  4 Experiments

###  4.1 Experimental Settings

OCMR Datasets. We generate four open-set 3D cross-modal retrieval (OCMR)
datasets, including OCAB, OCNT, OCES, OCMN, based on the public datasets ABO
[11], NTU [12], ESB [13], and ModelNet40 [14], respectively. These datasets
are split into seen and unseen categories, each object has three modalities
including multi-view, voxel, and point cloud.

Implemental Details. In our experiment, we choose all three modalities of 3D
objects. We set $\alpha=0.5$ for the hyper-parameters in Eq. 6, and
$\tau=0.75$ in Eq. 7. The two modules are trained separately with 40 epochs on
learning rate $lr=0.1$ and 120 epochs on $lr=0.001$, the random seed is fixed
as 2022 for all experiments.

###  4.2 Retrieval Performance

Compared Methods. As no methods are specifically designed for the open-set 3D
cross-modal retrieval to date, we refine the current state-of-the-art methods
from two tasks for comparison: close-set 3D cross-modal retrieval (SDML [15],
CMCL [1], MMSAE [16]), and open-set multi-modal recognition or retrieval
(PROSER [17], HGM2R [8]).

Evaluation Metrics. For a fair comparison, we employ the commonly used
retrieval metric, including Mean Average Precision (mAP), Normalized
Discounted Cumulative Gain (NDCG), Average Normalized Modified Retrieval Rank
(ANMRR), and the Precision-Recall Curve (PR-Curve). For the mAP and NDCG
metric, higher scores are better. For the ANMRR metric, the lower score is
better. We construct $6$ query-target types for cross-modal retrieval
according to these three modalities, including Image2Point (I2P), Image2Voxel
(I2V), Point2Image (P2I), Point2Voxel (P2V), Voxel2Image (V2I), and
Voxel2Point (V2P).

Comparison Analysis. We evaluate open-set 3D cross-modal retrieval results on
four datasets, quantitative results of SRCR framework and other state-of-the-
art methods are provided in Tab. 1 and Tab. 2. Results show that the proposed
method outperforms the other methods on all four datasets. We also provide the
Precision-Recall (PR) Curve to evaluate the performance of the proposed SRCR
framework and other compared methods, as illustrated in Fig. 3. The larger
area below the curve indicates better performance. From the results, we can
observe that our method outperforms all other compared methods. The better
performance indicates that by the residual-center embedding and hierarchical
structure learning, the proposed method has the capability to overcome
modality gaps while understanding the open-set categories.

Table 3: Ablation studies on OCNT dataset.

|  | mAP$\uparrow$ | NDCG$\uparrow$ | ANMRR$\downarrow$
---|---|---|---|---
On RCE | Direct Center | 0.0362 | 0.0194 | 0.9801
Category Center | 0.0433 | 0.0230 | 0.9772
On HSL | HSL w/o $\mathcal{E}_{m}$ | 0.1511 | 0.1081 | 0.8626
HSL w/o $\mathcal{E}_{m}$&$\mathcal{E}_{o}$ | 0.1474 | 0.1084 | 0.8636
GCN-based HSL | 0.2575 | 0.1640 | 0.7374
MLP-based HSL | 0.1553 | 0.1101 | 0.8380
| RCE+HSL | 0.2861 | 0.1585 | 0.7292

(a) PR Curve on OCAB.

(b) PR Curve on OCNT.

(c) PR Curve on OCES.

(d) PR Curve on OCMN.

Fig. 3:  The precision-recall curves comparison of Image2Point retrieval on
four datasets, respectively.

###  4.3 Ablation Study

We conduct ablation studies to verify the effectiveness of the proposed
modules. For the residual-center embedding module, we compare the proposed RCE
with the Direct Center and Category Center. The Direct Center denotes the
network that use auto-encoder to generate the center embedding directly
instead of residually, and Category Center denotes the network that generates
the category center rather than semantic center of each object. During the
ablation of the hierarchical structure learning module, we compared the
proposed HSL with naive structures (HSL w/o $\mathcal{E}_{m}$ and HSL w/o
$\mathcal{E}_{m}$&$\mathcal{E}_{o}$), where “w/o” denotes “without”. We also
replace the hypergraph-based correlation learning with MLP and GCN. As shown
in Tab. 3, the combination of RCE and HSL yields the best performance,
substituting either the embedding or the learning approach is observed to lead
to a decline in performance. These results demonstrate the proposed modules
can effectively obtain the center-embedding of objects and generalize it to
unseen categories.

##  5 Conclusion

In this paper, we propose the Structure-Aware Residual-Center Representation
(SRCR) framework for self-supervised open-set 3D cross-modal retrieval. We
utilize the Residual-Center Embedding (RCE) for each object by nested auto-
encoders to address the center deviation due to category distribution
differences, rather than directly mapping them to the modality or category
centerS. Besides, we construct a heterogeneous hypergraph structure based on
hierarchical inter-modality, intra-object, and implicit-category correlations,
and perform the Hierarchical Structure Learning (HSL) approach to leverage the
high-order correlations among objects for generalization. Extensive
experiments and ablation studies on four benchmarks demonstrate the
superiority of our proposed framework compared to state-of-the-art methods.

## References

  * [1] Longlong Jing, Elahe Vahdani, Jiaxing Tan, and Yingli Tian,  “Cross-modal Center Loss for 3D Cross-Modal Retrieval,”  in CVPR, 2021, pp. 3142–3151.
  * [2] Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen,  “Adversarial Cross-modal Retrieval,”  in ACMMM, 2017, pp. 154–162.
  * [3] Yanglin Feng, Hongyuan Zhu, Dezhong Peng, Xi Peng, and Peng Hu,  “RONO: Robust Discriminative Learning With Noisy Labels for 2D-3D Cross-Modal Retrieval,”  in CVPR, 2023, pp. 11610–11619.
  * [4] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu,  “Deep Canonical Correlation Analysis,”  in ICML. PMLR, 2013, pp. 1247–1255.
  * [5] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes,  “On Deep Multi-View Representation Learning,”  in ICML. PMLR, 2015, pp. 1083–1092.
  * [6] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman,  “Open-Set Recognition: A Good Closed-Set Classifier is All You Need?,”  arXiv preprint arXiv:2110.06207, 2021.
  * [7] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian,  “Adversarial Reciprocal Points Learning for Open Set Recognition,”  TPAMI, vol. 44, no. 11, pp. 8065–8081, 2021.
  * [8] Yifan Feng, Shuyi Ji, Yu-Shen Liu, Shaoyi Du, Qionghai Dai, and Yue Gao,  “Hypergraph-based Multi-Modal Representation for Open-Set 3D Object Retrieval,”  TPAMI, , no. 01, pp. 1–18, 2023.
  * [9] Fangxiang Feng, Xiaojie Wang, and Ruifan Li,  “Cross-Modal Retrieval with Correspondence AutoEncoder,”  in ACMMM, 2014, pp. 7–16.
  * [10] Yue Gao, Yifan Feng, Shuyi Ji, and Rongrong Ji,  “HGNN+: General Hypergraph Neural Networks,”  TPAMI, vol. 45, no. 3, pp. 3181–3199, 2022.
  * [11] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al.,  “ABO: Dataset and Benchmarks for Real-World 3D Object Understanding,”  in CVPR, 2022, pp. 21126–21136.
  * [12] Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung,  “On Visual Similarity based 3D Model Retrieval,”  in Computer graphics forum. Wiley Online Library, 2003, pp. 223–232.
  * [13] Subramaniam Jayanti, Yagnanarayanan Kalyanaraman, Natraj Iyer, and Karthik Ramani,  “Developing an Engineering Shape Benchmark for CAD Models,”  Computer-Aided Design, vol. 38, no. 9, pp. 939–953, 2006.
  * [14] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao,  “3D Shapenets: A Deep Representation for Volumetric Shapes,”  in CVPR, 2015, pp. 1912–1920.
  * [15] Peng Hu, Liangli Zhen, Dezhong Peng, and Pei Liu,  “Scalable Deep Multimodal Learning for Cross-Modal Retrieval,”  in SIGIR, 2019, pp. 635–644.
  * [16] Yiling Wu, Shuhui Wang, and Qingming Huang,  “Multi-Modal Semantic AutoEncoder for Cross-Modal Retrieval,”  Neurocomputing, vol. 331, pp. 165–175, 2019.
  * [17] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan,  “Learning Placeholders for Open-Set Recognition,”  in CVPR, 2021, pp. 4401–4410.